A Proposed Standard for the Reporting of Structural Equation Models With Ordinal Variables: Why Ordinal Data Should be Treated With Extra Care?
Gabriel Chun-Yeung Lee
Educational researchers, as well as researchers in other disciplines, often work with ordinal data, such as Likert item responses and test item scores.
- Pub. date: August 15, 2025
- Online Pub. date: August 09, 2025
- Pages: 423-442
- 23 Downloads
- 51 Views
- 0 Citations
Abstract:
Educational researchers, as well as researchers in other disciplines, often work with ordinal data, such as Likert item responses and test item scores. Critical questions arise when researchers attempt to implement statistical models to analyse ordinal data, given that many statistical techniques assume the data analysed to be continuous. Could ordinal data be treated as continuous data, that is, assuming the ordinal data to be continuous and then applying statistical techniques as if analysing continuous data? Why and why not? Focusing on structural equation models (SEMs), particularly confirmatory factor analysis (CFA), this article discusses an ongoing debate on the treatment of ordinal data and reports a short review on the practices of conducting and reporting SEMs, in the context of mathematics education research. The author reviewed 70 publications in mathematics education research that reported a study involving SEMs to analyse ordinal data, but less than half discussed how data were treated or guided readers through the analysis; it is therefore harder to repeat such an analysis and evaluate the results. This article invites methodological discussions on SEMs with ordinal variables in the practices of educational research. Subsequently, a standard for reporting SEMs with ordinal data is proposed, followed by an example. This standard contributes to educational research by enabling researchers (self and others) to evaluate SEMs reported. The example demonstrates, using real-life research data, how two different approaches for analysing ordinal data (as continuous or as a product of discretisation from some continuous distributions) can lead to results that disagree.
Keywords: Confirmatory factor analysis, Likert items, ordinal data, structural equation modelling.
0
References
Alabi, A. T., & Jelili, M. O. (2023). Clarifying likert scale misconceptions for improved application in urban studies. Quality and Quantity, 57, 1337-1350. https://doi.org/10.1007/s11135-022-01415-8
Almeida, D. (2000). A survey of mathematics undergraduates’ interaction with proof: Some implications for mathematics education. International Journal of Mathematical Education in Science and Technology, 31(6), 869-890. https://doi.org/10.1080/00207390050203360
American Statistical Association. (2022). Ethical guidelines for statistical practice. American Statistical Association. https://www.amstat.org/your-career/ethical-guidelines-for-statistical-practice
Andrews, P., & Diego-Mantecón, J. (2015). Instrument adaptation in cross-cultural studies of students’ mathematics-related beliefs: Learning from healthcare research. Compare: A Journal of Comparative and International Education, 45(4), 545-567. https://doi.org/10.1080/03057925.2014.884346
Brandenburg, N. (2024). Factor retention in ordered categorical variables: Benefits and costs of polychoric correlations in eigenvalue-based testing. Behavior Research Methods, 56, 7241-7260. https://doi.org/10.3758/s13428-024-02417-0
Brown, T. A. (2006). Confirmatory factor analysis for applied research. The Guilford Press.
Browne, M. W. (1984). Asymptotically distribution-free methods for the analysis of covariance structures. British Journal of Mathematical and Statistical Psychology, 37(1), 62-83. https://doi.org/10.1111/j.2044-8317.1984.tb00789.x
Cañete-Massé, C., Carbó-Carreté, M., Figueroa-Jiménez, M. D., Oviedo, G. R., Guerra-Balic, M., Javierre, C., Peró-Cebollero, M., & Guàrdia-Olmos, J. (2022). Confirmatory factor analysis with missing data in a small sample: Cognitive reserve in people with Down Syndrome. Quality and Quantity, 56, 3363-3377. https://doi.org/10.1007/s11135-021-01264-x
Flora, D. B., & Curran, P. J. (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9(4), 466-491. https://doi.org/10.1037/1082-989X.9.4.466
Foldnes, N., & Grønneberg, S. (2019). On identification and non-normal simulation in ordinal covariance and item response models. Psychometrika, 84(4), 1000-1017. https://doi.org/10.1007/s11336-019-09688-z
Foldnes, N., & Grønneberg, S. (2020). Pernicious polychorics: The impact and detection of underlying non-normality. Structural Equation Modeling: A Multidisciplinary Journal, 27(4), 525-543. https://doi.org/10.1080/10705511.2019.1673168
Foldnes, N., & Grønneberg, S. (2022). The sensitivity of structural equation modeling with ordinal data to underlying non-normality and observed distributional forms. Psychological Methods, 27(4), 541-567. https://doi.org/10.1037/met0000385
Frasier, B. J. (2010). Secondary school mathematics teachers’ conceptions of proof (Publication No. 3417066) [Doctoral dissertation, University of Massachusetts Lowell]. ProQuest Dissertations & Theses Global.
Garson, G. D. (2015). Structural equation modeling. Statistical Associates Publishing.
Geisler, S., Rolka, K., & Rach, S. (2023). Development of affect at the transition to university mathematics and its relation to dropout - identifying related learning situations and deriving possible support measures. Educational Studies in Mathematics, 113, 35-56. https://doi.org/10.1007/s10649-022-10200-1
Grønneberg, S., & Foldnes, N. (2024). Factor analyzing ordinal items requires substantive knowledge of response marginals. Psychological Methods, 29(1), 65-87. https://doi.org/10.1037/met0000495
Hair, J. F., Jr., Hult, G. T. M., Ringle, C. M., Sarstedt, M., Danks, N. P., & Ray, S. (2021). Partial least squares structural equation modeling (PLS-SEM) using R. Springer. https://doi.org/10.1007/978-3-030-80519-7
Holgado-Tello, F. P., Chacón-Moscoso, S., Barbero-García, I., & Vila-Abad, E. (2010). Polychoric versus Pearson correlations in exploratory and confirmatory factor analysis of ordinal variables. Quality and Quantity, 44, 153-166. https://doi.org/10.1007/s11135-008-9190-y
Jackson, D. L., Gillaspy, J. A., Jr., & Purc-Stephenson, R. (2009). Reporting practices in confirmatory factor analysis: An overview and some recommendations. Psychological Methods, 14(1), 6-23. https://doi.org/10.1037/a0014694
Jöreskog, K. G. (1967). Some contributions to maximum likelihood factor analysis. Psychometrika, 32(4), 443-482. https://doi.org/10.1007/BF02289658
Jöreskog, K. G. (1994). Structural equation modeling with ordinal variables. Multivariate Analysis and Its Applications: IMS Lecture Notes - Monograph Series, 24, 297-310. https://doi.org/10.1214/lnms/1215463803
Keçeli-Bozdağ, S., Uğurel, I., & Bukova-Güzel, E. (2015). Development of attitude scale towards proof and proving: The case of mathematics student teachers. Kastamonu Education Journal/Kastamonu Eğitim Dergisi, 23(4), 1585-1600. http://bit.ly/3IVbbcl
Khine, M. S. (Ed.). (2013). Application of structural equation modeling in educational research and practice. Sense Publishers. https://doi.org/10.1007/978-94-6209-332-4.
Kline, R. B. (2016). Principles and practice of structural equation modeling (4th ed.). The Guilford Press.
Knuth, E. J. (2002). Secondary school mathematics teachers’ conceptions of proof. Journal for Research in Mathematics Education, 33(5), 379-405. https://doi.org/10.2307/4149959
Kolbe, L., Oort, F., & Jak, S. (2021). Bivariate distributions underlying responses to ordinal variables. Psych, 3(4), 562-578. https://doi.org/10.3390/psych3040037
Kotelawala, U. M. (2007). Exploring teachers’ attitudes and beliefs about proving in the mathematics classroom (Publication Number 3266716) [Doctoral dissertation, Columbia University]. ProQuest Dissertations & Theses Global.
LeBeau, B., Ellison, S., & Aloe, A. M. (2021). Reproducible analyses in education research. Review of Research in Education, 45(1), 195-222. https://doi.org/10.3102/0091732X20985076
Lee, G. C.-Y. (2022). Hong Kong preservice teachers’ beliefs and attitudes towards teaching proof in school mathematics: A design-based research [Doctoral dissertation, University of Oxford]. Oxford University Research Archive. http://bit.ly/45ieKB5
Lenz, K., Reinhold, F., & Wittmann, G. (2024). Topic specificity of students’ conceptual and procedual fraction knowledge and its impact on errors. Research in Mathematics Education, 26(1), 45-69. https://doi.org/10.1080/14794802.2022.2135132
Li, C.-H. (2016a). Confirmatory factor analysis with ordinal data: Comparing robust maximum likelihood and diagonally weighted least squares. Behavior Research Methods, 48, 936-949. https://doi.org/10.3758/s13428-015-0619-7
Li, C.-H. (2016b). The performance of ML, DWLS, and ULS estimation with robust corrections in structural equation models with ordinal variables. Psychological Methods, 21(3), 369-387. https://doi.org/10.1037/MET0000093
Merkle, E. C., & Rosseel, Y. (2018). blavaan: Bayesian structural equation models via parameter expansion. Journal of Statistical Software, 85(4), 1-30. https://doi.org/10.18637/jss.v085.i04
Millsap, R. E., & Yun-Tein, J. (2004). Assessing factorial invariance in ordered-categorical measures. Multivariate Behavioral Research, 39(3), 479-515. https://doi.org/10.1207/S15327906MBR3903_4
Muthén, B. (1984). A general structural equation model with dichotomous, ordered categorical, and continuous latent variable indicators. Psychometrika, 49(1), 115-132. https://doi.org/10.1007/BF02294210
Narayanan, A. (2012). A review of eight software packages for structural equation modeling. Statistical Computing Software Reviews, 66(2), 129-138. https://doi.org/10.1080/00031305.2012.708641
Nyaumwe, L., & Buzuzi, G. (2007). Teachers’ attitudes towards proof of mathematical results in the secondary school curriculum: The case of Zimbabwe. Mathematics Education Research Journal, 19, 21-32. https://doi.org/10.1007/BF03217460
Olsson, U. H., Foss, T., Troye, S. V., & Howell, R. D. (2000). The performance of ML, GLS, and WLS estimation in structural equation modeling under conditions of misspecification and nonnormality. Structural Equation Modeling: A Multidisciplinary Journal, 7(4), 557-595. https://doi.org/10.1207/S15328007SEM0704_3
Putnick, D. L., & Bornstein, M. H. (2016). Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental Review, 41, 71-90. https://doi.org/10.1016/j.dr.2016.06.004
R Core Team. (2023). R: A language and environment for statistical computing (Version 4.3.1) [Computer Software]. R Foundation for Statistical Computing. https://www.R-project.org/
Rhemtulla, M., Brosseau-Liard, P. É., & Savalei, V. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17(3), 354-373. https://doi.org/10.1037/a0029315
Robitzsch, A. (2020). Why ordinal variables can (almost) always be treated as continuous variables: Clarifying assumptions of robust continuous and ordinal factor analysis estimation methods. Frontiers in Education, 5, Article 589965. https://doi.org/10.3389/feduc.2020.589965
Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1-36. https://doi.org/10.18637/jss.v048.i02
Savalei, V. (2014). Understanding robust corrections in structural equation modeling. Structural Equation Modeling: A Multidisciplinary Journal, 21(1), 149-160. https://doi.org/10.1080/10705511.2013.824793
Savalei, V. (2021). Improving fit indices in structural equation modeling with categorical data. Multivariate Behavioral Research, 56(3), 390-407. https://doi.org/10.1080/00273171.2020.1717922
Savalei, V., & Rhemtulla, M. (2013). The performance of robust test statistics with categorical data. British Journal of Mathematical and Statistical Psychology, 66(2), 201-223. https://doi.org/10.1111/j.2044-8317.2012.02049.x
Schreiber, J. B., Nora, A., Stage, F. K., Barlow, E. A., & King, J. (2006). Reporting structural equation modeling and confirmatory factor analysis results: A review. The Journal of Educational Research, 99(6), 323-338. https://doi.org/10.3200/JOER.99.6.323-338
Schumacker, R. E., & Lomax, R. G. (2004). A beginner’s guide to structural equation modeling (2nd ed.). Psychology Press. https://doi.org/10.4324/9781410610904
Stodden, V. (2015). Reproducing statistical results. Annual Review of Statistics and Its Application, 2, 1-19. https://doi.org/10.1146/annurev-statistics-010814-020127
Street, K. E. S., Malmberg, L.-E., & Stylianides, G. J. (2022). Changes in students’ self-efficacy when learning a new topic in mathematics: A micro-longitudinal study. Educational Studies in Mathematics, 111, 515-541. https://doi.org/10.1007/s10649-022-10165-1
Wright, D. B. (2003). Making friends with your data: Improving how statistics are conducted and reported. British Journal of Educational Psychology, 73(1), 123-136. https://doi.org/10.1348/000709903762869950
Wu, H., & Leung, S.-O. (2017). Can Likert scales be treated as interval scales? - A simulation study. Journal of Social Service Research, 43(4), 527-532. https://doi.org/10.1080/01488376.2017.1329775
Zengin, Y. (2017). The effects of GeoGebra software on pre-service mathematics teachers’ attitudes and views towards proof and proving. International Journal of Mathematical Education in Science and Technology, 48(7), 1002-1022. https://doi.org/10.1080/0020739X.2017.1298855
Zhang, Y., Yang, X., Sun, X., & Kaiser, G. (2023). The reciprocal relationship among Chinese senior secondary students’ intrinsic and extrinsic motivation and cognitive engagement in learning mathematics: A three-wave longitudinal study. ZDM Mathematics Education, 55, 399-412. https://doi.org/10.1007/s11858-022-01465-0