Tablet or Paper and Pen? Examining Mode Effects on German Elementary School Students’ Computation Skills with Curriculum-Based Measurements
Progress monitoring of academic achievement is an essential element to prevent learning disorders. A prominent approach is curriculum-based measuremen.
- Pub. date: November 15, 2020
- Pages: 669-680
- 1019 Downloads
- 1084 Views
- 13 Citations
Progress monitoring of academic achievement is an essential element to prevent learning disorders. A prominent approach is curriculum-based measurement (CBM). Various studies have documented positive effects of CBM on students’ achievement. Nevertheless, the use of CBM is associated with additional work for teachers. The use of tablets may be of help here. Yet, although many advantages of computer- or tablet-based assessments are being discussed in the literature (e. g. innovative item formats, adaptive testing, automated scoring and feedback), there are still concerns regarding the comparability of different assessment modes (paper-pencil vs. tablet). In the study presented, we analyze the CBM data of 98 fourth graders. They processed the exact same computation items once with paper and pen and once in a tablet application. The analyses point to comparable results in the test modes, although some significant deviations can be found at item level. In addition, the children report perceived benefits when working with the tablet.
Keywords: Progress monitoring, curriculum-based measurements, mode effects, tablet, elementary school.
References
American Educational Research Association. (2014). Standards for educational and psychological testing. AERA.
Arslan-Cansever, B. (2019). Investigation of third grade students' views on media literacy. International Journal of Educational Methodology, 5(1), 265-273. https://doi.org/10.12973/ijem.5.1.265
Bennett, R. E. (2003). Online assessment and the comparability of score meaning. ETS.
Bennett, R. E., Braswell, J., Oranje, A., Sandene, B., Kaplan, B. & Yan, F. (2008). Does it matter if I take my mathematics test on computer? A second empirical study of mode effects in NAEP. The Journal of Technology, Learning, and Assessment, 6(9), 1-38.
Blumenthal, Y, Hartke, B. & Voß, S. (2019). The role of evidence-based practice in German special education — State of research and discussion. Education Sciences, 9(2), 1-11. https://doi.org/10.3390/educsci9020106
Casale, G., Börnert-Ringleb, M., & Hillenbrand, C. (2020). Fordern auf Distanz? Sonder-pädagogische Unterstützung im Lernen und in der emotional-sozialen Entwicklung während der Schulschließungen 2020 gemäß den Regelungen der Bundesländer [Promoting at a distance? Special educational support in learning and emotional and social development during the school closures in 2020 in accordance with the regulations of the federal states]. Zeitschrift für Heilpädagogik, 71(5), 254-267.
Cozad, L. E., & Riccomini, P. J. (2016). Effects of digital-based math fluency interventions on learners with math difficulties: a review of the literature. The Journal of Special Education Apprenticeship, 5(2), 1-19.
German Federal Ministry of Education and Research (2006). IT-Ausstattung der allgemeinbildenden und berufsbildenden Schulen in Deutschland. Bestandsaufnahme 2006 und Entwicklung 2001 bis 2006 [IT-equipment in German schools. Stocktaking from 2006 and development from 2001 to 2006]. https://www.kmk.org/fileadmin/Dateien/pdf/Statistik/it-ausstattung_der_schulen.pdf
German Federal Ministry of Education and Research (2019). Wissenswertes zum digitalpakt schule [Useful information on the digital reform in schools]. https://www.bmbf.de/de/wissenswertes-zum-digitalpakt-schule-6496.php
Cayton-Hodges, G. A., Feng, G. & Pan, X. (2015). Tablet-based math assessment: What can we learn from math apps? Educational Technology & Society, 18(2), 3-20.
Clariana, R. & Wallace, P. (2002). Paper–based versus computer–based assessment: Key factors associated with the test mode effect. British Journal of Educational Technology, 33(5), 593-602. https://doi.org/10.1111/1467-8535.00294
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Erlbaum.
Deno, S. L. (1985). Curriculum-based measurement: The emerging alternative. Exceptional Children, 52(3), 219-232. https://doi.org/10.1177/001440298505200303
Deno, S. L. (2003). Developments in curriculum-based measurement. Journal of Special Education, 37(3), 184-192. https://doi.org/10.1177/00224669030370030801
Espin, C. A., Wayman, M. M., Deno, S. L., McMaster, K. L., & Rooij, M. de (2017). Data-based decision-making: Developing a method for capturing teachers’ understanding of CBM graphs. Learning Disabilities Research & Practice, 32(1), 8–21. https://doi.org/10.1111/ldrp.12123
Foegen, A., Jiban, C. & Deno, S. L. (2007). Progress monitoring measures in mathematics: A review of the literature. The Journal of Special Education, 41(2), 121-139. https://doi.org/10.1177/00224669070410020101
Foerster, N. & Souvignier, E. (2014). Learning progress assessment and goal setting: Effects on reading achievement, reading motivation and reading self-concept. Learning and Instruction, 32, 91-100. https://doi.org/10.1016/j.learninstruc.2014.02.002
Fuchs, L. S. (2004). The past, present, and future of curriculum-based measurement research. School Psychology Review, 33(2), 188-192.
Fuchs, L. S., Fuchs, D. & Hamlett, C. L. (1994). Strengthening the connection between assessment and instructional planning with expert systems. Exceptional Children, 61(2), 138-146.
Fuchs, L. S. & Fuchs, D. (1986). Effects of systematic formative evaluation: A meta-analysis. Exceptional Children, 53(3), 199-208. https://doi.org/10.1177/001440298605300301
Gebhardt, M., Heine, J.-H., Zeuch, N. & Förster, N. (2015). Lernverlaufsdiagnostik im Mathematikunterricht der zweiten Klasse: Raschanalysen und Empfehlungen zur Adaption eines Testverfahrens für den Einsatz in inklusiven Klassen [Progress monitoring for mathematics in second grade: Analysis and recommendations for the application of an diagnostic tool in inclusive classrooms]. Empirical special education/ Empirische Sonderpädagogik, 3, 206-222.
Grünke, M. (2006). Zur Effektivität von Fördermethoden bei Kindern und Jugendlichen mit Lernstörungen. Eine Synopse vorliegender Metaanalysen [Efficacy of interventions for students with learning disabilities. A synopsis of meta-analyses]. Childhood and development/ Kindheit und Entwicklung, 15(4), 239-254. https://doi.org/10.1026/0942-5403.15.4.239
Hargreaves, M., Shorrocks-Taylor, D., Swinnerton, B., Tait, K. & Threlfall, J. (2004). Computer or paper? That is the question: does the medium in which assessment questions are presented affect children’s performance in mathematics? Educational Research, 46(1), 29-42. https://doi.org/10.1080/0013188042000178809
Hartke, B. (2005). Schulische prävention – Welche maßnahmen haben sich bewährt [Prevention of learning difficulties – Which interventions stand the test]? In S. Ellinger & M. Wittrock (Hrsg.), Sonderpädagogik in der Regelschule. Konzepte, Forschung, Praxis [Special Education in Mainstream Schools. Concepts, Research, Practice] (pp. 11-37). Kohlhammer.
Hattie, J. (2013). Lernen sichtbar machen [Visible learning]. Schneider-Verlag.
Hebbecker, K. & Souvignier, E. (2018). Formatives Assessment im Leseunterricht der Grundschule – Implementation und Wirksamkeit eines modularen, materialgestützten Konzepts [Formative assessment of reading skills in primary schools – Implementation and efficacy of a modular concept]. Journal for Educational Science/ Zeitschrift für Erziehungswissenschaft, 21, 735-765. https://doi.org/10.1007/s11618-018-0834-y
Hensley, K. K. (2015). Examining the effects of paper-pencil-based and computer-based modes of assessment on mathematics curriculum-based measurement. [Doctoral dissertation, University of Iowa] Iowa Research Online. https://doi.org/10.17077/etd.ireseh1q
Hensley, K., Rankin, A., & Hosp, J. (2017). Comparing student performance on paper- and computer-based math curriculum-based measures. Assistive Technology, 29(3), 140-145. https://doi.org/10.1080/10400435.2016.1212129
Hesse, I. & Latzko, B. (2017). Diagnostik für Lehrkräfte [Assessment for teachers] (3rd ed.). Budrich.
Hosp, M. K., Hosp, J. L. & Howell, K. W. (2007). The ABC’s of CBM: A practical guide to curriculum-based measurement. Guilford.
Huber, C. & Grosche, M. (2012). Das response-to-intervention-Modell als Grundlage für einen inklusiven Paradigmenwechsel in der Sonderpädagogik [The response-to-intervention model as basis of an inclusive paradigm shift in special needs education]. Journal for Therapeutic Pedagogy/ Zeitschrift für Heilpädagogik, 8, 312-322.
Huff, K. L. & Sireci, S. G. (2001). Validity issues in computer-based testing. Educational Measurement: Issues and Practice, 20(3), 16-25. https://doi.org/10.1111/j.1745-3992.2001.tb00066.x
Jerrim, J. (2016). PISA 2012: how do results for the paper and computer tests compare? Assessment in Education: Principles, Policy & Practice, 23(4), 495-518. https://doi.org/10.1080/0969594X.2016.1147420
Jung, P.-G., McMaster, K. L., Kunkel, A. K., Shin, J. & Stecker, P M. (2018). Effects of data-based individualization for students with intensive learning needs: A meta-analysis. Learning Disabilities Research & Practice, 33(3), 144-155. https://doi.org/10.1111/ldrp.12172
Kesici, A., & Bindak, R. (2019). Does mathematics anxiety have any impact on secondary school pupils' friend choices? International Journal of Educational Methodology, 5(1), 109-116. https://doi.org/10.12973/ijem.5.1.123
Kingston, N. & Nash, B. (2011). Formative assessment: A meta‐analysis and a call for research. Educational Measurement: Issues and Practice, 30(4), 28-37. https://doi.org/10.1111/j.1745-3992.2011.00220.x
Klauer, K. J. (2014). Formative Leistungsdiagnostik: Historischer hintergrund und weiterentwicklung zur lernverlaufsdiagnostik [Formative achievement assessment: Historical background and evolution into progress monitoring]. In M. Hasselhorn, W. Schneider & U. Trautwein, U. (Eds.), Lernverlaufsdiagnostik [Progress monitoring] (Vol. 12., pp. 1-17). Hogrefe.
Kretschmann, R. (2009). Prävention [Prevention]. In B. Ahrbeck & M. Willmann (Eds.), Pädagogik bei Verhaltensstörungen [Pedagogy for behavioral disorders] (pp. 237-246). Kohlhammer.
Lottridge, S. M., Nicewander, W. A. & Mitzel, H. C. (2011) A comparison of paper and online tests using a within-subjects design and propensity score matching study. Multivariate Behavioral Research, 46(3), 544-566. https://doi.org/10.1080/00273171.2011.569408
McMaster, K. & Espin, C. (2007). Technical features of curriculum-based measurement in writing: A literature review. Journal of Special Education, 41 (2), 68-84. https://doi.org/10.1177/00224669070410020301
Nobel, K., Barwasser, A., Melzer, C., & Grünke, M. (2020). Webbasiertes distanzlernen - Erfahrungen im rahmen einer pilotstudie während der Covid-19-schulschließungen [Web-based distance learning - Experience in a pilot study during the Covid-19 school closures]. Journal for therapeutic pedagogy/ Zeitschrift für Heilpädagogik, 71(9), 465-479.
Educational Media Research Association Southwest (2016). KIM-Studie 2016. Kindheit, Internet, Medien. Basisstudie zum Medienumgang 6- bis 13-Jähriger in Deutschland [KIM- Survey 2016. Childhood, internet, media. Study on media behavior and habits of 6- to 13- year-olds in Germany]. https://www.mpfs.de/fileadmin/files/Studien/KIM/2016/KIM_2016_Web-PDF.pdf
Mühling, A., Gebhardt, M. & Diehl, K. (2017). Formative diagnostik durch die onlineplattform Levumi [Formative assessment with the website Levumi]. Computer Science Spectrum/ Informatik Spectrum, 40(6), 556-561. https://doi.org/10.1007/s00287-017-1069-7
Organisation for Economic Cooperation and Development. (2015). Students, computers and learning: Making the connection. https://www.oecd-ilibrary.org/education/students-computers-and-learning_9789264239555-en https://doi.org/10.1787/9789264239555-en
Peak, P. (2005). Recent trends in comparability studies. Pearson educational measurement. http://images.pearsonassessments.com/images/tmrs/tmrs_rg/TrendsCompStudies.pdf
Poggio, J., Glasnapp, D. R., Yang, X. & Poggio, A. J. (2005). A comparative evaluation of score results from computerized and paper & pencil mathematics testing in a large-scale state assessment program. Journal of Technology, Learning, and Assessment, 3(6), 1-30.
Pommerich, M. (2004). Developing computerized versions of paper-and-pencil tests: Mode effects for passage-based tests. Journal of Technology, Learning, and Assessment, 2(6), 1-44.
Pomplun, M., Frey, S. & Becker, D. F. (2002). The score equivalence of paper-and pencil and computerized versions of a speeded test of reading comprehension. Educational and Psychological Measurement, 62(2), 337-354. https://doi.org/10.1177/0013164402062002009
Redecker. C. & Johannessen, O. (2013). Changing assessment. Toward a new assessment paradigm using ICT. European Journal of Education, 47(1), 79-95. https://doi.org/10.1111/ejed.12018
Senkbeil, M. & Wittwer, J. (2007). Die Computervertrautheit von jugendlichen und wirkungen der computernutzung auf den fachlichen kompetenzerwerb [Media habits of adolescents and impacts of computer usage on skills development]. In M. Prenzel, C. Artelt, J. Baumert, W. Blum, M. Hammann, E. Klieme & R. Pekrun (Eds.), PISA 2006. Die Ergebnisse der dritten internationalen Vergleichsstudie [PISA 2006, the results of the third international comparative study] (pp. 277-308). Waxmann.
Shapiro, E. S., Keller, M. A., Lutz, J. G., Edwards Santoro, L. & Hintze, J. M. (2006). Curriculum-based measures and performance on state assessment and standardized tests: Reading and math performance in Pennsylvania. Journal of Psychoeducational Assessment, 24(1), 19-35. https://doi.org/10.1177/0734282905285237
Sikora, S. & Voß, S. (2017). Konzeption und Güte curriculumbasierter messverfahren zur erfassung der arithmetischen leistungsentwicklung in den klassenstufen 3 und 4 [Conception and psychometric properties of curriculum-based measurements for arithmetical skills in grade 3 and 4]. Empirical special education/ Empirische Sonderpädagogik, 9(3), 236-257.
Souvignier, E. & Förster, N. (2011). Effekte prozessorientierter diagnostik auf die entwicklung der lesekompetenz leseschwacher viertklässler [Impacts of process-oriented assessment on reading skills of poor readers in 4th grade]. Empirische Sonderpädagogik, 3(3), 243-255.
Souvignier, E., Förster, N. & Schulte, E. (2014). Wirksamkeit formativen assessments - Evaluation des ansatzes der lernverlaufsdiagnostik [Efficacy of formative assessment- Evaluation of progress monitoring]. In M. Hasselhorn, W. Schneider & U. Trautwein (Eds.), Lernverlaufsdiagnostik [Progress Monitoring] (Vol 12, pp. 221-237). Hogrefe.
Stecker, P. M., Lembke, E. S. & Foegen, A. (2008). Using progress-monitoring data to improve instructional decision making. Preventing School Failure: Alternative Education for Children and Youth, 52(2), 48-58. https://doi.org/10.3200/psfl.52.2.48-58
Stecker, P. M., Fuchs, L. S. & Fuchs, D. (2005). Using curriculum-based measurement to improve student achievement: Review of research. Psychology in the Schools, 42(8), 795-819. https://doi.org/10.1002/pits.20113
VanDerHeyden, A. M. & Burns, M. K. (2008). Examination of the utility of various measures of mathematics proficiency. Assessment for Effective Intervention, 33(4), 215-224. https://doi.org/10.1177/00131640021970600
Vispoel, W. P. (2000). Reviewing and changing answers on computerized fixed-item vocabulary tests. Educational and Psychological Measurement, 60(3), 371-384. https://doi.org/10.1177/00131640021970600
Voß, S. & Blumenthal, Y. (2019). Data-based decision-making. Zum konstrukt und verständnis datenbasierter förderentscheidungsprozesse im unterricht [Concept and insights on data-based decision-making for instructional purposes]. [Paper presentation]. Conference of Empirical Special Education Research, University of Siegen, Germany. https://doi.org/10.13140/RG.2.2.14823.80802
Voß, S. & Gebhardt, M. F. (2017). Themenheft Verlaufsdiagnostik [Special issue on progress monitoring]. Empirische Sonderpädagogik, 9(4), 95-97.
Voß, S. (2014). Curriculumbasierte messverfahren im mathematischen erstunterricht – Zur güte und anwendbarkeit einer adaption US-Amerikanischer Verfahren im deutschen Schulsystem [Curriculum-based measurement in first grade – Psychometric properties and adaptability of an US-American assessment tool in the German school system]. SVH.
Voß, S., Blumenthal, Y., Ehrich, K. & Mahlau, K. (2020). Multimodale diagnostik als ausgangspunkt für spezifische förderung – eine darlegung am beispiel der rechtschreibung [Multimodal assessment as starting point for special support – Explanation using the example of orthography]. Journal for Curative Education/ Zeitschrift für Heilpädagogik, 71, 88-99.
Voß, S., Sikora, S. & Mahlau, K. (2017). Vorschlag zur Konzeption eines curriculumbasierten messverfahrens zur erfassung der rechtschreibleistungen im grundschulbereich [Suggestion of a curriculum-based measurement for spelling skills in elementary schools]. Journal for therapeutic pedagogy/ Empirische Sonderpädagogik, 9(2), 184-194.
Wagner, D. L., Hammerschmidt‐Snidarich, S. M., Espin, C. A., Seifert, K., & McMaster, K. L. (2017). Pre‐service teachers’ interpretation of CBM progress monitoring data. Learning Disabilities Research & Practice, 32(1), 22-31. https://doi.org/10.1111/ldrp.12125
Wang, S., Jiao, H., Young, M. J., Brooks, T. & Olson, J. (2007). A meta-analysis of testing mode effects in grade K-12 mathematics tests. Educational and Psychological Measurement, 67(2), 219-238. https://doi.org/10.1177/0013164406288166
Wayman, M. M., Wallace, T., Wiley, H. I., Tichdt, R. & Espin, C. A. (2007). Literature synthesis on curriculum-based measurement in reading. The Journal of Special Education, 41(2), 85-120. https://doi.org/10.1177/00224669070410020401
Wilbert, J. & Linnemann, M. (2011). Kriterien zur analyse eines tests zur lernverlaufsdiagnostik [Criteria for the analysis of a progress monitoring tool]. Empirical special education/ Empirische Sonderpädagogik, 3(3), 225-242.