ON PAPER OR ON SCREEN? A STUDY ON THE PERCEPTIONS OF DIGITAL TESTS IN HIGHER EDUCATION
Article Sidebar
Published
Dec 24, 2018
Main Article Content
Abstract
This paper explores the potentialities and limitations of Computer-Based Testing (CBT) compared to traditional Paper-Based Testing (PBT). The aim is to verify whether, and to what extent, an electronic mode of assessment can become a suitable alternative to PBT, allowing the evaluation process to be managed more efficiently, especially within large higher education classes. The paper reports a study carried out at the University of Florence in 2016-17 involving 606 students, 443 of whom opted for CBT using their own devices, while 163 preferred PBT. Three hundred and seventy-two participants who experienced CBT also answered a questionnaire on their perceptions, preferences, and level of satisfaction. The results show that the students responded very positively to the digital system, especially the possibility to receive immediate feedback. Some critical issues emerged relating to on-screen reading, which suggests the need for careful design of testing tools.
Article Details
Section
Articles - Special Issue
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access)
References
Balogun, A., Balogun, S., & Onyencho, C. (2017). Test Anxiety and Academic Performance among Undergraduates: The Moderating Role of Achievement Motivation. The Spanish Journal of Psychology, 20(14), 1-8. doi: 10.1017/sjp.2017.5
Boevé, A. J., Meijer, R. R., Albers, C. J., Beetsma, Y., & Bosker, R. J. (2015). Introducing Computer-Based Testing in High-Stakes Exams in Higher Education: Results of a Field Experiment. PLoS ONE, 10(12), 1-13. doi: 10.1371/journal.pone.0143616
Dermo, J. (2009). e-Assessment and the student learning experience: A survey of student perceptions of e- assessment. British Journal of Educational Technology, 40(2), 203-214. doi: 10.1111/j.1467-8535.2008.00915.x
Deutsch, T., Herrmann, K., Frese, T., & Sandholzer, H. (2012). Implementing computer-based assessment - A web-based mock examination changes attitudes. Computers & Education, 58(4), 1068-1075. doi: 10.1016/j.compedu.2011.11.013
Fluck, A., Pullen, D., & Harper, C. (2009). Case Study of a Computer Based Examination System. Australasian Journal of Educational Technology, 25(4), 509-523. doi: 10.14742/ajet.1126
Gillies, C. (2016). To BYOD or not to BYOD: factors affecting academic acceptance of student mobile devices in the classroom. Research in Learning Technology, 24. doi: 10.3402/rlt.v24.30357
He, J., & Freeman, L. A. (2010). Are men more technology-oriented than women? The role of gender on the development of general computer self-efficacy of college students. Journal of Information Systems Education, 21(2), 203-212.
Hillier, M. (2014). The very idea of e-Exams: Student (pre) conceptions. In B. Hegarty, J. McDonald & S.-K. Loke (Eds.), Rhetoric and Reality: Critical perspectives on educational technology (pp. 77-88). Proceedings ASCILITE Dunedin 2014. Retrieved from https://bit.ly/2LK9dZt
Hillier, M. (2015). E-Exams with Student Owned Device: Student Voices. In Proceedings of the International Mobile Learning Festival 2015: Mobile Learning, MOOC’s and 21st Century Learning (pp. 582-608), Hong Kong SAR China. Retrieved from https://bit.ly/2yW2juS
Hochlehnert, A., Brass, K., Moeltner, A., & Juenger, J. (2011). Does Medical Students’ Preference of Test Format (Computer-based vs. Paper-based) have an Influence on Performance? BMC Medical Education, 11(1), 89-95. doi: 10.1186/1472-6920-11-89
International Test Commission (2014). ITC guidelines on quality control in scoring, test analysis, and reporting of test scores. International Journal of Testing, 14(3), 195-217. doi: 10.1080/15305058.2014.918040
INVALSI (2017). Organizzazione somministrazione prove INVALSI CBT 2017‐18. Retrieved from https://bit.ly/2LN0LJd
JISC (2010). Effective Assessment in a Digital Age. A Guide to Technology-Enhanced Assessment and Feedback. Bristol, UK: Higher Education Funding Council for England. Retrieved from https://bit.ly/2JDs4Vn
Jimoh, R. G., Shittu, A. K., & Kawu, Y. K. (2012). Journal of Computing: Students’ Perception of Computer Based Test (CBT) for Examining Undergraduate Chemistry Courses. Journal of Emerging Trends in Computing and Information Sciences, 3(2), 125-134. Retrieved from https://bit.ly/2sPUvJG
Kozma, R. B. (2009). Transforming education: Assessing and teaching 21st century skills. In F. Scheuermann & J. Bojornsson (Eds.), The transition to computer-based assessment (pp. 13–23). Ispra, Italy: European Commission. Joint Research Centre. Retrieved from https://bit.ly/2h2fRxe
Kuikka, M., Kitola, M., & Laakso, M. (2014). Challenges when introducing electronic exam. Research in Learning Technology, 22. doi: 10.3402/rlt.v22.22817
Leeson, H. (2006). The Mode Effect: A Literature Review of Human and Technological Issues in Computerized Testing. International Journal of Testing, 6(1), 1-24. doi: 10.1207/s15327574ijt0601_1
Leu, D.J., Kulikowich, J., Sedransk, N., & Coiro, J. (2009–2014). Assessing online reading comprehension: The ORCA Project. Washington, DC: Institute of Education Sciences, U.S. Department of Education. Retrieved from https://bit.ly/2CemshD
Luecht, R. M., & Sireci, S. G. (2011). A review of models for computer-based testing. Research report 2011–2012. New York, NY: The College Board. Retrieved from https://bit.ly/2ADnpRN
Nardi, A. (2018). Leggere e studiare sullo schermo. Dal design del testo digitale alla verifica degli apprendimenti basata sul computer [Tesi di dottorato]. Università degli Studi di Firenze, Italia. Retrieved from https://bit.ly/2t9kPOj
Nardi, A., & Ranieri, M. (2018). Comparing Paper-based and BYOD e-text examinations: Impact on students’ performance, self-efficacy and satisfaction. British Journal of Educational Technology. doi: 10.1111/bjet.12644
Nikou, S., & Economides, A. (2016). The impact of paper-based, computer-based and mobile-based self-assessment on students’ science motivation and achievement. Computers in Human Behavior, 55, 1241-1248. doi: 10.1016/j.chb.2015.09.025
Paek, P. (2005). Recent Trends in Comparability Studies. Pearson Educational Measurement Research Report 05-05. Retrieved from http://bit.ly/2iYK9W5
Sindre, G., & Vegendla, A. (2015). E-exams versus paper exams: A comparative analysis of cheating-related security threats and countermeasures. Norwegian Information Security Conference (NISK 2015). Retrieved from http://bit.ly/2yTecBH
Sorensen, E. (2013). Implementation and student perceptions of e-assessment in a Chemical Engineering module. European Journal of Engineering Education, 38(2), 172-185. doi: 10.1080/03043797.2012.760533
Stavert, B. (2013). Bring Your Own Device (BYOD) in Schools: 2013 Literature Review. Sydney, NSW: New South Wales Department of Education and Communities. Retrieved from https://bit.ly/2gQV8ge
Stenlund, T., Eklöf, H., & Lyrén, P. (2016). Group differences in test-taking behaviour: an example from a high-stakes testing program. Assessment in Education: Principles, Policy & Practice, 24(1), 4-20. doi: 10.1080/0969594X.2016.1142935
Terzis, V., & Economides, A.A. (2011). Computer based assessment: gender differences in perceptions and acceptance. Computers in Human Behavior, 27, 2108-2122. doi: 10.1016/j.chb.2011.06.005
Walker, R., & Handley, Z. (2016). Designing for learner engagement with computer-based testing. Research in Learning Technology, 24. doi: 10.3402/rlt.v24.30083
Wang, S., Jiao, H., Young, M., Brooks, T., & Olson, J. (2008). Comparability of Computer-Based and Paper-and- Pencil Testing in K–12 Reading Assessments. Educational and Psychological Measurement, 68(1), 5-24. doi: 10.1177/0013164407305592
Way, W.D., Davis, L.L., Keng, L., & Strain-Seymour, E. (2016). From standardization to personalization: The comparability of scores based on different testing conditions, modes, and devices. In F. Drasgow (Ed.), Technology in testing: Improving educational and psychological measurement, Vol 2. Abingdon, UK: Routledge.
Yamamoto, K. (2012). Outgrowing the Mode Effect Study of Paper and Computer Based Testing. Retrieved from https://bit.ly/2kiNKOz
Boevé, A. J., Meijer, R. R., Albers, C. J., Beetsma, Y., & Bosker, R. J. (2015). Introducing Computer-Based Testing in High-Stakes Exams in Higher Education: Results of a Field Experiment. PLoS ONE, 10(12), 1-13. doi: 10.1371/journal.pone.0143616
Dermo, J. (2009). e-Assessment and the student learning experience: A survey of student perceptions of e- assessment. British Journal of Educational Technology, 40(2), 203-214. doi: 10.1111/j.1467-8535.2008.00915.x
Deutsch, T., Herrmann, K., Frese, T., & Sandholzer, H. (2012). Implementing computer-based assessment - A web-based mock examination changes attitudes. Computers & Education, 58(4), 1068-1075. doi: 10.1016/j.compedu.2011.11.013
Fluck, A., Pullen, D., & Harper, C. (2009). Case Study of a Computer Based Examination System. Australasian Journal of Educational Technology, 25(4), 509-523. doi: 10.14742/ajet.1126
Gillies, C. (2016). To BYOD or not to BYOD: factors affecting academic acceptance of student mobile devices in the classroom. Research in Learning Technology, 24. doi: 10.3402/rlt.v24.30357
He, J., & Freeman, L. A. (2010). Are men more technology-oriented than women? The role of gender on the development of general computer self-efficacy of college students. Journal of Information Systems Education, 21(2), 203-212.
Hillier, M. (2014). The very idea of e-Exams: Student (pre) conceptions. In B. Hegarty, J. McDonald & S.-K. Loke (Eds.), Rhetoric and Reality: Critical perspectives on educational technology (pp. 77-88). Proceedings ASCILITE Dunedin 2014. Retrieved from https://bit.ly/2LK9dZt
Hillier, M. (2015). E-Exams with Student Owned Device: Student Voices. In Proceedings of the International Mobile Learning Festival 2015: Mobile Learning, MOOC’s and 21st Century Learning (pp. 582-608), Hong Kong SAR China. Retrieved from https://bit.ly/2yW2juS
Hochlehnert, A., Brass, K., Moeltner, A., & Juenger, J. (2011). Does Medical Students’ Preference of Test Format (Computer-based vs. Paper-based) have an Influence on Performance? BMC Medical Education, 11(1), 89-95. doi: 10.1186/1472-6920-11-89
International Test Commission (2014). ITC guidelines on quality control in scoring, test analysis, and reporting of test scores. International Journal of Testing, 14(3), 195-217. doi: 10.1080/15305058.2014.918040
INVALSI (2017). Organizzazione somministrazione prove INVALSI CBT 2017‐18. Retrieved from https://bit.ly/2LN0LJd
JISC (2010). Effective Assessment in a Digital Age. A Guide to Technology-Enhanced Assessment and Feedback. Bristol, UK: Higher Education Funding Council for England. Retrieved from https://bit.ly/2JDs4Vn
Jimoh, R. G., Shittu, A. K., & Kawu, Y. K. (2012). Journal of Computing: Students’ Perception of Computer Based Test (CBT) for Examining Undergraduate Chemistry Courses. Journal of Emerging Trends in Computing and Information Sciences, 3(2), 125-134. Retrieved from https://bit.ly/2sPUvJG
Kozma, R. B. (2009). Transforming education: Assessing and teaching 21st century skills. In F. Scheuermann & J. Bojornsson (Eds.), The transition to computer-based assessment (pp. 13–23). Ispra, Italy: European Commission. Joint Research Centre. Retrieved from https://bit.ly/2h2fRxe
Kuikka, M., Kitola, M., & Laakso, M. (2014). Challenges when introducing electronic exam. Research in Learning Technology, 22. doi: 10.3402/rlt.v22.22817
Leeson, H. (2006). The Mode Effect: A Literature Review of Human and Technological Issues in Computerized Testing. International Journal of Testing, 6(1), 1-24. doi: 10.1207/s15327574ijt0601_1
Leu, D.J., Kulikowich, J., Sedransk, N., & Coiro, J. (2009–2014). Assessing online reading comprehension: The ORCA Project. Washington, DC: Institute of Education Sciences, U.S. Department of Education. Retrieved from https://bit.ly/2CemshD
Luecht, R. M., & Sireci, S. G. (2011). A review of models for computer-based testing. Research report 2011–2012. New York, NY: The College Board. Retrieved from https://bit.ly/2ADnpRN
Nardi, A. (2018). Leggere e studiare sullo schermo. Dal design del testo digitale alla verifica degli apprendimenti basata sul computer [Tesi di dottorato]. Università degli Studi di Firenze, Italia. Retrieved from https://bit.ly/2t9kPOj
Nardi, A., & Ranieri, M. (2018). Comparing Paper-based and BYOD e-text examinations: Impact on students’ performance, self-efficacy and satisfaction. British Journal of Educational Technology. doi: 10.1111/bjet.12644
Nikou, S., & Economides, A. (2016). The impact of paper-based, computer-based and mobile-based self-assessment on students’ science motivation and achievement. Computers in Human Behavior, 55, 1241-1248. doi: 10.1016/j.chb.2015.09.025
Paek, P. (2005). Recent Trends in Comparability Studies. Pearson Educational Measurement Research Report 05-05. Retrieved from http://bit.ly/2iYK9W5
Sindre, G., & Vegendla, A. (2015). E-exams versus paper exams: A comparative analysis of cheating-related security threats and countermeasures. Norwegian Information Security Conference (NISK 2015). Retrieved from http://bit.ly/2yTecBH
Sorensen, E. (2013). Implementation and student perceptions of e-assessment in a Chemical Engineering module. European Journal of Engineering Education, 38(2), 172-185. doi: 10.1080/03043797.2012.760533
Stavert, B. (2013). Bring Your Own Device (BYOD) in Schools: 2013 Literature Review. Sydney, NSW: New South Wales Department of Education and Communities. Retrieved from https://bit.ly/2gQV8ge
Stenlund, T., Eklöf, H., & Lyrén, P. (2016). Group differences in test-taking behaviour: an example from a high-stakes testing program. Assessment in Education: Principles, Policy & Practice, 24(1), 4-20. doi: 10.1080/0969594X.2016.1142935
Terzis, V., & Economides, A.A. (2011). Computer based assessment: gender differences in perceptions and acceptance. Computers in Human Behavior, 27, 2108-2122. doi: 10.1016/j.chb.2011.06.005
Walker, R., & Handley, Z. (2016). Designing for learner engagement with computer-based testing. Research in Learning Technology, 24. doi: 10.3402/rlt.v24.30083
Wang, S., Jiao, H., Young, M., Brooks, T., & Olson, J. (2008). Comparability of Computer-Based and Paper-and- Pencil Testing in K–12 Reading Assessments. Educational and Psychological Measurement, 68(1), 5-24. doi: 10.1177/0013164407305592
Way, W.D., Davis, L.L., Keng, L., & Strain-Seymour, E. (2016). From standardization to personalization: The comparability of scores based on different testing conditions, modes, and devices. In F. Drasgow (Ed.), Technology in testing: Improving educational and psychological measurement, Vol 2. Abingdon, UK: Routledge.
Yamamoto, K. (2012). Outgrowing the Mode Effect Study of Paper and Computer Based Testing. Retrieved from https://bit.ly/2kiNKOz