A Review of Standardised Assessment Development Procedure and Algorithms for Computer Adaptive Testing: Applications and Relevance for Fourth Industrial Revolution
Abstract
Teaching and learning have gone online in response to the pandemic, which reveals the need for accurately tailored educational assessments to ascertain the extent to which learning outcomes or objectives are achieved. Computer Adaptive Testing (CAT) is a technology-driven form of assessment that tailors items to a candidate's ability level with empirically proven benefits over the fixed-form computer based test. A systematic review was employed which shows that item bank is a key requirement for CAT and the items must through a rigorous item development process to ensure and maintain quality  in terms of content, criterion constructs and internal consistency, determining the psychometric validation of behavioural measures while leveraging on variances of Item Response Theory (IRT). Following the item development stage is the need to compile validated items into administrable forms using advanced computer software for automatic test assembly and administration, such as FastTest which allows specifying empirically tried algorithms for CAT from start to termination of the test. This helps to ensure that assessment properly leverages the advantages that CAT holds. Furthermore, the review revealed that CAT has been widely applied with large-scale testing in various fields by educational, health and psychological professionals utilising different IRT models; however only in developed countries. This brings to bear the need for adoption in other parts of the world, for improvements in educational assessments. The interjections of 4IR with AI considering emerging technology aids the CAT algorithm for achieving expert and knowledge-based systems, being a requirement for survival in today’s world.
https://doi.org/10.26803/ijlter.20.5.1
Keywords
Full Text:
PDFReferences
Alabi, A. T., Issa, A. O., & Oyekunle, R. A. (2012). The use of a computer-based testing method for the conduct of examinations at the University of Ilorin. International Journal of Learning and Development, 2(3), 68-80.
Anderson, L., & Krathwohl, D.A. (2001). A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives. Longman.
APO Group. (2021). April 19. Coronavirus - South Africa: COVID-19 update 2021). africanews. https://www.africanews.com/2021/04/20/coronavirus-south-africa covid-19-update-19-april-2021//
Aybek, E. C., & Demirtasli, R. N. (2017). Computerised adaptive test (CAT) applications and item response theory models for polytomous items. International Journal of Research in Education and Science (IJRES), 3(2), 475-487. https://doi.org/10.21890/ijres.327907
BBC News. 2020. Coronavirus in South Africa: Restrictions ease as Covid-19 cases rise rapidly. https://www.bbc.com/news/world-africa-53093832
Bartneck, C., Lütge, C., Wagner, A., & Welsh, S. (2021). What Is AI? In C.Bartneck, C. Lütge, A. Wagner & S. Welsh (Eds.), An Introduction to Ethics in Robotics and AI. Cham: Springer. https://doi.org/10.1007/978-3-030-51110-4_2
Becker, K. A., & Bergstrom, B. A. (2013). Test administration models. Practical Assessment, Research, and Evaluation, 18(1), 14.
Builtin. (2019). Artificial Intelligence. What is Artificial Intelligence? How Does AI Work? https://builtin.com/artificial-intelligence
Butler-Adam, J. (2018). The fourth industrial revolution and education. South African Journal of Science, 114(5-6), 1-1. http://dx.doi.org/10.17159/sajs.2018/a0271
Cella, D., Gershon, R., Lai, J. S., & Choi, S. (2007). The future of outcomes measurement: item banking, tailored short-forms, and computerised adaptive assessment. Quality of Life Research, 16(1), 133-141.
COVID-19 Online Resource & News Portal. (2020). COVID-19 Corona Virus South African Resource Portal. https://sacoronavirus.co.za/
Downing, S. M. (2006). Twelve Steps for Effective Test Development. In S. Lane, M. R. Raymond & T. M. Haladyna (Eds.), Handbook of test development. Routledge. http://www.danangtimes.vn/Portals/0/Docs/314154729-0805852654.pdf#page=18
Eggen, T. J. H. M., & Straetmans, G. J. J. M. (2000). Computerised adaptive testing for classifying examinees into three categories. Educational and Psychological Measurement, 60(5), 713-734.
Eignor, D. R., Stocking, M. L., Way, W. D., & Steffen, M. (1993). Case Studies in Computer Adaptive Test Design Through Simulation. ETS Research Report Series, (2), 1-41.
Germain, M. L. (2006a). Development and preliminary validation of a psychometric measure of expertise: The generalized expertise measure (GEM) [Dissertation, Barry University, Florida].
Giouroglou, H., & Economides, A. (2004). State-of-the-Art and adaptive open-closed items in adaptive foreign language assessment. Proceedings of the 4th Hellenic Conference of the Association of Informational and Communication Technologies in Education (pp. 747-756). Athens.
Han, K. C. T. (2018). Conducting simulation studies for computerised adaptive testing using SimulCAT: An instructional piece. Journal of Educational Evaluation for Health Professions, 15(20). https://doi.org/10.3352/jeehp.2018.15.7
Izard, J. (2005). Trial testing and item analysis in test construction. International Institute for Educational Planning/UNESCO. http://www.sacmeq.org/sites/default/files/sacmeq/training-modules/sacmeq-training-module-7.pdf
Jozefowicz, R. F., Koeppen, B. M., Case, S., Galbraith, R., Swanson, D., & Glew, H. (2002). The quality of in-house medical school examinations. Academic Medicine, 77, 156–161.
Kane, L. T., Namdari, S., Plummer, O. R., Beredjiklian, P., Vaccaro, A., & Abboud, J. A. (2020). Use of Computerized Adaptive Testing to Develop More Concise Patient-Reported Outcome Measures. JBJS Open Access, 5(1). http://dx.doi.org/10.2106/JBJS.OA.19.00052
Kimura, T. (2017). The impacts of computer adaptive testing from a variety of perspectives. Journal of Educational Evaluation for Health Professionals, 14(12), 1-5.
Li, C., & Lalani, F. (2020, April 29). The COVID-19 Pandemic has changed education forever. This is how. World Economic Forum. https://www.weforum.org/agenda/2020/04/coronavirus-education-global-covid19-online-digital-learning/
Linacre, J. M. (2000). Computer-adaptive testing: A methodology whose time has come (No. 69, p. 58). Mathematics, Engineering, Science, Achievement (MESA) memorandum. https://www.cehd.umn.edu/EdPsych/C-Bas-R/Docs/Linacre2000_CAT.pdf
Luecht, R. M. (2006). Designing Tests for Pass-Fail Decisions Using Item Response Theory. In S. Lane, M. R. Raymond & T. M. Haladyna (Eds.) Handbook of test development. Routledge. http://www.danangtimes.vn/Portals/0/Docs/314154729-0805852654.pdf#page=18
McCallin, R. C. (2006). Test Administration. In S. Lane, M. R. Raymond, & T. M. Haladyna (Eds.) Handbook of test development. Routledge. http://www.danangtimes.vn/Portals/0/Docs/314154729-0805852654.pdf#page=18
National Research Council. (2004). "3 The Test Development Process." Redesigning the U.S. Naturalization Tests: Interim Report. The National Academies Press. https://doi.org/10.17226/11168
Oladele, J. I., Ayanwale, M. A., & Owolabi, H. O. (2020). Paradigm Shifts in Computer Adaptive Testing in Nigeria in Terms of Simulated Evidence. Journal of Social Sciences, 63(1-3, 9-20. Publication of Kamla-Raj Enterprises (KRE) Publishers. https://doi.org/10.31901/24566756.2020/63.1-3.2264
Penprase, B. E. (2018). The Fourth Industrial Revolution and Higher Education. In N. Gleason (Ed.), Higher Education in the Era of the Fourth Industrial Revolution. Singapore: Palgrave Macmillan. https://doi.org/10.1007/978-981-13-0194-0_9
Petersen, M. A., Aaronson, N. K., Chie, W. C. Conroy, T., Costantini, A., Hammerlid, E., Hjermstad, M. J., Kaasa, S., Loge, J. H., Velikova, G., Young, T., & Groenvold, M. (2016). Development of an item bank for computerised adaptive test (CAT) measurement of pain. Quality of Life Research, 25(1), 1-11. https://doi.org/10.1007/s11136-015-1069-5
Reckase, M. D. (2010). Designing item pools to optimise the functioning of a computerised adaptive test. Psychological Test and Assessment Modeling, 52(2), 127-141.
Schwab, K. (2016). The Fourth Industrial Revolution: what it means, how to respond. World Economic Forum. https://www.weforum.org/agenda/2016/01/the-fourth-industrial-revolution-what-it-means-and-how-to-respond/
Segall, D. O. (2005). Computerised adaptive testing. Encyclopedia of social measurement, 1, 429-438. http://iacat.org/sites/default/files/biblio/se04-01.pdf
Seo, D. G. (2017). Overview and current management of computerised adaptive testing in licensing/certification examinations. Journal of Educational Evaluation for Health Professions, 14. https://doi.org/10.3352/jeehp.2017.14.17
Singh, G., & Sagar, A. M. D. (2013). An Overview of Artificial Intelligence. SBIT Journal of Sciences and Technology, 2(1). https://doi.org/10.13140/RG.2.2.20660.19840
Staff Writer. (2020). September 16. South Africa moves to lockdown level 1-here are the changes. BUSINESSTECH. https://businesstech.co.za/news/trending/433943/south-africa-moves-to-lockdown-level-1-here-are-the-changes/
South African Government Disaster Management Act. (2020a). Alert level 1 lockdown regulations.
South African Disaster Management Act. (2020b). Alert level 5 lockdown regulations. https://bit.ly/33r7iUM
The Presidency, Republic of South Africa (2020). Statement by President Cyril Ramaphosa on South Africa's response to the coronavirus pandemic. https://bit.ly/3mezqlz
Theunissen, M. H., de Wolff, M. S., Deurloo, J. A., Vogels, A. G., & Reijneveld, S. A. (2020). Computerised adaptive testing to screen children for emotional and behavioural problems by preventive child healthcare. BMC paediatrics, 20(1), 1-7. https://bmcpediatr.biomedcentral.com/articles/10.1186/s12887-020-2018-1
Thompson, N. A. (2011). Advantages of computerised adaptive testing (CAT). https://assess.com/docs/Advantages-of-CAT-Testing.pdf
Thompson, N. A., & Weiss, D. A. (2011). A framework for the development of computerised adaptive tests. Practical Assessment, Research, and Evaluation, 16(1), 1.
Thompson, N. (2018, March 23). Overview: Steps to develop an adaptive test. Assessment Systems. https://assess.com/2018/03/23/develop-computerized-adaptive-test/
Thompson, N. (n.d.). Secure Online Testing with FastTest. https://assess.com/fasttest-secure-online-testing/
The World Bank Group. (2020a). How countries are using edtech (including online learning, radio, television, texting) to support access to remote learning during the COVID-19 Pandemic. https://bit.ly/3vULWvd
The World Bank Group. (2020b). Digital Technologies in Education. The use of information and communication technologies in education can play a crucial role in providing new and innovative forms of support to teachers, students, and the learning process more broadly. https://www.worldbank.org/en/topic/edutech
Van der Linden, W. J. (2005). A comparison of itemâ€selection methods for adaptive tests with content constraints. Journal of Educational Measurement, 42(3), 283-302. http://www.jstor.com/stable/20461794
Van der Linden, W. J., & Pashley, P. J. (2000). Item selection and ability estimation in adaptive testing. In W. J. Linden, W. J. van der Linden, & C. A. Glas (Eds.), Computerised adaptive testing: Theory and practice. https://doi.org/10.1007/0-306-47531-6_1
Vecchiatto, P., Kew, J., & Prinsloo, L. (2020) August 12. South Africa’s Ramaphosa Prepares to Ease Lockdown Rules. Bloomberg. https://www.bloomberg.com/news/articles/2020-08-12/south-africa-s-ramaphosa-prepares-to-ease-lockdown-restrictions
Veldkamp, B. P., & Verschoor, A. J. (2019). Robust Computerised Adaptive Testing. In B. P. Veldkamp & C. Sluijter (Eds.), Theoretical and Practical Advances in Computer-based Educational Measurement (pp. 291-305). Cham: Springer. https://library.oapen.org/bitstream/handle/20.500.12657/22945/1007216.pdf?sequence=1#page=291
Wang, Z., & Thompson, N. (2020). Digital Module 19: Foundations of IRT Estimation. Instructional Topics in Educational Measurement Series (ITEMS). https://ncme.elevate.commpartners.com
Wainer, H., Dorans, N. J., Flaugher, R., Green, B. F., & Mislevy, R. J. (2000). Computerised adaptive testing: A primer. Routledge.
Ware, Jr., J. E., Kosinski, M., Bjorner, J. B., Bayliss, M. S., Batenhorst, A., Dahlöf, C. G. H., Tepper, S., & Dowson, A. (2003). Applications of computerised adaptive testing (CAT) to the assessment of headache impact. Quality of Life Research, 12(8), 935-952. https://doi.org/10.1023/a:1026115230284
Way, W. D., Davis, L. L., & Fitzpatrick, S. (2006). Practical questions in introducing computerised adaptive testing for K-12 assessments. https://www.pearsonassessments.com/content/dam/school/global/clinical/us/assets/testnav/research-report-cat-for-k-12-assessments.pdf
Weiss, D. J., & Kingsbury, G. G. (1984). Application of computerised adaptive testing to educational problems. Journal of Educational Measurement, 21(4), 361–375. https://doi.org/10.1111/j.1745-3984.1984.tb01040.x
Wise, S. L., & Kingsbury, G. G. (2000). Practical Issues in Developing and Maintaining a Computerised Adaptive Testing Program. Psicológica, 21, 135-155.
Worldometers. (2020, November 28). COVID-19 Coronavirus Pandemic. https://www.worldometers.info/coronavirus/
Zhang, Y., Wang, D., Gao, X., Cai, Y., & Tu, D. (2019). Development of a computerised adaptive testing for internet addiction. Frontiers in Psychology, 10, 1010.
Refbacks
- There are currently no refbacks.
e-ISSN: 1694-2116
p-ISSN: 1694-2493