Evaluation of the Validity and Reliability of a Test System Based on the Degree of Confidence of the Student
Keywords:
Confidence based test, Evaluation, Validity, Reliability, Learning, University student, Security gradeCopyright (c) 2020 Iberoamerican Journal of Educational Evaluation
![Creative Commons License](http://i.creativecommons.org/l/by-nc-nd/3.0/88x31.png)
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 Unported License.
Abstract
Test-type exams based on the student's confidence (CBT) are scarcely used in the Spanish academic environment. This evaluation system emphasizes both the student's knowledge and the level of confidences he or she has of that knowledge. The results obtained in CBT were analyzed. The study was conducted in a school of Nursing and Physiotherapy during the academic years covered from 2009 to 2014. The validity and reliability of the CBT were compared to those obtained with a traditional correct-response grading system. Students acquired dexterity with the new scoring procedure quickly, choosing the most appropriate option for confidence level that was in accordance with their level of knowledge in a rational way. The validity of the correction system was higher in the CBT compared to the traditional test system, with a statistically significant difference. Reliability was analyzed in five groups of students using the Cronbach ? coefficient, with four of them having higher reliability with the CBT as compared to the traditional grading system and the fifth having the same level of reliability for the two types of tests. Confidence based testing has shown to be a procedure that provides advantages over the traditional test. CBT provides more information to the teacher, students adapt to it quickly and it has greater validity and at least equal reliability to the traditional grading system.
Downloads
References
Ahlgren, A. (1970). A hand-scoring system for confidence-weighted scores. Washington, DC: Department of Health, Education & Welfare, Office of Education.
Barr, D. A. y Burke, J. R. (2013). Using confidence-based marking in a laboratory setting: A tool for student self-assessment and learning. The Journal of Chiropractic Education, 27(1), 21-26. https://doi.org/10.7899/JCE-12-018
Ben–Shakhar, G. y Sinai, Y. (1991). Gender differences in multiple-choice tests: The role of differential guessing tendencies. Journal of Educational Measurement, 28(1), 23-35. https://doi.org/10.1111/j.1745-3984.1991.tb00341.x
Biggs, J. B. (1993). What do inventories of students’ learning process really measure? A theoretical review and clarification. British Journal of Educational Psychology, 63(1), 3–19. https://doi.org/10.1111/j.2044-8279.1993.tb01038.x
Bloom, B. S. (1956). Taxonomy of educational objectives: the classification of educational goal S: Handbook I, cognitive domain. Londres: Longman Group.
Budescu, D. y Bar-Hillel, M. (1993). To guess or not to guess: A decision-theoretic view of formula scoring. Journal of Educational Measurement, 30(4), 277-91. https://doi.org/10.1111/j.1745-3984.1993.tb00427.x
Chang, C. Y. y Cheng, W. Y. (2008). Science achievement and students’ self-confidence and interest in science: A Taiwanese representative sample study. International Journal of Science Education, 30(9), 1183-200. https://doi.org/10.1080/09500690701435384
Ebel, R.L. (1968). Valid confidence testing: Demonstration kit. Journal of Educational Measurement, 5(4), 353–354.
Entwistle, N., Hanley, M. y Ratcliffe, G. (1979). Approaches to learning and levels of understanding, British Journal of Educational Research, 5(1), 99-114. https://doi.org/10.10800141192790050110
Frary, R. B. (1982). A simulation study of reliability and validity of multiple–choice test scores under six response–scoring modes. Journal of Educational Statistics, 7(4), 333-51. https://doi.org/ 10.3102/10769986007004333
Gardner, W. C. (1969, septiembre). The use of confidence testing in the Academic Instructor Course. Comunicación presentada en el Annual Conference of the Military Testing Association, Nueva York.
Gardner-Medwin, A. R. (1995). Confidence assessment in the teaching of basic science. Association for Learning Technology Journal, 3(1), 80-85. https://doi.org/10.3402/ rlt.v3i1.9597
Gardner-Medwin, A. R. (1998). Updating with confidence: Do your students know what they don’t know?. Health Informatics, 4, 45-46.
Gardner–Medwin, A. R. (2006). Confidence-based marking towards deeper learning and better exams. En C. Bryan y K. Clegg (Eds.), Innovative Assessment in Higher Education (pp. 141-149). Londres: Routledge.
Gardner–Medwin, A. R. (2011). Reasonable doubt: uncertainty in education, science and law. Proceedings of the British Academy, 171, 465-83. https://doi.org/10.5871/bacad/9780197264843.003.0017
Gardner-Medwin, A. R. y Curtin, N. (2007). Certainty-Based Marking (CBM) for reflective learning and proper knowledge assessment. Recuperado de http://www.ucl.ac.uk/lapt/REAP_cbm.pdf
Gardner-Medwin, A. R. y Gahan, M. (2003, julio). Formative and summative confidence-based assessment. Comunicación presentada en el 7th International Computer–Aided Assessment Conference, Loughborough, Reino Unido. Recuperado de https://tmedwin.net/~ucgbarg/tea/caa03a.pdf
Hambleton, R. K., Roberts, D. M. y Traub, R. R. (1970). A comparison of the reliability and validity of two methods for assessing partial knowledge on a multiple-choice test. Journal of Educational Measurement, 7(2), 75-90. https://doi.org/10.1111/j.1745-3984.1970.tb00698.x
Hassmen, P. y Hunt, D. P. (1994). Human self–assessment in multiple–choice testing. Journal of Educational Measurement, 31(2), 149-60. https://doi.org/10.1111/j.1745-3984.1994.tb00440.x
Hevner, K. (1932). A method of correcting for guessing in true–false tests and empirical evidence in support of IT. The Journal of Social Psychology, 3(3), 359-62. https://doi.org/10.1080/00224545.1932.9919159
Hunt, D. P. (2003). The concept of knowledge and how to measure it. Journal of Intellectual Capital, 4(1), 100-13. https://doi.org/10.1108/14691930310455414
Kansup, W. y Hakstian, A. R. (1975). A comparison of several methods of assessing partial knowledge in multiple choice tests: I. Scoring procedures. Journal of Educational Measurement, 12(4), 219-30. https://doi.org/10.1111/j.1745-3984.1975.tb01024.x
Khan, K. S., Davies, D. A. y Gupta, J. K. (2001) Formative self–assessment using multiple true–false questions on the internet: Feedback according to confidence about correct knowledge. Medical Teacher, 23(2),158-63. https://doi.org/10.1080/0142159003107
Kleitman, S. y Stankov, L. (2001). Ecological and person-oriented aspects of metacognitive processes in test-taking. Journal of Applied Cognitive Psychology, 15(3),321-341. https://doi.org/10.1002/acp.705
Luetsch, K. y Burrows, J. (2016). Certainty rating in pre-and post-tests of study modules in an online clinical pharmacy course - A pilot study to evaluate teaching and learning. BMC Medical Education,16(1), 267-291. https://doi.org/10.1186/s12909-016-0783-1
Marton, F. y Säljo, R. (1976a). On qualitative differences of learning (I): outcome and process. British Journal of Educational Psychology,46(1), 4-11. https://doi.org/10.1111/j.2044-8279.1976.tb02980.x
Marton, F. y Säljo, R. (1976b). On qualitative differences in learning II: outcome as a function of the learner’s conception of the task. British Journal of Educational Psychology, 46(2), 115–27. https://doi.org/10.1111/j.2044-8279.1976.tb02304.x
Moore, D.A. y Healy, P.J. (2008). The trouble with overconfidence. Psychological Review, 115(2), 502-517.https://doi.org/10. 1037/0033-295X.115.2.502
Morales, P. (2006). Las pruebas objetivas: normas, modalidades y cuestiones discutidas. Recuperado dehttp://www.upcomillas. Es/personal/peter/otrosdocumentos /PruebasObjetivas. Pdf.
Morales, P. (2008). Estadística aplicada a las Ciencias Sociales. Madrid: Universidad Pontificia Comillas.
Morony, S., Kleitman, S., Lee, Y. P. y Stankov, L. (2013). Predicting achievement: confidence versus self-efficacy, anxiety, and self-concept in Confucian and European countries. International Journal of Educational Research,58, 79-96. https://doi.org/10.1016/j.ijer.2012.11.002
Panadero, E. y Alonso-Tapia, J. (2014). How do students self-regulate? Review of Zimmerman’s cyclical model of self-regulated learning. Anales de Psicología, 30(2), 450-462. https://doi.org/10.6018/analesps.30.2.167221
Pintrich, P. R. (2002). The role of metacognitive knowledge in learning, teaching and assessing. Theory into Practice, 41(4), 219-25. https://doi.org/10.1207/s15430421tip4104_3
Rippey, R. M. (1978), Interactive confidence test scoring and interpretation. Educational and Psychological Measurement, 38(1), 153-57. https://doi.org/10.1177/001316447803800122
Sharma, M. D. y Bewes, J. (2011). Self-monitoring: Confidence, academic achievement and gender differences in Fhysics. Journal of Learning Design, 4(3), 1-13. https://doi.org/10.5204/jld.v4i3.76
Stanger-Hall, K.F. (2012). Multiple-choice exams: An obstacle for higher-level thinking in introductory science classes. CBE Life Scientific Education.11(3), 294-306. https://doi.org/10.1187/cbe.11-11-0100
Stankov, L. (2000). Complexity, metacognition and fluid intelligence. Intelligence, 28(2), 121-43. https://doi.org/10.1016/S0160-2896(99)00033-1
Stankov, L., Lee, J. y Paek, J. (2009). Realism of confidence judgments. European Journal of Psychological Assessment, 25, 123-30. https://doi.org/10.1027/1015-5759.25.2.123
Stankov, L., Lee, J., Luo, W. y Hogan, D. J. (2012). Confidence: A better predictor of academic achievement self-efficacy, self-concept and anxiety?. Learning and Individual Differences, 22(6), 747-58. https://doi.org/10.1016/j.lindif.2012.05.013
Thomson, B. (1994). Guidelines for authors. Educational and Psychological Measurement, 54,837-47.
Urosa, B. (1995). La adivinación en las pruebas objetivas: Alternativas a la fórmula clásica de corrección. Tesis Doctoral, Universidad Pontificia Comillas, España.