INTERNATIONAL PERFORMANCE ASSESSMENT OF CRITICAL THINKING: FRAMEWORK FOR TRANSLATION AND ADAPTATION
Keywords:
Critical thinking, Performance assessment, Translation, Adaptation, International assessmentAbstract
Higher education institutions worldwide claim they impact students’ learning outcomes within and across academic domains. Critical thinking (CT) is prominent among the intended outcomes (Braun et al., 2020). In this context, there is increasing interest in ecologically valid performance assessments (PAs) of CT that can be used internationally (Zlatkin-Troitschanskaia et al., 2018). While several studies have aimed to measure and compare CT skills across countries, this typically has been done using multiple-choice questions. Only a few studies involve adaptation of PAs. Their results point to the need for a more refined adaptation process (Braun et al., 2020), especially in terms of functionally equivalent adaptation and redesign. Based on a review of previous approaches related to test adaptation, with focus on the challenges of achieving cultural responsiveness, we propose a conceptual framework for adapting PAs of CT for international studies. The framework differentiates between two stages and three adaptation designs. The first stage involves a collaborative approach to the design of PAs of CT. The second stage offers three design alternatives which differ in their emphasis on linguistic considerations and cultural responsiveness. While this paper focuses on PAs of CT for higher education, it may be applicable to pre-college education.
Downloads
References
Addey, C., & Sellar, S. (2017) Why do countries participate in PISA? Understanding the role of international large-scale assessments in global education policy. In A. Verger, M. Novelli, H.K. Altinyelken (Eds.), Global Education Policy and International Development: New Agendas, Issues and Policies (pp. 97-117) London: Bloomsbury.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
Basterra, M., Trumbull, E., & Solano-Flores, G. (2011). Cultural validity in assessment: Addressing linguistic and cultural diversity. NY: Routledge.
Bennet, R. (2020) Interpreting test-score comparisons. In A. I. Berman, E. H. Haertel, & J. W. Pellegrino (Eds.), Comparability of large-scale educational assessments: Issues and recommendations (pp. 205–225). Washington, DC: National Academy of Education. Retrieved from https://naeducation.org/wp-content/uploads/2020/05/Comparability-of-Large-Scale-Educational-Assessments_final.pdf
Berman, A. I., Haertel, E. H., & Pellegrino, J. W. (2020) Introduction: Framing the issues. In A. I. Berman, E. H. Haertel, & J. W. Pellegrino (Eds.), Comparability of large-scale educational assessments: Issues and recommendations (pp. 205–225). Washington, DC: National Academy of Education. Retrieved from https://naeducation.org/wp-content/uploads/2020/05/Comparability-of-Large-Scale-Educational-Assessments_final.pdf
Braun, H.I., Shavelson, R.J., Zlatkin-Troitschanskaia, O., & Borowiec, K. (2020) Performance Assessment of Critical Thinking: Conceptualization, Design, and Implementation. Frontiers in Education. 5:156. https://doi.org 10.3389/feduc.2020.00156
Brown, G. (2016). Handbook of Human and Social Conditions in Assessment. In G. Brown, & L. Harris (Eds.) Handbook of Human and Social Conditions in Assessment. New York: Routledge. https://doi.org/10.4324/9781315749136
Byrne, B.M., Shavelson, R.J., & Muthen, B.O. (1989). Testing for the equivalence of factor covariance and mean structures: The issue of partial measurement invariance. Psychological Bulletin, 105(3), 456-466.
DePascale, C., & Gong, B. (2020) Comparability of individual students’ scores on the “same test. In A. I. Berman, E. H. Haertel, & J. W. Pellegrino (Eds.), Comparability of large-scale educational assessments: Issues and recommendations (pp. 205–225). Washington, DC: National Academy of Education. Retrieved from https://naeducation.org/wp-content/uploads/2020/05/Comparability-of-Large-Scale-Educational-Assessments_final.pdf
Ercikan, K., & Solano-Flores, G. (2016). Section Discussion: Assessment and Sociocultural Context: A Bidirectional Relationship. (pp. 490–505) In G. Brown, & L. Harris (Eds.) Handbook of Human and Social Conditions in Assessment. New York: Routledge. https://doi.org/10.4324/9781315749136
Ercikan, K., & Por, H.H. (2020) Comparability in multilingual and multicultural assessment contexts. In A. I. Berman, E. H. Haertel, & J. W. Pellegrino (Eds.), Comparability of large-scale educational assessments: Issues and recommendations (pp. 205–225). Washington, DC: National Academy of Education. Retrieved from https://naeducation.org/wp-content/uploads/2020/05/Comparability-of-Large-Scale-Educational-Assessments_final.pdf
Hambleton, R. K. (2002). Adapting achievement tests into multiple languages for international assessments. In A. C. Porter & A. Gamoran (Eds.), Methodological advances in cross-national surveys of educational achievement (pp. 58-79). Washington, DC: National Academy Press.
Hambleton, R. K. (2005) Issues, Designs, and Technical Guidelines for Adapting Tests Into Multiple Languages and Cultures. In R.K. Hambleton, P.F. Merenda, C.D. Spielberger (Eds.) Adapting Educational and Psychological Tests for Cross-Cultural Assessment. Psychology Press.
Hambleton, R. K., & Zenisky, A. L. (2010). Translating and adapting tests for cross-cultural assessments. Cross-Cultural Research Methods in Psychology In D. Matsumoto & F. Van de Vijver (Eds.), Cross-Cultural Research Methods in Psychology (pp. 46-70). Cambridge: Cambridge University Press. doi:10.1017/CBO9780511779381.004
Hyytinen, H., & Toom, A. (2019). Developing a performance assessment task in the Finnish higher education context: Conceptual and empirical insights. British Journal of Educational Psychology, 89(3), 551–563. https://doi.org/10.1111/bjep.12283
Holtsch, D., Rohr-Mentele, S., Wenger, E., Eberle, F., & Shavelson, R. J. (2016). Challenges of a cross-national computer-based test adaptation. Empirical research in vocational education and training, 8(18), 1–32. https://doi.org/10.1186/s40461-016-0043-y
International Test Comission. (2017). The ITC Guidelines for Translating and Adapting Tests (Second edition). https://doi.org/10.1111/j.1464-0597.1975.tb00322.x
Keng, L., & Marion, S. (2020) Comparability of aggregated group scores on the “same test”. In A. I. Berman, E. H. Haertel, & J. W. Pellegrino (Eds.), Comparability of large-scale educational assessments: Issues and recommendations (pp. 205–225). Washington, DC: National Academy of Education. Retrieved from https://naeducation.org/wp-content/uploads/2020/05/Comparability-of-Large-Scale-Educational-Assessments_final.pdf
Liu, O. L., Frankel, L., & Roohr, K. C. (2014). Assessing critical thinking in higher education: current state and directions for next-generation assessments. ETS Res. Rep. Ser. 1, 1–23. https://doi.org/10.1002/ets2.12009
McClelland, D. C. (1973). Testing for competence rather than for “intelligence.” American Psychologist, 28(1), 1–14. https://doi.org/10.1037/h0034092
Mihailidis, P., & Thevenin, B. (2013). Media Literacy as a Core Competency for Engaged Citizenship in Participatory Democracy. American Behavioral Scientist, 57(11), 1611–1622. https://doi.org/10.1177/0002764213489015
Mislevy, R. J. (2018). Sociocognitive foundations of educational measurement. New York: Routledge. https://doi.org/10.4324/9781315871691
O’Leary, M., Reynolds, K., Guangming, L.; Ou, L.L., Belton, S., O’Reilly, N., & McKenna, J., (2020) Assessing Critical Thinking in Higher Education: Validity Evidence for the Use of the HEIghten™ Critical Thinking Test in Ireland. Journal of Higher Education Theory & Practice. Vol. 20 Issue 12, p115-130
Oser, F. K., & Biedermann, H. (2020). A three-level model for critical thinking: critical alertness, critical reflection, and critical analysis. In O. Zlatkin-Troitschanskaia (Eds.) Frontiers and Advances in Positive Learning in the Age of Information (PLATO). (pp. 89–106). Springer. https://doi.org/10.1007/978-3-030-26578-6_7
Perie, M. (2020) Comparability across different assessment systems. In A.I. Berman, E.H. Haertel, J.W. Pellegrino (Eds.), Comparability of Large-Scale Educational Assessments: Issues and Recommendations. National Academy of Education. Retrieved from https://naeducation.org/wp-content/uploads/2020/05/Comparability-of-Large-Scale-Educational-Assessments_final.pdf
Rogers, W. T., Gierl, M. J., Tardif, C., Lin, J., & Rinaldi, C. (2003). Differential Validity and Utility of Successive and Simultaneous Approaches to the Development of Equivalent Achievement Tests in French and English. Alberta Journal of Educational Research, 49(3), 290–304.
Rogers, W. T., Lin, J., & Rinaldi, C. (2011). Validity of the simultaneous approach to the development of equivalent achievement tests in English and French. Applied Measurement in Education, 24, 39-70.
Schendel, R., & Tolmie, A. (2017). Beyond translation: adapting a performance-task-based assessment of critical thinking ability for use in Rwanda. Assessment and Evaluation in Higher Education, 42(5), 673–689. https://doi.org/10.1080/02602938.2016.1177484.
Shavelson, R.J. (2010). Measuring college learning responsibly: Accountability in a new era. Stanford, CA: Stanford University Press.
Shavelson, R.J., Solano-Flores, G., & Kurpius, A. (2010) GS.4 Conceptual Framework. AHELO Module A: Adaptation and Translation of Performance Tasks. Council for Aid to Education.
Shavelson, R.J., Zlatkin-Troitschanskaia, O., Mariño, J. (2018). International Performance Assessment of Learning in Higher Education (iPAL): Research and Development. In: O. Zlatkin-Troitschanskaia, M. Toepper, H. Pant, C. Lautenbach, C. Kuhn (Eds.) Assessment of Learning Outcomes in Higher Education. Methodology of Educational Measurement and Assessment. Springer, Cham. https://doi.org/10.1007/978-3-319-74338-7_10
Shavelson, R.J., Zlatkin-Troitschanskaia, O., Beck, K., Schmidt, S., Mariño, J. (2019): Assessment of University Students’ Critical Thinking: Next Generation Performance Assessment, International Journal of Testing 19:4, 337-362. https://doi.org/10.1080/15305058.2018.1543309
Siegel, H. (2010). On Thinking Skills. In C. Winch (Eds). Teaching thinking skills (2nd ed.). New York, NY : Continuum International Pub. Group.
Sierci, S., & O’Riordan, M. (2020) Comparability when assessing individuals with disabilities. In A.I. Berman, E.H. Haertel, J.W. Pellegrino (Eds.), Comparability of Large-Scale Educational Assessments: Issues and Recommendations. National Academy of Education. Retrieved from https://naeducation.org/wp-content/uploads/2020/05/Comparability-of-Large-Scale-Educational-Assessments_final.pdf
Solano-Flores, G. (2011) Assessing the Cultural Validity of Assessment Practices: An Introduction. In M.R. Basterra, E. Trumbull, G. Solano-Flores. Cultural validity in assessment: Addressing linguistic and cultural diversity. NY: Routledge. https://doi.org/10.4324/9780203850954
Solano-Flores, G. (2012). Smarter Balanced Assessment Consortium: Translation accommodations framework for testing English language learners in mathematics.
Solano-Flores, G. (2019a). Examining Cultural Responsiveness in Large-Scale Assessment: The Matrix of Evidence for Validity Argumentation. Frontiers in Education, 4(June), 1–9. https://doi.org/10.3389/feduc.2019.00043 Retrieved from https://www.frontiersin.org/articles/10.3389/feduc.2019.00043/full
Solano-Flores, G. (2019b). The participation of Latin American Countries in International Assessments: Assessment Capacity, Validity, and Fairness. In, L. E. Suter, E. Smith & B. D. Denman, B. D.T (Eds.), Sage Handbook on Comparative Studies in Education: Practices and Experiences in student schooling and learning (pp. 139-161). Thousand Oaks, CA: Sage.
Solano-Flores, G., Javanovic, J., Shavelson, R. J., & Bachman, M. (1999). On the development and evaluation of a shell for generating science performance assessments. International Journal of Science Education, 21(3), 293–315. https://doi.org/10.1080/095006999290714
Solano-Flores, G., Trumbull, E., & Nelson-Barber, S. (2002). Concurrent Development of Dual Language Assessments: An Alternative to Translating Tests for Linguistic Minorities. International Journal of Testing, 2(2), 107–129. https://doi.org/10.1207/s15327574ijt0202_2
Solano-Flores, G., Backhoff, E., & Contreras-Niño, L. Á. (2009). Theory of Test Translation Error. International Journal of Testing, 9(2), 78–91. https://doi.org/10.1080/15305050902880835
Solano-Flores, G., Chía, M., Shavelson, R.J., & Kurpius, A. (2010) GS.36. Translation Guide. AHELO Module A. Council for Aid to Education.
Solano-Flores, G., Shade, C., & Chrzanowski, A. (2014). Smarter Balanced Assessment Consortium: Item Accessibility and Language Variation Conceptual Framework. https://portal.smarterbalanced.org/library/en/item-accessibility-and-language-variation-conceptual-framework.pdf
Survey Research Center, Institute for Social Research, University of Michigan (2016). Guidelines for Best Practice in Cross-Cultural Surveys. http://ccsg.isr.umich.edu/images/PDFs/CCSG_Full_Guidelines_2016_Version.pdf
Stecher, B. M., Klein, S. P., Solano-Flores, G., McCaffrey, D., Robyn, A., Shavelson, R. J., & Haertel, E. (2000). The Effects of Content, Format, and Inquiry Level on Science Performance Assessment Scores. Applied Measurement in Education, 13(2), 139–160. https://doi.org/10.1207/S15324818AME1302_2
Tremblay, K., Lalancette, D., & Roseveare, D. (2012). Assessment of higher education learning outcomes (AHELO): Rationale, challenges and initial insights from the feasibility study. OECD (Vol. 1). https://doi.org/10.1007/978-94-6091-867-4 http://www.oecd.org/education/skills-beyond-school/AHELOFSReportVolume1.pdf
van de Vijver, F. J. R., & Poortinga, Y. H. (2004). Conceptual and methodological issues in adapting tests. Adapting Educational and Psychological Tests for Cross-Cultural Assessment, 39–63. https://doi.org/10.4324/9781410611758
van de Vijver, F. J. R., & Poortinga, Y. H. (2016). On item pools, swimming pools, birds with webbed feet, and the professionalization of multilingual assessment. In C. S. Wells & M. Faulkner-Bond (Eds.), Educational measurement: From foundations to future (pp. 273–290). The Guilford Press.
van de Vijver, F. J. R. (2016). Assessment in education in multicultural populations. In G. T. L. Brown and L. Harris (Eds) Handbook of Human and Social Conditions of Assessment. (pp. 436–453). New York, NY: Routledge
Zhao, X., & Solano-Flores, G. (2020). Testing across languages in international comparisons: cultural adaptation of consensus-based test translation review procedures, Journal of Multilingual and Multicultural Development. https://doi.org/10.1080/01434632.2020.1852242
Zahner, D., & Ciolfi, A. (2018). International Comparison of a Performance-Based Assessment in Higher Education. In: O. Zlatkin-Troitschanskaia, M. Toepper, H.A. Pant, C. Lautenbach and C. Kuhn (Eds.), Assessment of Learning Outcomes in Higher Education Cross-National Comparisons and Perspectives. Springer. https://doi.org/10.1007/978-3-319-74338-7_11.
Zlatkin-Troitschanskaia, O., Toepper, M., Molerov, D., Buske, R., Brückner, S., Pant, H., Hofmann, S., & Hansen-Schirra, S. (2018). Adapting and Validating the Collegiate Learning Assessment to Measure Generic Academic Skills of Students in Germany: Implications for International Assessment Studies in Higher Education. In: O. Zlatkin-Troitschanskaia, M. Toepper, H.A. Pant, C. Lautenbach and C. Kuhn (Eds.), Assessment of Learning Outcomes in Higher Education Cross-National Comparisons and Perspectives. Springer. https://doi.org/10.1007/978-3-319-74338-7_12.