Keywords:
Academic competence, Domain-specific and generic skills, Tests, Assessments, Higher Education, Meta-analysisAbstract
In the 10-year research program "Modelling and Measuring Competencies in Higher Education (KoKoHs)" that ran from 2011 until 2020, more than 100 theoretically sound models and corresponding tests were developed to analyze the level and development of students’ competencies. Using these newly developed assessments, generic and domain-specific competencies of more than 75,000 students at more than 350 German universities in numerous disciplines (e.g., economics, teacher education) were measured in 39 interdisciplinary collaborative KoKoHs projects.
Method: In these measurements, a nationally and internationally unique database was created that contains information on the level, acquisition, and development of competencies, as well as corresponding personal and contextual influences, among students at different stages throughout their studies. A comprehensive meta-analysis of the students’ competencies as well as of the central factors influencing the acquisition of competencies in higher education had not been conducted so far. The existing KoKoHs data had only been analyzed in isolation within the individual KoKoHs projects until then. Accordingly, the obtained results related only to the individual project level. To address this research deficit, the meta-study presented in this paper conducted a research synthesis, i.e., a systematic and structured meta-analysis of the KoKoHs database.
Results: The meta-study describes the acquisition of competencies in higher education across domains and provides empirical evidence on generalizable versus specific influencing factors and, thus, evidence-based practical knowledge for higher education practice and policy.
Downloads
References
Adelman, C. (2009). The Bologna Process for U.S. Eyes: Re-learning Higher Education in the Age of Convergence. Institute for Higher Education Policy.
AERA (American Educational Research Association). (2011). Code of ethics. Educational Researcher, 40, 145–156.
AERA, APA, & NCME (2014). Standards for educational and psychological testing. AERA.
Behrendt, S., Dammann, E., ?tef?nic?, F., Markert, B. & Nickolaus, R. (2015). Physical-technical prior competencies of engineering students. Empirical Research in Vocational Education and Training, 7(2), DOI 10.1186/s40461-015-0013-9.
Berliner, D. C. (2002). Educational research: The hardest science of all. Educational Researcher, 31 (8), 18–20.
Biggs, J., & Tang, C. (2011). Teaching for Quality Learning at University. McGraw-Hill and Open University Press.
Blömeke, S., Gustafsson, J.-E., & Shavelson, R. (2015): Beyond dichotomies: Competence viewed as a continuum. Zeitschrift für Psychologie, 223 (1), 3–14.
Blossfeld, H.-P., Roßbach, H.-G., & von Maurice, J. (Eds.) (2011). Education as a Lifelong Process – The German National Educational Panel Study (NEPS). Zeitschrift für Erziehungswissenschaft: Sonderheft 14.
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2010). Introduction to meta-analysis. Wiley.
Brändle, T., & Häuberer, J. (2015). Social capital of non-traditional students at a German university. Do traditional and non-traditional students access different social resources? International Journal of Higher Education, 1 (4), 92–105.
Brückner, S., Förster, M., Zlatkin-Troitschanskaia, O., Happ, R., Walstad, W. B., Yamaoka, M., & Asano, T. (2015). Gender effects in assessment of economic knowledge and under-standing: Differences among undergraduate business and economics students in Germany, Japan, and the United States. Peabody Journal of Education, 90(4), 503-518.
Coates, H. (2014). Higher education learning outcomes assessment: International perspectives. Peter Lang.
DIHK (Association of German Chambers of Industry and Commerce) (2015). Ausbildung 2015. Ergebnisse einer DIHK-Online-Unternehmensbefragung. [Training 2015. Results of a DIHK online company survey.] Available online at https://www.dihk.de/ressourcen/downloads/dihk-ausbildungsumfrage-2015.pdf.
Gessler, M., & Siemer, C. (2020). Umbrella review: Methodological review of reviews published in peer-reviewed journals with a substantial focus on vocational education and training research. International Journal for Research in Vocational Education and Training, 7 (1), 91–125. https://doi.org/10.13152/IJRVET.7.1.5
Hammer, S., Carlson, S., Ehmke, T., Koch-Priewe, B., Köker, A., Ohm, U., Rosenbrock, S., & Schulze, N. (2015). Kompetenz von Lehramtsstudierenden in Deutsch als Zweitsprache. Zeitschrift für Pädagogik [Competence of student teachers in German as a second language. Journal for Pedagogy], 61, 32–54.
Happ, R., Nagel, M., Zlatkin-Troitschanskaia, O., & Schmidt, S. (2019). How migration back-ground affects master degree students’ knowledge of business and economics. Studies in Higher Education, 1–16. doi.org/10.1080/03075079.2019.1640670
HRK (German Rectors` Conference) (2012). Kompetenzorientierung im Studium. Vom Konzept zur Umsetzung. [Competence Orientation in Studies. From concept to implementation.] Available online at https://www.hrk-nexus.de/fileadmin/redaktion/hrk-nexus/07-Downloads/07-02-Publikationen/impulse_Onlineversion.pdf
Heublein, U. (2014). Student Drop-out from German Higher Education Institutions. European Journal of Education, 49 (4), 497–513.
Hunter, J. E., & Schmidt, E. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Sage.
Kaiser, G. & König, J. (2020). Analyses and Validation of Central Assessment Instruments of the Research Program TEDS-M. In O. Zlatkin-Troitschanskaia, H. A. Pant, M. Toepper & C. Lautenbach (Eds.), Student Learning in German Higher Education: Innovative Measurement Approaches and Research Results (pp. 29-52). Springer VS.
Kraitzek, A., Förster, M., & Zlatkin-Troitschanskaia, O. (2020). Influences on Master’s Degree Students’ Economic Knowledge. In O. Zlatkin-Troitschanskaia, H. A. Pant, M. Toepper & C. Lautenbach (Eds.), Student Learning in German Higher Education: Innovative Measurement Approaches and Research Results (pp. 401-429). Springer.
Kuhn, C., & Zlatkin-Troitschanskaia, O. (2011). Assessment of Competencies Among University Students and Graduates: Analyzing the State of Research and Perspectives (Working Papers Chair of Business Education Issue No. 59). Mainz: Johannes Gutenberg University.
Kuhn, C., Zlatkin-Troitschanskaia, O., Lindmeier, A., Jeschke, C., Saas, H., & Heinze, A. (2020). Relationships between Domain-Specifc Knowledge, Generic Attributes, and Instructional Skills – Results from a Comparative Study with Pre- and In-Service Teachers of Mathematics and Economics. In O. Zlatkin-Troitschanskaia, H. A. Pant, M. Toepper & C. Lautenbach (Eds.), Student Learning in German Higher Education: Innovative Measurement Approaches and Research Results (pp. 75-104). Springer.
Kühling-Thees, C., Happ, R., Zlatkin-Troitschanskaia, O. & Pant, H.A. (2020a). The Impact of Entry Preconditions on Student Dropout and Subject Change in Business and Economics. In O. Zlatkin-Troitschanskaia, H. A. Pant, M. Toepper & C. Lautenbach (Eds.), Student Learning in German Higher Education: Innovative Measurement Approaches and Research Results (pp. 351–370). Springer VS.
Lemmrich, S., Hecker, S.-L., Klein, S., Ehmke, T., Koch-Priewe, B., Köker, A. & Ohm, U. (2020). Linguistically Responsive Teaching in Multilingual Classrooms Development of a Perfor-mance-Oriented Test to Assess Teachers’ Competence. In O. Zlatkin-Troitschanskaia, H. A. Pant, M. Toepper & C. Lautenbach (Eds.), Student Learning in German Higher Educa-tion: Innovative Measurement Approaches and Research Results (pp. 125–140). Springer VS.
Lörz, M., Netz, N., & Quast, H. (2016). Why do students from underprivileged families less often intend to study abroad? Higher Education, 72 (2), 153–174.
Mislevy, R. J. (2016). How Developments in Psychology and Technology Challenge Validity Argumentation. Journal of Educational Measurement, 53 (3), 265–292.
Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003). On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1, 3–67.
Montgomery, C., & Rupp, A. (2005). A meta-analysis for exploring the diverse causes and effects of stress in teachers. Canadian Journal of Education, 28 (3), 461–488.
Müller, S., & Schneider, T. (2013). Educational pathways and dropout from higher education in Germany. Longitudinal and Life Course Studies, 4 (3), 218–241.
NRC (National Research Council) (2012). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas. The National Academics Press.
Oakley, A. (2017). Foreword. In D. Gough, S. Oliver and J. Thomas (Eds.), An Introduction to Systematic Reviews (pp. xiii–xvi). Sage.
OECD (Organisation for Economic Co-operation and Development) (2015). Education at a Glance 2015: OECD Indicators. OECD Publishing.
OECD (Organisation for Economic Co-operation and Development) (2012). Assessment of Higher Education Learning Outcomes. Feasibility Study Report: Volume 1. Design and Implementation. http://www.oecd.org/edu/skills-beyond-school/AHELOFSReportVolume1.pdf
OECD (Organisation for Economic Co-operation and Development) (2008). Measuring im-provements in learning outcomes: Best practices to assess the value-added of schools. OECD.
Pellegrino, J. W., Chudowsky, N., & Glaser, R. (2001). Knowing what students know: The science and design of educational assessment. National Academy Press.
Reimer, D., & Pollak, R. (2010). Educational expansion and its consequences for vertical and horizontal inequalities in access to higher education in West Germany. European Sociological Review, 26 (4), 415–430.
Reinhardt, F., Deribo, T., Zlatkin-Troitschanskaia, O., Happ, R., & Nell-Müller, S. (2020). The Influence of Refugee Students’ Personal Characteristics on Study Success in Online Education. Journal of International Migration and Integration, 1–22.
Roberts, B. W., Kuncel, N. R., Viechtbauer, W., & Bogg, T. (2007). Meta-analysis in personality psychology: A primer. In R. W. Robins, R. C. Fraley and R. F. Krueger (Eds.), Handbook of research methods in personality psychology (pp. 652-672). New York, NY, US: The Guilford Press.
Rychen, D. S., & Salganik, L. H. (Ed.). (2003). Key Competencies for a Successful Life and a Well-Functioning Society. Hogrefe & Huber.
Saas, H., Kuhn, C., & Zlatkin-Troitschanskaia, O. (2020). Ein videobasiertes Lehr-Lernformat als innovativer hochschuldidaktischer Ansatz in der wirtschaftspädagogischen Lehrerausbildung. Zeitschrift für Erziehungswissenschaft. [A video-based teaching-learning format as an innovative university didactic approach in business education teacher training. Journal for Educational Science.]
Schlax, J., Zlatkin-Troitschanskaia, O., Kühling-Thees, C., & Brückner, S. (2020). Influences on the Development of Economic Knowledge over the first Academic Year: Results of a Germany-Wide Longitudinal Study. In O. Zlatkin-Troitschanskaia, H. A. Pant, M. Toepper & C. Lautenbach (Eds.), Student Learning in German Higher Education: Innovative Measurement Approaches and Research Results (pp. 371-399). Springer.
Siebert-Ott, G., Fischbach, J., & Schindler, K. (2015). Akademische Textkompetenzen modellieren – Entwicklung und Beschreibung eines Kompetenzmodells für das Bewerten von Schülertexten. [Modelling Academic Text Competencies - Developing and describing a competency model for assessing student texts.] In B. Koch-Priewe, A. Köker, J. Seifried & E. Wuttke (Eds.), Kompetenzerwerb an Hochschulen: Modellierung und Messung [Competence acquisition at universities: Modelling and measurement] (pp. 129-151). Klinkhardt.
Vogelsang, C., Borowski, A., Kugelmeyer, C., Riese, J., Buschhüter, D., Enkrott, P., Kempin, M., Reinhold, P., Schecker, H., & Schröder, J. (2020). Development of Prospective Teachers’ Professional Knowledge and Skills during a One-Semester School Internship. In O. Zlatkin-Troitschanskaia, H. A. Pant, M. Toepper & C. Lautenbach (Eds.), Student Learning in German Higher Education: Innovative Measurement Approaches and Research Results (pp. 105-124). Springer.
Wagenaar, R. (2017). What do we know - What should we know? Measuring and comparing achievements of learning in European Higher Education: initiating the new CALO-HEE approach. In O. Zlatkin-Troitschanskaia, H.A. Pant, M. Toepper, C. Kuhn & C. Lautenbach (Eds.) (2018), Assessment of Learning Outcomes in Higher Education – Cross-national Comparisons and Perspectives. Springer.
Weinert, F. E. (2001). Concept of Competence: A Conceptual Clarification. In D. S. Rychen & L. H. Salganik (Eds.), Defining and Selecting Key Competencies (pp. 45-65). Hogrefe and Huber Publishers.
Wissenschaftsrat [German National Science Council] (2015). Empfehlungen zum Verhältnis von Hochschulbildung und Arbeitsmarkt. Zweiter Teil der Empfehlungen zur Qualifizierung von Fachkräften vor dem Hintergrund des demographischen Wandels. [Recommendations on the relationship between higher education and the labour market. Second part of the recommendations on the qualification of skilled workers against the background of demographic change.] Wissenschaftsrat. Available online at https://www.wissenschaftsrat.de/download/archiv/4925-15.pdf.
Wissenschaftsrat [German National Science Council] (2012). Prüfungsnoten an Hochschulen im Prüfungsjahr 2010. Arbeitsbericht. Mit einem Wissenschaftspolitischen Kommentar des Wissenschaftsrates. [Examination grades at universities in the examination year 2010. Working report. With a science policy commentary by the Science Council.] Wissenschaftsrat. Available online at https://www.wissenschaftsrat.de/download/archiv/2627-12.pdf.
Zlatkin-Troitschanskaia, O., Kuhn, C. & Toepper, M. (2014). Modelling and assessing higher education learning outcomes in Germany. In H. Coates (Ed.). Higher Education Learning Outcomes Assessment – International Perspectives (pp. 213-235). Peter Lang.
Zlatkin-Troitschanskaia, O., Pant, H. A. & Coates, H. (2016). Assessing Student Learning Out-comes in Higher Education – Challenges and International Perspectives. Assessment and Evaluation in Higher Education, 41 (5), 655-661.
Zlatkin-Troitschanskaia, O., Pant, H. A., Lautenbach, C., Molerov, D., Toepper, M. and Brückner, S. (2017a). Modeling and Measuring Competencies in Higher Education: Approaches to Challenges in Higher Education Policy and Practice. Springer VS.
Zlatkin-Troitschanskaia, O., Pant. H. A., Nagel, M.-T., Molerov, D., Lautenbach, C. and Toepper, M. (2020b). Portfolio of KoKoHs Assessments - Test Instruments for Modeling and Measuring Domain-specific and Generic Competencies of Higher Education Students and Graduates. pfalzdruck.
Zlatkin-Troitschanskaia, O., Pant, H. A., Toepper, M. and Lautenbach, C. (Eds.) (2020a). Student Learning in German Higher Education: Innovative Modelling and Measurement Approaches and Research Results. Springer VS.
Zlatkin-Troitschanskaia, O., Pant, H. A., Toepper, M., Lautenbach, C. and Molerov, D. (2017b). Valid Competency Assessment in Higher Education: Framework, Results, and Further Perspectives of the German Research Program KoKoHs. AERA Open, 3 (1), 1-22.
Zlatkin-Troitschanskaia, O., Shavelson, R. J. and Kuhn, C. (2015). The international state of re-search on measurement of competency in higher education. Studies in Higher Education, 40 (3), 393-411.
Zlatkin-Troitschanskaia, O., Toepper, M., Pant, H. A., Lautenbach, C. and Kuhn, C., (Eds.) (2018). Assessment of Learning Outcomes in Higher Education – Cross-national Comparisons and Perspectives. Springer VS.