Do Students Read Teacher Evaluation Surveys when Participation Incentives are Applied? An Empirical Approach
Keywords:
Educational quality, Teacher evaluation, Teacher effectiveness, Universities, QuestionnairesAbstract
The purpose of this study is to reveal the extent to which university read teacher evaluation surveys when participation incentives are applied. Researchers carry out a quantitative study, in which an experimental methodology with two groups is adopted. The first group performs the assessment of the teacher in a scenario free of incentives; the second completes the survey in an incentivized participation scenario. In addition, the research considers two types of questionnaires: on the one hand, Likert scales, on the other hand, scales with behavioral episodes or bars. The research uses descriptive analysis, Student´s t-test, and analysis of correlations through the Pearson correlation coefficient. The findings reveal differences in the investment of time when participation incentives are applied. It can be concluded that the instruments with Likert scales do not favor the correct reading and completion of surveys when the evaluation introduces rewards. However, this situation can be improved using bars questionnaires. The present study sheds light on a problem practically ignored by previous literature, but also introduces alternatives for improvement.
Downloads
References
Abrami, P. C. y D’Apollonia, S. (1997). Navigating student ratings of instruction. American Psychologist, 52(11), 1198-1208. Recuperado de http://psycnet.apa.org/buy/1997-43129-004
Ballantyne, C. (2003). Online evaluations of teaching: An examination of current practice and considerations for the future. New Directions for Teaching and Learning, 96, 103-112. https://doi.org/10.1002/tl.127
Bernardin, H. J. (1977). Behavioural expectation scales versus summated scales. Journal of Applied Psychology, 62(4), 422-427. Recuperado de http://psycnet.apa.org/record/1978-09104-001
Boring, A. (2017). Gender biases in student evaluations of teaching. Journal of Public Economics, 145, 27-41. https://doi.org/10.1016/j.jpubeco.2016.11.006
Buendía, L. (1997). La investigación por encuesta. En L. Buendía, P. Colás y F. Hernández Pina (Eds.), Métodos de investigación en psicopedagogía (pp. 120-154). Madrid: McGraw-Hill.
Buendía, L. (1994). El proceso de investigación. En L. Buendía y P. Colás (Eds.), Investigación educativa (pp. 69-108). Sevilla: Alfar.
Cañadas, I. y Cuétara, I. De. (2018). Estudio psicométrico y validación de un cuestionario para la evaluación del profesorado universitario de enseñanza a distancia. Revista de Estudios de Investigación en Psicología y Educación, 5(2), 102-112. https://doi.org/10.17979/reipe.2018.5.2.3701
Darwin, S. (2017). What contemporary work are student ratings actually doing in higher education? Studies in Educational Evaluation, 54, 13-21. https://doi.org/10.1016/j.stueduc.2016.08.002
De-Juanas Oliva, A. y Beltrán Llera, J. A. (2013). Valoraciones de los estudiantes de ciencias de la educación sobre la calidad de la docencia universitaria. Educación XX1, 17(1), 59-82. https://doi.org/10.5944/educxx1.17.1.10705
Dickinson, T. L. y Zellinger, P. M. (1980). A comparison of the behaviorally anchored rating and mixed standard scale formats. Journal of Applied Psychology, 65(2), 147-154. https://doi.org/10.1037//0021-9010.65.2.147
Dommeyer, C. J., Baum, P., Hanna, R. W. y Chapman, K. S. (2004). Gathering faculty teaching evaluations by in-class and online surveys: Their effects on response rates and evaluations. Assessment y Evaluation in Higher Education, 29(5), 611-623. https://doi.org/10.1080/02602930410001689171
Escobar-Pérez, J. y Cuervo-Martínez, Á. (2008). Validez de contenido y juicio de expertos: Una aproximación a su utilización. Avances en Medición, 6, 27-36.
Feistauer, D. y Richter, T. (2016). How reliable are students’ evaluations of teaching quality? A variance components approach. Assessment y Evaluation in Higher Education, 47(8), 1-17. https://doi.org/10.1080/02602938.2016.1261083
Feldman, K. A. (1978). Course characteristics and college students’ ratings of their teachers: What we know and what we don’t. Research in Higher Education, 9(3), 199-242. https://doi.org/10.1007/BF00976997
Fernández Millán, J. M. y Fernández Navas, M. (2013). Elaboración de una escala de evaluación de desempeño para educadores sociales en centros de protección de menores. Intangible Capital, 9(3), 571-589. https://doi.org/10.3926/ic.410
Ficapal-Cusí, P., Torrent-Sellens, J., Boada-Grau, J. y Sánchez-García, J.-C. (2013). Evaluación del e-learning en la formación para el empleo: Estructura factorial y fiabilidad. Revista de Educación, 361, 9-7. https://doi.org/10.4438/1988-592X-RE-2013-361-232
Franklin, J. (2001). Interpreting the numbers: Using a narrative to help others read student evaluations of your teaching accurately. New Directions for Teaching and Learning, 87, 85-100. https://doi.org/10.1002/tl.10001
Galbraith, C. S. y Merrill, G. B. (2012). Predicting student achievement in university-level business and economics classes: Peer observation of classroom instruction and student ratings of teaching effectiveness. College Teaching, 60(2), 48-55. https://doi.org/10.1080/87567555.2011.627896
Gannaway, D., Green, T. y Mertova, P. (2017). So how big is big? Investigating the impact of class size on ratings in student evaluation. Assessment y Evaluation in Higher Education, 8(2), 1-10. https://doi.org/10.1080/02602938.2017.1317327
George, D. y Mallery, P. (2003). SPSS for Windows step by step: A simple guide and reference. Los Ángeles, CA: Allyn and Bacon.
Griffin, B. W. (2004). Grading leniency, grade discrepancy, and student ratings of instruction. Contemporary Educational Psychology, 29(4), 410-425. https://doi.org/10.1016/J.CEDPSYCH.2003.11.001
Guzmán, J. C. (2018). Las buenas prácticas de enseñanza de los profesores de educación superior. REICE. Revista Iberoamericana sobre Calidad, Eficacia y Cambio en Educación, 16(2), 133-149. https://doi.org/10.15366/reice2018.16.2.008
Harari, O. y Zedeck, S. (1973). Development of behaviorally anchored scales for the evaluation of faculty teaching. Journal of Applied Psychology, 58(2), 261-265. https://doi.org/10.1037/h0035633
Hernández Pina, F. (1997). Diseños de investigación experimental. En L. Buendía, P. Colás y F. Hernández Pina (Eds.), Métodos de investigación en psicopedagogía (pp. 91-117). Madrid: McGraw-Hill.
Jacobs, R., Kafry, D. y Zedeck, S. (1980). Expectations of behaviorally anchored rating scales. Personnel Psychology, 33(3), 595-640. https://doi.org/10.1111/j.1744-6570.1980.tb00486.x
Johnson, T. D. (2003). Online student ratings: Will students respond? New Directions for Teaching and Learning, 96, 49-59. https://doi.org/doi: 10.1002/tl.122
Linse, A. R. (2017). Interpreting and using student ratings data: Guidance for faculty serving as administrators and on evaluation committees. Studies in Educational Evaluation, 54, 94-106. https://doi.org/10.1016/j.stueduc.2016.12.004
Lizasoain, L., Etxeberria, J. y Lukas, J. F. (2017). Propuesta de un nuevo cuestionario de evaluación de los profesores de la Universidad del País Vasco. Estudio psicométrico, dimensional y diferencial. RELIEVE. Revista Electrónica de Investigación y Evaluación Educativa, 23(1), 1-21. https://doi.org/10.7203/relieve.23.2.10436
Luna Serrano, E. (2015). Validación de constructo de un cuestionario de evaluación de la competencia docente. Revista Electronica de Investigación Educativa, 17(3), 27-45.
Marsh, W. (1982). SEEQ: A reliable, valid, and useful instrument for collecting students’ evaluations of university teaching. British Journal of Educational Psychology, 52(2), 77-95. https://doi.org/10.1111/j.2044-8279.1982.tb02505.x
Marsh, W. (1987). Students’ evaluations of university teaching: Research findings, methodological issues, and directions for future research. International Journal of Educational Research, 11(3), 253-388. https://doi.org/10.1016/0883-0355(87)90001-2
Marsh, W. (1991). A multidimensional perspective on students’ evaluations of teaching effectiveness-reply to Abrami and Dapollonia (1991). Journal of Educational Psychology, 83(3), 416-421. https://doi.org/10.1037//0022-0663.83.3.416
Martin-Raugh, M., Tannenbaum, R. J., Tocci, C. M. y Reese, C. (2016). Behaviourally anchored rating scales: An application for evaluating teaching practice. Teaching and Teacher Education, 59, 414-419. https://doi.org/10.1016/j.tate.2016.07.026
Matosas-López, L. y Leguey-Galán, S. (2018). Implementación de behavioral anchored rating scales (BARS) para la evaluación del profesorado universitario en asignaturas de modalidad online. En C. Monge López, P. Gómez Hernández y R. Herrero Marcos (Eds.), Actas del I Congreso Virtual Internacional y III Congreso Virtual Iberoamericano sobre Recursos Educativos Innovadores CIREI (pp. 204-208). Madrid: Fundación General de la Universidad de Alcalá.
Matosas-López, L., Aguado-Franco, J. C. y Gómez-Galán, J. (2019). Constructing an instrument with behavioral scales to assess teaching quality in blended learning modalities. Journal of New Approaches in Educational Research, 8(2).
Matosas-López, L., Leguey-Galán, S. y Leguey-Galán, S. (2019). Evaluación de la calidad y la eficiencia docente en el contexto de la educación superior: Alternativas de mejora. En J. Gómez-Galán, A. Martín-Padilla y H. Cobos (Ed.), La educación superior en el siglo XXI: Una mirada multidisciplinaria (pp. 240-257). Wheaton, IL: Editorial UMET.
Mayorga Fernández, M. J. y Ruiz Baeza, V. M. (2002). Muestreos utilizados en investigación educativa en España. RELIEVE. Revista Electrónica de Investigación y Evaluación Educativa, 8(2), 195-165.
McCann, S. y Gardner, C. (2014). Student personality differences are related to their responses on instructor evaluation forms. Assessment y Evaluation in Higher Education, 39(4), 1-15. https://doi.org/10.1080/02602938.2013.845647
McClain, L., Gulbis, A. y Hays, D. (2018). Honesty on student evaluations of teaching: Effectiveness, purpose, and timing matter! Assessment and Evaluation in Higher Education, 43(3), 369-385. https://doi.org/10.1080/02602938.2017.1350828
McPherson, M. A. (2006). Determinants of how students evaluate teachers. The Journal of Economic Education, 37(1), 3-20. https://doi.org/10.3200/JECE.37.1.3-20
Molero López-Barajas, D. M. y Ruiz Carrascosa, J. (2005). La evaluación de la docencia universitaria. Dimensiones y variables más relevantes. Revista de Investigación Educativa, 23(1), 57-84. Recuperado de http://revistas.um.es/rie/article/view/98341
Moreno Olivos, T. (2018). La evaluación docente en la universidad: Visiones de los alumnos. REICE. Revista Iberoamericana sobre Calidad, Eficacia y Cambio en Educación, 3(16), 87-102 . https://doi.org/10.15366/reice2018.16.3.005
Morley, D. D. (2012). Claims about the reliability of student evaluations of instruction: The ecological fallacy rides again. Studies in Educational Evaluation, 38(1), 15-20. https://doi.org/10.1016/j.stueduc.2012.01.001
Muñoz Cantero, J. M., Ríos De Deus, M. P. y Abalde Paz, E. (2002). Evaluación docente vs evaluación de la calidad. RELIEVE, 8(2), 103-134.
Nair, C. S. y Adams, P. (2009). Survey platform: A factor influencing online survey delivery and response rate. Quality in Higher Education, 15(3), 291-296. https://doi.org/10.1080/13538320903399091
Nasser-Abu Alhija, F. y Fresko, B. (2009). Student evaluation of instruction: What can be learned from students’ written comments? Studies in Educational Evaluation, 35(1), 37-44. https://doi.org/10.1016/j.stueduc.2009.01.002
Nulty, D. D. (2008). The adequacy of response rates to online and paper surveys: What can be done? Assessment y Evaluation in Higher Education, 33(3), 301-314. https://doi.org/10.1080/02602930701293231
Nygaard, C. y Belluigi, D. Z. (2011). A proposed methodology for contextualised evaluation in higher education. Assessment y Evaluation in Higher Education, 36(6), 657-671. https://doi.org/10.1080/02602931003650037
Reyero, D. (2014). La excelencia docente universitaria. Análisis y propuestas para una mejor evaluación del profesorado universitario. Educación XX1, 17(2), 125-143. https://doi.org/10.5944/educxx1.17.2.11482
Ruiz Carrascosa, J. (2000). La evaluación de la enseñanza por los alumnos en el plan nacional de evaluación de la calidad de las universidades. Construcción de un instrumento de valoración. Revista de Investigación Educativa, 18(2), 433-445.
Sharon, A. T. y Bartlett, C. J. (1969). Effect of instructional conditions in producing leniency on two types of rating scales. Personnel Psychology, 22(3), 251-263. https://doi.org/10.1111/j.1744-6570.1969.tb00330.x
Sorenson, D. L. y Reiner, C. (2003). Charting the uncharted seas of online student ratings of instruction. New Directions for Teaching and Learning, 96(1), 1-24. https://doi.org/10.1002/tl.118
Spooren, P. (2010). On the credibility of the judge. A cross-classified multilevel analysis on students’ evaluation of teaching. Studies in Educational Evaluation, 36(4), 121-131. https://doi.org/10.1016/j.stueduc.2011.02.001
Spooren, P., Mortelmans, D. y Christiaens, W. (2014). Assessing the validity and reliability of a quick scan for student’s evaluation of teaching. Results from confirmatory factor analysis and G theory. Studies in Educational Evaluation, 43, 88-94. https://doi.org/10.1016/j.stueduc.2014.03.001
Stanny, C. J. y Arruda, J. E. (2017). A comparison of student evaluations of teaching with online and paper-based administration. Scholarship of Teaching and Learning in Psychology, 3(3), 198-207. https://doi.org/10.1037/stl0000087
Stoskopf, C. H., Glik, D. C., Baker, S. L., Ciesla, J. R. y Cover, C. M. (1992). The reliability and construct validity of a behaviorally anchored rating scale used to measure nursing assistant performance. Evaluational Review, 16(3), 333-345.
Stowell, J. R., Addison, W. E. y Smith, J. L. (2012). Comparison of online and classroom-based student evaluations of instruction. Assessment y Evaluation in Higher Education, 37(4), 465-473. https://doi.org/10.1080/02602938.2010.545869
Tejedor Tejedor, F. J. (2009). Evaluación del profesorado universitario: Enfoque metodológico y algunas aportaciones de la investigación. Estudios sobre Educación, 16, 79-102.