Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

Interrogating evaluation 2015 induction

•
•

This is a short session interrogating the purpose and values of carrying out course evaluations in a university context.

1 of 21
Download to read offline
Interrogating Course Evaluation UWC Induction 20 March 2015 Rita Kizito - kizitorita@gmail.com
Aim of this part of the session • Interrogate ‘evaluation’ from own perspective • Come up with an evaluation strategy
Common Purpose of evaluation • Required by universities near end of each course * • Can be used by administration/departments as an important element in making decisions about promotion • Could be a source of great pride or trepidation (Cashin, 1999; Clayson, 2009)
Purposes of evaluation audit development “quality assurance” (Biggs, 2003; Edström, 2008; Patton, 1997) appraising teachers developing/ improving courses and teaching effectiveness “quality enhancement”
What should the focus of the evaluation be?
What should the focus of the evaluation be? • To check that something is working What is that something ?(e.g. course, degree programme, activity) What do we mean by ‘working’ ? (what goal should be achieved?) • To figure out how ‘it’ can be improved
What should the focus of the evaluation be? Teaching ( teacher) Learning (outcomes) Teaching (process) Learning (process) “Rather than ratings, teachers should be asked to include their course analyses in their teaching portfolio in order to show their ability to both analyze the student learning experience and the quality of the student learning outcomes , and to improve these with adequate course development measures” (Edström, 2008, p.104).
“Evaluation is often viewed as a test of effectiveness – of materials, teaching methods or whatnot - but this is the least important aspect of it. The most important is to provide intelligence on how to improve these things” What should the purpose of evaluation be ? ( Bruner, 1966, as cited in Ramsden, 2003, p. 233).
Typical questions • Have student attitudes changed? • Has the lecturer approaches changed? • Have student learnt something in class? • What is happening in class? • Have classroom practices changed? • Are students engaged? • Am I meeting students’ needs?
Evaluation strategies – student attitudes & perceptions Have student attitudes changed? • Likert-scale questions & surveys • Could be used to measure the effect of a course, a degree or a long term change over a number of years
Evaluation strategies – Lecturer approaches Have lecturer approaches to teaching changed? • Teaching Practices Inventories • Can help departments reflect on teaching, by allowing comparisons between different courses/departments
Evaluation strategies – Student learning Have students learned anything? • Measure student performance • Important to standardise testing (using locally developed or already developed tools) • Important to conduct pre and post-testing, before and after a learning intervention Is there evidence that the Graduate Attributes have been adopted by the students?
Evaluation strategies – classroom practices What is going on in the class? • Video recordings • Peer observations (fellow lecturers) using observation protocols • This data can be used in your Teaching Portfolio Have classroom practices changed? • Use of innovative teaching & learning practices • Providing useful feedback etc.
Evaluation strategies – student engagement Are students engaged? • Observation of about 10 students to see if they are engaged by recording student activity • What teaching & learning activities result in high levels of student engagement?
Evaluation strategies – student learning needs Are students learning needs being met? • Focus groups or Interviews • Anonymous surveys/midterm surveys (online, or on paper) – letter to the facilitator • Assignment by assignment survey (time it took, approach, resources used etc. ) • Keep, Start, Stop
Kirkpatrick’s (1994) four level model Level What is measured Examples 1 Reaction - changes in perception , satisfaction levels How students feel about the learning experience Feedback forms 2 Learning – changes in knowledge, skills, attitudes Increase in student knowledge and skills Informal/informal assessment before and after learning interventions 3 Changes in behavior/ practices How far learning is applied in practice resulting in personal changes Observations and interviews of students over time 4 Results - noticeable changes in results or conditions. How far the module/course impacts on program or institutional factors ( student performance, retention, throughput). Use institutional data to identify whether the program/module shifts the nature of student participation/ performance/engagement. Student surveys.
Some Drawbacks • Students are not always good at evaluating teaching effectiveness. Popularity is usually mistaken for good teaching • Response rate is always less than 100% • Susceptible to bias - voluntary participation/polarization • Questions are usually general –missing the finer details of practice
The UWC Evaluation Guideline document Instruments GENERIC EVALUATION DISCIPLINE SPECIFIC Question Bank A standard set of questions that students are invited to answer anonymously at the end of each course. ( often, the results are given weight in promotion and tenure decisions) Run centrally and completed by all students to provide the institution with data on how students are performing with regards to institutional strategies and plans
Your evaluation strategy Which evaluation methods could you consider implementing in your teaching?
References • Biggs, J. (2003). Teaching for quality learning at university: What the student does, Buckingham, UK: SRHE and Open University Press. • Cashin, W. E. (1999). Student ratings of teaching: uses and misuses. Ch 2, In P. Seldin and Associates (Eds). Changing practices in Evaluating Teaching: A practical guide to improved faculty performance and promotion/tenure decisions. Pp 25-44. Bolton, MA. Anker Publishing. • Clayson, D. E. (2009). Student evaluations of teaching: Are they related to what students learn? A meta-analysis and review of the literature. Journal of Marketing Education. 31 (1), 16-30. • Edström, K (2008), Doing course evaluation as if it matters most. Higher Education Research and Development, 27 :2: 95-106 • Kirkpatrick, D.L. (1994). Evaluating Training Programs. The Four Levels. San Francisco: Berrett- Koehler. • Ramsden, P (2003). Learning to teach in higher education ( 2nd edition). London: Routledge Falmer. • Schimpf, N (2015) Evidence-based Active Learning: Evaluating Classroom Practices. Presentation made at the University of the Western Cape
Thank you

More Related Content

Interrogating evaluation 2015 induction

  • 1. Interrogating Course Evaluation UWC Induction 20 March 2015 Rita Kizito - kizitorita@gmail.com
  • 2. Aim of this part of the session • Interrogate ‘evaluation’ from own perspective • Come up with an evaluation strategy
  • 3. Common Purpose of evaluation • Required by universities near end of each course * • Can be used by administration/departments as an important element in making decisions about promotion • Could be a source of great pride or trepidation (Cashin, 1999; Clayson, 2009)
  • 4. Purposes of evaluation audit development “quality assurance” (Biggs, 2003; Edström, 2008; Patton, 1997) appraising teachers developing/ improving courses and teaching effectiveness “quality enhancement”
  • 5. What should the focus of the evaluation be?
  • 6. What should the focus of the evaluation be? • To check that something is working What is that something ?(e.g. course, degree programme, activity) What do we mean by ‘working’ ? (what goal should be achieved?) • To figure out how ‘it’ can be improved
  • 7. What should the focus of the evaluation be? Teaching ( teacher) Learning (outcomes) Teaching (process) Learning (process) “Rather than ratings, teachers should be asked to include their course analyses in their teaching portfolio in order to show their ability to both analyze the student learning experience and the quality of the student learning outcomes , and to improve these with adequate course development measures” (Edström, 2008, p.104).
  • 8. “Evaluation is often viewed as a test of effectiveness – of materials, teaching methods or whatnot - but this is the least important aspect of it. The most important is to provide intelligence on how to improve these things” What should the purpose of evaluation be ? ( Bruner, 1966, as cited in Ramsden, 2003, p. 233).
  • 9. Typical questions • Have student attitudes changed? • Has the lecturer approaches changed? • Have student learnt something in class? • What is happening in class? • Have classroom practices changed? • Are students engaged? • Am I meeting students’ needs?
  • 10. Evaluation strategies – student attitudes & perceptions Have student attitudes changed? • Likert-scale questions & surveys • Could be used to measure the effect of a course, a degree or a long term change over a number of years
  • 11. Evaluation strategies – Lecturer approaches Have lecturer approaches to teaching changed? • Teaching Practices Inventories • Can help departments reflect on teaching, by allowing comparisons between different courses/departments
  • 12. Evaluation strategies – Student learning Have students learned anything? • Measure student performance • Important to standardise testing (using locally developed or already developed tools) • Important to conduct pre and post-testing, before and after a learning intervention Is there evidence that the Graduate Attributes have been adopted by the students?
  • 13. Evaluation strategies – classroom practices What is going on in the class? • Video recordings • Peer observations (fellow lecturers) using observation protocols • This data can be used in your Teaching Portfolio Have classroom practices changed? • Use of innovative teaching & learning practices • Providing useful feedback etc.
  • 14. Evaluation strategies – student engagement Are students engaged? • Observation of about 10 students to see if they are engaged by recording student activity • What teaching & learning activities result in high levels of student engagement?
  • 15. Evaluation strategies – student learning needs Are students learning needs being met? • Focus groups or Interviews • Anonymous surveys/midterm surveys (online, or on paper) – letter to the facilitator • Assignment by assignment survey (time it took, approach, resources used etc. ) • Keep, Start, Stop
  • 16. Kirkpatrick’s (1994) four level model Level What is measured Examples 1 Reaction - changes in perception , satisfaction levels How students feel about the learning experience Feedback forms 2 Learning – changes in knowledge, skills, attitudes Increase in student knowledge and skills Informal/informal assessment before and after learning interventions 3 Changes in behavior/ practices How far learning is applied in practice resulting in personal changes Observations and interviews of students over time 4 Results - noticeable changes in results or conditions. How far the module/course impacts on program or institutional factors ( student performance, retention, throughput). Use institutional data to identify whether the program/module shifts the nature of student participation/ performance/engagement. Student surveys.
  • 17. Some Drawbacks • Students are not always good at evaluating teaching effectiveness. Popularity is usually mistaken for good teaching • Response rate is always less than 100% • Susceptible to bias - voluntary participation/polarization • Questions are usually general –missing the finer details of practice
  • 18. The UWC Evaluation Guideline document Instruments GENERIC EVALUATION DISCIPLINE SPECIFIC Question Bank A standard set of questions that students are invited to answer anonymously at the end of each course. ( often, the results are given weight in promotion and tenure decisions) Run centrally and completed by all students to provide the institution with data on how students are performing with regards to institutional strategies and plans
  • 19. Your evaluation strategy Which evaluation methods could you consider implementing in your teaching?
  • 20. References • Biggs, J. (2003). Teaching for quality learning at university: What the student does, Buckingham, UK: SRHE and Open University Press. • Cashin, W. E. (1999). Student ratings of teaching: uses and misuses. Ch 2, In P. Seldin and Associates (Eds). Changing practices in Evaluating Teaching: A practical guide to improved faculty performance and promotion/tenure decisions. Pp 25-44. Bolton, MA. Anker Publishing. • Clayson, D. E. (2009). Student evaluations of teaching: Are they related to what students learn? A meta-analysis and review of the literature. Journal of Marketing Education. 31 (1), 16-30. • Edström, K (2008), Doing course evaluation as if it matters most. Higher Education Research and Development, 27 :2: 95-106 • Kirkpatrick, D.L. (1994). Evaluating Training Programs. The Four Levels. San Francisco: Berrett- Koehler. • Ramsden, P (2003). Learning to teach in higher education ( 2nd edition). London: Routledge Falmer. • Schimpf, N (2015) Evidence-based Active Learning: Evaluating Classroom Practices. Presentation made at the University of the Western Cape