44 The Evaluation of Doctoral Level “Development and Learning” and “Instructional Planning and Evaluation” Courses Nilay T. BÜMEN* Abstract The present study aimed to evaluate two doctoral level courses that are offered in Ege University. The study employed both quantitative and qualitative methods. Achievement tests and attitude scales were administered to the students in a one group pretest-posttest design. In terms of gender, 61% of the students were females and 39% were males. Semi-structured interviews were carried out with lecturers and selected students. Results showed that there were significant differences in achievement and attitude scores between the beginning and end of the semester. The students were satisfied with the lecturers in terms of instructional & communicational skills. However, they were not satisfied in terms of presentations and exams. Besides, they preferred the courses to be elective and mentioned that the courses should have been started at the beginning of the graduate programs, especially for research assistants and lecturers. Although the excessive course contents and the insufficiency of time were the areas of compliance among the lecturers, they considered that the courses reached their objectives in general. Key Words Lecturer Training, Curriculum Evaluation, Instructional Planning and Evaluation, Development and Learning * Correspondance: Assist. Prof. Dr. Nilay T. BÜMEN, Ege University Education Faculty, Department of Educational Sciences, Bornova Campus, 35040 Bornova, Izmir – Turkey. E-mail: nilay.bumen@ege.edu.tr Educational Sciences: Theory & Practice 6 (1) • January 2006 • 44-52 © 2005 E¤itim Dan›flmanl›¤› ve Araflt›rmalar› ‹letiflim Hizmetleri Tic. Ltd. fiti. (EDAM) BÜMEN / The Evaluation of Doctoral Level “Development and Learning” and “Instructional... • 45 Various suggestions to improve academic personnel have been made in the literature. However, there are limited studies related to field specific teaching activities and their validation. Particularly, clearly defining the services called “teaching” becomes harder. As the era, economic conditions, and academic personals’ values have been changed, so have the comments on how to improve academic personal (Orsmond & Stiles, 2002). Accordingly, a considerable effort has been allocated to creating performance indicators that address the three major functions of a university: research, service, and teaching and learning. Of the three functions, teaching and learning has received the most attention (Burke & Serban, 1998; Whiteley, Porter, & Fenske, 1992; cited in Alberto, Colbeck & Terenzini 2001). However, new faculty members lack the commitment and experience in this field. They are product of post graduate education that prepares them only to conduct research in an area of knowledge. This is an important task, but it does not prepare them for the full range of faculty responsibilities, specifically for teaching (Gaff, 1994). Furthermore, the new faculty members pass through a critical period for learning the job and forming attitudes about it during the initial years, they may easily move from a more liberal and idealistic perspectives to a more conventional and bureaucratic ones (Reynolds, 1992). From this point of view, Murray and Holmes (1997) noted that as long as lecturers in higher education are not trained, they will not be instructors. In order to establish effective faculty development programs, a step by step approach and an understanding of effective faculty development programs are required. Therefore, it is necessary to be informed about the models that have been developed so far. Examples of these models are the Concerns-Based Adoption Model (CBAM) (Loucks-Horsley & Stiegelbauer, 1970), Spark’s Model (Spark, 1983), RPTIM (Readiness, Planning, Training, Implementation and Maintenance) Model (Wood, 1989), and Lawler and King’s (2000) Adult Learning Model (cited in Akp›nar-Wilsing & Paykoç, 2004, p.72). When the principles and values underlie the instructional dimensions of these models are investigated, an “effective teaching” concept arises. Ramsden (1992) offered six key principles of an effective teaching in higher education: a clear explanation of complex subject matters; a conscientious consideration for students; an ap- 46 • EDUCATIONAL SCIENCES: THEORY&PRACTIGE propriate assessment and feedback; clear goals and intellectual challenge; student independence, control and active engagement; and learning from students. Various answers have been given to how to evaluate effective teaching. Research indicated that numerous methods have been used to evaluate the lecturers’ performances (Collins, 2002). These methods are: a) standardized questionnaires for students, b) evaluations made by the chair of the department, c) qualitative and quantitative evaluations made by the other lecturers in the same program, and d) evaluations made by the lecturers from different faculties or departments (Knutson, Schmidgall & Sciarini, 1996). Among these strategies, evaluation made by students is one of the most common methods. This method is eventually accepted as a self evaluation method of institution. Besides, many researchers implied that this method was the “only reliable” or “notable” evaluation method for evaluating lecturers (Seldin, 1984; Centra, 1977; Marsh, 1984; McKeachie, 1990; Ramsden, 1992; cited in Collins, 2002, p.82). Although, Aleamoni (1981) and Cohen (1980) indicated that evaluations made by students are reliable and are affected by grades; McKeachie (1979) proposed that students’ expectations are the most effective element in this type of an evaluation. Marsh’s (1987) review recognized that variables such as the size of the class, prior subject interest, workload, difficulty, and expected grade could have a small effect as much as 5% to 25% on student ratings. Moreover, Tagomori and Bishop (1995) examined 200 questionnaires used by accredited schools of education in the US and found that over 75% had items that were confusing, ambiguous, or subjective. They also found that a high proportion had problems with relatively straightforward issue such as item design suggests that few could have paid much attention to a more complex question of construct validity. Collins (2002) suggested that evaluation methods based on the students’ evaluations could not be used alone. Rather it should be jointed with other methods. Husbands (1997) found that there were variations in the ratings of individual teachers both within the same course by mode of teaching and among courses for the same teaching mode. Course characteristics appeared to have a greater effect on these discrepancies than teacher characteristics. The strongest effects were from enrolment size, the number of lecturers teaching a course, and general BÜMEN / The Evaluation of Doctoral Level “Development and Learning” and “Instructional... • 47 satisfaction with the course. Kember and Wong (2000) examined in the students’ perceptions of a good and poor teaching from interviews with 55 Hong Kong undergraduate university students. The interview transcripts suggested that the perceptions of teaching quality form as an interplay between the student’s conceptions of learning and their beliefs about the teaching of the lecturer. Students’ beliefs about learning can be placed on a continuum between passive and active learning. Their perceptions of the instructors’ beliefs about teaching range between transmissive and non-traditional teaching. The quality of teaching is conceived in four categories that are the quadrants formed by the intersections of the representations of beliefs about learning and perceptions of teaching. The quadrants are examined in turn to reveal how students with active and passive beliefs about learning could conceive quality in transmissive and non-traditional teaching. When we look at the teaching in higher education in Turkey, much attention has been given to “teacher role” of academic personal. Some approaches in this subject have discussed and started to be applied. For example, the Higher Education Council (2000) decided that all doctoral students take two pedagogical formation courses: “Instructional Planning and Evaluation” (IPE) and “Development and Learning” (DL). The aim of the present study was to evaluate the effectiveness of these pedagogical formation courses in Ege University. The study has two steps. Prior hypothesis for evaluating the effectiveness of the pedagogical formation courses were: a) There is a significant difference in the achievement scores between the pretest and posttest in DL courses. b) There is a significant difference in the achievement scores between the pretest and posttest in IPE courses. c) There is a significant difference in the attitude scores between the pretest and posttest in DL courses. d) There is a significant difference in the attitude scores between the pretest and posttest in IPE courses. To get more detailed and in depth view from the quantitative data, these research questions were investigated: e) What are the students’ opinions about effectiveness in DL courses? 48 • EDUCATIONAL SCIENCES: THEORY&PRACTIGE f) What are the students’ opinions about effectiveness in IPE courses? g) What are the lecturers’ opinions about the effectiveness in two courses? Method Both quantitative and qualitative research methods were employed to collect the data. The mixed procedures developed in response to a need to clarify the intent of mixing the quantitative and qualitative data in a single study. The concurrent triangulation strategy (Creswell, 2003) was chosen for a particular research approach. The achievement and attitude scales were administered with one group pretest-posttests design. To do this, achievement tests and attitude scales for each course were developed and their reliability and validity were computed. The reliability coefficient (KR-20) of the Instructional Planning and Evaluation course achievement test was 0.73, and the Development and Learning course achievement test was 0.75. The Cronbach reliability coefficients of the attitude scales were 0.95 for IPE, and 0.92 for DL. In terms of gender, 61% of the students were females and 39% were males. Semi-structured interviews were carried out with lecturers (n = 4) and selected students (n = 12). Interview guides for students and lecturers were prepared. At the end of the semester, interviews had been conducted. Before starting the interviews, the interviewees were informed about the purpose of the research and were reminded of the confidentiality of responses. All interviews were recorded with the permission of the participants and transcribed in two weeks. The transcriptions were later member-checked; that is, they were taken back to the participants for approval to increase the validity of the interviews. With the feedback received from the interviewees, necessary adjustments were made to the transcriptions. Because the researcher was the lecturer as well in these two courses, other interviewers executed the interviews with her students. During the study, quantitative data were analyzed using a t-test for paired samples in SPSS (Statistical Package for Social Sciences) version 10.0. The qualitative data were gathered and a content analysis technique (Y›ld›r›m & fiimflek, 2005) was used. The data were reviewed and the codes were attached to the chunks of meaningfully divided data. During this process, some certain themes proved to be unimportant and some new themes were emerged. Finally, exp- BÜMEN / The Evaluation of Doctoral Level “Development and Learning” and “Instructional... • 49 lanations were developed by looking at the patterns, contrasts, relationships, and respondents’ feedbacks. Results and Discussion According to the findings, there is a significant difference in achievement scores between the beginning and end of the semester. It was seen that the success level of the each course was around 65%. Similarly, there is a significant difference in the attitude scores between the pretest and posttest. So it can be claimed that these courses changed the students’ attitudes to a positive direction through the semester. These two quantitative results can be considered as indicators for attaining the course objectives in the semester. According to the comments made by the students, interviews were pleased with the lecturers about instructional and communicational skills and mentioned that courses were beneficial. However, they were not satisfied with presentations and examinations. The presentations in the courses were considered not only for usefulness but also for difficulty. Students preferred projects and/or homework rather than exams in doctorate levels. Besides, they preferred that these courses be compulsory for those who are tenure track. They think that for other doctoral students, these courses should be elective. They mentioned that these courses can be chosen in the beginning of the graduate program, especially for lecturers and research assistants. The course lecturers considered that the course contents were excessive and more time interval was necessary for practice. While there are different points of views about planning and projects suggested by the lecturers of the DL course, the IPE lecturer was pleased with the projects and assessments. However, they considered that the courses reached their objectives in general. On the other hand, three lecturers agreed that these courses be compulsory for doctoral students. All lecturers considered that timing of these courses is reasonable and if they were chosen by the students in early grades or graduate level, they would not be beneficial. In this research, problems that IPE lecturer faced during the course were consistent with the research by Yeler (2002). In his study, it is submitted that IPE course was having an extent with its current content, and insufficiency of causing students to gain subject content in 5 hours a week, and necessity of dividing content into a few courses filling each other in the gap. 50 • EDUCATIONAL SCIENCES: THEORY&PRACTIGE Some doctoral students (n = 3) mentioned that they had participated in the course reluctantly and that the lecturers had observed unwillingness on students in the beginning of the semester. These findings were supported the findings of Akp›nar-Wilsing and Paykoç (2004). According to the results of these studies, those who are in charge of subject research stated that they were not willing to take IPE courses due to it’s timing and overloaded work but they had to participate because the course was mandatory. In other words, they asserted that the timing of the course was not suitable in regards to their workload. But, the Higher Education Council suggested that these courses be taken after proficiency. Thus, taking this important suggestion into consideration may solve some problems related with timing. Besides, these courses are recognized as useful in terms of taking the job conditions of learners into consideration on adult education, learner’s participation in the decision making, and their motivation (Lawler & King, 2000, Galbraith, 1991). Therefore, preparing brochures and/or booklets related with the goals of DL and IPE courses, content and timing by cooperating with three Institutions, will be suitable and useful. Taking a closer look at universities which offer DL and IPE courses to doctoral students, it is observed that not all universities with the faculty of education provide such courses. After surfing their web sites on the internet, it is determined that merely eight of all universities provide such courses. It is also determined that performing decisive planning on lecturer training and efforts on solving such problems will be necessary. Students’ evaluations can be useful on permanent research and improvement of the effectiveness of DL and IPE courses. Research done at developed countries revealed that 60-70% of university students would like to evaluate the lecturers. Although, Öztürk (1999) and Yefliltafl & Öztürk (2000) emphasized the difficulty of using this method at Turkish State Universities, potential problems can be overcome by applying necessary adjustments. Not only this method improves the lecturers and effectiveness of the courses, but also it strengthens organizational development and stability, as well. BÜMEN / The Evaluation of Doctoral Level “Development and Learning” and “Instructional... • 51 Kaynakça / References Akp›nar-Wilsing, N., & Paykoç, F. (2004). Needs of future faculty members in relation to instructional planning, effective teaching and evaluation: A case study. E¤itim ve Bilim, 29 (133), 71-82. Alberto, F. C., Colbeck, C. L., & Terenzini, P. T. (2001). Developing performance indicators for assessing classroom teaching practices and student learning: The case of engineering. Research in Higher Education, 42 (3), 327-352. Aleomoni, L. M. (1981). Students ratings of instruction. In Millman, J. (Ed.) Handbook of Teacher Evaluation (pp. 110-145) Beverly Hills: Sage. Bryant F. B., & Yarnold, P. R. (1998). Principal-component analysis and exploratory and confirmatory factor analysis. In L.G. Grimm & P. R. Yarnold (Ed.), Reading and understanding multivariate statistics (pp. 99-137). Washington: American Psychological Association pub. Burke, J. C., & Serban, A. M. (1998). Performance funding for public higher education: Fad or trend? New directions for institutional research. San Francisco: Jossey-Bass. Büyüköztürk, fi. (2003). Sosyal bilimler için veri analizi el kitab›. Ankara: PegemA Yay›nc›l›k. Centra, J. (1977). Students raitings of instruction and their relationship to student learning. American Educational Research Journal, 14, 17-24. Cohen, P.A. (1980). Using student ratings feedback for improving college instruction: A meta-analyis of findings. Research in Higher Education. 13, 321-341. Collins, A. B. (2002). Üniversite ö¤rencileri ö¤retim elemanlar›n›n baflar›s›n› de¤erlendirebilir mi? ‹kilemler ve problemler. Ankara Üniversitesi E¤itim Bilimleri Fakültesi Dergisi, 35 (1-2), 81-91. Creswell, J. W. (2003). Research design: qualitative, quantitative, and mixed method approaches (2nd ed.). Thousand Oaks, CA: Sage. Çavdar, A. (2000). Üniversitelerde akademik yükseltmeler. Ankara: Tübitak Matbaas›. Devlet Planlama Teflkilat› (DPT). (2000). Sekizinci befl y›ll›k kalk›nma plan› yüksekö¤retim özel ihtisas komisyon raporu. Ankara. Erginer, A. (1997). Abant ‹zzet Baysal Üniversitesi E¤itim Fakültesi S›n›f Ö¤retmenli¤i bölümünde ö¤retim hizmeti veren ö¤retim elemanlar›n›n yeterlikleri. Yay›nlanmam›fl yüksek lisans tezi, Abant ‹zzet Baysal Üniversitesi Sosyal Bilimler Enstitüsü, Bolu. Galbraith, M. W. (Ed.) (1991). Adult learning methods: A guide for effective instruction. Malabar, Florida: Krieger Pub. Gaff, J. G. (1994). Faculty development. Liberal Education, 80 (4), 16-22. Hodkinson, S., & Taylor, A. (2002). Initiation rites: The case of new university lecturers. Innovations in Education and Teaching International, 39 (4), 256-264. Husbands, C. T. (1997). Variations in students’ evaluations of teachers’ lecturing in different courses on which they lecture: A study at the London School of Economics and Political Science. Higher Education, 33, 51–70. Kavak, Y. (1986). E¤itim fakültelerindeki ö¤retim elemanlar›n›n yeterlikleri ve e¤itim ihtiyaçlar›. Yay›nlanmam›fl doktora tezi, Hacettepe Üniversitesi Sosyal Bilimler Enstitüsü, Ankara. Kember, D., & Wong, A. (2000). Implications for evaluation from a study of students’ perceptions of good and poor teaching. Higher Education, 40, 69–97. Knutson, B., Schmidgall, S. R., & Sciarini, M. (1996). Teaching evaluations in CHRIE member schools: Perceptions of the faculty. Hospitality and Tourism Educator, 8 (4), 27-32. 52 • EDUCATIONAL SCIENCES: THEORY&PRACTIGE Lawler, P., & King, K. P. (2000). Planning for effective faculty development: Using adult learning strategies. Malabar, FL: Krieger. Marsh, H. W. (1984). Student evaluation of university teaching: Research findings, methodological issues, and directions for future research. International Journal of Educational Research, 11, 253-388. McKeachie, W. J. (1979). Students ratings of faculty: A reprise. Acedeme. 65, 384-397. McKeachie, W. J. (1990). Research on college teaching: The historical background. Journal of Educational Psychology, 82, 189-200. Murray, R., & Holmes, S. (1997). Partnerships in staff development: An institutional case study. Studies in Higher Education, 22 (1), 67-84. Orsmond, P., & Stiles, M. (2002). University teaching: A challenge to staff development. Innovations in Education and Teaching International, 39 (4), 253-255. Öztürk, Y. (1999). Ö¤retim elemanlar›n›n ders vermesinin de¤erlendirilmesinin kamu üniversitelerinde uygulanabilirli¤i. Millî E¤itim Dergisi, 141, 61-63. Patton, M. Q. (1990). Qualitative evaluation and research methods. Thousand Oaks, CA: Sage. Prosser, M., & Trigwell, K. (1999). Understanding learning and teaching: The experince of higher education. Buckingham: SRHE and Open University Press. Ramsden, P. (1992). Learning to teach in higher education. London: Kagan. Reynolds, A. (1992). Charting the changes in junior faculty. Journal of Higher Education, 63, 637-652. Rust, C. (2000). Do initial training courses have an impact on university teaching? The evidence from two evaluative studies of one course. Innovations in Education and Training International, 37 (3), 254-262. Seldin, P. (1984). Changing practices in faculty evaluation. San Francisco: Josey-Bass. Semerci, Ç. & Kara, A. (2004). E¤itim derslerinin doktora ö¤rencilerinin ö¤retmenlik tutumlar›na etkisi. E¤itim ve Bilim, 29 (131), 70-73. Tabachnick, B. G., & Fidell, L. S. (1996). Using multivariate statistics (3rd ed.) New York: Harper Collins College Publishers. Tagamori, H., & Bishop, L. (1995). Student evaluation of teaching: Flaws in the instruments’. Thought and Action: The National Education Association Higher Education Journal, 11, 63–78. Turgut, M. F. & Baykul, Y. (1992). Ölçekleme teknikleri. Ankara: ÖSYM Yay›nlar›. Whiteley, M. A., Porter, J. D., & Fenske, R. H. (1992). The premier for institutional research. Florida: Association for Institutional Research. Yeler, M. (2002). Ö¤retimde planlama ve de¤erlendirme dersine iliflkin ö¤renci görüflleri. Burdur E¤itim Fakültesi Dergisi, 3 (3), 131-142. Yefliltafl, M. & Öztürk, Y. (2000). Ö¤retim elemanlar›n›n ders vermedeki baflar›lar›n›n de¤erlendirilmesi sisteminin Türk kamu üniversitelerinde uygulanabilirli¤i üzerine bir araflt›rma. Hacettepe Üniversitesi E¤itim Fakültesi Dergisi, 19, 156-165. Y›ld›r›m, A. & fiimflek, H. (2005). Sosyal bilimlerde nitel araflt›rma yöntemleri. Ankara: Seçkin. Yüksek Ö¤retim Kanunu (2547 SK, 04.11.1981). Resmi Gazete. Say›: 17506, 06.11.1981. Yüksek Ö¤retim Kurulu (YÖK). (2000). Lisansüstü E¤itim. B.30.0.ATK.0.00.00.05/082607-6961, Ankara.