We examined the effect of two interventions on both the reliability and validity of regulatory ju... more We examined the effect of two interventions on both the reliability and validity of regulatory judgments: adjusting the regulatory instrument and attending a consensus meeting. We adjusted the regulatory instrument. With a randomized controlled trial (RCT) we examined the effect of the adjustments we made to the instrument. In the consensus meeting inspectors discussed cases and had to reach consensus about the order of the cases. We used a before and after case study to assess the effect of the consensus meeting. We compared the judgments assigned in the RCT with the unadjusted instrument with the judgments assigned with the unadjusted instrument after the consensus meeting. Moreover we explored the effect of increasing the number of inspectors per regulatory visit based on the estimates of the two interventions. The consensus meeting improved the agreement between inspectors; the variance between inspectors was smallest (0.03) and the reliability coefficient was highest (0.59). Va...
Many education systems are developing towards more lateral structures where schools collaborate i... more Many education systems are developing towards more lateral structures where schools collaborate in networks to improve and provide (inclusive) education. These structures call for bottom-up models of network evaluation and accountability instead of the current hierarchical arrangements where single schools are evaluated by a central agency. This paper builds on available research about network effectiveness to present evolving models of network evaluation. Network effectiveness can be defined as the achievement of positive network level outcomes that cannot be attained by individual organizational participants acting alone. Models of network evaluation need to take into account the relations between network members, the structure of the network, its processes and its internal mechanism to enforce norms in order to understand the achievement and outcomes of the network and how these may evolve over time. A range of suitable evaluation models are presented in this paper, as well as a tentative school inspection framework which is inspired by these models. The final section will present examples from Inspectorates of Education in Northern Ireland and Scotland who have developed newer inspection models to evaluate the effectiveness of a range of different networks.
Vier mythes over onderwijs. Rede uitgesproken bij het aanvaarden van het ambt van bijzonder hoogl... more Vier mythes over onderwijs. Rede uitgesproken bij het aanvaarden van het ambt van bijzonder hoogleraar Algemene Onderwijskunde aan de Inter-Continental University of the Caribbean op 19 maart 2015
Page 1. praktijkserie Basisschoo Page 2. Page 3. praktijkserie Basisschoo Prof. dr. Frans JG Jans... more Page 1. praktijkserie Basisschoo Page 2. Page 3. praktijkserie Basisschoo Prof. dr. Frans JG Janssens De onderwijsinspectie komt op bezoek 3).Kluwer a Wotters Kluwer business Page 4. Page 5. De onderwijsinspectie komt op bezoek Page 6. ...
APA PsycNET Our Apologies! - The following features are not available with your current Browser c... more APA PsycNET Our Apologies! - The following features are not available with your current Browser configuration. - alerts user that their session is about to expire - display, print, save, export, and email selected records - get My ...
Page 1. http://aje.sagepub.com/ American Journal of Evaluation http://aje.sagepub. com/content/30... more Page 1. http://aje.sagepub.com/ American Journal of Evaluation http://aje.sagepub. com/content/30/3/411 The online version of this article can be found at: DOI: 10.1177/1098214009341016 2009 30: 411 American Journal of ...
In the scientific literature about reliability, the main approach to increasing reliability seems... more In the scientific literature about reliability, the main approach to increasing reliability seems to involve increasing the number of observers and improving the instrument used. Other aspects for improving reliability - like the training of raters - seem to receive less notice. It is worth asking whether this technical approach could be complemented by training the user of the instrument. A systematic meta-analytical review of the research literature was performed to answer this question and examine the effectiveness of planned interventions for improving interrater reliability of health care professionals. The databases of PubMed (MEDLINE), Embase, Omega and PsycINFO were searched. The inclusion criteria were met by 57 studies. Details extracted from the studies included the study design, the number of observers and the number of observed cases, the intervention, the type of instrument (whether or not it was highly technical), and statistical information about the agreement before and after the intervention. Interventions were categorized into three groups: training of professionals, improving the diagnostic instrument and a combination of training and improving the instrument. A meta-analysis was performed by means of linear regression. The interventions were arranged according to their effectiveness in improving the diagnostic instrument (mean change: β = 0.13), training combined with improving the instrument (mean change: β = 0.10) and training (mean change: β = 0.09). On average, although all types of interventions are effective, improving the diagnostic instrument seems to be the most effective. Especially when highly technical instruments were concerned, improvement proved to be very effective (β = 0.52). Because instrumental variables constitute a major source of error, improving the instrument is an important approach. However, this review offers solid arguments that can complement the literature and practice, with a focus on training the user of the instrument.
Page 1. PEABODY JOURNAL OF EDUCATION, 76(3&4), 41-56 Copyright ? 2001, Lawrence Erlbaum A... more Page 1. PEABODY JOURNAL OF EDUCATION, 76(3&4), 41-56 Copyright ? 2001, Lawrence Erlbaum Associates, Inc. Schools Make a Difference, but Each Difference Is Different: On Dutch Schools and Educational Equality: Trends and Challenges ...
EJ850607 - Analyzing the Assumptions of a Policy Program: An Ex-Ante Evaluation of &a... more EJ850607 - Analyzing the Assumptions of a Policy Program: An Ex-Ante Evaluation of "Educational Governance" in the Netherlands.
We examined the effect of two interventions on both the reliability and validity of regulatory ju... more We examined the effect of two interventions on both the reliability and validity of regulatory judgments: adjusting the regulatory instrument and attending a consensus meeting. We adjusted the regulatory instrument. With a randomized controlled trial (RCT) we examined the effect of the adjustments we made to the instrument. In the consensus meeting inspectors discussed cases and had to reach consensus about the order of the cases. We used a before and after case study to assess the effect of the consensus meeting. We compared the judgments assigned in the RCT with the unadjusted instrument with the judgments assigned with the unadjusted instrument after the consensus meeting. Moreover we explored the effect of increasing the number of inspectors per regulatory visit based on the estimates of the two interventions. The consensus meeting improved the agreement between inspectors; the variance between inspectors was smallest (0.03) and the reliability coefficient was highest (0.59). Va...
Many education systems are developing towards more lateral structures where schools collaborate i... more Many education systems are developing towards more lateral structures where schools collaborate in networks to improve and provide (inclusive) education. These structures call for bottom-up models of network evaluation and accountability instead of the current hierarchical arrangements where single schools are evaluated by a central agency. This paper builds on available research about network effectiveness to present evolving models of network evaluation. Network effectiveness can be defined as the achievement of positive network level outcomes that cannot be attained by individual organizational participants acting alone. Models of network evaluation need to take into account the relations between network members, the structure of the network, its processes and its internal mechanism to enforce norms in order to understand the achievement and outcomes of the network and how these may evolve over time. A range of suitable evaluation models are presented in this paper, as well as a tentative school inspection framework which is inspired by these models. The final section will present examples from Inspectorates of Education in Northern Ireland and Scotland who have developed newer inspection models to evaluate the effectiveness of a range of different networks.
Vier mythes over onderwijs. Rede uitgesproken bij het aanvaarden van het ambt van bijzonder hoogl... more Vier mythes over onderwijs. Rede uitgesproken bij het aanvaarden van het ambt van bijzonder hoogleraar Algemene Onderwijskunde aan de Inter-Continental University of the Caribbean op 19 maart 2015
Page 1. praktijkserie Basisschoo Page 2. Page 3. praktijkserie Basisschoo Prof. dr. Frans JG Jans... more Page 1. praktijkserie Basisschoo Page 2. Page 3. praktijkserie Basisschoo Prof. dr. Frans JG Janssens De onderwijsinspectie komt op bezoek 3).Kluwer a Wotters Kluwer business Page 4. Page 5. De onderwijsinspectie komt op bezoek Page 6. ...
APA PsycNET Our Apologies! - The following features are not available with your current Browser c... more APA PsycNET Our Apologies! - The following features are not available with your current Browser configuration. - alerts user that their session is about to expire - display, print, save, export, and email selected records - get My ...
Page 1. http://aje.sagepub.com/ American Journal of Evaluation http://aje.sagepub. com/content/30... more Page 1. http://aje.sagepub.com/ American Journal of Evaluation http://aje.sagepub. com/content/30/3/411 The online version of this article can be found at: DOI: 10.1177/1098214009341016 2009 30: 411 American Journal of ...
In the scientific literature about reliability, the main approach to increasing reliability seems... more In the scientific literature about reliability, the main approach to increasing reliability seems to involve increasing the number of observers and improving the instrument used. Other aspects for improving reliability - like the training of raters - seem to receive less notice. It is worth asking whether this technical approach could be complemented by training the user of the instrument. A systematic meta-analytical review of the research literature was performed to answer this question and examine the effectiveness of planned interventions for improving interrater reliability of health care professionals. The databases of PubMed (MEDLINE), Embase, Omega and PsycINFO were searched. The inclusion criteria were met by 57 studies. Details extracted from the studies included the study design, the number of observers and the number of observed cases, the intervention, the type of instrument (whether or not it was highly technical), and statistical information about the agreement before and after the intervention. Interventions were categorized into three groups: training of professionals, improving the diagnostic instrument and a combination of training and improving the instrument. A meta-analysis was performed by means of linear regression. The interventions were arranged according to their effectiveness in improving the diagnostic instrument (mean change: β = 0.13), training combined with improving the instrument (mean change: β = 0.10) and training (mean change: β = 0.09). On average, although all types of interventions are effective, improving the diagnostic instrument seems to be the most effective. Especially when highly technical instruments were concerned, improvement proved to be very effective (β = 0.52). Because instrumental variables constitute a major source of error, improving the instrument is an important approach. However, this review offers solid arguments that can complement the literature and practice, with a focus on training the user of the instrument.
Page 1. PEABODY JOURNAL OF EDUCATION, 76(3&4), 41-56 Copyright ? 2001, Lawrence Erlbaum A... more Page 1. PEABODY JOURNAL OF EDUCATION, 76(3&4), 41-56 Copyright ? 2001, Lawrence Erlbaum Associates, Inc. Schools Make a Difference, but Each Difference Is Different: On Dutch Schools and Educational Equality: Trends and Challenges ...
EJ850607 - Analyzing the Assumptions of a Policy Program: An Ex-Ante Evaluation of &a... more EJ850607 - Analyzing the Assumptions of a Policy Program: An Ex-Ante Evaluation of "Educational Governance" in the Netherlands.
Uploads
Papers by Frans J G Janssens