Studies in Educational Evaluation 33 (2007) 101–119 www.elsevier.com/stueduc MATCHING INTERNAL AND EXTERNAL EVALUATION IN AN ERA OF ACCOUNTABILITY AND SCHOOL DEVELOPMENT: LESSONS FROM A FLEMISH PERSPECTIVE Jan Vanhoof and Peter Van Petegem Institute of Education and Information Sciences Antwerp University, Belgium Abstract Contributions on quality assurance in education often argue that a complementary and integrated relation between internal and external evaluation of schools would be advisable. This article sets out to investigate what role internal evaluation should - or could - fulfil in relation to external evaluation. It concludes that the arguments for an integration of both are overly positive. The question as to whether accountability and school improvement can be reconciled is indeed a complex one and requires a carefully qualified answer. Looking at some Flemish experiences, we set out a scenario that provokes reflection. Introduction The relationship between the internal and external evaluation of educational institutions has long been a subject of much discussion with regard to the subject of quality assurance in education. External evaluations in the form of school inspections appear always to have enjoyed a somewhat bizarre existence. While everyone accepts the need for them, the way in which they have been carried out has always been heavily criticised (Norton Grubb, 1999; Nevo, 2001). For this reason - and also in response to recent trends with regard to decentralization and increasing autonomy for schools - evaluation methods have been developed in many countries which permit more participatory and self-directed 0191-491X/04/$ – see front matter # 2006 Published by Elsevier Ltd. doi:10.1016/j.stueduc.2007.04.001 102 J. Vanhoof, P.V. Petegem / Studies in Educational Evaluation 33 (2007) 101–119 forms of evaluation (McNamara & O'Hara, 2005; Robinson & Cousins, 2004). In the last ten years self-evaluation has also become commonplace in many schools in Flanders (Belgium). Educational umbrella organisations, universities and schools themselves have developed tools and methodologies to enable schools to evaluate educational quality autonomously. However, although the existence of self-evaluations is welcomed, the quality of these self-evaluations is nonetheless open to question (Cousins & Earl, 1995; Forss, Cracknell, & Samset, 1994; MVG, 2006; Van Petegem, Verhoeven, Buvens, & Vanhoof, 2005). As long as schools are financed or subsidized by governments, forms of monitoring will always have to be devised and schools will have to render account. The form this ought to take is currently subject to considerable change, however (Van Petegem, Vanhoof, Daems & Mahieu, 2005). At present the primary responsibility for educational quality lies with the school. The withdrawal of government is making schools increasingly autonomous in terms of their freedom to formulate and conduct their own policy. In return for this autonomy, schools are being required to evaluate their own educational quality and to come up with their own plans for improvement. The responsibilities for quality assurance in education are spread across various partners. The parallel existence of these responsibilities has led the government, the educational inspectorate and schools to look for a way in which internal and external evaluations can be matched with each other. This is the subject of this article. The search for an appropriate balance between internal and external evaluation is underway in multiple locations and in many countries. We would like to propose one possible scenario given that specific interpretations are at present somewhat thin on the ground (Christie, Ross, & Klein, 2004). The description of, and reflection upon, the Flemish situation will serve as an instructive case study illustrating this international problem. Quality Assurance: Different Responses to Differing Expectations Quality assurance implies having an idea of what quality involves. It would therefore be inappropriate to approach this topic without first exploring what we mean by educational quality. In the following section we will attempt to arrive at an operational definition of quality. Educational Quality: A Variety of Expectations We will arrive at our interpretation of the concept of quality step by step. Although the following reflection might seem somewhat redundant at first sight, it will become apparent later on in this article that the definition of this concept is crucial to sketching out scenarios for the role of self-evaluation and school audits in a (Flemish) quality assurance system. Our point of departure is that: “quality means complying with expectations” (Harrington & Harrington, 1994). A school can be said to be functioning optimally (in other words, to be providing good quality education) when it meets expectations. These expectations may be dictated by the government, the school board, parents, pupils and, of course, also by the school management and the teaching staff of the school itself. Examples J. Vanhoof, P.V. Petegem / Studies in Educational Evaluation 33 (2007) 101–119 103 of expectations are: raising pupil enrolment; achievement in mathematics literacy; excellence and guaranteeing equal opportunities. These expectations come from both internal as well as external stakeholders. Although they are often parallel or complementary, they can also be contradictory. In the first instance, the expectations refer to the attainment of pre-set objectives. Quality is thus, first and foremost, meeting expectations in terms of results (i.e., effectiveness). However, effectiveness alone does not amount to quality. How these results are reached is also of importance. It is, therefore, impossible to judge the quality of a school without taking into account the entire process which leads to the attainment of objectives. Efficiency, for example, is an integral part of the expectations drawn up by the various stakeholders (i.e., value for money), but it is to be clearly distinguished from expectations in terms of results. The various stakeholders may also have clear expectations with regard to pupil-teacher interactions or the didactic approach used. Expectations are often not explicitly stated, however. The expectations of all stakeholders must, therefore, be ascertained and discussed. Quality is thus not a characteristic of - or a task for - school management and teaching staff alone. Quality of education is something on which the government, the school board, school management, teaching staff, pupils, parents, the business world and higher education must agree through dialogue. Quality is related to all levels within the school and to all stakeholders (Hendriks, Doolaard, & Bosker, 2002). It is not a sum total of all expectations, but rather the integration and assessment of a school-specific whole. This means that a useful definition of educational quality must, of necessity, be separated from specific content and requires an abstract description. This brings us to the following definition of quality: (educational) quality is meeting the expectations shared by the stakeholders in an appropriate manner. The (shared) expectations are, however, of a diverse nature and with view to the interpretation of quality assurance/care it is important to clearly distinguish their respective origins. We distinguish between school-internal and school-external expectations. Within the group of school-external expectations we make a further distinction between expectations which are anchored in the law and expectations which are not legally anchored. Expectations which are anchored in the law are those expectations which the government (acting on behalf of society) imposes on schools and which can be forcibly imposed. If schools fail to meet these expectations the government can- and must - take steps to sanction these schools. School-external expectations which are not legally anchored derive from stakeholders whose expectations are not legally laid down. This group of stakeholders is extremely broad and still growing (e.g., schools to which pupils move on; recipients of pupils who have successfully obtained their graduation diplomas; parents (and parents associations); pastoral services and a very wide variety of interest groups and organisations, such as the Flemish cyclists association). These stakeholders may well make demands of schools and/or participate in decision-making with regard to how educational quality is to be determined, but they do not have the power to sanction, as the government has. Finally there are school-internal expectations. These are those aspects which the school emphasizes over and beyond the minimum requirements expected by society (Van Petegem, 1999). These expectations may, for example, be philosophical in nature (reflecting a particular life vision) or may relate to the didactic methods used. 104 J. Vanhoof, P.V. Petegem / Studies in Educational Evaluation 33 (2007) 101–119 Each type of expectation requires its own interpretation of quality assurance and of the role of self-evaluation and school audits in the quality assurance system as a whole. These expectations are highly diverse (Alkin, 2004). Quality Assurance as Response Quality assurance is an umbrella concept which covers all activities undertaken to investigate, monitor, improve - and perhaps also even to make public - the quality of schools. The difference between internal and external quality assurance essentially comes down to the question of who bears responsibility (Nevo, 2001; Scriven, 1991). If these activities are undertaken by the school itself, we are dealing with internal quality assurance. Internal quality assurance means that the monitoring, development and improvement of educational quality takes place within the school, whereas, in the case of external quality assurance the initiative for undertaking quality assurance activities lies with persons or institutions which are outside the school (e.g., the educational inspectorate or an accreditation institution) (Nevo, 2001). External evaluations by the inspectorate usually focus on policy, legislation and regulations, and educational performance (i.e., the statutory expectations). Internal evaluation can in principle concern itself with whatever topic the school believes to be important. Many of the activities which schools undertake in the context of internal quality assurance are of a self-evaluatory nature. Van Petegem describes self-evaluation as: (…) the process, largely initiated by the school itself, whereby carefully chosen participants describe and evaluate the functioning of the school in a systematic manner for the purposes of taking decisions or undertaking initiatives in the context of [aspects of] overall school [policy] development. (2005, p. 104) Self-evaluation is therefore not an end in itself, but – as described here – it is explicitly related to school development and pupil learning. The article sets out to investigate – within the Flemish context – what role internal evaluation should – or could – fulfil in relation to external evaluation. In approaching this, however, it is important to appreciate that there are two bodies, in addition to the schools themselves, which play an important role in the Flemish quality assurance system. Since the royal decree of 17 July 1991, the principal monitoring functions have been entrusted to the educational inspectorate and an advisory and supporting role has been given to the educational support services. The inspectorate verifies by means of school audits that the school is complying with its social task and that community monies are being used in a responsible manner. On the basis of this, the inspectorate submits a recommendation as to whether or not schools should be recognized or subsidized. The pedagogical supervisory services are related to the school networks and support teaching staff, school managers and school administrators in the achievement of the pedagogical plan. J. Vanhoof, P.V. Petegem / Studies in Educational Evaluation 33 (2007) 101–119 105 Perspectives on Quality Assurance There are a variety of possible arguments for integrating internal and external evaluation. Recent trends towards the emphasis of the combination of internal and external evaluation originate in the demands for school accountability and school-internal quality development. For the sake of conceptual clarity this distinction between an accountability perspective and a school improvement perspective is briefly explained below. Thereafter, we will investigate the role of self-evaluation in quality assurance as a whole. Broadly speaking, two perspectives can be distinguished in the area of quality assurance: the accountability perspective and the school improvement perspective (focused on improvement) (Nisbet, 1988; Cheng, 1996). The distinction between the two perspectives is based on different answers to the questions of (1) whether quality assurance is primarily concerned with monitoring and accountability or rather with development and improvement and (2) who determines quality of education, in other words: the government or the school itself. In the accountability perspective the central focus is on the exercise of control. Schools are expected to achieve pre-determined objectives. These objectives are imposed by the government (i.e., society) and have been established within a legislative framework. It should be noted that they do not necessarily have to bear any relation to the output of schools. Furthermore, the government may also formulate expectations regarding educational processes (e.g., attention for participatory thinking). In this perspective, quality assurance is externally directed and focused on uniformity, the attainment of minimum objectives and an efficient functioning. This takes place, in the first instance, at the initiative of school-external actors. In the case of the school improvement perspective, on the other hand, the starting point is a shared vision of a desired educational quality. The prime concern here is constituted by aims with regard to output and processes to be defined by the school itself. In this case, quality assurance is focused on the differences between schools, teaching staff and pupils and the intention is to initiate a dialogue. Quality assurance takes place, in the first instance, on the initiative of school-internal actors and factors. Until recently the accountability perspective was the exclusive terrain of external evaluation and the school improvement perspective of internal evaluation. This distinction is, however, gradually becoming blurred, indicating that the traditional relationship between internal and external evaluation needs to be rethought. Self-Evaluation: Accountability and/or School Improvement? The question as to whether accountability and school improvement can be reconciled is a complex one and requires a carefully qualified answer. We will begin by examining the strong arguments in favour of integrating internal and external evaluations, and we will then go on to examine a number of observations in relation to integration which indicate the need for close critical examination. 106 J. Vanhoof, P.V. Petegem / Studies in Educational Evaluation 33 (2007) 101–119 Self-Evaluation: Accountability and School Improvement As we will demonstrate below, the perspectives of accountability and school improvement each require a different interpretation of the relationship between internal and external evaluation. In the accountability perspective, internal evaluation is subordinate to the external evaluation, whereas in the school development perspective the reverse is true. In other words, we are offering here a Flemish interpretation of the advantages on both sides suggested by Nevo (2001). Accountability: Internal Evaluation in the Service of External Evaluation In the general conclusions of Van Hoyweghen’s “self-evaluation project” we read that: school self-evaluation activities appear to be capable of forming a good basis for external school audits. In monitoring quality, internal and external evaluation can be complementary, but external evaluation remains necessary by reason of its legitimizing function (2002, p. 16]. We distinguish three roles that internal evaluation may take to support external evaluation: a scope-broadening role, an interpretation-fostering role and an implementing role. A scope-broadening role. External evaluations are sometimes criticized because in aiming at comparability and generalizability they end up by focusing on aspects which all schools have in common. Consequently, there is a danger that aspects revealed will only be the most obvious ones and there is an additional danger that local needs and priorities will be ignored (Nevo, 2001). This is a problem for a government which allows its policy to be inspired by the results of audits. Internal evaluations can play a scope-broadening role here (Christie et al., 2004). It appears that an external evaluation can be conducted with greater depth and higher quality when school self-evaluation is involved in the audit (Van Hoyweghen, 2002). The participants in the school self-evaluations are usually more familiar with the specific nature of the local school context and communicate better with the (local) school community. By focusing attention on additional data which reveal the unique character of particular schools, they can broaden the focus of external evaluations. An interpretation-promoting role. An accountability perspective usually operates from a criterion-oriented and norm-focused frame of reference when evaluating the quality of schools. These are important and legitimate angles of approach in themselves, but have the disadvantage that the local perspective is lost. In order to correctly interpret the findings of an external evaluation, it is recommended that a school-focused frame of reference also be used. Specifically, the results of a self-evaluation can play a productive role in the interpretation of data collected during an audit. They add a local perspective to the interpretation (Nevo, 2001). An implementing role. A great deal is also expected from the integration of selfevaluation and external evaluation with regard to the relationship between schools and the inspectorate: J. Vanhoof, P.V. Petegem / Studies in Educational Evaluation 33 (2007) 101–119 107 With the preservation of everyone’s responsibility and independent judgement formation on the part of the inspectorate, both inspectors and schools – by the more active involvement of the schools – can interact on a more equal basis during the audit process. (Van Hoyweghen, 2002) Schools with experience in self-evaluation have a greater chance of adopting a constructive attitude with regard to school audits and to make more productive use of the results of an external evaluation (Nevo, 2002). Dealing with data from external evaluation after all requires skills from schools which they cannot simply be assumed to have (such as interpretation of data; looking for explanations of findings; and the application of the results in policy discussions) (Van Petegem, Verhoeven, et al., 2005). The impact of the inspection reports, for example, appears to be enhanced if the audit is combined with some form of internal evaluation. Moreover, an internal evaluator provokes less opposition and remains in place after the evaluation in order to implement the proposed action (Christie et al., 2004; Love, 1991). School Development: External Evaluation in the Service of Internal Evaluation As we have seen above, internal evaluation can be of service to external evaluation. However, self-evaluation is first and foremost an activity aimed at helping create the foundation - on a school-internal basis - for a particular dynamic of development, modification and renewal. That said, however, external evaluation can also play a variety of roles in the internal evaluation of schools. A scope-broadening role. Schools are increasingly expected to formulate their own policy and self-evaluation plays an important role in this. School development can only take place, however, in the presence of other and broader ranges of vision which go beyond the individual and the subjective. In the event of a lack of distance and a lack of objectivity, a school can suffer from organizational blindness (McBeath & Myers, 2002). Apart from the control perspective external evaluation clearly has an additional task, namely, providing feedback on the functioning of the school and providing fresh, challenging and stimulating ideas. The bodies responsible for conducting the external evaluation can thus bring greater depth and breadth to school self-evaluation. A stimulating role. Schools only undergo external evaluation sporadically. The existing interpretation of external evaluation in Flanders appears at presentto fail, both in terms of content and organization, in offering sufficient guarantees with respect to school development. That does not mean, however, that an external evaluation cannot serve as a motivation for the initiation of an internal quality assurance campaign within the school (Nevo, 2002). It cannot be assumed that all schools will automatically conduct selfevaluations and the role of external evaluation can therefore also be that of acting as a prompt towards doing so. A legitimizing role. Internal evaluation can also serve - in itself - as a point of departure to hold schools accountable towards parents, the local community and members of the school team. The data gathered during the self-evaluation appear at first sight to be of clear value in this respect. That said, however, the credibility of these data is open to 108 J. Vanhoof, P.V. Petegem / Studies in Educational Evaluation 33 (2007) 101–119 serious reservations in the absence of a link to some external form of evaluation (Nevo, 2001). Although validated tools do exist in order to guarantee systematic data collection in the conduct of a self-evaluation, accusations or suspicions – whether intentional or not – of subjectivity are always a risk (Scriven, 1991). As a consequence, the results of selfevaluation tend to be greeted with a degree of scepticism and are often dismissed as invalid and unreliable. In this way, the self-evaluation misses its objectives of contributing to school development. External evaluation can play a role in counter-acting this in the sense that external evaluation can legitimize the results of self-evaluation by confirming their validity and reliability (Christie et al., 2004). From the above we can conclude that internal and external forms of evaluation ought to fulfil integrated roles in quality assurance. Each form of evaluation can be of service to the other. According to this logic the presence of one form of evaluation thus appears to enhance the value of the other form of evaluation. The most suitable relationship between internal and external evaluation is thus not a mere adding-up of two separate aspects. Instead, thinking in terms of multiplication seems more appropriate. This means not only that these aspects can strengthen each other when combined, but also that if one is absent, the other loses value. Self-Evaluation: Accountability or School Improvement There are indications, however, that the accountability and school improvement described positive. Working towards accountability and school it might appear at first sight. This has to do with functions. arguments for an integration of above may be overly and naïvely development is not as self-evident as the presence of different evaluatory The Formative and Summative Functions of Evaluation The argument above was based in large measure on an assumption that there is a desire on the part of the government and of schools to understand what goes on in schools and what needs to be improved. The objective here would thus be to a large extent formative: suitable information and feedback is sought with a view to policy development. On the basis of formative evaluation data it is possible to remedy when necessary and provide additional supervision (McBeath & Myers, 2002). This is apparent, for example, from the proposition that internal evaluation can be a scope-broadening and interpretationfostering complementary element within the accountability perspective in external evaluation. Thanks to the self-evaluations, the government can obtain a greater quantity of valid data and interpretations on which to base its policy. Evaluation also has a summative function, however (Scriven, 1967). This summative function includes the evaluation of schools with a view to the question as to whether they are eligible to be financed or subsidized by the government. The objective here is not giving feedback, but rather determining results. The most important task of the inspection team responsible for an audit is that of submitting an opinion with regard to the recognition and subsidizing of the school. At the end of the day these external evaluations are therefore clearly summative and it is here that the somewhat rosy picture painted above J. Vanhoof, P.V. Petegem / Studies in Educational Evaluation 33 (2007) 101–119 109 and supported by the advocates of integration comes up against a less than pastel-shaded reality. When the function of external evaluation is in large measure summative, using selfevaluation for both accountability purposes as well as school improvement becomes more complicated. When schools are aware of the summative nature of an evaluation or selfevaluation, side-effects arise which immediately reduce the chances of the successful achievement of formative functions. Summative functions and obligation always involve the danger that the justification and account- rendering aspect may predominate at the expense of desire for improvement. It is, for example, quite possible in schools which fear a negative evaluation that there may be instances of putting on a show, window-dressing and spin. There is also a very real risk of self-serving results (Watling & Arlow, 2002). Van Petegem (1999, p. 30) refers to a “strategic” use of self-evaluation. The prime aim of schools in a summative evaluation is, after all, to present themselves as positive as possible. As a consequence, any readiness to reflect critically on their own functioning – a precondition where school development is concerned – is to a large extent eradicated. School development leans heavily on the concept of “organizational learning”. Two findings in the field of human learning and reflection behaviour shed an interesting light on the problematics of the incompatibility of formative and summative functions indicated above. Motivational psychology, for example, strongly emphasizes the importance of intrinsic motivation as a facilitator of thorough learning. According to numerous motivation theories, humans are keen to learn by nature and motivated to develop themselves through challenges. The prospect and perception of having to learn due to the presence of an external obligation arising from an impending inspection and with a possibility that sanctions will be imposed (i.e., extrinsic motivation) would have a pernicious influence on this intrinsic motivation (Deci & Ryan, 1985; Lens & Depreeuw, 1998). Those involved are no longer learning because they want to but because they have to. A similar situation is described in the literature with regard to the evaluation of pupils. Here too, a distinction is always made between formative and summative evaluatory functions (Aschbacher, Koency, & Schacter, 1995). Moreover, it also has been demonstrated that the critical and reflective attitude, and the desire to improve which is so important for learning and which is the object of formative evaluation methods, disappear with the prospect of summative evaluation. In essence school development is a process of human learning with respect to the participants in the school and - mutatis mutandis - also a learning process at the organizational level (i.e., the organization itself learns). It ought to come as no surprise therefore that similar problems can present themselves in learning at the school level. Implications for the Relationship Between Internal and External Evaluation The work of the Flemish educational inspectorate (as an example of external evaluation) is currently both of a formative and summative nature. Although the audit report is in the first instance a summative report, it nonetheless contains valuable feedback for schools. If the audit report now had to be (partly) based on the self-evaluation of schools, this self-evaluation would, in effect, be mortgaged in advance. Successfully implementing a self-evaluation aimed simultaneously at school development and accountability is therefore not as simple as the advocates of integration would have us 110 J. Vanhoof, P.V. Petegem / Studies in Educational Evaluation 33 (2007) 101–119 believe. Furthermore, if the function of an external evaluation is summative, one has to ask whether a self-evaluation can play any role in this at all. There is a real risk that schools will see self-evaluation as an obligation and as a task which is not connected to school development (Hopkins, 1987; Renkema, 2002). In matching external and internal evaluation therefore, careful thought must be given as to what the function of the evaluation is. Depending on this, different answers must be formulated to the question as to what the relationship between the two ought to be. This leads therefore to an argument for a strict separation between accountability and school development. Conditions for a Successful Combination of Self-Evaluations and School Audits Self-evaluations and school audits are two components of a single entity which is called quality assurance. There may very well be valid arguments for integrating the two, but it also appears that such arguments may be based on overly positive assumptions regarding the possible marriage of the formative and summative functions of evaluation. In the following section we will set out a number of guidelines and points for consideration when combining these two forms of evaluation in a sound quality assurance system based on the considerations described above. Make a Strict Distinction Between Accountability and School Development A school audit can, in principle, be set up with either a summative or a formative focus. Although the prime concern in the first instance is a determination of results, the data gathered for this purpose can yield valuable feedback for schools. This is not the case in a self-evaluation. Self-evaluation must be primarily concerned with school development. Intrinsic motivation is a necessary pre-condition to getting a sound self-evaluation process off the ground (McBeath, 1999). From this perspective, self-evaluation is not an appropriate method for the gathering of data intended for a summative evaluation. A selfevaluation cannot take place under heavy pressure from outside the school. The selfevaluation results might be very far removed from the actual functioning of the school and also prove unusable both for accountability as well as school improvement purposes (Davies & Rudd, 2001). It is, therefore, advisable to start with a strict distinction between the determination of results and development. Regard Evaluation as a Process and not as a One-off Activity In the context of quality assurance in education, evaluation must yield insights into extremely complex processes and it is overly ambitious to imagine that these can be captured in a single snapshot. Effective evaluation is a process of analysis, presentation and discussion of findings and of confrontation with supplementary data (Nevo, 2001). This process cannot be adequately conducted in a single, one-off evaluation moment. A selfevaluation as a preparation to a one-off school audit ignores the process-related character of self-evaluations. In self-evaluation the emphasis has to be on the process (Watling & Arlow, 2002). Certainly, this can yield a product (e.g. a report) as an interim result, but the J. Vanhoof, P.V. Petegem / Studies in Educational Evaluation 33 (2007) 101–119 111 report in itself is of little real significance (Van Petegem, 1999). The preparation for the self-evaluation and the follow-up based on the report are much more important. Select Themes for Self-Evaluation Carefully Given that the government obliges/encourages schools to carry out self-evaluations, it is very important that the themes chosen for self-evaluation are seen as relevant by schools. The dialogue that self-evaluation requires can only be initiated if those participating in the self-evaluation have the feeling that the themes examined are genuinely important (Scheerens, 2004). This means that they must, at least, be either personally involved or possess relevant expertise (Hoy & Miskel, 2001). The choice of themes to be evaluated must be left up to the school. If the government imposes external aspects, there is a risk that schools will only conduct self-evaluation in order to comply with regulations and not because they are intrinsically motivated to do so or from a school-internal need for quality assurance (Van Petegem, Verhoeven, et al., 2005). We would also like to emphasize that the reasons for conducting school audits are different from those which underlie selfevaluations and consequently audits have a different focus. Frames of reference suitable for external evaluation are thus not necessarily good frames of reference for internal evaluations. Recognize and Encourage the Professionalism of Others Schools, educational support services and inspectors, each have complementary roles to fulfil in the existing system of quality assurance and this can only work well if the different parties involved recognize each other’s professionalism and conduct themselves in an atmosphere of sufficient trust (O'Sullivan, 2004). Trust can, however, only be gained by placing trust in others (Fullan, 1993). There is a world of difference between a formal obligation and a strong encouragement. If the desire is to encourage teaching staff and school managers to adopt a critical attitude which will facilitate school development, they must be approached as professionals and as equal partners (Earley, 1998; Mace, 2002; McBeath, 1999). In other words, the attitude of external evaluators may well be of decisive importance in encouraging internal quality assurance. Guarantee the Quality of Self-Evaluations Professionalism may not simply be assumed, however. Conducting self-evaluations requires schools to possess not only a number of technical self-evaluation skills, but also a sufficient level of policy effectiveness (Cheng, 1990; Van Petegem, Verhoeven, et al., 2005). Two important tasks must be addressed in order to guarantee the quality of selfevaluations: supporting schools in the conduct of self-evaluation, and monitoring the quality of those self-evaluations. 112 J. Vanhoof, P.V. Petegem / Studies in Educational Evaluation 33 (2007) 101–119 Support Schools in Conducting Self-Evaluations A government which expects high quality self-evaluations from schools may not ignore the professional development and support which are necessary to accomplish this. The Flemish educational inspectorate accepts that most schools are familiar with the concept of self-evaluation, but that they define this concept in their own - and usually limited – way. Schools often award themselves a positive evaluation whereas in reality they are still at the beginning of a long process of improvement (Van Petegem, Verhoeven, et al., 2005). Not all schools have the development level and the willingness to operate autonomously and independently. Moreover, they may not have the resources required - in terms of methodology and contents - to arrive at good self-evaluations. The type of support which schools require can vary from financial support and the supply of materials and ready-to-use tools; to training and support in situ. Neither content-related nor methodological support must be underestimated, therefore. Monitor the Quality of Self-Evaluations At present it is unclear as to what extent schools possess sufficient organizational effectiveness and capacity for self-evaluation, i.e., to design strategies, methodologies and criteria for high-quality self-evaluation for themselves. Moreover, the question as to the differences which currently exist between schools has so far remained unanswered or rests on assumptions (Robinson & Cousins, 2004; Rogers & Hough, 1995). This makes it difficult to estimate what can be expected from schools in this regard. There is unanimity with regard to the importance of self-evaluation as an element in quality assurance in schools, but things become more complicated when it comes to determining which criteria can be used to evaluate the quality of a self-evaluation. A combination of self-evaluation and school audits, however, presupposes a minimum acceptable quality of self-evaluation and the tools to record that quality. In other words, an array of tools needs to be developed for meta-evaluation. Once these are operational it should be possible to identify the strengths and weaknesses of Flemish schools with regard to self-evaluation. The step towards more focused support can also be implemented in a manner which is tailored to the needs of individual schools. Implications for Possible Scenarios for a Successful Combination of External and Internal Evaluation We have demonstrated that both external and internal evaluations are necessary components of quality assurance systems. However, these cannot be combined carelessly. We would now like to move on – keeping the above points for consideration in mind – to arrive at possible scenarios for the achievement of a successful combination of auditing and self-evaluation by schools. By rigorously applying the points formulated above, we would like to set out a scenario which may serve to provoke reflection. Our point of departure is a distinction between two groups of school-external expectations (which may or may not be anchored in legislation) and school-internal expectations. We also make an additional J. Vanhoof, P.V. Petegem / Studies in Educational Evaluation 33 (2007) 101–119 113 distinction between accountability and school development. In total, therefore, there are six conditions to be met when elaborating a system of quality assurance. Table 1: The Six Domains of Quality Assurance Legally-anchored expectations Non-legally anchored external expectations School-internal expectations Accountability Domain 1 Domain 3 Domain 5 School development Domain 2 Domain 4 Domain 6 The full implementation of this framework is an extremely complex task and it is very difficult to arrive at a consensus on this given the importance of ideological assumptions. In the following section we will focus on the role which self-evaluations, school audits and support services must play in the various domains. For each domain we will examine whether internal and external evaluations should run in parallel, sequentially or cooperatively (Kyriakides & Campbell, 2004). The domains with regard to non-legally anchored and school-internal expectations are handled together throughout, however. Our intention here is, after all, to focus primarily on the distinction between expectations whether or not these are legally anchored. Nor are we going to explore more specific forms of self-evaluation such as formative visits by colleagues and the input of “critical friends” in self-evaluations or the specific ways in which data should assembled in conducting selfevaluations (Swaffield & MacBeath, 2005). This does not mean, however, that we do not subscribe to the potential of these forms of quality assurance. Domain 1: Accountability for Legally-Anchored Expectations In this first aspect of quality assurance we must adopt – on the basis of the arguments described above – two clear standpoints: (1) it is only in this case that an obligatory school audit must play a role and (2) there is no room here for self-evaluation. In essence, the inspectorate is authorized to check whether schools comply with the expectations which the government imposes upon them. From a strictly legal point of view their responsibilities are more restricted than is the case in the present situation. There should, therefore, be a “pared down” form of school audit which is restricted to those aspects which are actually stipulated by law and which could be termed a “core audit”. Core audits take place at the initiative of the government and are obligatory. The report of a core audit is first and foremost summative and limited, as standard, to those aspects which are imposed on schools by the government. This does not mean, of course, that the audit may not look at other processes in the school, but this is done as a means to an end and not as an objective per se. After all, it is not possible to talk about the quality of a product without acquiring an understanding of how that product is created (Michielsens, 1999). 114 J. Vanhoof, P.V. Petegem / Studies in Educational Evaluation 33 (2007) 101–119 We have already argued above that the validity and reliability of self-evaluations is insufficiently guaranteed in the event of an obligatory summative evaluation. This means, therefore, that self-evaluation as such does not have a role to play in this first aspect of quality assurance. In this domain conducting self-evaluations as a preparation to (core) school audits is thus not an appropriate method. The reverse may be possible, however: the results of a core audit may provide useful information which could then be included in a self-evaluation. In this case, we find ourselves in the second domain of quality assurance. Domain 2: School Development with Regard to Legally-Anchored Expectations The legally-anchored expectations which schools are obliged to meet can also be the subject of school development. This is because in many cases the objectives imposed by the government are also shared by schools and also because schools are themselves aware of the importance of complying with these expectations. In these school development processes a role may be assigned to the core audit report, for the educational support services and for self-evaluation. A central element in this domain of quality assurance is self-evaluations by school focused on the extent to which the school concerned complies with the expectations of the government. The initiative for undertaking such a self-evaluation ought preferably to be taken by the school or a supporting body and not by the government as monitoring party. The aim of this self-evaluation is thus also formative and focused on development. Schools want to describe and evaluate their own functioning with respect to legal expectations with a view to taking decisions or undertaking initiatives which make it possible to respond (even) better to these expectations. In principle, self-evaluations in this domain can be independent of the core audit, but they can also be combined. Possible objectives of selfevaluations with legal expectations as their focus are anticipating, analysing and discussing the findings of a core audit and establishing suitable implementation methods. In this sense, the core audit report also has an important informative value in this domain of quality assurance. This report must not only contain findings with regard to the expectations investigated, but must also indicate how obstacles observed can best be tackled in the opinion of the educational inspectorate. There must, therefore, be a proper follow-up to the core audit, although ideally this ought not to be carried out by the inspectorate itself. The inspectorate writes its report and explains its content and this is where its role ends. Once the audit has been completed, the educational support services must be closely involved in what follows in order to guarantee the formative role of the evaluation. Another important element here is the creation of formative (external) evaluation moments which are independent of the core audit. The time interval between (core) audits is at present too great to permit them to play a systematic role in the process of school development. Yet school development is a continuing process. Initiatives need to be taken, therefore, to make contact with schools in a formative manner with an eye to establishing their strengths and weaknesses in relation to compliance with legal expectations. In the first instance, the supervision services have an important role to play in this. J. Vanhoof, P.V. Petegem / Studies in Educational Evaluation 33 (2007) 101–119 115 Domains 3 and 5: Accountability of Non-Legally Anchored and School-internal Expectations In principle, schools are not accountable to the government with regard to external expectations which are not legally-anchored, but they may render direct or indirect account to those bodies and (groups) of people who originally formulated these expectations. The same applies to school-internal expectations. In both these domains of quality assurance it is essentially a matter of opening communication between schools and the persons and groups concerned. According to Flemish educational experts, schools are currently somewhat closed. How and to whom account ought to be rendered are questions which have yet to be clarified, however. This is a social discussion which has yet to get under way in Flanders. Hence it is particularly difficult to determine at present what roles could be allotted to self-evaluation and school audits in quality assurance as a whole. Both are means to an end which can only be employed once a clear aim has been identified. A determining factor is, for example, the role to be given to the government (or its inspectorate) in guaranteeing this form of accountability. Domains 4 and 6: School Development with Regard to Non-Legally Anchored and School-Internal Expectations School development is concerned with much more than merely the “minimum” which the inspectors come to audit. The non-legally anchored and school-internal expectations ought also to be part of this. It is in the first instance up to the school itself to describe and evaluate its own functioning through cycles of self-evaluation. The context-related nature of the non-legally anchored and school-internal expectations makes it difficult to establish general frameworks, of course. Yet this does not mean that external evaluations cannot have a scope-broadening, encouraging and legitimizing impact on self-evaluation activities. In these domains of quality assurance there can thus also be a role for external evaluations by the educational inspectorate and supervisory services. At present, schools do not have to render account to the inspectorate with regard to expectations in these domains. That said, however, the audits of the inspectorate can play a valuable role. In the course of the core audit, the inspectorate does, after all, collect data with regard to aspects which go beyond the merely legal. We have commented above that the report of the core audit in this scenario restricts itself to those aspects which the government stipulates for schools. At the request of the school other aspects of the functioning of the school could also be included in a supplement to the core audit report. This section of the audit report would have a purely formative function and the inspectorate would then give its opinion on aspects which form part of the audit, at the request of schools. It must be possible for the inspectorate - at the request of and in consultation with the school - to shine its expert light on part of the route travelled in the direction which the school wishes to go. A further step might also be taken here. At the request of the school under inspection, the inspectorate could expand the summative core audit by including formative theme audits. These (non-obligatory) theme audits focus on aspects which go beyond the legal expectations per se, such as the pedagogical/didactic policy of the school, 116 J. Vanhoof, P.V. Petegem / Studies in Educational Evaluation 33 (2007) 101–119 or its policy implementation capacity. On the initiative of the school the contents of the core audit could be supplemented with some form of external formative evaluation. Educational support services also have a role to play here. It is in the first instance up to them to provide schools with formative evaluation moments regarding expectations which exceed the purely legal. Schools must be able to turn to the supervisory service for support both in terms of content and with regard to methodology. This might take a wide variety of useful forms: providing additional training; the supervision of self-evaluations; making tools and methods available; setting up partnerships between schools which participate in self-evaluation and arranging exchanges of formative visits around a joint framework; establishing forms of audits, etc. Conclusion The specific implementation of the role of self-evaluation and school audits as part of an overall quality assurance system is no easy matter. Educational policy-makers in various Western countries are currently struggling with this problem. Questioning existing practices and looking for possible alternatives often seems only to leave us with yet more questions. This is a necessary process, however, if we are to create a comprehensive and better-functioning quality assurance system. In this connection it is important to arrive at a shared vision with regard to what the objectives of such a system ought to be. Only when these objectives are clear, will we be in a position to clarify the respective roles which selfevaluations and school audits can play in quality assurance as a whole (Kyriakides & Campbell, 2004). Fundamental reflections are thus crucially important in relation to the quality assurance system. In attempting to arrive at a possible scenario, we have very strongly emphasized the difference between formative and summative forms of evaluation. The initiative for school development always lies with the school itself. Unless schools specifically request this, the inspectorate should, in this scenario, limit itself to its core tasks. The inspectors must conduct a two-track policy - strictly and transparently: an accountability-oriented policy track; and a development-oriented policy track. In addition to this, possibilities must be created - or further expanded - so that schools can call on external actors in order to ensure the quality of internal evaluation. This article contains material for reflection. We have set out arguments both for and against an integration of self-evaluations and school audits in schools’ rendering account and development. In order to make the implications of these arguments more specific we have developed a possible scenario. Although we are favourably inclined towards the logic behind this scenario it is not our intention to provide definite answers. Our purpose in setting out this scenario is to formulate critical reflections in connection with existing quality assurance systems and to offer suggestions for possible alternatives. First and foremost our intention was to question a number of assumptions as these assumptions stand in the way of the critical examination of the present quality assurance system which is so desperately needed. References Alkin, M.C. (Ed.) (2004). Evaluation roots. Tracing theorists' views and influences. London: Sage. J. Vanhoof, P.V. Petegem / Studies in Educational Evaluation 33 (2007) 101–119 117 Aschbacher, P.R., Koency, G., & Schacter, J. (1995). Alternative assessment guidebook. Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing. Cheng, Y. (1990). Conception of school effectiveness and models of school evaluation: A dynamic perspective. CUHK Education Journal, 18 (1), 47-61. Cheng, Y. (1996). The pursuit of school effectiveness. Hong Kong: Hong Kong Institute of Educational Research. Christie, C.A., Ross, R.M., & Klein, B.M. (2004). Moving toward collaboration by creating a participatory internal-external evaluation team: A case study. Studies in Educational Evaluation, 30, 125134. Cousins, J., & Earl, L. (1995). Participatory evaluation in education: Studies in evaluation use and organizational learning. London: Falmer. Davies, D., & Rudd, P. (2001). Evaluating school self-evaluation. Berkshire: National Foundation for Educational Research. Deci, E.L., & Ryan, R.M. (1985). Intrinsic motivation and self-determination in human behavior. New York: Plenum. Earley, P. (Ed.). (1998). School improvement after inspection: School and LEA responses. London: Chapman. Forss, K., Cracknell, B., & Samset, K. (1994). Can evaluation help an organisation to learn? Evaluation Review, 18 (5), 574-592. Fullan, M. (1993). Change forces. Probing the depths of educational reform. London: Falmer. Harrington, H.J., & Harrington, J.S. (1994). Total improvement management: The next generation in performance improvement. New York: McGraw-Hill. Hendriks, M.A., Doolaard, S., & Bosker, R. J. (2002). Using school effectiveness as a knowledge base for self-evaluation in Dutch schools: The ZEBO-project. In A. Visscher & R. Coe (Eds.), School improvement through performance feedback (pp. 114-142). Lisse: Swets & Zeitlinger. Hopkins, D. (1987). Improving the quality of schooling. In D. Hopkins (Ed.), Improving the quality of schooling. Lessons from the OECD International School Improvement Project (pp. 192-197). London: Falmer. Hoy, W., & Miskel, C. (2001). Educational administration: Theory, research and practice. Boston: McGraw-Hill. Kyriakides, L., & Campbell, R.J. (2004). School self-evaluation and school improvement: A critique of values and procedures. Studies in Educational Evaluation, 30 (1), 23-36. Lens, W., & Depreeuw, E. (1998). Studiemotivatie en faalangst nader bekeken. [A closer look at motivation to study and fear of failure]. Leuven: University Press. 118 J. Vanhoof, P.V. Petegem / Studies in Educational Evaluation 33 (2007) 101–119 Love, A. (1991). Internal evaluation: Building organizations from within. London: Sage. Mace, J. (Ed.) (2002). Self-assessment – Take a look at yourself. London: Learning and Skills Development Agency. McBeath, J. (1999). Schools must speak for themselves. London: Routledge. McBeath, J., & Myers, K. (2002). Self-evaluation: What's in it for schools? London: Routledge/ Falmer. McNamara, G., & O'Hara, J. (2005). Internal review and self-evaluation – The chosen route to school improvement in Ireland? Studies in Educational Evaluation, 31, 267-282. Michielsens, P. (1999). Bezorgd om Kwaliteit! Concepten, ideeën en visies met betrekking tot externe evaluatie. [Concerned about quality! Concepts, ideas and visions relating to external evaluation]. Informatie Vernieuwing Onderwijs, 74, 33-42. MVG. (2006). Onderwijsspiegel Schooljaar 2004-2005. Verslag over de toestand van het onderwijs. [Educational Mirror, 2004-2005]. Brussels: Ministry of the Flemish Community. Nevo, D. (2001). School evaluation: internal or external? Studies in Educational Evaluation, 27 (2), 95-106. Nevo, D. (2002). Dialogue evaluation: Combining internal and external evaluation. In D. Nevo (Ed.), School-based evaluation: An international perspective. Oxford: Elsevier Science. Nisbet, J. (1988). Rapporteur's report. In Council of Europe/Scottish Council for Research in Education (Eds.) The evaluation of educational programmes: Methods, uses, and benefits (pp. 1-9). Amsterdam: Swets & Zeitlinger. Norton Grubb, W. (1999). Improvement or control? A U.S. view of English inspection. In C. Cullingford (Ed.), An inspector calls: OFSTED and its effect on school standards. London: Kogan Page. O'Sullivan, R.G. (2004). Practicing evaluation: A collaborative approach. London: Sage. Renkema, G. (2002). Kwaliteitszorg: Wat levert het op? [Quality control: What does it bring in?]. Basisschoolmanagement, 15 (8), 14-19. Robinson, T.T., & Cousins, J.B. (2004). Internal participatory evaluation as an organizational learning system: A longitudinal case study. Studies in Educational Evaluation, 30, 1-22. Rogers, P.J., & Hough, G. (1995). Improving the effectiveness of evaluations: making the link to organizational theory. Evaluation and Program Planning, 18 (4), 321-332. Scheerens, J. (2004). The evaluation culture. Studies In Educational Evaluation, 30 (2), 105-124. Scriven, M. (1967). The methodology of evaluation. In R.W. Tyler, R.M. Gagne & M. Scriven (Eds.), Perspectives in Curriculum Evaluation (pp. 39-83). Chicago: Rand McNally. Scriven, M. (1991). Evaluation thesaurus (4th ed.). Newbury Park, CA: Sage. J. Vanhoof, P.V. Petegem / Studies in Educational Evaluation 33 (2007) 101–119 119 Swaffield, S., & MacBeath, J. (2005). School self-evaluation and the role of a critical friend. Cambridge Journal of Education, 35 (2), 239 - 252. Van Hoyweghen, D. (2002). Hoe complementair zijn interne en externe evaluatie? [How complementary are internal and external evaluation?]. Kwaliteitszorg in het onderwijs, 2, 39-70. Van Petegem, P. (1999). Zelfevaluatie ter verbetering en/of ter verantwoording van scholen? [Selfevaluation to improve and/or hold accountable?]. Informatie vernieuwing onderwijs, 74, 21-32. Van Petegem, P. (2005). Vormgeven aan schoolbeleid: Effectieve-scholenonderzoek als inspiratiebron voor de zelfevaluatie van scholen. [Shaping school policy: School effectiveness research as a source of inspiration for school self evaluation]. Leuven: Acco. Van Petegem, P., Vanhoof, J., Daems, F., & Mahieu, P. (2005). Publishing information on individual schools? Educational Research and Evaluation: An International Journal on Theory and Practice, 11 (1), 45-60. Van Petegem, P., Verhoeven, J.C., Buvens, I., & Vanhoof, J. (2005). Zelfevaluatie en beleidseffectiviteit in Vlaamse scholen. Het gelijke onderwijskansenbeleid als casus. [Self evaluation and policy effectiveness of schools. A case study of the Flemish equal chances policy]. Gent: Academia Press. Watling, R., & Arlow, M. (2002). Wishful thinking: Lessons from the internal and external evaluations of an innovatory education project in Northern Ireland. Evaluation & Research in Education, 16 (3), 166-181. The Authors JAN VANHOOF studied educational sciences at the Catholic University of Leuven (Belgium) and is currently a PhD student at the Institute of Education and Information Sciences of the University of Antwerp (Belgium). His interests are in educational quality assurance in general and in the conduct of high quality school self evaluation in particular. PETER VAN PETEGEM is a full professor at the Institute of Education and Information Sciences of the University of Antwerp (Belgium) and is the head of the research group EduBROn (www.edubron.be). His research and writing interests include school effectiveness, educational quality assurance, school and teacher self evaluation, and learning and instruction. Correspondence: <Jan.vanhoof@ua.ac.be>