Using Portfolios for External Assessment: An Experiment in Portugal Maria Teresa Torres Pereira de Eça Abstract This article describes some of the conclusions reached from an experiment using portfolios for art external assessment at the end of secondary education (age 17+) conducted in five Portuguese schools in 2001–2003. Several positive outcomes were found. Students found portfolios to be motivating and fostering constructive learning, dialogue and co-operation between students and teachers. The new assessment procedures developed communities of assessors enabling some increased consistency of examination results and positive professional development opportunities. JADE 24.2 ©NSEAD/Blackwell Publishing Ltd 2005 However some weaknesses were detected such as potential bias related to the degree of teacher aid and practical problems as, for example, time-consuming in-service teacher training. 209 210 Maria Teresa Torres Pereira de Eça 1. Methodology The experiment was part of a PhD research study [1]; the research took place between 2000 and 2004. Quantitative and qualitative research methods were used; data collection included observation, documents, interviews, questionnaires, teachers’ and external observers’ reports and the results of portfolio marking exercises. The participants in the experiment were selected from a list of volunteers gleaned from an initial survey questionnaire about the current Portuguese art examinations conducted in Portugal during 2001 with 44 art teachers and 104 students. The design of the assessment instrument and procedures was developed in three stages: • Stage 1: Development: Rationales, draft specifications of the assessment instrument and assessment procedures. • Stage 2: Trial 1 – Piloting: Negotiate with participants; trial with a reasonable sample, evaluate and refine the draft specifications for the assessment instrument and assessment procedures. • Stage 3: Trial 2 – Main Trial: Implementation: trial a larger scale experimental assessment and evaluate the specifications for the assessment instrument and assessment procedures. A new framework for art external assessment was initially designed, first, by drafting general guidelines including rationales, contents, the evidence to be assessed and procedures necessary to establish an assessment community. The design of the instrument (Portfolio) and assessment procedures was developed through a pilot study involving seven art teachers and 51 art students in one secondary school in Portugal. The rationales were discussed; teachers and students negotiated the domains of knowledge and evidence to be assessed – as well as criteria and grade descriptors. After the detailed instructions for students and teachers were written, students developed their portfolios according to a project brief chosen by them. The teachers in the pilot were trained to assess the students’ portfolios during five one-day meetings using visual exemplars of students’ work to reach a common interpretation of criteria and standards. After the students’ portfolios were completed, moderation procedures were tried out. The final evaluation of the pilot study helped to redefine the instructions, criteria and procedures used in the main trial. At the second stage, the revised instrument and procedures were tried out with a larger sample including five schools from different Portuguese regions (10 art teachers and 117 art students). 2. Design of the examination instruments and procedures The rationale The specification of the instrument followed the rationale for art and design education described by Swift & Steers [2] who argued that art education should address three fundamental principles: difference, plurality and independent thought. These principles are embodied in the promotion of risk-taking, personal enquiry and challenging established orthodoxies, especially those associated with cultural hierarchies. Thus, learning in the arts should develop disciplinary knowledge, creative and critical thinking, and the skills to interrogate dominant ideologies. The rationale was further inspired by four dimensions of student understanding in the arts proposed by Ross, et al [3]: 1. Conventionalisation: an awareness and ability to use the conventions of art forms. 2. Appropriation: embracing for personal use, the available expressive form. 3. Transformation: the creative search for personal knowledge. 4. Publication: the placing of the outcome in the public domain. These dimensions operated as four basic quadrants from which to systematise the contents and skills of art and design for assessment purposes in JADE 24.2 ©NSEAD/Blackwell Publishing Ltd 2005 Figure 1: Types of evidence for Portfolio Reports or notes about previous experiences, interests, etc. (visual/written) Student Portfolio Folder, exhibition; workjournals; CD; Webpage, etc. Preliminary studies, developmental records (Visual/ written) Final Products (visual): Paintings, drawings, sculptures, printings; graphic design; product design; multimedia photographs, films, video records of performances, installations, exhibitions, etc. Self-assessment report. Interviews: written or oral: tape, video, digital record about the students’ intentions, progress, investigation, achievement, presentations, evaluation. Records of self-assessment and ‘crits’ Investigation reports and data critical inquiry (Written and visual) the form of a blueprint which attempted to foster a holistic vision of art and design, that promoted risktaking, personal enquiry and critical skills. The instrument The portfolio was adopted as the assessment instrument for the new examination. In compiling a portfolio, students explore a theme, plan, elaborate, present and evaluate their work [4]. In light of the Portuguese tradition of one single examination period at the end of the art course, it was considered appropriate to develop a single portfolio as the final art and design examination. A decision was taken that the examination should be conducted in the last term of the last academic year during normal art and design class time (22 hours). Students and teachers would receive all the instructions at least one month before the examination to allow for preparation time. The portfolio was devised as an open-ended project to explore a theme that should include selected evidence of students’ intentions, motivations, JADE 24.2 ©NSEAD/Blackwell Publishing Ltd 2005 critical inquiry, explorative and developmental studies, final products and records of self-assessment. Self-assessment evidence was included in order to ensure students’ voices were heard and their intentions made clear. The evidence to be selected for assessment by students was schematised as above in figure 1. The assessment criteria The design of the instrument was limited, in particular, by the need to establish criteria that would be subject to common interpretation by users. The agreed criteria were: C1: Record personal ideas, intentions, experiences, information and opinions in visual and other forms. C2: Critically analyse sources from visual culture showing understanding of purposes, meanings and contexts C3: Develop ideas through purposeful experimentation, exploration and evaluation. 211 Maria Teresa Torres Pereira de Eça 212 Maria Teresa Torres Pereira de Eça C4: Present a coherent and organised sample of works and final product revealing a personal and informed response that realises their intentions. C5: Evaluate and justify the qualities of the work. The assessment procedures In-service teacher training was provided through five one-day meetings and five on-line discussion meetings. The meetings included explanation and discussion of instructions and simulation of marking using examples of students’ portfolios designed to achieve consensus when interpreting criteria. In order to achieve consistent results a procedure for external verification of the internal marks was also designed. A significant sample of students’ portfolios was available to one or more external assessors for checking the fairness and consistency of internal marking. The first trial The pilot exercise was centred on negotiation. According to an appointed external observer, ‘the design was based on teacher ownership: sharing power and constructing knowledge instead of an elitist, top down vision of the assessment’. Teacher ownership of the assessment was ensured and emphasised. According to one teacher: Examinations used to be imposed by the government; the question papers only express the view of the people in the national awarding board about what they value in art and design or about what they think should be assessed. With this experiment I felt that things can be different; we can also express our views about what must be assessed and we ended up by enlarging the vision… (TR: Ana Paula; 18 January 2003). Students’ opinions also were taken into account during the pilot exercise: It was quite odd during the examination preparation, because the teacher was always asking us if we agreed with the instructions and so on; I think it was a sign of respect for us; it was helpful not just because we could express our opinions but also because by talking about it we better understood the instructions (SR: Manuel; 15 January 2003). The main trial (2002–2003) The second trial included five schools in diverse geographical locations and with different resources: Schools P; B, K, V and A. Teachers’ approaches to art education and students’ motivations were also very diverse according to the context. At School P, in the teacher’s view students’ poor attitudes and lack of motivation were a consequence of negative educational practices: they have a passive attitude towards art disciplines; because they are used and were trained as passive objects to make pre-determined tasks (J report, 6 June 2003). Students at P school were unfamiliar with the new assessment requirements, their previous learning in art was underpinned by a formalist approach, and because of that they could not perform well in the majority of tasks. [During the course] It was not the same thing; the preparation/search was just finding an image and after we did the work using the image; for example an impressionist or abstract work. We are used to making drawings (MTEP; Studio Art) to experiment with specific techniques the teacher ask us to experiment… The work is the same for everyone; for example a face and then we work the form; light; colour, etc. (SR Sergio, 2nd June 2003). School B was a contrast. According to the teacher; the students were competitive and well motivated. The teacher had no problems applying the new instrument with the students because she already assessed students in a similar way and fostered the kind of knowledge, understanding and skills required by the new assessment instrument in her classes. So, the tasks were not unfamiliar and students had the JADE 24.2 ©NSEAD/Blackwell Publishing Ltd 2005 necessary training to perform within the constraints and opportunities presented. Previous experience seemed to be a key factor for the success of the new assessment instrument in this case. In general, students considered that the new assessment instrument was valid and they enjoyed the relative freedom of choice: The portfolio was good essentially because we had to make different works; it was not prescriptive; we had a theme and we had to develop the work… From our heads; not like ‘go and draw, draw a bench with a monkey’… I did a portfolio with things I like to do and showing what I wanted to show (SR Telma, 2nd June 2003). The portfolio demands more creativity; it is much more enjoyable and motivating for us; but it is also more difficult because of that (SR Sara, 2nd June 2003). At School A, implementing the new assessment raised several difficulties: The main difficulties in implementing the portfolio with my art class were related to the rationale of the portfolio; it was very difficult to require students to think independently because they are used to following detailed prescriptions for each task (M’s report, 27 June 2003). However, the teacher ended up with a generally positive implementation of the portfolio task. – this was probably a consequence of the great deal of support the students received from their teacher. At School K, students were quite excited about the new instrument; they seemed to understand the tasks as authentic and liked the more personal way of working in the arts: The portfolio was more real than the usual work and tests we do in the art classes; it is about our life; not school life or tasks that school thinks are important for us (SR Ruben, 9 June 2003). JADE 24.2 ©NSEAD/Blackwell Publishing Ltd 2005 The new instrument had been piloted at School V, so the students and teachers were familiar with it. Consequently the conditions in this school were quite different from the other participating schools. since we started the portfolio in September [pilot] students had the time to be prepared; little by little they learned skills of critical reflection and self-evaluation. They learn how to plan the different steps of the work in the time to respond to the deadlines. It was good to continue to use portfolios after the first experience [pilot] at the beginning of the year; students acquired so many learning experiences, they had the opportunity to develop organised ways of thinking and making; they developed independent skills and little by little they pushed their own barriers; they learned how to realise their own intentions and how to make projects that motivated them (TR AP, 25 June 2003). 3. Some conclusions from the experiment in Portugal Several positive aspects of the experimental assessment instrument and procedures were found, especially with regard to aspects such as content, response and face validity. Portfolios were viewed by students as motivational and fostering constructive learning, dialogue and cooperation between students and teachers. The new assessment procedures developed communities of assessors enabling some increased consistency of examination results and positive professional development opportunities. However some weaknesses were detected, such as potential bias related to the degree of teacher aid and practical problems as, for example, time consuming in-service teacher training. Validity The assessment instrument integrated a wide range of methods of inquiry, media, and domains of art and design, allowing students to develop personal projects in which they could personalise social issues and reveal important cognitive and metacognitive skills. The portfolio, as an 213 Maria Teresa Torres Pereira de Eça 214 Maria Teresa Torres Pereira de Eça extended task, provided valid and authentic tasks related to the art curriculum, and the use of portfolios for assessment purposes provided a new perspective on learning. The portfolio connected process and product and students were engaged in learning as an interactive process. Student learning was well documented and they actively explored and evaluated their learning through engaging in discussion with their teachers, other students or peers. Collaboration, dialogue and reflection became essential processes in the construction of the portfolio. The framework [5] used for designing the assessment specifications was intended to accommodate postmodern views of art and education and to relocate student experience, and contemporary realities of the visual arts, by encouraging students to ‘develop a critical awareness of the visual culture they encounter every day’ [6]. It was intended to reduce tensions between assessment discourses and the heterogeneity of practice by moving towards a more inclusive approach to teaching and learning in art education. During the trials of the new assessment procedures it was found that in certain cases students integrated everyday life experiences and reflected on the social meanings of art through exploration of issues such as racism, sexual orientation, otherness, etc. They used a variety of methods of investigation to understand social issues, including interviewing members of the community and independent searches for source material. But this was not the case for all students. The role of the teachers and their beliefs significantly influenced students’ choices. In some cases teachers were not able to go beyond the formalist art concepts they had been taught and they did not encourage critical analyses of visual culture. Such teachers were nervous about letting students investigate themes they were not comfortable with themselves, and convinced their students to engage with ‘safer’ and more conventional projects. The in-service training meetings were insufficient to encourage such teachers to make profound changes in their attitudes. The on-line meetings while were intended to provide support for teachers and more space for dialogue were of limited help in attaining these goals. Using Internet resources for dialogue about art and design learning and assessment may become more useful in the future; at the present time it was found that both teachers and their students found it difficult to use such tools because of their unfamiliarity. Moreover digital reproductions of students’ studio art works are not appropriate for fully appreciating and discussing the visual characteristics of portfolios, except perhaps when the student’s chosen media is itself digital. After the trials it was evident that the teacher’s role is fundamental in preparing and managing the learning environment, and it was crucial when students challenged conventions or methods of work through truly creative proposals or when they attained unexpected outcomes that revealed their own intentions, personal experience and awareness of social and visual culture issues. Some teachers acted as facilitators through the substantive dialogue that took place between the student and the teacher, using teaching methods based on collaborative learning and partnership. This enabled assessment to be integrated into the teaching and learning cycle. However, one negative consequence of the importance of the teacher’s role in the assessment was concern that the instrument was strongly biased by the degree of aid afforded by the teacher. The portfolio approach was found to respect the students’ voices and personal styles. Students were given a considerable degree of decision-making autonomy: selection of themes, selection of works for inclusion in portfolios and the reasons for inclusion were negotiated with their teachers. Students felt that with portfolio assessment they gained more ownership of the learning and assessment process. The conclusion was drawn that students need to have decision-making powers about themes for JADE 24.2 ©NSEAD/Blackwell Publishing Ltd 2005 project briefs and about which work is chosen for inclusion in portfolios if they are to experience ownership real commitment. It increased the content validity of the assessment instrument and presented more opportunities for student motivation and independent learning. However, for some students, such a level of autonomy was difficult; while they were aware that they were expected to be thoughtful about the selection of work for inclusion, this presented problems because they had not developed a capacity for critical reflection in previous years. From the research it was also apparent that some students found self-evaluation difficult. With portfolio assessment, students need to take responsibility for self-directed learning and for developing and maintaining their portfolios. The student interviews confirmed that this requirement was understood in principle, but nevertheless they were not always able to evaluate their own progress and achievement. Interactive learning, tutorials, interviews, one-toone sessions, discussion groups, peer critique, were all found to be helpful in this regard. But developing a capacity to select appropriate evidence of attainment of particular competencies takes time for some students – more time than this experiment allowed. The breadth and flexibility of the assessment instrument accommodated a variety of different visions of art and design processes and making which challenged the old habit of setting teacherdirected prescriptive tasks. It does, however, require art teachers who are able to help students develop independent critical and self-reflective skills. It also demands dialogue between teachers and students, mutual understanding of aims and intentions and negotiation of tasks and approaches. From the trials it was evident that portfolios can provide very useful feedback for students and teachers. Assessment was seen in Wolf’s terms as ‘an episode of learning’ [7] providing qualitative information in the course of portfolio development through constructive dialogue between students and teachers [8]. JADE 24.2 ©NSEAD/Blackwell Publishing Ltd 2005 Reliability The development of communities of assessors for inter-rater reliability purposes can be used as positive professional development opportunities [9]. Boughton’s concept of a ‘community of judges’ [10] and the example of the English moderation procedures was used to help design the assessment procedures. It was clearly established that reliability of results achieved in the trials increased in comparison with the current Portuguese system which uses a single assessor. As noted by several authors, assessors can reach a considerable degree of consensus through the use of standardisation and moderation procedures [11]. Using the new assessment procedures required considerable time for training and for dialogue between teachers. The increased burden on teachers needs to be acknowledged – the quantity of work to be assessed, the standardisation meetings and reports to be written – all contributed to teachers’ workload. This was only possible in the trials because the teachers who participated in them were convinced that it was important for their own professional development and for the fairness of art and design students’ assessment. One feature of the experiment’s success was that teachers had enhanced confidence in their powers to make assessment decisions. The new procedures did not undermine teachers’ professionalism. The current Portuguese reliance on teachers’ connoisseurship was maintained, but the shift was made to dependence on a community of assessors achieving consensus. Impact From the research it appeared that the effects of the new examination were significant for students, teachers, schools and the curriculum. A principal outcome of the new assessment instrument could be an increase in the status of studio activity and the raising of art and design educational standards. The teachers in the pilot and the trial reported that they changed the way that they thought of their teaching, classroom 215 Maria Teresa Torres Pereira de Eça 216 Maria Teresa Torres Pereira de Eça practice and assessment methods. These changes happened because the teachers who were already competent practitioners reflected on the proposed rationales for the experience and decided, with the support of their peers, that they had to make changes, because they agreed with the proposals and believed that their students had the right to be involved in and be responsible for their own learning and assessment. Effects upon students The portfolio was a way to improve our knowledge and at the same time it was preparing us for the future (SR Mafalda, questionnaire, June 2003). The questionnaire responses revealed that all the teachers and a great majority of students saw the new instrument as a good learning tool and a sound foundation for university. From the sample included in the research it was apparent that the impact of portfolios increased students in-depth study, active and independent learning, awareness of their own learning strategies, motivation and interest in their own achievements and performance. Inevitably this demanded increased work and students who have grown used to being tacit observers might resent having to work harder. Nevertheless students felt there were benefits in the new assessment system; it seemed more interesting and relevant. As Klenowski [12] pointed out, it is not only teachers who need to learn about the pedagogical implications of using portfolios for assessment and learning; students need specific teaching and support to develop the necessary cognitive processes. It was evident from the research that students need skills in independent study, group work, self-reflection, self-assessment, self-evaluation and questioning. Some students were not prepared to develop their portfolios because they lacked background skills in critical inquiry, in defining problems, self-discipline and self-evaluation. The teachers saw that the source of the problem was that their students lacked the necessary skills both to judge specific problems in understanding and to set realistic targets to remedy problems within reasonable time frames. However, where teachers created classroom environments in which students worked together to understand teachers’ comments about their work, then peer and selfassessment provided the training that students needed to judge their own learning and to begin to take action to improve their performance. It was evident that changes are needed to the role of students, but the period during which the role of the students changes needs to be handled carefully and the students have to be supported as they learn to become active, responsible learners. This requires time. Effects upon teachers It was evident that teachers initially felt overloaded by the portfolio experience – it marked a significant change in the Portuguese context. However they did not see it as a negative experience. On the contrary, teachers appreciated the opportunity to learn, improve their relationship with their students and assessment practices. The experience helped me to understand my students better. Through the dialogue established during the portfolio, I was able to understand them as individuals, looking to their particularities and respecting their differences (TR Ana Paula, questionnaire, June 2003) Now, I think that I can assess students with more consistency and I can give more valuable assessment feedback to students (TR Carolina, questionnaire, June 2003). After the main trial at least nine teachers appeared convinced about the positive qualities of portfolio assessment. Their responses revealed that their perceptions of the necessary knowledge, understanding and skills in art education had changed. They were convinced of the advantages of portfolios as a learning strategy and as an instrument for summative assessment. JADE 24.2 ©NSEAD/Blackwell Publishing Ltd 2005 It was evident that to increase its validity the new assessment instrument would require a long period of time for students to become familiar with a culture of self-reflection that is only established through long years of education. It also became increasingly clear, as Black et al [13] argue, that the teachers also needed to train their students to take responsibility for their own learning and assessment. The teacher’s role is to ‘scaffold’ this process – that is, to provide a framework of appropriate targets and to give support throughout the course of attaining them. In Portugal, implementation of changes in classroom assessment would call for profound changes in both teachers’ perceptions of their own role in relation to their students and in their classroom practice. Knowledge acquisition must be conceptualised as an active process rather than passive. Cognitive processes used by students such as self-regulation and self-monitoring are fundamental to understanding individual development and achievement and have to be fostered by teachers. This kind of assessment therefore has to be viewed as an integral part of the curriculum combining summative and formative forms; it would be necessary for teachers to change prescriptive modes of teaching and probably to revise fundamentally their rationales for art and design education. Effects upon schools and curriculum Teachers saw the portfolio as a valid instrument for learning and assessment with strong consequences for the educational experience. As stated by José, the introduction of portfolio assessment could be used as an agent for curriculum reform: However, we live in a period of social and technological change and our students should have access to a different kind of learning, fostering the generation of personal ideas and the portfolio as an assessment instrument is a good strategy to introduce new pedagogical practices (J’s report, 6 June 2003). JADE 24.2 ©NSEAD/Blackwell Publishing Ltd 2005 The teachers agreed that portfolio could function as an agent for the reform of established educational orthodoxy, and to promote diversity and plurality of approach, because it was a form of assessment that was integrated with the learning process and focused on students as active subjects rather than objects for passive reception of information: That’s why I think this is not just about assessment; portfolio is more than an assessment instrument; it is a way of teaching and learning (TR Rui, 25 June 2003). However, certain problems need to be acknowledged. Schools need to provide greater and more equal resources for students. The learning milieu is a fundamental resource and schools must recognise its importance. Another important consideration for schools is the allocation of time for teachers’ in-service training, meetings and marking. Teachers need to be motivated and to understand the justification if they are to be expected to take on the burdens of change. They need time to plan, discuss and mark students’ work; they also need opportunities for in-service training, peer-reviews or meetings in order to create an effective community of assessors. 4. Further research This research study was not expected to resolve all the problems of validity and reliability in external art assessment and some problematic issues were not fully solved. Portfolio assessment achieved greater validity than the current Portuguese art and design examination tests but whether or not more equality and fairness was obtained is questionable. Some students were disadvantaged because of poor access to resources and teacher aid. Even if the required knowledge, skills, competencies and criteria are established through dialogue and agreement, allowing some degree of teachers’ and learners’ ownership, external assessment still denies diversity and plurality of approach for some groups of students and schools. Greater fairness might be 217 Maria Teresa Torres Pereira de Eça 218 Maria Teresa Torres Pereira de Eça obtained by using negotiated assessment [14]. Only embryonic forms of negotiated assessment were used in this research, however the possibility of negotiated assessment as an element in external assessment could be explored in further research. Other methods and forms of assessment need to be investigated that might, as Rayment [15] suggests, ‘identify methods of providing valid and reliable evaluation procedures which can perform both formative and summative functions’. References 1. Eça, T. (2004) Developing a New Conceptual Framework for Pre-University Art Examinations in Portugal. Unpublished PhD thesis, University of Surrey, Roehampton. 2. Swift, J. & Steers, J. (1999) A Manifesto for Art in Schools, Journal of Art & Design Education, Vol 18, No. 1, pp. 7–13. 3. Ross, M. et al (1993) Assessing Achievement in the Arts. Buckingam: Open University Press, p. 51. 4. Lindström, L. (1999) The Multiple Uses of Portfolio Assessment, in Piironen, L. [Ed.] Portfolio Assessment in Secondary Art Education and Final Examination. Helsinki: University of Art and Design. 8. Wiggins, G. P. (1993) Assessing Student Performance: Exploring the Purpose and Limits of Testing. San Francisco: Jossey-Bass Education Series, p. 195. 9. See, for example, Klenowski, V. ( 2003) Developing Portfolios For Learning And Assessment. London: RoutledgeFalmer; Schönau, D. W. (1996) Nationwide Assessment of Studio Work in the Visual Arts: Actual Practice and Research in the Netherlands, in Boughton, D. et al [Eds.] Evaluating and Assessing the Visual Arts in Education. New York: Teachers College Press. 10. Boughton, D. (1997) Reconsidering Issues of Assessment and Achievement Standards in Art Education, Studies in Art Education, Vol 38, No. 4, pp. 199–213. 11. See Beattie, D. K. (1997) Visual Arts Criteria, Objectives, and Standards: A Revisit, Studies in Art Education, Vol 38, No. 4, pp. 217–31; Blaikie, F. (1996) Qualitative Assessment of Studio Art: Problems, Definitions and Solutions, Canadian Review of Art Education, Vol 23, No. 1, pp. 17–30; (Finland) Department of Art Education (1999), Report of EU Comenius 3.1 Project, pp. 7-16; Boughton, D. op. cit; Steers, J. (2003) Art and Design, in White, J. [Ed.] Rethinking The School Curriculum: Values, Aims and Purposes. London: RoutledgeFalmer. 5. All the materials used in the experience and results are displayed online in Portuguese at www.geocities.com/aiea2000/guiao.htm (Accessed December 2004). 12. Klenowski, V. op. cit. 6. Freedman, K. (2003) Teaching Visual Culture: Curriculum, Aesthetics and the Social Life of Art. New York: Teachers College, Columbia University, p. 11. 15. Rayment, T. (1999) Assessing National Curriculum Art AT2, Knowledge and Understanding: A Small-Scale Project at Key-Stage 3, Journal of Art & Design Education, Vol 18, No. 2, pp. 188–193. 7. Wolf, D. et al (1991) To Use Their Minds Well: Investigating New Forms of Assessment, in Grant, G. [Ed.] Review of Research in Education. Washington, DC: American Research Association, p. 183. 13. Black, et al (2003), p. 59. 14. Ross, M. et al op. cit. JADE 24.2 ©NSEAD/Blackwell Publishing Ltd 2005