Guidance on writing aims and intended learning outcomes Dr Peter Kahn Educational Development, Centre for Lifelong Learning University of Liverpool 3rd November 2010 Introduction A clear and appropriate statement of aims and intended learning outcomes forms an important element in programme design, in quality assurance and in focusing student learning. It is thus important that aims and intended learning outcomes are written in ways that facilitate these processes. The purpose of this paper is to provide guidance on the scope and construction of aims and intended learning outcomes, and on links with the wider programme. The paper concentrates on aspects of writing aims and learning outcomes that experience from reviewing documentation and from working with colleagues indicates are often most problematic. Thus while we introduce how to write aims and learning outcomes, we focus in more detail on the following:   broadening the scope of aims; indicating level in an intended learning outcome. In addition, we consider in Appendix 1 the relationship between intended learning outcomes and standards of performance. A consideration of this relationship enables us to clarify whether intended learning outcomes should be set for the typical or threshold student, and also to help ascertain the level of detail that is required. Appendix 2 and Appendix 3 provide guidance on both possible skills and other attributes, and verbs to be included in intended learning outcomes. A broader approach to aims A programme aim is a general statement about the purpose of the programme. Aims are thus primarily concerned with what the programme hopes to achieve, and they are typically written in terms of teaching intention rather than the learning of the student. Note, however, that pre-requisites for running the programme should not be recast as aims, as is the case for the following 'aims':   to employ an appropriate variety of teaching and assessment methods to meet the programme's aims and learning outcomes; to use learning resources effectively and efficiently to meet the programme's aims and learning outcomes. While the aims of a programme will primarily concern the students themselves, wider aims may also be relevant. Indeed, it is often only by considering a wide range of aims that the real distinctiveness of a programme is seen to emerge. An appreciation of the distinctiveness of the programme is useful in programme design and in conveying to students and others its unique features. It will thus be helpful to consider several types of aim as indicated in Table 1. There is, evidently, overlap between the categories in Table 1 and it is further possible to set aims that incorporate more than one of these categories, as in the following example:  to enhance students' capacity to engage in extended project work, both on an independent basis and in collaboration with their peers, and thereby to prepare students for further academic study and employment. Type of aim Student-based Department or Subject based Employer-based Society-based Examples of aims To explore the central features of the discipline; to attract students who will benefit from studying in a researchenriched environment; to open access to the study of a range of specialist areas within the discipline; to focus on the contested nature of knowledge within the discipline. To provide a seed-bed for ideas that can be exploited in research programmes within the department; to provide the disciplinary community with new members; to constitute an example of best practice to other departments in how to teach the discipline. To meet the requirements of potential employers in a specific sector. To positively impact on the social fabric of the local community; to widen participation within the body of students studying the discipline; to contribute to society through the development of a sense of civic responsibility in the students on the programme. Table 1, Types of aim Writing learning outcomes An intended learning outcome is a concise description of what a student will have learnt at the end of some learning process. One of the main advantages to stating the intended learning outcomes (simply now referred as learning outcomes) from a course of study is the way in which this allows one explicitly to consider the ways in which the goals for student learning are constructively aligned with both the methods used for teaching and supporting learning and the assessment on the programme. Given their key role in helping to shape the educational process, it will be useful to review key aspects of writing learning outcomes. This review will also provide an effective foundation for the next section, which looks at what is perhaps the most challenging aspect of writing a learning outcome; namely indicating the level of the outcome. A review of how to write learning outcomes A learning outcome typically consists of sentence that begins with a phrase such as ‘At the end of this programme it is expected that you will be able to’, which is then followed by three elements (examples for these aspects of a learning outcomes are provided in Table 2):    an active verb (often with an associated adverb); an object of the verb (indicating on what the learner is acting); a phrase that indicates the context or provides a condition. Verb Critically evaluate Recognise Object new technical, regulatory and policy developments in law any risks or safety aspects Context especially in relation to notions of justice1 that may be involved in the operation of computing equipment within a given context Table 2, Examples of the three elements of a learning outcome The actual content of each learning outcome is, of course, shaped by a number of actors. We thus highlight here some of the more important issues that need to be addressed:       1 Categories of outcome — In general, learning outcomes describe intended learning in terms of knowledge, understanding, skills and other attributes. In particular, the University's template for a Programme Specification requests information on knowledge and understanding, and on intellectual, practical and transferable skills (see Appendix 2 for a list of various skills.) Balanced outcomes — There should be an appropriate balance between the subject, personal and professional learning outcomes. Different views — It is possible to view a programme in three different ways: a collection of course units; a number of years; and an entire programme. Viewing a programme in these ways can help to ensure coherence between the different course units, across the various years of the programme and within the programme as a whole. It is therefore useful to state the learning that is intended to result from each course unit (course unit outcomes), from each year of the programme and from the programme as a whole (programme outcomes). Realistic – Learning outcomes must be achievable within the constraints of time and other resources, and the learning and teaching context provided. Developmental – Learning outcomes should be developed in part using feedback from previous learners (i.e. about what they identify and value). Understandable – The context and meaning of learning outcomes should be understood easily by interested parties, e.g. colleagues, learners and employers. This example, and other examples, are taken from documentation submitted within the University for quality assurance purposes.  Process versus product – Learning outcomes should not simply constitute a statement of the process by which the learning takes place. For instance, you will want to develop in students 'the ability to plan and implement a research project' rather than simply get them to 'have some experience of research'. Indicating level in learning outcomes When writing learning outcomes that concern knowledge and understanding, staff often simply provide a statement of the content that is covered. This, at least, is the perspective that a review of quality assurance documentation provides. Take for instance the following outcome: Knowledge and understanding of specific disciplines including periodontology, preventive dentistry, dental, dental radiography and orthodontics. Does the understanding have to be critical in any sense or connected to relevant areas of professional practice? The outcome as stated relies, in effect, on a set of accepted norms for the kind of understanding that is expected within the discipline in question. Indeed, with a few stylistic changes, a syllabus can therefore suffice for a statement of learning outcomes. Learning outcomes are redundant. Learning outcomes, however, offer a significant advantage over a syllabus by providing an explicit indication of the abilities that students actually should be learning. Where the nature of the intended learning is clearly spelled out learning outcomes can, for instance, be of use in more directly considering the appropriateness of specific teaching methods, and also in framing appropriate assessment tasks, criteria and feedback on student learning (see for instance Biggs, 1999). In addition, learning outcomes can assist in personal development planning for students, providing a basis for planning, activity and reviewing. A syllabus cannot easily serve all of these functions. Indeed, basing teaching simply around a syllabus tends to lead to viewing teaching as a fixed activity that is never adapted to the specific abilities that students are trying to master. It is worth noting that a clear match should be evident in a Programme Specification between the programme outcomes, the methods employed for teaching and supporting learning, and the assessment. What is clear, furthermore, is that an explicit indication of level2 is needed if a learning outcome is to fully specify the intended learning. Such an indication of level actually helps to set the expected standard that must be achieved. In disciplines where knowledge is organised on a strongly hierarchical basis, however, one might still argue that it should not matter whether one favours a syllabus or a statement of learning outcomes. After all, in such disciplines the level of understanding one can argue that the level is at least implicitly indicated within a syllabus. Take for instance the following outcome: At the end of the programme, students will have an understanding of how accounting and information systems informs and is informed by economic, organisational, social and political contexts. Here one could argue that the level is implicitly provided by the sophistication of the subject matter. But this still fails to articulate the nature of the understanding that is required. Does the understanding need to be systematic, original or critical? Is there any need to recognise 2 We use the term 'level' here to indicate a hierarchical categorisation of learning outcomes: this categorisation is closely related to the year of study. We follow the National Qualifications Framework which spells out the following levels: Certificate (year 1); Intermediate (year 2); Honours (year 3); Masters; Doctoral. the interplay between the specified contexts? We still need to go beyond an implicit indication of level if a learning outcome is to help shape student learning. Further ways to specify level How then can one take advantage of the additional possibilities for specifying level that are afforded by a set of learning outcomes? The approach suggested here is, first of all, to specify the nature of the understanding that is sought. Perhaps the most useful categorisation for different types of understanding is that provided by Bloom (1964). He views understanding according to the following hierarchy:  knowledge; comprehension; application; analysis; synthesis; evaluation. (See Appendix 3 for verbs to use in learning outcomes that reflect this hierarchy.) Are we looking for a systematic understanding, or will a haphazard grasp suffice? Similarly, the intended understanding may be critical, conceptual or so on. It is also relatively easy to fail to indicate level when making statements about skills. When dealing with intellectual skills and practical skills, then the disciplinary context may, as above, provide a partial indication of the level, but even this will not hold where transferable skills are concerned. For instance, one might consider the following learning outcome: You will be able to work in a team. But students are able to work in a team by the end of primary school! As stated, this outcome provides little help in designing an appropriate programme. It therefore again helps to specify more clearly the actual nature of the ability in question. Such a more detailed consideration of the ability in question also needs to take account of a number of further considerations. Level may be provided in part by framing learning outcomes that require students to have mastered the interplay between two or more abilities. We might therefore consider developing the above outcome about working in a team to the as follows: You will be able to give an oral presentation as part of a team, based around the earlier joint solution of a problem. However, even indicating such interplay will often fail fully to articulate the nature of the learning that occurs. In particular, it is important to take into account the context in which these abilities are developed. We can consider the following elements that help to make up this context:    the extent of autonomy involved; the inherent complexity of the context; the degree of originality. A learning outcome will clearly be more challenging if the student is expected to attain it without detailed guidance from a lecturer. Similarly, the context that frames the learning will also influence the level of difficulty. An outcome that requires students to adapt their response as the situation changes in an unpredictable fashion is clearly more challenging than one in which the demands placed on the student remains constant. Meanwhile, the degree of originality that is involved in carrying out some task will again affect its level. This is more obvious towards Masters and PhD level, providing a key distinguishing factor at these levels from earlier levels. We might thus develop our outcome related to team-work to the following: You will be able to give an oral presentation in a team to specialist and non-specialist audiences, critically evaluating both the team's performance and your own contribution to the team. Taken together, all of these above considerations combine to indicate the level of an outcome, giving arise to a explicit hierarchy within any set of learning outcomes. The features that indicate level are summarised within Table 3. Such an explicit focus on level does, of course, not dispense with the need for the context that is provided by the subject. Judgements against national standards, for instance, are difficult to make without reference to specific subject matter. Hence the National Qualifications Framework (see Quality Assurance Agency for Higher Education (QAA), 2001), especially at undergraduate level serves to provide a context for different subject benchmark statements (see QAA, 2002) rather than to provide a primary point of reference. What is apparent, however, is that without an explicit indication of level learning outcomes are far less useful. Factor Words and phrases to help introduce factor Nature of ability Coherent knowledge; systematic comprehension; application to unseen problem; conceptual analysis; wide-ranging synthesis; critical evaluation; generate alternative scenarios. Combination of abilities Interplay between; in light of; taking into account the impact of; while also; an integrated approach; at the same time as. Autonomy Within a structured and managed environment; some self direction, awareness of own learning; initiate and carry out projects; manage or advance own learning; monitoring progress; personal responsibility; autonomous planning; evaluating your own performance; enhancing your skills. Context Specialist and non–specialist audiences; variable demands; use scholarly reviews and primary sources; effectively adapting to new demands; unpredictable situations; incomplete data; appreciate uncertainty; identify a range of solutions; managing dynamically complex work. Originality Established techniques; standard methods of solution; current problems; devise novel approaches; recent research; creative responses; informed by the forefront of the discipline; at the forefront of discipline; innovations in practice; extending the theoretical basis; limits of knowledge; originality in the application of knowledge; creation and interpretation of new knowledge; meriting publication. Table 3, Factors that contribute to an indication of level in a learning outcome, beyond a statement of the subject matter in question Conclusion The Higher Education sector as a whole has now accepted aims and intended learning outcomes as integral elements of good practice in teaching and the support of learning. However, if intended learning outcomes in particular are created simply for quality assurance purposes, and are not drawn upon in designing effective programmes and enabling student learning, then it is understandable that they are seen as a burden. This paper, in particular, has sought to explore ways in which learning outcomes need to be specified if they are to be useful more widely, contributing to the design of both teaching and assessment. Appendix 1: Learning outcomes and standards of performance The relationship between learning outcomes and student performance is a complex one. It is, of course, true that some students will be able to achieve the specified learning outcomes whereas others will not, even at a minimal standard of performance. Meanwhile, many students will learn more than is specified within the learning outcome. Thus, in an overall sense, a statement of learning outcomes does provide a coarse way of indicating a standard of performance. Yet at the same time, a learning outcome is not designed to fully specify student achievement. Two students may achieve the same learning outcome, but to differing extents. A tension is thus present, one which is often resolved by simply ignoring the relevance of learning outcomes to assessment. We will explore in this appendix a number of ways in which learning outcomes can contribute to making judgements about student performance. In the process we will clarify two important issues that concern writing learning outcomes: whether the learning outcomes reflect either the threshold requirements that all students are expected to achieve or the expected achievement of a typical or modal learner; and the level of detail that is required. Threshold or modal? If a learning outcome states that students 'will be able to do something on completion of the course', then this suggests that a threshold outcome is in play. Even here, however, the language may hide a modal outcome. Introducing the idea of that it is 'expected that students will be able to do something on completion of a course' makes the modal character of a learning outcome more explicit. Unless there are good reasons to specify learning outcomes in threshold terms (e.g. in certain professional disciplines) it would usually be expected that they would be framed in modal terms. This will make it clear that our programmes exceed those of the benchmark statements. However, one might wish to use threshold learning outcomes to help shape grade descriptors that are positive, say, even when describing performance for a Third Class honours degree. Assessment criteria and grade descriptors The relationship between learning outcomes and student performance is clearest when considering both assessment criteria and grade descriptors. A set of assessment criteria provides a framework in which judgement can be made about a student's performance. As an example we can take one of the assessment criteria from a Postgraduate Certificate in Learning and Teaching in Higher Education: Extent to which the work is analytical in style and approach, with critical understanding and interpretation. Furthermore, a clear match will be evident between any assessment criteria and the learning outcomes. Indeed, the statement of learning outcomes may well be used to help decide upon the criteria. We can see, for instance, the correspondence between the above example and one of the related programme outcomes: Ability to engage in self-directed analysis and synthesis of teaching practices, and evaluation of the impact of that practice upon the students concerned. An assessment process, however, usually results in specifying a student's performance at one of a number of grades. One typically thinks of the usual degree classifications or of a percentage scale. The assessment criteria will thus usually lead to a set of grade descriptors, or in certain cases a marking scheme or model answer. Such further specification of the standard against which a student's work is judged are, of course, necessary required to ensure the reliability of the assessment. It is worth emphasising here that grade descriptors are likely to incorporate the following two elements:   adjectives that describe the degree to which a particular criteria has been demonstrated, e.g. good, fully, partially, excellent. (Note: these adjectives will not normally figure in the statement of a learning outcome, as they refer more directly to the standard of performance against the learning outcome) aspects of student performance that only figure at the higher or lower grades of performance. For instance a learning outcome at modal level may incorporate elements that are not present within the work of a failing candidate or a First Class performance may introduce a new element that goes beyond achievement at modal level. In particular, there will be a clear match between threshold outcomes and grade descriptors for a Third Class degree and between modal outcomes and the border between Upper and Lower Seconds. It is worth comparing the programme outcome given above with the grade descriptors that refer to the ability to engage in analysis: (0-39%) Limited grasp of the relevant ideas and issues; (40-49%) Grasp of the relevant ideas and issues is evident, although limited; (50-59%) Analytical in style and approach at times, although descriptive and prescriptive for the greater part; (6069%) Analytical in style and approach, with some critical interpretation; (70-100%) Highly analytical in style and approach, with critical understanding and interpretation.3 The comparison suggests strongly that the programme outcome has been set at modal rather than threshold level. Finally, it is worth noting that a match should also be evident between marking schemes and the actual learning outcomes and any associated assessment criteria, although the links will typically be less explicit. In conclusion, one might also think of a continuum of the ways in which learning outcomes and assessment criteria seek to articulate student performance. Learning outcomes in which level is clearly articulated provide a coarser approach while assessment criteria and grade descriptors provide the basis for a more fine-grained approach. Appendix 2: Skills and other attributes 3 Note that one often includes finer divisions of firsts, e.g. 70-79% and fails, e.g. 30-39%. Intellectual skills Critical thinking – capacity to abstract, analyse and make critical judgement Problem solving – ability to solve problems in an efficient and effective manner, drawing on problem solving strategies as relevant Problem posing – ability to frame problems in a fashion that is amenable to their solution. Synthesis and analysis of data and information Planning, conducting and report on research project/dissertation Critical reflection and evaluation Translation Expression – able to make a reasoned argument for a particular point of view Decision-Making – able to draw reasoned conclusions Practical skills Planning and executing safely a series of experiments or independent research Using library, electronic and online resources Using reporting skills Mapping and modelling Audit production Speaking, reading and writing a foreign language at near-native proficiency Treating patients in certain defined clinical procedures Peer review – able to comment on the performance or work of a peer, identifying strengths and making constructive suggestions for improvement where appropriate Transferable skills Information Retrieval – ability independently to gather, sift, synthesise and organise material from various sources (including library, electronic and online resources), and to critically evaluate its significance. Presentation – capacity to make oral presentations, using appropriate media for a target audience Numeracy – ability to appreciate issues of selection, accuracy, uncertainty and approximation with number) Literacy – the capacity both to make written presentations using appropriate language for a target population and to collect and integrate evidence to formulate and test a hypothesis Computer Literacy – ability to use word processing, database, spreadsheet and presentation software and the use of the Internet Networking Teamwork – recognising and identifying views of others and working constructively with them Negotiation - understand group dynamics and intercultural backgrounds in the use of negotiating skills to reach objectives Time Management – ability to schedule tasks in order of importance Applying Subject Knowledge – use of discipline specific knowledge in everyday situations Research – ability to plan and implement an effective research project. Improving own Learning – ability to improve one's own learning through planning, monitoring, critical reflection, evaluate and adapt strategies for one's learning Other attributes Willingness to update knowledge – understand the need for Life Long Learning Listening – ability to listen effectively and make a constructive contribution to a discussion Commercial Awareness – working effectively within externally or poorly defined constraints as in a business environment Initiative – able to take action unprompted and assume responsibility Creativity – able to be innovative and apply lateral thinking in problem solving and decision making Stress Tolerance – able to use personal resources effectively to meet challenges Self-confident – able to maintain independence of thought and be self-reliant Independence – capacity for self-discipline, motivation and diligence Self-management – capacity for self-appraisal, reflection and time management Adaptability – ability to respond positively to changing circumstances Self-awareness – awareness of own strengths and weaknesses and to be able to work as part of a multidisciplinary team Ethical appreciation – a willingness to ascertain the ethical implications of proposed courses of actions or situations and to take the necessary steps to ensure that result from this analysis Professionalism Appendix 3: Verbs for use in Learning Outcomes Use of the following verbs can help in the construction of learning outcomes: Knowledge: What do you expect learners to know?  define; state; list; outline Comprehension: How do learners convey what they have understood?  explain; identify; discuss; describe; interpret Application: How do learners use a theory or information in a new situation?  demonstrate; apply; operate; employ; illustrate. Analysis: How do learners break down material and ideas into constituent parts to show how they relate to each other and how they are organised?  Distinguish; appraise; debate; solve; differentiate; contrast; examine; investigate; calculate; question; analyse; test; criticise. Synthesis: How do learners work with elements and combine them in a way that was not there before?  devise; manage; design; organise; create; plan; construct; formulate. Evaluation: How do learners construct an argument, compare opposing arguments, make judgments?  Judge; evaluate; appraise; criticise; assess. Acknowledgements This paper has been informed by Bingham (1999), discussions within a Working Group at the University of Manchester on aims and intended learning outcomes, and the University of Manchester 's Student Satisfaction Survey (2003). The original version of this paper was developed by the author for the Academic Framework and Regulation Project at the University of Manchester, 2003. References Biggs J (1999) Teaching for Quality Learning at University, Society for Research into Higher Education/Open University Press, Buckingham. Bingham R (1999) Learning Outcomes: a guide, Learning and Teaching Institute, Sheffield Hallam University, Sheffield. Bloom B S (1964) Taxonomy of Educational Objectives, David McKay. Quality Assurance Agency for Higher Education (QAA) (2001) The framework for higher education qualifications in England, Wales and Northern Ireland, QAA, Gloucester [Online, accessed 7 May 2003] http://www.qaa.ac.uk/crntwork/nqf/ewni2001/contents.htm QAA (2002) Benchmarking academic standards, QAA, Gloucester [Online, accessed 7 May 2003] http://www.qaa.ac.uk/crntwork/benchmark/index.htm Further Reading Biggs J (1999) Teaching for Quality Learning at University, Kogan Page, London