Assessing English for Professional Purposes - Introduction

Near final version of the ‘Introduction’ in Knoch, U. & Macqueen, S. (2020). Assessing English for professional purposes. Abingdon: Routledge. Introduction Modern life has become ever more dependent on ‘expert systems’, such as the airline industry, judicial and health systems (Giddens, 1990). At the same time, expertise itself is increasingly mobile. Doctors, engineers, programmers, nurses, accountants, pilots, teachers and many other professions, including workers with a plethora of hybrid professional capabilities, move across national borders in response to demands for the diverse array of expertises that sustain modern systems. A professional qualification now offers employment potential well beyond the boundaries of the state in which it was earned with the key proviso that the migrant professional has the ability to encode their expertise in the language of the destination nation-state. Hence, a need for language assessment arises at the governance boundaries of nations, professional certification bodies and workplaces. In this movement of expertise, standard varieties of the English language have accrued a high economic and cultural value. This means that it is readily used as a gate-keeping mechanism for participation in the professional workforce of migration destinations that have English as an official or national language. Commercial English language tests classify the language abilities of millions of test-takers annually, some of whom will undergo testing multiple times. As both a symbol and tool of globalization, English language skills are valued even in corporations which are predominantly non-English speaking. Therefore evidence of English language ability in the form of test scores converts to a competitive advantage for professional graduates seeking to work in both English-speaking and non-English-speaking countries. Some professions, especially those in industries serving global connectivity such as pilots, air traffic controllers and call centre staff, require people to work in lingua franca varieties of English such as Aviation English, that are not tied to national boundaries. These varieties too, are the objects of language tests. The regulatory role of English language tests is, in part, a response to the management of risk. Many professionals work in high-risk jobs, where communication breakdowns can lead to serious consequences. Even when the stakes are not so high, poor communication or miscommunication has negative effects on work practices such as relations with clients and the efficient achievment of work tasks. Language asessments for gate-keeping and screening purposes are efforts to avert the risk of miscommunication. Just what constitutes ‘risk’, however, is in the eye of the beholder. For skilled migration applicants, tests themselves, as potential barriers to employment, pose a risk. For skilled migration policy makers, the risk may have a political basis that is more concerned with immigration quotas and skill oversupply or shortages than effective communication. At the same time, ‘expert systems’ implicate laypeople in their expert procedures. The layperson converts to the role of ‘patient’ in the health professional world and is suddenly, and consequentially, surrounded by the need to understand the knowledge, activities, systems and relationships encoded in the language used there. A layperson who is a client signing off on a contract is legally commiting to an agreement about their comprehension of professional matters that are encoded in specialist register. The comprehension of laypeople as jurors in 1 legal proceedings is gravely consequential. Thus, even for laypeople, there is risk associated with not understanding the language used in professional worlds. This book is about various kinds of language assessment processes that are carried out in relation to participation in professionally-oriented settings. Our contribution comes some forty years after the first standardized English test for specific purposes, and with the benefit of the extensive development in research, theory and practice that has occurred since. Our aim is to recontextualize this body of work in the current times of rapid globalization, increasing transnational migration flows, massive technologization, English language spread and shift and increasingly complex infrastructures of professional regulation and statehood. We take account of these various dynamics and offer both a theoretical basis and practical guide for the development and validation of language assessment for professional purposes. What we hope to achieve is a renewed agenda for inquiry into Language Assessments for Professional Purposes (LAPP). It is one which may seem uncomfortably broad to some, but in the interests of greater awareness of the extent of language assessment activities and better assessment practices in society generally, it is an agenda worth pursuing we think. It has been nearly 20 years since Dan Douglas published his seminal text entitled ‘Assessing Languages for Specific Purposes’, and in the years since, new language assessment for professional purposes contexts have emerged or been formalized through various policies. Douglas’s work remains foundational and we do not attempt to repeat, revise or replace it here. We take up where he left off, by extending the remit of the field and theorizations about its various tools: those for understanding constructs, for carrying out needs analyses, for developing assessment instruments, for validation and for evaluating policy. Definition of LAPP and notes on terminology With the broader remit mentioned above in mind, we define language assessment for professional purposes (LAPP) as any assessment process, carried out by and for invested parties, which is used to determine a person's ability to understand and/or use the language of a professionally-oriented domain to a specified or necessary level. Although we use the term ‘test’ when appropriate throughout this book, and we draw upon the preponderance of test-based research and theory, it can be seen in the examples above, that there are non-test language evaluations that are highly consequential in terms of participation in a professional domain. Therefore, we use the term ‘assessment’ as the more general term, and ‘test’ to refer to instruments delivered under test conditions, such as time and resource-access constraints, often with multiple administrations. In the spirit of the broader remit, we refer to people who are assessed as ‘assessees’, and as ‘test-takers’ when we are discussing test instruments. This broadening allows us to consider assessment practices which are not explicitly named as such. Similarly, LAPP is intended to cover any language assessment in or related to workplaces and professional domains generally. We use the terms ‘domain insider’, ‘domain expert’ and ‘subject matter expert’ interchangeably throughout the book to refer to professionals with training and experience in a specific profession. We understand that this is a very simplistic use of the term, as professionals and professional specialisations differ greatly in and across workplaces, and the ‘experts’ therefore differ in their experiences. However, for consistency, we have decided to stick to this use of the term as it is currently used in the literature on language assessments for specific purposes. 2 Overview The chapters in this book are relatively self-contained but interrelated. Wherever text in a chapter relates to another chapter, we have noted this within the chapter. Readers may wish to read the six chapters in the book consecutively, or they may want to select individual chapters of interest from which they can explore the leads into other chapters. In Chapter 1, Scope, we set out the territory for the remainder of the book. The broadened conceptualisation of language assessment for professional purposes (LAPP) is introduced by drawing on two media stories we use as a basis for expanding LAPP into assessments for insiders and outsiders taking part, or seeking to take part in professional domains. We introduce the sociological concept of risk to conceptualize LAPP as a social exercise in risk management. We explore risk from the viewpoints of different stakeholder groups, including how the management of ‘language-associated risk’ implies responsibilities for different stakeholders in striving for assessment practices that are as beneficial and as fair as possible. We also consider the implications of rapid sociotechnological development for LAPP. In Chapter 2, Construct, we interrogate how constructs are represented in the language assessment literature, and we explore how they manifest in assessment practices and stakeholder activities. We propose a socially-oriented theory of construct for LAPP, starting with the dimensions of an assessment construct and then suggesting how these relate to the micro and macro contexts of an assessment. We then turn our attention to language varieties and professional registers, distinguished by Michael Halliday as ‘ways of speaking’ and ‘systems of meaning’ respectively (1985, p. 41). Because constructs are determined by what is sampled in an assessment procedure, we draw up a model for sampling from the gamut of linguistic repertoires that inhabit professional workplaces. In Chapter 3, Needs, we look closely at one key ingredient in the development of LAPPs, the needs analysis. We propose a broadened model of needs analysis, building on similar work in curriculum design, and discuss each component of our model in detail, drawing on methodological considerations in collecting this kind of information, where appropriate. The chapter concludes with a description of a needs analysis conducted for a large-scale LAPP, the Occupational English Test (OET), a language screening test for overseas-trained health professionals. Chapter 4, Development, then describes the various stages of LAPP development following a needs analysis. We start by describing the process of reconciling the findings of a needs analysis and developing a test blueprint and test specifications. We do this by taking test developers through a series of practical stages aimed at arriving at a representative list of tasks which could be included in a test blueprint. The remaining part of the chapter focusses on some key considerations specific to LAPP development, including the elicitation and development of scoring criteria based on indigenous criteria (that is, the criteria valued by domain insiders), setting standards on LAPP by drawing on the judgements of domain insiders and considerations associated with the operational use of a LAPP. In Chapter 5, Validation, we draw on the argument-based approach to validation, a framework proposed by researchers in educational assessment (e.g. Kane, 2006) to guide validation activities, to set out a comprehensive framework which researchers and test 3 developers can draw on when validating the score interpretations and uses of LAPPs. The framework we propose is designed to be flexible and adaptable to different contexts. This framework can be used to identify weaknesses in a chain of argumentation and to set research priorities. Where existing research on LAPPs is available, we have integrated that into our description of the framework. In Chapter 6, Policy, we discuss the fuzzy distinction between language assessments and language-associated policies. We take a broad view of policy, following work by Elana Shohamy and Tim McNamara, to include the covert work done by language assessments. We return to the notion of construct, particularly the public-facing dimension, in the question of the uses of language-based assessments as social sorting mechanisms. We consider the rise of national and international frameworks (e.g. the Common European Framework of Reference) which serve as supra-organisational mechanisms for linking different test instruments across different jurisdictions. We return to the principles of benefit and fairness, and the notion of language-associated risk in the making and enacting of policies, for both workplace insiders as well as outsiders. A hierarchy of affordances of assessment instruments proposed in this chapter is intended to guide policy decisions. We conclude the chapter by arguing that the type of validation work we describe in Chapter 5 may need to be complemented by policy evaluation in certain contexts. For this reason, we draw on the argumentative approach to policy evaluation and set out a framework that researchers may use as a basis for interrogating language-associated policies. In the concluding chapter, we return to the question of the use of linguistic evidence as a basis for decisions. We distinguish between two types of evidence: assessment evidence on which scores are based and validation evidence which may reveal something about how trustworthy a score is for a particular use. We revisit the tools, theories and concepts presented in this book under the overarching theme of ‘evidence’. We suggest some future directions for research and how a broadened agenda for assessment in professional worlds and workplaces might proceed. A broadened agenda Language Assessments for Professional Purposes (LAPPs) and Language Assessment for Specific Purposes (LSP) (the latter also focusses on language assessments for academic purposes) have been a key part of the language assessment literature for several decades. Because of the explicit connections made between the assessment instrument and the Target Language Use domain (TLU domain), these areas of inquiry have paid particular attention to the interaction between domain-related knowledge (or content) and language ability. We feel it is time to take stock of our directions in theory and practice as a basis for new developments and increased awareness of the risks and responsibilities associated with language assessment. In view of the various uses of these assessments, either as high-stakes sorting mechanisms or risk-reducing devices, our intention is to stretch the now-comfortable boundaries of the field. Doing so will offer some alternative directions to the centripetal force created by well-known commercial tests which attract attention away from the myriad of language assessing practices that are carried out in the maintenance of expert systems. In order to do this, we have explored within our field, and we have ventured beyond our discipline to offer a broader proposed agenda and some tools for working with it. At the same time, we pay close attention to how our methods are evolving in relation to our more traditional object of inquiry, the standardized test. While we consider our traditional objects 4 well worth the continued pursuit, we hope that we have offered some new lenses for familiar concepts, as well as some new avenues to pursue. References Giddens, A. (1990). The consequences of modernity. Cambridge: Polity. Halliday, M. A. K. (1985). Register variation. In M. A. K. Halliday & R. Hasan, Language, context and text: Aspects of language in a social-semiotic perspective (pp. 29-43). Victoria: Deakin University Press. 5