Assessing Reading

by J. Charles Alderson
DZULFIKAR LAZUARDI MASYHURI(18070835001)
ENDAH YULIA EKA (18070835032)
DWI HANUM MASRUROH (18070835032)

Alderson, J. C. (2005). Assessing

reading. Ernst Klett Sprachen.
 AGENDA
1. The nature of reading
2. Variables that affect the nature of reading
3. Research into the assessment of reading
4. The reader: defining the construct of reading ability
5. A framework for test design
6. Test in the real world: test purposes
7. Techniques for reading ability
8. The development of reading ability
9. The way forward: assessing the interaction between
reader and text process and strategies
 1. Nature of reading
process and product
• Process means the interaction between a reader and
the text
• It has series of acts proceeding from one step to
another step
• There are three method that frequently used to
examine reading process: introspection, think-aloud,
and verbal retrospection in interviews
• Product means the reader understanding about the
text that have been read
• The product of reading will also different between one
reader with another reader based on their knowledge
and experience
 Nature of reading
level of understanding
• Literal meaning of text
• Inferred meanings
• Readers’ critical evaluation of text
 Nature of reading
bottom-up and top-down approach
• Bottom-up approaches are serial models where
the reader begins with the printed word,
recognise graphic stimuli, decodes them into
sound, recognises words and decodes meanings.
• Top-down approaches emphasise the importance
of information stored in the brain which act as
filters for incoming information and the readers’
contribution over the incoming text.
 Nature of reading
implications for test design
• Reading might be tested within a content-focused
• Students should be tested on a range of relevant skills
and strategies
• Students should be encouraged to read longer texts
rather than short snippets
• Background knowledge should be recognised as
influencing all comprehension
• Tests should be open to the possibility of multiple
interpretations
• Group tasks might be devised for a discussion of
students interpretation of text
 Nature of reading
implication for test design
• Extensive reading should not be discouraged by the
assessment procedures
• The importance of identification skills needs to be explored
• There will be settings and tests where it will be impossible
to reduce extrinsic motivation to a minimum and to
emphasise enjoyment
• Also there will be occasion when integrated testing
• Exploration is needed of ways in which synthesis and
evaluation skills, metacognitive knowledge and skills
monitoring can be assessed
• Test designers need to consider to what extent their test
reflect and build upon what recent research into reading
suggests about the process, not just product
2. Variables that affect the nature of
reading
Schemata and
Background
Knowledge

Metalinguistics Formal
knowledge Reader Schemata:
and knowledge of
Metacognition
Variables language

Knowledge of
genre/text
type
Variables that affect the nature of reading

Content
Schemata

Reader Knowledge
Cultural
of subject
knowledge Variables matter

Knowledge
of the
world
Variables that affect the nature of reading

Criticisms
of schema
theory

Real world Reader Reader

versus test skills and
taking Variables abilities

Reader
purpose in
reading
Variables that affect the nature of reading

Reader
motivation

Beginning
readers and Reader Reader affect
variables
fluent readers

Reader
characteristics
Variables that affect the nature of reading

Text topic
and content

Text Text type

readability and genre

Text
Variables

Traditional Literary and

linguistics non-literary
variables text

Text
organisation
 Variables that affect the nature of
reading
• Typographical features
• Verbal and non-verbal information
• The medium of text presentation
3. Research into the assessment of
reading
Language
questions

Reading
Types of
and
questions
intelligence

Factors
affecting the
difficulty of
Use of reading test
dictionaries items Testing
in reading skills
tests

Role of Role of
vocabulary grammar in
in reading reading
tests tests
 4. CONSTRUCT OF READING

• Factors that affect reading are relevant in determining assessment

• Alderson recommends dependency on all possible linguistics
features of a text to measure reading constructs.
• Impacting variables of reading need to be considered while
designing the test
• Importance of readers background knowledge is less relevant than
linguistic background (language of the target texts)
• A variable to be neutralized and controlled
• The demanding the task is, the higher that linguistic threshold goes.
• Making a task demanding has to do with variables of topic
knowledge, background knowledge, text language, and task type
• The use of texts that reflect what academic readers are
supposed to do
• “Texts are taken from magazines, journals, books, and
newspapers. Texts have been written for a non-specialist
audience. All the topics are of general interests. They deal
with issues which are interesting, recognizably appropriate
and accessible to candidates entering postgraduate or
undergraduate courses. At least one text contains detailed
logical argument.” (IELTS Handbook, 1999:6)
• IELTS attempts to measure the following constructs: (IELTS
Specifications, December 1989)
1. Identifying structure, content, sequence of events and procedures
2. Following instructions
3. Finding main ideas which the writer has attempted to make salient
4. Identifying the underlying theme or concept
5. Identifying ideas in the text, and relationships between them, e.g.
probability, solution, cause, effect
6. Identifying, distinguishing, and comparing facts, evidence, opinions,
implications, definitions and Formulating a hypothesis from underlying
theme, concept and evidence Then Reaching a conclusion by relating
supporting evidence to the main idea.
Constructs of reading and constructs of communicative language
ability
• North showed how we can relate the construct of reading to the
construct of communicative language ability in the following
extracts:
• Strategic Competence in reading
(Mastery Level) Can adapt style and speed of reading to
different texts and purposes.
(Effectiveness Level) Can read with a large degree of
independence, using appropriate reference courses selectively
• Linguistic Competence in reading
(Mastery Plus Level) Can understand texts which contain
(Effectiveness Level) Has a broad active reading vocabulary,
but may experience some difficulty with low-frequency idioms. Can
understand grammatical patterns, and vocabulary ordinarily
encountered in academic/professional reading
• Discourse Competence in reading
(Mastery Level) Can distinguish in details the various parts of
the treatments of a theme and understand their interrelations.
(Effectiveness Level) Can separate the main ideas in details
from lesser ones. Can recognize the line of argument in the treatment
of the issue presented, though not necessarily in detail.
• Sociolinguistic Competence in reading
(Mastery Plus Level) Can understand a wide variety of slang
and pertinent cultural references. Can appreciate humor and subtle or
culture-dependent nuances of meaning or style.
(Effectiveness Level) Can understand many socio-linguistics and
cultural references.
• But....
• According to Assessing Reading book by J. Charles Alderson
(2000) there is a text called DIALANG on page 125 which
Alderson critically examines the test specifications of the
European Commission's DIALANG diagnostic approach
towards reading In the IELTS test/UCLES.
• He outlines descriptors of reading proficiency, where he
comments on the tendency for assessors to promote both
global and specific skills, and approaches to communicative
reading tests.
 Task Characterization and Reading Comprehension

• The different components and the levels of reading passage

might interact with each other And reading is interactive in
that the reader’s background knowledge and other attributes
interact with the content of the text.
With regard to the fact that reading involves interaction between
readers and texts, it can be concluded that characteristics of
both, the reader and the text, can affect the reading process
(Alderson, 2000).
• Alderson want to continue for test design using Bachman and
Palmer (1996) framework
 5. Bachman and Palmer (1996) framework

• According to Bachman and Palmer (1996) task characteristics can be

synonymous with test method facets. Therefore, the characteristics
evaluation of the reading comprehension tests may be the optimal
means for the assessment of the learners’ task performance.
• Task characteristics model presented by Bachman and Palmer
(1996) and modified as test task characteristics (Bachman and
Palmer, 2010), was developed based on “test method facets model”
presented by Bachman (1990).
• Bachman (1990) provides a framework of task characteristics. This
framework includes a set of features that describe five aspects of
tasks: setting, test rubrics, input, expected response, and
relationship between input and response.
 Characteristics of the setting refer to all physical conditions under which
testing takes place. This includes the physical setting, participants, and the
time of task.
 The characteristic of rubric include: the organization (structure) of the
test, instructions, the duration of the test as a whole and of the individual
parts, and how the language that is used is evaluated and scored.
In a test task the instructions should be as explicit and clear as possible
because students should know exactly what to do and the teachers can make
safe decisions based on their performance on the test.
Structure of the test task shows how the different part of the test are put
together and presented to the test takers. This aspect deals with the number
of parts or tasks, the salience of the parts/tasks which shows how the
different parts are clearly distinguished from each other, the order or
sequence of parts/tasks, relative importance of tasks, and the number of
items in each part. Time allotment is the amount of time for each part of test
and the entire test. Scoring method specifies how numbers will be assigned to
test takers’ performance.
 characteristic of the input. Input consists of the material contained
in a given test task which the test takers are going to process in
some way and to which they are expected to respond. Input can be
studied from different aspects namely format and language. By
format Palmer means the way in which the input is presented.
Format includes channel, form, language, length, type, degree of
speediness, and vehicle.
• Channel determines the way of presenting the input which can be
aural, visual, or both. Also input can take the form of language, non-
language, or both. If the form is language, it can be the testees’
native language or target language. In terms of length, input can be
just a single word, a phrase, a sentence, a paragraph, or an
extended discourse.
• Type of input may be either an item or a prompt.
• The purpose of an item is to elicit either a selected or a limited
response. An example of a test item is the familiar multiple-choice
question.
• The purpose of a prompt is to elicit an extended production
response. A test task as summary writing contains input of this
type. Degree of speediness refers to the rate at which the test taker
has to process the information. Vehicle is the means by which input
is delivered which can be live or reproduced
 Characteristics of the expected response are another aspect of test
task features. Two factors are important: format and type of
response. The explanation of format is the same as what was said
about input.
• Genesee and Upshur (1996) state that when selecting language
tests, the response characteristics or demands of the test task
should be taken into consideration.
• They state that based on the response characteristics, test methods
can be described as closed-ended, limited, and open ended.
• For describing the type of response other measurement specialists
use three categories: selected, limited production, and extended
production.
• In a selected response no production is needed. example of
selected response is a multiple-choice test task
• A limited production response includes a single word or phrase, and may
be as long as a single sentence or utterance. In short completion items
and cloze test this type of response is required.
• If the test taker has to provide a response that is longer than a single
sentence and its range is from two sentences to a whole composition,
then the response is named an extended production one.
• The summary test task used in this study requires an extended response
and the test takers are to provide a summary of about 100 words. Other
features relating to response are the degree of speediness and the
language of expected response.
 The characteristic of relationship between input and response is
defined as broad scope or narrow scope. A broad scope requires the
language user to process a lot of input. The ‘main idea question’
that deals with the context of entire passage. The question that
focused on a specific detail or limited amount of the passage would
be a narrow scope.
6.Test development
 Banchman and Palmer (1996:87) regard test development as having
three components:
• Design
• Operationalisation
• Administration
 An alternative model taken from Alderson et al (1995) involving the
following stages of test construction and evaluation:
• Identifying test purpose
• Developing test specifications
• Guidelines for and training of item/task writers and moderations of
their products
• Pre-testing, analysis of results and revision of test
• Training examiners and administrators
• Monitoring examiner reliability
• Reporting scores and setting pass marks
• Test validation
• Post-test reports
• Developing and improving tests.
 7. Techniques for reading ability
• Discrete- point versus integrative technique
• The cloze test and gap-filling tests
• Multiple- choice techniques
• Alternative objective techniques:
• Matching Techniques
• Ordering Tasks
• Dichotomous Items
• Editing tests
• Alternative integrated approaches: the C-test and the cloze elide test
• Short answer tests: the free-recall test, the summary test and the gapped
summary
• Information-transfer techniques
• ‘Real-life’ method: the relationship between text types and test tasks
• Informal methods of assessment
 8. The development of reading ability
• National frameworks of attainment: reading
english as a first language, modern foreign
languages
• Reading scales: the ACTFL proficiency guidelines ,
the ALTE framework for language tests, band
describtors of foreign-language academic reading
performance ->two potraits(good and marginal
reader)
• Suites of tests of reading: the UCLES main suite,
certificates of communicative skill in english
(CCSE)
 9. The way forward: assessing the interaction
between reader and text process and strategies

• Process
• Strategies: learner strategies, reading strategies,
characterisation of strategies in textbooks and by
teachers, strategies during test-taking
• Insights into process: methods for eliciting or assessing:
introspection, interview and talk-back, classroom
conversations, immediate-recall protocols, miscue
analysis, self-assessment
• Miscellaneous methods used by researchers: encoding
time, word-identification processes, word-guessing
processes, metacognition, measures of achievement
• Computer-based testing and assessment
Reference
Assessing Reading. (2000). J. Charles Alderson.
Cambridge: Cambridge University Press.