Listening. Learning. Leading.

ETS Global Institute

®

2014 Course Catalog

Capacity Building in the Service of Learning
 Global Institute

Contents
About ETS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
ETS® Global Institute. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
What does the ETS® Global Institute do?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
What does the ETS Global Institute offer?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Why train with ETS?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
ETS® Global Institute Faculty. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
A Sampling of Global Institute Trainer Biographies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4–6
Highlights. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Countries Served. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Scheduled Courses. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Sample Agenda. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Customized Courses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Customized Course Descriptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Session Topic Descriptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12–14
Administering and Scoring Performance Assessments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Conceptual Overview of Psychometric Concepts in Testing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Design and Development of Performance-Based Assessments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Designing a Field Test. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Developing Test Specifications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Equating Test Scores. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
ETS Audit Process. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Evidence-Centered Design. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Introduction to Item Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Introduction to Reliability and Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Item Banking. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Item Writing Guidelines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Measuring Critical Thinking. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Multiple-Choice Item Writing and Review Workshop. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Online Scoring Network. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Scaling Test Scores. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Standard Setting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Test Development Process. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Test Fairness and Differential Item Functioning (DIF). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Test Security Procedures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Frequently Asked Questions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Expression of Interest. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16–18

ETS® Global Institute 2014 Course Catalog

www.ets.org/globalinstitute
 Global Institute

About ETS
At ETS, we advance quality and equity in education for people worldwide by creating assessments
based on rigorous research. Founded as a nonprofit in 1947, ETS develops, administers and scores
more than 50 million tests annually — including the TOEFL® and TOEIC® tests, the GRE® tests and
The Praxis Series™ assessments — in more than 180 countries, at over 9,000 locations worldwide.
In addition to assessments, we conduct educational research, analysis and policy studies, and we
develop a variety of customized services and products for:
• teacher certification
• English language learning
• elementary, secondary and post-secondary education

ETS serves individual students, their parents, educational institutions and government agencies. We
help teachers teach, students learn, and parents measure the educational and intellectual progress
of their children by:
• Listening to educators, parents and critics
• Learning what students and their institutions need
• Leading in the development of new and innovative products and services

Our Mission: To advance quality and equity in education by providing fair and valid assessments,
research and related services. Our products and services measure knowledge and skills, promote
learning and educational performance, and support education and professional development for
all people worldwide.
Our Vision: To be recognized as the global leader in providing fair and valid assessments, research
and related products and services to help individuals, parents, teachers, educational institutions,
businesses, governments, countries, states and school districts, as well as measurement specialists
and researchers.
Our Values: Social responsibility, equity, opportunity and quality. We practice these values by
listening to educators, parents and critics. We learn what students and the institutions they attend
need. We lead in the development of products and services to help teachers teach, students learn
and parents measure the intellectual progress of their children.

1 ETS® Global Institute 2014 Course Catalog

www.ets.org/globalinstitute
 Global Institute

ETS® Global Institute

What does the ETS® Global Institute do?
The ETS® Global Institute is the international training and technical assistance arm of
Educational Testing Service (ETS). Its mission is to advance the fields of testing, assessment
and education by making ETS staff resources available to educators, policymakers and testing
professionals worldwide.
The ETS® Global Institute carries out this mission by:
• c ollaborating with ministries of education, universities and professional organizations to
explore today’s increasingly complex educational measurement challenges
• p
 roviding a place where the world’s test creators and policymakers can exchange ideas on
topical issues
• p
 roviding practical, “how-to” advice to educators, policymakers and testing professionals
through seminars and courses

What does the ETS Global Institute offer?

A distinguished faculty of professionals and external assessment experts participate in
three modes of training:
• Scheduled 3–5 day courses on topics related to large-scale standardized testing
• Customized courses tailored to specific client needs
• Distance learning “mini sessions”

Training and Capacity Building

The ETS Global Institute helps build local capacity by providing a professional development
opportunity that enhances the knowledge base of educational leaders, classroom teachers and
school directors. Additionally, these workshops introduce clients to ETS’s other research-based
professional development programs.

Why train with ETS?

After participating in ETS Global Institute programs, clients have said that they were able to return
to their jobs and immediately put the knowledge they gained into practice.
Through their participation, they:
• d
 eveloped an understanding of how standards and test specifications must be aligned to
test items
• learned how to determine the quality of tests and the most important aspects of classical test
theory and item-response theory
• d
 iscovered how to conduct psychometric analysis of test results and how to base suggestions
for improvement on test results
• learned the principles and benefits of test-score scaling and equating in large-scale assessments

How might you and your institution or organization benefit from the
same knowledge?

ETS® Global Institute 2014 Course Catalog 2

www.ets.org/globalinstitute
 Global Institute

ETS® Global Institute Faculty

The ETS Global Institute faculty is made up of practicing professionals who bring firsthand
experience to all courses. These testing specialists include:
• Test developers (with expertise in subjects including math, science, ESL/EFL and history)
• Psychometricians (measurement statisticians)
• Managers of testing programs
• Information technology specialists
• Experts in test security
• Communications professionals
• Trainers of teachers and school leaders

Language
Instruction is normally delivered in English; however, some customization in other languages may
be possible.

Fees
Course fees vary by program. Please contact the ETS Global Institute at globalinstitute@ets.org
for more information.

Location
The ETS Global Institute is based at the ETS headquarters in Princeton, New Jersey, USA.

3 ETS® Global Institute 2014 Course Catalog

www.ets.org/globalinstitute
 Global Institute

A Sampling of Global Institute

Trainer Biographies
Eugenio J. Gonzalez, Ph.D.
Eugene J. Gonzalez is currently the Director of the IEA-ETS Research Institute (IERI), a collaborative
effort between the International Association for the Evaluation of Educational Achievement
(IEA) and ETS that focuses on improving the science of large-scale assessments. The IERI
undertakes activities around three broad areas of work that include research studies related to
the development and implementation of large-scale assessments; professional development and
training; and dissemination of research findings and information gathered through large-scale
assessments. He is also head of the Research and Analysis Unit at the IEA-Data processing and
Research Center.
Gonzalez formerly served as the Director of Quality Control and Field Operations for the National
Assessment of Educational Progress (NAEP), and as Director of International Operations and Data
Analysis in the International Study Centre (ISC) at Boston College. In this role, he oversaw the
development and implementation of international operations and data analysis procedures for
the Trends in International Mathematics and Science Study (TIMSS) and Progress in International
Reading Literacy Study (PIRLS). Gonzalez received his Ph.D. in educational research, measurement and
evaluation in 1994 from Boston College. He received his undergraduate degree in psychology from the
Universidad Católica Andres Bello in Caracas, Venezuela, and is a licensed psychologist in Venezuela.

Richard J. Tannenbaum, Ph.D.

Richard J. Tannenbaum is the Senior Research Director for the Center for Validity Research in the
Research & Development division at ETS. In this capacity, he oversees the fundamental and applied
research that supports the validity associated with score use and score interpretation for testing
programs serving higher education, K–12 students and educator licensure. Tannenbaum also
co-directs the Validity Initiative, which supports and conducts innovative research in various areas
related to validity.
Prior to his current position, he was the Director of the Licensure and Certification Research
area (within the Validity Center). Tannenbaum’s primary responsibilities included providing
measurement expertise in support of educator licensure and building both standard-setting
capabilities and a standard-setting research agenda.
He joined the ETS in 1988 as an associate research scientist and has held progressively higher-level
positions over the years, including positions in strategic business units.
Tannenbaum earned a Ph.D. in industrial/organizational psychology from Old Dominion University.
He has published numerous articles, book chapters and technical papers. His areas of expertise
include standard setting, assessment development, licensure and certification, and validation.
He also has earned the ETS Presidential Award for Extraordinary Accomplishments for his innovative
design and implementation of a multistate standard-setting process.

ETS® Global Institute 2014 Course Catalog 4

www.ets.org/globalinstitute
 Global Institute

Jean C. Williams, Research Project Manager

Jean C. Williams is a research project manager working with human constructed-response (CR)
scoring. She joined ETS in 2001 as an assessment specialist working primarily in the area of item
development and scoring for writing ETS’s PPST® and GRE® assessments, and the College Board® SAT®
and CLEP® assessments, along with working on an LSAT® pilot assessment.
Since then, she has worked with a range of tests containing constructed-response items and
has been responsible for item and rubric development and review, overseeing CR scoring and
monitoring scoring leaders, and developing scoring and training procedures and policies. She
served as Coordinator for the Praxis™ and School Leaders Series Educational Leadership tests and
led development of the revised School Leaders Licensure Assessment.
In her current role, Williams is involved in developing rater and scoring leader policies and training
materials for the Teachscape/MET project, along with pursuing interests in rater note-taking effects
and working on a longitudinal study of undergraduate writing assessment. Over the years she
has taken on increasing responsibilities with various CR policy and standards initiatives, and she
is currently involved with the development and implementation planning for ETS’s new online
scoring system.

Samuel A. (“Skip”) Livingston

Samuel A. (“Skip”) Livingston has been at ETS since 1974, where his work has included the
development of performance tests in the health professions, research on methods of equating
test scores, and the planning and coordination of statistical operations for college placement
tests, college outcomes assessments, and teacher and school administrator licensing tests.
He has served as a consultant to professional associations, government agencies and private
corporations. He also has served on the board of advisory editors for three professional journals
and is the author of articles and book chapters on test score equating, item analysis, reliability
estimation and standard setting.

Xuan (Adele) Tan, Ph.D.

Xuan (Adele) Tan is a psychometrician in the Research & Development division of ETS. She earned
her Ph.D. in educational psychology with an emphasis in measurement, evaluation and cognition
at the University of Alberta in 2006. She joined ETS’s teacher quality group as an associate
psychometrician in 2006. She serves as the Statistical Coordinator for a set of test titles in the
Texas Teacher Certification Program and in The Praxis Series where her operational work focuses on
classical test theory, test-score equating and differential item functioning. Her research interests
include mixed-format test equating, CR scoring shift and differential item functioning.

Diego Zapata-Rivera, Ph.D.

Diego Zapata-Rivera is a research scientist in the Research & Development division of ETS. He
earned a Ph.D. in computer science (with a focus on artificial intelligence in education) from
the University of Saskatchewan in 2003. His current research interests include the design,
development and evaluation of adaptive, assessment-based gaming and learning environments,
evidence-centered design and evaluation, virtual communities, English Language Learning (ELL),
Bayesian student modeling, open student models and the design and evaluation of innovative
score reports that clearly communicate assessment information to educational stakeholders.
Zapata-Rivera has produced more than 90 publications including journal articles, book chapters
and technical papers. He has also served as a reviewer for several international conferences and
journals. He has been a committee member and organizer of international conferences and
workshops in his research areas.

5 ETS® Global Institute 2014 Course Catalog

www.ets.org/globalinstitute
 Global Institute

Philip Everson, Ph.D.

Philip Everson is currently a principle assessment designer specializing in ELL in the Assessment
division of ETS. ETS ELL products and services include the TOEFL and TOEIC tests among others. He
has led the design of a number of new products, including the new TOEIC Speaking test and the
new TOEIC Writing test. He served as a director of English-language tests during the design and
implementation of the TOEFL iBT® test. He has served as Director of Product Development for ELL.
From 1995 to 2000, Everson supervised the ELL test development staff of the Assessment division,
the group responsible for the creation of content for the TOEFL test, ETS’s largest computer-based
test. He began work in TOEFL test development as an item writer in 1990.
Prior to joining ETS, Everson taught English at the University of Delaware, Temple University
(Philadelphia) and Temple University Japan (Tokyo). He holds a Ph.D. in English literature from the
University of Delaware.

Highlights
Global Institute Reflects ETS’s Worldwide Mission
As the international training and technical assistance arm of ETS, the Global Institute seeks to
advance the fields of testing, assessment and education by making ETS staff resources available
to educators, policymakers and testing professionals worldwide.
It collaborates with ministries of education, universities and professional organizations around
the world, offering courses related to large-scale standardized testing either in Princeton or in
host countries.
We want to share what we have learned about testing and test takers with our international
colleagues around the world. In turn, we hope to learn about what assessment challenges our
international colleagues in education have faced in their own countries.

Highlights of Global Institute Initiatives

The ETS Global Institute has hosted workshops on a variety of measurement topics for audiences
from around the world. For example, the Global Institute conducted performance assessment
workshops for the United Nations Human Resource unit (New York City); large-scale assessment
courses for the Zhejiang Provincial Education Examinations Authority (China), the Agency of
Quality in Education (Chile), and World Bank (Ethiopia); and psychometric methods courses for
governmental testing agencies from Nigeria, Taiwan, Mexico, Malaysia and Vietnam.

ETS® Global Institute 2014 Course Catalog 6

www.ets.org/globalinstitute
 Global Institute

Countries Served

• Albania • Hong Kong • Republic of Georgia

• Argentina • Honduras • Russia
• Azerbaijan • India • Saudi Arabia
• Barbados • Japan • Singapore
• Bolivia • Kazakhstan • South Africa
• Botswana • Kyrgyzstan • Sweden
• Brazil • Macedonia • Taiwan
• Canada • Malaysia • Tanzania
• Chile • Mexico • Thailand
• China • Nicaragua • UAE
• Colombia • Nigeria • UK (England, Scotland)
• Costa Rica • Norway • Uzbekistan
• Cyprus • Oman
• Ethiopia • Pakistan

7 ETS® Global Institute 2014 Course Catalog

www.ets.org/globalinstitute
 Global Institute

Scheduled Courses
Held at ETS headquarters in Princeton, New Jersey, USA

Date Title Content

March 3–7, 2014 ; Design and Development • Deciding what to measure • Item review and pre-testing
June 2–6, 2014 of Large-Scale Assessments • Developing specifications • Item analysis and differential
• Determining question item functioning
(item) types and writing to • Performance measures
specifications • Follow-up studies
to determine
predictive validity

April 28 –May 2, 2014 Development of Computer- • Evolution of computer-based • Computer-based

Based Testing Programs testing (CBT) delivery models
• Considerations in CBT • Consideration of students
• Item banking with special needs in
• Automated scoring test design
• Test security issues • Score reporting

September 22–26, 2014 Design and Development • Design and development • Test development procedures
of National Assessments of national assessments • Constructed-response
• Development of frameworks development and scoring
and test content • Communication of results
specifications to key stakeholders
• Psychometric models
• Item banking

December 1–5, 2014 Psychometrics Methods • Overview of psychometric • Scale development

Content concepts • Item response theory (IRT)
• Developing psychometric • Item/test analyses
specifications • Item banking
• Reliability and standard error • Standard setting
of measurement • Validity
• Automated essay grading • Test-score equating
and scaling

Note: Current course schedules require a minimum of eight participants.

ETS® Global Institute 2014 Course Catalog 8

www.ets.org/globalinstitute
 Global Institute

Sample Agenda
Design and Development of Large-Scale Assessments

Topic Learning Objective

Test Development Process This workshop session reviews the general steps of the ETS test development process.

Developing Test Test specifications are the blueprints that test developers use when creating a test. This
Specifications session describes how test specifications are developed.

Evidence-Centered Design This session presents a method of designing tests that directly links item and test
development to the knowledge, skills and abilities to be tested, claims to be made and
evidence to be collected.

Item Writing/Reviewing Guidelines This session reviews general item writing and reviewing guidelines applicable to
multiple-choice tests. Time permitting, participants practice writing and reviewing
items using the guidelines.

Design and Development of This session outlines the guidelines and procedures for developing performance
Performance Assessments assessments (non-multiple-choice assessments). Rubrics and essays from ETS tests
are used as examples.

Administering and Scoring This session outlines the methods and logistics of essay scoring. Time permitting,
Performance Assessments participants also gain experience reviewing essays.

Conceptual Overview of This session provides a summary of the basic psychometric concepts behind test
Psychometric Concepts in Testing development: item analysis, differential item functioning (DIF), test-score equating and
scaling, score interpretation, reliability, validity and fairness.

Equating and Scaling of Test Score This session discusses the statistical issues involved in constructing comparable
test forms.

Standard Setting This session provides an overview of the methodologies involved in establishing cut
scores in large-scale assessments.

Test Fairness and This session discusses the issues surrounding test fairness and equity vs. equality. It also
Differential Item reviews the international principles and guidelines for fairness in testing. The session
Functioning (DIF) also introduces DIF, the analysis of how matched people in different groups perform
on a test item.

9 ETS® Global Institute 2014 Course Catalog

www.ets.org/globalinstitute
 Global Institute

Customized Courses
In addition to offering regularly scheduled courses, the ETS Global Institute creates and
conducts courses customized to the specific needs of our clients. We have developed courses
to meet the expressed training requirements of ministries of education, universities and other
organizations worldwide.
Customized courses are held in Princeton, New Jersey, and can be of varying lengths. The
curriculum is shaped according to the needs of the client. Below is a listing of recent customized
courses along with the countries that requested them.

Examples of past ETS Global Institute customized courses

Item Writing and Review in Content Areas – Chile, England, Ethiopia, Kyrgyzstan, Thailand
Issues in Teacher Licensure Testing – Chile, Malaysia
Development of Computer-Based Testing Programs – Canada, Hong Kong,
Netherlands, Taiwan
Developing Large-Scale Performance Assessments – Barbados, Nicaragua
Psychometric Methods – Chile, Colombia, Malaysia, Saudi Arabia
Developing a National Assessment Program – Barbados, Republic of Georgia
Design and Development of Language Assessments – Thailand
Upon request, a customized course can also include a tour of ETS’s operations facilities; time in
ETS’s Carl Campbell Brigham Library, which houses a comprehensive collection of educational
testing and psychometric literature; or a visit to a local school or university.

To register or for more information, contact:

ETS Global Institute
Rosedale Road, MS 04-L
Princeton, NJ 08541, USA
Phone: +1-609-252-8699
Email: globalinstitute@ets.org
Visit: www.ets.org/globalinstitute

ETS® Global Institute 2014 Course Catalog 10

www.ets.org/globalinstitute
 Global Institute

Customized Course Descriptions

Item Writing and Reviewing in Content Areas
Course Content: Item writing/review procedures • Setting cut scores and score scaling • SAT I and II overview
• Advanced Placement® overview • Test security and administration • Life cycle of a test • Equating test scores

Issues in Teacher Licensure Testing

Course Content: Evidence-centered models for teacher licensure testing • Score reporting • Portfolio
assessments • Statistical methods for licensure testing • Standard setting • Equating procedures for multiple
selection/performance assessments • Operational issues • Registration systems

Development of Computer-Based Testing Programs

Course Content: Evolution of computer-based testing (CBT) • Key issues in CBT • Security in the CBT test
development process • Developing and managing item pools for computer adaptive tests • Managing the
transition to a CBT program • Psychometric research on test security for CBT • Developing and managing a
CBT network and electronic irregularity reporting • Conducting test security investigations for CBT

Design of Large-Scale Performance Assessments

Course Content: Development and evaluation of portfolios • Advanced placement models • Structure
of large-scale performance scoring • Preparing and reviewing essay questions and scoring guide
• Planning and carrying out validity studies • Developing training materials • e-rater® scoring engine
• Key psychometric concepts • Assessing classroom performance

Psychometric Methods
Course Content: Overview of psychometric concepts • Developing psychometric specifications • Validity
• Reliability and standard error of measurement • Test and item fairness • Equating test scores and scale
development • Item analysis • Test analyses • Measurement models/IRT • Automated essay grading

Developing a National Assessment Program

Course Content: Development of U.S. National Assessment of Educational Progress • Managing a
large-scale assessment program • Design, sampling and analysis • Writing test items for national assessments
• Current status of national and international assessments • Developing specifications/curriculum
frameworks for national assessments • Model for developing a national assessment • Test administration/test
security • Reporting national assessment results • Developing accessible databases • Scoring essays using
e-rater scoring engine

Design and Development of Language Assessments

Course Content: Managing a second-language assessment program • Design of test specifications
• Procedures for training item writers • Assessment of listening comprehension skills • Performance
assessments (speaking/writing item rubric development) • Oral proficiency interview procedures
• Contemporary research issues in language assessment • Benchmarking to the Common European
Framework (CEF)

11 ETS® Global Institute 2014 Course Catalog

www.ets.org/globalinstitute
 Global Institute

Session Topic Descriptions

Below are descriptions of the individual sessions available through the ETS Global Institute.

Administering and Scoring Performance Assessments

Target audience: Test Developers, Test Administrators
Learning objectives: This session outlines the methods and logistics of essay scoring. Participants also gain
experience reviewing essays.
Approximate length: 4+ hours
Noteworthy feature: Interactive

Conceptual Overview of Psychometric Concepts in Testing

Target audience: Policymakers, Test Developers, Measurement Statisticians (Psychometricians)
Learning objectives: This session provides a summary of the basic psychometric concepts behind test
development: item analysis, differential item functioning, equating test scores and scale development,
scaling, score interpretation, reliability, validity and fairness.
Approximate length: 2–4 hours

Design and Development of Performance-Based Assessments

Target audience: Test Developers
Learning objectives: This session outlines the guidelines and procedures for developing performance
assessment (non-multiple-choice assessments). Rubrics and essays from ETS tests are used as examples.
Approximate length: 2+ hours

Designing a Field Test

Target audience: Policymakers, Test Developers, Measurement Statisticians
Learning objectives: This session discusses the purpose of a field test and the roles that test developers
and psychometricians play in designing and administering these tests.
Approximate length: 1.5+ hours

Developing Test Specifications

Target audience: Policymakers, Test Developers
Learning objectives: Test specifications are the blueprints that test developers use when creating a test.
This session describes how test specifications are developed.
Approximate length: 1+ hour

Equating Test Scores

Target audience: Policymakers, Measurement Statisticians
Learning objectives: This session discusses the benefits and methods of reviewing various forms of a
given test to determine whether they are of equal or comparable difficulty.
Approximate length: 1–1.5+ hours

ETS® Global Institute 2014 Course Catalog 12

www.ets.org/globalinstitute
 Global Institute

ETS Audit Process

Target audience: Policymakers, Test Administrators
Learning objectives: This session discusses the process by which ETS determines whether tests and
products adhere to company standards for quality and fairness.
Approximate length: 2 hours

Evidence-Centered Design
Target audience: Policymakers, Test Developers
Learning objectives: This session presents a method of designing tests that directly links item and test
development to the knowledge, skills and abilities to be tested, claims to be made, and evidence to
be collected.
Approximate length: 1.5+ hours

Introduction to Item Analysis

Target audience: Test Developers, Measurement Statisticians
Learning objectives: This session discusses how and when items and tests should be analyzed.
Approximate length: 2+ hours

Introduction to Reliability and Validity

Target audience: Policymakers, Test Developers, Measurement Statisticians
Learning objectives: This session defines reliability and validity and methods of determining reliability
of scores and validity of tests.
Approximate length: 2+ hours

Item Banking
Target audience: Policymakers, Test Developers, Measurement Statisticians
Learning objectives: This session provides the history, procedures and uses of item banks; and allows
participants to view screenshots and operating demos of the ETS item banking system.
Approximate length: 2+ hours

Item Writing Guidelines

Target audience: Policymakers, Test Developers
Learning objectives: This session reviews general item writing guidelines, applicable to both multiple-
choice and constructed-response items.
Approximate length: 1.5+ hours

Measuring Critical Thinking

Target audience: Policymakers, Test Developers
Learning objectives: This session defines critical thinking and provides models of measuring critical
thinking using both multiple-choice and constructed-response items.
Approximate length: 1.5+ hours

13 ETS® Global Institute 2014 Course Catalog

www.ets.org/globalinstitute
 Global Institute

Multiple-Choice Item Writing and Review Workshop

Target audience: Test Developers
Learning objectives: This session teaches the guidelines specific to multiple-choice item writing.
Participants practice writing and reviewing items using guidelines.
Approximate length: 1–2+ days
Noteworthy feature: Very interactive

Online Scoring Network

Target audience: Policymakers, Test Administrators
Learning objectives: This course introduces the features and processes of ETS’s Online Scoring
Network (OSN).
Approximate length: 1.5+ hours
Noteworthy feature: Requires special network connection to log in to the OSN.

Scaling Test Scores

Target audience: Policymakers, Measurement Statisticians
Learning objectives: Many assessments assign scores to test-taker performance. This session discusses
the types of score scales that can be applied to your assessments.
Approximate length: 1–1.5 hours

Standard Setting
Target audience: Policymakers, Test Developers, Researchers
Learning objectives: This session defines standard setting and discusses the process
of performing a standard-setting session.
Approximate length: 1.5 hours–1 day long

Test Development Process

Target audience: Policymakers, Test Developers
Learning objectives: This session reviews the general steps of the ETS test development process.
Approximate length: 1+ hour

Test Fairness and Differential Item Functioning (DIF)

Target audience: Test Developers, Measurement Statisticians
Learning objectives: This session discusses test fairness — equity vs. equality — and reviews the
international principles and guidelines for item writing. This course also introduces DIF, the analysis
of how matched people in different groups perform on an item.
Approximate length: 2+ hours

Test Security Procedures

Target audience: Policymakers, Test Administrators
Learning objectives: This course addresses the three main steps of test security:
• Prevention
• Detection
• Remediation
Approximate length: 1 hour

ETS® Global Institute 2014 Course Catalog 14

www.ets.org/globalinstitute
 Global Institute

Frequently Asked Questions

What does the course fee cover?
The course fee covers instruction, course materials and lunches for each day of the course. All
charges associated with air travel, ground transportation, overnight accommodations and meals
other than lunch are paid by the client.

Where can we stay while attending an ETS Global Institute?

The Chauncey Conference Center is located on the ETS campus, within walking distance of the
building where courses are given. Complimentary breakfast is included in the room rate.
Chauncey Conference Center Hyatt Place® Princeton Residence Inn® Princeton
Rosedale Road Located at Princeton MarketFair at Carnegie Center
Princeton, NJ 08541 3565 U.S. Route 1 Located at Princeton MarketFair
+1-609-921-3600 Princeton, NJ 08540 3563 U.S. Route 1
+1-609-734-5600 +1-609-720-0200 Princeton, NJ 08540
www.chauncey.com +1-609-720-0201 Fax +1-609-799-0550
princeton.place.hyatt.com +1-609-799-2626 Fax
Please try one of the other hotels if www. marriott.com
Sonesta ES Suites®
Chauncey is not available.
4375 U.S. Route 1 South
Princeton, NJ 08540
+1-609-951-0009
+1-609-951-0696 Fax
www.sonesta.com

What ground transportation is available from airports or train stations?

By Train. From New York, Newark, Philadelphia and other stations along the Northeast Corridor line
of Amtrak®, take any train scheduled to stop at Princeton Junction (New Jersey Transit trains are
also available). A shuttle train runs from Princeton Junction to Princeton Station. From there it is a
short taxi ride to ETS. The Washington Metroliner stops in Trenton. Taxis are available there as well
(Approximate taxi fares to ETS: $14 from Princeton Downtown Station, $22 from Princeton Junction
Railroad Station and $40 from the Trenton Railroad Station).
By Air. Bus service is available between the Chauncey Conference Center and Newark and JFK
airports. For reservations call +1-800-385-4000. Also, shuttle service is available through Olympic
Airporter at +1-800-822-9797.
Princeton Taxi Information:
Roy’s Car Service at +1-609-430-8342 King Limousine Service at +1-609-951-0444
Nassau Taxi Stand at +1-609-924-1222 Touch Of Class Limo at +1-800-367-6317
A-1 AAA Princeton Taxi at +1-609-921-1177 Flyte Tyme Limo at +1-800-920-5466 (LIMO)
Gem Limousine Service at +1-800-223-1161

Where can I find information about the weather in Princeton, N.J.?

Please check www.weather.com for weather information.

How will I locate different buildings at the ETS Campus?

Please refer to the ETS Princeton Campus map at www.ets.org/about/locations.

15 ETS® Global Institute 2014 Course Catalog

www.ets.org/globalinstitute
 Global Institute

Expression of Interest
If you are interested in taking an ETS Global Institute scheduled course, or requesting a customized
course, please fill out the information below. In addition to our scheduled courses, the ETS Global
Institute specializes in developing customized programs uniquely tailored to meet the specific
needs of our international clients.

YOUR NAME

INSTITUTION NAME 

EMAIL ADDRESS

FAX NUMBER

Scheduled Courses

❒ Design and Development of Large-Scale Assessments

Course Content: Deciding what to measure • Developing specifications • Determining question (item) types
and writing to specifications • Item review and pre-testing • Item analysis and differential item functioning
• Performance measures • Follow-up studies to determine predictive validity
March 3–7; and June 2–6, 2014

❒ Development of Computer-Based Testing Programs

Course Content: Evolution of computer-based testing (CBT) • Considerations in CBT • Item banking
• Automated scoring • Test security issues • Computer-based delivery models • Consideration of students
with special needs in test design • Score reporting
April 28–May 2, 2014

❒ Design and Development of National Assessments

Course Content: Design and development of national assessments • Development of frameworks and test
content specifications • Psychometric models • Item banking • Test development procedures • Constructed-
response development and scoring • Communication of results to key stakeholders
September 22–26, 2014

❒ Psychometric Methods
Course Content: Overview of psychometric concepts • Developing psychometric specifications •
Reliability and standard error of measurement • Automated essay grading • Equating test scores and scale
development • Item response theory (IRT) • Item/test analyses • Item banking • Standard setting • Validity
December 1–5, 2014

ETS® Global Institute 2014 Course Catalog 16

www.ets.org/globalinstitute
 Global Institute

Courses of Interest

❒ Design and Development of Large-Scale Assessments

Course Content: Deciding what to measure • Developing specifications • Determining question (item) types
and writing to specifications • Item review and pre-testing • Item analysis and differential item functioning
• Performance measures • Follow-up studies to determine predictive validity

❒ Development of Computer-Based Testing Programs Content

Course Content: Evolution of computer-based testing (CBT) • Item banking • Automated scoring • Test
security issues • Computer-based delivery models • Consideration of students with special needs in test
design • Score reporting

❒ Design and Development of Teacher Assessments

Course Content: Establishing testing framework based on national standards • Using evidence-centered
design and job analysis methodology to establish test validity • Item writing and reviewing to meet content
specifications • Use of external experts • Standard setting to establish cut scores • Performance assessments
• Test equating • Application of IRT

❒ Design and Development of Language Assessments

Course Content: Managing second language assessment program • Design of test specifications
• Procedures for training of item writers • Assessment of reading comprehension skills • Assessment of
listening comprehension skills • Performance assessment (Speaking principles) • Performance assessment
(Writing principles) • Contemporary research issues in language assessment

❒ Psychometric Methods
Course Content: Overview of psychometric concepts • Developing psychometric specifications •
Reliability and standard error of measurement • Automated essay grading • Equating test scores and scale
development • Item response theory • Item/test analyses • Item banking • Standard setting • Validity

❒ Design and Development of National Assessments

Course Content: Design and development of national assessments • Development of frameworks and test
content specifications • Psychometric models • Item banking • Test development procedures • Constructed
response development and scoring • Communication of results to key stakeholders

❒ Design and Development of Performance Assessments

Course Content: Psychometric concepts for performance assessments • Evidence-centered design
development of scoring rubrics • Performance assessment item writing and review • Scoring
performance assessments • Use of technology in performance assessments • Performance assessment
in computer-based testing programs • Performance assessment in teacher evaluation programs
• Standard setting • Current research

17 ETS® Global Institute 2014 Course Catalog

www.ets.org/globalinstitute
 Global Institute

Customized Courses
Preferred Topic (see list of customized courses or describe on separate sheet of paper)

Name and contact information for other person(s) interested in

ETS Global Institute courses

NAME

INSTITUTION NAME

ADDRESS LINE 1

ADDRESS LINE 2

FAX NUMBER TELEPHONE NUMBER

EMAIL ADDRESS

Please mail or fax to:

ETS Global Institute
Rosedale Road, MS 04-L
Princeton, NJ 08541, USA
Phone: +1-609-252-8699
Email: globalinstitute@ets.org
Visit: www.ets.org/globalinstitute

ETS® Global Institute 2014 Course Catalog 18

www.ets.org/globalinstitute
 Global Institute

Notes:

19 ETS® Global Institute 2014 Course Catalog

www.ets.org/globalinstitute
ETS Global Institute
®

2014 Course Catalog

Capacity Building in the Service of Learning

772161

Copyright © 2014 by Educational Testing Service. All rights reserved. ETS, the ETS logo, LISTENING. LEARNING. LEADING., E-RATER, GRE, PPST, TOEFL,
TOEFL IBT and TOEIC are registered trademarks of Educational Testing Service (ETS). PRAXIS and THE PRAXIS SERIES are trademarks of ETS. Advanced
Placement, CLEP, LSAT and SAT are registered trademarks of the College Board. All other trademarks are the property of their respective owners. 24754