June 2010

R E P O RT

e-Learning for Educators

Effects of On-Line Professional
Development on Teachers and
their Students:
Executive Summary
of Four Randomized Trials

Laura M. O’Dwyer, Jessica Masters,

Sheralyn Dash, Raquel Magidin De Kramer,
Andrea Humez, & Michael Russell

Technology and Assessment Study Collaborative

Lynch School of Education
Boston College
Chestnut Hill, MA 02467

For full report, please see:

www.intasc.org http://www.bc.edu/research/intasc/researchprojects/eLearning/efe.shtml
inTASC REPORT: e-Learning for Educators: Executive Summary of Four Randomized Trials, June 2010
2

Introduction
Over the past decade, there has been growing interest in on-line delivery of professional de-
velopment (OPD). This interest is driven largely by a desire to improve access to and convenience
of professional development, as well as improve cost-efficiency. Despite this increased interest in
OPD, there is a dearth of scientifically-based research exploring the effects of OPD on teacher and
student outcomes. To address this lack of evidence, researchers at Boston College conducted a set of
four randomized controlled trials that examined the effect of OPD on teacher knowledge, teacher
practices, and student achievement. For two trials, the effect of teachers’ participation on student
practices was also examined.
The four studies summarized in this executive brief are part of a larger project known as the e-
Learning for Educators (efe) Project. The efe Project is a ten-state initiative designed to expand each
state’s capacity to deliver high-quality OPD that addresses teacher quality and student achievement
needs. As part of the initiative, four randomized controlled trials were conducted with teachers
from multiple states to evaluate the effects of OPD on teachers’ knowledge and instructional prac-
tices, and on students’ content knowledge and practices. The four independent trials employed the
same research design, but focused on a single grade level and subject area: fourth grade English
language arts (ELA), fifth grade mathematics, seventh grade ELA, and eighth grade mathematics.
This executive brief summarizes findings from each of the four trials.

Study Design
Each trial was conducted across multiple states between January 2007 and June 2009 and in-
cluded three rounds of data collection, each spanning three school semesters. The trials compared
changes in teachers’ content knowledge and instructional practices, and student achievement be-
tween participants assigned to either a control or treatment group. For the two ELA trials, effects on
student practices were also examined. For each trial, teachers who volunteered to participate were
randomly assigned to the treatment or control group; students were grouped based on the assign-
ment of their teacher.
All teachers completed a pre-survey in the spring semester of their first year of participation.
Teachers in the treatment group then participated in a series of three OPD workshops: the first in
the spring semester of their first year of participation, the second in the fall semester of their second
year of participation, and the third in the spring semester of their second year of participation. Each
workshop was sustained over seven weeks and required approximately 4–6 hours of participation
per week from teachers. Therefore, teachers in the treatment group participated in approximately
100 hours of OPD related to best practices for teaching specific topics within mathematics or ELA.
Teachers in the control group were not restricted from participating in their normal professional
development activities (online or face-to-face). All teachers completed a post-survey at the end of
the spring semester of their second year of participation.
inTASC REPORT: e-Learning for Educators: Executive Summary of Four Randomized Trials, June 2010
3

Although the teachers participated in the study for three school semesters, the student partici-
pants were drawn from only two school semesters- the fall and spring semesters of the teachers’ sec-
ond year of participation. These students completed a pre-test as close as possible to the beginning
of the school year and a post-test as close as possible to the end of the school year.
For each trial, the OPD workshops were developed by the Education Development Center
(EDC) through a collaborative process among various stakeholders in the efe initiative. The work-
shops were designed to include both theoretical information as well as pedagogical techniques that
could be immediately applied in the classroom. The OPD workshops implemented a learning com-
munity model, which combined independent activities and activities to be completed with stu-
dents in the classroom, and placed a strong emphasis on facilitated peer-to-peer discussions. Each
workshop consisted of one orientation session and six sessions of content. Each of the six content
sessions involved three components: readings, activities, and discussions. The readings were drawn
from articles, book chapters, or reports. The activities often required teachers to view online videos
or work with existing classroom materials. For the discussion component, each teacher participant
was asked to respond to one or more questions related to the readings and activities via the work-
shop discussion board. In addition to completing the readings, activities, and discussions of the six
content sessions, the teacher participants were required to create a final product, which involved
developing an action plan or lesson plan based on the workshop content.
Table 1 shows the numbers of teachers and students who participated in each of the four trials:

Table 1: Number of Teacher and Student Participants

Control Treatment
Total
Group Group

4th Grade Number of Teachers 61 49 110

EL A Trial Number of Students 922 766 1,688

7th Grade Number of Teachers 45 35 80

ELA Trial Number of Students 1,225 831 2,056

5th Grade Number of Teachers 45 34 79

Mathematics Trial Number of Students 790 648 1,438

8th Grade Number of Teachers 43 28 71

Mathematics Trial Number of Students 1,062 799 1,889

Data analyses were conducted to estimate treatment effects for teachers and students. Analysis
of Covariance (ANCOVA) was used to estimate the treatment effects for the teacher outcomes,
and hierarchical linear modeling (HLM) was used to estimate the treatment effect for student out-
comes.
inTASC REPORT: e-Learning for Educators: Executive Summary of Four Randomized Trials, June 2010
4

Summary of Findings
Collectively, the four trials provide strong evidence that participation in a coordinated series of
three OPD workshops has positive effects on teachers’ instructional practices and content knowl-
edge. Compared to the control group teachers, larger changes in instructional practices occurred in
each trial for teachers in the treatment group. In many cases, the effect of the OPD workshops on
instructional practices was large. Across all four trials, larger changes in teacher content knowledge
also occurred for teachers in the treatment group. In most cases, the size of the effect was medium
or large.
In addition, the results of the trials provide evidence that teachers’ participation in a coordi-
nated set of three OPD workshops can have positive effects on their students. Although these effects
are smaller and occur less consistently across all the sub-domains targeted by the courses, in each
trial, a statistically significant effect was found for at least one student measure.
Table 2 (next page) provides a summary of the findings within each trial.
inTASC REPORT: e-Learning for Educators: Executive Summary of Four Randomized Trials, June 2010
5

Table 2: Summary of Statistically Significant Findings

4th ELA 7th ELA 5th Math 8th Math

TEACHER
Algebraic Proportional
Writing †3 Writing †2
Thinking †3 Reasoning †3
Instructional Geometric
Vocabulary †3 Vocabulary †2 Fractions †3
Practices Measurement †2
Reading Reading
Measurement †3 Functions †3
Comprehension †2 Comprehension
Algebraic Proportional
Writing †3 —
Thinking †2 Reasoning1
Geometric
Vocabulary †1 Vocabulary †1 Fractions1
Knowledge Measurement1
Reading Reading
Measurement †3 Functions1
Comprehension †3 Comprehension1
Composite ELA †3 Composite ELA †2 Composite Math †3 Composite Math †2
STUDENT
Proportional
Writing1 Writing1 Algebraic Thinking1
Reasoning1
Geometric
Vocabulary †1 Vocabulary1 Fractions1
Knowledge Measurement †1
Reading Reading
Measurement1 Functions †1
Comprehension1 Comprehension1
Composite ELA †1 Composite ELA1 Composite Math †1 Composite Math †1
Writing †1 Writing1 — —
Practice Reading Reading
— —
Comprehension1 Comprehension †1
† indicates statistically significant effect.
1
indicates a small effect size.
2
indicates a moderate effect size.
3
indicates a large effect size.

As shown in Table 2, findings from the fourth grade ELA trial indicate that teachers in the
treatment group had, on average, higher scores on the measures of instructional practices for writ-
ing, vocabulary, and reading comprehension than the teachers in the control group. The size of the
effects was large for writing and vocabulary, and medium for reading comprehension. Analyses also
indicate that teachers in the treatment group had higher scores on the measures of content knowl-
edge for writing, vocabulary, reading comprehension, and overall ELA than the teachers in the
control group. These effects were large for teachers’ reading comprehension scores, writing knowl-
edge scores, and overall ELA knowledge scores, and small for vocabulary knowledge scores.
Analysis of student data indicates that there were changes in student practices related to both
inTASC REPORT: e-Learning for Educators: Executive Summary of Four Randomized Trials, June 2010
6

reading comprehension and writing for students whose teachers were assigned to the treatment
group and participated in the OPD courses. These differences, however, were relatively small and
were statistically significant for writing, but not reading comprehension. Larger changes in stu-
dent knowledge scores were also found for students whose teachers were assigned to the treatment
group. Again, these differences were small, and were statistically significant for the overall ELA
score and vocabulary, but not for reading comprehension and writing.
Findings for the grade 7 ELA trial indicate that teachers who participated in the treatment group
had, on average, higher scores on the measures of instructional practices for writing and vocabu-
lary, but not reading comprehension, when compared to the teachers in the control group. The size
of the effects was medium for writing and vocabulary, and negligible for reading comprehension.
Analyses also indicate that, when compared to the control group, participation in the OPD courses
had statistically significant effects on teachers’ vocabulary and overall ELA knowledge scores, but
did not have a significant effect on reading comprehension scores. Teachers’ knowledge scores in
writing were not reported due to unreliability in the writing knowledge scale. These effects were
medium for teachers’ overall ELA scores and small for both vocabulary and reading comprehen-
sion scores. Analysis of student data indicates that there were changes in student practices related
to both reading comprehension and writing for students whose teachers were assigned to the treat-
ment group. These differences, however, were relatively small and were statistically significant for
reading comprehension, but not writing. Changes in student knowledge scores were also found for
students whose teachers participated in the OPD courses. However, differences between the control
and treatment groups were negligible and not statistically significant.
Findings from the grade 5 Mathematics trial indicate that teachers who participated in the
treatment group had, on average, higher scores on the measures of instructional practices for alge-
braic thinking, fractions, and measurement than the teachers in the control group. The size of the ef-
fects was large for all three content areas. Analyses also indicate that, compared to the control group,
participation in the OPD courses had statistically significant effects on teachers’ algebraic thinking,
measurement, and overall mathematics knowledge scores, but did not have a significant effect on
fractions knowledge scores. These effects were large for teachers’ overall knowledge of mathemat-
ics and measurement scores, medium for algebraic thinking scores, and small for knowledge of
fractions. Analysis of student data indicates that there were changes in student content knowledge
scores and that, compared to the control group, these changes were larger for students whose teach-
ers participated in the OPD courses. These differences, however, were relatively small and were
statistically significant for the overall mathematics scores, but not for the sub-scales.
Finally, findings from the grade 8 mathematics trial indicate that teachers who participated in
the treatment group, on average, had higher scores on the measures of instructional practices for
proportional reasoning, geometric measurement, and functions than the teachers in the control
group. The size of the effects was large for proportional reasoning and functions, and moderate for
geometric measurement. Analyses also indicate that, compared to the teachers in the control group,
participation in the OPD courses had a larger effect on proportional reasoning, geometric measure-
ment, and overall mathematics knowledge scores, and that the effect for the overall mathematics
knowledge scores was statistically significant. The effect for overall mathematics score was moder-
inTASC REPORT: e-Learning for Educators: Executive Summary of Four Randomized Trials, June 2010
7

ate, and small for proportional reasoning, geometric measurement, and function scores. Analysis of
student data indicates that there were changes in student content knowledge scores and that these
changes were larger for students whose teachers participated in the OPD courses. These differences
were relatively small, and statistically significant for geometric measurement, functions, and overall
mathematics scores, but not for proportional reasoning.

Summary
The series of four randomized controlled trials summarized here examined the effect that a se-
ries of three OPD courses had on teachers’ knowledge and instructional practices, and subsequent
effects on student achievement. Collectively, the four trials provide strong evidence that participa-
tion in a coordinated series of three OPD workshops has positive effects on teachers’ instructional
practices and content knowledge. Across all four trials, larger changes in teacher content knowledge
also occurred for teachers in the treatment group. Each trial also provides evidence that participa-
tion in a coordinated set of three OPD workshops also has effects on students of the teachers in the
workshops. Although these effects are smaller and occur less consistently across all the sub-domains
targeted by the courses, in each trial, a statistically significant effect was found for at least one student
measure. This is noteworthy given the timing of the collection of student outcome measures. For
each trial, student measures were collected within weeks after teachers completed the final work-
shop. Moreover, two of the three workshops were completed during the same year in which student
measures were collected. This timing provided very little time for teachers to alter their practices,
become comfortable with these practices, and subsequently tailor these practices to their teaching
style and the needs of their students. In addition, in some cases, students were likely exposed to
content that was targeted by the workshops prior to the teacher’s completion of the workshop.
Despite these shortcomings, the set of four randomized trials presented here provide a body of
evidence that a series of online professional development courses that target specific student learn-
ing needs can have positive effects on teacher knowledge and instructional practices. Importantly,
this study provides a body of evidence that teachers’ participation in a coordinated series of OPD
courses have effects that translate into improvements in targeted student outcomes. While further
research is warranted to better understand the design features that maximize the effects of OPD and
which explore the effects across a broader range of content areas and grade levels, the findings from
this study support the continued implementation of online professional development similar to that
developed and delivered in the e-Learning for Educators project.