Ist626 Qms 1 8
Ist626 Qms 1 8
Ist626 Qms 1 8
Statement of Work
The Online Teaching Workshop is intended for CSUMB faculty who will be
incorporating online instruction into their curriculum delivery. The workshop is comprised of
five sections. Two of the sections are the focus of this statement of work: Quality Matters
Standards 1-4 and 5-8. Created in Adobe Captivate, the modules feature the following
instructional tools:
Interactive tools that generate automated responses and feedback based on data entry and
other methods
A variety of self-check activities, including multiple choice, drag and drop, and true or
false quizzes
target audience needs to acquire the skills illustrated in order to create valuable online
instructional content.
Online Presence and Interaction
The goal of both modules is to apply the Quality Matters rubric to an online course that best
supports student engagement via online learning. The QMS 1-4 module will instruct learners
about the following standards: Course Overview and Introduction; Learning Objectives;
Assessment and Measurement; and Instructional Materials. The QMS 5-8 module will instruct
learners about the following standards: Course Activities and Learner Interaction; Course
Technology; Learner Support; and Accessibility and Usability. The two content modules were
developed separately, with the intention of delivering them to the learners as a single module.
Content Delivery
Online education is a great opportunity for busy adults who are unable to attend a
traditional university. California State University offers a variety of online courses, which are
created using rigorous standards. New programs are added yearly. At the end of the Online
Teaching Workshop, a student participating in this module will be able to elaborate about the
importance of the Quality Matters Standards, and the role the standards play in CSUMBs online
curriculum.
Current technologies in online education include access to e-books, multimedia
presentations, and audio and video conferencing. All of these platforms include images,
narrations, and audiovisuals. The software required for producing and sharing digital content
includes several types of software programs and applications, such as e-learning platforms or
Learning Management Systems, (LMS), screen-sharing online forum programs such as Zoom
and Blackboard Collaborate, and online resources such as YouTube and Atomic Learning.
Without the proper structure these elements can become overwhelming and their inclusions can
be more distracting to the learner than helpful in communicating content.
Crafting good learning objectives that drive the structure of the course, and relating
digital resources to the course objectives is an essential key that our modules will cover.
Writing assessments to match course objectives will also be a focus of the modules presented.
Overall we want to promote the best practices for developing an effective and engaging online
class with all of the Quality Matters Standards considered and utilized within the course.
Assessment
The Quality Matters modules will instruct the faculty how to develop a course for an
online environment. The modules will focus on creating lesson specific learning objectives
(rather than course level learning objectives), helping students who are struggling with
accessibility, and designing assessments that provide evidence that students successfully
achieved the learning outcomes set out for a given course. The module will be divided into two
sections, identifying and assessing QMS 1-4 and QMS 5-8. By the end of the complete module,
learners will be able to:
a) Create a course that includes a solid introduction and overview
b) Identify the benefits of using the QMS rubric
c) Create learning objectives that are supported by assessment and measurement
d) Identify ways to provide learner support
e) Identify learner interactions and activities that will support student learning
f) Provide effective accessibility and usability through the course technology
Learner Profiles
Information about learners was gathered in four ways: live interviews with two CSUMB
faculty (Brian Robertson and Hawra Rabaan), interviewed using Zoom, https://www.zoom.us/);
Google Form-based survey of faculty (Brian Robertson, Hawra Rabaan, and Donald Fischer, see
Appendix B and C); a conversation with Dr. Miguel Lara, the SME on this project; and the
Learner Support Analysis section of the ADD document created by MIST CSUMB students in
the initial design of the Teaching Online Workshop (Avina et.al, 2014).
Both faculty that were interviewed in a live session were non-tenured faculty, have
earned a Masters degree, and have taught at the college level for no more than two years. They
were both currently going through the Online Teaching Workshop using the modules already in
place and taking the course in a self-paced, rather than cohort-based, manner.These two faculty
members represent one of two groups that are the target audience for the course (Avina et.al,
2014). The other group of learners are tenured faculty, over 40 years of age who have earned
Ph.D degrees and have taught at the college level for over 6 years. Learners in both groups are
highly motivated to learn because they have self-selected themselves for this course based on a
self-assessment of their current readiness to teach online. They are participating voluntarily and
spending their free time to improve their skills, with no additional compensation.
Both Mr. Robinson and Ms. Rabaan are participating in the Online Teaching Workshop
to help them prepare to teach in CSUMBs online Computer Science Program. This was the
primary target audience for the course at the current time, but this could be expanded in the
future to include any faculty on the CSUMB campus (Dr. Lara, conversation).
The faculty learners interviewed were both motivated to participate in the Online
Teaching Workshop due to their desire to teach online more effectively. Both had experience
taking courses online as students but had varied experiences with teaching online. They felt that
the 20 hours (total course time including modules other than Quality Matters) allotted for the
course was reasonable, however they did value flexibility in content delivery so they might move
through the course according to their personal schedule and learning needs. They did value their
time and although 20 hours in 2 weeks seemed reasonable, any longer time would be seen as a
burden unless it was spread out over a longer period. It was also noted that the exercises in the
current Quality Matters module would be more valuable if they were related to a practical
application, such as guiding the learner to create deliverables for courses that the faculty were
developing as instructors. The ability to receive feedback on that work, while valuable, may
prove to be time-consuming, may not fit in a 2 week 20 hour time frame or allow for flexible,
asynchronous learning. They were comfortable with working with other faculty collaboratively
and receiving feedback from others if this were a component of the course.
With respect to the learners comfort with technology, the development team found that
the learners that were interviewed in person and via the survey supported the profile of a learner
that is rather tech-savvy. All interviewees were familiar with using computers for routine
communications (email, forums, videoconferencing). Comfort with mobile use was slightly less
than usage of a computer to perform basic tasks. All learners had access to a computer and most
had a smartphone and/or tablet as well (see Appendix C for more detail). They will access the
Online Teaching Workshop using all of these devices, but will most likely use a PC or Apple
computer rather than mobile devices to access course materials.
With respect to the learners comfort and experience with online learning there were
some areas where the learners were less experienced and less comfortable than their experience
and comfort with technology. Based on the survey given, learners had experience with teaching
and had previously taken a course or workshop that provided guidance in teaching and learning,
however not all had taken a course that provided guidance in online teaching and learning. The
survey also showed that all learners were familiar with LMS systems as a student and as an
instructor. The learners were less comfortable with writing learning objectives, understanding the
difference between learning outcomes and course objectives, writing assessments and aligning
learning objectives and course assessments. The idea of active learning was something that not
all learners surveyed were familiar with (see Appendix C for more detail). During interviews
there was greater uncertainty about the teaching and learning aspect of online teaching than the
technical aspect.
Finally, the survey probed the learners familiarity with resources available for students
and faculty with respect to accessibility and technology support. Learners were not as familiar
with resources for teaching students with disabilities and were similarly unfamiliar with the
accessibility guidelines for developing online courses that are accessible to all students. The
learners knowledge of the technical resources for faculty was greater than their knowledge of
resources for students.
Overall the learners interviewed via live interview or online survey are knowledgeable of
the challenges of teaching online and are motivated to take the Online Teaching Workshop to
help them be more effective instructors. They have the requisite technological background and
have the tools to use computers and software effectively to perform the tasks required for setting
up and administering the course, as well as creating course materials. The only small exception
to this is their discomfort with being recorded, which is common as being on camera is awkward
and unfamiliar to most people.
The learners in the Online Teaching Workshop appear to have a greater knowledge of
technology than knowledge of effective instruction and student support. The areas where the
learners are less certain is in the areas of: support of students with disabilities; technical support
for students; and creation of course materials and resources (See Appendix C). In the creation of
course materials and resources the learners appear to be able to create digital course materials,
but lack guidance in the structuring of their course through the construction of learning
objectives, understanding the difference between learning outcomes and objectives and aligning
their learning objectives to assessment. The Quality Matters rubric provides guidance in these
areas and it would be helpful to include many examples and guidance in the writing of learning
objectives, the distinction between outcomes and objectives and the construction of assessments
that are aligned with standards.
Another area where learners require support is in their knowledge of accessibility and
technical resources and support for learners (see Appendix C). This is an area that can also be
addressed by the Quality Matters Rubric modules. Support for learners was a great concern
expressed by the instructors in the interviews. They felt it was important for student engagement,
motivation, performance and retention. They felt it was a challenge to motivate students online
due to the lack of personal interaction and they were looking for support in this area.
was the original intention the two faculty interviewed were working asynchronously so both
modalities are accommodated with the course design.
The entire Online Teaching Workshop is designed to take 20 hours with the QM module
taking 6 of those hours. Time is a valuable resource for faculty, however faculty will not be
monetarily compensated for this work and participation is entirely voluntary.
The initial analysis, design and development, and implementation of the QM learning
module will take place during the Spring Term B session of IST 626: Advanced Instructional
Design (MIST, CSUMB). Students in this course will be completing these tasks and passing the
completed modules to Dr. Lara for feedback and refinement. Dr. Lara will take the completed
projects and incorporate them into the current Online Teaching Workshop within iLearn. The
implementation and evaluation will be conducted by Dr. Lara and other faculty at CSUMB
following sufficient use of the module for instructor training. Students will not be monetarily
compensated for their work on this project, but they will get credit for graduation in the MIST
program. Faculty administering the module will receive their normal compensation. Additional
support for technology and accessibility are available through CSUMBs faculty resources and
will not require any specialized training of support staff.
Task/Work Analysis
Content and procedures
As stated previously the learners are a mix of masters and doctorate level
instructors/professors who are highly motivated to learn, tech-savvy and are familiar with LMS
systems. However the learners have a greater knowledge of technology than knowledge of
effective instruction and student support. The module will be divided into two sections,
identifying and assessing QMS 1-4 and QMS 5-8.
10
The learner will require access to a computer, the internet and a compatible browser to
complete and engage with the lesson. The module can be accessed at anytime and from any
location, however it would be beneficial to have a quiet learning environment to help the learner
focus and concentrate on the lesson. The module is an asynchronous lesson and does not require
group interaction or the learner to take the course on a specific day or time. However, the
learners are expected to complete the module within a two week / twenty hour timeframe.
The learners are not required to come to the workshop with prior knowledge of the course
subject matter. All instruction will cover relevant topics and the learners will be given progress
assessments throughout the module that require them to answer a series of questions prior to
advancing to the next section. Question types will include a combination of true or false,
matching, fill in the blank and multiple choice. The modules would culminate in an overall
assessment of understanding.
Goals and instructional objectives
The overall goal of the workshop is to teach faculty members how to utilize technology
to direct and support the online learning environment, as well as ensure they are familiar with the
requirements of Cal State Onlines Quality Online Learning and Teaching (QOLT) and the
Quality Matters (QM) rubrics, so they can apply them in their own online courses. The course
objectives will be achieved by mastery of the learning objectives listed below.
The modules will include instruction on the following areas:
Standard 1: Course overview and introduction. Upon completing this section,
the learner will understand the purpose and structure of an online course and be
able to answer questions on how to set up an organized course structure. The
11
learner will also establish course guidelines and clear expectations for online
students.
Standard 2: Learning objectives. The learner will develop outcomes that are
measurable and consistent with the course-level objectives. The learner will
understand the importance of and how to draw relationships between learning
objectives, competencies and course activities that are clearly stated and suited to
the level of the course.
Standard 4: Instructional materials. The learner will comprehend how and why
instructional materials contribute to the achievement of stated course and
module/unit learning objectives and competencies, and how the materials are to
be used for learning activities.
Standard 5: Course Activities and Learner Interaction. The faculty learner will
be able to produce curriculum that uses interactivity to engage the online student
and promotes the achievement of the stated learning objectives.
12
13
new concept is presented.. The two cumulative knowledge exams will be presented at the end of
each sections lessons (QMS 104 and QMS 5-8). hen they have finished reviewing the content
and completed all progress checks for each section, learners will be prompted that they may
proceed forward to the knowledge exam if they are ready.
The final knowledge exam for each section will consist of 10 questions - ten relating to
Quality Matters Standards 1 - 4 and ten questions relating to Standards 5 - 8. Content will be
graded and reported to the corresponding Learning Management System. In order to receive a
passing score learners must achieve a minimum score of 80% on each. The following are
examples of format and feedback to be provided.
True/False
Multiple Choice
Matching
Fill in the blank
The final component in the testing plan for the Quality Matters instructional material will
be developed by the administrators of the Online Teaching Workshop, as it is outside the scope
of this development project. As part of the Learning Management System (LMS) on which the
QMS modules will be hosted, the administrators of the workshop will include feedback forums
and practical exercises related to the content delivered.
Organizational, Delivery, and Management Strategies
Content will be delivered via eLearning module that will be accessed within a Learning
Management System. The content for Quality Matters Standards 1 through 8 will be developed
using Adobe Captivate, packaged for SCORM delivery, and able to be viewed in HTML/Flash.
Content will be organized within the eLearning module according to learning-related sequencing
themes that builds on learners skills and knowledge. The Quality Matters Standards themselves
14
will be the guiding topic headers. In addition to sequencing content in relation to Quality Matters
Standards, the content developed for this training will consist of the following segments to
ensure learner orientation.
Navigational Guidance. The first slide will contain direction and information on how to
proceed through the module content. The guidance will be presented both as part of the narration
and onscreen prompt text.
Module Introduction. The introduction will contain the topics below.
Gain Attention: Primes the learner's mind to be open, receptive and inquisitive about new
information.
Overview: Whats in it for me?
Learning Objectives: What will I be learning?
Method/Media/GUI Overview: How will I be learning the new information?
Evaluation: How will I be evaluated?
Transition: Focuses the learner back to the topic for presentation.
Presentation of Content. This section will break down the topics that make up the
Quality Matters Standards content. Information will be chunked according to each of the
standards.
prior to being tested. Techniques will include opportunities to review Quality Matters Standards,
15
identify whether standards have been achieved or not, and how to ensure standards have been
implemented. Progress checks will not be graded.
While the progress checks will be in quiz form rather than in an activity format, feedback
given will contribute to the learners understanding through the use of concept repetition. In the
case of an incorrect response, an explanation why the given response is inaccurate and a
description of the desired response will be given to reinforce the concept being tested.
Knowledge Exam. This segment will aim to test the performance of the student through
evaluation. This will be administered after the learner has reviewed all content, completed all
progress checks, and is ready to move forward to evaluation. As a cumulative test, the final
knowledge exam will contain questions previously experienced by the user during the progress
tests as well as new questions that address higher level understanding.
Instruction
All aspects of course design, development and implementation will adhere to the
following proven strategies.
Storyboard that will detail the content and function of all elements, pages and media, as well as
all identified references pertaining to Quality Matters Standards. Content will be delivered via
web based eLearning module. This media will provide a Graphic User Interface, navigation,
16
external resources, additional media elements where appropriate, and present content to the
learners in a linear fashion.
Development
Development of the Captivate lessons has been split into two teams addressing four
standards each (1-4 and 5-8). The teams have completed two eLearning programs that will be
used as part of the Quality Matters module in the Online Teaching Workshop. The two
Captivate lessons will be developed over an eight week period. Teams will provide a storyboard
with narration included for the client. A prototype will be provided to the SME to ensure the
design complements previous lessons created. After approval, the team will produce content,
interactions, and narration for the lessons. Usability testing will take place after completion of
the project and changes made to the lessons based on feedback from testing.
Implementation
The Captivate lessons will be embedded by the SME into the Online Teaching Workshop
for the learners to access. The intended date of access is September 1st, 2015. This module will
be part of the overall workshop that professors must take to convert their teaching methods from
face to face, to an online only class. Learners will access the course through iLearn and
complete modules based on the current format of the workshop. The workshop is designed to
help professors understand the best practices in converting content from face-to-face to online.
VI. Assessment
Summative evaluation
The summative evaluation is composed of two separate instances of evaluation. The first
consists of multiple-choice learner self-checks at the end of each instructional unit. These self-
17
checks are intended to keep the learner engaged with the instruction as well as to offer corrective
feedback if necessary. When the learner chooses the correct answer, the feedback will state that
the learner selected the correct answer, state why, and allow the learner to continue to the next
question or to the rest of the learning module if that was the last question. When the learner
chooses the incorrect answer, the feedback provided will inform the learner that their selection
was not correct, state why, and then ask the learner to choose again. This will repeat until the
learner chooses the correct answer.
The second evaluation will be a learning evaluation assessment which will consist of a
pre-test and a post-test, administered as part of the usability testing interviews. Both the pre- and
post-test assessments will entail answering the same set of questions. This will allow for an
evaluation of the learning effectiveness of the instructional module. Although a statistical
analysis of the testing results will be out of the scope of work for this project, we recommend
one be conducted at a later date by the administrators of the Online Teaching Workshop. A
statistical evaluation should be conducted comparing the pretest and posttest in order to assess
the statistical significance of the learning outcome. The pretest and posttest will be conducted
using Google Forms (Appendix D). This will allow consistent administration, collection of data,
and evaluation both with the present set of test subjects and additional subjects moving forward.
18
statistical significance of the learning. This section describes the protocols we recommend be
followed in order to ensure accurate results.
Prior to completing the module for Quality Matters 1-4 and 5-8 the learner will be asked
to take the pretest (Appendix D). Immediately following the pre-test the learner will participate
in the designated module. After the completion of the module the learner will be asked to take
the posttest. After completing the posttest the learner will be asked to fill out the usability form.
The information will be recorded and evaluated in order to better the instructional module. No
personally identifiable information will be collected with any assessment.
All assessments and instructions for the assessments will be accessed and delivered
through Google forms. A link to each of the specific assessments will be available within the
CSUMB iLearn system where the modules will be hosted.
Formative/usability evaluation
Formative assessments will progress in tandem with the design of the instruction and then
19
The final evaluation will be conducted as an assessment of the instructional design in its
entirety using Google Forms. This evaluation will be completed by participants who will be
directly involved with the future use of the instructional product. The feedback obtained from
these users will be used to correct and fine-tune the instruction, thus ensuring an error-free user
experience. These assessors, who are also future users, will also be asked to provide feedback
regarding the match between the stated learning objectives within the module and whether the
evaluators felt that those objectives were met. In addition to assessing instructional goals and
usability, questions regarding the principles of multimedia will also be evaluated (Appendix F).
Usability Summary
Usability testing was conducted on a preliminary version of both the QMS 1-4 module
and QMS 5-8 module. Because the number of participating users was limited to three, the
feedback received is helpful but not conclusive. Additional usability testing is recommended
with a larger test pool of participants.
Of the three subjects that participated in the preliminary testing, all three were college
educated males, ranging from early forties to late sixties in age. Subjects #1 and #2 come from a
professional education background, working as instructors at the college level. As professional
educators, Subjects #1 and #2 have had prior experience with the QMS Standards and were
familiar with the instructional content prior to reviewing the modules. Subject #3 works in the
high tech industry and has no background in professional education.
Testing Protocols
Each subject was given access to and completed a Pre-test using a Google Docs form
prior to experiencing the instructional modules. Subjects then went through the instructional
modules and completed a Post-test for each afterwards. The Pre-test and Post-test contained the
20
same questions in order to measure the impact of the instruction. A Usability Survey was
completed by each user after finishing the instruction and tests in order to capture direct
feedback on the user interface of the instructional modules.
Subjects #1 & 2 were observed remotely via a video conference. The subjects screens
were shared with the administrator in order to enable observation of mouse movements. Audio
and video of each of the subjects was also captured, and the subjects were encouraged to verbally
communicate their thought process as they moved through the instructional modules. Both
Subject #1 and Subject #2 went through the usability process in a quiet home office environment
familiar to them. Subject #3 was observed in person, also in a quiet home office environment.
Test Execution
Subjects #1 & 2 completed the Pre-test for QMS Module 1-4 and proceeded directly to
the Pre-test for QMS Module 5-8. Both subjects then proceeded to undertake the instructional
modules, again in sequence - QMS 1-4 followed directly by QMS 5-8. Upon completing the
instruction, the subjects completed both Post-tests and Usability Surveys, again in sequence.
Due to the procedure followed by Subjects #1 & 2, feedback was combined on the
Usability Surveys for the different modules. For example, a feature not included in QMS 1-4 was
commented on in the QMS 1-4 Usability Survey, instead of on the survey for QMS 5-8, where
the feature was included. For this reason, the testing procedure was altered slightly for Subject
#3, the last to go through the process. The intention was to isolate feedback to the specific
module under review.
Subject #3 completed all steps for QMS 1-4 in sequence (Pretest, instructional module,
Posttest and Usability Survey) before taking a break and repeating the sequence for QMS 5-8.
QMS 1-4 Results
21
Methodology. By assigning a value of 1 for each correct answer, the Pretest results
were compared with the Postest results as well as between each subject. A full statistical analysis
was not conducted due to the small data pool available.
QMS 1-4 results. Subject #1 and Subject #2 scored 85.71% on the Pretest (12 correct out
of 14 possible). Subject #3 scored 8 of 14, or 57.14%. After completing the instruction and
taking the posttest, Subject #1 increased his score by 7.14%, to 92.86%. Subject #2 maintained
the same score of 85.71%. Subject #3, the user with no professional educational experience and
therefore no prior exposure to the content, increased his score by 35.71%. This would indicate
that a user with no prior experience with the Quality Matters material would experience a
significant benefit from completing the QMS 1-4 module. Please see Charts 1 and 2 for an
illustration of results.
22
Verbal feedback received from Subject #2 during the instructional module quizzes and
the Pre- and Post-tests indicates the Subject was aware of the correct answer based on the QMS
Standards, but disagreed with them. Answers were chosen based on personal belief rather than
content presented. At the time the usability tests were conducted, the feedback portion of the
quizzes had not been implemented. It is the opinion of this researcher that the incorporation of
informative feedback to quiz questions included in the module will increase the success rate of
the learners.
23
Methodology. By assigning a value of 1 for each correct answer, the Pre-test results
were compared with the Post-test results as well as between each subject. A full statistical
analysis was not conducted due to the small data pool available.
QMS 5-8 results. Subject #1 and Subject #2 scored 85.71% on the Pre-test (6 correct out
of 7 possible). Subject #3 scored 3 out of 7, or 43%. After completing the instruction and taking
the Post-test, Subject #1 increased his score by 0%, maintaining the 85.71% score. Subject #2
increased their score by 14.29% to a score of 100%. Subject #3, the user with no professional
educational experience and therefore no prior exposure to the content, increased his score by
57% to achieve all correct answers. This would indicate that a user with no prior experience with
the Quality Matters material would experience a significant benefit from completing the QMS 58 module. See Charts 3 and 4 below for an illustration of results.
24
25
Feedback received in the Usability Survey indicated that users found navigation of the
courses could be improved, and again referenced inconsistencies in interactions as part of the
confusion. Subjects also expressed that the automatic advance of slides was not preferable, as
they would rather have control and spend as much time as they need to review content. Overall,
users rated the instruction well, indicated that their attention was held, and that the pace was
acceptable. The results of the usability survey demonstrate the overall effectiveness of both the
QMS 1-4 and 5-8 learning modules, but also the areas that should be improved to better
accommodate the learner.
Conclusion
Learners from both QMS modules were able to access and improve their knowledge after
completing the content. Although the pool of subjects was too small to provide conclusive
evidence, the given data demonstrates that learners had a generally positive opinion about the
modules and the content. Feedback collected during usability and testing will be used and
implemented into course content, especially those referencing consistency, quiz feedback, and
navigability. The increase between Pre- and Post-test results indicate that the learners
demonstrated and increase in knowledge and that the content was beneficial in completing a final
QMS summative evaluation.
VII.
26
Appendices
Appendix A
General Learner Questions (in-person questions)
1. As an introduction, please describe your academic background and career path? What
path led you to this interview?
2. Have you ever taken a course online?
a. What did you like about it? Dislike?
3. Are you planning to convert one of your classes to an online format?
a. Next fall? Or next spring?
b. What is the topic of the class you plan to convert?
c. What motivated you to teach an online class?
4. Do you feel you currently have the technical skills to teach an online class?
a. What areas do you feel you may need more training?
b. Do you feel you need to upgrade your computer equipment to participate in an
online course or program of study?
27
Appendix B
Learner Survey (taken from Google Forms, http://goo.gl/forms/VGvbdCveE1)
28
29
30
31
32
33
Appendix C
Summary of Responses to Google Form
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
Appendix D
Pre and posttest
53
54
Appendix E
Usability survey
55
56
57
58
References
Avina, C. Bettencourt, J., Bradley, A., Osterweil, A. Rincon, A., and Simmons, J. (2014)Ist 626
Abbreviated Design Document CSUMB Online Tutorials. California, United States. California State
University Monterey Bay.