REPUBLIC OF THE PHILIPPINES DEPARTMENT OF NATIONAL DEFENSE. DEC 02 Bi OFFICE ORDER No. J&!_, s. 2017 TO ALL OCD EMPLOYEES SUBJECT Guidelines on the Implementation of the Four (4) Levels of Evaluation of Training Courses 1. RATIONALE The Office of Civil Defense (OCD), pursuant to its mandate in Disaster Risk Reduction and Management (DRRM) as stipulated in Republic Act (RA) 10121, has been conducting DRRM training and other capacity-building activities for various stakeholders. Using the train-the-trainers approach, the OCD was able to create a pool of DRRM subject matter experts to help strengthen the capacities of government agencies, civil society organizations, private sector groups, and communities. As new approaches and standards in DRRM emerge, the OCD continues to explore innovations to improve its training methodologies To effectively deliver quality DRRM training, there is a need to conduct comprehensive evaluation of the training courses specifically in terms of the reaction obtained by the trainees, the leaming acquired, the resultant change in behavior, and the corresponding long-term impacts of the training to the ‘communities. Such parameters denote the four (4) levels of training evaluation To prescribe the guidelines on the conduct of four (4) levels of evaluation, this Office Order is hereby prepared. 2. PURPOSE The purpose of this Office Order is to prescribe guidelines on the implementation of the four (4) levels of evaluation on the training courses of the OCD, 3. GOAL The goal of this Office Order is to enhance the delivery of training courses of the OCD based on the results of the four (4) levels of evaluation. “Telefax: OCD Capacty-Bulling and Training Service (+652) 912-4830, (+632) 421-1928 “fee ofthe Adminstrator, OCD (+692) 012.2424 Email cbts@ocd gow ph ceddrrmetraning@gmal.com Website: ww. cod gov ph4. OBJECTIVE The objectives of this Office Order are as follows: 4.1 To provide the general guidelines that must be observed in the implementation of four (4) levels of evaluation of training courses of the OCD; and 4.2 To prescribe the steps, conditions, and considerations for administering the four (4) levels of evaluation. 5. SCOPE This Office Order shall cover all training courses conducted by the OCD Central Office and Regional Offices for internal and external stakeholders. 6. DEFINITION OF TERMS 6.1 6.2. 63 6.4, 6.5, 6.6. 67. After Training Activity Report (ATAR) refers to the summary of the pertinent details on the conduct of a training or capacity-building activity, including the background, objectives, highlights of proceedings, outputs, and other relevant information. Civil Defense Education and Training Program (CDETP) refers to broad category of training program offered by the OCD that aims to equip the OCD employees and other civil defense practitioners with knowledge, skills, attitude, and other competencies for more effective and efficient delivery of tasks and responsibility given to them under the RA 10121 Evaluation refers to the process that employs the use of both qualitative and quantitative methods including case studies, surveys and research statistical analysis and mode! building, among others. Evaluation Tool refers to the instrument used to collect data for subsequent analysis and interpretation. Evaluation tools are used for surveys, tests, interviews, focused-group observations and performance records, among others. Impact Level refers to the highest level of evaluation that focuses on assessing the long-term effects of training on the organization, community or society. Impact level is usually measured within three (3) to five (5) years after the conduct of the training Learning Level refers to the second level of evaluation that focuses on measuring knowledge, attitude and/or skills gained by the trainees. Learning level is usually measured before and after the training. Module (or Session or Unit) refers to a standard segment or part of a Training Course6.8. 6.9. 6.10. 6.11 6.12, 6.13. 6.14, 6.15. 6.16. Module Evaluation Form refers to the evaluation tool accomplished by the trainees every after training module, It is used to assess the delivery of the said module and the performance of the instructor. National Disaster Risk Reduction and Management Education and Training Program (NDRRMETP) refers to the broad category of training program that aims to heighten the knowledge in DRRM and strengthen the capacities of OCD's extemal stakeholders involved in DRRM Outcome Level refers to the third level of evaluation that focuses on measuring the extent to which trainees applied the learning in their workplace or organization. Outcome level is usually measured within three (3) to six (6) months after the conduct of the training, Pre-Test refers to the evaluation tool administered to measure the participant's baseline knowledge or preparedness for the training, Post-Test refers to the evaluation tool administered to students after the completion of the training to measure the extent of knowledge gained. Reaction Level refers to the first level of evaluation that focuses on measuring the personal feedback and learning experiences of the trainees. Learning level is usually measured during and at the end of the training. Training Course refers to the specific training and capacity-building activities that form part of the NDRRMETP and CDETP. Training Evaluation Form refers to the evaluation tool to be accomplished by the trainees after completing all the training modules. It is used to assess the overall administrative and substantive proceedings of the training. Training Team refers to a group that is responsible for the overall conduct of the training. It consists of course monitor, trainers, facilitators, and secretariat, among others. POLICY STATEMENT This Office Order has been prepared to emphasize the significance of using evaluation methods in enhancing the implementation of training courses under the National Disaster Risk Reduction and Management Education and Training Program (NDRRMETP) and Civil Defense Education and Training Program (CDETP). Hence, the evaluation methods articulated herein are expected to promote innovation in and revamping of training courses as may be deemed necessary.8. GUIDELINES 8.1 7.2. General Guidelines TAA The OCD Capacity-Building and Training Service (CBTS), in collaboration with all concemed services, offices, divisions, sections and units, shall be responsible for the planning, formulation, development, implementation, and enhancement of evaluation tools applicable for all training courses. . The tools for the four (4) levels of evaluation shall be subjected to appropriate consultation, review, and testing prior to full implementation. . The results of the four (4) levels of evaluation shall be duly reported to the OCD Central Office through an After Training Activity Report (ATAR) for consolidation of the CBTS. The CBTS shall utilize the consolidated findings of the four (4) levels of evaluation for the continued enhancement of the training courses of the OCD. Specific Guidelines 721. Level 1: Reaction 7.2.1.1. The Module Evaluation Form shall be administered by the training team at the end of every module (Annex A: Module Evaluation Form). 7.2.1.2. The overall rating of the module shall be processed and analyzed by looking at the frequencies obtained and their corresponding percentages. 7.2.1.3. The Training Evaluation Form shall be administered by the training team at the end of all the training modules (Annex B: Training Evaluation Form). 7.2.1.4. The overall rating of the training shall be processed and analyzed by looking at the frequencies obtained and their corresponding percentages. 7.2.1.5. The results of the evaluation of the modules and the training shall be consolidated and interpreted by the training team. Recommendations shall be formulated out of the results for inclusion in the ATAR7.2.2 Level 2: Learning 7.23 7.2.24 7.2.2.2 7.2.2.3 7.2.2.4 7225 7.2.2.6 7.2.2.7 All training courses of the OCD shall have mandatory pre-tests and post-tests. The standard questions and answers for the pre-tests and post-tests shall depend on the type of training. The pre-test shall be administered by the training ‘team before the start of the first module. The results of the pre-test shall be processed and analyzed using the following Measures of Central Tendency: Measures | Description _ ] Mean Average value of all the scores ___| obtained for the pre-test _ Mode ‘Soore that was obtained the most the pre-test Median | Middle most value of all the scores _ obtained for the pre-test The post-test shall be administered by the training team after the last module. The results of the post-test shall be processed and analyzed using the following Measures of Central Tendency: Measures ‘Mean [Average value of all the scores | obtained for the post-test Mode Score that was obtained the most for I the post-test Median | meade, most value of all the scores | obtained for the post-test The results of the pre-test and post-test shall be compared for appropriate analysis and interpretation by the training team. A recommended action shall be formulated out of the results. Level 3: Outcome 7.2.3.4 The standard evaluation tool and methodologies for measuring the resultant change in behavior shall be unique for every training course.7.2.3.2 After the conduct of the training course, the resultant change in behavior of the trainees shall be observed and documented by the training team within three (3) to six (6) months. 7.2.3.3 The observed changes in the behavior of the trainees shall be subjected to appropriate analysis and interpretation by the training team. Recommendations for the training shall be formulated out of the evaluation results. 7.2.3.4 Specific guidelines and protocols on the conduct of the Level 3 Evaluation shall be developed by the CBTS, in collaboration with all concerned OCD services, offices, divisions, sections, and units. 7.2.4 Level 4: Impact 7.2.4.1 The standard evaluation tool and methodologies for measuring the impacts of the training to the organization or community shall be unique for every training course of the OCD. 7.2.4.2 After the conduct of the training, the impacts of the training to the organization or community shall be observed and documented by the training team within three (3) to five (5) years 7.24.3 The observed impacts of the training to the organization or community shall be subjected to appropriate analysis and interpretation by the training team. Recommendations for the training shall be formulated out of the evaluation results 7.244 Specific guidelines and protocols on the conduct of the Level 4 Evaluation shall be developed by the CBTS, in collaboration with all concerned OCD services, offices, divisions, sections, and units. 9. SEPARABILITY CLAUSE In the event that any provision or part of this Office Order is deemed invalid by a court of law or competent authority, the remaining provisions hereof which are not affected thereby shall continue to be in full force and effect. 10. REPEALING CLAUSE All existing issuances inconsistent herewith are hereby repealed or modified accordingly11, EFFECTIVITY This Office Order shall take effect immediately upon approval of the signatory.ANNEX B: TRAINING EVALUATION FORM. Trainee's Name Name of Training Course ‘Agency/Organization Date(s) Venue On a scale of 1 to 5, pl 1 ] 3 3 a 3 Poor | __ Average Good Very Good Excellent 7 z 3 a 5 REMARKS | ‘Objectives 1 t Materials and Visual Aids ‘Methodologies: Topics Resource Persons ‘Support Staff 1 Coxtrainees Time Management [Venue ‘Overall Training Course Rating | L L What did you like the MOST about the training course? What did you like the LEAST about the training course? What are your recommendations for improving the training course? jank you for your partclpation?ANNEX A: MODULE EVALUATION FORM Trainee's Name Name of Training Course ‘Agency/Organization Date(s) A Venue Please fill-in the required information. [Name of TraineriResource Person [Module Number [Module Name On a scale of 1 to 5, please check the box that best reflects your evaluation. ‘Waterials and Visual Ads 1 2 3 4 5 ] Poor Average Good | Very Good Excellent c =a a a 5 [REMARKS Contents T Methodologies | Level of ineraction Wastery of Topic Time Allocation ‘Overall Instructor Rating ‘Overall Module Rating T Other comments/observations: Thank you for your participation! @