Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

Expertise, Consumer-Oriented, and Program-Oriented Evaluation Approaches

•Download as PPTX, PDF•
•

All information referenced from: Fitzpatrick, J., Sanders, J., & Worthen, B. (2011). Program evaluation: Alternative approaches and practical guidelines (4th ed.). Upper Saddle River, N.J.: Pearson Education.

1 of 20
Expertise, Consumer-Oriented, and Program-Oriented Evaluation Approaches John H. Curry, Ph.D.
Expertise-Oriented Approach •The oldest type of formal evaluation •Relies on professional expertise to judge the quality of an institution, program, product, or activity.
Types of Expertise-Oriented Evaluations •Formal review system •Informal review system •Ad hoc panel review •Ad hoc individual review
Formal Review Systems •Mostly used in accreditation •Examine existing structure of organization •Examine published standards •Follow a specified schedule •Increasing emphasis on outcomes •Use the opinions of multiple experts •Status of those being evaluated is affected by the results
Informal Review Systems •Used primarily for evaluations that lack published standards or follow a specified review schedule. •Use multiple reviewers •Status of those being reviewed is affected by results •Examples: Peer review of articles, theses or dissertation committees
Ad Hoc Panel Reviews •Occur at irregular intervals when circumstances demand •Reviews not related to institutionalized evaluation or standards •Usually one-shot evaluations prompted by a particular, time-bound need for evaluative information
Examples of Ad Hoc Panels •Panels to develop standards •Funding agency review panels •Blue ribbon panels
Ad Hoc Individual Reviews •Review of any entity by an individual selected for his/her expertise •Usually to judge value or make recommendations •Example: Employment of a consultant to review an educational, social, or commercial program
Consumer-Oriented Evaluation •Much like expertise-oriented •Helps inform decisions on what to purchase or trade •Judge the quality of something, establish value, merit or worth of a product •Audience is broader: The purchasing public, and they are not known to the evaluator.
Consumer-Oriented, cont. •Developed by Scriven •Much like needs assessments •Functional analysis of product •IMPORTANT to: •IDENTIFY the criteria correctly •DEVELOP standards to judge those criteria •COLLECT data •SYNTHESIZE information to make a final judgment
Sample Consumer-Oriented Criteria •Need •Market •Performance (Field Trials) •Performance (Consumer) •Performance (Critical Comparisons) •Performance (Long-Term) •Performance (Side Effects) •Performance (Process) •Performance (Causation) •Performance (Statistical Significance) •Performance (Educational Significance) •Cost-effectiveness •Extended Support
Consumer-Oriented Examples •Consumer Reports •US News and World Report
Program-Oriented Approaches •Focus on learning some key features of the program, and then serve to help the evaluator decide which questions should be asked. •Most common type: The objectives- oriented evaluation
Objectives-Oriented Evaluation •Distinguishing feature: some activity is specified, and then the evaluation tries to determine the extent to which those objectives are achieved.
Tylerian Evaluation •Steps: •Establish broad goals or objectives •Classify the goals or objectives •Define the objectives in behavioral terms •Find situations in which achievement of objectives can be shown •Develop or select measurement techniques •Collect performance data •Compare performance data with behaviorally stated objectives •Generally discredited today as oversimplifying a complex system
Provus Discrepancy Evaluation •Steps: •Agree on standards (objectives) •Determine whether a discrepancy exists between performance and standards •Using that information, decide whether to improve, maintain, or terminate the program or some aspect of it
Theory-driven evaluation •Program theory is “the construction of a plausible and sensible model for how a program is supposed to work.” •Programs fail to achieve goals for two different reasons: •The program isn’t delivered as planned and, therefore, isn’t ready to be tested (implementation failure) •The program is delivered as planned, and the results indicate that the program theory was incorrect (theory failure)
Theory-driven evaluation, cont. •Steps: •Engage relevant stakeholders •Develop a first draft of program theory •Present a draft to stake holders for discussion, reaction, and input •Conduct a plausibility check •Communicate findings to key stakeholders •Probe arrows for model specificity •Finalize program impact theory
Goal-Free Evaluation •The evaluator purposely avoids becoming aware of the program goals •Predetermined goals are not permitted to narrow the focus of the evaluation study •Goal-free evaluation focuses on the actual outcomes rather than intended program outcomes •The goal-free evaluator has minimal contact with the program manager and staff •Goal-free evaluation increases the likelihood that unanticipated side effects will be noted
Reference •Fitzpatrick, J., Sanders, J., & Worthen, B. (2011). Program evaluation: Alternative approaches and practical guidelines (4th ed.). Upper Saddle River, N.J.: Pearson Education.

More Related Content

Expertise, Consumer-Oriented, and Program-Oriented Evaluation Approaches

  • 2. Expertise-Oriented Approach •The oldest type of formal evaluation •Relies on professional expertise to judge the quality of an institution, program, product, or activity.
  • 3. Types of Expertise-Oriented Evaluations •Formal review system •Informal review system •Ad hoc panel review •Ad hoc individual review
  • 4. Formal Review Systems •Mostly used in accreditation •Examine existing structure of organization •Examine published standards •Follow a specified schedule •Increasing emphasis on outcomes •Use the opinions of multiple experts •Status of those being evaluated is affected by the results
  • 5. Informal Review Systems •Used primarily for evaluations that lack published standards or follow a specified review schedule. •Use multiple reviewers •Status of those being reviewed is affected by results •Examples: Peer review of articles, theses or dissertation committees
  • 6. Ad Hoc Panel Reviews •Occur at irregular intervals when circumstances demand •Reviews not related to institutionalized evaluation or standards •Usually one-shot evaluations prompted by a particular, time-bound need for evaluative information
  • 7. Examples of Ad Hoc Panels •Panels to develop standards •Funding agency review panels •Blue ribbon panels
  • 8. Ad Hoc Individual Reviews •Review of any entity by an individual selected for his/her expertise •Usually to judge value or make recommendations •Example: Employment of a consultant to review an educational, social, or commercial program
  • 9. Consumer-Oriented Evaluation •Much like expertise-oriented •Helps inform decisions on what to purchase or trade •Judge the quality of something, establish value, merit or worth of a product •Audience is broader: The purchasing public, and they are not known to the evaluator.
  • 10. Consumer-Oriented, cont. •Developed by Scriven •Much like needs assessments •Functional analysis of product •IMPORTANT to: •IDENTIFY the criteria correctly •DEVELOP standards to judge those criteria •COLLECT data •SYNTHESIZE information to make a final judgment
  • 11. Sample Consumer-Oriented Criteria •Need •Market •Performance (Field Trials) •Performance (Consumer) •Performance (Critical Comparisons) •Performance (Long-Term) •Performance (Side Effects) •Performance (Process) •Performance (Causation) •Performance (Statistical Significance) •Performance (Educational Significance) •Cost-effectiveness •Extended Support
  • 13. Program-Oriented Approaches •Focus on learning some key features of the program, and then serve to help the evaluator decide which questions should be asked. •Most common type: The objectives- oriented evaluation
  • 14. Objectives-Oriented Evaluation •Distinguishing feature: some activity is specified, and then the evaluation tries to determine the extent to which those objectives are achieved.
  • 15. Tylerian Evaluation •Steps: •Establish broad goals or objectives •Classify the goals or objectives •Define the objectives in behavioral terms •Find situations in which achievement of objectives can be shown •Develop or select measurement techniques •Collect performance data •Compare performance data with behaviorally stated objectives •Generally discredited today as oversimplifying a complex system
  • 16. Provus Discrepancy Evaluation •Steps: •Agree on standards (objectives) •Determine whether a discrepancy exists between performance and standards •Using that information, decide whether to improve, maintain, or terminate the program or some aspect of it
  • 17. Theory-driven evaluation •Program theory is “the construction of a plausible and sensible model for how a program is supposed to work.” •Programs fail to achieve goals for two different reasons: •The program isn’t delivered as planned and, therefore, isn’t ready to be tested (implementation failure) •The program is delivered as planned, and the results indicate that the program theory was incorrect (theory failure)
  • 18. Theory-driven evaluation, cont. •Steps: •Engage relevant stakeholders •Develop a first draft of program theory •Present a draft to stake holders for discussion, reaction, and input •Conduct a plausibility check •Communicate findings to key stakeholders •Probe arrows for model specificity •Finalize program impact theory
  • 19. Goal-Free Evaluation •The evaluator purposely avoids becoming aware of the program goals •Predetermined goals are not permitted to narrow the focus of the evaluation study •Goal-free evaluation focuses on the actual outcomes rather than intended program outcomes •The goal-free evaluator has minimal contact with the program manager and staff •Goal-free evaluation increases the likelihood that unanticipated side effects will be noted
  • 20. Reference •Fitzpatrick, J., Sanders, J., & Worthen, B. (2011). Program evaluation: Alternative approaches and practical guidelines (4th ed.). Upper Saddle River, N.J.: Pearson Education.