Skip to main content

Thomas Archibald

Virginia Tech, Agricultural, Leadership, and Community Education, Faculty Member

Followers

151

Following

80

Co-authors

8

Public Views

I am an associate professor and extension specialist in the Agricultural, Leadership, and Community Education department at Virginia Tech. My work is primarily focused on program evaluation, with a specific focus on evaluation capacity building (ECB) within the Cooperative Extension system. I am also the Director of the Feed the Future Senegal Jeunesse en Agriculture (Youth in Agriculture) project, a 5-year, $4 million project supported by USAID/Senegal. I am a recipient of the 2017 Maria Guttentag Promising New Evaluator award of the American Evaluation Association. I serve on the Board of Directors of the Eastern Evaluation Research Society and on the Editorial Board of the American Journal of Evaluation and New Directions for Evaluation.

I received my PhD in Adult and Extension Education from Cornell University. Working with Arthur (Butch) Wilson, William Trochim, and Stacey Langwick, my dissertation research was a qualitative sociological study of three cases which involved efforts to connect research and practice in non-formal educational contexts. Specifically, I focused on the epistemological politics inherent in the evidence-based program and evidence-based practice movements in non-formal education.

During my graduate studies, I was also a Graduate Research Assistant with the Cornell Office for Research on Evaluation under the direction of William Trochim. There, I worked with a dynamic team on a multi-year, multi-site research project in which we developed, implemented, and conducted research on a systems thinking-influenced approach to strengthening the capacity of non-formal educators to meaningfully evaluate their own education and outreach programs.
Supervisors: Arthur (Butch) Wilson , William Trochim, and Stacey Langwick

less

InterestsView All (14)

Uploads

Papers by Thomas Archibald

Community Organizing in/as Adult Education

Community organizing and adult education overlap in a number of ways, yet other than some notable... more Community organizing and adult education overlap in a number of ways, yet other than some notable exceptions, those overlaps remain inadequately explicated and insufficiently represented in contemporary adult education literature. The purpose of this paper is to explore the conceptual and practical overlaps of adult education and community organizing, in an effort to elevate an adult education praxis that promotes participation, equity, and justice.

Consumer Response to Mechanically Tenderized Beef (MTB) and MTB Labels: An Exploratory Focus Group Study

Journal of Food Protection

In May 2016, labeling of certain nonintact mechanically tenderized beef (MTB) products was mandat... more In May 2016, labeling of certain nonintact mechanically tenderized beef (MTB) products was mandated in the United States. MTB products should be handled differently by the consumer because pathogens can be transferred from the exterior to the interior of the meat during the tenderization process. Without labeling, it is difficult to visually distinguish between some intact beef and MTB products, which is a concern because MTB products require higher internal cooking temperatures for safety. An exploratory study was conducted to understand consumer understanding of MTB products and consumer responses to the new label. Thirteen focus groups were convened in rural and urban settings across Virginia and North Carolina between December 2015 and May 2016. Sessions were audiorecorded, transcribed verbatim, and analyzed through constant-comparison thematic analysis. Although MTB products were commonly bought, prepared, and consumed, consumer awareness of MTB products and the MTB process was...

The Critically Reflective Evaluator: Adult Education's Contributions to Evaluation for Social Justice

New Directions for Evaluation

International Positive Youth Development: Challenges and Opportunities in Policy and Practice

Journal of Youth Development

This special issue of the Journal of Youth Development on international positive youth developmen... more This special issue of the Journal of Youth Development on international positive youth development brings together diverse perspectives of research, practice, and policy regarding the challenges and opportunities facing international efforts to promote the positive development of youth. To represent a plurality of perspectives on international PYD, the authors sought to represent a variety of continents, countries, genders, races, ethnicities, sectors, and levels of experience with publishing. There is also variety in the types of articles such as case studies, presentations of frameworks and curricula, letter-style commentary by youth, and cutting-edge research methodologies and studies. A few key themes frame the volume, including (a) diversity in definitions of youth and operationalizations of PYD; (b) the multifaceted and multidimensional ways in which PYD is integrated into different sectors, domains, and types of programmatic and organizational contexts; and (c) the importance...

Positive Youth Development in Senegal: A Case Study of 4-H Senegal

by Thomas Archibald, Kathleen Jamison, and Fatou Diouf

Journal of Youth Development , 2021

African youth face great challenges and great opportunities. Positive youth development (PYD) pro... more African youth face great challenges and great opportunities. Positive youth development (PYD) programs have the potential to support youth in Africa to reach their full potential. Yet PYD approaches must be tailored if they are to be contextually and culturally responsive and relevant. The purpose of this paper is to provide an evidence-informed description of one such program, the 4-H Senegal PYD program. We first provide a conceptual background for the program, informed especially by the literature on global and cross-cultural PYD. We then share empirical results of a qualitative case study focused on the establishment, scale-up, and institutionalization of the 4-H Senegal program. Our case study results elucidate culturally responsive PYD program adaptations that corroborate conceptual framings of cross

Refining and Measuring the Construct of Evaluative Thinking: An Exploratory Factor Analysis of the Evaluative Thinking Inventory Journal of MultiDisciplinary Evaluation

by Jane Buckley and Thomas Archibald

Journal of MultiDisciplinary Evaluation, 2020

Background: Evaluative thinking has emerged as a key construct in evaluation, especially for eval... more Background: Evaluative thinking has emerged as a key construct in evaluation, especially for evaluation practitioners and researchers interested in evaluation capacity building (ECB). Yet, despite increasing calls for more research on evaluation and, more specifically, for more research on ECB, until recently little empirical inquiry on the dimensions of evaluative thinking has been conducted.

Purpose: To address that lack, the purpose of the study presented in this paper is to refine the construct of evaluative thinking by exploring its underlying dimensions and to ascertain the internal consistency of an instrument developed to measure evaluative thinking, the Evaluative Thinking Inventory (ETI).

Setting: The ETI was developed as part of an ECB initiative focused on non-formal science, engineering, technology, and math (STEM) education in the United States, and was tested as part of a study focused on evaluating gifted education programs, also in the United States.

Intervention: Not applicable.

Research design: Survey research and exploratory factor analysis (EFA).

Data collection & analysis: The ETI was administered to participants in a study measuring the effectiveness of a tool used to conduct internal evaluations of gifted education programs. SPSS was used to conduct an EFA on 96 completed ETIs. Cronbach’s alpha was used to estimate the internal consistency of the instrument.

Findings: The analysis of the ETI revealed a two-factor model of evaluative thinking (i.e., believe in and practice evaluation and pose thoughtful questions and seek alternatives). This study also provided internal consistency evidence for the ETI showing alpha reliabilities for the two factors ranging from 0.80 to 0.82. The ETI has potentially wide applicability in research and practice in ECB and in the field of evaluation more generally.

Factors Affecting Evaluation Culture within a Non-Formal Educational Organization

Evaluation and Program Planning, 2018

While research has been done on many aspects of evaluation within a variety of contexts and organ... more While research has been done on many aspects of evaluation within a variety of contexts and organizations, there is a lack of research surrounding the culture of evaluation. This study set out to examine this evaluative culture in one of the world's largest non-formal educational organizations through the use of an online survey and quantitative methodology. A path model was developed to examine the factors affecting evaluation culture. Results show perception regarding evaluation, program area, college major, location, training in evaluation, degree level, and years of experience explained 28% of the variance within evaluation culture. Results also found that the culture of evaluation is greatly impacted by leadership. By taking a closer look at the evaluation culture of a large non-formal educational organization, much can be learned about how to better develop and support evaluative work in other similar organizations and programs.

Whose Extension Counts? A Plurality of Extensions and Their Implications for Credible Evidence Debates

Journal of Human Sciences and Extension, 2019

The debate over what counts as credible evidence often occurs on a methodological level (i.e., ab... more The debate over what counts as credible evidence often occurs on a methodological level (i.e., about what technical applications of systematic inquiry provide believable, justifiable claims about a program). Less often, it occurs on an epistemological level (i.e., about what ways of knowing are appropriate for making claims about a program). Even less often, the debate touches on ontological concerns (i.e., about what conceptualizations of reality, in general or in relation to a specific program, are in play when we wish to make claims about that program). For example, whether we understand Extension to be a vehicle for the dissemination of scientific knowledge or a site of grassroots democracy matters when we seek to evaluate Extension with credibility. The purpose of this paper is to examine the credible evidence debates through an ontological lens, showing why and how different narratives (or different realities) of Extension must be considered when we seek credible evidence about Extension.

What’s the Problem Represented to Be? Problem Definition Critique as a Tool for Evaluative Thinking

American Journal of Evaluation, 2019

Problem definition in program planning and evaluation is rarely problematized. In this article, I... more Problem definition in program planning and evaluation is rarely problematized. In this article, I discuss why the lack of problem problematization is itself problematic—in other words, why treating problems as self-evident can pose a risk for evaluation practice. Then, to help avoid such risks, I suggest Carol Bacchi’s “What’s the Problem Represented to Be?” approach as a useful tool to focus evaluative thinking on the frequently tacit step of problem definition. Bacchi’s approach, informed by feminist and poststructuralist epistemologies, is designed to facilitate critical interrogation of policies and programs, focusing on the social and value-laden ways in which problems are initially defined. I propose that evaluators can fruitfully use Bacchi’s tool to promote evaluative thinking about the assumptions that inhere problem representation and thus can help promote better evaluation.

The Graduate Extension Scholars Program: Professional Development to Connect Research and Education

by Ayla A Wilk and Thomas Archibald

Journal of Extension, 2018

The Graduate Extension Scholars program represents a novel approach that brings together graduate... more The Graduate Extension Scholars program represents a novel approach that brings together graduate students, 4-H agents, and secondary agriculture teachers in an educational resource development project. We conducted process evaluation research based on program goals for the pilot implementation year using participant interviews and program artifacts. Program participants reported an overall positive experience and identified new programming and partnerships that would not have been possible without involvement in the program. Goals were enacted to varying degrees, with challenges occurring related to collaborative planning, educational module development, and building of partnerships between Extension and school-based educators. Recommendations for Extension program development are identified.

Using a Participatory Approach to Investigate a Leadership Program's Theory of Change

by Thomas Archibald, B. Burbaugh, and Megan Seibel

The linkages between interventions and outcomes of leadership initiatives have been insufficientl... more The linkages between interventions and outcomes of leadership initiatives have been insufficiently studied. To better understand these links, the Virginia agricultural leadership program conducted a pathway mapping session to investigate the program's theory of change. In this novel process, program participants engaged in collaborative brainstorming to identify outcomes and connect them to program activities. This process helped participants reflect on their learning and helped program staff identify the mechanisms (i.e., learning activities) that led to desired outcomes. The group produced a visual model that represents the program's theory of change depicting how the program develops capacity for leadership. Using program participants to identify the connections between pedagogy, practice, and outcomes can inform the evidence-base and development of evaluation plans for leadership development programs.

Evaluation Champions: What They Need and Where They Fit in Organizational Learning

Evaluation champions in four state Extension systems described needs for training and support and... more Evaluation champions in four state Extension systems described needs for training and support and perspectives on building evaluation capacity in their organization as part of a larger interview study exploring their roles, growth, and motivations. These 40 evaluation leaders identified needs for basic and advanced evaluation skills training, technical assistance, and practical learning via mentoring and project teams. Recommendations for organizational change in evaluation capacity included " top-down " investments in communication, training, and practical support, as well as increased " bottom-up " efforts by champions like themselves to advocate, model best practice, and contribute to training and mentoring peers. Implications for professional development and evaluation capacity building in Extension and other community-based organizations are discussed.

Evaluation Champions: What They Do, Why They Do It, and Why It Matters to Organizations

Evaluation champions are individuals who serve as catalysts for building evaluation capacity with... more Evaluation champions are individuals who serve as catalysts for building evaluation capacity within an organization. They advocate for the importance of program evaluation, model good evaluation behaviors, and mentor their peers in program evaluation skills and competencies. Interviews with 40 peer-nominated champions in four purposively-sampled Extension organizations identified the roles, contexts, and motivations of staff who act as evaluation champions. Findings underline the importance—and the limits—of mentors and project teams in building evaluation capacity in complex organizations. Implications for practice, research, and policy are discussed.

« La pensée évaluative » : Une activité mystérieuse et quotidienne

by Thomas Archibald and Laurent Ogoueli Moussavou

La pensée évaluative est, en bref, la pensée critique appliquée aux contextes de l'évaluation. Ce... more La pensée évaluative est, en bref, la pensée critique appliquée aux contextes de l'évaluation. Ce concept est relativement nouveau dans l'évaluation des programmes et politiques publiques. Bien qu'il reste un peu vague, il est important car il est utilisé par la plupart d'entre nous, de façon quasi-quotidienne. Ainsi, si l'évaluation formelle peut intimider les acteurs du domaine éducatif, la nature commune et quotidienne de la pensée évaluative peut représenter une ouverture pour les aider à apprendre à évaluer.

Assumptions, Conjectures, and Other Miracles: The Application of Evaluative Thinking to Theory of Change Models in Community Development

Unexamined and unjustified assumptions are the Achilles heel of development programs. In this pap... more Unexamined and unjustified assumptions are the Achilles heel of development programs. In this paper, we describe an evaluation capacity building (ECB) approach designed to help community development practitioners work more effectively with assumptions through the intentional infusion of evaluative thinking (ET) into the program planning, monitoring, and evaluation process. We focus specifically on one component of our ET promotion approach involving the creation and analysis of theory of change (ToC) models. We describe our recent efforts to pilot this ET ECB approach with Catholic Relief Services (CRS) in Ethiopia and Zambia. The use of ToC models, plus the addition of ET, is a way to encourage individual and organizational learning and adaptive management that supports more reflective and responsive programming.

Defining and teaching evaluative thinking: Insights from research on critical thinking

Evaluative thinking (ET) is an increasingly important topic in the field of evaluation, particula... more Evaluative thinking (ET) is an increasingly important topic in the field of evaluation, particularly among people involved in evaluation capacity building (ECB). Yet it is a construct in need of clarification, especially if it is to be meaningfully discussed, promoted, and researched. To that end, we propose that ET is essentially critical thinking applied to contexts of evaluation. We argue that ECB, and the field of evaluation more generally, would benefit from an explicit and transparent appropriation of well-established concepts and teaching strategies derived from the long history of work on critical thinking. In this paper, based on previous work in the fields of education, cognitive science and critical thinking, as well as on our experience as ECB practitioners, we propose several guiding principles and specific strategies for teaching ET that draw directly from research on the teaching of critical thinking.

‘‘They Just Know’’: The epistemological politics of ‘‘evidence-based’’ non-formal education

Evaluation and Program Planning, 2015

Community education and outreach programs should be evidence-based. This dictum seems at once war... more Community education and outreach programs should be evidence-based. This dictum seems at once warranted, welcome, and slightly platitudinous. However, the ‘‘evidence based’’ movement’s more narrow definition of evidence—privileging randomized controlled trials as the ‘‘gold standard’’—has fomented much debate. Such debate, though insightful, often lacks grounding in actual practice. To address that lack, the purpose of the study presented in this paper was to examine what actually happens, in practice, when people support the implementation of evidence-based programs (EBPs) or engage in related efforts to make non-formal education more ‘‘evidence-based.’’ Focusing on three cases—two adolescent sexual health projects (one in the United States and one in Kenya) and one more general youth development organization—I used qualitative methods to address the questions: (1) How is evidence-based program and evidence-based practice work actually practiced? (2) What perspectives and assumptions about what non-formal education is are manifested through that work? and (3) What conflicts and tensions emerge through that work related to those perspectives and assumptions? Informed by theoretical perspectives on the intersection of science, expertise, and democracy, I conclude that the current dominant approach to making non-formal education more evidence-based by way of EBPs is seriously flawed.

Rethinking Empowerment: Theories of Power and the Potential for Emancipatory Praxis

In this paper, we make the case that " empowerment " is still a concept which requires closer scr... more In this paper, we make the case that " empowerment " is still a concept which requires closer scrutiny, for both practical and theoretical reasons. Building on critiques of empowerment and on calls to more consciously address the power relations inherent in adult education, we lay the groundwork for a practical theory of action whereby education practitioners and researchers can work towards salvaging empowerment's emancipatory potential.

Relationships between Quantitative Measures of Evaluation Plan and Program Modeling Quality and a Qualitative Measure of Participant Perceptions of an Evaluation Capacity Building Approach

Despite a heightened emphasis on building evaluation capacity and evaluation quality, there is st... more Despite a heightened emphasis on building evaluation capacity and evaluation quality, there is still a lack of tools available to identify high quality evaluation. In the context of a longitudinal study testing the Systems Evaluation Protocol (SEP), rubrics were designed to assess the quality of evaluation plans and models. The rubrics were tested for reliability and internal consistency and used to calculate quantitative quality scores. Qualitative interview data were also collected and analyzed using a priori codes. A mixed methods approach was used to synthesize quantitative and qualitative data and explore trends. Consistencies between qualitative and quantitative data were found for attitude and capacity and disconnects between qualitative and quantitative data were found for knowledge, cyberinfrastructure, time, and quality. The approach to data integration demonstrated in this paper represents a novel way to tap the generative potential of divergence that arises when different methods produce contradictory results.

Book Reviews by Thomas Archibald

Review of Being an Evaluator: Your Practical Guide to Evaluation, by Donna Podems.

American Journal of Evaluation, 2020

Community Organizing in/as Adult Education

Community organizing and adult education overlap in a number of ways, yet other than some notable... more Community organizing and adult education overlap in a number of ways, yet other than some notable exceptions, those overlaps remain inadequately explicated and insufficiently represented in contemporary adult education literature. The purpose of this paper is to explore the conceptual and practical overlaps of adult education and community organizing, in an effort to elevate an adult education praxis that promotes participation, equity, and justice.

Consumer Response to Mechanically Tenderized Beef (MTB) and MTB Labels: An Exploratory Focus Group Study

Journal of Food Protection

In May 2016, labeling of certain nonintact mechanically tenderized beef (MTB) products was mandat... more In May 2016, labeling of certain nonintact mechanically tenderized beef (MTB) products was mandated in the United States. MTB products should be handled differently by the consumer because pathogens can be transferred from the exterior to the interior of the meat during the tenderization process. Without labeling, it is difficult to visually distinguish between some intact beef and MTB products, which is a concern because MTB products require higher internal cooking temperatures for safety. An exploratory study was conducted to understand consumer understanding of MTB products and consumer responses to the new label. Thirteen focus groups were convened in rural and urban settings across Virginia and North Carolina between December 2015 and May 2016. Sessions were audiorecorded, transcribed verbatim, and analyzed through constant-comparison thematic analysis. Although MTB products were commonly bought, prepared, and consumed, consumer awareness of MTB products and the MTB process was...

The Critically Reflective Evaluator: Adult Education's Contributions to Evaluation for Social Justice

New Directions for Evaluation

International Positive Youth Development: Challenges and Opportunities in Policy and Practice

Journal of Youth Development

This special issue of the Journal of Youth Development on international positive youth developmen... more This special issue of the Journal of Youth Development on international positive youth development brings together diverse perspectives of research, practice, and policy regarding the challenges and opportunities facing international efforts to promote the positive development of youth. To represent a plurality of perspectives on international PYD, the authors sought to represent a variety of continents, countries, genders, races, ethnicities, sectors, and levels of experience with publishing. There is also variety in the types of articles such as case studies, presentations of frameworks and curricula, letter-style commentary by youth, and cutting-edge research methodologies and studies. A few key themes frame the volume, including (a) diversity in definitions of youth and operationalizations of PYD; (b) the multifaceted and multidimensional ways in which PYD is integrated into different sectors, domains, and types of programmatic and organizational contexts; and (c) the importance...

Positive Youth Development in Senegal: A Case Study of 4-H Senegal

by Thomas Archibald, Kathleen Jamison, and Fatou Diouf

Journal of Youth Development , 2021

African youth face great challenges and great opportunities. Positive youth development (PYD) pro... more African youth face great challenges and great opportunities. Positive youth development (PYD) programs have the potential to support youth in Africa to reach their full potential. Yet PYD approaches must be tailored if they are to be contextually and culturally responsive and relevant. The purpose of this paper is to provide an evidence-informed description of one such program, the 4-H Senegal PYD program. We first provide a conceptual background for the program, informed especially by the literature on global and cross-cultural PYD. We then share empirical results of a qualitative case study focused on the establishment, scale-up, and institutionalization of the 4-H Senegal program. Our case study results elucidate culturally responsive PYD program adaptations that corroborate conceptual framings of cross

Refining and Measuring the Construct of Evaluative Thinking: An Exploratory Factor Analysis of the Evaluative Thinking Inventory Journal of MultiDisciplinary Evaluation

by Jane Buckley and Thomas Archibald

Journal of MultiDisciplinary Evaluation, 2020

Background: Evaluative thinking has emerged as a key construct in evaluation, especially for eval... more Background: Evaluative thinking has emerged as a key construct in evaluation, especially for evaluation practitioners and researchers interested in evaluation capacity building (ECB). Yet, despite increasing calls for more research on evaluation and, more specifically, for more research on ECB, until recently little empirical inquiry on the dimensions of evaluative thinking has been conducted.

Purpose: To address that lack, the purpose of the study presented in this paper is to refine the construct of evaluative thinking by exploring its underlying dimensions and to ascertain the internal consistency of an instrument developed to measure evaluative thinking, the Evaluative Thinking Inventory (ETI).

Setting: The ETI was developed as part of an ECB initiative focused on non-formal science, engineering, technology, and math (STEM) education in the United States, and was tested as part of a study focused on evaluating gifted education programs, also in the United States.

Intervention: Not applicable.

Research design: Survey research and exploratory factor analysis (EFA).

Data collection & analysis: The ETI was administered to participants in a study measuring the effectiveness of a tool used to conduct internal evaluations of gifted education programs. SPSS was used to conduct an EFA on 96 completed ETIs. Cronbach’s alpha was used to estimate the internal consistency of the instrument.

Findings: The analysis of the ETI revealed a two-factor model of evaluative thinking (i.e., believe in and practice evaluation and pose thoughtful questions and seek alternatives). This study also provided internal consistency evidence for the ETI showing alpha reliabilities for the two factors ranging from 0.80 to 0.82. The ETI has potentially wide applicability in research and practice in ECB and in the field of evaluation more generally.

Factors Affecting Evaluation Culture within a Non-Formal Educational Organization

Evaluation and Program Planning, 2018

While research has been done on many aspects of evaluation within a variety of contexts and organ... more While research has been done on many aspects of evaluation within a variety of contexts and organizations, there is a lack of research surrounding the culture of evaluation. This study set out to examine this evaluative culture in one of the world's largest non-formal educational organizations through the use of an online survey and quantitative methodology. A path model was developed to examine the factors affecting evaluation culture. Results show perception regarding evaluation, program area, college major, location, training in evaluation, degree level, and years of experience explained 28% of the variance within evaluation culture. Results also found that the culture of evaluation is greatly impacted by leadership. By taking a closer look at the evaluation culture of a large non-formal educational organization, much can be learned about how to better develop and support evaluative work in other similar organizations and programs.

Whose Extension Counts? A Plurality of Extensions and Their Implications for Credible Evidence Debates

Journal of Human Sciences and Extension, 2019

The debate over what counts as credible evidence often occurs on a methodological level (i.e., ab... more The debate over what counts as credible evidence often occurs on a methodological level (i.e., about what technical applications of systematic inquiry provide believable, justifiable claims about a program). Less often, it occurs on an epistemological level (i.e., about what ways of knowing are appropriate for making claims about a program). Even less often, the debate touches on ontological concerns (i.e., about what conceptualizations of reality, in general or in relation to a specific program, are in play when we wish to make claims about that program). For example, whether we understand Extension to be a vehicle for the dissemination of scientific knowledge or a site of grassroots democracy matters when we seek to evaluate Extension with credibility. The purpose of this paper is to examine the credible evidence debates through an ontological lens, showing why and how different narratives (or different realities) of Extension must be considered when we seek credible evidence about Extension.

What’s the Problem Represented to Be? Problem Definition Critique as a Tool for Evaluative Thinking

American Journal of Evaluation, 2019

Problem definition in program planning and evaluation is rarely problematized. In this article, I... more Problem definition in program planning and evaluation is rarely problematized. In this article, I discuss why the lack of problem problematization is itself problematic—in other words, why treating problems as self-evident can pose a risk for evaluation practice. Then, to help avoid such risks, I suggest Carol Bacchi’s “What’s the Problem Represented to Be?” approach as a useful tool to focus evaluative thinking on the frequently tacit step of problem definition. Bacchi’s approach, informed by feminist and poststructuralist epistemologies, is designed to facilitate critical interrogation of policies and programs, focusing on the social and value-laden ways in which problems are initially defined. I propose that evaluators can fruitfully use Bacchi’s tool to promote evaluative thinking about the assumptions that inhere problem representation and thus can help promote better evaluation.

The Graduate Extension Scholars Program: Professional Development to Connect Research and Education

by Ayla A Wilk and Thomas Archibald

Journal of Extension, 2018

The Graduate Extension Scholars program represents a novel approach that brings together graduate... more The Graduate Extension Scholars program represents a novel approach that brings together graduate students, 4-H agents, and secondary agriculture teachers in an educational resource development project. We conducted process evaluation research based on program goals for the pilot implementation year using participant interviews and program artifacts. Program participants reported an overall positive experience and identified new programming and partnerships that would not have been possible without involvement in the program. Goals were enacted to varying degrees, with challenges occurring related to collaborative planning, educational module development, and building of partnerships between Extension and school-based educators. Recommendations for Extension program development are identified.

Using a Participatory Approach to Investigate a Leadership Program's Theory of Change

by Thomas Archibald, B. Burbaugh, and Megan Seibel

The linkages between interventions and outcomes of leadership initiatives have been insufficientl... more The linkages between interventions and outcomes of leadership initiatives have been insufficiently studied. To better understand these links, the Virginia agricultural leadership program conducted a pathway mapping session to investigate the program's theory of change. In this novel process, program participants engaged in collaborative brainstorming to identify outcomes and connect them to program activities. This process helped participants reflect on their learning and helped program staff identify the mechanisms (i.e., learning activities) that led to desired outcomes. The group produced a visual model that represents the program's theory of change depicting how the program develops capacity for leadership. Using program participants to identify the connections between pedagogy, practice, and outcomes can inform the evidence-base and development of evaluation plans for leadership development programs.

Evaluation Champions: What They Need and Where They Fit in Organizational Learning

Evaluation champions in four state Extension systems described needs for training and support and... more Evaluation champions in four state Extension systems described needs for training and support and perspectives on building evaluation capacity in their organization as part of a larger interview study exploring their roles, growth, and motivations. These 40 evaluation leaders identified needs for basic and advanced evaluation skills training, technical assistance, and practical learning via mentoring and project teams. Recommendations for organizational change in evaluation capacity included " top-down " investments in communication, training, and practical support, as well as increased " bottom-up " efforts by champions like themselves to advocate, model best practice, and contribute to training and mentoring peers. Implications for professional development and evaluation capacity building in Extension and other community-based organizations are discussed.

Evaluation Champions: What They Do, Why They Do It, and Why It Matters to Organizations

Evaluation champions are individuals who serve as catalysts for building evaluation capacity with... more Evaluation champions are individuals who serve as catalysts for building evaluation capacity within an organization. They advocate for the importance of program evaluation, model good evaluation behaviors, and mentor their peers in program evaluation skills and competencies. Interviews with 40 peer-nominated champions in four purposively-sampled Extension organizations identified the roles, contexts, and motivations of staff who act as evaluation champions. Findings underline the importance—and the limits—of mentors and project teams in building evaluation capacity in complex organizations. Implications for practice, research, and policy are discussed.

« La pensée évaluative » : Une activité mystérieuse et quotidienne

by Thomas Archibald and Laurent Ogoueli Moussavou

La pensée évaluative est, en bref, la pensée critique appliquée aux contextes de l'évaluation. Ce... more La pensée évaluative est, en bref, la pensée critique appliquée aux contextes de l'évaluation. Ce concept est relativement nouveau dans l'évaluation des programmes et politiques publiques. Bien qu'il reste un peu vague, il est important car il est utilisé par la plupart d'entre nous, de façon quasi-quotidienne. Ainsi, si l'évaluation formelle peut intimider les acteurs du domaine éducatif, la nature commune et quotidienne de la pensée évaluative peut représenter une ouverture pour les aider à apprendre à évaluer.

Assumptions, Conjectures, and Other Miracles: The Application of Evaluative Thinking to Theory of Change Models in Community Development

Unexamined and unjustified assumptions are the Achilles heel of development programs. In this pap... more Unexamined and unjustified assumptions are the Achilles heel of development programs. In this paper, we describe an evaluation capacity building (ECB) approach designed to help community development practitioners work more effectively with assumptions through the intentional infusion of evaluative thinking (ET) into the program planning, monitoring, and evaluation process. We focus specifically on one component of our ET promotion approach involving the creation and analysis of theory of change (ToC) models. We describe our recent efforts to pilot this ET ECB approach with Catholic Relief Services (CRS) in Ethiopia and Zambia. The use of ToC models, plus the addition of ET, is a way to encourage individual and organizational learning and adaptive management that supports more reflective and responsive programming.

Defining and teaching evaluative thinking: Insights from research on critical thinking

Evaluative thinking (ET) is an increasingly important topic in the field of evaluation, particula... more Evaluative thinking (ET) is an increasingly important topic in the field of evaluation, particularly among people involved in evaluation capacity building (ECB). Yet it is a construct in need of clarification, especially if it is to be meaningfully discussed, promoted, and researched. To that end, we propose that ET is essentially critical thinking applied to contexts of evaluation. We argue that ECB, and the field of evaluation more generally, would benefit from an explicit and transparent appropriation of well-established concepts and teaching strategies derived from the long history of work on critical thinking. In this paper, based on previous work in the fields of education, cognitive science and critical thinking, as well as on our experience as ECB practitioners, we propose several guiding principles and specific strategies for teaching ET that draw directly from research on the teaching of critical thinking.

‘‘They Just Know’’: The epistemological politics of ‘‘evidence-based’’ non-formal education

Evaluation and Program Planning, 2015

Community education and outreach programs should be evidence-based. This dictum seems at once war... more Community education and outreach programs should be evidence-based. This dictum seems at once warranted, welcome, and slightly platitudinous. However, the ‘‘evidence based’’ movement’s more narrow definition of evidence—privileging randomized controlled trials as the ‘‘gold standard’’—has fomented much debate. Such debate, though insightful, often lacks grounding in actual practice. To address that lack, the purpose of the study presented in this paper was to examine what actually happens, in practice, when people support the implementation of evidence-based programs (EBPs) or engage in related efforts to make non-formal education more ‘‘evidence-based.’’ Focusing on three cases—two adolescent sexual health projects (one in the United States and one in Kenya) and one more general youth development organization—I used qualitative methods to address the questions: (1) How is evidence-based program and evidence-based practice work actually practiced? (2) What perspectives and assumptions about what non-formal education is are manifested through that work? and (3) What conflicts and tensions emerge through that work related to those perspectives and assumptions? Informed by theoretical perspectives on the intersection of science, expertise, and democracy, I conclude that the current dominant approach to making non-formal education more evidence-based by way of EBPs is seriously flawed.

Rethinking Empowerment: Theories of Power and the Potential for Emancipatory Praxis

In this paper, we make the case that " empowerment " is still a concept which requires closer scr... more In this paper, we make the case that " empowerment " is still a concept which requires closer scrutiny, for both practical and theoretical reasons. Building on critiques of empowerment and on calls to more consciously address the power relations inherent in adult education, we lay the groundwork for a practical theory of action whereby education practitioners and researchers can work towards salvaging empowerment's emancipatory potential.

Relationships between Quantitative Measures of Evaluation Plan and Program Modeling Quality and a Qualitative Measure of Participant Perceptions of an Evaluation Capacity Building Approach

Despite a heightened emphasis on building evaluation capacity and evaluation quality, there is st... more Despite a heightened emphasis on building evaluation capacity and evaluation quality, there is still a lack of tools available to identify high quality evaluation. In the context of a longitudinal study testing the Systems Evaluation Protocol (SEP), rubrics were designed to assess the quality of evaluation plans and models. The rubrics were tested for reliability and internal consistency and used to calculate quantitative quality scores. Qualitative interview data were also collected and analyzed using a priori codes. A mixed methods approach was used to synthesize quantitative and qualitative data and explore trends. Consistencies between qualitative and quantitative data were found for attitude and capacity and disconnects between qualitative and quantitative data were found for knowledge, cyberinfrastructure, time, and quality. The approach to data integration demonstrated in this paper represents a novel way to tap the generative potential of divergence that arises when different methods produce contradictory results.

Review of Being an Evaluator: Your Practical Guide to Evaluation, by Donna Podems.

American Journal of Evaluation, 2020

Review of Schwandt, T. A. (2015). Evaluation Foundations Revisited: Cultivating a Life of the Mind for Practice

Designing an Evaluation Culture: Exploring the Relationship of Evaluation Capacity Building and Evaluation Culture

by Courtney Vengrin and Thomas Archibald

Without a culture of evaluation, evaluation capacity building work cannot be successful. While mu... more Without a culture of evaluation, evaluation capacity building work cannot be successful. While much has been done to develop the area of evaluation capacity building, evaluation culture has been somewhat overlooked. One cannot exist without the other and they must work in tandem in order to achieve lasting change within the organizational culture. Evaluation capacity building does not only build the capacity of an organization to preform evaluations, but it also builds the culture of evaluation, as a part of the overall organizational culture. A culture of evaluation should be designed by those doing the capacity building work as well as those within the organization undergoing capacity building efforts. Consciously considering how to design a culture of evaluation can provide a platform for evaluation capacity building work. It’s time that we move forward and open up the dialogue surrounding evaluation culture and its critical relationship to evaluation capacity building.