Some musings on evaluating the impacts of integrated systems research
- 4. …but often need for “deep” understanding of system & context • Not always easy for evaluators to gain such an understanding • Actual decision-making processes (or reasons for reforms, etc.) often inaccessible • Insider knowledge useful & important but high potential for confirmation bias • Consensus on methods for addressing bias in large n studies not same degree as for small n • Historians & political scientists often debate reasons for big changes in history—things often not black & white, so A diagram developed for the U.S. military mapping spheres of influence in Afghanistan
- 5. How to link systems changes to changes in state? Sphere of Control (Things research teams can control) Sphere of Interest (Development impacts) Sphere of Influence (Changes in capacity, behavior, practice & policy that hopefully can be influenced—including improving or even transforming systems) Even when we can evidence contributions to systems-level change, go back to same small n evaluation challenges And things much more complex with systems transformation: relationships in whole system will change in unexpected ways
- 6. Towards new paradigm for systems-focused R4D impact evaluation (& R4D impact expectations)? • Arguably, the demand for direct impact evidence has made R4D less effective at inducing (or contributing to) desirable systems transformation • We (and donors) need to recognize (and embrace) the inherent complexity, and, in turn, limitations • Systems oriented R4D should focus— through well targeted research & engagement—on increasing the likelihood that the system will pivot (or jump) towards a more positive trajectory— without assuming control or ability to predict exactly what this trajectory will be • Evaluation can stop there if a jump has not happened—or if it has—further