Capturing complex change: Is it really all about confidence?

by 22nd May 2017
Holding those in power to account: A community meeting in Bangladesh Holding those in power to account: A community meeting in Bangladesh

For those of us that sing the praises of social accountability (citizen-driven initiatives to hold those in power to account), making a claim about “impact” (or transformative change) is a challenge we face on a daily basis. And CARE’s not alone. The title of the first session at an NGO political economy analysis working group at which I’m presenting this week (“Building the Evidence-base for Social Accountability”) speaks to the same concern.

Some argue that we need more longitudinal studies. Others advocate the use of Randomised Control Trials (RCTs). And a recent study CARE conducted on the influence of Community Score Cards on reproductive health-related outcomes in Malawi shows that RCTs have a place, and do demonstrate that social accountability makes a difference.

But, once you consider that outcomes are behavioural changes of real (complicated) people, you quickly see, as Marina Apgar recently suggested, why we need to move “beyond measurement of linear pre-defined change and intervention-effect alone and [use] mixed-methods to help us understand emergent complex social change.” Social accountability outcomes (such as mayors changing budgets to benefit poorer areas, or even procuring a new ambulance) don’t fit neatly into boxes. They rely on our capacity to influence behaviour, and this is behaviour we can’t (fully) control. So, we need to better explain HOW change happened, not merely to assert that it did. 

Recognising this has led CARE to explore various theory-based methods such as Most Significant Change and Outcome Mapping. With a particular emphasis on the change process, we are now piloting Contribution Tracing with Pamoja Evaluation Services in Ghana and Bangladesh to help us better understand CARE’s contribution to social accountability outcomes.

Contribution Tracing is all about increasing your confidence in making claims about impact. Essentially, you make a “claim” about your intervention’s role in achieving an outcome that really happened (your contribution), and then find evidence to defend your claim.

To do this, like other theory-based methods, you need a hypothesis (a proposed explanation) about how you think change happened. You then review the connection between different steps (or components) in that process:

intervention outcome graphic

You identify evidence that would help support (or undermine) your proposed explanation using the four tests of Process Tracing (Straws-in-the-wind, Hoops, Smoking Guns, Doubly Decisive).

What matters is not how much evidence you have, but how good that evidence is to help confirm that each part of your proposed explanation for your claim really exists (“probative value”).

In Contribution Tracing, you use Baysian (Confidence) Updating to assign a probability (how likely it is) that the various components of your contribution claim exist; and ultimately whether your claim holds true. You then update your confidence after gathering data precisely tailored to your claim (increasing or decreasing the probability using the four tests), compare this against rival explanations, and then put it up for “trial”, inviting others in to peer review your claim.

We’re right at the beginning of the journey, but to me, what our learning already suggests is that:

  • You can show your contribution, even when change processes are complex;
  • You can make credible impact claims, without a counterfactual;
  • You can tighten up your loose theory of change as you go along, and;
  • You may not need to gather as much data as you think you do to prove it.

But don’t take my word for it; listen to some reflections from staff on the experience so far. And watch this space for more to come.

Tom Aston

Currently I am working as the monitoring and evaluation and research lead for the inclusive governance team. I am particularly looking at the application of theory-based evaluation methods such as contribution tracing and outcome mapping.

I joined CARE International UK in 2012, providing support to the Latin America and the Caribbean and Middle East and North Africa regions, particularly in conducting political economy analyses, and conducting studies on social accountability and advocacy.

I have an MSc in Development Administration and Planning from University College London (UCL) and am doing a PhD on the political economy of cash transfers, with Bolivia as a case study. Previously I worked for CARE Bolivia and as a consultant for the ODI and UCL on issues of social protection and disaster risk reduction.

One good thing I've read

For those of you looking to unlock the activist inside you I recommend Rebecca Solnit’s Hope in the dark: Untold histories, wild possibilities.

Email: Aston@careinternational.org