First, we had to prepare for the evaluation, supported by Pamoja Evaluation Services: formulate and agree on a testable contribution claim; develop the evaluation plan and data to collect; prepare the Contribution Tracing ‘workbook’; and create the key informant interview questionnaires.
A key part of the preparatory work was to formulate and agree on a claim of change we can test. Contribution Tracing centres on testing a claim by investigating whether evidence exists to support or reject it. A claim is simply a statement about the role a project may have played in bringing about change. At GSAM, we have many claims we wish to test and so this became our first challenge! After several rounds of discussion, we decided to test the following claim:
The GSAM Contribution Claim: “GSAM's facilitation of citizen's oversight on capital projects has improved District Assemblies’ responsiveness to citizen's concerns.”
As we began to understand how Contribution Tracing works in practice, particularly in how the approach deals with data collection, we realised that there was something truly different about it. As the evaluation team lead, Samuel Addai-Boateng of CARE International in Ghana, noted: “Unlike previous data collection exercises we have done under the GSAM project, we set out not only to conduct interviews, but to collect all available evidence that would help us verify our claim. This was also the first time the team was applying Contribution Tracing and so it came with some uncertainty and eagerness, at the same time.”
Guided by the various types of evidence used in the four process tracing tests – the hoop test, smoking gun, doubly-decisive and straw-in-the-wind – the team was on a mission to build a watertight dossier for the case on hand. With the support of a semi-structured interview guide and recording apps on their phones, the team interacted with citizens in four communities situated in two of Ghana’s administrative districts: Afigya Kwabre and Wassa East. These were selected as the study areas out of the 100 districts where the project is being implemented. The team also interacted with local government officials, identified as key informants, as well as staff of civil society organisations that were implementing the GSAM project in these districts.
Interview after interview, community after community, the exercise progressed smoothly and a lot of interesting evidence – videos, photos, documents, recordings, etc – was gathered, which could be classified under the various types of evidence. For instance, in Afigya Kwabre, members of the evaluation team, who were not known in that district, conducted an interview with a top official from the District Assembly. He confirmed how the GSAM project had contributed to making the assembly responsive to citizens’ concerns. The video recording of this interview has, indeed, become a doubly-decisive piece of evidence. The team again recorded voices of citizens making comments that clearly confirmed the claim. This could straightforwardly act as ‘love-to-see’ evidence.
Reports and photos of project activities that had contributed to the claim were also gathered, as was expected. In addition, we obtained photos of some portions of the Assembly’s documents. Although these were ‘straw-in-the-wind’ evidence, they corroborated other evidence to give more credence to our claim. At the end of the data collection exercise, it became obvious that the team had learnt several lessons, as summarised below, which are worth sharing with any group interested in using the Contribution Tracing approach.
1. Be observant and ‘hungry’ for new evidence
Prior to the data collection, the team had outlined pieces of evidence, mainly ‘hoop test’ evidence, which were likely to be collected. Whilst conducting the interviews, other evidence, some of which could be classified as ‘smoking gun’ (love to see) evidence, was discovered.
“In Wawase (Afigya Kwabre), we noticed that the chief had documented on a board at the palace major events or activities that had taken place in the community. Interestingly, one of these activities was the sod-cutting ceremony for the community centre project, in which the Chief acknowledged the role played by the GSAM project and its partner, Access to Life Foundation, in the ceremony.” – Michael Tettey, ISODEC.
2. Probe, probe, probe
Some evidence was only discovered after team members probed further.
“In Afigya Kwabre, for instance, the Planning Officer had indicated there were no reports of the Assembly that referred to the GSAM project and issues related to citizen monitoring of the Community Centre project. However, when we probed further, it emerged that the Assembly had reported the status of the Community Centre project in one of its quarterly reports to the National Development Planning Commission (NDPC). We were then allowed to take a shot of the pages that contained this information and it serves as another piece of evidence for our claim.” – Samuel Asamoah-Boateng, Oxfam.
3. Pre-testing is key
Although the team pre-tested the data collection tools, it became evident during the data collection that further iterations were required to ‘get it right’. Evaluation team member Samuel Addai-Boateng observed, “As we collected the data, we noticed there were some gaps with the interview guide and these gaps could have been fixed if we had done a more thorough pre-testing in the study communities.”
4. Be flexible
The team had initially planned to have interviews with a number of key informants in the study communities. However, it did not take long for the team to realise it was best to complement the interviews with focus group discussions.
“In Wassa East, we brought the key informants from the community together to do a focus group discussion. In the end, it was an interesting and revealing discussion as community members reinforced each other’s views, whilst also maintaining individual point of views.” – Francisca Agyekum, Access to Life Foundation.
5. Gathering evidence may take longer than planned
Whilst it was the expectation of the team that the evidence required could be collected within two days, it became obvious after the two days that not all the evidence had been collected within that period. This was because a few of the key informants could not readily provide some of the evidence required and so the team had to follow up later, as Peter Porekuu of CARE International in Ghana notes: “We had to follow up later on the civil society partner to collect certain activity reports which contained useful pieces of information that could support our claims.” Consequently, it took more than a week to get all pieces of evidence together.
Currently, the team is analysing the pieces of evidence gathered and will be sharing the outcome of the evaluation in due course. Stay tuned…
The GSAM project is a five-year project that seeks to “strengthen citizens’ oversight of capital projects to improve local government transparency, accountability and performance in 100 districts of Ghana.” It is being funded by the United States Agency for International Development (USAID) and implemented by the GSAM Consortium – CARE, Oxfam in Ghana and ISODEC – and 27 local civil society organisation partners.
The evaluation team (nicknamed the ‘Dream Team’) comprises GSAM staff: Samuel Addai Boateng, Michael Tettey, Samuel Asamoah Boateng, Francisca Adjekum, Peter Porekuu and Mohammed Nurudeen Salifu. This blog draws from a live case study of a Contribution Tracing pilot evaluation of the GSAM project. The pilot forms part of a learning partnership called the Innovations in Capturing Complex Change Project, between Pamoja Evaluation Services, CARE International UK and CARE Ghana and CARE Bangladesh country offices. For more on the Contribution Tracing approach, please see this blog series on Insights.
