One of the privileges in working with Tearfund is to be able to visit communities and speak with people about how the work of our partners has helped bring about positive change into the lives of their families and communities.
In February 2024 I was fortunate to be able to lead an evaluation of the work of Tearfund’s partner, ACROSS in Rumbek, South Sudan. The project in Rumbek has been going for a number of years, had been evaluated three years ago, and ACROSS was looking to see what difference their work had made and how they could learn from that to help them design and develop a new project for their next cycle of work.
There are many and varied evaluation methods, but as we were keen to hear from the community members themselves, create a process where ACROSS staff could get direct feedback about the impact of their work, and look at the broader impact of the project rather than the details of specific activities, we chose to use the Most Significant Change (MSC) technique.
MSC is a participatory evaluation method where project participants share stories about the most significant change they have experienced due to the project.
It is effective at generating a large amount of data, direct from the project participants themselves in their own words, and provides a way of analysing the data to draw out themes, identify strengths and weaknesses, and understand how change has occurred. MSC requires participants to engage and be involved in the evaluation process.
The process started with a team training session with a group of story collectors who would do the interviews in the communities. This was a useful time of refining the questions and discussing the MSC approach and what we hoped to achieve from it. It also gave a helpful introduction to the project with the staff team.
Once in the communities we met with groups of people who had been a part of the project, whether they were Farmers’ Group or Savings Group members, Water, Sanitation and Hygiene (WASH) Group members, or other group members. We held general discussions about the work, what they had done, what they had achieved and what they hoped for the future. In the story collection phase the evaluation team sat with small groups of community members to hear their individual stories of change.
As well as some basic questions to locate the story for analysis, three main questions were asked:
After the group members heard each other’s stories they had to select the story that best represented the change that they had experienced through the work of the project.
Throughout the week 98 stories of change were collected. These were each “coded” or categorised according to common themes as I sat outside in the warm evening South Sudanese air in the ACROSS compound.
On the final day of the evaluation the evaluation and ACROSS staff team met together to review the stories that had been selected by each group. From those stories they had to select one story for each category or area of change. In that way we had three different groups of stories: every individual’s story, the most significant stories as chosen by the community groups, and then a final selection of four stories that were felt to best describe the change that the project had been able to achieve. Discussion about these stories led to ideas for the next project that came from the staff themselves based on what they had heard from community members.
The coding enabled me to analyse the stories according to each area of change: what were the common themes about what had worked well, were there gaps or weaknesses that were identified in the project, and what were the most effective activities that had brought about positive change?
Using MSC enabled us to hear from project participants in their own words (albeit through translation by the the story-collectors) and combined with the usual document reviews and interviews with other people such as local government and project staff, gave a detailed and rich understanding of the reach and impact of the project. It also enabled us to develop a set of recommendations to inform and guide ACROSS as they designed their new work.
MSC is not always the most appropriate method for evaluations, but if the aim is to hear directly from the participants and there is time to do the whole process, it is an effective and rigorous approach to evaluation.
Phil Lindsay is Tearfund's Effectiveness and Humanitarian Team Leader
