Finding outcomes, understanding progress
At Prism we’re kicking off a new year-long learning partnership with a national organisation in Australia which includes using the practise of ‘outcome harvesting’ to identify and understand their contribution in a very complex system. Outcome harvesting is a method that helps to reflect on impact and measure progress in such complex systems. It about-faces traditional measures of progress and rather than predicting the outcomes that will be caused by your work, allows for examination of what is actually changing in a system, and determining what your credible contribution to that is.
The great benefit of the method in my mind is that if your work is long- term, systemic, distributed, adaptive, and political it is highly likely that your work will contribute to outcomes that are both intended and unintended, and potentially both positive and negative.
More traditional approaches to measuring progress take a linear approach - what objectives did you set for yourself and did you achieve them - or focus only on the ultimate and positive impacts and exclude a lot of the signs of progress that relate to shifting relationships, policies and resources in a broader system.
Outcomes harvesting asks: how did your work influence the actions, circumstances, practices and policies of other people and institutions? Outcomes here are shifts in others; prompted, shaped or conditioned in some way by the work of your organisation or initiative, both directly or alongside the work of many others, as is often the case.
Of course, the goals you have matter and accountability to delivering on objectives is important. Outcomes harvesting can help here too. As you collect outcomes you create a database organised by the things that matter to you and your understanding – the goals that you have, the level of change (e.g. individual, place-based, institutional, systemic), the role that you played (e.g. collaboration, research, advocacy, direct support, coalition-building) and others who were involved or contributed. You can track impacts you might not have predicted and hence didn’t set as objectives. You can start to build networks to see how these outcomes might connect in clusters of outcomes or to series of work or organisations who are working in related areas. You can start to see patterns and develop insights that are not visible when you are deep in doing the work or if you are constrained by looking for what you intended to do or imagined may result. To understand it more I’m sharing three examples of how I used outcomes harvesting across different projects.
I first started using outcome harvesting as a method about seven years ago – working on a review of the global advocacy work of Transparency International Secretariat (Berlin). The method made it possible to verify and understand where the Transparency International Secretariat was contributing to impact across a wide array of subjects (climate finance, clean contracting, infrastructure integrity, political capture, company ownership transparency.). This review identified 93 outcomes and clear enablers for where and how the TI Secretariat and Movement were able to align to create real influence, such as in their work on the transparency of company ownership information.
An example of an outcomes mapping from Transparency International Secretariat project
A few years later in 2021, along with my long term collaborator Annabel Brown, we used the method for another project involving Transparency International – their partnership with the Organized Crime and Corruption Reporting Project called The Global Anti-Corruption Consortium. Through the Consortium investigative journalists were working with advocates to support them to use the findings of investigations, resulting in real accountability for the corrupt and underlying reforms to prevent the same kinds of acts in the future. The (very impressive) set of outcomes from the first phase of their partnership demonstrated, in particular, the premise the work was based on: when the two organisations both directed their attention at a certain case, there were 5x as many outcomes compared to instances funded by the Consortium partnership where they worked independently.
While they were of course aware of the impact of key investigations, outcomes harvesting was uniquely placed to demonstrate the power of the partnership by creating a dataset that could compare outcomes when they worked together from when they did not.
An important part of this evaluation was sampling the dataset of 228 separate outcomes to verify clusters of outcomes. We reviewed a range of materials and interviewed a fascinating mix of people to verify these including members of the European Parliament, other civil society organisations or activists, ex-FBI agents and many others. We were able to verify 90% of our sample of outcomes. This allowed us to have a high confidence in both the change having happened and in the contribution of the Global Anti-Corruption Consortium to that change.
Another initiative where we’ve used outcomes harvesting is Yuwaya Ngarra-li – where we have used it as part of an embedded learning and evaluation approach alongside colleagues at the Dharriwaa Elders Group (DEG) and University of New South Wales (UNSW). Outcomes harvesting has not only helped to understand the contribution of Yuwaya Ngarra-li to place-based and systemic outcomes (you can see a summary of 4 years of outcomes in this report) but also provided real insight to the team around their practice.
Yuwaya Ngarra-li's approach to research and evaluation
As detailed in a Yuwaya Ngarra-li Briefing Paper Processes to enable community-led, systemic and collaborative change, early on in our use of outcomes harvesting we saw that around 45% of outcomes – all contributing to long term goals set by the Elders Council of the DEG – were from responsive, not planned, work. These outcomes emerged from responding directly to the community, at times of crisis or to political moments. This insight helped to shift mindsets around how to manage workloads and flows – adjusting what was “planned” and accepting more clearly the value of the responsive work of the organisation and initiative.
A powerful part of how Yuwaya Ngarra-li has adopted outcomes harvesting is that it sits alongside a number of other important evaluation approaches – including community-based research, linked administrative data, reflective practice and case study development. This mixed method approach critically enables the initiative to triangulate data and insights on an ongoing basis.
As part of Yuwaya Ngarra-li, we are seeing the benefit of outcomes harvesting in helping to be more attentive to signs of progress.
It helps you be alive to the shifts that are happening around you that you may not otherwise count as progress (but undoubtedly are) and helps you understand and position your contribution amongst others’.
For my colleagues working at Dharriwaa Elders Group in Walgett, who are dealing with the compound and long-term crises of colonisation, trauma, over-policing, destruction of country and culture, there is some hope to be gained from seeing their yearly impact in this way – a record of improvements to peoples’ lives, of preventing changes that could harm country further, and of growing the respect and influence of elders to realise their bigger vision for change.