in content lists or on the content page.
More and more often we face challenges whose outcomes extend beyond a single organisation, field of science, sector or state, thus requiring a multidisciplinary approach and different practices and mechanisms. These wicked societal problems call for systemic change. Innovations aimed at systemic change are often generated in networks of different actions, actors and environmental factors and as a combined effect of them, which makes them complex.
Efforts are being made to influence complex phenomena by various means, including by us here at Sitra. But how can we evaluate the effectiveness of measures aimed at generating systemic change? What tools do we have for managing desired impact?
Basically, impact evaluation is interested in cause-and-effect relationships, or causality. The key questions of impact evaluation concern how and to what extent the activity being studied has produced outcomes and impact.
The purpose of impact evaluation is to verify and test, as well as to specify and complement, theories of impact based on systematic analysis.
“The more complicated the object of evaluation becomes, the more challenging it is to verify the relationship between cause and effect.”
The suitability of approaches and methods used for impact evaluation depends on the nature of the object and context of evaluation as well as on what information is needed. In simple interventions, the connection between cause and effect is direct and causality is linear. If there is high-quality data available, it is possible to reliably verify outcomes and impact. The more complicated the object of evaluation becomes, the more challenging it is to verify the relationship between cause and effect. As regards methods, the need for systems thinking and qualitative data becomes more crucial when dynamics of a system is complex (see e.g Hargreaves 2010; Preskill et al. 2014)
Adaptability vs predictability
Metaphorically, we can say that complex systems do not resemble machines as much as they do constantly changing organisms. The cause-and-effect relationships can be unpredictable and have effects in various ways. Their identification is also made difficult by the long-time span of such changes. In their evaluation, it is more essential to understand their logic of impact than to use a predefined metering system.
When operations have complex features, in impact evaluation it is necessary to also identify unpredictable and unintended impacts. Simple interventions can be described as a logically advancing model, summed up using the sentence construction “if x then y”. However, when it comes to innovative, searching and experimental (explorative) activity the question to be asked in evaluation is rather “What if?”.
“A clear goal and vision is the essence of everything.”
Even though the pathways to impact may be blurred, crossed overlapping and unpredictable, and found through doing and trying, the impact goal, and the kind of societal impact sought, may be clear. In fact, when influencing complicated or complex systems, the outlining, description and management of the intended impact and vision for the future becomes even more important.
Theory-based evaluation
Theory-based evaluation has been regarded as a well-suited approach for interventions concerning operating environments with complex features. In theory-based evaluation, the starting point consists of different theories of change that describe how impact is supposed to be generated.
There are various tools for outlining and describing theories of change, ranging from diagrams to stories. The advantage of definitions of impact and impact stories is that they depict assumptions of impact in versatile ways. For example, the societal impact of research is principally a complicated phenomenon, which can be described with the help of impact case studies stories, (see also examples from the UK). Approaches such as tracking of processes and contribution analysis can be used as methods of impact evaluation.
How do we describe Sitra’s impact?
By nature, Sitra’s impact goals are long-term strategic objectives. They depict the last step of the input-output-outcome-impact results chain and define a shared direction of what kind of a society we want to build.
Our approach to impact evaluation is guided by the nature of our operations and the need for knowledge that supports the activities aimed at societal impact. We consider it essential to outline and increase our understanding of how our future-oriented work promotes our impact objectives and how we could enhance that work further. External impact evaluation produces independent information that supports us in reaching this goal.
“We want to better understand how our future-oriented work promotes our impact objectives.”
We also consider it important that we ourselves outline and define impact chains. With these definitions, we want to structure assumptions of impact and theories of change that steer our work, and to encourage others to express and define their pathways towards societal impact.
The definitions of impact (or “impact claims”) we have produced focus particularly on the last step of the results chain – from outcomes to impact. Definitions of impact have laid foundations for theory-based evaluation that external evaluators have produced for us. The definitions have also helped us in our internal evaluation and development work.
It is important to note that the defined projects may also have other outcomes and impact which may not have been taken into account in these definitions. Therefore, from the viewpoint of a specific project, the definitions are not comprehensive and do not take account of every viewpoint, but they make explicit a project’s contribution to the organisation’s shared goals for societal impact.
Recommended
Have some more.