Many concepts and strategies from different sources are used in academic areas ofprogramme evaluation. In this section, we will highlight some of them and make them tangible through tasks and exercises.

CIPP Model

The Stufflebeam CIPP model is a very useful way to organise the evaluation of programs.

All what we have to say in the CIPP model about Context, Input, Process, Product, actions, goals and outcomes is based on the Core Values of our educational organisation and comes back to it.

It is the assignment for the management and the board of an educational organization to formulate the core values together with the team and stakeholders. For INTENSE, the core values can be found in the Charter, the training programme for Transition Coaches, discussions with stakeholders and the Transition Plan for the organization. See below.

To get more insight into the application of the model, we recommend to work through the following article carefully:

http://openjournals.libs.uga.edu/index.php/jheoe/article/download/628/482

«  Exercise!

Find more information about the CIPP model on the Internet. Create your own short summary of the CIPP model.

Model of the Contribution Analysis

 “Questions of cause and effect are critical to assess the performance. When it is not practical to develop a project to assess the performance, contribution analysis can provide credible assessments of cause and effect. Verifying the theory of change that the programme is based on and paying attention to other factors that may influence the outcomes, provides reasonable evidence about the contribution being made by the programme.”

John Mayne

Contribution Analysis: An approach to exploring cause and effect. 2008

To be clear: It is not intended to prove direct causality. It would be unlikely in the complex social world we work within for any intervention to be the only influence on particular outcomes. Therefore, we must formulate an attribution problem. To what extent are observed results due to programme activities?

Contribution analysis is a structured approach to understand to what extent observed outcomes are a consequence of a particular activity, as opposed to other factors.  It is not intended to prove direct causality. It would be unlikely in the complex social world we work within for any intervention to be the only influence on particular outcomes.

Contribution analysis focuses on increasing understanding of why the observed results have occurred, the roles played by the activity, and other internal and external factors. By gathering and providing evidence around a line of reasoning (a Theory of Change), a reasonable conclusion can be drawn. There are six iterative (repeating) steps in Contribution analysis. Each step building the contribution story and addressing weaknesses in the previous stage. See below.

«  Step 1. Set out the attribution problem to be addressed.

This step is necessary for formulating all the different questions.

• Acknowledge the attribution problem. The fundamental attribution error explains why we often judge others harshly while letting ourselves off the hook at the same time by rationalizing our own (unethical) behaviour. In Contribution Analysis one must be always aware of the attribution error.
• Determine the specific cause-effect question being addressed.
• Determine the level of confidence required.
• Explore the type of contribution expected
• Determine the other key influencing factors
• Assess the plausibility of the expected contribution in relation to the size of the programme

«  Step 2. Develop the Theory of Change. (ToC)

This step together with the Results Chain are the key tools of Contribution analysis. The ToC and the Results chain are directly related to each other.

Every intervention by means of a project (INTENSE) is based on assumptions about what the project (by interventions) will bring about. So, a Theory of Change is an explanation of why you believe your project will lead to positive outcomes. When an organization wants to create a Theory of Change one must think about a few thinks:

1. What are the assumptions in realising the goals of INTENSE?
2. What are the risks?

Representing a ToC:

• Determine the level of detail
• Determine the expected contribution of the programme
• List the assumptions underlying the ToC
• Include consideration of other factors that may influence outcomes
• Determine how much the ToC is contested

3. How to realise a Results chain in relation to the ToC?
4. Formulate the Results chain in terms of inputs – activities – outputs- (intermediate) outcomes and at the end impacts.

«  Step 3: Gather the existing evidence on the Theory of Change.

Assess the logic of the links in the ToC. Reviewing the strengths and weaknesses of the logic, the plausibility of the various assumptions in the theory and the extent to which they are contested, will give a good indication of where concrete evidence is most needed. Gather the evidence. Evidence to validate the theory of change is needed in three areas: observed results, assumptions about the theory of change, and other influencing factors.

• Evidence on results and activities
• Evidence on assumptions
• Evidence on other influencing factors

«  Step 4: Assemble and assess the contribution story, and challenges to it.

The contribution story, as developed so far, can now be assembled and assessed critically. There are questions to be asked at this stage. Some significant examples:

• How credible is the story overall?
• Which links in the results chain are strong?
• Do stakeholders agree with the story?
• Is it clear what results have been achieved?
• Are key assumptions validated?

«  Step 5. Seek out additional evidence.

• Identify what new data is needed.
• Adjust the ToC.
• Gather more evidence.

«  Step 6. Revise and strengthen the Contribution story.

• New evidence will build a more credible Contribution story.
• Supporting the weaker parts of the earlier version or suggesting modifications to the ToC.
• It is unlikely that the revised story will be fool proof, but it will be stronger and more credible.

Contribution analysis works best as a repeating process. Thus, at this point the analysis may return to Step 4 and reassess the strengths and weaknesses of the contribution story.

The following graphic provides a differentiated insight into the ToC of the project INTENSE. It is a central component of the evaluation design and therefore Basis ToC for all individually developed ToCs of the partners.

 

Source: University of Malaga

In order to gather information and data, the following instruments can be used, depending on the partner who chooses them:

• Official statistics
• Official reports
• Interviews
• Questionnaire or Survey
• Scales and Graphs
• Tests
• Observation: self-report, diaries, checklist.

For more input please read the following documents:

• https://www2.gov.scot/Resource/Doc/175356/0116687.pdf
• https://www.betterevaluation.org/sites/default/files/ILAC_Brief16_Contribution_Analysis.pdf

OECD Criteria

Another important element is the criteria used to guide the evaluation. The department DAC-OECD has developed a set of criteria that are proposed as activities. Please read the evaluation criteria of the DAC-OECD.

https://www.oecd.org/development/evaluation/qualitystandards.pdf

Theory of change

“Every intervention in an organization is based on assumptions that together form a theory. The Theory of Change approach tests the theory of performers – because nothing is as practical as a good theory.” (Erik Snel, 2013)

The result chain shown below visualizes the serial connection behind Theory of Change (ToC), which according to Wikipedia

“… explains the process of change by outlining causal linkages in an initiative, i.e., its shorter-term, intermediate, and longer-term outcomes. The identified changes are mapped – as the “outcomes pathway” – showing each outcome in logical relationship to all the others, as well as chronological flow. The links between outcomes are explained by “rationales” or statements of why one outcome is thought to be a prerequisite for another”.

The ToC also describes in parallel the course of the implementation process. The plan you can create from this is, so to speak, the guide to successful implementation. It also helps to change the way people think in an organization and establish a collaborative process. As you know, nothing changes without rethinking that takes place in people.

A successful implementation process depends on many factors, such as appropriate training, theoretical knowledge, management support and above all, a clear commitment to change. If this commitment does not exist, it is almost impossible to implement INTENSE as an overall concept or in parts.

For more input please watch the following video: https://youtu.be/KRptX_DNL2Q

«  Exercise

…by answering the following question briefly. In your opinion: what can the role of the Theory of Change in your work as a coach?

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License

The European Commission support for the production of this publication does not constitute an endorsement of the contents which reflects the views only of the authors, and the Commission cannot be held responsible for any use which may be made of the information contained therein.