Evaluation Design
Intro
A design is an action plan for the steps to be performed in the evaluation to identify changes in the evaluated system. The evaluation can specify three scenarios that determine which type of design to use. The first two are based on the assumption that groups are comparable. The third is an alternative when comparison of groups are impossible:
First scenario: Randomised-controlled studies. These are designs that are executed when you have full control over the launch of the programme as well as the participants. This includes the control over programme completion times, the selection of participants, the allocation to different versions or modalities of the programme, the resources used by them, etc. In this case, the design is typical for the field of experimental research.
Second scenario: Not randomised group designs. If it is not possible to have complete control over the programme or participants, control over previous designs must be abandoned. These are situations in which you can develop evaluation plans that are similar in form and organization to the above-mentioned ones. However, as there is no complete control, the results should be considered with caution. In other words, although the design is like the first scenario, the inability to control all aspects of the process threatens the validity of the results. For this reason, the results mentioned should be evaluated rather critically and complementary ways must be found in order to achieve reliable results.
Third scenario: Coherence studies. It is a scenario in which there is no guarantee that there is control over the various and significant elements. These studies are specifically looking for alternatives that are independent of the comparison of measures between 2 or more groups.
These three scenarios can be organised using two basic approaches:
Counterfactual designs: They consist of designs that use factual groups. A factual or counterfactual group is the group that does not use the programme, but in characteristics exactly matches the group to which it is applied. Within these designs, repeat-controlled studies are most effective. In the differentiated application, parts can also be organised as non-random group designs, although they are less meaningful.
Non-counterfactual designs: These are those where you cannot rely on group-related facts and thus the comparison between groups is not possible. This kind of design includes coherence designs.
Causal attribution
One of the basic elements of the design is to get the causal attribution for the programme. To find out more, please read the following document: https://www.unicef-irc.org/publications/751-overview-strategies-for-causal-attribution-methodological-briefs-impact-evaluation.html
« Exercise!
- A programme to control aggressiveness is conducted in a detention centre with acceptable results. Nevertheless, it was decided to start another programme based on mutual mindfulness. In order to check the effectiveness of this new programme compared to the other one, a group of participants was selected as appropriate. Indicate which evaluation design can be used and justify your answer.
- An English teacher uses a novel method based on working with teachers from other subjects. For example, with the economics and social studies teacher, they develop the budget of a fictitious company in English. Imagine you have to register with the student group to collect relevant data. The course runs for a whole semester and you need the permission of the school administration, the teachers and the participating students. Based on this information, make a design proposal for the evaluation that takes into account that more than one student can be contacted to collect the data. Justify your suggestion briefly.
- Imagine you are working in the Immigration centre trying to adapt your working procedure to the recently approved regulations. Your perception is that the introduction of this scheme will result in a deterioration of users’ attention. In addition, the processing time seems to have increased considerably until the validation of the user documentation. Likewise, the number of users per social worker has increased by an average of 3 people. In view of this situation, indicate which evaluation design you could use to de facto underpin this situation and justify your proposal.
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License
The European Commission support for the production of this publication does not constitute an endorsement of the contents which reflects the views only of the authors, and the Commission cannot be held responsible for any use which may be made of the information contained therein.