Cursos virtuales CAF

Economia ambiental y finanzas sostenibles-Modulo 3 -Podcast 5

Economia ambiental y finanzas sostenibles-Modulo 3 -Podcast 4

Economia ambiental y finanzas sostenibles-Modulo 3 -Podcast 3

Economia ambiental y finanzas sostenibles-Modulo 3 -Podcast 2

Economia ambiental y finanzas sostenibles-Modulo 3 -Podcast 1

In practice, the theory of change process might depend on whether a clearly defined intervention already exists, or if we start from a problem that we have identified, and we seek to define a program (or intervention) to solve it.

Actually, these are two sides of the same coin; the crucial part is to recognize our starting point.

In the rural, scattered areas of Latin America, the students’ academic performance tends to be lower than in more urban areas.

This can be observed clearly in the results of standardized tests. It is a common problem faced by our countries and, as expected, this gap leads to a persistent inequality in future employment, income and quality of life opportunities available for people who are born and grow up in rural areas.

A policy that might be used to attempt to reduce the academic performance gap would be attracting better qualified teachers to rural areas in order to increase the quality of education.

For example, monetary incentives could be used to encourage a teacher to work in a rural area for a certain period of time in exchange for a monthly bonus equal to 15% of their basic salary.

However, several interesting questions arise when one thinks about the possible impact of this policy:

To what extent academic performance depends on teacher quality? Are there other determinants that complement teacher quality, and which should be addressed? They might include an adequate learning infrastructure, access to technology and to information, among others.

What type of teachers will we attract with a 15% bonus? Highly qualified teachers? Teachers of intermediate or low-quality skills?

Using a monetary incentive, would we attract those who are better prepared? Or those who have less (and worse) alternatives in urban areas? This will depend on how the incentive positions are filled. Is the applicant’s teaching quality and/or experience assessed?

Is it enough to offer a monetary incentive to attract teachers, or do we need to modify other characteristics of the rural area to make it more attractive?

Is a 15% bonus enough? Or for how long should an additional incentive be offered? The bonus should exceed the costs arising from the teacher’s relocation to a rural area.

What if, instead of attracting better teachers, we create training programs for teachers who are already in rural areas?

When implementing the monetary incentive policy to attract better teachers, we have to take all these questions into account (and several others).

What is it for?

Several public and private institutions implement programs to improve society’s welfare. Some of these programs are effective and achieve their goals, but it is not uncommon for some projects to fail at accomplishing the expected outcomes.

Therefore, the theory of change allows us to:

Understand what the specific need to be addressed through a certain program is (a specific problem to be solved must be identified).

Streamline ideas about the policy’s components, and how they will be combined to reach the expected outcomes.

Set medium- and long-term intermediate outcomes and develop their corresponding indicators.

Enable the development of multiple impact indicators, both in the short- and long-term.

Focus on just one intervention with specific outputs.

Identify the channels whereby we seek to impact the intermediates outcomes and impact indicators.

Ensure knowledge and understanding of the requirements to implement an effective policy and, therefore, to achieve the desired change.

What questions does the Theory of Change respond?

Through the theory of change we can verify if a program’s design is appropriate, if the number of provided outputs accomplishes a specific intermediate outcome, and, therefore, if a final impact is achieved.

It also contributes to identifying problem determinants and how these are connected throughout the entire causal chain, and even to determining, to a certain extent, the effectiveness of an action.

The theory of change allows us to recognize:

If the different actions, interventions or activities carried out within the programcan help us meet the population’s need.

What are the assumptions and conditions to ensure that the intervention will have an effect on intermediate outcomes and the impacts sought.

Recognizes immediate outcomes, mechanisms or channels whereby the program would be expected to work and translate into improved well-being.

Identifies the expected final outcomes to improve the population’s well-being.

Recognizes what the different determinants of a problem are.

What potential actions can be taken to meet the identified need.

Only an impact evaluation can verify if the pathway proposed within the theory of change or value chain is met, and in what magnitude.

One question that remains to be answered is: How do we estimate or quantify the impact of a randomized experiment?

We have given details about the conditions the experiment must meet in terms of the selection of the treatment and control groups. We also know that the assignment between groups must be random in order to have "identical" groups before the intervention.

But all this ... how do we use it?

We want the assess a program by analyzing its impact on an outcome indicator. If we go back to our previous example, the outcome indicator would be height, and the program would be the school meals program.

We then estimate the impact by taking the average outcome of the group that received the treatment, minus the average outcome of the group that did not receive the treatment.

The latter group tells us how much a child would have grown without the treatment and additionally takes into account special conditions that occur at the time of the intervention (all other factors that we must isolate), which is why we remove this growth in order to find the impact attributable exclusively to the program (or treatment).

If we use graph A as a given result, we find that both children in the treatment group and the control group grew between 2016 and 2018.

As mentioned previously, the beneficiary children grew 10 cm on average. This is because they started with an average height of 120 cm in 2016, and after two years, in 2018, they reached 130 cm. On the other hand, the children in the control group grew 8 centimeters on average, because they started with an average height of 120 cm (equal to that of the children in the treatment group), and two years later their average height was 128 cm.

Therefore, we may conclude that in the absence of the program the growth would be 8 cm and with the school meals program the growth would be 10 cm, which means that the program has a positive impact of 2 cm on average.

Ways to randomize

Flipping a coin, as we have already seen, can randomly assign individuals to two groups.

A lottery where all the names of potential participants (those individuals who will be part of the treatment group and the control group) are put into a bag. The names of the first number of people to be part of the treatment are drawn. If, for example, there are 200 potential participants, the first 100 names are assigned to the treatment group and all those remaining in the bag are assigned to the control group.

Another way, if we want to select half of the participants for the treatment group and the other half for the control group, is to determine that people with IDs ending in an even number will participate in the treatment and odd numbers will not participate in the treatment. This selection mechanism is completely verifiable and is a condition independent of the individuals’ decision. In other words, there is no way to modify it in order to be included in the treatment or the control group.

Produce a random number (it can be done in Excel or in a statistical package), sort it and take one yes, one no. Or take the first half for one group and the second half for another group.

The positive aspect of these randomization methods is that they are completely transparent and verifiable for any individual who is present during randomization or who wants to verify it in the case of identification documents or the random number produced by a computer.

Being able to verify randomization is extremely important, as it is proof that the evaluator or policy implementer does not want to or has no interest in favoring or disfavoring any individual by giving him/her the program or treatment.

Additionally, any of these randomization methods has the benefit that the selection is not related to the outcome variable. For example, the first letter of your name does not make you grow faster or slower, nor does the last number of your ID, and so on.

There may be cases where we want to randomize into more than two groups. In this scenario we must find a way to randomize correctly. A coin would not be useful in this situation. Let's think, for example, that we want to randomize into six groups, in which case a dice would be ideal to perform a random assignment.

Why a control group is necessary and what are the conditions it must meet?

We cannot assess the impact of a program by looking only at the outcome of individuals benefited ... we will understand why!

Let’s recall our impact evaluation question: How does the provision of school meals impact the height of beneficiary children?

In the following figure we have the outcome (height) of the children who participated in the school meals program during 2017, the period identified by the green line. The figure shows us that before the program, in 2016, the beneficiary children had an average height of 120 cm. After the program, in 2018, the beneficiary children had an average height of 130 cm.

Given this outcome, could we say that the impact of the school meals program on the height of beneficiary children is 10 centimeters on average per year of intervention?

The answer is NO... because we would expect children to grow up over time anyway, even in the absence of the school meals program.

Let's think: what other factors can modify the normal development of children's height?

Eating habits

Health condition

Physical activity

Between 2016 and 2018 (before and after the intervention), two things happen:

1) School meals program +

2) Other factors that may in turn modify the normal course of the outcome variable (child height).

In the comparison we make in the graph, we do not know whether we are measuring (1) or we are measuring (2), or to what extent both factors are combined.

Therefore, we need to know whether the growth that children experience when they participate in the program is greater than, less than, or equal to what they would experience without the program, the counterfactual. In other words, to know the impact of the program we need to know what would have happened to the height of those children if they had not participated and compare that height with what we can actually observe in 2017.

Since we cannot know what would have happened without the meals program (because the children have already benefited from it) we need a control group (or comparison group). That is, another group of children that has not benefited from the program and that will help us to approximate the counterfactual that we cannot observe.

• What constitutes a good control group?

A group identical to the treatment group composed by non-beneficiaries.

In our example, the best control would be "literally" a group composed by the twin siblings of the children benefited by the school meals program. Since thetwins are not beneficiaries of the school meals program, they will be fed at home.

• The control group must meet the following conditions:

1) It is a group that shows us what would have happened to those treated had they not received the treatment, the counterfactual.

2) The effects of the program should be the same for both groups (treatment or control). This means that if the units in the control group are the ones that receive the treatment (instead of what actually happens: those in the treatment group are the beneficiaries), the impact would be exactly the same as the one we are going to measure.  In other words, the groups are similar to each other, and it is irrelevant which individuals specifically within each group receive the treatment.

3) External factors, which affect all children, should have the same effect on the comparison (control) group and the treatment group.