How can we use the results of an impact evaluation?

When we obtain the results of an impact evaluation for a given program, we use it to verify compliance with the goal set by the policy or plan being evaluated.

It is the input used to estimate the costs and benefits associated with the program, and what will allow us to compare it with other policy alternatives that have the same goal.

In other words, we can compare evaluations to achieve the best policy in terms of greater benefits and lower costs.

This helps us to review its design and thus improve the use of scarce resources invested in the program or to improve the management of the program.

This makes management more efficient; that is, with the same resources or budget we obtain better results.

Redesigns must be perceived as an improvement of the program or project that was evaluated. It serves to verify the effectiveness of the investment in terms of money and efforts made within the policy or plan.

The amount of resources invested does not always guarantee a positive, optimal or expected outcome.

We can use these results to argue why it is relevant to invest more resources (from government, NGO, private sector with an interest in social responsibility projects, or multilateral projects) in our policy or plan.

We can redefine the program by anticipating strategic behaviors (incentives) that the policy creates for both beneficiaries and non-beneficiaries.

We can use them as a very comprehensive accountability mechanism. Even more comprehensive than the follow-up and monitoring indicators traditionally used. For the latter, we are unable to know or explain exactly what explains their evolution.

We must understand administrative data as a repository of information on the functioning of public or private policy. They serve us to follow up on these policies.

In general, it is advisable that our indicator be SMART (Specific, Measurable, Achievable, Relevant, Time-bound).

Let's see if the School Attendance indicator meets the characteristics of a SMART indicator:

Specific: attendance refers to the specific action of attending or not attending classes.

Measurable: it is possible to determine on a daily basis how many children attend school.

Achievable: it is relatively easy to measure. It is sufficient to take a daily list by classroom, for example, of who attends and who does not.

Relevant: with attendance we measure directly an indicator that is relevant to evaluate the school's performance.

Time-bound: we can calculate the daily attendance or number of days in a month, semester or year. Calculate a percentage of attendance for a given period of time.

Now, knowing the results of this impact evaluation, we can think about what this evaluation helps us to understand based on the points mentioned above.

The program has no effects on the school performance of laptop beneficiaries.

This undermines the initial idea we had about the positive impact that laptops could have on learning.

But it does have impacts on cognitive tests.

There are three important conditions to be taken into account that may explain the outcome:

1) The program was implemented in remote rural areas of Peru where there may be limited access to electricity or internet service.

2) The laptop did NOT include software designed to improve skills in the areas of math or language.

3) There was no major link between the use of the laptop and the classroom curriculum.

The conditions mentioned above can tell us why the outcomes differ from what was expected.

The program could be redesigned taking into account the three previous points.

This could comprise the following actions:

a) Verify connectivity of households of beneficiary children.

b) Include teachers’ training to incorporate the use of laptops in the classroom content as well as useful programs for this purpose.

c) Advise parents not to neglect the supervision or care of their children, since children may use the laptop to play games instead of spending time studying.

It cannot be assumed that laptop use will be related only to school activities.

There could be a positive externality effect if, for example, the laptop is used by other members of the household and this has an impact on labor participation.

To better understand this first definition of impact evaluation, we can use an example:

“We want to assess the impact of the induction received by workers on their first day of work on their productivity.

Specifically, we are going to work with a company that collects letters from mailboxes.

…the productivity of the postman is measured by the number of letters he/she collects or the number of mailboxes checked in a day.

The goal of an induction is to familiarize the worker with procedures, schedules and routes.

In principle we would expect that a postman who attends the induction will have better tools to collect as many letters as possible from their first days of work.

But this is something we BELIEVE SHOULD HAPPEN… AND WE SHOULD CHECK IF WE ARE RIGHT.

What we achieve through the evaluation of the induction on the first day of work of the postmen (program) is to know how many letters a postman who receives the induction will collect on his/her first day of work (the beneficiary outcome) with respect to the number of letters that this same postman would collect if he/she had not received the induction (the counterfactual outcome).

For example, let's consider that the postman who received induction can collect 80 letters per day (10 letters per 8 hours of work), whereas if the postman had not received induction, he/she would have collected 64 letters per day (8 letters per 8 hours of work).

This indicates that the effect of the induction on the productivity of the postman is 16 letters per day. This difference measures exactly what impact the induction has on postmen productivity.”

This example is useful to guide the letter collection company in quantifying: The benefits it receives from the induction / whether this induction program proves to be cost effective for the company.

In other words, based on the outcome of the evaluation, a cost-benefit analysis of the induction can be made.

Postmen who receive induction collect 10 letters per hour and postmen who do not receive induction collect 8 letters per hour.

This means that induction causes postmen to collect 2 additional letters per hour.

Postmen with induction are 25% more productive in one hour of work.

If the company needs to collect 40 letters per hour throughout the city, this would be done by 4 postmen with induction (4*10=40) or 5 postmen without induction (5*8=40).

The company needs to assess whether it is more profitable to invest in induction (pay the cost of induction for the 4 postmen) or to hire five postmen without providing induction.

This would be the way to find out which of the two policies is the most profitable or beneficial for the company.

Dengue podcast 2

Dengue podcast 3

Dengue podcast 1

FF3E - Curso B - Mercado de destino Europa - Docente Josep Bertran

DGPA - Módulo Salud - Podcast 1

Educación financiera Módulo 2 apartado 6 sección 3  item a - La formación: el desarrollo profesional de los docentes