What do we (not) know about the benefits of households' electrification?

Household electrification has recently become a controversial topic. When The Economist declared that “electricity does not change poor lives as much as was thought,” a number of non-profits and industry representatives wrote indignant responses (CEO of SolarAid, CrossBoundary) arguing that electricity, including distributed solar power, can be a game-changer for rural communities.

Considering these diverging opinions, this blog post reports early findings from a systematic review that the Initiative for Sustainable Energy Policy (ISEP) has recently completed. The study reviews academic research on the benefits of household electrification on the following key outcomes: energy expenditures, household income, household savings, business creation, and education.

The study only includes research that conducts statistical tests and reports confidence intervals around estimated effects. It focuses only on household electrification, excluding community and industrial uses. While this kind of precision may be a little dull for some readers, it is absolutely necessary if we want to uncover the facts. The last thing a serious debate on a key development issue needs is inflated claims based on vague definitions and shallow research.

To foreshadow, we find that The Economist had it partially right. The body of evidence does not support the claim that there is a straightforward causal effect of household electricity access on socio-economic development. And yet, we also find that in the right circumstances, household electrification can be beneficial. As is almost always the case with the real world, the issues are complicated and the only thing we really know is that we do not know much at all.

But before discussing these teaser results in any depth, I am going to do something far more important by explaining what exactly it means to evaluate the benefits of household electrification.

Impact evaluation is hard

It would be great if we could just compare households with electricity to those without, or compare households that recently obtained electricity to their former selves. While these simple approaches remain popular in the development consulting business, they are unreliable and misleading.

If we compare households with electricity to those without, we can establish that they are different. However, we cannot say with any certainty why that is. It could have something to do with education, wealth, attitudes, religion, geography, or diet. It could have something to do with an infinite number of differences between the households.

If we compare households with electricity to a time before electrification, we can again easily establish a difference, but we cannot attribute it to electricity. Any number of things are changing over time, and not just electricity access.

For these reasons, comparisons across households or before-after electrification are not useful for impact evaluation. Instead, we need to ask a far more difficult question: if a household without electricity had access to it, all else constant, what would the result be?

Randomised controlled trials to the rescue?

One popular approach to dealing with these issues is the randomised controlled trial (RCT). Here researchers randomise access to electricity, so that the “treatment” and “control” groups are identical except for electricity access. This approach removes the selection and omitted variable bias problems by design, which is why some economists argue that the method is a “gold standard” for impact evaluation.

Unfortunately, the RCT does not fully solve the problem. While it does offer correct estimates, a number of challenges remain:

• It is far easier to randomise small interventions, such as distributing solar lanterns to households, than large programmes, such as village electrification.
• Measuring outcomes over long periods of time requires a lot of patience. If a researcher starts an RCT today, results on the 10-year impact of household electrification will be available around the year 2030.

Suppose we go ahead and conduct an impressive number of excellent studies. If these studies focus on distributed solar power and short periods of time, as I and my collaborators have done in rural India, we learn about the benefits of solar power in the short-run. However, these lessons may not generalise to grid extension or to longer periods of time. The RCT, then, is no panacea. It is immensely helpful, but it does not solve the problem entirely.

Many researchers continue to develop and use quasi-experimental methods, such as regression discontinuities, instrumental variables, or difference-in-difference estimators. They mimic the randomisation process and can often be applied to existing data. But quasi-experimental methods require a lot of assumptions about the data generation process, and these assumptions can never be fully tested.

To summarise, impact evaluation is really hard. We can learn over time if we are very disciplined and continue to accumulate high-quality studies for systematic review and meta-analysis. When we have hundreds of excellent studies in our hands, published in refereed journals and reflecting a diverse set of conditions, we can perhaps say something definitive. However, as I will now show, we are far from that point.

The results of ISEP’s systematic review

In our systematic review, we identified 30 studies that focused on our outcomes of interest at the household level. Together, they reported 64 statistical estimates of impact. Of these, 47 are positive and the rest either neutral (16) or negative (1).

These numbers suggest that household electrification could be highly beneficial. 47 out of 64 is not a slam-dunk, but it is quite an impressive number. If three-quarters of estimates are positive, it seems likely that household electrification is a good thing.

Unfortunately, the reality is a little more complicated:

• RCT evaluations produce only 15 positive effects out of 25 (60%).
• Difference-in-difference evaluations, where (i) changes among households that gain access to electricity are compared to (ii) changes among households that never gain access to electricity, produce 12 positive effects out of 16 (75%).
• Other methods produce 20 positive effects out of 23 (87%).
• Positive effects are clearer for outcomes such as reduced kerosene expenditure than for broader development outcomes, such as income or education.

These differences in outcomes and methods are a clear sign that we are missing something. RCTs find fewer positive outcomes than other researchers by a wide margin. While energy access seems to improve consistently, effects on broader development outcomes are less clear.

Policy and research recommendations

Governments continue to electrify households to win elections. Funders will continue to fund interventions that produce compelling human interest stories. Industry associations and the development-industrial non-profit complex continue to scream murder whenever they are challenged. The Economist will continue to signal centrist virtue by writing sceptical pieces, so that industry associations and the development-industrial non-profit complex can continue to scream murder.

This scepticism aside, here are my suggestions:

1. Governments, development agencies, and foundations should fund impact evaluations on a large scale and over long periods of time. It is deeply disturbing that billions of dollars of public money are invested every year in household electrification without an aggressive impact evaluation push.

2. Everyone should acknowledge the deep uncertainties surrounding the benefits of household electrification. While it is obvious that electricity is necessary for an economy to grow over time, the value of household electrification is uncertain.

4. Anyone proposing a household electrification intervention on a large scale should commit to an unbiased and neutral impact evaluation using appropriate methods.
5. Nobody should ever make grandiose claims about (the lack of) benefits of household electrification based on a handful of studies. Conduct rigorous systematic review, and ruthlessly exclude any and all studies that fail to meet the stringent criteria for rigorous social science.