Can drawing on preliminary findings boost the impact of evidence on policymaking?

The long duration of much academic research reduces its potential to influence time-sensitive policy decisions. Can this research-policy gap be closed by delivering initial results to the policy-maker early in the research cycle?  Existing evidence and new theory say yes, but only if current incentives in the profession are restructured.

Policymakers constantly bemoan researchers’ inability to communicate their findings in time for policy decisions. Researchers aim for excellence, devoting as much time to data collection, analysis and peer review as necessary to meet the profession’s exacting standards.  The result is a tension between policymakers’ desire for timeliness and researchers’ striving for rigour. How is it to be resolved?

Many case studies in the field of economic development show that positive research-policy outcomes shared two characteristics. First, researchers presented their findings in short, policy-focused notes. Second, they initiated a dialogue with the policymaker early in the research cycle.

Vendors at Tagrin Ferry Terminal

Building on this evidence of association, we have called, in a forthcoming article in the World Bank Economic Review, for the introduction of provisional assessments—preliminary findings delivered to policymakers not only to provide information, but also to start an exchange—, with accompanying changes in professional incentives to encourage researchers to produce them.

Drawing on comparable experiences can provide reliable guidance in policy decisions

To give an example, consider the proposal to offer remedial classes for children who had failed the Ghanaian Basic Education Certificate Examination. Would the pass rate among those re-sitting it improve sufficiently to justify the investment?  The researchers had access to accurate information on the number of affected children and the cost per pupil, but not on the expected increase in the pass rate.  Instead of conducting a randomised controlled trial, the so-called gold standard for measuring impact, whose implementation can take months, if not years, to complete, the researchers drew on experience with similar programmes in other countries.  By projecting increased earnings for certificate holders from current Ghanaian wages, the researchers had sufficient information to complete a preliminary cost-benefit analysis and produce a provisional assessment.

Three characteristics of a strong provisional assessment:

This example shows the three characteristics of an effective provisional assessment: a well-specified policy issue; reliance on extant, available information, with guesstimates or assumptions to fill in the blanks; and use of standard techniques.

Robustness arises as a central issue. In the Ghanaian example, the provisional assessment predicted a net benefit for a wide range of estimates of impact, so a positive recommendation was safe. In many cases, however, a provisional assessment will lead to the conclusion that more data and analysis are required before a recommendation can be made with any confidence.

An exchange of information, however incomplete or indirect, may improve the chances that the decision, when it does come, will be the right one, in the sense that the chosen course of action is welfare-superior when viewed ex ante at the time of decision.

When this occurs, the provisional assessment must set out clearly what additional research is necessary. This both protects the researcher’s professional integrity and, most importantly, opens a dialogue with the policymaker aimed at agreeing on a timetable that ensures that the findings of additional research are on hand prior to any decision.

Provisional assessments, even with incomplete information may prove superior to traditional wait-and-see approaches

In our paper, we develop a theoretical model of the researcher-policymaker interaction that captures these tensions, hazards and opportunities. A project or programme is under consideration. Both the exact timing of the simple ‘yes-no’ decision of whether to adopt it and its social profitability are uncertain. There are two actors, a policy-maker (D) and a researcher (R). As ‘insiders’ in their respective fields, D is better informed about timing, R about social profitability, but neither is perfectly so. An exchange of information, however incomplete or indirect, may improve the chances that the decision, when it does come, will be the right one, in the sense that the chosen course of action is welfare-superior when viewed ex ante at the time of decision.

While preparing her analysis, R knows that she may submit her advice too late. This risk must be weighed against the improvement in quality that would result from taking longer over it. Now suppose she can test the waters by submitting a provisional assessment, accompanied by the offer of a more definitive analysis to follow if required. This provisional assessment also goads D into playing an active role in the ‘game’ between them. We prove that the resulting option has a positive value, relative to a single submission, in the sense of increasing the probability that D will decide correctly. A system based on provisional assessments will do better, on average, than the traditional approach of waiting until the research is complete before the findings are shared with policymakers, by which time many of the decisions may already have been taken.

Yet some scope for error – and improvement – remains. An ambitious D may use a provisional assessment, even if it calls for more research, to push through a pet project. Pet project or no, had R completed the study and then submitted a recommendation, D might have been persuaded to decide otherwise. That some D’s are more open to expert advice than others is, moreover, a fact of life. Local R’s are surely better informed as to who is amenable, an advantage that should be exploited.

Rejigging incentives

If both case studies and theory confirm the superiority of provisional assessments, why are they not in widespread use?   The answer lies in incentives.  In economics, and probably most other disciplines, high-quality research published in peer-reviewed journals is the measuring rod of success.  Provisional assessments, however, are unlikely to be accepted for publication in today’s journals, even applied ones.  Prevailing incentives, then, encourage timeliness only to the extent that researchers want to get their results into print.

The obvious solution is to subsidise provisional assessments.  We suggest a two-part payment – the first covering the relatively modest cost of producing the assessment, the second paid after delivery as an added incentive. Governments and donors should be willing funders, since provisional assessments realise one of their most sought-after objectives – policy impact – at low cost.  Additional funding for any mutually agreed, follow-up research should be especially attractive; for thorough research and analysis would now inform policy in good time.

Not all economists will like this idea.  Many will prefer to tackle the vital task of contributing to our basic understanding of development.  Yet many others are keen to have a more immediate policy impact: for them, funding for provisional assessments will open a door through which they will gladly march.  If this happens, the economics profession at large should see this as a valuable initiative and seek ways of supporting provisional assessments through, for example, quick-turnaround reviews and journals specialising in such contributions.