Directed by
Ideas for growth
Menu
Directed by
Ideas for growth
Menu
Projects
Publications

Can drawing on preliminary findings boost the impact of evidence on policymaking?

The long duration of much academic research reduces its potential to influence time-sensitive policy decisions. Can this research-policy gap be closed by delivering initial results to the policy-maker early in the research cycle?  Existing evidence and new theory say yes, but only if current incentives in the profession are restructured.

Policymakers constantly bemoan researchers’ inability to communicate their findings in time for policy decisions. Researchers aim for excellence, devoting as much time to data collection, analysis and peer review as necessary to meet the profession’s exacting standards.  The result is a tension between policymakers’ desire for timeliness and researchers’ striving for rigour. How is it to be resolved?

Many case studies in the field of economic development show that positive research-policy outcomes shared two characteristics. First, researchers presented their findings in short, policy-focused notes. Second, they initiated a dialogue with the policymaker early in the research cycle.

Vendors at Tagrin Ferry Terminal

Vendors at Tagrin Ferry Terminal

Building on this evidence of association, we have called, in a forthcoming article in the World Bank Economic Review, for the introduction of provisional assessments—preliminary findings delivered to policymakers not only to provide information, but also to start an exchange—, with accompanying changes in professional incentives to encourage researchers to produce them.

Drawing on comparable experiences can provide reliable guidance in policy decisions

To give an example, consider the proposal to offer remedial classes for children who had failed the Ghanaian Basic Education Certificate Examination. Would the pass rate among those re-sitting it improve sufficiently to justify the investment?  The researchers had access to accurate information on the number of affected children and the cost per pupil, but not on the expected increase in the pass rate.  Instead of conducting a randomised controlled trial, the so-called gold standard for measuring impact, whose implementation can take months, if not years, to complete, the researchers drew on experience with similar programmes in other countries.  By projecting increased earnings for certificate holders from current Ghanaian wages, the researchers had sufficient information to complete a preliminary cost-benefit analysis and produce a provisional assessment.

Three characteristics of a strong provisional assessment:

This example shows the three characteristics of an effective provisional assessment: a well-specified policy issue; reliance on extant, available information, with guesstimates or assumptions to fill in the blanks; and use of standard techniques.

Robustness arises as a central issue. In the Ghanaian example, the provisional assessment predicted a net benefit for a wide range of estimates of impact, so a positive recommendation was safe. In many cases, however, a provisional assessment will lead to the conclusion that more data and analysis are required before a recommendation can be made with any confidence.

An exchange of information, however incomplete or indirect, may improve the chances that the decision, when it does come, will be the right one, in the sense that the chosen course of action is welfare-superior when viewed ex ante at the time of decision.

When this occurs, the provisional assessment must set out clearly what additional research is necessary. This both protects the researcher’s professional integrity and, most importantly, opens a dialogue with the policymaker aimed at agreeing on a timetable that ensures that the findings of additional research are on hand prior to any decision.

Provisional assessments, even with incomplete information may prove superior to traditional wait-and-see approaches

In our paper, we develop a theoretical model of the researcher-policymaker interaction that captures these tensions, hazards and opportunities. A project or programme is under consideration. Both the exact timing of the simple ‘yes-no’ decision of whether to adopt it and its social profitability are uncertain. There are two actors, a policy-maker (D) and a researcher (R). As ‘insiders’ in their respective fields, D is better informed about timing, R about social profitability, but neither is perfectly so. An exchange of information, however incomplete or indirect, may improve the chances that the decision, when it does come, will be the right one, in the sense that the chosen course of action is welfare-superior when viewed ex ante at the time of decision.

While preparing her analysis, R knows that she may submit her advice too late. This risk must be weighed against the improvement in quality that would result from taking longer over it. Now suppose she can test the waters by submitting a provisional assessment, accompanied by the offer of a more definitive analysis to follow if required. This provisional assessment also goads D into playing an active role in the ‘game’ between them. We prove that the resulting option has a positive value, relative to a single submission, in the sense of increasing the probability that D will decide correctly. A system based on provisional assessments will do better, on average, than the traditional approach of waiting until the research is complete before the findings are shared with policymakers, by which time many of the decisions may already have been taken.

Yet some scope for error – and improvement – remains. An ambitious D may use a provisional assessment, even if it calls for more research, to push through a pet project. Pet project or no, had R completed the study and then submitted a recommendation, D might have been persuaded to decide otherwise. That some D’s are more open to expert advice than others is, moreover, a fact of life. Local R’s are surely better informed as to who is amenable, an advantage that should be exploited.

Rejigging incentives

If both case studies and theory confirm the superiority of provisional assessments, why are they not in widespread use?   The answer lies in incentives.  In economics, and probably most other disciplines, high-quality research published in peer-reviewed journals is the measuring rod of success.  Provisional assessments, however, are unlikely to be accepted for publication in today’s journals, even applied ones.  Prevailing incentives, then, encourage timeliness only to the extent that researchers want to get their results into print.

The obvious solution is to subsidise provisional assessments.  We suggest a two-part payment – the first covering the relatively modest cost of producing the assessment, the second paid after delivery as an added incentive. Governments and donors should be willing funders, since provisional assessments realise one of their most sought-after objectives – policy impact – at low cost.  Additional funding for any mutually agreed, follow-up research should be especially attractive; for thorough research and analysis would now inform policy in good time.

Not all economists will like this idea.  Many will prefer to tackle the vital task of contributing to our basic understanding of development.  Yet many others are keen to have a more immediate policy impact: for them, funding for provisional assessments will open a door through which they will gladly march.  If this happens, the economics profession at large should see this as a valuable initiative and seek ways of supporting provisional assessments through, for example, quick-turnaround reviews and journals specialising in such contributions.

Comments

Leave a Reply

Comments will be held for moderation. Your email address will not be published.

Time limit is exhausted. Please reload CAPTCHA.

  • Khalil Ahmed Arbi 8 Aug 2016 - 04:56

    The policy makers should make informed decision making, otherwise don’t. They can not simply go for hunch based decisions. Researchers are not an ATM machine which has ready information in their hand every time. Obviously to generate some knowledge the researchers need time to provide accurate and precise information. It comes on the shoulders of policy makers that when do they “order” for an information. Normally the policy makers are too late in ordering an information and to cover up this inefficiency they put blame on researchers for late recovery of information. A policy maker is usually a politician having non researcher background and he/she has no clue about the meticulous attitude of researcher. A researcher simply can not give false or hunch based information. The possible solution to this paradox is consistent existence of “research think tank” in every ministry which deals with the long term objectives of the ministry and subsequent short term procedures/actions in coherence with the long term objectives of the ministry. The window shooing attitude of information should be avoided and a permanent think tank for each ministry be established with clear TORs.