How to increase the value of COVID-19 data: Lessons and reflections from PAD’s multi-country survey

How can researchers ensure that data collection in the midst of a pandemic is worth the time and effort?

As the COVID-19 pandemic gathered pace in mid-March, Precision Agriculture for Development (PAD) launched a rapid, multi-country phone survey to monitor the impact of the disease on our end-users. We were not the only ones with this idea, however. Convenience surveys –  data collection from whichever group is closest to hand – have proven explosively popular during the outbreak, as researchers have scrambled to contribute to the nascent evidence-base. In this blog, we reflect on how researchers can ensure that this unprecedented data drive is worth the time and effort.

According to Innovation for Poverty Action’s (IPA) RECOVR Hub, at least 141 remote COVID-19 surveys have been initiated in 76 developing countries since March (Figure 1). This data collection involves non-trivial direct costs, including large surveyor teams, state-of-the-art data management platforms, multiple hardware, and many, many hours of respondents’ time. The opportunity costs may be greater still, particularly if surveys divert resources from service delivery or other research agendas. These costs are justified only if they are outweighed by the benefits of the evidence for policymaking and programming. Below, we draw on our own experience to suggest eight ways that coronavirus surveys can be better designed, implemented and shared to inform the pandemic response and ensure value-for-money.

Source: IPA’s Research for Effective COVID-19 Responses (RECOVR) Research Hub (retrieved on: 10 August 2020)

During survey design

1. Identify the evidence gaps

To allocate scarce resources effectively, policymakers need to know how COVID-19 and related restrictions are affecting different groups. A key risk with convenience surveys is that sectors or populations of lesser interest to researchers are excluded from the evidence-base. A thematic break-down of the studies listed in IPA’s RECOVR hub suggests that research on COVID-19’s impacts has concentrated on some areas more than others (Figure 2). The relative dearth of attention to agriculture is concerning, given that two-thirds of working-age adults in developing countries work in this sector. Going forward, researchers should be cognizant of these imbalances and seek to address them. Donors and policymakers can assist by providing additional funds for research on neglected populations.

Source: IPA’s Research for Effective COVID-19 Responses (RECOVR) Research Hub (retrieved on: 10 August 2020)

2. Strive to harmonise

Harmonising data collection has benefits for both researchers and policymakers. For researchers, coordination can lead to better instrument design, larger datasets, and more precise estimates of COVID-19’s effects. For policymakers, comparable data facilitates needs assessment and targeting. Fortunately, there are some excellent coordination platforms for research on COVID-19. COVID-19 Economic Impact Surveys, a joint initiative by the International Growth Centre and IPA, has coordinated the instrument design of 32 research groups, including PAD, to generate a comparable, cross-country dataset for key variables.

3. Seek diverse perspectives

As Ranil Dissanayake from the Centre for Global Development notes, unprecedented challenges like COVID-19 make “identifying the right questions especially hard”. There is little theory and evidence to draw on, and researchers frequently find themselves investigating phenomena outside of their field of expertise. Given this uncertainty, incorporating a diversity of perspectives during research design helps to ensure that important questions - and therefore relevant data - are not missed. In addition to speaking to policymakers and practitioners about their data requirements, researchers could, and should do more to involve target populations in research design. Developing better techniques for remote or socially-distanced qualitative research would help generate insights into subjects’ challenges and concerns.

During implementation

4. Balance speed with sub-group representation

Differential attrition is often a consequence of rapid and remote data collection. Since strategies for minimising “drop out” are typically time-intensive or difficult to implement during lockdown, important subgroups have been under-represented in datasets on COVID-19’s impacts. For example, women are systematically excluded from official data on confirmed COVID-19 cases in Pakistan, preventing the development of a gendered policy response. PAD’s COVID-19 survey also achieved lower response rates amongst women, as well as substantial attrition overall (Table 1).  Going forward, researchers should consider testing and sharing approaches for reducing attrition in remote surveys. Effective solutions may be as simple as increasing the number and varying the timing of call attempts.

Note: This table shows response rates by gender for PAD’s COVID-19 survey from two states in India.

5. Triangulate your sources

A surprising finding from our data collection in Kenya is that farmers are more sanguine about the pandemic’s effect on input markets than agro-dealers. Over two-thirds of the smallholders whom we interviewed said that they would use at least as many inputs this season as last season, and 92% of those that had visited agro-dealers had reportedly purchased their required inputs. By contrast, agro-dealers described steeply diminished demand for their products. The mismatch highlights the importance of triangulating data from different sources, including ones outside of your main population of interest, to generate a more complete picture.

6. Prioritise continuous monitoring over “snap shots”

In situations of extreme uncertainty and unpredictability such as COVID-19, regular monitoring will always be more valuable than one-off surveys. Frequent data collection enables measurement of important temporal trends and changes. For example, in one Indian state, we found that farmers’ fears about the pandemic shifted as the lockdown lengthened, from a member of their family contracting the disease to losing work and income. Moreover, learnings from initial survey rounds can be used to strengthen the research design over time. Following evidence that farmers had reduced their meal sizes in May, PAD’s Kenya team revised its questionnaire to measure whether the decreased consumption was seasonal or linked to COVID-19. Finally, in low-resource settings, researchers that monitor vulnerable populations can raise timely alerts if their situation deteriorates, thereby providing a crucial early-warning function.

During dissemination

7. Transparency improves research impact

Data on the impacts of COVID-19 is an urgent and vital public good. Wherever possible, researchers should publish their datasets (including the survey instruments, code, and research methods employed) online. Publishing them increases the impact of the evidence: the research is immediately more credible, more usable, and more shareable. It also improves the data quality because researchers are forced to check and re-check their work. When full transparency is not an option due to the sensitive subject matter, researchers should at least try to specify their sample frames, sampling approach, and attrition rates so that unsuspecting readers do not generalise results to the wrong populations.

8.   …as does effective communication

The breath-taking pace of the evidence generation on COVID-19’s effects risks confusing, rather than informing, time-strapped decision-makers. New findings are frequently conflicting, scattered across platforms, and often embedded in lengthy documents. Researchers can help by distilling key messages from the deluge. Sharing papers via one-stop-shop hubs for coronavirus research, such as the RECOVR Hub and CORE Net, is an important first step. But synthesising and summarising the evolving evidence-base is likely to be more useful. In Uganda, for example, researchers have established a Slack channel to collate new findings before communicating them to policymakers. Innovative products, such as PAD’s interactive dashboard, may also increase the impact of research.

While confirmed cases of COVID-19 continue to rise around the world, demand for data on the pandemic’s socio-economic impacts will remain strong. Researchers must ensure that the quantity of evidence generation does not compromise its quality, if they are to help alleviate the dreadful toll of the pandemic.

Disclaimer: The views expressed in this post are those of the authors based on their experience and on prior research and do not necessarily reflect the views of the IGC