COVID-19 has tremendously challenged organizations working in the field in all corners of the world, including Rwanda and the United Republic of Tanzania (Tanzania). Movement limitation and social-distancing measures have affected the way work is organized, including the collection of data for programme evaluation.

This situation affected UNCDF and our partners’ work greatly as well, given that even though the COVID-19 pandemic is still ongoing, our programmatic and monitoring activities must continue. A number of fieldwork activities, such as baseline studies, were necessary to conduct even after protection measures for COVID-19 were enacted in both countries.

Since 2017, UNCDF has been leading poverty reduction projects that aim to improve the financial-inclusion levels of refugees and host communities in Rwanda and Tanzania. Our programmes focus on individuals participating in savings groups and emphasize improvements in digital and financial literacy as well as soft and hard business skills. The objective is to empower refugees and their host communities so that they can become active stakeholders in the digital economy of their countries.

So, what do we do if going to the field and face-to-face interviews are not possible to monitor and evaluate our projects?

In close collaboration with our partners in the field, we sought out the best and most health-conscious solutions to continue our monitoring activities. Mobile phones were the obvious answer so we modified our procedures to use this channel.

Our partners in the region are Three Stones International Rwanda (TSIR), a firm in Rwanda specialized in conducting evaluations, and the Kigoma Youth Agricultural Development Organization (KIYADO), an implementing NGO in Tanzania. With both partners we needed to collect data for their baseline and midline evaluations from April to June 2020.

The baseline survey by TSIR aimed to understand the financial knowledge, skills, attitudes and behaviors, and the socio-economic profiles of the beneficiaries of the programme Expanding Financial Access and Digital and Financial Literacy in Rwanda (REFAD).

KIYADO works together with UNCDF Tanzania for the Kigoma Joint Programme, and conducted a mid-line survey in May 2020 to measure the project’s progress towards reaching development outcomes and find out the gaps in the programme design that may result in poor execution of the project.

Pros and cons of collecting data via telephone survey

Before making the decision to move from face-to-face to mobile-phone interviews, UNCDF, TSIR and KIYADO assessed the risk of exposure to the virus, interviewees’ willingness to participate in outdoor activities, the impact of government’s restrictions on mobility and the team's capacity and preparedness.

Table 1: Pros and Cons of Telephone Data Collection Method

Top three challenges of collecting data via phone

Both partner organizations implementing the surveys faced the same three challenges.

1. Respondents do not respond to phone calls

In rural and underserved areas, weak connectivity and low mobile-phone ownership make it difficult to reach all target respondents.
For the baseline collection, TSIR gathered mobile phone numbers from other UNCDF partners: Inkomoko, World Relief, Umutanguha Finance Company and the local government. However samples from the local government were not large enough. During the survey, TSIR had to call the same numbers three times over three days. When such calls did not pay off, they would contact neighbours, spouses and/or savings group leaders. If not effective, TSIR used an oversampling method to find substitutes.

KIYADO’s approach differed because they were collecting mid-line data on the same sample as their baseline survey carried out in December 2019. KIYADO relied on village-based agents and savings groups’ leaders to allow individuals living in remote and underserved areas to move to locations with good signals during the survey day. Those target respondents without mobile devices nor active SIM cards were grouped together with others more equipped, while everyone made sure that phone batteries were fully charged for the exercise.

2. Respondents do not trust the voice on the other end of the phone

Responding to complete strangers over the phone, rather than meeting up and seeing the person asking questions, reduces the interviewee’s willingness and openness to answer to financial and livelihood questions, as well as trust in the usage of his or her personal data.

To address this issue TSIR sent an official letter to all relevant local government representatives and civil society influencers while keeping Rwandan government informed. In addition, enumerators followed a standard protocol which included asking for consent after clarifying the purpose of the data collection in the context of the project, as well as their roles and responsibilities within TSIR.

KIYADO obtained respondents’ consent through village-based agents and savings groups’ leaders, after communicating the objective and methodology clearly and ahead of time. KIYADO planned for the survey in consultation and coordination with village-based agents and savings groups’ leaders, based on the specificities of the location and respondent profiles. Logistics aside, KIYADO reformulated certain questions to make them more clear or less direct in order to avoid offending anyone. Enumerators received training on questioning techniques on asking questions while putting interviewees at ease.

3. Data quality control

Unstable connectivity, hardware malfunction and oversights in data collection and entry may affect the speed and quality of the exercise.

TSIR and KIYADO implemented the following quality control procedures.

Table 2: The Quality Control Procedures of TSIR and KIYADO

Recommendations for mobile-based data collection

As the survey’s and the evaluations where completed, UNCDF asked TSIR and KIYADO to share their unique experience on mobile-based data collection and three recommendations that could help other practitioners.

(1) Communicating and preparing clearly: Clearly describing objectives, roles, methodology and types of questions before asking for consent helps to ensure respondents’ buy-in as they feel safer and more comfortable answering to a stranger on the other end of the line.

(2) Understanding the risks: Conducting a thorough risk assessment before and throughout the activity highly contributes to being prepared and reactive or proactive as the enabling environment and the uncertainties on the epidemic evolve on short notice.

(3) Data quality controls should never be neglected: Optimizing data quality and checkpoints include identifying the most appropriate tools for the context and the people, training enumerators extensively while monitoring daily that data is well captured, protected and reliably usable.

In short, using mobile phones allowed TSIR and KIYADO to efficiently perform evaluation activities during the pandemic, with lower costs and fewer delays.

About Three Stones International (TSIR)

TSIR is an evaluation company based in Rwanda with rich experience in conducting evaluations and social research assignments in more than ten countries. TSIR is a partner of UNCDF in Rwanda to evaluate interventions supporting savings groups and their members in refugee-hosting areas. They have a sample population of 1247, made up of 54 percent women and 14 percent youth in Rwanda.

About Kigoma Youth Agricultural Development Organization (KIYADO)

KIYADO is a Tanzanian NGO conducting livelihood interventions in agriculture, financial inclusion and business development in the Kigoma Region. KIYADO is a partner of UNCDF that supports savings groups and members in refugee-hosting communities, with special attention to youth and women. KIYADO has a sample population of 259, made up of 71 percent women and 16 percent youth.