Expertise and Leadership from Local Partners: A Q&A on data collection strategies and lessons learned in Bangladesh and Tanzania

August 15, 2024

By Joyanta Roy and Catherine Kahabuka

In 2023, Data for Impact (D4I) published an assessment report entitled High Impact Practices (HIPs) in Family Planning (FP): A qualitative assessment of quality and scale of implementation for three service delivery HIPs in Bangladesh and Tanzania.

The assessment focused on the extent that three HIPs are implemented across several USAID-funded health service delivery projects in Bangladesh and Tanzania. The HIPs covered in the assessment relate to community health workers, mobile outreach service delivery, and immediate postpartum family planning. Prior to this assessment, no implementation standards for HIPs had been established beyond the HIP technical briefs. As such, HIP core components were developed with core components referring to specific guidance for implementation. Data collection included the administration of core component checklists to rank individual core component implementation using a scale of 1 (limited) to 4 (foundational) and key informant interviews. A total of 156 individuals (83 male and 73 female) participated in data collection.

The D4I HIPs assessment—like nearly all USAID-funded research and evaluation work—relied heavily on the contributions and leadership from local partners in Bangladesh and Tanzania. Of particular importance was their expertise in interview techniques, group facilitation, data management, and analysis along with in-depth knowledge of country context.

With the assessment now completed, D4I reflected on the strategies and lessons learned with Joyanta Roy from the Capacity Building Service Group (CBSG) in Bangladesh and Catherine Kahabuka from the Centre for Research Mentorship and Support (CREMES International) in Tanzania.

Joyanta Roy, CBSG

CBSG engaged with the four organizations who agreed to participate in the assessment (Chemonics, Pathfinder International, Save the Children, and Social Marketing Company) to introduce the HIP assessment objectives and data collection strategies and to distribute checklists in advance. This method effectively gathered detailed insights and secured organizational buy-in.

We adapted to COVID-19 restrictions by using Zoom for the small group discussions and phone calls or WhatsApp for key informant interviews (KIIs) and ensured accurate transcription through audio recordings. Team meetings were conducted after each small group discussion and KII, which was a critical function to note any learnings or corrective measures needed.

We also used our in-house team of interviewers, moderators, and note-takers to translate the checklist, prepare the IRB application, and participate in training sessions. This greatly aided in having skillful moderation and transcription.

Catherine Kahabuka, CREMES

Our primary goal in leading data collection teams is to ensure the collection of high-quality data. To achieve this, the following key strategies have been instrumental:

• Comprehensive training: A cornerstone of our approach is the thorough training of our teams. Through our CREMES Research Mentorship Program, we recruit and strengthen the capacity of young researchers, not only in research skills but also in research ethics, community entry, and cultural sensitivity.
• Deep understanding of project context: We dedicate time to ensure that all team members have familiarity with the evaluation objectives and broader project goals. This includes detailed briefings on the expected outcomes and specific elements the projects aim to achieve.
• Familiarity with data collection tools: To ensure each team member is thoroughly familiar with the data collection tools, we conduct detailed reviews of each question and overall tool structure, ensuring that the team can probe effectively and gather comprehensive data. For the HIPs assessment, it was vital for the team to be well-versed in the HIPs so they could effectively question and evaluate the implementation of these practices.

Joyanta Roy, CBSG

CBSG uses appreciative inquiry techniques in our FP/RH research, particularly during data collection of sensitive issues. By highlighting positive examples, such as successful FP stories that have helped overcome social taboos, we encourage the participants to feel comfortable and open in sharing their information freely.

At the project level, we thoroughly reviewed project progress reports. At the district level, we reviewed the FP Management Information System data. This approach enabled us to gain a deep understanding of the local context, which in turn allowed for the collection of objective data to inform discussions and interviews.

Catherine Kahabuka, CREMES

Here are CREMES International’s approaches to eliciting insightful, actionable information:

1. Building rapport and trust: We prioritize training our teams on the importance of developing rapport. Establishing trust is crucial to making respondents feel comfortable and encouraged to openly share their views and stories.
2. Guaranteeing confidentiality: We emphasize the importance of confidentiality. Respondents may have ongoing relationships with project staff or government officials, which can make them hesitant to share openly. We ensure all team members are equipped with scripts that reassure respondents that their information will be handled with the utmost privacy and that their identities will not be disclosed under any circumstances.
3. Conducting private interviews: Privacy is paramount, especially when evaluating sensitive programs. Interviews are conducted in a setting that guarantees privacy. This is important when project staff wish to observe; we politely explain that their presence might inhibit honest discussion and is therefore not permitted.
4. Using effective probing techniques: To deepen the quality of responses, we ensure the interviewers are well-versed in probing techniques, such as echoing back what the respondents say or requesting specific examples.
5. Customizing interview settings: Lastly, we are very careful about how we configure interview settings. Sometimes, group interviews are useful, but we always consider the dynamics of who is in the room. Ensuring there are no power imbalances or potential conflicts of interest is vital for fostering an environment where everyone can speak freely.

Joyanta Roy, CBSG

For data management, we implement systematic organization methods, ensure accurate recording, maintain confidentiality, and employ version control.

For data analysis, we employ advanced qualitative analysis techniques, including thematic analysis, coding, constant comparison, and triangulation, to extract themes, patterns, and insights from qualitative data. In addition, we enhance the validity of our findings through the rigorous examination of secondary sources such as literature, reports, and datasets to verify the authenticity of the data.

Catherine Kahabuka, CREMES

Efficiently managing qualitative data is paramount at CREMES International, particularly given the depth of responses collected by our skilled interviewers. Our approach to qualitative data is meticulously structured from the beginning of the study design and ensures that the evaluation questions guide each step.

• Structured coding framework with ATLAS.ti: To handle the complexity and volume of data, we employ a detailed coding frame or codebook informed by the evaluation questions. We use ATLAS.ti, a tool for qualitative analysis, to systematically code and organize large volumes of data. This efficiently extracts relevant information from the interview responses.
• Effective data management system: Alongside ATLAS.ti, we use a robust Excel system designed to effectively summarize and organize data from the code reports to capture and organize comprehensive insights for each evaluation question.

Joyanta Roy, CBSG

Over the past 20 years, we have expanded our portfolio, primarily working on capacity strengthening, research, and evaluation projects funded by USAID and other U.S. government agencies such as the U.S. Department of Agriculture and the State Department. CBSG collaborates with U.S.-based implementing partners and has successfully completed more than 150 activities, primarily in Bangladesh and across South Asia. Projects funded by the U.S. government require strict adherence to timelines and high-quality deliverables. We consistently perform risk analysis, implement mitigation strategies, remain highly responsive, and are committed to maintaining the highest standards of quality.

Catherine Kahabuka, CREMES

CREMES International primarily supports monitoring and evaluation activities and has observed that evaluations of USAID-funded projects are often implemented within tight timeframes. USAID funds public health and education programs in Tanzania that have extensive reach and impact. The need for quick data collection and analysis underscores the importance of having robust monitoring and evaluation frameworks in place to ensure that programs can adapt swiftly to emerging needs and challenges, maximizing their effectiveness and impact.

A significant challenge is the limited time allotted to initiating and completing evaluations of USAID-funded projects. To address this challenge, we have a robust pool of skilled research assistants and junior consultants. Further, our mentorship program nurtures junior researchers, providing us with a steady stream of professionals ready to support our projects.

Another challenge we often face is the timely availability of funds. To mitigate this, we maintain reserve funds to ensure data collection can commence promptly, even before our first invoice has been paid. This financial strategy enables us to avoid delays and maintain momentum in our projects, ensuring that tight deadlines are met and results are delivered as expected.

---

Authors: