USAID Integrated Health Program Evaluation Report: Impact Evaluation Results

USAID Integrated Health Program Evaluation Report: Impact Evaluation Results
USAID Integrated Health Program Evaluation Report Impact Evaluation Results Cover
Abstract: This report presents results from an impact evaluation conducted by Data for Impact (D4I) following six years of the United States Agency for International Development (USAID) Integrated Health Program’s (IHP)
implementation in nine provinces of the Democratic Republic of the Congo (DRC). The IHP focuses, in part, on increasing utilization of health facility-based maternal and child healthcare and family planning services. The impact evaluation investigates the extent to which changes in healthy behaviors and health outcomes are attributable to the USAID IHP.

The primary analysis for the evaluation used a quasi-experimental design based on a propensity score matched difference-in-differences (DID) model fit to data collected through the DRC’s routine health information system (RHIS). The matching process was successful in balancing the distribution of all selected RHIS indicators across comparison and intervention sites. Restricting the propensity scores to a region of common support only led to the exclusion of 2.3 percent (104 of 4,504) of intervention facilities and 1.5 percent (101 of 6,937) of comparison facilities. The common trends assumption suggested no significant differences between the comparison and intervention groups in the preintervention time series trends for all but three of the RHIS indicators (i.e., new acceptors of modern contraceptive methods, bed net distribution during first antenatal care clinic visits, and measles vaccinations). Not satisfying the common trends assumption served to reduce the stringency of the analysis and undermined the ability to appropriately interpret the results for implicated indicators.

Given the time series nature of the available RHIS data, we conducted a secondary analysis that used a controlled interrupted time series approach. This analysis was conducted on selected indicators only for which suitable RHIS data elements were available to serve as comparator values (i.e., denominators). Another secondary analysis was done that compared intervention health zone (HZ) facilities that had received intensified support from the IHP with other intervention HZs that had not received this intensified support (the latter group served as comparison facilities).

Based on adjusted p-values from the primary analysis, significant changes were observed in six of the 13 RHIS indicators. Five of the significant changes suggested impacts in the anticipated direction. Of these, the largest magnitude was observed for new acceptors of modern contraceptive methods, adding an additional 4.06 women per 1,000 women of reproductive age when compared to non-intervention areas. A significant change in the unanticipated direction was observed for complicated malaria treatment, which decreased by 3.18 cases per 1,000 children under 5 years of age. The remaining seven of the 13 indicators did not show significant changes.

For future projects, USAID may draw its attention to those indicators that show little to no movement in the anticipated direction, such as treatment of complicated malaria, and prevalence of malnutrition in children, and consider whether any adjustments are warranted in future programs. Findings from this
portion of the evaluation will be triangulated with those from the health system surveys and the qualitative data collection, and D4I will make final recommendations in the endline evaluation report
Shortname: TR-24-589 D4I
Author(s): David Hotchkiss, Jonathan Niles Matt Worges, Janna Wisniewski, Paul-Samson Lusamba-Dikassa
Year: 2024
Language: English
Region(s): CONGO, THE DEMOCRATIC REPUBLIC OF THE