Evaluation of Integrated Community Case Management in Ethiopia

Independent Prospective Evaluation of Integrated Community Case Management and Development and Implementation of a Method for Real-time Mortality Monitoring in the Oromia Region, Ethiopia

The purpose of this study is to to measure the effect of the HEP+ICCM program relative to routine HEP approach in rural Ethiopia on changes in coverage of case management of common childhood illnesses and severe acute malnutrition, reductions in mortality among children under the age of five, and improvements in nutritional status using a rigorous evaluation design.

STUDY BACKGROUND AND RATIONAL Most low-income countries are making slow progress in addressing childhood mortality - too slow to achieve the fourth Millennium Development Goal by 2015. Countries, donors and development agencies are responding to the situation by redoubling efforts to stimulate and support child survival activities in countries, particularly in Africa. The Catalytic Initiative to Save a Million Lives (CI) is a partnership between donor agencies committed to accelerate progress toward MDGs 4 and 5 through support of scale-up of proven high impact interventions. UNICEF and the Canadian International Development Agency (CIDA), as part of the CI, are undertaking a major initiative to reduce mortality among children less than five years of age in several countries in sub-Saharan Africa, including Ethiopia.

Ethiopia has a high under-five mortality rate, estimated at 123 per 1000 live births by the 2005 Demographic and Health Survey. The country has made a commitment to achieve the MDG 4 target of a two-thirds reduction in mortality by 2015 as reflected through the launching of the national Health Extension Program (HEP) in 2004, with continued support from UNICEF and CI funding. The key program strategy is to train and support approximately 30,000 Health Extension Workers (HEWs) to provide promotive, preventive, and selected curative health care services at the community level. Previously, the HEWs treated diarrhea with ORS and malaria with rapid diagnostic kits and ACTs; pneumonia cases were referred to health centers.

Ethiopia has recently adopted a policy to expand management of pneumonia among children under-five with antibiotics to communities through HEWs. This provides a unique opportunity to accelerate increases in coverage of treatment of pneumonia, which is one of the greatest killers of children under-five in Ethiopia. This expansion, when combined with the existing community management of malaria, diarrhea and severe acute malnutrition, is referred to as integrated community case management (HEP+ICCM) and is largely supported through the Catalytic Initiative by CIDA and UNICEF and will focus on its first stage on five regions of the country: Amhara, Benishangul-Gumaz, Oromia, SNNP and Tigray. The leading strategy of the HEP+ICCM initiative is to increase the capacity of the HEWs to effectively assess, classify, and manage the leading causes of preventable child mortality including pneumonia, malaria, diarrhea, and severe acute malnutrition. In the focus regions, plans have been developed to introduce CCM of childhood pneumonia with cotrimoxazole and diarrhea with ORS and zinc, malaria with ACTs, and malnutrition with therapeutic feeding. The program plans to conduct refresher trainings of HEWs and their supervisors, strengthen supervision, logistical support and the ICCM system overall. The initial implementation will start in August 2010 and to be phased in Oromia region.

The effectiveness of the ICCM varies across specific country contexts and depends on the strength of implementation. It is therefore essential to evaluate the effectiveness of the scale-up strategy to provide a basis for future program improvement and global evidence on effective strategies for accelerating reductions in under-five mortality.

OBJECTIVES The Institute for International Programs at the Johns Hopkins University Bloomberg School of Public Health (IIP-JHU) has been commissioned by CIDA and UNICEF to conduct an independent prospective evaluation of the HEP+ICCM.

This study objective is to conduct a full prospective evaluation of the impact of the Ethiopian HEP+ICCM of common childhood illnesses implemented through HEWs on mortality and nutritional status of children under the age of five.

The specific objectives of the study are:

i. To measure the effect of the HEP+ICCM program relative to routine HEP approach in the country over an 18 month period on changes in coverage of case management of common childhood illnesses and severe acute malnutrition, reductions in mortality among children under the age of five, and improvements in nutritional status using a rigorous evaluation design; ii. To track program implementation inputs, processes, outputs and contextual factors; and iii. To measure implementation strength and assess the quality of child health care.

METHODOLOGY In Oromia region, implementation of the HEP+ICCM program will be phased allowing the possibility to identify comparison areas for the evaluation. The prospective evaluation will use a cluster randomized design with stratification by zone, within which woredas are randomly assigned to intervention and comparison arms using a restricted randomization procedure. Coverage of childhood interventions will be measured at baseline and end-line with a midterm "quality of care implementation snapshot" to assess the strength of implementation. Mortality estimates for both the intervention and comparison woredas will be generated retrospectively through the end-line household survey. Program implementation and contextual factors will be documented throughout the evaluation period.

The evaluation will be focused the Jimma and West Hararge zones. Woredas within the two evaluation zones will be randomly assigned to intervention and comparison areas. Intervention areas are referred to as phase 1 area and comparison areas referred to as phase 2 areas. The phase 1 and 2 woredas will be randomly assigned through an adapted, restricted randomization process, balanced by the presence of malaria, food security and the zone. Urban areas were excluded since the urban HEP is a recent addition and implemented differently. In Jimma zone, the seventeen woredas will be divided into nine phase 1 woredas and eight phase 2 woredas. In West Haraghe zone, there will be seven woredas in phase 1 and phase 2 areas.

The ORHB has committed to at least an 18-month phase-in between the identified intervention and comparison areas in these two areas. Phase 1 woredas will implement HEP+ICCM while the Phase 2 woredas will continue to offer services included in the routine HEP. Phase 1 implementation in the two evaluation zones will begin in February 2011, following the baseline coverage survey.

DATA SOURCES Household Surveys: The primary data for the evaluation will come from baseline and end-line household surveys. The baseline survey was conducted in January - February 2011 and the end-line survey will be conducted 18 months following the end of the HEW training as part of phase one. These surveys will measure coverage of the child health interventions of interest and nutritional status among children under five. The end-line survey will include, in addition to coverage measures, a full birth history for women 15-49 years old to measure retrospectively mortality among children under five. Partners report that phase one implementation was completed in July 2011; therefore the end-line survey is planned for January 2013.

For the baseline survey, we estimate that 3,700 households (1850 in each arm) are required to detect a change in coverage indicators between baseline and end-line. For the end-line survey, sample size of households is calculated to detect a difference in differences of 20 percentage point in under-five mortality between intervention and comparison areas with 80% power. We estimate a conservative sample of about 15,157 households in each arm of the study will be required, making a total sample size of 30,314 households.

Measurement of implementation strength and assessment of quality of child health care data: A community-level survey of HEWs is planned to provide a "snapshot" of HEP +ICCM implementation strength in the two evaluation zones. The data collected will comprise core indicators of implementation strength, as well as indicators on demand-generation activities, utilization and the quality of services provided by HEWs.

The study will sample functional 104 health posts in the intervention areas and 46 in the comparison areas. Study participants will include: 1) HEWs performing case management of childhood illnesses, 2) sick children 2-59 months of age presenting at health posts for consultations and their caretakers and 3) sick children 2-59 months in the communities surrounding health posts.

ANALYSIS The main analysis of the data will involve comparison of trends in coverage and impact indicators between intervention and comparison arms, using appropriate statistical models that adjust for clustering at cluster level. Adequacy of implementation in each arm of the evaluation will be determined using data from baseline surveys, process documentation data and the implementation snapshot survey.

Using data from the end-line survey, under-five mortality will be computed on 18-month period before the baseline survey (which will represent the baseline under-five mortality) and on 18-month period after full implementation (the end-line under-five mortality). Precaution will be taken to measure the end-line mortality only on period starting from time when full implementation (training and deployment of HEWs with drugs) has been completed. Full implementation in intervention woredas was achieved in July 2011, six months after the completion of the baseline survey.

The first level of analysis will assess "intention to treat", which consists of analyzing the data from the intervention and comparison arms as per the start-up design. This analysis will compare changes between baseline and end-line in under-five mortality rates as well as nutritional status among children under-five in the intervention and comparison areas. Difference in differences analysis will be conducted to estimate the impact of the ICCM on mortality. Proportion hazard models will also be applied to control for possible differences in baseline characteristics and contextual factors.

An important secondary objective of the HEP+ICCM is to increase equity in access to high impact, cost-effective preventative and curative interventions. The evaluation team will assess changes in inequalities associated with the implementation of the HEP+ICCM program interventions by examining project outcomes and impact across socioeconomic and demographic subgroups.

Inclusion Criteria:

• Surveys:

  • Members of households in evaluation woredas in Jimma and West Haraghe zones of Ethiopia.

• Quality of Care Implementation snapshot:

  • Health posts: All functional health posts in the study zones will be included in the sampling frame. In cases where an HEW is providing services, but an official health post structure has not been constructed, the HEW's primary location for providing case management services will be considered as the health post.

  • HEWs: All HEWs providing case management services in selected health posts will be included.

  • Patients must meet the following criteria:

    • Between 2 and 59 months of age;
    • Described as sick by the caretaker. Sick children must have at least one of the following complaints: signs or symptoms of severe illness (change in consciousness/lethargy, convulsions, vomiting everything, not eating or drinking); fever/malaria; cough, fast/difficulty breathing, pneumonia; diarrhea/vomiting; ear problem; measles; nutrition or feeding problems;

Exclusion Criteria:

• Surveys:

  • Members of households NOT residing in the evaluation woredas in Jimma and West Haraghe zones of Ethiopia.
  • Members of households in urban communities.
  • Participant does not consent to study procedures

• Quality of Care Implementation snapshot:

  • Non-functional health posts

  • HEW does not consent to procedure.

  • Patients will be excluded from the study:

    • If this is not the initial consultation for the current illness episode: patient has been seen at the health post, by the HEWs (including home/community).
    • Patient is younger than 2 months or older than 59 months.
    • Caretaker does not consent to procedure.