Keywords

1 Introduction

A key goal of a health system is to provide the necessary healthcare services to all those who are in need; thereby improving the health status of the population. International organisations such as WHO and United Nations, in partnership with governments of countries, have developed global initiatives to monitor performances of the health system. One concrete example is the United Nations’ Sustainable Development Goals (SDGs) which target major health problems alongside other related problems alongside other related issues and constitutes a coordinating framework for these efforts (WHO 2010). Universal Health Coverage (UHC) is one of the goals of this initiative. UHC has been defined as the ability of all people who need health services to receive them without incurring financial hardship (WHO 2010; Kieny and Evans 2013). To monitor countries’ health progress and performances towards supporting UHC, a range of health determinants’ indicators have been developed. These indicators are measurable and time-bound. Examples of these health indicators include; children under 5 sleeping under insecticide Treated Nets (ITNs) and births delivered in a birth facility.

Health indicators are powerful tools used at all levels of the health system, for monitoring and evaluating and communicating information about the population’s health (Mant 2001; Klazinga et al. 2001; Maniz 2003). They could be used to track how well (quality) and how far (quantity) countries’ health system are performing (Klazinga et al. 2001; Maniz 2003).

An indicator has two parts; numerator and denominator that go into the formula for calculating them (Klazinga et al. 2001; Maniz 2003). Numerators are the things we count, i.e. infants immunized or new cases of TB. Denominators are the group with which the things we count are compared, i.e. total population or all births in a year. In the health system, indicators could also be used to measure a variety of dimensions concerning the health situation of the population; mortality, morbidity, health status.

There exist different types of indicators which could be used to monitor health progress, namely; health status, health systems, risk factors and health service coverage. Of important to this paper are health service coverage indicators.

These indicators reflect the extent to which people in need actually receive the health service they need (Tanahashi 1978; WHO 2009). This group of indicators measure the effectiveness of health programs. They help service providers to understand how effective an intervention is, and whether one target group is reached more effectively than another. They also help to identify underserved areas or regions which need more attention (Boerma et al. 2014). Most UHC indicators fall in this group.

Indicators in this group use population data as denominator, as it facilitates comparison of health status or health service provision over time and space. Coverage indicators are equally specific and valid for small and large geographic entities, and for any given time interval (WHO 2009). Examples of such indicators are; immunization coverage, HIV testing coverage, and delivery in facility rate. A common challenge is that there is lack of reliable population data to calculate them.

Population data is usually not accessible for smaller areas such as communities or districts (WHO 2009; Linard et al. 2012). Often, available population data is either outdated, not available for current administrative entities, not available for certain target populations (such as women of child-bearing age), or they may be duplicated sources of population data that do not necessarily provide the same figure (Linard et al. 2012; Leegwater et al. 2015; Bharti et al. 2016). The purpose of this paper is twofold. First, we are empirically exploring the understanding and use of health indicators by health program managers for decision-making. Second, we draw implications from this situation related to local practices and global initiatives such as UHC.

This study is derived from an ongoing implementation of an electronic system as a measure to strengthen Health Management Information system (HMIS) in Cameroon. This system is the main tool used for the management of health information nationwide. In this system, indicators are core element of data analysis, used to measure healthcare services.

2 Related Literature

In this section, we present relevant literature on health indicators to establish an understanding of what they are, what makes a good or poor indicator, and how they are used in health management.

Indicator(s) is developed by international organizations, reference groups and interagency groups, countries, academics, advocacy groups and others. From a performance management perspective, Flowers et al. (2005) describe indicator as a measure used to express the behavior of a system. In the public health context, indicators are specific tools for programme management, including the analysis and diagnosis of problems, and for taking correctives actions (Sahay et al. 2009).

Indicators are key statistical measure used to describe a situation, track progress and performance over time and to compare entities doing similar work. They could be used as a guide to decision making and set priorities (Donabedian 1966). They help to inform policy and policy-makers and can be used to improve quality of care and promote accountability (Donabedian 1966; Mainz 2003).

The literature explains that the process of selecting indicators should be systematic and based on facts rather than on feelings (Mant 2001). The process should involve all stakeholders and be based on “who wants the indicator”, “how it is used” and “by whom” (Mant 2001; Klazinga et al. 2001). Heywood et al. (2001) explains that the process should employ understanding, discipline, teamwork and negotiation. Therefore, an ideal indicator should: (i) be based on agreed definitions; (ii) measure what it is intended to measure; (iii) give the same results if used by different people in different places; (iv) be simple to calculate using readily available data; (v) fit local needs, capacity and culture and the decisions to be made; (vi) be highly sensitive to changes in the situation concerned; (vii) permit useful comparison; (viii) be evidence-based (Heywood et al. 2001; Mainz 2003; Larson and Mercer 2004).

Indicators can be classified as: input, process, output, outcome and impact. Input indicators are the resources needed to implement work. Input indicators measure resources, both human and financial, allocated to a particular program. Process indicators measure whether planned activities did take place. For example, in antenatal care, process indicators can be antenatal 1 st visit before 20 weeks rate, antenatal client re-test rate, and post-natal visit with 2 weeks rate. Output indicators measure first level results associated to an intervention. They are defined as what we produce. Some examples include; Infant 1 st PCR test around 6 week uptake or delivery by caesarean section rate. Outcome indicators measure what we have achieved and should be linked to concrete goals. Examples include; delivery at facility for women under 18 years and live birth under 250 grams in facility rate. Finally, impact indicators are the cumulative effect of the overall program (Mainz 2003; Heywood et al. 2001).

Indicators can be calculated in different ways; ratios, proportions, or rate (Worning et al. 1992; Mainz 2003). Ratios are numbers expressed in relation to another by dividing one number by the other. Here, the numerator is not part of the denominator. Proportion measures a part or amount that is part of a whole. The numerator is part of the denominator. Rate represents the frequency of an event in a specified period. In calculating a rate indicator, the numerator is the number of occurrences of an event during a period of time. The denominator is the number of person exposed to that event in the time period. These indicators are illustrated in Table 1 below.

Table 1. Different types of indicators
Full size table

Population data is required as denominator to calculate these types of indicators. Since these are integral for health management, the lack of population data poses a challenge.

3 Research Context

The empirical setting within which the study was conducted is Cameroon. It is a low-income country, situated in the sub-Saharan Africa (SSA) region. It has an estimated population of 20.6 million (Chen et al. 2004). In Cameroon, basic public and social amenities for the vulnerable are either absent or inadequate. Nationally, 29.7% of the population does not have access to safe drinking water and 66.9% lack adequate sanitation, resulting in regular outbreaks of cholera and other water-related diseases (UNICEF 2015). The burden of healthcare financing is born largely by households through out-of-pocket payments (OOP). The government of Cameroon spends an average of USD 61 as per capita per person on health. Out of this amount, only USD 17 paid by the state, USD 8 comes from international donors, and USD 36 is OOP (Cameroon Economic Update 2013). Over the past two decades, health indicators have remained poor, and in some cases even worsened. Cameroon is struggling with high mortality and morbidity especially in rural communities. Mortality rate for children under 5 is 148 per 1000 live births, ranking Cameroon as 18th amongst 20 countries in the world with the highest mortality rate. Only 13% of children under the age of five sleep under insecticide-treated nets, in a country where malaria accounts for more than 40% of all deaths in this age group. Maternal mortality rate is alarmingly high, 670 per 100,000 births as compared to 546 per 100,000 live births in SSA. In addition, many women and girls have limited access to, and utilization of, prevention-of-mother-to-child transmission (PMTCT) services, resulting in HIV infection transfer to children (UNICEF 2015).

The healthcare system adheres to the district health approach, organised in three levels: the operational level, corresponding to district health care; hub of all health interventions; the intermediate level which is responsible for technical support, while the central level deals with the development of health policies. Different programmes operate at all three levels, engaged in the provision of specialised services such as maternal and child health, malaria, HIV/AIDS, TB, and are supported by different donor agencies. The health system suffers from qualitative and quantitative shortage of human resources, and lack of technical and managerial expertise (Cameroon Economic Update 2013).

The health management information system (HMIS) in Cameroon is fragmented and characterised by vertical and fragmented information system and non-standardized data collection methods. DHIS2, the electronic tool used is housed at “Cellular National d’Informations Sanitaires” (CNIS), the department responsible for the management of health information in the country.

4 Research Design and Methods

The study is drawn from an interpretative strand (Walsham 1995). Interpretive research in Information Systems (IS) is useful as it helps researchers understand the problem in the contextual nature (Klein and Myers 1999).

Data was collected by the first author using qualitative methods. It included interviews, group discussion, and a document review. A total of 2 focus group discussions and 7 interviews were conducted in January and in July 2016, with Maternal and child health Programme Managers at district and national levels; data managers at district healthcare facilities, Matron in-charge of data management in wards, and Sister-in-charge of health facilities. 22 health and program managers participated in the study, of which 13 were females. Ten had no access to a computer and only 6 had Internet access. The document review included annual reports, strategy documents, and program reports.

Purposive sampling technique was used to select interviewees (Creswell 2007). This technique is used to achieve a homogeneous sample; that is sample of cases who share the same characteristics e.g. background or occupation. In this study, the interviewees shared a similar occupation; i.e. involved in data management at their respective facilities. An interview guide with broad themes around data management was used, focusing on understanding and use of health indicators. Permission to conduct the study was obtained from the Office of Regional Health delegate of each region and signed informed consent was obtained from each interviewee. The principle of data saturation was applied; i.e. interviews were ended when further probing were not adding new information. Permission to audio record interviews was obtained from interviewees at the start of every discussion and interviews were transcribed.

Data analysis was driven by the interpretive process and a descriptive approach with content analysis was used to analyse the data (Elo and Kyngas 2008). The interviews were transcribed verbatim. The interviews were read through several times to obtain a sense of the whole. The text was then divided into condensed meaning units. Open codes were used to group headings into categories to formulate a general description of the research questions. The interviews revealed themes which are interrelated. During this process, the data, themes and topics was discussed among the authors and based on these discussions the data was revisited.

5 Analysis

The analyses of the interviews and review of documents revealed the following:

A general observation in Cameroon is that population data is either outdated or not available. In the absence of reliable census data, the National Bureau of Statistics issues population figures per province and district to the Ministry of Public Health annually. However, for areas below the districts, it provides percentage per population group and annual projected growth rate per age group that has to be calculated by those who want to use population data. Usually, healthcare managers lack the necessary numeracy skills to perform such calculations. Also, these figures are not publicly available.

Data is collected at health facilities, and reported upward through health districts to the national level. At health facilities, daily registers are used to record activities from various units of the facility; outpatient, antenatal unit, labour and delivery, immunization and in-patient. At the end of each month, data from these registers are collated manually in to the Monthly Reporting ActivityFootnote 1 (MRA) and also on Program Template forms. Thereafter, these forms are forwarded to the district office where data is captured electronically and their respective databases. MRA is captured into DHIS2 while program-specific data is captured on pre-designed Excel templates, hence forwarded to the regional level. At the regional level, data is also aggregated and synchronized to get the profile of the entire region before it is forwarded to the national level where analyses is done. We observed that the MRA is not comprehensive, as it does not contain all the data elements of health programs, as explained by one district manager:

“Previously, programs manage have their own reporting system but since the new CNIS director was appointed, he is trying to put some order in the system; starting by standardizing data collection tools. The data elements in MRA were selected based on the 100 indicator datasets as stipulated by WHO.”

Theme 1: Issues relating to Data Quality

Data Quality - Data review and giving feedback

It is essential to perform data quality checks. Exploring issues of data quality, we observed that data reviews and feedbacks are not standardized processes of data management. They are often done haphazardly and not frequently conducted at most facilities. District managers explain that due to lack of feedback, data clerks have cultivated the habit of falsifying data:

“When data clerks submit data and receive no feedback, they believe nobody checks the data submitted. Thus, they have cultivated the habit of cooking data and submitting.”

Despite being aware that neither data quality reviews are done nor feedbacks provided, district managers said they trust the quality of data: “I only type and submit I do not use data. I cannot say anything about the quality of data. But I trust the data submitted by the matron in-charge”

Theme 2: Understanding of and use of health indicators

Understanding indicators: Nurse Managers at District & Regional Health facilities

We observed that health managers at operational levels seldom discuss about indicators and do not share the same level of understanding of indicators. In particular, facility level managers interviewed were not aware that there is a difference between an indicator and data element. They were neither able to understand what makes up an indicator nor how an indicator is calculated. They were also not able to differentiate a data element from an indicator. During the discussions, they would refer to “indicators” but when probing it became apparent that what they meant were “data elements”. When asked to present a list of monthly priority indicators, a matron in-charge showed a list of monthly summary data element forms, noting: “these are all our indicators that we report monthly. I write everything here on these forms and submit to them.” A similar situation was observed among district program managers. When asked to give example of some of the SDGs indicators in the program she manages she noted:

“Last week I went to the hospital for a visit. As I was walking out of the hospital, I caught sight of this beautiful picture with the following caption “breastfeeding is a sustainable development goal as will reduce infant mortality”. That was the very first time I heard about sustainable development goals. I was so happy.”

While attending a monthly district meeting, Program managers were presenting monthly reports on facility visits conducted in their sub-districts. In the course of presenting these reports, they use phrases such as “low ANC coverage, low immunization coverage” as outcome of their visits. It was not clear from the presentations how they arrived at the decisions.

Use of indicators: Nurse Managers at Healthcare Facilities.

We also observed that health managers’ use of indicators was minimal. A nurse manager explained that she uses indicators when preparing her performance-based business plan to help allocate targets (excerpt of plan presented on table below). A general complaint was the difficulties to do the calculations as illustrated in Table 2 below. For example, column 2 depicts how population data are presented to managers at health facilities.

Table 2. Level of achievement of the objectives of the last quarter business plan
Full size table

A matron in-charge explains how she uses indicators:

“At the beginning of the year we have priority activities and also set which are reviewed annually. For example I know in this hospital, our consultation at the end of last year was 40,000. For this year, we should move up to about 45,000. If that is not achieved, we will know that we have not performed well and need to do something so that we can achieve more consultations. Two years ago, our consultations moved from 20,000 to 30,000. As a result of this increase, the director saw that the consultation area has become small. He expanded the waiting area, bought more chairs so that patients should sit down while waiting to be seen by a doctor.”

In the explanation above, the manager talked of “indicators” but what she is referring to are “data elements”. She added that to calculate indicators, they would use the performance of the previous year or quarter to evaluate their facility’s performance against set targets. To set indicator targets, it was observed that there are no clearly defined criteria to use. With reference to the Table 2 above, Columns 3 and 4, the nurse manager from another facility shared her experience:

“As we prepare the business plan for PBF, we have working sessions with the PBF team. They will say for example, on the indicator: “Outpatient consultations (new cases): nurse”, since we achieved 4025 in the last quarter, what do you think if we shift your objective for next quarter to 660 that is 4025 +660 = 4685. What do we do, we just have to accept? They do not take into consideration any other factor. It is our responsibility to ensure that the number cases are achieved at the end of the quarter. If we do not achieve it, we shall not benefit a single franc on this particular indicator.”

Another facility manager added:

“To set a target, we merely take the performance for the previous quarter, and add at least 5%. Five percentages is just an amount we decide to use. The idea is to encourage them to perform better than the previous quarter. However, there is a challenge because most often those targets sets are not achieved.”

Use of Indicators at District Level

District managers use indicators to compile quarterly reports as explained by one of them:

“Programs have specific indicator targets. Take for example, the indicator “immunization coverage.” I am a statistician so when data arrives my office, after capturing, I do my calculation base on the targets and population projection I have. Hence, I compare the results (figures) with what was submitted for the previous months to know when I have achieved the target or not.”

Another Program manager shared her experiences:

“For example, as a HIV/AIDS program manager, this program is a priority program. I have monthly targets that were set and sent to me from Yaoundé [the capital in Cameroon] at the beginning of the year. Consider, for example, the indicator, “Treatment coverage among HIV/AIDS positive pregnant women”. Based on the data submitted, at the end of each month, I calculate the indicator and send the report to the national office in Yaoundé.”

Furthermore, the district program manager explains:

“If for examples the targets are not met, I will have to wait for instructions from Yaoundé before going out or planning for an intervention because Yaoundé makes all the decisions.”

Use of Indicators at the National Level

A director at the national explained that all decisions are made at the central level, the reason why data analyses are done there; at the national level. He added that indicators are used to evaluate program, but lamented on the quality of data. For example, he mentioned that most of the indicators routinely collected for his program are irrelevant. He also added that district managers are not involved in decision-making as this is the responsibility of those at the central level:

“Districts are not involved in decision-making. Except for the HIV/AIDS program, where program managers at district level have targets. Reasons being it is a priority program, with lots of international funding from PEPFAR. However, about which SDGs my program is reporting against, SDGs have not yet been made published. They are still waiting for the indicators and targets.”

Further, he added, indicators are often discussed only when preparing annual reports. This is the responsibility of the national program manager for monitoring and evaluation.

6 Discussion

The study provides tangible examples of health managers’ understanding and use of indicators in healthcare settings. So how transferable are the findings? We found that these behavior were similar to other countries reported by Silvia et al. (2013), Thaizy et al. (2015).

The world’s political agenda for the next 15 years (2016–2030) on health systems’ strengthening is to reduce poverty through SDGs in general and UHC goals in particular. These are global initiative to monitor countries health performance. UHC are time-bound and measurable indicators to be monitored and reported against, annually by each country. Though internationally developed, most of the indicators are similar to those in a country’s operational plans. Thus, decision and interventions made by policy makers and healthcare providers should be based on these indicators.

At the operational level, data to generate these indicators come from various health interventions done at district and communities levels. Therefore, it is important that healthcare professionals understand indicators and also know how they are calculated, as it might help them to monitor and evaluate their performances, and in the event of an outbreak, they intervene immediately.

While managers are should understand health indicators, it is important the government provides reliable population-based data. In Cameroon, population-data is available for the regions, and for administrative units below the district, projected population estimates are provided instead, and it is the responsibility of the district or facility managers to calculate these estimates, who in most cases lack the necessary numeracy skills.

A reliable source of population data is through census, but census is not frequently conducted. Data available is either outdated or does not exist in administrative unit below the district. However, there are other reliable sources of population-data. Citing an example from the developed countries, in Norway, the civil registration and vital statistics (CRVS) systems are reliable sources of population data. The government has introduced a method whereby these systems are maintained through incentives given to bother public and private entities and citizens to engage in proper and timely registration of vital events (Nielsen et al. 2015). Thus, the saying goes “population-data is the true denominator for development” (Purcell 2016).

Population-based provide an important piece of mosaic of evidence for decision-makers as well as healthcare providers. It can be used to assess the magnitude of health problems of which population are most vulnerable, such as to track and evaluate the effectiveness of health intervention for UHC.

Furthermore, health care providers dealing with indicators need to understand their meanings and what goes into calculating it (Sahay et al. 2009). In addition, indicators should be standardized as it improves data quality (Maniz 2003; Flower et al. 2005).

7 Conclusion

The study found that healthcare managers have difficulties understanding data elements and indicators. In the absence of population data, they use alternative method to calculate indicators. Although managers have priority indicators they report on monthly, coverage indicators are not calculated using population as denominator. This study extends research on how health information is used by healthcare providers based on a case study in Cameroon. Though this study was limited to data and program managers at district and regional levels of the health systems, the findings raise issues that have wider applicability in the implementation of electronic health information system as well as how health indicators such as UHC goals are calculated.

The primary concerns of case study research centers around validity and reliability (Merriam 1985). To address these concerns; for validity, we employed the process of triangulation, i.e. the use of a variety of data sources (interviews, focus group discussions and documents review) as opposed to relying solely upon one source. We have also included verbatim quotations (Johnson 1997) in the analysis section, consulted with senior directors at the ministry of health. To ensure reliability, the following measures were employed; interviews were recorded and transcribed, and during data analysis themes identified were discussed and agreed among the authors, before it was included in the paper (Roberts et al. 2006).