You are on page 1of 12
RESEARCH ARTICLE Assessing the Quality of Care for Pneumonia in Integrated Community Case Management: A Cross-SectionalCreative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability Statement: Consent from participants was only granted to use the data for future research purposes where ethical approval has been granted. For qualifying researchers, access to the data will be guaranteed by contacting Dr Sylvia Meek, Malaria Consortium Technical Director, at s. meek@malariaconsortium.org . Funding: The study was conducted through COMDIS-HSD ( http://comdis-hsd.leeds.ac.uk ), a research programme consortium funded by UK aid from the UK government ( www.gov.uk/government/ Background Pneumonia is the leading infectious cause of mortality in children under five worldwide. Community-level interventions, such as integrated community case management, have great potential to reduce the burden of pneumonia, as well as other diseases, especially in remote populations. However, there are still questions as to whether community health workers (CHW) are able to accurately assess symptoms of pneumonia and prescribe appropriate treatment. This research addresses limitations of previous studies using inno- vative methodology to assess the accuracy of respiratory rate measurement by CHWs and provides new evidence on the quality of care given for children with symptoms of pneumo- nia. It is one of few that assesses CHW performance in their usual setting, with independent re-examination by experts, following a considerable period of time post-training of CHWs. Methods In this cross-sectional mixed methods study, 1,497 CHW consultations, conducted by 90 CHWs in two districts of Luapula province, Zambia, were directly observed, with measure- ment of respiratory rate for children with suspected pneumonia recorded by video. Using the video footage, a retrospective reference standard assessment of respiratory rate was conducted by experts. Counts taken by CHWs were compared against the reference stan- dard and appropriateness of the treatment prescribed by CHWs was assessed. To supple- ment observational findings, three focus group discussions and nine in depth interviews with CHWs were conducted. Results and Conclusion The findings support existing literature that CHWs are capable of measuring respiratory rates and providing appropriate treatment, with 81% and 78% agreement, respectively, organisations/department-for-international- between CHWs and experts. Accuracy in diagnosis could be strengthened through further PLOS ONE | DOI:10.1371/journal.pone.0152204 March 24, 2016 1 / 12 " id="pdf-obj-0-3" src="pdf-obj-0-3.jpg">

RESEARCH ARTICLE

Assessing the Quality of Care for Pneumonia in Integrated Community Case Management:

A Cross-Sectional Mixed Methods Study

Chomba Sinyangwe 1 , Kirstie Graham 2 , Sarala Nicholas 2 , Rebecca King 3 , Samuel Mukupa 1 , Karin Källander 2 , Helen Counihan 2 *, Mark Montague 4 , James Tibenderana 4 , Prudence Hamade 2

1 Malaria Consortium, Mansa, Luapula Province, Zambia, 2 Malaria Consortium, London, United Kingdom,
  • 1 Malaria Consortium, Mansa, Luapula Province, Zambia, 2 Malaria Consortium, London, United Kingdom,

  • 3 The Nuffield Centre for International Health & Development, University of Leeds, Leeds, United Kingdom,

  • 4 Malaria Consortium, Kampala, Uganda

* h.counihan@malariaconsortium.org

* h.counihan@malariaconsortium.org

 

Abstract

  • OPEN ACCESS

Citation: Sinyangwe C, Graham K, Nicholas S, King R, Mukupa S, Källander K, et al. (2016) Assessing the Quality of Care for Pneumonia in Integrated Community Case Management: A Cross-Sectional Mixed Methods Study. PLoS ONE 11(3): e0152204.

doi:10.1371/journal.pone.0152204

Editor: Guy N Brock, Ohio State University College of Medicine, UNITED STATES

Received: October 26, 2015 Accepted: March 10, 2016 Published: March 24, 2016

Copyright: © 2016 Sinyangwe et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability Statement: Consent from participants was only granted to use the data for future research purposes where ethical approval has been granted. For qualifying researchers, access to the data will be guaranteed by contacting Dr Sylvia Meek, Malaria Consortium Technical Director, at s. meek@malariaconsortium.org.

Funding: The study was conducted through COMDIS-HSD (http://comdis-hsd.leeds.ac.uk), a

research programme consortium funded by UK aid from the UK government (www.gov.uk/government/

Background

Pneumonia is the leading infectious cause of mortality in children under five worldwide. Community-level interventions, such as integrated community case management, have great potential to reduce the burden of pneumonia, as well as other diseases, especially in remote populations. However, there are still questions as to whether community health workers (CHW) are able to accurately assess symptoms of pneumonia and prescribe appropriate treatment. This research addresses limitations of previous studies using inno- vative methodology to assess the accuracy of respiratory rate measurement by CHWs and provides new evidence on the quality of care given for children with symptoms of pneumo- nia. It is one of few that assesses CHW performance in their usual setting, with independent re-examination by experts, following a considerable period of time post-training of CHWs.

Methods

In this cross-sectional mixed methods study, 1,497 CHW consultations, conducted by 90 CHWs in two districts of Luapula province, Zambia, were directly observed, with measure- ment of respiratory rate for children with suspected pneumonia recorded by video. Using the video footage, a retrospective reference standard assessment of respiratory rate was conducted by experts. Counts taken by CHWs were compared against the reference stan- dard and appropriateness of the treatment prescribed by CHWs was assessed. To supple- ment observational findings, three focus group discussions and nine in depth interviews with CHWs were conducted.

Results and Conclusion

The findings support existing literature that CHWs are capable of measuring respiratory rates and providing appropriate treatment, with 81% and 78% agreement, respectively,

Quality of Care for Pneumonia in iCCM

Quality of Care for Pneumonia in iCCM

development). KG, MM, SN, PH and RK were funded through this grant. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. This research study was conducted alongside implementation of iCCM in Luapula Province in Zambia funded by CIDA (now Global Affairs Canada, www.international.gc.ca) from 20092013 (grant number M012955-001). CS, SM and HC were funded through this grant. Some of the activities and materials for the study were therefore supported by funding from CIDA.

Competing Interests: The authors have declared that no competing interests exist.

training and the development of improved diagnostic tools appropriate for resource-poor settings.

Introduction

Pneumonia accounts for 15% of all childhood deaths and kills more children than any other infectious disease worldwide, with an estimated 935,000 deaths of children under five per year attributable to this infection in 2013 [1]. The World Health Organization (WHO) and UNICEF estimate that through implementation of several key interventions, a substantial proportion of childhood pneumonia deaths could be averted [2]. The Global Action Plan for Prevention and Control of Pneumonia [2] outlines these key interventions; case management at all levels (including community); vaccination; prevention and management of HIV infection; improve- ment of nutrition and reduction of low birth weight; and control of indoor air pollution. Community-level interventions have an important contribution to make through improv- ing accessibility, uptake and appropriate use of services. Evidence has shown that pneumonia case management alone of infants and preschool children by community health workers (CHWs) resulted in a 36% reduction in pneumonia mortality [3]. However, the necessity of targeting not only pneumonia but also other important causes of childhood illnesses (especially malaria and diarrhoea) in an integrated way has also been recognised [3, 4]. Additionally, within the community it has been shown that increasing the number of illnesses CHWs treat increases demand for their services [5]. One outcome of the growing emphasis on community-level interventions has been the inte- grated community case management (iCCM) strategy for malaria, pneumonia, diarrhoea in chil- dren under five years of age, sometimes including newborn health and nutrition [6]. iCCM is currently being rolled-out in several countries across Asia, South America and sub-Saharan Africa. The overall aim of iCCM is to support and strengthen community-based case manage- ment in hard-to-reach communities by providing free diagnosis and treatment to children under five years old, in addition to training, job aids and supervision of CHWs. The iCCM package also includes behaviour change communication to caregivers and the wider community. Interventions, such as iCCM, have great potential to increase access to health care and reduce the burden of pneumonia, especially in marginalised populations [6, 7]. However, increasing access to medicines must be implemented in parallel with equivalent efforts to understand and address the potential implications, which include ensuring children receive appropriate treatment for their condition, as well as defining the potential risk of contributing towards the development of drug resistance. Despite existing evidence, there remains an ongo- ing debate regarding the ability of CHWs to accurately diagnose the signs and symptoms of suspected pneumonia and provide appropriate treatment [8]. Evidence on CHWsability is still quite limited, with studies often having significant methodological constraints. In particu- lar, most have been conducted in formal health care settings and not in the CHWs usual place of work [911]. This approach has the potential to introduce considerable bias and brings into question the extent to which findings are transferable to the community setting. This research assessed the accuracy of respiratory rate measurement by CHWs using inno- vative methodology, and evaluated their ability to appropriately prescribe treatment for chil- dren with suspected pneumonia. It builds on previous studies to inform future implementation of iCCM, evaluating CHW performance in their usual context, with independent re-examina- tion by experts and after a significant period of time post training [9, 1215]. The research pre- sented in this manuscript is one component of a mixed methods cross-sectional study that

Quality of Care for Pneumonia in iCCM

Quality of Care for Pneumonia in iCCM

overall aimed to identify the percentage of CHWs who adhered to iCCM guidelines. Data exploring the rational use of antibiotics will be presented in a separate manuscript.

Materials and Methods

Study setting

Data was collected in Samfya and Kawambwa districts of Luapula province, Zambia between October and December 2012. Kawambwa and Samfya have a population of 130,680 and 191,980 and are in the north and south of the province, respectively [16].

Study design

1,497 CHWsconsultations, conducted by 90 CHWs over six weeks, were directly observed by nine non-clinical researchers in the CHWs usual place of work, with measurement of respira- tory rate by CHWs for 698 children recorded by video. The video camera was mounted on a tripod and placed in a discreet position prior to consultations. Researchers were instructed to record the measurement of respiratory rate by video for all children for whom the CHW con- ducted a respiratory rate count i.e. only those with suspected pneumonia. According to CHWstraining, the respiratory rate of a child should be measured using an acute respiratory infection (ARI) timer in all children presenting with cough or difficulty in breathing. For children 211 and 1259 months, a respiratory rate of 50 and 40 indicates suspected pneumonia, respec- tively. Researchers visited ten CHWs each, spending three days with each CHW. To ensure there were sufficient numbers of children with suspected pneumonia attending the CHW dur- ing the two day observation period, the community was sensitised during the first day to encourage them to bring any sick children under five years of age for treatment during the sub- sequent two days. Video footage was later used to conduct a retrospective reference standard assessment of respiratory rate by two pairs of experts. Experts were national-level certified mas- ter trainers in Integrated Management of Childhood Illnesses (IMCI) in Zambia. Each pair of experts viewed the video footage together, but recorded the respiratory rate count indepen- dently, and subsequently compared results. An average count was taken for expert counts within five breaths of each other. An inter-expert count variation of more than five prompted a re-assessment. For the most part, experts were blinded to the CHW count, as the respiratory rate measured by the CHW was recorded as a separate video clip following the initial record- ing. Three focus group discussions (FGDs) and nine in depth interviews (IDIs) with CHWs were also conducted to supplement observational findings. Researchers were all Zambian nationals from Luapula province with mixed experience of conducting research. However, all researchers took part in seven days of training, delivered in two parts, to ensure they had suffi- cient understanding to conduct the study. This included a detailed discussion on the appropri- ate translation of each of the questions in the semi-structured interview guides. All qualitative study components were audio-recorded, conducted in the local language (Bemba), and subse- quently translated and transcribed into English. The IDIs were transcribed verbatim and the FGDs were transcribed using the Fairnotes approach [17]. The CHWs who were selected as interview respondents were asked to participate at the end of the second day of observation. The FGDs were held once the observation component of the study was completed for all 90 CHWs.

Sample size

The sample size for the observation of CHWs was calculated to identify the proportion of CHWs who adhere to iCCM guidelines. A final sample size of 90 CHWs was calculated,

Quality of Care for Pneumonia in iCCM

Quality of Care for Pneumonia in iCCM

assuming 50% of CHWs would be adherent to iCCM guidelines, with 95% confidence, at a pre- cision of 10%, and including an adjustment of 10% to account for CHW non-response and data excluded from final analysis for any reason.

Sampling

CHWs were selected for observation by simple random sampling from the total number (N = 440) of CHWs trained in iCCM in the two study districts, as part of the iCCM programme in Luapula Province funded by the Canadian International Development Agency (CIDA) from 2009 to 2013. CHWs had received training on iCCM at least one year prior to the study. For the IDIs, nine CHWs in total were systematically sampled, with the eighth CHW observed by each researcher invited to participate. For the three FGDs, 24 CHWs were selected to participate by convenience sampling from all those observed in the observation component of the study.

Ethics

This study was approved by the School of Medicine Research Ethics Committee at the Univer- sity of Leeds and the University of Zambia Biomedical Research Ethics Committee. Written consent was obtained from each CHW prior to the observation period, and from each caregiver of children under five who presented during the two days. Additional written consent was obtained from the CHWs who were subsequently selected for the IDIs, and those who partici- pated in the FGDs.

Data analysis

Quantitative analysis. All quantitative data were analysed using Stata version 12 (Stata- Corp LP). Of the 1,497 consultations that were observed, seven (<0.005%) were excluded due to:

no data being recorded (n = 2), child was aged more than five years (n = 1), or if the childs age was not specified (n = 4). 698 videos were recorded where the CHW measured the respiratory rate of the child. Of this number, 537 videos (77%) were included in the analysis and reviewed by experts, to compare the counts taken by the CHW against a reference standard. 161 videos (23%) were excluded for various reasons that included poor video quality, missing information or child was breastfeeding, crying during measurement of the respiratory rate or where the expert ratings of respiratory rate differed by more than five counts and had not been recounted. Descriptive results are summarised as frequencies, percentages or as a median with inter- quartile range (IQR). Indicators are presented as percentages with 95% confidence intervals (CI) adjusting for clustering of children by CHW. Pearson chi-squared test accounting for clus- tering by CHW was used to assess association between each indicator and age group of the children. Scatterplots and Bland-Altman plots [18] were produced to assess the agreement in ratings of respiratory counts between experts and CHWs. The Bland-Altman method plots the differ- ence in ratings (y-axis) against the average of the ratings (x-axis) for every child observed. Superimposed upon these figures is the mean difference in ratings between experts and CHWs (the bias) and the 95% limits of agreement defined as +/-2 standard deviations of the bias. These limits indicate a plausible range of values where 95% of the observed differences should lie. The narrower the range of values, the better the agreement. Levels of significance were obtained by the adjusted Wald test. Furthermore, the proportion of respiratory rate counts by CHWs within five, three and two breaths of the expert count was calculated. Children were classified as having fast breathing if they presented with a respiratory rate of more than or equal to 50 breaths per minute or 40 breaths per minute for younger (211 months) and older children (1259 months) respectively. Children with normal breathing had

Quality of Care for Pneumonia in iCCM

Quality of Care for Pneumonia in iCCM

respiratory rates below these cut-off points. Cohens Kappa was obtained to assess the level of agreement between the classification of fast and normal breathing by experts and CHWs. Sen- sitivity and specificity analyses were also performed, to determine the proportion of children who were correctly classified as having fast or normal breathing. Fast breathing in children with cough or difficulty breathing is indicative of suspected pneu- monia and should be treated with antibiotics, as per the iCCM guidelines. The sensitivity and specificity of treatment given by the CHW was calculated. Sensitivity and specificity indicates the proportion of children observed who were correctly prescribed antibiotics for fast breath- ing, as assessed by an expert, and the proportion of children observed who had normal breath- ing and were not given antibiotics, respectively. Cohens Kappa was calculated to assess overall agreement of treatment given, indicating the proportion of children observed who were appro- priately treated, receiving antibiotics for fast breathing and not receiving antibiotics for normal breathing, as assessed by an expert. Qualitative data analysis. Qualitative data was managed using NVivo 10 (QSR Interna- tional) and analysed by thematic analysis. The coding frame consisted of both a priori and grounded codes, and was developed and agreed by RK, KG and CS based on an initial analysis of three IDI transcripts. The remainder of the transcripts were analysed by KG, with any changes to the coding frame or queries discussed with RK and CS as necessary. Summaries of each theme were also reviewed by each researcher before data was consolidated and written up. Unfortunately, it was not possible for the study participants to review final transcripts or pro- vide feedback on study findings.

Results

Participants

The median age of the 90 CHWs observed was 44 years (IQR: 3849) and 19% were female. The majority (77%) had attained a secondary education and 90% worked in the same commu- nity as from where they originated. 99% of CHWs reported that they had worked for more than one year as a CHW and 44% reported that they had worked more than five years as a CHW. All participants had received the six-day iCCM training supported by Malaria Consor- tium as part of the CIDA-funded implementation of iCCM in Luapula Province. However, 79% of the CHWs had also previously received six-week CHW training from the Ministry of Health. This proportion increased to 87% if CHW and community IMCI training were consid- ered. The median number of types of training received was three and only one CHW reported having no additional training. The median number of children observed per CHW was 14 (IQR: 1021) during the two days of observation. 50% of children observed were male and 25% were less than 12 months of age (Table 1). The characteristics of all children were similar, regardless of whether or not the measurement of respiratory rate was recorded on video. There were nine in depth interviews conducted with CHWs, seven were male, one was female, and the gender of one participant was not recorded by the researcher. The FGDs involved 24 CHWs, consisting of seven females and 17 males, with eight participants per group. IDIs and FGDs were approximately 40 minutes and two hours in length, respectively.

Assessment of respiratory rate in children: comparison of CHW and expert respiratory rate counts

Overall, there is a strong correlation between expert and CHW respiratory counts (Pearson correlation coefficient = 0.78) (Fig 1A). The mean difference in respiratory rate counts between

Quality of Care for Pneumonia in iCCM

Quality of Care for Pneumonia in iCCM

Table 1. Characteristics of children observed.

 

All children (N = 1490)

Children on video (N = 537)

Other (N = 952)

Childs age

n

%

n

%

n

%

211 months

368

25

147

27

221

23

1259 months

1122

75

390

73

732

77

 

All children (N = 1486)

Children on video (N = 535)

Other (N = 951)

Childs sex

n

%

n

%

n

%

Male

739

50

260

48

479

50

Female

747

50

275

51

472

50

 

All children (N = 1480)

Children on video (N = 536)

Other (N = 944)

Guardian

n

%

n

%

n

%

Mother

1380

93

503

94

877

93

Father

55

4

21

4

34

4

Other

46

3

12

2

34

4

doi:10.1371/journal.pone.0152204.t001

experts and CHW (bias) as obtained by the Bland-Altman method (Fig 1B) was -0.74 (95% CI:

-1.96 to 0.49), which is not statistically different from 0 (Adjusted Wald Test, p = 0.23). How- ever, the variability (as shown by the scatter around the mean difference) is not consistent; while CHWscounts are in general higher than those of experts, when average respiratory counts are greater than 60, CHWscounts are lower than experts. Further, the standard devia- tion of the mean difference is large (SD = 9.0) giving a wide limits of agreement between -18.8 to 17.3. Examining the percentage of CHW respiratory counts that are within two, three or five breaths of expert count showed 46% (95% CI: 39 to 53) were within two breaths and 55% (95% CI: 46 to 63) and 67% (95% CI: 58 to 75) were within three or five breaths respectively.

Quality of Care for Pneumonia in iCCM Table 1. Characteristics of children observed. All children (N

Fig 1. Relationship between expert and CHW respiratory count for all children. (A) Scatterplot, (B) Bland Altman plot.

doi:10.1371/journal.pone.0152204.g001

Quality of Care for Pneumonia in iCCM

Quality of Care for Pneumonia in iCCM

When examined by age group, the correlation between respiratory rate counts by experts

and CHWs is also high at 0.74 and 0.70 for children aged 211 months and 1259 months respectively. The mean difference in ratings for children aged 211 months is 2.04 (95% CI:

  • 0.02 to 4.06) and -1.77 (95% CI: -2.92 to -0.64) for 1259 months. Both are significantly differ-

ent from 0 (Adjusted Wald Test, p = 0.05 and p = < 0.01, respectively). For children aged 211

months, the data implies that CHWs consistently scored respiratory rates lower than experts compared with the older age group, where it suggests that CHWs consistently scored respira- tory rates higher than experts. For both groups, there is inconsistent and wide variation in the

difference in ratings, with the limits of agreement lying between -19.7 and 23.8 for the younger group and -17.7 and 14.2 for children aged 1259 months. In terms of the percentage of CHW respiratory counts that were within two, three or five breaths of experts, for the younger age group, this equates to 40% (95% CI: 32 to 49), 50% (95% CI: 40 to 60) and 61% (95% CI: 51 to 71) respectively. For the older age group, 48% (95% CI:

40 to 57), 56% (95% CI: 48 to 65) and 69% (95% CI: 60 to 77) were within two, three or five breaths respectively. In terms of fast breathing, 40% of children (213/537) were classified as having fast breathing by CHW compared to 33% (180/537) by experts. Both CHWs and experts found a higher per- centage of children with fast breathing in the younger age group (p = < 0.01). CHWs diag- nosed 54% of children aged 211 months to have fast breathing compared to 34% of children aged 1259 months. Experts diagnosed 56% of children aged 211 months to have fast breath- ing compared to 25% of children aged 1259 months. CHWs were 81% in agreement with experts when classifying children as having fast or nor- mal breathing, with a sensitivity and specificity of 81% and 81%, respectively (Table 2). Similar agreement was found amongst younger and older children, however sensitivity was slightly lower when CHWs diagnosed older children. Cohens Kappa statistic ranged between 0.54 to

  • 0.63 denoting moderate agreement.

When the participants of the FGDs were asked to describe how a child with pneumonia would be diagnosed, one or two participants took the lead and described the process in depth. Whilst the others were strongly in agreement, it does not give a representative assessment of knowledge within the group, only of one or two individuals. However, the knowledge of those who responded was quite high and followed the key steps as outlined in their training materials. All three of the CHW focus groups noted the need to count the respiratory rate or the number of breaths to diagnose suspected pneumonia, and the use of an ARI timer. Furthermore, all groups detailed correctly the different categories for fast breathing depending on the age of the child. Two of the focus groups also mentioned asking the caregiver to hold the child still or to ensure that there is no movement, taking off the child's

Table 2. CHW classification of fast and normal breathing compared to expert classification.

 

All children

 

Children 211 months

 

Children 1259 months

Expert

Expert

Expert

CHW

Fast breathing

Normal breathing

Fast breathing

Normal breathing

Fast breathing

Normal breathing

Fast breathing

145

68

67

12

78

56

Normal breathing

35

289

15

53

20

236

Agreement (95% CI)

81 (7685)

 

82 (7388)

 

81 (7585)

Kappa (κ) value

0.59

0.63

0.54

Sensitivity (95% CI)

81 (7387)

 

82 (6990)

 

80 (7087)

Specicity, (95% CI)

81 (7586)

 

82 (7089)

 

81 (7486)

doi:10.1371/journal.pone.0152204.t002

Quality of Care for Pneumonia in iCCM

Quality of Care for Pneumonia in iCCM

clothes, asking the age of the child and checking for chest in-drawing, which indicates the need for urgent referral. One of the focus groups also added the need to restart the timer if the child moves. Another specifically mentioned looking for fast breathing and counting the breaths for one minute. The in depth interviews with individual CHWs also seemed to indicate a high level of knowledge regarding pneumonia diagnosis and assessment of respiratory rate. Some partici- pants shared more detail than others, thus appearing more knowledgeable, however this is not necessarily indicative of a higher level of knowledge as these are qualitative assessments. Nevertheless, it is interesting to note that of the CHWs who specified the age categories for fast breathing, all except one were correct. The CHW who was not correct simply said that "I count when the child breathes out and when the child breathes 35 per minute then I know that the child has pneumonia" (CHW IDI 6). Another CHW highlighted that "we know the differ- ence in breaths per minute according to the child's age" (CHW IDI 7), but did not elaborate further with specific age groups or number of breaths. Furthermore, all of the CHWs men- tioned that they would count the breaths of the child and the majority said they would use a timer, although one called it a "heart timer" (CHW IDI 1). A few further elaborated that they would count breaths for one minute. On the other hand, there were a few who did not mention asking the age of the child, or defined age categories for fast breathing. Also, only a few individuals specifically mentioned chest indrawing, or referring to checking "how the patient is looking under the ribs" (CHW IDI 9). There were a couple of others, in addition to these few, who highlighted that they would need to remove the child's clothes "so that the stomach and chest are exposed" (CHW IDI 8), which is in line with the normal procedure for checking respiratory rate, but may also imply that they were looking for chest indrawing.

Treatment of suspected pneumonia

Appropriate treatment is defined as receiving antibiotics for fast breathing, as assessed by an expert, and not receiving antibiotics for normal breathing. Analysis indicated that 78% of chil- dren were appropriately treated following measurement of their respiratory rate, indicating overall agreement (Table 3). There was no significant difference between the two age groups (p = 0.80). Cohens Kappa statistic was 0.51 indicating moderate agreement. Based on the expert assessment of respiratory rate, the sensitivity of treatment prescribed by CHWs was 73% (Table 3). There was no significant difference (p = < 0.01) between the pro- portion of children with fast breathing, as assessed by an expert, who were correctly prescribed antibiotics between the two age groups. Specificity of treatment, with children with normal breathing not receiving antibiotics, was 80%.

Table 3. Medication prescribed according to assessment of fast breathing by expert and child's age.

 

All children

Children 211 months

 

Children 1259 months

Expert

Expert

Expert

Received antibiotics

Fast breathing

Normal breathing

Fast breathing

Normal breathing

Fast breathing

Normal breathing

Yes

131

71

62

14

69

57

No

49

286

20

51

29

235

Agreement (95% CI)

78 (7382)

77 (6784)

78 (7283)

Kappa (κ) value

0.51

0.54

0.47

Sensitivity (95% CI)

73 (6381)

76 (6186)

70 (5979)

Specicity (95% CI)

80 (7386)

78 (6588)

81 (7386)

doi:10.1371/journal.pone.0152204.t003

Quality of Care for Pneumonia in iCCM

Quality of Care for Pneumonia in iCCM

Discussion

The level of agreement found in this study between CHWsclassification of children with fast and normal breathing, and expert classification (reference standard) (81%) is within the range shown in other studies of 75 to 85% [9, 12, 14, 15]. Table 4 presents a detailed comparison of the data collected in this study with existing literature. The findings from our study are compa- rable, taking into account the size of the study, location of observations, the duration of time post-training when the study took place and the methodology used to assess performance of CHWs. Whilst the mean difference of -0.74 in respiratory rate counts between experts and CHWs is not statistically significant from 0, the limits of agreement are wide and there is inconsistent variation in the differences over the range of measurements, showing poor agreement between CHWs and experts. However, it is worth noting that achieving strong agreement on respiratory counts is difficult, even between experts [19]. The findings suggest that there is a tendency for CHWs to measure rates closer to 60 per minute, regardless of whether the respiratory rate is high or low. This may be due to the fact that the ARI timer used to measure the respiratory rate ticks once per second, however, this finding was also reported by Noordam et al. [20] regardless of the device used to assess respiratory rate. The data indicate that there is a significantly higher percentage of younger children with fast breathing, as documented in the literature [21]. Also, it implies that younger children are potentially harder to diagnose as the variability between expert and CHW respiratory rate counts is slightly greater, however the sensitivity and specificity for the classification of fast breathing were not affected. The qualitative findings from FGDs and IDIs with CHWs also indicate that there is a good understanding of how to diagnose suspected pneumonia and measure respiratory rate, includ- ing the different cut-off values for fast breathing according to age. The data presented here provides further evidence that CHWs are capable of assessing chil- dren with suspected pneumonia using the tools provided, but accuracy of respiratory rate mea- surement could be strengthened through further training and the development of improved diagnostic devices, such as revised respiratory rate counters, appropriate for use by CHWs in these remote contexts.

Table 4. Comparison of data with literature on CHW performance in measuring respiratory rate.

Country

No of

No of

Method a

Time

Location Sensitivity

Specicity

Agreement

κ-

+/- 5

+/- 3

+/- 2

CHWs

consultations

post-

(%)

(%)

(%)

value

breaths

breaths

breaths

 

(total)

training

(%)

(%)

(%)

Zambia

90

538

DO+RE

>1 year

Community

81

81

81

0.58

67

55

46

Malawi [12]

131

382

DO+RE

Max. 23

Community

59

82

77

0.35

-

-

-

 

months

Uganda [14]

57

-

DO+RR

-

Health

-

-

75

-

49

39

-

 

facility

Uganda [9]

14

13 (182)

DO

3 days

Health

-

-

84.6

0.665

-

-

64

 

facility

Uganda [15]

96

576

DO

2 days

Health

75

83

79

0.75

71

-

-

 

facility

Bangladesh

120

1,166

DO

64.5

Community

67.7 c

95.2 c

89 c

--

-

-

[13]

months b

a RR: register review, RE: re-examination, DO: direct observation b median experience c cases classied as: mild, severe, very severe

doi:10.1371/journal.pone.0152204.t004

Quality of Care for Pneumonia in iCCM

Quality of Care for Pneumonia in iCCM

This study contributes new evidence to the existing literature on the quality of care provided for children with suspected pneumonia by CHWs at community level through iCCM. A num- ber of studies have conducted assessments of CHWsability to accurately measure and classify respiratory rates of children. However, these studies rarely evaluate CHWsperformance in their day-to-day environment, without direct and immediate validation of respiratory rate by a clinician, and seldom after a considerable period of time has elapsed post-training [9, 1315]. For example, in a study conducted by Mukanga et al.[9], 14 CHWs were directly observed at a health facility by a paediatrician and laboratory scientist, two weeks post training, as the objec- tive was to assess CHW competence after training. It has been recommended that studies are conducted in real life settingto gain a more accurate picture of their normal performance level [9, 15]. Rowe et al.[22] found that CHWs were less prone to errors in a hospital setting, with direct or non-direct observation, compared with working unobserved in their usual con- text. Furthermore, as detailed by Cardemil et al.[12], direct observation with re-examination was found to be the most reliable and appropriate method for accurate assessment of CHW performance in diagnosing uncomplicated cough and fast breathing. This assessment of meth- ods was conducted in the CHWs day-to-day context, rather than a health facility, with the chil- dren involved representing a similar case mix to normal [12]. In our study, the observation of CHW consultations took place in the CHWs place of work and involved direct observation with re-examination. It is also hoped that the Hawthorne effect [23] of behaviour changing as a result of being observed, was minimised as only one non-clini- cal researcher was involved and was present with each CHW for three days, and expert assess- ment of the respiratory rate was conducted independently at a later date. Furthermore, research by Pringle and Stewart-Evans [24] suggests that the use of a video to observe consulta- tions does not affect health worker performance, although it is acknowledged that these find- ings are from a developed country context with higher-level health workers. While our study involved slightly less CHWs than other studies, approximately 1,500 sick child consultations were observed in total, with measurement of respiratory rate recorded in 698. CHWs involved in the study had not received any formal training for at least one year, but did receive monthly supportive supervision from health facility staff. When prescription of antibiotics is based on the reference standard classification of fast breathing, a high percentage of children (73%) were correctly prescribed antibiotics (sensitiv- ity). This tends to be higher than is reported in literature, with 40% and 63% reported by Mukanga et al. and Cardemil et al., respectively [9, 12]. Specificity of treatment in our study was 80%, compared with 75% reported by Cardemil et al [12]. This high level of specificity is encouraging, and helps to address residual concerns regarding the use of antibiotics at commu- nity level and the risk of over prescription. Furthermore, considering both children with or without fast breathing, a high proportion were appropriately treated (78%) (overall agreement). In conclusion, the findings of the study indicate that CHWs can provide appropriate treat- ment for children under five years old with suspected pneumonia, however further training and the development of improved or new diagnostic tools, such as pulse oximeters [25], for assessing symptoms of pneumonia for use by CHWs in remote settings in low and middle income countries could further assist the accuracy of diagnosis and the appropriateness of treatment provided. This study has a number of limitations. In terms of sampling, CHWs located on islands in the swamps in Samfya were not included, as it was not logistically possible to reach them within the timeframe and resources available. Also, stockouts were not assessed during this study, so it is not possible to determine if these influenced treatment given by the CHWs. It is acknowl- edged that CHWs overall classification of disease (i.e. pneumonia, malaria, diarrhoea) was not

Quality of Care for Pneumonia in iCCM

Quality of Care for Pneumonia in iCCM

recorded as part of this study, only the measurement of respiratory rate and the subsequent action taken. The data presented here also does not intend to comment on CHWs overall iden- tification and assessment of diagnosis of pneumonia symptoms (i.e. cough, chest indrawing). Furthermore, a large number of the videos collected needed to be excluded from the analysis, in part due to the challenges of collecting data within this context. In addition, it is appreciated that the results from the FGDs are not representative of all participants, as they tended to be led by a few participants, whilst others were less actively involved.

Acknowledgments

We acknowledge the hard work of the community health workers in Luapula Province, Zam- bia, as well as the participation of the caregivers and children in this study. Also, we appreciate the support of Dr Penelope Kalesha, Child Health Specialist from the Ministry of Community Development, Mother and Child Health in Zambia, in helping to identify appropriate experts to retrospectively assess respiratory rate. Thanks also to Dr Sylvia Meek for providing valuable oversight.

Author Contributions

Conceived and designed the experiments: CS KG PH MM JT. Performed the experiments: CS KG SM. Analyzed the data: KG CS SN RK HC KK PH. Contributed reagents/materials/analysis tools: SN KG RK CS. Wrote the paper: CS KG SN RK SM KK HC MM JT PH.

References

  • 1. Liu L, Oza S, Hogan D, Perin J, Rudan I, Lawn JE, et al. Global, regional, and national causes of child mortality in 200013, with projections to inform post-2015 priorities: an updated systematic analysis. Lancet. 2014; 385(9996): 43040.

  • 2. World Health Organization, United Nations Children's Fund. Global action plan for prevention and con- trol of pneumonia. Geneva, Switzerland: World Health Organization; 2009.

  • 3. Sazawal S, Black RE, Pneumonia Case Management Trials Group. Effect of pneumonia case manage- ment on mortality in neonates, infants, and preschool children: a meta-analysis of community-based tri- als. Lancet Infect Dis. 2003; 3(9): 54756. PMID: 12954560

  • 4. World Health Organization, United Nations Children's Fund. WHO/UNICEF Joint Statement: Manage- ment of pneumonia in community settings. New York, USA: United Nations Children's Fund/World Health Organization; 2004.

  • 5. Curtale F, Siwakoti B, Lagrosa C, LaRaja M, Guerra R. Improving skills and utilization of community health volunteers in Nepal. Soc Sci Med. 1995; 40(8): 111725. PMID: 7597465

  • 6. World Health Organization, United Nations Children's Fund. WHO/UNICEF joint statement: integrated community case management (iCCM). New York, USA: United Nations Children's Fund; 2012.

  • 7. Das J, Lassi Z, Salam R, Bhutta Z. Effect of community based interventions on childhood diarrhea and pneumonia: uptake of treatment modalities and impact on mortality. BMC Public Health. 2013; 13 (Suppl 3): S29. doi: 10.1186/1471-2458-13-S3-S29 Epub 2013 Sep 17. PMID: 24564451

  • 8. Druetz T, Siekmans K, Goossens S, Ridde V, Haddad S. The community case management of pneu- monia in Africa: a review of the evidence. Health Policy Plan. 2013; 30(2): 25366. doi: 10.1093/ heapol/czt104 PMID: 24371218

  • 9. Mukanga D, Babirye R, Peterson S, Pariyo GW, Ojiambo G, Tibenderana JK, et al. Can lay community health workers be trained to use diagnostics to distinguish and treat malaria and pneumonia in chil- dren? Lessons from rural Uganda. Trop Med Int Health. 2011; 16(10): 123442. doi: 10.1111/j.1365- 3156.2011.02831.x PMID: 21752163

  • 10. Kelly J, Osamba B, Garg R, Hamel M, Lewis J, Rowe S, et al. Community health worker performance in the management of multiple childhood illnesses: Siaya district, Kenya, 19972001. Am J Public Health Res. 2001; 91(10): 161724.

  • 11. Rowe SY, Kelly JM, Olewe MA, Kleinbaum DG, McGowan JE, McFarland DA, et al. Effect of multiple interventions on community health workers' adherence to clinical guidelines in Siaya district, Kenya. Trans R Soc Trop Med Hyg. 2007; 101(2): 188202. PMID: 17064747

Quality of Care for Pneumonia in iCCM

Quality of Care for Pneumonia in iCCM

  • 12. Cardemil CV, Gilroy KE, Callaghan-Koru JA, Nsona H, Bryce J. Comparison of methods for assessing quality of care for community case management of sick children: an application with community health workers in Malawi. Am J Trop Med Hyg. 2012; 87(5 Suppl): 12736. doi: 10.4269/ajtmh.2012.12-0389 PMID: 23136288

  • 13. Hadi A. Management of acute respiratory infections by community health volunteers: experience of Bangladesh Rural Advancement Committee (BRAC). Bull World Health Organ. 2003; 81(3): 1839. PMID: 12764514

  • 14. Kalyango JN, Rutebemberwa E, Alfven T, Ssali S, Peterson S, Karamagi C. Performance of community health workers under integrated community case management of childhood illnesses in eastern Uganda. Malar J. 2012; 11: 282. doi: 10.1186/1475-2875-11-282 PMID: 22905758

  • 15. Kallander K, Tomson G, Nsabagasani X, Sabiiti JN, Pariyo G, Peterson S. Can community health work- ers and caretakers recognise pneumonia in children? Experiences from western Uganda. Trans R Soc Trop Med Hyg. 2006; 100(10): 95663. PMID: 16455119

  • 16. Republic of Zambia Central Statistical Office. Zambia 2010 census of population and housing: prelimi- nary population figures. Lusaka, Zambia: Republic of Zambia Central Statistical Office; 2011.

  • 17. Halcomb EJ, Davidson PM. Is verbatim transcription of interview data always necessary? Appl Nurs Res. 2006; 19(1): 3842. PMID: 16455440

  • 18. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986; 327(8476): 30710.

  • 19. Simoes EA, Roark R, Berman S, Esler LL, Murphy J. Respiratory rate: measurement of variability over time and accuracy at different counting periods. Arch Dis Child. 1991; 66(10): 1199203. PMID:
    1953002

  • 20. Noordam AC, Barberá Laínez Y, Sadruddin S, van Heck PM, Chono AO, Acaye GL, et al. The use of counting beads to improve the classification of fast breathing in low-resource settings: a multi-country review. Health Policy Plan. 2015; 30(6): 696704. doi: 10.1093/heapol/czu047 Epub 2014 Jun 27. PMID: 24974104

  • 21. Walker CLF, Rudan I, Liu L, Nair H, Theodoratou E, Bhutta ZA, et al. Global burden of childhood pneu- monia and diarrhoea. Lancet. 2013; 381(9875): 140516. doi: 10.1016/S0140-6736(13)60222-6 PMID:
    23582727

  • 22. Rowe SY, Olewe MA, Kleinbaum DG, McGowan JE Jr., McFarland DA, Rochat R, et al. The influence of observation and setting on community health workers' practices. Int J Qual Health Care. 2006; 18(4): 299305. PMID: 16675475

  • 23. Earl-Slater A. The handbook of clinical trials and other research. Abingdon, UK: Radcliffe Medical Press Ltd; 2002.

  • 24. Pringle M, Stewart-Evans C. Does awareness of being video recorded affect doctors' consultation behaviour? Br J Gen Pract. 1990; 40(340): 4558. PMID: 2271278

  • 25. Floyd J, Wu L, Hay Burgess D, Izadnegahdar R, Mukanga D, Ghani AC. Evaluating the impact of pulse oximetry on childhood pneumonia mortality in resource-poor settings. Nature. 2015; 528(7580): S53S9. doi: 10.1038/nature16043 PMID: 26633766