REPORTS

BRIEF

Using a Learning Coach to Teach Residents Evidence-based Medicine

Paul George, MD; Shmuel Reis, MD, MHPE; Melissa Nothnagle MD, MSc

BACKGROUND AND OBJECTIVES: Medical educators have used multiple interventions to teach evidence-based medicine (EBM) and information mastery. Most of these interventions are applied uniformly to a group of residents. We developed a curriculum to increase residents EBM and information mastery skills that would meet individual learning needs. METHODS: Two cohorts of second year residents (n=26) in the Family Medicine Residency Program at Brown University participated in the intervention. We delivered the curriculum through monthly individual meetings with a learning coach over 1 year. Before and after participating, residents completed a survey assessing their attitudes toward EBM and a quiz assessing their knowledge of EBM. Semi-structured interviews with each resident were done after the intervention and analyzed using qualitative methods. RESULTS: At the conclusion of our intervention, residents attitudes toward EBM and information mastery were overwhelmingly positive, EBM knowledge quiz scores increased by 31.8%, and reported use of EBM in real time during patient encounters increased. CONCLUSIONS: An intervention using a learning coach to provide one-on-one EBM and information mastery instruction to residents improved residents attitudes, knowledge, and use of both in the clinical setting. (Fam Med 2012;44(5):351-5.)

vidence-based medicine (EBM) is the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients.1 Educators have used multiple interventions to teach EBM. However, while many physicians believe that EBM is important, translating EBM skills into practice has proven difficult. Lack of time, knowledge, and resources contribute to physician difficulty implementing EBM.2 EBM proponents argue that it can optimize patient care.3 Additionally, the Accreditation Council for Graduate Medical Education
FAMILY MEDICINE

(ACGME) requires that residents demonstrate competency in information mastery or the ability to locate, appraise, and assimilate evidence from scientific studies.4,5 Residency curricula to promote EBM and information mastery skills show promise. A community-based internal medicine residency using a multi-component EBM intervention improved residents practice of EBM.6 Another intervention demonstrated efficacy of team mentoring in teaching EBM skills.7 A family medicine residency redesigned its EBM curriculum with the introduction of resident workshops and Web-based

tools and found improved resident comfort with EBM.8 Other interventions also demonstrate success in improving residents EBM skills.9-12 In our family medicine residency, we noted wide variability in our residents prior experience and level of comfort with EBM. We did not feel that standardized curriculum aimed at the entire group would meet individual learning needs. In the context of a larger intervention to promote self-directed learning skills,13 we set out to increase residents skill in finding and applying the best medical evidence to the care of patients in real time. We created the role of a learning coach, a family physician skilled in EBM, who would facilitate individual resident learning of EBM and information mastery skills. We based the learning coachs role on Collins cognitive apprenticeship model. Collins proposes six strategies to develop learners cognitive skills: modeling, coaching, scaffolding, articulation, reflection, and exploration.14,15 The learning coach used these methods during meetings with residents. First, he modeled use of EBM skills and then coached residents by providing specific feedback

From the Department of Family Medicine, Memorial Hospital of Rhode Island and Alpert Medical School, Brown University (Drs George and Nothnagle); and Department of Family Medicine, Clalit Health Services, Haifa District and Technion-Israel Institute of Technology, Haifa, Israel (Dr Reis).

VOL. 44, NO. 5 MAY 2012

351

BRIEF REPORTS

on their EBM skills. This one-on-one teaching allowed optimal scaffolding, as the coach tailored initial support for learning EBM to individual needs and withdrew support as a learner gained competence. Articulation involved questioning residents and stimulating them to ask questions. Finally, the coach encouraged residents to reflect on their strengths and weaknesses in EBM and helped them explore and formulate their own learning goals. This paper describes the design, implementation, and evaluation of a curricular intervention tailored to individual residents using a learning coach to develop their EBM skills.

Table 1: Evidence-based Medicine (EBM) Curriculum Topics

EBM Curriculum Introduction to Evidence-based Medicine Introduction to Online Evidence-based Medicine Resources Introduction to PubMed Basic Biostatistics Introduction to Guidelines Formulating PICO Questions Study Design Clinical Question Search Strategies

Methods

We conducted this study from 2008 2010 at the Brown University Family Medicine Residency, which has 39 residents and is based in a community hospital in Rhode Island. The intervention group consisted of two classes of second-year residents. The hospitals Institutional Review Board approved the study, and participants gave informed consent. Based on a review of published EBM curricula and on discussions held with faculty to assess resident EBM learning needs, we chose topics for inclusion, emphasizing information mastery skills (see Table 1). The learning coach delivered the curriculum to residents through monthly 1-hour meetings, with half of each meeting devoted to EBM training. Meetings began with the coach soliciting a clinical question from the resident, then helping refine the question and use electronic resources to search for answers. Additionally, the coach covered one topic per meeting from the curriculum to ensure that each resident received a standardized EBM curriculum. We selected the learning coach, a family physician and recent residency graduate, because of his EBM skills and trusted relationship with residents. A senior physician and medical educator (SR) mentored the coach for 1 year prior to and during the intervention.
352 MAY 2012 VOL. 44, NO. 5

We evaluated the curriculum using both quantitative and qualitative methods. To detect changes in resident attitudes and behaviors regarding EBM and information mastery, we administered a 10-item survey using a 5-point Likert scale. Residents completed surveys at the beginning and end of the year during which they received the curriculum. Each resident also completed an 18item EBM knowledge quiz before and after the curriculum (available from the corresponding author on request). Six questions were adapted from the Fresno Test of Competence in Evidence-Based Medicine.16 We developed the remaining questions based on discussions with residency faculty. The knowledge test was not designed to be summative; thus we did not use a passing score nor a standard-setting procedure. Faculty reviewed the evaluation tools to ensure acceptability and comprehensibility of the content. Two investigators independently scored all quizzes. To assess changes in EBM skills over time, the learning coach rated residents EBM ability after each meeting using an anchored 5-point scale. He completed the ratings immediately following each coaching meeting and did not view them again until the intervention was complete. We measured changes in resident attitudes and knowledge on the pre- and post-intervention survey and knowledge quiz using Wilcoxon signed ranks tests and used

intraclass correlation to assess interrater reliability on knowledge quiz scores. We used time-series analysis to assess changes in the coachs ratings from the first to last sessions. An independent interviewer met with the first cohort of participants 6 months after the intervention to assess the impact of the larger intervention on self-directed learning. Two investigators independently read all the transcripts and extracted comments pertinent to the EBM curriculum. They independently reviewed these comments and identified themes, then met together to create a comprehensive list of themes discussed by the residents, reaching consensus on these through discussion and review of original transcripts.

Results

Twenty-six residents participated in the curriculum and attended an average of 5.5 meetings with the coach (range three to eight sessions). Twenty-one residents (81%) completed both pre- and post-intervention surveys and quizzes. Resident surveys demonstrated favorable attitudes about EBM at baseline, with significant improvements in several items after the intervention (Table 2). Residents demonstrated improved knowledge of EBM, as evidenced by significant improvement in total scores on the EBM quiz from baseline to after the intervention by 31.8% (P<.001) (Table 3). Inter-rater reliability using intraclass correlation was 0.75.

FAMILY MEDICINE

BRIEF REPORTS

Table 2: Resident Pre- and Post-intervention Surveys of Evidence-based Medicine (EBM) Attitudes
Pretest* Learning EBM is an important component of my residency training. Learning EBM is important for my professional development. Using EBM is important in making medical decisions in my outpatient clinic. Using EBM is important in making medical decisions on the inpatient service. I frequently look up the evidence for the medical decisions I make in my outpatient clinic. I frequently look up the evidence for medical decisions I make on the inpatient service. Practicing EBM is time consuming. I can easily navigate online resources for EBM. I feel comfortable interpreting the positive predictive value and negative predictive value of a diagnostic test. I feel comfortable interpreting P values.
n=21 * 1=strongly disagree, 2=do not agree, 3=neutral, 4=agree, 5=strongly agree Wilcoxon signed rank test Statistically significant change

Posttest* 4.81 4.90 4.76 4.76 4.19

P Value .32 .56 .66 .41 .01 .11 .35 .01 .01 .01

4.71 4.86 4.81 4.67 3.71 3.76 3.62 3.43 3.10 3.33

4.14 3.33 4.10

3.81 4.05

Table 3: Resident Pre- and Post-evidence-based Medicine (EBM) Quiz Questions

Evidence-based Medicine Quiz Write a focused clinical question that will help you organize a search of the clinical literature for an answer to whether a strep culture should be performed. (Maximum score=3) Name as many resources as you can think of that you may go to find an answer to the above question. Describe the advantages and disadvantages of each. (Maximum score=6) Total quiz score (Maximum score=41)
n=21 * Wilcoxon signed rank test Statistically significant change NAnot applicable

Pre-EBM Intervention 1.05

Post-EBM Intervention 1.81

Percent Change NA

P Value* .003

3.00

4.24

NA

.004

18.05

23.71

31.82%

<.001

Learning coach ratings of the residents EBM skills increased from a mean of 2.0 (representing almost complete dependence on the learning coach) during session one to a mean of 4.2 (indicating self-directed EBM skills) during the final session (P<.001) (Figure 1). These ratings indicate progressive improvement in EBM skills and knowledge through the intervention. Eleven of the 12 year-one participants completed post-intervention interviews. Table 4 includes themes and illustrative quotes. Common themes included the importance of

one-on-one coaching in delivering the curriculum and a change in residency culture placing greater value on EBM.

Discussion

Our intervention, using a learning coach to teach EBM one on one, proved both feasible and effective. Although each resident met with the coach an average of only 56 hours (half of which was devoted to EBM), quantitative and qualitative data showed significant changes in knowledge and skills. The intervention addressed key skills emphasized in the

literature on EBM education and information mastery: formulating clinical questions, searching EBM resources for answers, and most importantly, applying their knowledge to patient care in real time.17-19 Our intervention required a single faculty member with a one half per day per week (0.1 FTE) time commitment and some expertise in EBM and basic epidemiology. This time commitment includes development of curriculum for topics covered and would likely be less at smaller residency programs. With appropriate faculty development, multiple

FAMILY MEDICINE

VOL. 44, NO. 5 MAY 2012

353

BRIEF REPORTS

faculty members could assume the coach role, distributing the intensive time commitment on the faculty side. Otherwise, resources necessary for our intervention are minimal, with the greatest other resource being access to Web-based EBM resources. Our study has several limitations. This pilot study occurred at a single site without a control group, limiting the validity of our results. We cannot be certain EBM ability did not increase as a function of time. A hidden agenda of the intervention, the effect on residents of EBM modeling by the coach, cannot be adequately represented by self-report survey data and ratings by a single assessor. Observing actual physician behavior, such as shared decision making with patients, and patient outcomes would add strength to our results. Finally, although our residents EBM knowledge scores improved after the intervention, they averaged only 58% correct on the posttest.

Figure 1: Average Coachs Ratings of Evidencebased Medicine Performance by Session

Table 4: Themes and Illustrative Quotes From Evidence-based Medicine Cirriculum

Themes Using wider range of electronic resources Resident Quotes Instead of just using Up to Date I think people are using different resources. I see people using the Brown library system more. I see people using the Brown library and using Dynamed and using other sources to find information. At least I know my resources, so if I cant find in one resource I can go up to the next one and look it up. The key is to know where to find the answer; not to know the answer all the time. Anytime I have questions in clinic or in any kind of patient care, I know where to go look it up. And which sites to go to connecting to the Brown library and going to different sites there. Sometimes if the patient needs to know the answer I go through online resources with the patient. Okay, lets check what it says and I asked the patient to look at the online resources and we both go through that. Im trying to find more answers. Im trying to get better at answers that are evidence based. I think what was very beneficial was the kind of questions that he showed us, and all the databases that he actually shared with us, because to be frank when I learned all those and I used to be on call with one of my seniors from third year they didnt knowI mean they know a lot, but they didnt know all the resources [the coach] shared with us. I mean it [residency culture] has changed since they started self-directed learning. I came in my first year and I did not know which resources to look up. The learning coach showed us how to actually learn and how to update ourselves. Its very important when youre working and youre a family physician. I still think doing one-on-one with him was important because we do have a lot of questions as to how we could go to this site, how we could use this resource. Again I think that will get lost if its a noon conference. I think it worked more with the face to face contact with him and just one to one.

Demonstrating greater confidence and comfort in finding clinical information Possessing ability to find best medical evidence for patients in real time/point of care

EBM valued in residency culture

Importance of one-on-one format for curriculum delivery

354

MAY 2012 VOL. 44, NO. 5

FAMILY MEDICINE

BRIEF REPORTS

However, the more important goal of our intervention (and that of most EBM curricula, we believe) was to promote information mastery to support lifelong learning. This was what the coachs ratings assessed: the skill of looking up information, motivation to engage in it, and perceived self-efficacy in information retrieval, rather than EBM knowledge per se. Resident interviews and coachs ratings showed important gains in point-of-care EBM skills. The relationship between epidemiologic knowledge and ability to use EBM effectively warrants further study. In our program, using a learning coach to teach EBM to second-year residents improved residents attitudes, knowledge, and use of EBM. This model may be a feasible alternative to a uniform curriculum that addresses variable learning needs of residents.
ACKNOWLEDGMENTS: This project was supported by US Department of Health and Human Services Primary Care Training Grant D58HP08298. A description of this intervention and preliminary data were presented at the Society of Teachers of Family Medicine (STFM) 2009 Annual Spring Conference in Denver, and at the 2010 STFM Annual Spring Conference in Vancouver. CORRESPONDING AUTHOR: Address correspondence to Dr George, Memorial Hospital of Rhode Island, 111 Brewster Street, Pawtucket, RI 02860. 401-729-2235. Fax: 401-729-2923. Paul_george@brown.edu.

References
1. Sackett DL, Rosenberg WC, Muir Gray JA, Haynes RB, Richardson WS. Evidence-based medicine: what it is and what it isnt. BMJ 1996;312(7023):71-2. 2. McColl A, Smith H, White P, Field J. General practitioners perceptions of the route to evidence-based medicine: a questionnaire survey. BMJ 1998;316(7128):3615. 3. Haynes RB. What kind of evidence is it that evidence-based medicine advocates want healthcare providers and consumers to pay attention to? BMC Health Serv Res 2002;2(1):3. 4. www.acgme.org/outcome/comp/generalcompetenciesstandards21307.pdf. Accessed April 18, 2011. 5. Slawson DC, Shaughnessy AF. Evidencebased medicine: should we be teaching information management instead? Acad Med 2005;80:685-9. 6. Straus SE, Ball C, Balcombe N, Sheldon J, McAlister F. Teaching evidence-based medicine skills can change practice in a community hospital. J Gen Intern Med 2005;20(4):340-3. 7. Mascola A. Guided mentorship in evidencebased medicine for psychiatry: a pilot cohort study supporting a promising method of real-time clinical instruction. Acad Psychiatry 2008;32:475-83. 8. Allan GM, Korownyk C, Tan A, Hindle H, Kung L, Manca D. Developing an integrated evidence-based medicine curriculum for family medicine residency at the University of Alberta. Acad Med 2008;83(6):581-7. 9. Thom DH, Haugen J, Sommers PS, Lovett P. Description and evaluation of an EBM curriculum using a block rotation. BMC Med Educ 2004;4:19. 10. Haines SJ, Nicholas JS. Teaching evidencebased medicine to surgical subspecialty residents. J Am Coll Surg 2003;197(2):285-9.

11. Schilling K, Wiecha J, Polineni D, Khalil S. An interactive Web-based curriculum on evidencebased medicine: design and effectiveness. Fam Med 2006;38(2):126-32. 12. Burneo JG, Jenkins ME, Bussire M, UWO Evidence-Based Neurology Group. Evaluating a formal evidence-based clinical practice curriculum in a neurology residency program. J Neurol Sci 2006;250(1-2):10-9. 13. Nothnagle M, Goldman R, Quirk M, Reis S. Promoting self directed learning skills in residency. Acad Med 2010;85(12):1874-9. 14. Collins A. Cognitive apprenticeship. Cambridge handbook of the learning sciences. Cambridge, UK: Cambridge University Press, 2006:47-60. 15. Collins A, Brown JS, Duguid P. Cognitive apprenticeship: teaching the crafts of reading, writing and mathematics. In: Resnick LB, ed. Knowing, learning and instruction: essays in honor of Robert Glaser. Hillsdale, NJ: Lawrence Erlbaum Associates Inc, 1989:453-94. 16. Ramos KD, Schafer S, Tracz SM. Validation of the Fresno Test of Competence in evidencebased medicine. BMJ 2003;326(7384):319-21. 17. Bergus GR, Emerson M. Family medicine residents do not ask better-formulated clinical questions as they advance in their training. Fam Med 2005;379(7):486-90. 18. Kim S, Willett LR, Murphy DJ, ORourke K, Sharma R, Shea JA. Impact of an evidencebased medicine curriculum on resident use of electronic resources: a randomized controlled study. J Gen Intern Med 2008;23(11):1804-8. 19. Rao G. Physician numeracy: essential skills for practicing evidence-based medicine. Fam Med 2008;40(5):354-8.

FAMILY MEDICINE

VOL. 44, NO. 5 MAY 2012

355