ORIGINAL REPORTS

Is the Evaluation of the Personal Statement a Reliable Component of the General Surgery Residency Application?
Bobbie Ann Adair White, MA,* Mark Sadoski, PhD, Scott Thomas, MD, and Mohsen Shabahang, PhD *The Ofce of Faculty Development, Texas A&M Health Science Center, College of Medicine, Temple, Texas; The Ofce of Educational Development, Texas A&M Health Science Center, College of Medicine, Bryan, Texas; Department of General Surgery, Scott & White Healthcare, Temple, Texas; Department of General Surgery, Geisinger Medical Center, Danville, Pennsylvania
BACKGROUND: Each year, fourth-year medical students

spend considerable time writing and rewriting their personal statements. However, there is little evidence of what role the personal statement plays in deciding which applicants will be invited for an interview.
OBJECTIVE: To evaluate the inter-rater reliability of a surgical selection committees ratings for both the personal statement and the application summary parts of the residency application. DESIGN: We completed a retrospective analysis of the 2007

correlations ranged from 0.83 to 0.63 with an overall average of 0.09.

CONCLUSION: These results demonstrate that the personal

statements lacked objective criteria for evaluation. (J Surg 69: 340-343. 2012 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.)
KEY WORDS: residency application, match, personal statement, application criteria COMPETENCIES: Interpersonal and Communication Skills,

2008 Scott & White surgical residency application pool. From a total pool of 174 residency applications, we selected 8 (5%) applications randomly to be evaluated by 4 experienced members of the selection committee. The 4 committee members rated each personal statement on a 7-point scale, from negativewould not invite for an interview to positivewill invite for an interview. They rated respective application summaries separately on a similar 7-point scale. Committee members also listed their top three reasons for assigning their scores.
METHODS: Rating scores for the personal statements and the

Professionalism, Systems Based Practice

INTRODUCTION
Each year, fourth-year medical students spend considerable time writing, rewriting, and seeking editorial advice about the content of their personal statements. Students believe that the quality of the personal statement may determine their chances for a residency interview. Many medical schools provide a list of dos and donts, and some distribute guidelines for writing the personal statement. But the personal statement is ultimately personal, and evaluators are faced with varied styles, format, and content. It is difcult to measure objectively what makes a personal statement a good one, although there have been attempts in the literature to characterize it more objectively.1 Also poorly understood is what role the personal statement plays in deciding which applicants will be invited for an interview. The National Resident Matching Program (NRMP) attempted to capture the relative contributions of application components through their 2010 NRMP Program Director Survey2 (for additional details about their survey, see http://www.nrmp.org). Sixty-eight percent of responding directors cited the personal statement was a factor in interview selection. This percentage is higher than that of grades in required clerkships (62%), the
1931-7204/$30.00 doi:10.1016/j.jsurg.2011.12.003

applications were analyzed for inter-rater correlation. The qualitative data (ie, reasons for the scores) were reviewed to help the investigators prole the reasons given for very positive and very negative scores.
RESULTS: For the application summaries, the correlations be-

tween each pair of raters ranged from 0.79 to 0.94 with an overall average of 0.88. For the personal statements, inter-rater

Correspondence: Inquiries to Bobbie Ann Adair White, MA, The Ofce of Faculty Development, Texas A&M Health Science Center, College of Medicine, 2401 South 31st Street, Temple, TX 76508; fax: (254) 724-6810; e-mail: bawhite@medicine.tamhsc.edu Presented in part at the Association of American Medical Colleges, Joint Student Affairs and Careers in Medicine Professional Development Conference, June 11, 2011.

340

Journal of Surgical Education 2012 Association of Program Directors in Surgery Published by Elsevier Inc. All rights reserved.

deans letter (60%), and class ranking (59%). The only application components that had a higher percentage were letters of recommendation in the specialty (71%) and the United States Medical Licensing Examination (USMLE) step 1 score (73%). The survey also attempted to quantify the importance of the factors used in ranking applicants using a 5-point scale. On that scale, personal statements fell at 3.4, whereas interactions with faculty and interpersonal skills were of the highest at 4.7. Although all specialties were surveyed in the NRMP 2010 Program Director Survey, for our purposes we will review only general surgery data. In all, 402 surveys were sent with a 36.8% response rate. Sixty-ve percent of the 148 general surgery programs completing the survey indicated that personal statements were a factor in selecting applicants to interview. The importance of the personal statement in ranking an applicant averaged 3.3 on a 5-point scale. However, in 1998, Joseph T. Crane et al. completed a similar survey specically for emergency medicine resident selection that had a 79.7% return. They found that personal statements were the least important factor in resident selection with an average of 2.75 on a 5-point scale. A study by Taylor et al.3 found that personal statements were the last factor considered for interview invitations by directors of obstetrics and gynecology, whereas it was the second most important factor for family practice residency directors. Similarly, they found that obstetrics and gynecology directors ranked the personal statement last, whereas family medicine directors ranked it third for making decisions about ranking.3 Because of the mixed results the personal statement seems to have as a factor in the interview selection process, we thought we would attempt to get a closer look at what a selection committee found to be appealing.

students personal statement. Each part was given a different identication number known only to the research coordinator. The application summary was given an 8-digit number, whereas the personal statement was given a number that ranged from 1 to 174 to avoid the possibility of raters connecting the 2 parts of the application. After deidentication and number assignment, 8 application summaries with corresponding personal statements were pulled randomly from the stack of applications by the research coordinator. Although the application was split in 2 parts and deidentied, each rater received all 8 summaries and all 8 personal statements so that later we could compare the ratings of the personal statement with that of the application summary. To assess inter-rater reliability, all 4 raters reviewed the complete applications in two parts. Each rater was experienced with reviewing applications during interview season. Using 7-point rating scales (Appendices A and B), each rater rated all the application summaries and then rated all the personal statements. Additionally, we asked the raters to specify their top 3 reasons for rating the statement or summary sheet the way they did (see Appendix A). Inter-rater reliability was calculated to determine consistency across graders, and we computed the degree of correlation between ratings for the personal statements and the application summaries. This study was approved by Scott & White Healthcares Institutional Review Board.

RESULTS
We correlated the ratings of all possible pairs of raters. For the application summaries, the correlations between each pair of raters ranged from 0.79 to 0.94 with an overall average of 0.88, indicating a high degree of agreement between raters on what is valued in an applicant or application summary. All these individual correlations were statistically signicant (p 0.05). For the personal statements, inter-rater reliability had a range of 0.83 to 0.63 with an overall average of 0.09 (see Table 1 for a full listing of the personal statement scores). None of these individual correlations was statistically signicant except one (p 0.05). These results demonstrated that common criteria for evaluation of the personal statement was lacking. The qualitative data regarding the personal statements reected the inconsistency found in the correlations of the ratings. A single personal statement could receive 4 completely contradictory comments from separate raters:

METHODS
This study used qualitative and quantitative methods. A total of 174 applications were submitted to the surgical residency program of Scott & White Memorial Hospital in Temple, Texas for the 2007-2008 academic year. The applications were divided into 2 parts: an application summary and the personal statement. The application summary included a summary of objective data (college and medical school attended, scores on USMLE I and II, class rank, surgery grade, publications, applicant summary/accomplishments found on curriculum vitae, and hobbies), whereas the second part solely consisted of the

TABLE 1. Ratings Listed for Each Personal Statement Provided by the Selection Committee Members (R1R4)* Personal Statement Scores Faculty and Resident Raters R1 R2 R3 R4 PS 1 4 6 4 6 PS 2 6 6 5 5 PS 3 3 3 6 4 PS 4 7 2 6 3 PS 5 1 2 6 3 PS 6 4 7 2 3 PS 7 4 6 2 6 PS 8 1 3 5 2

*The ratings were based on the scale in Appendix B. Journal of Surgical Education Volume 69/Number 3 May/June 2012 341

1. Mentions CV essentially . . . explaining away some of his problems (rating 1) 2. Uses context of surgery to say a lot about themselves (rating 7) 3. No ending . . . hard to get past failure of Step 2; like the passion, but worry about making right decisions and priorities (rating 3) 4. Service to others correlates well to surgical career, Interesting storywanted to meet the candidate . . . poor step 2, prelim, but good explanation (rating 6). Although it does not seem possible, these comments were referring to the same statement, where the student explains what happened with their step 2 score. Another example of comments from separate raters is as follows: 1. Humble, good quote, succinct . . . (rating 7) 2. Too short, uses a family example that is not dramatic, clich (rating 2) 3. Would have given a 7 if grammar error . . . was not there . . . sounds like someone I would want to train (rating 6) 4. Short, does not say much, does not explain/give examples . . . (rating 3). The personal statement is described as too short with a rating of 2 and then as succinct with a rating of 7, which makes these evaluations contradictory. Even for the 1 personal statement that seemed to have consistent ratings, the remarks were different: 1. Good rst paragraphattention getting and relevant; describes why surgery, gives a goal that is attainable (rating 6) 2. Eloquent, good examples, could give more insight (rating 6) 3. Like that he/she understand the approach, not overcondent . . . weird comment about consulting other services was not sure what to think about that (rating 5) 4. Clearly shows patient ownership, acknowledges multifaceted job of surgeon, need for team, middle of the road, no wow factor (rating 5) Based on numbers alone, every rater agreed this was a good personal statement; however, the criteria for which it was deemed good were not consistent. This nding illustrates the subjective evaluation of the personal statement.

sonal statement to be a reliable measure for interview offers and that different evaluators will judge the same statement differently. Unfortunately, we cannot generalize our results as our small sample size was chosen originally to establish inter-rater reliability within a larger study. Thus, this small size was not powered for generalization and may be vulnerable to errors of statistical inference. We had hoped to nd out how the rating of the personal statement correlated with the rating of the application summary and how both together predicted the likelihood of gaining an interview. Because of the high degree of unreliability, in the ratings of the personal statements, such analyses were precluded. Although our original objective was not met, our nding that the inter-rater reliability of the personal statement was unstable was signicant enough to share with students and faculty that are engaged in the match process. This nding suggests an issue in much need of continued research: What is expected in a personal statement? Although it is advisable that students do not write anything offensive or grammatically incorrect in their personal statement, the content can vary and its appeal will be variable.4 To ensure objectivity in a potentially subjective personal statement evaluation, institutions need to determine what characteristics a personal statement should exhibit to merit an interview and stay close to those criteria in scoring them. Wayne State Universitys Radiology Program began this process, which is illustrated in Smith et al.4 Another program that found success in standardization of the screening process is the University of Pennsylvania. Their surgical program had an organizational specialist help standardize the screening and interviewing process; the adjustments made improved the programs ability to match the candidates t with faculty expectations.5 Although these are 2 great examples of programs standardizing the application process, to our knowledge, this is not being accomplished widely.

ACKNOWLEDGMENTS
A special thank you to Gale G. Hannigan, PhD, MLS, MPH for her editorial, creative, and supportive contributions and to June Lubowinski, MLS, MA for her help with literature reviews.

REFERENCES
1. Crane JT, Ferraro CM. Selection criteria for emergency

DISCUSSION
Our results suggest that application summaries enjoy a substantial degree of objectivity, whereas personal statements are a much more subjective measure, indicating that different things appeal to different people. Thus, a personal statement would not be a stable indicator of being granted an interview in our study. If these results are generalizable to other institutions, then students should know that they cannot count on the per342

medicine residency applicants. Acad Emerg Med. 2000;7: 54-60.

2. National Resident Matching Program. Data Release and

Research Committee. Results of the 2010 NRMP Program Director Survey. Washington, DC: National Resident Matching Program; 2010. Available at: http://www. nrmp.org/data/programresultsbyspecialty2010v3.pdf.

Journal of Surgical Education Volume 69/Number 3 May/June 2012

3. Taylor CA, Weinstein L, Mayhew HE. The process of res-

ident selection: a view from the residency directors desk. Obstet Gynecol. 1995;85:299-303.
4. Smith EA, Weyhing B, Moody Y, Smith WL. A critical

analysis of personal statements submitted by radiology residency applicants. Acad Radiol. 2005;12:1024-1028.
5. Kelz RR, Mullen JL, Kaiser LR, et al. Prevention of surgical

_________________________________________ _________________________________________ _________________________________________ 7 ___ Positive: Will invite for an interview _________________________________________ _________________________________________ _________________________________________

resident attrition by a novel selection strategy. Ann Surg. 2010:537-543.

Appendix B
Personal Statement Checklist 1-7 Reviewer _______________________________ Personal Statement #____________ Please rate the personal statement and list your top three reasons for your rating. 1 ___ Negative: Would not invite for an interview _________________________________________ _________________________________________ _________________________________________ 2 ___ _________________________________________ _________________________________________ _________________________________________ 3 ___ _________________________________________ _________________________________________ _________________________________________ 4 ___ _________________________________________ 5 ___ _________________________________________ _________________________________________ _________________________________________ 6 ___ _________________________________________ _________________________________________ _________________________________________ 7 ___ Positive: Will invite for an interview _________________________________________ _________________________________________ _________________________________________

Appendix A
Application Checklist 1-7 Reviewer _______________________________ Application Summary #__________ Please rate the application and list your top reasons for your rating (objective application information). 1 ___ Negative: Would not invite for an interview _________________________________________ _________________________________________ _________________________________________ 2 ___ _________________________________________ _________________________________________ _________________________________________ _________________________________________ 3 ___ _________________________________________ _________________________________________ _________________________________________ _________________________________________ 4 ___ _________________________________________ _________________________________________ _________________________________________ 5 ___ _________________________________________ _________________________________________ _________________________________________ 6 ___

Journal of Surgical Education Volume 69/Number 3 May/June 2012

343