The Journal of Continuing Education in the Health Professions, Volume 23, pp. 4-12. Printed in the U.S.A.

Copyright 0 2003 The Alliance

for Continuing Medical Education, the Society for Academic Continuing Medical Education, and the Council on CME, Association for
Hospital Medical Education. All rights reserved.

Theoretical Foundations
Multisource Feedback in the Assessment of
Physician Competencies
Jocelyn Lockyer, PhD
Abstract
Multisource feedback (MSF), or 360-degree employee evaluation, is a questionnaire-based
assessment method in which ratees are evaluated by peers, patients, and coworkers on key
performance behaviors. Although widely used in industrial settings to assess performance,
the method is gaining acceptance as a quality improvement method in health systems. This
article describes M S E identifies the key aspects of MSF program design, summarizes some
of the salient empirical research in medicine, and discusses possible limitations for MSF as
an assessment tool in health care. In industry and in health care, experience suggests that
MSF is most likely to succeed and result in changes in performance when attention is paid to
structural and psychometric aspects of program design and implementation.A carefilly selected
steering committee ensures that the behaviors examined are appropriate, the communication
package is clear; and the threats posed to individuals are minimized. The instruments that are
developed must be tested to ensure that they are reliable, achieve a generalizability coefficient
of Ep2 = .70, have face and content validity, and examine variance in performance ratings to
understand whether ratings are attributable to how the physician performs and not tofactors
beyond the physician S control (e.g., gender; age, or setting). Research shows that reliable data
can be generated with a reasonable number of respondents, and physicians will use the feed-
back to contemplate and initiate changes in practice. Pe$ormance may be affected byfamiliarity
between rater and ratee and sociodemographic and continuing medical education charac-
teristics; however; little of the variance in performance is explained by factors outside the
physician S control. MSF is not a replacement for audit when clinical outcomes need to be
assessed. Howevel; when interpersonal, communication, professionalism, or teamwork behav-
iors need to be assessed and guidance given, it is one of the better tools that may be adopted
and implemented to provide feedback and guide performance.
Key Words: Continuing medical education (CME), 360-degree evaluation, medical educa-
tion, multisource feedback, patient surveys, peer assessment, physician assessment, physician
competence, graduate medical education

The assessment and maintenance of physician
competence have been given worldwide attention,'
Reprint requests: Jocelyn Lockyer, PhD, Director,
Continuing Medical Education and Professional
Development, University of Calgary, 3330 Hospital Drive
NW, Calgary, AB, T2N 4NI; e-mail: lockyer@ucalgary.ca.
partly in response to concerns about physician per-
fomiance2and patient as well as demands
for accountability to patients and funding agen-
cies.'-' These concerns have shifted the concept
of competence from a narrow definition of the
ability to perform technical medical acts into
much broader aspects of competence. For exam-
ple, the Accreditation Council for Graduate Med-
ical Education (ACGME) has declared that physi-
cians need to attain competence in six domains:
patient care, medical knowledge, practice-based
learning and improvement, interpersonal and

4
 Lockyer
communication skills, professionalism, and sys-
tems-based practice.6 Physicians are encouraged
to continue to improve their medical skills and
knowledge and asked to reflect on the care they
provide; collaborate with other members of the
health care team; recognize their roles within
overall systems of care; and deliver their ser-
vices with integrity, honesty, and compassion,
exhibiting appropriate personal and interpersonal
professional behaviors. New ways of thinking
about competence have resulted in a quest to find
new approaches to assess practicing physicians
and guide their development.
Assessment tools and accreditation criteria
have been established to ensure competency inte-
gration into undergraduate medical education, as
well as residency education in the United States
and Canada. Paper- and computer-based exami-
nations, objective structured clinical examina-
tions (OSCEs), and in-training assessment reviews
are the traditional ways of evaluating and mea-
suring these competencies over the course of a clin-
ical rotation, a year, or a residency. It is not as easy
for practicing physicians to evaluate themselves
on these newer competencies in routine and reg-
ular ways.
Multisource Feedback
Industry developed multisource feedback (MSF)
systems (360-degree evaluation7) to support
employee decision making and quality improve-
ment. It was seen as a practical approach to assess-
ing competence in the workplace, and, as such, it
developed without a theoretical base. Nonetheless,
MSF is recognized as useful in assessing broad
c~mpetencies.~*~
MSF requires that questionnaires be designed
to determine competence for specific behavioral
tasks or approaches. They are administered on
behalf of the person being assessed. The goal is to
look at a person’s work from a variety of per-
spectives, including those at the same level in the
organizational chart, those above, and those at
lower levels. The look from all perspectives helps
5
to frame a more complete picture of performance.
In industry, the questionnaires obtain information
from the person being assessed, hisher peers, sub-
ordinates, supervisors, and, occasionally, clients.
They assess observable behaviors such as written
and oral communication skill, team-building abil-
ity, collegial interaction, and problem-solving abil-
ity. The person being assessed receives feedback
with hisher own aggregate ratings (for items and
scales) along with the mean ratings for others
being assessed at the same time. Feedback also
might compare the individual’s self-assessment
data with those provided by raters.
As medical services are provided increasingly
within organizationalsystems that demand account-
ability to funding agencies and patients, it is not sur-
prising that the tools used to assess managers also
would be seen as ways to provide physicians with
feedback in areas such as communication skills and
interpersonalrelationships.I0J1Similarly, as licens-
ing and professional organizations struggle to
ensure that the practice of medicine includes col-
legiality, communication slulls, and professional-
ism, as well as medical expertise, MSF has been
identified as a mechanism to provide this feedback
to In medical settings, physician col-
leagues (peers), coworkers, and patients are asked
to complete surveys about the physician.
MSF is a survey-based method with flexibil-
ity. Questionnaire items and domains can be
changed quickly to encompass new perspectives
and needs. For example, the American Board of
Internal Medicine (ABIM)I5 has instruments for
patient and peer assessment as part of its mainte-
nance of certification program. Its patient instru-
ment focuses on professionalism and interper-
sonal and communication skills, whereas the peer
instrument covers professionalism, medical knowl-
edge, and patient care. In Alberta, Canada, the
College of Physicians and Surgeons of Alberta
(CPSA) has instruments for peers, coworkers, and
patients across a variety of medical and surgical
disciplines.I8These instruments cover the breadth
of the ACGME competencies as well as office and
support staff issues.
 MSF in Assessing Physician Competencies
Table 1 Comparison of Studies
Reference Number
Studies in Present Text
North East United States 16, 17
Internist
Alberta 12-14, 18-21
ABIM 15
ABIM =American Board of Internal Medicine.
Designing an MSF Program
Standardizedinstruments for MSF do not exist; yet
research and experience from m e d i ~ i n e ' * - ~ ~ . ~ ~ of
and i n d ~ s t r y ~suggest
-~ that structural and psy-
chometric requirements should be considered
when MSF programs are being developed or
adapted for use. Resistance can be anticipated
because people are concerned about a process of
evaluation that is public and includes colleagues,
supervisors, and patients. Individual concerns will
range from anxiety about their colleagues' and
patients' ability to assess them, the fear of getting
"bad" results, and mistrust about the system's
ability to handle their data confidentially to reper-
cussions that might accrue from the assessment.
Successful MSF appears to be dependent on
four facets:
organizational support
steering committee work
monitoring
psychometric design and testing
Although each facet of MSF design and imple-
mentation requires a longer forum for a full descrip-
t i ~ n , ~the
" following two sections offer an outline
of the salient aspects of each of these components.
Physician Group
and Study Size Ratees
3 16 internists from New York, MD peers;
Pennsylvania, and New Jersey; hospital-based
ABIM certification registered nurses
received 5-1 5 yr previously
257 family physicians, Medical colleagues,
200 surgeons; licensed in coworkers, patients,
province at least 5 yr self
356 diplomates of the ABIM; Patient, peer
average physician received
certification in 1989
Examples of how these processes were managed
come from three sets of studies, each comprising
one or more publications or reports. The first group
- * studies,
' the North East United States (NE U.S.)
~ , ' ~undertaken between 1988
internist ~ t u d i e s , ' was
and 1989 and included ABIM certificants practic-
ing in New York, Pennsylvania,and New Jersey. The
second group of studies, the Alberta ~tudies,'~-~~,'~-"
was conducted between 1996and 200 1 and included
family p h y s i ~ i a n s ' * and- ' ~ ~surgeon^'^^^^
~~ practicing
in Alberta. The third study, the ABLM study, included
data about internists collected between 1998 and
2Oo2.l5A brief comparison of the studies is provided
in Table 1.
Administering MSF
The organization must be committed to MSF and
believe that MSF is the best way to provide salient
feedback that is not being provided to physicians
through other mechanisms. Implementation of
MSF requires communication about the MSF pro-
gram itself. Participants (both ratees and raters)
must understand the purpose and goals of the pro-
gram and how it will be of value to the individu-
als and the organization. Decisions need to be
made early and widely communicated regarding
how the data will be ~ s e d . ~ ,In ~ ,some
' * organiza-
6
 Lockyer
tions, data are used only for quality improvement
(formative) purposes. In other organizations, the
data are used as part of reward or discipline struc-
tures. In Alberta, for example, a decision was
made early that the MSF program would be dis-
tinct from the disciplinekomplaints responsibili-
ties that the CPSA had, and MSF data would not
be used to revoke medical licenses.12Training of
raters is helpful to ensure that people understand
the purpose, the dimensions being examined, and
the questions being asked as they clearly influence
the quality and accuracy of data that raters provide.
Communication with raters and ratees needs to be
continuous as the program begins, raters are
recruited, and ratees receive results. Further, once
gaps in performance are identified, support sys-
tems need to be developed or made accessible to
help ratees improve.
A strategically selected steering committee
is needed to guide and validate the p r o c e ~ s . In ~ ~ , ' ~ raters for each physician must be determined. Too
conjunction with psychometric consultants, this
committee determines the domains to be assessed,
the items that can be observed, the people who will
provide the data for each item, the scale(s) that will
be used for assessment, and the number of raters
that will be required for each instrument.
Monitoring of the overall program generally
includes informal and formal monitoring. Informal
monitoring is achieved through meetings to solicit
feedback and encouraging people to submit their
thoughts about the program through electronic
mail list serves and newsletters. Formal monitor-
ing is asking raters and ratees to complete surveys
in which their perceptions of the program, the
~ , ' ~ , ~ ~ena as feasibility and acceptance of feedback.
questions, or data received are a s s e ~ s e d . ~Ratee
acceptance of datal6.l7and information about how
they used the data can also be obtained.20,2'This
allows the organization to revisit the communica-
tion program and the steering committee to revisit
the items andor sources used, as needed.
Psychometric Design and Assessment
Careful psychometric design and assessment of
psychometric properties help to ensure that the
instruments are valid and reliable.
7
Generally, this process begins with a struc-
tured approach to item generation to ensure con-
tent ~alidity.'~.'~Key domains (e.g., interpersonal
skills, communication skills) to be covered by the
instrument are determined. Items are developed for
each of the domains, giving consideration to the
type of information each source (e.g., patient or
peer) can provide. Designers must assure them-
selves that the proposed rater for that item actually
has the ability to observe or experience the behav-
iors being assessed. When instruments are drafted,
they can be sent to each of the physicians, who will
be part of the MSF process to ensure that they
have an opportunity to react to the content and
instrument revisions can be made.'2.'9Tables 2 to
4 provide examples of questions developed in
Albertals to assess interpersonal and communica-
tion skills from patients, coworkers, and peers.
Early in design, an appropriate number of
few raters may provide idiosyncratic data and
data with low reliability (i.e., low internal con-
sistency). Requiring too many raters may make it
difficult for some physicians to recruit sufficient
numbers of raters who can actually answer the
items accurately, thus affecting the validity of the
data.
Once instruments are developed, it is possible
to run a pilot project to assess their psychometric
properties. At this point, the goals are to establish
instrument reliability, ensure that the instruments
attain an acceptable generalizability coefficient
(generally Ep2 = .70), assess criterion and con-
struct validity, and examine such related phenom-
Reliability is done by assessing the Cronbach
(Y levels of instruments and scales to ensure inter-
nal consistency. Assessments of reliability show
that it is possible to develop instruments with
Cronbach a levels > .90'2-143'9 indicating high
internal consistency. Generalizability,22a more
advanced analysis of reliability, determines
whether there are sufficient numbers of raters and
items on the instrument to attain an appropriate
generalizability coefficient (generally EP2= .70).
These analyses show that one can obtain accept-
 MSF in Assessing Physician Competencies

Table 2 Sample Questions to Assess Communication Skills from Patients

1 = Strongly 5 = Strongly
Answer Questions about This Physician Disagree 3 = Neutral Agree UA
Based on your last visit to this physician,
Did your physician explain your illness or 0 0 0 0
injury to you thoroughly?
Did your physician adequately explain 0 0 0 0
treatment choices?
If a prescription for medication was given, 0 0 0 0
did your physician clearly explain how
and when to take the medication?

UA = unable to assess.

able results with a reasonable number of items
and raters. For example, the NE U.S. studies
achieved .70 with an 11-item instrument using 10
to 11 physician peer raters16and 10 to 15 nurses
for the 13 nurse assessment of the physicians.”
The Alberta estimated that 6 peer
raters (26 items), 6 coworkers (17 items), and 22
patients (45 items) provided adequate depend-
ability (Ep2 > .70) of ratings. The ABIM data
attained Ep2 = .67 with 25 patients and .61 with
10 peers.lS
Data to permit an examination of the corre-
lations between MSF data and other ratings are dif-
ficult to find. In part, instruments examining these
aspects of physician behavior are not available.
One studyis tests the ABIM ratings with the rat-
ings that physicians received 5 or more years ear-
lier from their residency program directors. The
study revealed small correlations with prior ratings
for humanistic qualities but not with overall clin-
ical competence ratings. Similarly, small correla-
tions were found between peer ratings and program
director ratings.
Most assessments have focused on aspects of
construct validity or those phenomena that might
affect performance ratings. Rater-ratee familiar-
ity is a common concern because MSF systems
depend on ratees identifying appropriate raters.
There is the concern that one’s friends will be
selected and bias the results upward, thereby nul-
lifying objectivity. Studies conducted in the north-
eastern United States’6J7showed that increased

Table 3 Sample Questions to Assess Interpersonal and Communication Skills from Coworkers
(e.g., Nurses, Pharmacists, Physiotherapists)

1 = Among 5 = Among
Compared with Physicians I Know, This One the Worst 3 = Average the Best UA
Communicates effectively with patients. 0 0 0 0
Is available for consultation about mutual patients. 0 0 0 0
Writes hospital orders clearly. 0 0 0 0
Is courteous to coworkers. 0 0 0 0

UA = unable to assess.

8
 Lockyer
Table 4 Sample Questions to Assess Interpersonal and Communication Skills from Peers
1 = Strongly
Compared with Physicians I Know, This One
Communicates effectively with patients.
Communicates effectively with other
health care professionals.
Handles transfer of care effectively.
UA = unable to assess.
contact and ability to observe the ratee increased
ratings somewhat, but this explained < 7% of the
variance. In the Alberta studies,I2the highest rat-
ings were provided by those who did not know the
ratee well. However, there were few raters who did
not know their physician well, leading them to note
that this finding was trivial. The ABIM data15
showed that peers who shared patients with ratees
for longer periods of time tended to rate them
higher. That study also found that those patients
who spent more time in the physician’s care rated
the physician higher. In both cases, the correlations
were very low, although significant.
In Alberta, continuing medical education
(CME) participation was studied.I2The combined
data for both surgeons and family physicians
showed that higher peer ratings correlated with
longer call hours, younger physicians, and hold-
ing an academic appointment. Higher ratings from
coworkers were significantly correlated with being
a family physician versus a surgeon, being female
and a younger age, and doing less CME. Higher
patient ratings were correlated with being a fam-
ily physician versus surgeon, being female, work-
ing fewer hours, having fewer hours on call, and
doing less CME.14The ABIM dataI5 showed that
the health of the patient correlated with overall rat-
ing, with those in better health rating the physi-
cian higher, and with female physicians receiving
higher ratings. Ail of these correlations were rel-
atively small, generally r < .20.
Convergence, the correlation between rater
sources or between self and rater sources, often is
5 = Strongly
Disagree 3 = Neutral Agree UA
0 0 0 0
0 0 0 0
0 0 0 0
examined in MSF. Such analyses are important
because they provide information about the gap
between self and other as that may affect ratee’s
ability to accept the validity of the data. Low con-
vergence between sources reconfirms the need to
have several sources to provide complete data. The
Alberta analysisl2 of family physician perfor-
mance found that self data did not correlate with
patient, referring physician, medical colleague,
or coworker data. When surgeon and family physi-
cian data were grouped and analyzed together,
self-assessment correlated with coworker ratings
but not significantly with either peer or patient rat-
i n g ~The. ~ Alberta
~ studies1*-14and the U.S. stud-
ies“,” revealed that there were significant corre-
lations between sources (other than self), but the
correlations were generally low, suggesting that
coworkers and peers provided different insights.
Only the NE U.S. s t u d i e ~ ’examined
~ ? ~ ~ accep-
tance of data. They showed that acceptance was
associated with the source of the data. For exam-
ple, physicians were more comfortable with nurses
rating them on communication skills and human-
istic qualities than on overall clinical skills.”
When the same physicians were asked whether
nurses should assess physicians for credentialing
and peer review processes, 29.8% felt that these
data should not be used for communication skill,
27.9% felt that they should not be used for human-
istic quality assessment, and 45.2% indicated that
they should not be used for assessing overall clin-
ical skills.16 However, 84% of the physicians
believed that peer ratings should be used to eval-
9
 MSF in Assessing Physician Competencies
uate the overall skills of practicing for credentialing
and recertification. ”
The extent to which ratees use their data to
contemplate or initiate change was examined in
both the Alberta’4.’0*2’and the ABIMIS studies.
This is important because changing behavior is one
of the primary objectives of MSF processes. The
Alberta data show that 84% of the family physi-
cians and 7 I % of the surgeons used their feedback
data to contemplate change.*()Fewer actually ini-
tiated change (68% of family physicians and 27%
of surgeons). Family physicians were more likely
to make changes in areas they could control (e.g.,
improved communication with patients, better
explanations about medications, and providing
patient material) rather than changes that required
others to be involved (e.g. staff improvement) or
incurred costs (e.g. telephone systems). The ABIM
study15found a majority of physicians believed that
feedback from peers and patients would improve
the quality of the medical care they deliver. About
40% reported an intent to change their communi-
cation strategieswith patients, and 28% indicated that
they would change their communication strategies
with their peers.
Discussion
The opportunity to find valid, reliable, feasible, and
acceptable methods to assess the competency of
practicing physicians undoubtedly will continue.
It is unlikely that a single tool will be found to
guide physician improvement or to satisfy all
funding agencies that physicians are achieving
satisfactory levels of performance. Nonetheless,
MSF shows promise as a means of assessing
physician competence across a broad range of
competencies.
MSF appears to be acceptable for assessing
some aspects of the ACGME competency domains.
When MSF includes self-assessment, the data can
confirm physicians’ impressions about how well
they are doing in practice or identify gaps between
the physicians’ impressions and those of others.
Such gaps may be helpful in identifying areas for
10
Lessons for Practice
Multisource feedback (MSF) offers
another method of providing physicians
with feedback about their performance.
Instruments used in MSF can be
designed to assess Accreditation
Council for Graduate Medical
Education competencies, including
patient care, interpersonal and com-
munication sltills, professionalism, and
systems- based practice . Medical
knowledge and practice-based learn-
ing competencies are probably better
assessed by other tools.
MSF systems can be designed so that
they are valid and reliable.
Physicians will accept MSF data, and
many use it to contemplate and/or ini-
tiate change in practice.
attention and learning. If mentors are part of the
follow-up program, a mentor can use “gap” infor-
mation to guide discussions and validate a need
for change. When peers, coworkers, and patients
can observe physicians, they can provide valuable
data about interpersonal and communication skills,
professionalism, and selected aspects of patient
care and systems-based practice. For example,
patients may be the best source to tell a physician
whether treatment options were discussed, med-
ication side effects were provided in sufficient
detail, or the interaction was compassionate and
respectful. Physician peers, on the other hand,
can respond to questions about the quality of
information provided in referral letters and eval-
uate their colleagues’ use of resources or ability
to appropriately transfer care. Coworkers such as
nurses or pharmacists can respond to questions
about collegial relationships and information pro-
vided about mutual patients.
 Lockyer
There are medical competencies that may not
be addressed optimally with MSF. For example,
medical knowledge probably can be better assessed
more precisely through traditional means, such as
multiple-choice examinations. Similarly, medical
audit and third-party data sharing techniques offer
more specific information about quality of care pro-
vided in clinical situations.
Studies examining MSF for its psychomet-
ric properties show that MSF instruments can be
developed that are reliable, with enough raters to
achieve an appropriate generalizability coeffi-
cient. 12~15-17Studies examining variance in ratings
show that although some variance in ratings can
be accounted for by rater and ratee characteris-
tics, on the whole, these correlations are relatively
low and explain relatively little of the vari-
ance.l4*l5 Physicians accept MSF, if they feel that
raters have the ability to provide ratings in the
areas being a s ~ e s s e d , ' ~although
,'~ not all physi-
cians will use the data to contemplate or initiate
change.20,21Additional work is warranted to
develop instruments to assess competencies such
as those of the ACGME and to ensure that each
competency is measured by robust scales.
Because MSF has developed largely on atheo-
retical grounds, the time has arrived to develop
theories that may explain the phenomenon of
MSF. Several compelling areas for research
include organizational context, change, learn-
ing, differences in self and others' perceptions,
self-assessment as a skill, and the use of feedback
data.
Acknowledgments
I would like to thank Claudio Violato, who was my
PhD supervisor and colleague for the Alberta stud-
ies. This work was undertaken as part of the can-
didacy examination process at the University of
Calgary in 2002. I would also like to acknowledge
the support of J. Swiniarski, B. Ward, R. Burns, and
L. Ohlhauser at the College of Physicians and Sur-
geons of Alberta, who invited us to join them in
developing and testing the Alberta PAR instru-
11
ments. Thanks also to D. Dauphinee and D. Black-
more, Medical Council of Canada, who financially
supported the surgeon feedback work, and to the
Society for Academic Continuing Medical Edu-
cation for supporting the family physician feedback
work. Special thanks to H. Fidler, who provided
incredible SPSS and research management support
through all phases of the Alberta studies.
References
1. Southgate L, Hays RB, Norcini J, et al.
Setting performance standards for medical
practice: a theoretical framework. Med Educ
2001; 35:474481.
2. Southgate L, Cox J, David T, et al. The assess-
ment of poorly performing doctors: the devel-
opment of the assessment programmes for the
General Medical Council's Performance
Procedures. Med Educ 2001; 35(Suppl 1):2-8.
3. Wilson RM, Harrison BT, Gibberd RW,
Hamilton JW. An analysis of adverse events
from the Quality in Australian Health Care
Study. Med J Aust 1999; 170:411415.
4. Kohn L, Corrigan J, Donaldson M, eds. To err
is human: building a safer health care system.
Washington, DC: Institute of Medicine, 1999.
5. Societal Needs Working Group Report. Skills
for the new millennium: CanMEDS 2000
Project. Ann R Coll Phys Surg Can 1996;
29:206-2 16.
6. ACGME Outcome Project. Accreditation
Council for Graduate Medical Education Web
site. Available at:
http://www.acgme.org/Outcome. Accessed
January 9,2003.
7. Edwards MR, Ewen AJ. 360" feedback: the
powerful new model for employee assessment
and performance improvement. New York:
-
American Management Association, 1996.
8. Bracken DW, Timnxeck CW, Church AH, eds.
The handbook of multisource feedback: the
comprehensive resource for designing and
implementing MSF processes. San Francisco:
Jossey-Bass, 2001.
9. Tornow WW, London M, CCL Associates.
Maximizing the value of 360-degree feedback:
 MSF in Assessing Physician Competencies
a process for successful individual and organi-
zational development. San Francisco: Jossey-
Bass Publishers, 1998.
10. Connors GP, Munro TW. 360-degree physi-
cian evaluations. Healthcare Executive 2001 ;
5 :58-59.
11. Jako RA. Evolution of multisource feedback
in a dynamic environment. In: Bracken DW,
Timreck CW, Church AH, eds. The handbook
of multisource feedback: the comprehensive
resource for designing and implementing MSF
processes. San Francisco: Jossey-Bass,
1999:403417.
12. Hall W, Violato C, Lewkonia R, et al.
Assessment of physician performance in
Alberta: The Physician Achievement Review
Project. Can Med Assoc J 1999; 161:52-57.
13. Violato C, Marini A, Toews J, Lockyer J,
Fidler H. Using peers, consulting physicians,
patients, co-workers and self to assess physi-
cians. Acad Med 1997; 70:57-63.
14. Lockyer JM. Role of socio demographic vari-
ables and continuing medical education in
explaining multi source feedback ratings and
use of feedback by physicians [dissertation].
Calgary, AB: University of Calgary, 2002.
15. Lipner RS, Blank LL, Leas BF, Fortna GS.
The value of patient and peer ratings in recer-
tification. Acad Med 2002; 77( 10
Suppl):64-66.
12
16. Ramsey PG, Carline JD, Inui TS, Larson EB,
LoGerfo JP, Wenrich MD. Use of peer ratings
to evaluate physician performance. JAMA
1993; 269~1655-1660.
17. Wenrich MD, Carline ID, Giles LM, Ramsey
PG. Ratings of the performances of practicing
internists by hospital-based registered nurses.
Acad Med 1993; 68:680-687.
18. College of Physicians and Surgeons of
Alberta, Physician Achievement Review
Program. Available at: http://www.par-pro-
gram.org/ or http://www.cpsa.ab.ca/college-
programdpar-programap. Accessed January
9, 2003.
19. Violato C, Lockyer JM, Fidler H. Multi source
feedback: a method of assessing surgical prac-
tice. BMJ 2003; 326546-548.
20. Fidler H, Lockyer J, Toews J, Violato C.
Changing physicians’ practices: the effect of
individual feedback. Acad Med 1999;
74:702-7 14.
21. Lockyer J, Violato C, Fidler H. Likelihood of
change: a study assessing surgeon use of multi
source feedback data. Teach Learn Med 2003.
In press.
22. Marcolides GA. Generalizability theory. In:
Tinsley HEA, Brown SD, eds. Handbook of
multivariate statistics and mathematical mod-
eling. San Diego, CA: Academic Press, 2000:
557--551.