Rating Satisfaction Research: Is it c]Poor,
0Fair, Good, 0Very Good, or c]Excellent?
Michele T Di Palo
Satisfaction is one of the core outcome measures for
health care. It is intuitively more appealing than mea-
sures of health care effectiveness or efficiency that are
more diffi’cult to understand. Satisfaction with health
care is a measure with a long history in the social
sciences. Most current research is less interested in cor-
relations between patient characteristics and satisfac-
tion and more focused on improving the quality of care
and service delivered to patients and health plan meni-
bers. While this article provides a brief overview of
historical satisfaction literature, the primary focus is
on current trends in measuring satisfaction, including
efforts by the National Committee for Quality Assur-
ance and the Foundation for Accountability. Limita-
tions of current satisfaction measures are discussed;
alternatives to using satisfaction to understand the ex-
perience of the health care system are suggested.
Key words. Patient satisfaction: Satisfaction with
care; Member satisfaction.
Introduction
Unlike the Rolling Stones in the 1960s, health care
consumers today can get plenty of reported satisfaction.
Hospitals and managed care organizations publicize
high satisfaction scores in local newspapers. Regional
coalitions, such as the Cleveland Health Quality Choice
Program and Massachusetts Health Care Purchasers
Group, publish satisfaction results on a regular basis.
Of all the standard outcome measures in health care,
Address correspondence to Michele T. Di Palo, ABD, Director,
Outcomes and Program Assessment, Blue Cross Blue Shield of Mas-
sachusetts, 100 Sunimer Street MS 01/07, Boston, MA 02110.
Submitted for publication J U ~ Y25, 1997; accepted in revised form
August 14, 1997.
0 1997 by t h e Amcrican College of Rheumatology.
422
satisfaction is the one that generates the most interest
on the part of consumers [l). But what is the quality
of current satisfaction research, and why is it so im-
portant?
Why is satisfaction with health care important?
The place of satisfaction in health care outcomes
research is well recognized. Satisfaction with care has
a long history in both academic social science (2) and
consumer research. It is included as one of the out-
comes measures in all of the standardized systems cur-
rently being used or developed for health care perfor-
mance. The National Committee for Quality Assurance
(NCQA) (3) includes a general satisfaction survey in its
Health Plan Employer Data and Information Set (HED-
IS) 3.0. The Foundation for Accountability (FACCT)
endorses both general and disease-specific satisfaction
measures in Accountability! Helping Americans Make
Better Healthcare Decisions (4). The Joint Commission
for the Accreditation of Healthcare Organizations ex-
plains its approach to satisfaction as part of the ORYX
system ( 5 ) .
The impact of health care cannot be fully understood
without considering the satisfaction of patients. Along
with measures of effectiveness (outcomes) and effi-
ciency (the ratio of benefit to cost), satisfaction is one
of the core outcome measures in health care. It is the
only 1 of the 3 that considers the affective reaction of
the patient to the care. The “feelings” aspect of satis-
faction gives it a more democratic air than outcomes
and efficiency research. The most effective, efficient
treatment or program is of limited use if it does not
fulfill the needs of the individuals or groups receiving
the service, i.e., if it does not produce satisfaction.
The public interest in satisfaction
The public has a strong, active interest in satisfac-
tion. Recent work with consumer groups has demon-
0893-7524/97/$5.00
Arthritis Care and Research
strated that most people currently have limited interest
in quality and cost information. In early 1997, academ-
ic and legislative experts in health care quality agreed
that public interest in such information is virtually
nonexistent (6).
Available quality of care information typically pre-
sents the rate of some treatment in an age- or gender-
specific population. The percentage of women receiv-
ing prenatal care in the first trimester, a measure in
HEDIS, is likely to be of little interest to men or women
who are not planning a family or are past childbearing
years. Even if a woman is planning to start a family,
the experience of a health plan’s entire population may
have little meaning. She may be confident that she will
obtain the care she needs when she needs it; the ex-
perience of others may have little meaning for her.
As increasing numbers of workers enroll in managed
care plans, the relevance of cost diminishes. Most
health maintenance organizations (HMOs)require only
a small co-payment for an office visit. Hospital stays
are entirely covered, and minimum length of stay may
become legislatively mandated. Under these circum-
stances, the relative cost of two alternative treatments
is of little interest to the average patient.
Satisfaction is a readily accepted concept that ap-
plies to all populations and can be expressed in very
personal terms. Restaurant, hotel, and movie reviews
all summarize whether a critic was satisfied with her
or his experience. It is precisely this “personal” aspect
of satisfaction that carries the most weight when con-
sumers express a preference for the sort of information
they want when making health care decisions. Mea-
sures of outcome and benefit relative to cost are less
common and less easy to explain. Initial research in-
dicates that health care consumers are more likely to
reject measures that they do not understand (1).
The Internet offers another example of the popular
importance of satisfaction as an outcome measure. Key
word searches on the phrases “patient satisfaction,”
“health care quality,” and “health status assessment”
were performed on July 2, 1997, using HotBot, a com-
mercial search engine on the Internet, and MEDLINE,
an electronic index for clinical and professional re-
search on health care issues.
While the absolute number of links or citations for
patient satisfaction were virtually identical-the Inter-
net showed 3,273 while MEDLINE had 3,265-the rel-
ative number when compared to citations regarding
health care quality and health status assessment was
vastly different. The Internet’s populist view showed
satisfaction to have far more links than either health
care quality, 1,886, or health status assessment, 118.
By contrast, patient satisfaction is a distant third be-
hind health status assessment and health care quality
Rating Satisfaction Research 423
in MEDLINE citations. Health status assessment had
8,502 citations; health care quality showed 7,293.
Professional research favors health care topics other
than patient satisfaction. There were more than 1,800
researchers and over one hundred sessions at the 1997
annual meeting of the Association for Health Services
Research. There were multiple-session tracks for access
to health care issues, for measuring quality and out-
comes, for examining the costs of health care, on the
relationship between health care and ethics and law,
and on health care management and state reforms.
There was a single presentation on satisfaction, and
the audience did not fill the room.
While not the favorite of researchers, satisfaction
does appear to be the “people’s choice” when it comes
to health care outcome measures. Satisfaction has the
“I know what I like” appeal. Perhaps the problem is
translating “I know what I like” into reliable health
care research.
A brief overview of satisfaction research
Early patient satisfaction citations are almost a half
century old (2). Much of this early research was con-
ducted as part of broader sociological research into
Americans’ expressed satisfaction with different areas
of their lives. Medical sociology as a separate subfield
of inquiry grew in the 1960s and 1970s. With its ex-
pansion came an enhanced focus on understanding the
behavior of the principle actors in medical care-phy-
sicians and patients.
Much of the patient-based research concentrated on
the relationship of sociodemographic characteristics to
care seeking, reactions to pain, and satisfaction. Many
of the variables that had been shown to make a dif-
ference in other sociological research proved to be un-
related to patient satisfaction. There was no clear pat-
tern of association between satisfaction levels and
patients’ age, social class, sex, race, occupation, or re-
ligion (7-9). Educational level was not clearly related
to satisfaction,although lower income appeared to lead
to lower satisfaction with medical care (8,9). Age also
proved to be equivocal, with some studies suggesting
older people were more satisfied with the medical care
they received than younger people (7).
Satisfaction was of interest not only as an outcome,
but as an intervening factor that influenced patient ad-
herence (10). Both satisfaction and adherence are pre-
sumed to have an effect on other outcomes such as
health status, continuity of care, and subsequent need
for inpatient care (11).
Research in the 1980s began to focus more on the
relationship between patient satisfaction and the or-
ganization of medical care. With the rise of HMOs and
424 Di Palo Vol. 10,No. 6, December 1997

other forms of managed care, the effect of different termined by the interaction of this system of beliefs,
health care and payment structures on satisfaction as- the actual occurrences of the encounter, and the com-
sumed new importance. Most of the earlier studies parison of that encounter to others like it.
were conducted in staff or clinic model HMOs and
showed that fee-for-service models were associated Whose satisfaction?
with higher levels of satisfaction. However, at least one
In its overview of satisfaction, FACCT distinguishes
study found that while the patients who were part of
between 3 different forms: satisfaction with a health
a HMO initially had the lowest level of satisfaction,
plan, satisfaction with a physician, and satisfaction
their satisfaction increased the longer they stayed in
with a hospital (5). Health plan satisfaction items are
the HMO, while the satisfaction of patients in the fee-
aimed at the consumer as member. Member satisfaction
for-service system decreased across time (12).
items ask about things that are more directly controlled
At the same time, there was exploratory research on
by the plan itself. NCQA’s health plan-related questions
the ability of patients to judge the competence of their
ask members to rate the range of services covered by
physicians (13). Studies showed that ratings of tech-
the health plan, the availability of information from the
nical competence were highly dependent on the com-
plan, the availability of information about the costs of
munication skills of the physician. Physicians who
care, the length of time spent filling out forms, and the
were judged to be good communicators were also be-
amount of premium paid. These member satisfaction
lieved to be technically competent (7). Although tech-
items are more closely tied to service-oriented ques-
nical competency is a requisite for effective care, these
tions common in consumer satisfaction surveys con-
findings directed efforts away horn attempts to have
ducted by airlines or retail stores. While of importance
patients assess care and focused research back on pa-
to understanding overall satisfaction with the managed
tient satisfaction.
care experience, these items are not related to the actual
experience of care.
What is satisfaction? Satisfaction with physicians and hospitals focuses
on patients and their interactions with individual or
In a review article that studied the content of satis-
institutional care givers. Patient satisfaction addresses
faction surveys, John Ware and colleagues described at
aspects of the medical care process. Physician satis-
least 10 distinct dimensions of satisfaction: accessibil-
faction questions that examine issues of access to care,
ity/convenience, availability of resources, continuity of
thoroughness of care, interaction with physician, and
care, efficacy/outcomes of care, finances, humaneness,
outcome of care are contained in NCQA’s satisfaction
information gathering, information giving, pleasant-
survey. These items are intended to help the plan and
ness of surroundings, and quality/competence of care
the public decide how well a particular plan’s network
giver (14). These constructs have remained fairly con-
of physicians performs relative to another plan’s net-
sistent over time and have been adopted in many of
work. The results are highly ambiguous in those areas
the most widely used patient satisfaction surveys
of the country where competing plans contract with
(15,16).
the same physicians. Under these circumstances, vari-
Much of the work in patient satisfaction focuses on
ations in patient satisfaction are difficult to understand
methods and tools to measure satisfaction (14). Yet,
and interpret.
cataloguing the most prevalent questions asked about
Hospital satisfaction typically incorporates a broad
satisfaction does not provide a definition of satisfac-
range of experiences; hospital administration, ameni-
tion. Quantifying satisfaction, through standardized
ties, and care may all be covered by a single question-
surveys, has been more important than qualifying sat-
naire. Individual items ask about the convenience of
isfaction by establishing explicit definitions. Theories
hospital parking, the time required to register, the com-
of patient satisfaction are rare.
fort of the room and quality of the food, and the
Most of the theoretical research on patient satisfac-
promptness and compassion of the attending staff. Un-
tion is built on attitude theory and job satisfaction re-
til recently, hospital satisfaction surveys were restricted
search (17). Patient satisfaction is defined as an indi-
to the institutional experience. They seldom addressed
vidual’s positive evaluations of distinct dimensions of
patient experiences beyond the hospital walls.
health care (18). These evaluations are based on the
patients’ expectations (set of beliefs), values (whether
aspects of the health care encounter are important), and
Satisfaction today: national efforts to collect
sense of entitlement (believing he or she has a right to
satisfaction data
expect a particular outcome). Patients bring these at- As the article in this issue by Mason indicates (“Out-
titudes to the health care encounter. Satisfaction is de- comes Measurement in Today’s Health Care Environ-
Arthritis Care and Research
ment,” p. 355-a), current health services research is
no longer being conducted exclusively by academics
or sponsored by foundations. As a result of the con-
sumer movement and the call for accountability, health
services research in the 1990s is driven by the need to
measure the current state of health care and demon-
strate improvement. Spurred by pressures to be ac-
credited and to be accountable for their populations,
HMOs are conducting research that examines the pro-
cesses and outcomes of care. Satisfaction has been
adopted in the two most prominent national efforts to
standardize measurement across health plans-NCQA
and FACCT.
The NCQA includes satisfaction as a measure in its
HEDIS. The HEDIS satisfaction survey is conducted
annually. Standardized for 1996 (HEDIS 3.0), health
plans must contract with an independent vendor who
handles all aspects of member contact. A randomly
selected sample of health plan members receives a
printed survey that contains a broad range of satisfac-
tion items, the Short Form 1 2 (SF-12; 19), and a check-
list of selected chronic conditions, including arthritis,
migraine, hypertension, depression, and ulcers. The
HEDIS satisfaction survey contains questions on sat-
isfaction with a health plan’s service as well as with
physician care.
While FACCT has said that it is “not ready to declare
a definitive strategy for assessing consumer satisfac-
tion,” it has endorsed NCQA’s Annual Member Health
Care Survey as a measure of general satisfaction (4). In
additional to this generic measure, FACCT has focused
on the outcomes of 3 discrete diseases: breast cancer,
major affective disorders, and diabetes. It has included
satisfaction as one of its measures. Exemplifying its
broader policy to promulgate existing measures rather
than develop its own, FACCT has adopted a 32-item
satisfaction survey developed and used by an Oregon
health plan for assessing satisfaction with care for
breast cancer (20). The Behavioral Healthcare Rating
of Satisfaction developed at the Department of Com-
munity Health at the Florida Mental Health Institute
was chosen for satisfaction with treatment for major
affective disorders (21). Satisfaction with diabetes care
is measured using selected items from the Group
Health Association of America Consumer Satisfaction
Survey (15).
The use of standardized satisfaction surveys pro-
vides employers and individuals the ability to compare
the relative satisfaction expressed by members of dif-
ferent health plans. However, having a method that is
capable of producing comparable data does not nec-
essarily produce the most meaningful data.
Questions have already arisen about the usefulness
of individual items and about the structure of the
Rating Satisfaction Research 425
NCQA satisfaction survey. Health care plans took part
in a structured study that compared the standardized
NCQA satisfaction form with a modified version of the
survey (22). The SF-12 was included in the satisfaction
survey to adjust results based on the self-reported
health status of individual members. A list of comor-
bidities including hypertension, arthritis, heart dis-
eases, diabetes, and a number of other chronic con-
ditions was also presented. Statistical analyses were
conducted to determine whether the whole SF-12 and
a summation of all comorbidities were useful in ex-
plaining variation in the global satisfaction items. The
results strongly suggested that none of the comorbid-
ities and only 4 items on the SF-12 (general health,
reported health transition, mental health, and social
functioning) affect global satisfaction (22).
Questions about the response scale used were also
addressed. The 5-point scale, typical in standardized
satisfaction surveys, implicitly assumes that respon-
dents have had certain experiences and have opinions
about those experiences. Forced to evaluate experiences
about which they have no knowledge, a respondent may
either choose a random response to the question or may
choose an answer that is consistent with other experi-
ences. Providing a “No Opinion” or “No Experience”
response resulted in satistically significant higher cor-
relations with the global satisfaction items (22).
NCQA changed the response categories for HEDIS
3.0, which was then used to collect 1996 data. The
choice “No Experience” was incorporated into the
form. However, the entire SF-12 and the full list of
comorbidities was retained. Given how generally ac-
cepted the SF-12 is and how widely it is held that
outcomes measures need to be risk adjusted, it may
take longer to acknowledge that . . . there is little ev-
“
idence . . . that supports its (the SF-12’s)value (taking
the scale as a whole) for risk adjustment of satisfaction
data” (22). Without proven value, these 29 items take
up most of one page of a 4-page survey. As NCQA
continues to improve its satisfaction survey, it must
balance the length of the survey against the usefulness
of the information it collects.
Measuring satisfaction with health care: issues
and (some) answers
Collecting satisfaction data may be the easiest part
of the process. Understanding and using satisfaction
information is a challenge that all groups interested in
improvement face. The single session at the Associa-
tion for Health Services Research that discussed patient
satisfaction was titled “Methodological Issues Sur-
rounding the Use of Satisfaction Data” (23). Based on
information provided by health plan executives, 6
426 DiPalo
problems with satisfaction data were identified. The 6
areas also represent core questions that should be con-
sidered whenever a satisfaction survey is being con-
ducted. They form a primer for satisfaction researchers
and should guide thought by both researchers and con-
sumers whenever satisfaction information is presented.
Sampling. Always a concern in determining wheth-
er data are representative of an entire population, sam-
pling becomes more important if satisfaction for dif-
ferent subpopulations is of interest. If a health plan
wants to find out about overall satisfaction with the
service it delivers, a relatively small sample of the pop-
ulation might be sufficient. But what if it wants to com-
pare the satisfaction of young single people to young
married people? What about comparing young single
people to young married people to young married peo-
ple with children? What if groups with particular con-
ditions or diseases are of interest?
Providers face a similar challenge. If overall satis-
faction is of interest, then sampling is fairly straight-
forward. But what if the interest is rheumatic diseases?
What is the appropriate proportion of patients with
arthritis, fibromyalgia, and lupus to use in a sample
that determines satisfaction with care for rheumatic
conditions?
Advocacy groups that are interested in how people
with a particular condition or disease fare in the health
care system also need to consider sampling. Once the
group moves beyond the overall level of satisfaction
with care provided, the satisfaction of a number of
subpopulations may be of interest. Are those who are
most severely affected more or less happy with the care
they are receiving? Does being treated by a specialist,
as opposed to a primary care physician, affect how care
is rated?
The questions being asked determine what groups
need to be in the sample and in what proportion in
order to guarantee that the sample accurately reflects
the whole population. Provided the research questions
are specified before the survey is conducted, statistical
techniques can be used to determine the proper size
and composition of the sample. But if analyses are de-
signed post hoc, groups may be misrepresented or un-
derrepresented in the survey sample, and erroneous
conclusions can be drawn.
Method. As in any research, good results only occur
if good methods are used. All aspects of the satisfaction
survey need to considered: What are the implications
of different forms of administration? Are standardized
or specialized surveys more applicable? What is the
most appropriate distribution interval? What is the cor-
rect trade-off between collecting detailed information
Vol. 10,No. 6, December 1997
and burden on the respondent? Will rating or reporting
the interactions yield the most useful information?
Social science research has shown that different
methods of administering surveys can influence re-
sponse rates. Personal face-to-face interviews used in
federally funded national health surveys can obtain
response rates of 85% to over 95% (24). Telephone
surveys average 70-85%, with mailed surveys having
the lowest response rates (in the range of 60-70% or
lower) (25).
Current practice is to use mailed or telephone sat-
isfaction surveys that are easily quantifiable. There is
some evidence that older people, people with certain
disabilities, and those who tire easily or who have
problems with chronic fatigue, as is the case with sev-
eral rheumatic diseases, decline participation in tele-
phone surveys (26). These groups could be unknow-
ingly underrepresented in telephone surveys of
satisfaction.
As the number of national and regional organizations
and coalitions increases, the question of whether to use
standardized satisfaction forms will arise more fre-
quently. Academic researchers interested in a partic-
ular aspect of satisfaction or in experimenting with new
ways to measure satisfaction may continue to use sit-
uation-specific surveys that are unrelated to existing
forms in wider use. Health plans and hospitals, which
are under increasing pressure to release data publicly,
are more likely to have to consider using standardized
surveys. Comparability of results is an issue in these
situations.
Standardized forms typically contain the most ge-
neric questions-questions that are applicable across
settings and populations. Satisfaction research appears
to be following the direction of other medical research,
focusing on populations and conditions that are most
in the public eye or that affect large numbers of people
(4). In order to accurately capture aspects of care for a
given condition, advocacy groups for lesser known
conditions may be forced to develop their own satis-
faction surveys. Unlike standardized surveys, these
special efforts will neither be validated or be backed
by a history of use. This can put the advocacy group
at a disadvantage if they try to use their results to sup-
port claims of lower patient satisfaction than is dem-
onstrated by well-established tools.
Selecting intervals for collecting data depends on
how the data are to be used. The patient satisfaction
survey used by NCQA is part of their annual battery
of measures. While convenient for employers and con-
sumers, who typically can switch health care plans
only once a year, the results may not be suitable for
quality improvement efforts. Quality improvement ac-
tivities require more fkequent measurement cycles to
Arthritis Care and Research
track the effect of an improvement effort. Under these
conditions, it is possible for health plan members or
patients to find themselves in multiple rounds of sat-
isfaction surveys. Trying to understand satisfaction
may have the paradoxical effect of lowering it through
constant monitoring.
The more direct interest repondents have in the topic
of the survey, the more likely they are to complete it.
Still, longer surveys, particularly longer written sur-
veys, are likely to result in higher rates of nonresponse
(24). Telephone surveys on health-related topics have
been successful at lengths up to one hour or more (26).
Face-to-face surveys presumably have the greatest lat-
itude in length of survey, probably related to the per-
sonal contact the interviewer has with the respondent
(28). However, the same personal touch that allows the
interviewer to spend more time collecting information
from a respondent raises the possibility of an inter-
viewer effect, possibly biasing the results (24).
One current controversy is whether surveys that ask
about a health care encounter should have the patient
rate the interaction or report on its contents. The tra-
ditional approach has been to have members or pa-
tients subjectively rate specific aspects of a health care
encounter. An individual who has received health care
could be asked about his or her level of satisfaction
with the time spent in the waiting room, or about the
communication he or she had with the physician. Sat-
isfaction levels are typically standardized to the 5-point
scale familiar to anyone who has completed a consum-
er survey: very satisfied, satisfied, neither satisfied nor
dissatisfied, dissatisfied, very dissatisfied. A variant on
satisfaction ratings is to substitute excellent, very good,
good, fair, and poor for the 5-point satisfaction scale.
NCQA uses this scale in their annual satisfaction sur-
vey conducted as a part of the HEDIS data set (29).
In surveys that rely on rating, experience has shown
that data responses cluster at the high end of the range
(30).When there is little “spread” in results, meaningful
differences are difficult to establish. The HMO for which
I work recently completed two years of patient satisfac-
tion surveys. On a 5-point scale, with 5 being “very
satisfied,” the average score for the satisfaction with
treatment scale was 4.6 (31). These results may not be
surprising-if members do not like their physicians they
are likely to “vote with their feet” and find another doc-
tor-but the uniformly high scores do little to help iden-
tify physicians providing poorer quality of care.
When reporting methods are used, the patient is
asked what happened in a health care encounter, not
for opinions about that encounter. Rather than being
dependent on a subjective rating of a visit, this method
ascertains whether or not certain aspects of care oc-
curred. By structuring the survey to collect information
Rating Satisfaction Research 427
on agreed upon standards of good care, reporting can
help determine whether expected care is occurring. For
treating chronic conditions such as arthritis, where pa-
tient education and pain control are central to good
clinical management, patients can be asked whether
they are getting the information they need to care for
themselves, whether the physician has asked them
about their burden of pain, and whether the physician
has discussed pain management with them. Reporting
can act as an indirect measure of basic technical com-
petency by collecting information on what patients re-
call about their visits and comparing their experiences
to accepted standards.
Responses. What response rate is adequate? What
are the implications of declining response rates? How
do proxy respondents affect expressed levels of satis-
faction?
Response rate questions are based on concerns about
nonresponse bias. If the population that chooses not to
respond differs from the population that has returned
the satisfaction survey, then the results may not truly
represent the whole population. While there are statis-
tical methods that can adjust for some of the possible
sources of nonresponse bias, the best solution is to have
an adequate response rate. “Adequate” is often deter-
mined by the organization administering or sponsoring
the survey. While consulting firms that conduct satis-
faction surveys may consider a 3 0 4 5 % return rate “ad-
equate,” FACCT and NCQA have not specified a re-
quired return rate. NCQA does report the return rate for
each health plan. The range is as wide as 2 6 7 0 % (32).
Satisfaction surveys are returned in “waves.” As-
suming the survey is mailed out, a certain percentage
of respondents will return the survey by the deadline
on the form without further contact. These surveys rep-
resent the first wave. A reminder, typically a mailed
postcard, will then be sent to those who have not re-
turned their forms. The group that mails their surveys
back after this postcard is the second wave. From this
point on, a series of reminders will be sent to people
who have not yet responded; each represents a differ-
ent response wave.
Initial analysis of results to the HEDIS satisfaction
survey has shown that satisfaction varies systemati-
cally by wave. Members who respond in the first wave
tend to show higher levels of satisfaction than succes-
sive waves (32). The more “waves” needed to obtain
a particular response rate, the more likely satisfaction
will be lower. “Wave” information is seldom presented,
yet can affect reported levels of satisfaction.
Clearly, satisfaction information is best collected di-
rectly from individuals who receive the care or service
in question. When dealing with satisfaction with health
428 Di Palo
care, there are some circumstances under which the
ideal is not always possible. Satisfaction with care for
patients who are acutely ill, permanently unable to re-
spond due to a chronic condition or severe injury, or
simply too young may be of interest to health plans,
employers, or advocacy groups. In these situations all
responses are proxy responses. In others instances, there
may be a blend of proxy and intended respondents.
Research has shown that, under certain circumstanc-
es, intended respondents’ and proxy respondents’ sat-
isfaction ratings are moderately correlated (33). The
strength of the correlation is stronger if the proxy sees
the intended respondent more often. It is stronger still
if the proxy lives with the respondent. This variation
suggests that proxy reporting should be minimized when-
ever possible, and when used, suitable proxies should
be clearly defined (i.e., relationship to the intended re-
spondent, whether respondent is primary caregiver).
Analytic issues. Is there a need to adjust satisfaction
results based on population characteristics, such as se-
verity of illness?
The issue of severity of illness affects both research
in satisfaction and health outcomes research. Few re-
searchers would argue that there is a need to adjust for
outcomes based on severity of illness (34). The answer
is less clear with satisfaction. The HEDIS survey in-
dicates that several chronic conditions that would nor-
mally be used to adjust results have no significant effect
on expressed satisfaction (22). Data from the Medical
Outcomes Study also fail to show a link between dif-
ferences in physical health status and satisfaction.
There is, however, an association between mental
health status and satisfaction (35). While preliminary,
these studies suggest that adjusted satisfaction results
based on severity of illness may be of limited use.
Results. How do you interpret differences in satis-
faction? What differences are actionable?
Satisfaction with health care scores tend to cluster
in the high end of the available range. Given the lack
of distribution, how can health plans, hospitals, pur-
chasers, or patients interpret small variations in re-
ported satisfaction? It is possible to determine whether
there are “statistically significant” differences between
satisfaction scores. But statistically significant differ-
ences may not provide enough help in interpreting sat-
isfaction results. The previously mentioned satisfaction
survey completed by the HMO for which I work in-
volved almost 100,000 people. The sheer size of the
sample means that any observed difference is statisti-
cally significant.
The results showed that women were less likely than
men to recommend their primary care physicians to
Vol. 10,No. 6, December 1997
family and friends. Information from the survey was
discussed within the organization and distributed to
primary care physicians. But how to act on the infor-
mation to improve satisfaction is unclear; we certainly
do not want physicians to try to recruit more men in
order to increase their satisfaction ratings.
Use of results. Will the results be used within an
organization or will they be distributed more widely?
What is the best reporting format?
Satisfaction surveys have at least two purposes. The
results can be used to identify opportunities for im-
provement or they can act as a marketing tool. If the
survey is strictly for internal use, more specific,tougher
questions that may generate less favorable responses
may be asked. A survey for external distribution is less
likely to focus on a plan’s shortcomings and more likely
to try to find ways to emphasize the positive. Especially
in competitive markets, satisfaction surveys that are
available to the public will be aimed at collecting and
spreading good news.
Standardized efforts, such as NCQA, do not solve
the problem of satisfaction surveys designed to collect
and report the most favorable reviews. Paradoxically,
NCQA may increase the likelihood of health plans de-
veloping their own satisfaction surveys with a mar-
keting orientation. With its strict rules on how health
plans can report data to the public, it may encourage
the use of independent surveys that can be used with-
out restrictions or disclaimers in advertisements.
Of all the problems discussed, reporting format is
generating the most interest and research. The Agency
for Health Care Policy and Research is funding work
that is aimed at understanding how consumers use data
and what format of data is most effective. Consumer
Reports has demonstrated that if complicated data are
presented in a simple, straightforward way, consumers
will use them to help make purchasing decisions. The
best example may be the “half-moon” tables that sum-
marize the repair records of late model autos that ap-
pear in the magazine annually.
In an attempt to emulate that success, regional busi-
ness coalitions and health care plans have experimented
with displaying satisfaction data. Stars, apples, and half-
moons have all been used, but the core issue remains
the same: what is the most appropriate depiction of a
difference of some number of percentage points? Should
the plan or physician or hospital with the “worst” sat-
isfaction score get 1 star, while the “best” gets 5 stars?
What if, as is typically the case, the numerical differ-
ences between plans are small? In a recent example, a
health plan with an 87.3% satisfaction rating received
2 of 5 stars. A plan that scored 89.9 received 3. Are 2
points difference, out of 100,worth a star? (36).
Arthritis Care and Research
Satisfaction and problem identification
Today’s satisfaction surveys, which primarily focus
on the one-to-one, patient-caregiver relationship, may
be better suited to marketing health care providers than
to health care improvement. Asked to judge their own
doctor, nurse, or hospital, the overwhelming majority
of patients say that they are satisfied. Under these cir-
cumstances, satisfaction surveys can do little to help
improve health care.
Perhaps a more appropriate construct for under-
standing the patient experience of health care and for
identifying ways to improve health care encounters lies
in a concept mentioned previously: moving from rating
satisfaction to reporting problems. The Picker Institute
is a nonprofit organization that advocates problem
identification and reporting as a patient-centered ap-
proach to improving health care ( 3 7 ) .
Moving away from rating and into reporting means
moving from perception (“how satisfied”) to events
(“how many times”). Satisfaction surveys that address
the issue of access by asking a patient how satisfied
she or he is with the “usual wait” for an appointment
with a primary care physician may be both concep-
tualizing access in too limited a way as well as placing
the emphasis on the emotional over the factual. A pa-
tient may be perfectly satisfied with the wait time to
see the primary care physician, but can still feel that
the managed care “gatekeeper” system acts as a barrier
to more specialized care. Asking whether patients have
ever faced a delay or denial in receiving needed care
more directly addresses concerns about a broader form
of access to the health care system. Reporting “how
many times” rather than “how satisfied” also removes
some of the uncertainty that surrounds using the results
of satisfaction surveys to improve health care. The
HEDIS annual satisfaction survey already asks health
plan members “how many times” they have been ad-
vised to quit smoking by their physician (29).
Access to medical care is 1 of the 8 domains of care
that patients have identified as being important ( 3 7 ) .
The other 7 are: respect for patients as individuals,
coordination among caregivers, access to needed in-
formation at appropriate times, physically comfortable
surroundings, emotional support and assurance, the
involvement of family and friends in care, and tran-
sition and continuity from care to community. The pa-
tient survey designed by the Picker Institute seeks to
understand the experience of care “through the pa-
tients’ eyes.” It contains items that help identify prob-
lems with each of the 8 domains of care. In addition,
pain management is addressed in a separate section of
the survey so that patients can report their caregivers’
effectiveness in controlling pain ( 3 8 ) . While not fitting
Rating Satisfaction Research 429
neatly into any of the identified domains, Picker ac-
knowledges that pain management can affect the phys-
ical and psychological state of patients and deserves
attention.
Organized so that it moves patients chronologically
through their experience of care, the Picker survey asks
about problems encountered throughout the health
care system. Results are presented as the percentage of
patients reporting problems with each of the aspects
of care and for each individual item.
This problem-oriented approach raises the interest-
ing possibility that a single form of satisfaction cannot
be all things to all health care constituencies. Consum-
ers may prefer hearing how many of their neighbors
recommend their health care plan or hospital. At the
same time, health plan and hospital executives may
need to use a problem-oriented approach to identify
ways to improve their systems so that more neighbors
will recommend them. The question may not be “What
is satisfaction?” but “Who wants to know?”
Rating satisfaction research
The public interest in health is excellent. The
intuitive appeal of satisfaction as an outcome measure
for health care is undeniable. Its emphasis on the per-
sonal aspects of care and service and its use in other
domains of life (general consumer research, restaurant
and hotel reviews) make it familiar and easy to un-
derstand.
The current environment for satisfaction research is
excellent. The most prominent national outcome
measurement efforts (NCQA and FACCT)both support
and require satisfaction surveys.
Satisfaction research also has an excellenthistory.
Most of the associations between sociodemographic
variables, the organization of medical care, and phy-
sician-patient communication have already been ex-
plored. Current interest is on the aspects of care and
service that result in people recommending their health
care plan or provider.
Work on the theory of health care satisfaction is
!Jfair. The few studies that have emerged borrow
heavily from attitude and job satisfaction literature.
However, the specification of items for general satis-
faction surveys is excellent. Differences in health
plan, physician, and hospital satisfaction have been
discussed, and items specific to each identified. The
multidimensional constructs of satisfaction have been
well documented and validated.
Using satisfaction information as a driver of quality
improvement is good. The rapid incorporation of
satisfaction research into health care has produced
(healthy?) skepticism about all aspects of survey de-
430 Di Pafo
sign, data collection, data analysis, and data interpre-
tation. Familiarity with the surveys will breed either
contempt or comfort.
NCQA and FACCT are working with interested
health care systems to collect satisfaction data. State
and national efforts are exploring ways to transform
those data into useful consumer information. New
ways of conceptualizing satisfaction may lead to more
precise information on how to improve care. The future
of satisfaction research is excellent.
REFERENCES
I. Oregon pinpoints what consumers really want to know
about their health plan. Rep Med Guidelines Outcomes
Res 1996;7:7-9.
2. Koos EL. Metropolis-what city people think of their
medical services. Am J Public Health 1955;45:1551-7.
3. National Committee for Quality Assurance. URL: http:
//www.ncqa.org./news/hedismeas.htm; June 1997.
4. Foundation for Accountability. Accountability! Helping
Americans make better healthcare decisions. Portland,
(OR): Foundation for Accountability; 1996.
5. Joint Commission for the Accreditation of Healthcare
Organizations. URL: http://www.JCAHO.org/perfmeas/ 28. White AA. A response rate calculation in RRD tele-
oryxloryxfrm.htm; June 1997.
6. Public demand for quality information ‘almost nil’.
Managed Care Outlook 1997;10:9.
7. Di Matteo MR, Hays R. The significance of patients’
perceptions of physician conduct. J Community Health
1980;6:18-34.
8. Greenley JR, Schoenherr RA. Organization effects on
client satisfaction with humaneness of service. J Health
SOCBehav 1981;22:2-18.
9. Romm FJ, Hulka BS. Care process and patient outcome
in diabetes mellitus. Med Care 1976;14:765-76.
10. Kincey J, Bradshaw P, Ley P. Patients’ satisfaction and
reported acceptance of advice in general practice. J R
Coll Gen Practitioners 1975;25:558-66.
11. Lochman J. Factors related to patients’ satisfaction with
their medical care. J Community Health 1983;9:91-109.
12. Ross CE, Wheaton B, Duff RS. Client satisfaction and
the organization of medical practice: why time counts.
J Health SOCBehav 1981;22:243-55.
13. Segall A, Burnett M. Patient evaluation of physician role
performance. SOCSci Med 1980;14A:269-78.
14. Ware JE, Davies-Avery A, Stewart AL. The measurement
and meaning of patient satisfaction. Health Med Care
Serv Rev 1978;l:l-12.
15. Davies AR, Ware JE.GHAA’s consumer satisfaction sur-
vey. 2nd ed. Washington (DC): GHAA; 1991.
16. Ware JE, Hays RD. Methods for measuring patient sat-
isfaction with specific medical outcomes. Med Care
1988;26:393-402.
17. Fishbein M, Ajzen I. Belief, attitude, intention and be-
Vol. 10, No. 6, December 1997
havior: an introduction to theory and research. Reading
(MA): Addison-Wesley; 1975.
18. Linder-Pelz S. Toward a theory of patient satisfaction.
SOCSci Med 1982;16:577-82.
19. Ware JE, Kosinski M, Keller SD. A 1 2 item short-form
health survey (SF-12): construction of scales and pre-
liminary tests of reliability and validity. Med Care 1996;
32 :2 20-3 3.
20. Skokan L. Satisfaction with breast cancer care survey.
Portland (OR): Providence Health System; 1994.
21. Ward J. The behavioral healthcare rating of satisfaction
(BHRS). Tampa: University of Southern Florida; 1995.
22. Kippen LS, Strasser S, Joshi M. Improving the quality
of the NCQA annual member health care survey version
1.0. Am J Managed Care 1997;3:719-30.
23. Davies A. Methological issues surrounding the use of
satisfaction data. Association for Health Services Re-
search Conference; 1997 June 16; Chicago, IL.
24. Aday LA. Designing and conducting healthsurveys. San
Francisco: Jossey-Bass; 1989.
25. Marcus A, Crane LA. Telephone surveys in public
health research. Med Care 1986;24:97-112.
26. Cannell C, Thornberry OT, Fuchsberg R. Research on
interviewing techniques: sociological methodology. San
Francisco: Jossey-Bass; 1982.
27. Fleming GV, Andersen R.Can access be improved while
controlling costs? Chicago: Pluribus Press; 1986.
phone surveys: current practices. Washington (DC):
American Statistical Association; 1983.
29. National Committee on Quality Assurance. Member sat-
isfaction survey for HEDIS v. 3.0. Washington (DC):Na-
tional Committee on Quality Assurance; 1997.
30. Mason J, Ashe L, Souza J. Using patient satisfaction data
to improve physician performance. Institute for Health-
care Improvement Conference; 1996 Dec 4-7; New Or-
leans, LA.
31. Blue Cross Blue Shield of Massachusetts. Health services
evaluation: contributions to the management of care. Bos-
ton: Blue Cross Blue Shield of Massachusetts; 1996.
32. Kosinski M. Methological issues surrounding the use of
satisfaction data. Association for Health Services Re-
search Conference; 1997 June 16; Chicago, IL.
33. Epstein A, Hall J, Tognetti J, Son L, Conant L Jr. Using
proxies to evaluate quality of life. Med Care 1989;27:
S91-8.
34. Hannan EL, Kilburn H Jr, Racz M, Shields E, Chassin
MR. Improving the outcomes of coronary artery bypass
surgery in New York State. JAMA 1994;271:761-6.
35. Marshall GN, Hays RD, Maze1 R. Health status and sat-
isfaction with health care: results from the Medical Out-
comes Study. J Consult Clin Pyschol 1996;64:380-90.
36. Pham A. Insurer rejects ratings. Boston Globe 1996 Oct
18;C-1.
37. Picker Institute results. URL: http://www.amhpi.com/
eyeonpatients; June 1996.
38. The Picker Institute. Patient satisfaction form MS2.4.
Boston: The Picker Institute; 1995.