Professional Psychology: Research and Practice 1996, Vol. 27, No.

2, 202-208

Copyright 1996 by the American Psychological Association, Inc. 0735-7028/96/$3.00

Relationship of Satisfaction to Symptom Change, Follow-Up Adjustment, and Clinical Significance

Gene Pekarik
Antioch New England Graduate School

Christian B. Wolff
Washburn University

The relationship of client satisfaction to outcome was investigated for adult outpatients (N = 152) from 3 urban community mental health centers. Clients completed a problem self-rating and the Brief Symptom Inventory (BSI) at intake, 10 weeks later, and 5 months later. Therapists' ratings of client adjustment were obtained at intake and termination. Clients' ratings of satisfaction with treatment were obtained at 10-week and 5-month follow-up.Correlations between satisfaction and client measures of outcome (client rating and BSI) based on pre-post changes, posttreatment adjustment, and Jacobson and Truax's (1991) method of measuring clinical significance were not significant.Correlations between satisfaction and therapist outcome ratings were significantbut low for pre-post changes and clinical significance.The utility of client satisfaction as an outcome measure is discussed.

Professionals in the mental health fields, including clinical psychologists, have shown an ambivalence toward the formal assessment of psychotherapy outcome. Although an increasingly large number of outcome studies have appeared in recent decades, there has been a distinct lack of published outcome research originating from standard practice settings (Linden & Wen, 1990; Stiles, Shapiro, & Elliott, 1986 ). The debate over health care reform and managed-competition solutions appears to be giving impetus to increased evaluation of psychotherapy outcome (Sleek, 1994). As occurred during the last era of federal initiatives relevant to assessing quality of care (i.e., the extension and expansion of the community mental health centers program through the Public Health Service Act ammendments in 1975 and the Mental Health Systems Act in 1980), it is likely that client satisfaction will become a popular means of evaluating psychotherapy services. Indeed, there is already evidence that satisfaction assessment is the core of many current quality-assurance procedures (Greenfield & Attkisson, 1989; Lambert, Shapiro, & Bergin, 1986; Winegar, 1992). Satisfaction measures have several virtues, including ease of administration, high face validity, and appeal as indexes of

GENEPEKARIKreceivedhis PhD in clinical psychologyfrom State Universityof New Yorkat Stony Brook in 1977. He is currently the director of the Center for Research on PsychologicalPractice and a core faculty member of the clinical psychologydoctoral program at Antioch New England Graduate School, Keene, New Hampshire. His main research interests concern psychosocial treatment process and outcome in applied settings. CHRISTIANI. WOLFFreceived his MA in psychologywith an emphasis in clinical skills from Washburn University in 1993. He is currently employed as a family reunification counselor at Youthworks, Inc., in Medford, Oregon. CORRESPONDENCE CONCERNING THIS ARTICLE should be addressed to Gene Pekarik, Department of ClinicalPsychology,Antioch New England Graduate School, 40 AvonStreet, Keene, New Hampshire 03431-3516. 202

treatment acceptability (Berger, 1983; Lebow, 1982). One of the weaknesses of satisfaction measures is that there has been little systematic investigation of their relationship to other outcome measures (Greenfield & Attkisson, 1989), although their use as a quality-assurance measure presumes a high relationship with other, more traditional measures of outcome. Research has indicated that the correlations between satisfaction and other outcome measures are low to modest, with correlations generally ranging approximately from zero to .40 (Attkisson & Zwick, 1982; Carscaddon, George, & Wells, 1990; Edwards, Yarvis, Mueller, & Langsley, 1978; Fiester, 1979; Garfield, Prager, & Bergin, 1971; Greenfield & Attkisson, 1989; Nguyen, Attkisson, & Stegner, 1983; Pekarik, 1992; Wilier & Miller, 1978). Advocates of satisfaction assessment (e.g., Berger, 1983; Greenfield & Attkisson, 1989; Larsen, Attkisson, Hargreaves, & Nguyen, 1979) have cited compelling logical reasons to assess client satisfaction, but research done to date suggests that it is quite independent of traditional outcome measures. Much further research is needed to understand the nature of client satisfaction and its relation to outcome, for there are several limitations of the current research literature. A general and important limitation is that relatively few studies have addressed this relationship. A second limitation is that most research has been conducted in inpatient or college counseling-clinic settings. In addition to this general need for further study of satisfaction and outcome in standard practice settings, there are other specific issues that warrant investigation: 1. There is evidence that therapist and client ratings of outcome at treatment termination or follow-up are heavily influenced by absolute adjustment at that point regardless of degree of improvement from pretreatment (Lambert & Hill, 1994; Lambert, Shapiro, & Bergin, 1986). It is quite plausible that satisfaction, virtually always assessed at termination or followup, would be similarly influenced by such end-point adjustment. The meaning of the construct satisfaction could be clari-

SATISFACTION AND OUTCOME fled by addressing the relative relationship o f e n d - p o i n t adjustm e n t versus degree o f i m p r o v e m e n t (pre- to p o s t t r e a t m e n t changes) to satisfaction. 2. A l t h o u g h a few studies have investigated the relationship o f satisfaction to different o u t c o m e measures, n o n e h a s identified the degree o f relationship between satisfaction a n d each o f several o u t c o m e m e a s u r e s b y using multivariate statistical procedures. T h a t is, at present, we do n o t k n o w w h i c h o u t c o m e measures are m o s t a n d least related to satisfaction. 3. Recently, researchers have noted that statistically significant change may not be clinically significant change and have devised ways to identify the latter (Jacobson & Truax, 1991 ). By more rigorously identifying clinical successes and failures through the use o f Jacobson's clinical significance procedure, a more valid relationship between satisfaction and outcome may be found. T h e true relationship between satisfaction a n d outcome may have been obscured in previous studies by reliance on weak methods o f defining client improvement. Indeed, a recent study at a university clinic found satisfaction was more related to s y m p t o m change when measured by clinical significance procedures t h a n by more traditional methods ( A n k u t a & Abeles, 1993 ). Investigation o f the issues j u s t o u t l i n e d m a y aid o u r understanding o f the relationship between satisfaction a n d o u t c o m e a n d o u r u n d e r s t a n d i n g o f the n a t u r e o f client satisfaction. T h e purpose of this study was to investigate satisfaction by addressing these issues, specifically t h e relationship o f satisfaction to different o u t c o m e m e a s u r e s a n d different p r o c e d u r e s for defining outcome, a n d to do so in a s t a n d a r d practice setting. Method

203

cognitive-behavioral (n = 6), eclectic (n = 4), gestalt (n = 2 ), Adlerian (n = 1 ), and reality therapy (n = 1 ).

D e p e n d e n t M e a s u r e s a n d Procedures
At their intake appointment, clients were given a consent form, and when consent was granted, were asked to complete a set of measures of adjustment The same measures were readministered 10 weeks and 5 months after intake, at which time they also completed a satisfaction measure. Readministrations were done by research assistants over the phone or by mail (about 50% by each method), depending on the preference expressed by clients on the consent forms. Derogatis (1977) reported that narrative (oral) and written administration of the Symptom Checklist-90 (SCL-90), a longer version of the BSI measure used in this study, yields comparable results. At intake and termination of treatment, therapist measures of client adjustment were obtained. The Brief Symptom Inventory (BSI; Derogatis & Spencer, 1982) was administered to all clients at intake, at l0 weeks, and 5 months after intake. The BSI is a 53-item list of psychiatric symptoms derived from the longer SCL-90, with which it correlates highly: BSI and SCL-90 symptom dimension score correlations range from .92 to .99 (Derogatis & Spencer, 1982). Clients are asked to rate the degree to which they have experienced each item in the preceding week. Ratings range from 0 (not at all) to 4 (extremely). BSI scores were obtained by summing the distress level reported for each item. The therapist problem rating was obtained at intake and again at termination. This measure and the client problem rating were derived from the target complaints described by Mintz and Kiesler (1982). The therapist was asked to "Name the one or two problems that are most likely to be treatment targets for this client." The therapist was then asked to indicate on a 13-point continuum how much each problem bothered the client. Ratings ranged from 1 (not at all) to 13 (couldn't be worse). Ifa second major problem was indicated, this procedure was again implemented. The client problem rating was obtained at intake. The client was asked, "Name the one or two problems for which you are most seeking assistance." The client was then asked to use the same 13-point continuum used by therapists to rate how much the problem bothered them. Ifa second significant di~culty was indicated, this procedure was again implemented. Each problem was rerated at 10 weeks and 5 months after intake. The client satisfaction rating was obtained 10 weeks and 5 months after intake. The client was asked, (a) "Overall, how satisfied are you with the services you received?"; (b) "Would you recommend this agency to others seeking help?"; (c) "If you were to seek help again, would you return to this agency?"; and (d) "How would you rate your therapist?" Ratings were obtained on 5-point Likert scales. These questions encompass those recommended by Berger ( 1983 ) and Larsen et al. ( 1979 ) in their reviews of satisfaction techniques.

Clients
Consecutive nonemergency outpatient admissions were recruited for this study at three urban community mental health centers in the Midwest. Of 247 who agreed to participate, 176 (71%) provided 10-week follow-up data. Twenty-four of the participants were children (age 17 and younger). For the purposes of this study, only data obtained from adults were used. The 152 adult clients who supplied follow-up information had the following characteristics: 47% were female, 91% were White, average education was 14 years, 35% were married, average income was $15,000, and average age was 32. Using criteria from the revised third edition of the

Diagnostic and Statistical Manual of Mental Disorders ( DSM-III-R;

American Psychiatric Association, 1987) disorders were generally mild to moderate in severity. Most common were adjustment disorders (48%), dysthymia (13%), and personality disorders (8%). No other disorder or group of disorders accounted for more than 6% of the cases. Demographic and diagnostic characteristics of a random client sample who terminated in the 6 months preceding the study (n = 879) were compared with the clients who provided study information. No differences were found except that study participants had higher income ( M = $15,000, SD = 1,100 for participants; M = 12,900, SD = 337 for random sample), t(1,053) = 46.67, p < .001.

Clinical Sign ificance A n a l y s e s Clinically significant change. Determining whether a client has improved to a clinically significant (CS) degree involved a two-step process as outlined by Jacobson and Truax ( 1991 ). First, it was determined whether a client had improved to a statistically significant degree by calculating reliable change (RC) for each of the outcome measures. The formula that was used for calculating RC is as follows:
RC = x2 - xl/Sd~.

Therapists
Twenty-two therapists volunteered to participate in this study and had the following characteristics: 12 were male and l0 female; they averaged 7 years of postgraduate experience; 19 had master's degrees, and 3 had PhDs. Their theoretical orientations were family systems (n = 8 ),

( 1)

That is, pretest-posttest difference scores were divided by the standard error of difference between the two test scores. The standard error of difference between the two scores was eaicu-

204 lated directly from the standard error of measurement following formula:

PEKARIK AND WOLFF

(SE) using the

(2)

Sd~ = 2(SE) 2.

An RC of 1.96 or greater was considered to be a statistically significant change (i.e., a reliable change). The second step involved in determining CS was to determine whether a client had crossed an established cutoff point into a functional population range from a dysfunctional population range. For each of the outcome measures used in this study, the cutoff point was established using methods derived from those recommended by Jacobson and Truax ( 1991 ). For client and therapist ratings, the cutoff for the functional range was the score that fell two standard deviations below the mean intake score for each measure. Because nonpatient normative data were available for the BS1, these data were used in determining that measure's cutoff score, as recommended by Jacobson and Truax ( 1991 ). The cutoff score for the BSI was 1.5 standard deviations above the nonpatient mean. The logic and methods used to arrive at this cutoff are described in the Appendix. Both crossover into the functional population and the achievement of RC were required for a client to be considered to have improved to a clinically significant degree. The specific methods used to determine RC and CS depended on the nature of the data available and are described in detail in the Appendix. The RC and CS scores provided by these calculations are as follows: BSI RC = 33, CS cutoff = 40; client ratings RC = 5, CS cutoff = 5; therapist ratings RC = 4, CS cutoff = 6. Clinicallysignificantfailure. Clients who both failed to achieve RC and had a follow-up score higher than the cutoff score (that separated the functional and dysfunctional populations) were regarded as providing no indication of treatment benefit on a given measure. It would be unreasonable to classify such clients as treatment "failures" if they entered treatment with intake scores or ratings already in or near the functional range, however. For such clients, achievement of RC would entail follow-up scores in the superior end of the normal range--"supernormal" functioning. To account for such clients in identifying clinically significant failures (CSF), minimum intake dysfunction was calculated for each measure so that CS would not entail follow-up functioning in the superior end of the normal (functional) range. Minimum intake dysfunction scores or ratings that made clients eligible for categorization as CSF were 57, 8, and 7 for the BSI, client ratings, and therapist ratings, respectively. The method used to calculate these scores is presented in the Appendix. Clients were classified as CSF failures on a measure if they met the minimal intake dysfunction score or rating, failed to achieve RC, and failed to cross over the cutoffscore. Unclassified. Clients were categorized as unclassified and excluded from clinical significance analyses if they (a) achieved RC but did not cross over into the functional range, (b) crossed from the dysfunctional to the functional range without achieving RC, or (c) had intake scores in the functional range. Results

Larsen et al., 1979; Lebow, 1982; Nguyen, Attkisson, & Stegner, 1983), suggesting t h a t the satisfaction data were representative o f public clinic settings. Intercorrelations a m o n g i t e m s were fairly high, ranging f r o m .58 to .82. These data are presented in Table 1. T h e satisfaction scores o b t a i n e d 10 weeks after intake were f o u n d to b e significantly a n d highly correlated with satisfaction scores o b t a i n e d 5 m o n t h s after intake ( r = .79, p < .01 ), indicating t h a t 10-week satisfaction is representative o f longer t e r m satisfaction. Because data were s o m e w h a t m o r e c o m p l e t e at 10 weeks t h a n 5 m o n t h s , all s u b s e q u e n t analyses used 10week i n f o r m a t i o n .

Relationship of Satisfaction to Therapeutic Change

For each individual o u t c o m e m e a s u r e (client rating, therapist rating, BSI ), intake to follow-up rating ( p r e - p o s t ) changes were calculated for each individual participant. A significant b u t small relationship was f o u n d between intake to 10-week changes in therapist ratings a n d client satisfaction ( r = - . 2 1 , p < .05). T h a t is, as therapist-rated changes increased, client satisfaction i m p r o v e d ( a n u m e r i c a l l y low satisfaction score represented high client satisfaction). N o significant relationship was f o u n d between client rating change a n d satisfaction ( r = - . 0 7 ) or between BSI score change a n d satisfaction ( r = - . 0 6 ).

Relationship of Follow-Up Adjustment With Satisfaction

Pearson correlations calculated between satisfaction a n d 10week follow-up scores a n d ratings for each o u t c o m e m e a s u r e revealed n o significant relationships. T h e correlations between satisfaction a n d follow-up therapist ratings, client ratings, a n d BSI scores were, respectively, r = . 18,. 10, a n d .08.

Relationship of Satisfaction to Clinically Significant Improvement

J a c o b s o n a n d T r u a x ' s ( 1991 ) m e t h o d for calculating clinical significance was applied to each o u t c o m e m e a s u r e in order to identify clients w h o achieved a n d failed to achieve clinically significant change for each measure. T h e results o f the application are presented in Table 2. Point-biserial correlations were p e r f o r m e d with clinically significant success versus clinically significant failure as a dichotom o u s variable a n d degree of satisfaction as a n ordinal variable for each o u t c o m e measure. T h e y revealed a significant b u t low

Table 1

Descriptive Statistics and Satisfaction Survey Results

Correlations a m o n g the o u t c o m e m e a s u r e s were low to m o d erate. At intake, BSI scores correlated .38 a n d .20 with client a n d therapist ratings, respectively. Client a n d therapist ratings c o r r e l a t e d . 19 with each other. These correlations indicate t h a t these measures are relatively i n d e p e n d e n t o f one another. The overall satisfaction ratings were high ( M = 1.88 per item, SD = .92) a n d similar to those r e p o r t e d b y other investigators (Berger, 1983; Essex, Fox, & G r o o m , 1981; Fiester, 1979;

Intercorrelations Among Items on the Client Satisfaction Rating Obtained 10 Weeks After Intake
Item I. 2. 3. 4. Item l Item2 Item 3 Item 4 l .71" .67* .58* 2 -.82* .58* 3 4

-.60*

--

*p<.01.

SATISFACTION AND OUTCOME Table 2

205

Clients Identified as Clinically Significant Successes and Failures by Therapist Rating, Client Rating, and BSI
Measure Therapist rating n % Posttest score M CSS 43 29 4.05 1.29 6.23 1.54 55 35 3.09 1.38 7.98 2.1 34 21 24.5 11.4 63.1 25.7 CSF 86 59 9.52 1.60 0.70 1.61 71 46 9.42 2.18 1.70 1.83 46 28 92.5 32.2 1.39 22.4 80 51 162 100 30 19 156 100 Unclassified 17 12 Total 146 100

SD
Pre-post difference M

SD
Client rating n % Posttest score M

SD
Pre-post difference M

SD
BSI n % Posttest score M

SD
Pre-post difference M

SD

Note. BSI = Brief Symptom Inventory; CSS = clinically significant

success;CSF = clinicallysignificantfailure; Total = clinicallysignificant successes + clinically significantfailures + unclassified.

correlation between degree of satisfaction and clinical significance status for therapist ratings (r = -.30, p < .05) but not for client ratings ( r = - . 0 9 ) or for BSI scores ( r = -.01 ). The same pattern was found in t-test results: Clients who had achieved clinically significant change on the therapist rating had better satisfaction ( M = 5.88, SD = 2.25) than did clinical failures ( M = 8.21, SD = 4.06), t(144) = -3.52, p < .05. Satisfaction did not differ for those who achieved clinically significant change and failure based on the client ratings, t(154) = -.98, and BSI, t(160) = -.03. Originally we planned to assess the relationship of satisfaction to the outcome measures by using a two-step process: First, univariate statistics would identify outcome measures related to satisfaction, then those measures found significant would be used in multiple regression within each of the ways of calculating outcome used in this section. Because only a single outcome measure, therapist rating, was found to be related to satisfaction using the more liberal univariate tests (and that relationship was very small), the multiple regression tests were not performed. Discussion The results of this study suggest that there was no relationship between client measures (BSI and client problem ratings) of

therapeutic change and client satisfaction and that there was a significant but very modest correlation between therapist ratings and satisfaction. The findings of this research, which was conducted using clients from standard practice settings, are generally consistent with previous studies, which generally find low correlations between satisfaction and other outcome measures (Attkisson & Zwick, 1982; Carscaddon et al., 1990; Fiester, 1979; Larsen et al., 1979; WiUer & Miller, 1978). However, the current findings are toward the low end of the range typically reported. This may well be due to some methodological differences between this study and the others, specifically other studies' use of global measures of adjustment, very brief follow-up periods, exclusion of dropouts, or reliance on therapist measures. Virtually all the previous research on the relationship of outcome to satisfaction suffer from one or more of these limitations. All of these might logically contribute to higher satisfaction outcome correlations than found in the current study. The reliance on therapist ratings by other investigations is especially important given this study's finding that only therapist ratings were significantly correlated with satisfaction. This study did not support reports and speculations by Lambert et al. (1986) that satisfaction is highly related to end-point (posttreatment) outcome scores. Jacobson and Truax's ( 1991 ) methods of determining clinical significance to assess the relationship between client satisfaction and outcome provided stricter criteria by which clients were classified as therapeutic successes or failures. The extreme differences between clinically significant successes and clinically significant failures can be seen in the data presented in Table 2. Ironically, however, the relationship of client ratings and BSI scores to satisfaction was low and nonsignificant even with the extreme groups identified with Jacobson and Truax's method. Only the relationship of therapist ratings to satisfaction was significant. Such findings are counterintuitive in that one would expect to find a greater relationship between client measures of problem change and their satisfaction with the therapeutic experience. The results of this study appear to contradict a recent study that reported a stronger relationship between satisfaction and outcome when outcome was "clinically significant" rather than indicative of"moderate symptom change" (Ankuta & Abeles, 1993). That study, however, did not use traditional satisfaction measures to assess satisfaction, but, rather, the sum of three items that assessed treatment benefit, amount of problem change, and current functioning (all of which assess adjustment rather than satisfaction) and a single item that assessed satisfaction. Such a "satisfaction" measure is clearly primarily an adjustment measure--it is understandable that it was found related to another adjustment measure. The fact that this study used stricter criteria (clinical significance) to define and separate those who were therapeutic successes from those who had failed and still found little in the way of significant relationships between outcome and satisfaction provides stronger evidence than previously available of the negligible relationship between satisfaction and outcome. The clinical-significance definitions dearly produced extreme groups of successes and failures, yet even these were not related to satisfac-

206

PEKARIK AND WOLFF Attkisson, C. C., & Zwick, R. (1982). The Client SatisfactionQuestionnaire: Psychometric properties and correlations with service utilization and psychotherapyoutcome. Evaluation andProgram Planning, 5, 223-237. Berger, M. ( 1983). Towardmaximizing the utility of consumer satisfaction as an outcome measure. In M. J. Lambert, E. R. Christensen, & S. S. Dejulio (Eds.), The assessment of psychotherapy outcome (pp. 56-80). New York: Wiley. Carscaddon, D. M., George, M., & Wells,G. (1990). Rural community mental health consumer satisfaction and psychiatric symptoms. Community Mental Health Journal 26, 309-318. Derogatis, L. ( 1977). SCL-90: Administration, scoring, and procedures manual-l for the R(evised) version. Baltimore, MD: Johns Hopkins UniversitySchool of Medicine. Derogatis, L., & Spencer, P. (1982). The brief symptom inventory: Administration, scoring and procedures manual-l. Baltimore, MD: Clinical Psychometric Research. Edwards, D., Yarvis, R., Mueller,D., & Langsley,D. (1978). Does patient satisfaction correlate with success? Hospital and Community Psychiatry, 29, 188-190. Essex, D. W., Fox, J. A., & Groom, J. M. (1981). The development, factor analysis, and revision of a client satisfaction form. Community Mental Health Journal, 17. 226-235. Fiester, A. R. ( 1979). Goal attainment and satisfaction for CMHC clients. American Journal of Community Psychology, 7, 181-188. Garfield, S. L., Prager, R. A., & Berg, in, A. E. (1971). Evaluation of outcome in psychotherapy. Journal of Consulting and Clinical Psychology, 37, 307-313. Greenfield, T. K., & Attkisson, D. D. (1989). Steps toward a multifactorial satisfaction scale for primary care and mental health services. Evaluation and Program Planning, 12, 271-278. Jacobson, N. S., & Truax, P. ( 1991 ). Clinical significance:A statistical approach to defining meaningful change in psychotherapyresearch. Journal of Consulting and Clinical Psychology, 59, 12-19. Lambert, M. J., & Hill, C. E. (1994). Assessing psychotherapy outcomes and processes. In S. L. Garfield & A. E. Bergin (Eds.), Handbook of psychotherapy and behavior change ( 4th ed., pp. 72-113 ). New York: Wiley. Lambert, M. J., Shapiro, D. A., & Bergin, A. E. (1986). The effectiveness of psychotherapy. In S. L. Garfield & A. E. Bergin(Eds.), Handbook of psychotherapy and behavior change ( 3rd ed., pp. 157-212). New York: Wiley. Larsen, D. L., Attkisson, C. C., Hargreaves, W. A., & Nguyen, T. D. (1979). Assessmentof client/patient satisfaction: Development of a general scale. Evaluation and Program Planning, 5, 233-237. Lebow, J. (1982). Consumer satisfaction with mental health treatment. Psychological Bulletin, 91, 244-259. Linden, W., & Wen, E K. (1990). Therapy outcome research, health care policy, and the continuing lack of accumulated knowledge. Professional Psychology: Research and Practice, 21,482-488. Mental Health SystemsAct of 1980, Pub. L. No. 96-398. Mintz, J., & Kiesler, D. J. ( 1982). Individualized measures of psychotherapy outcome. In P. C. Kendall & J. N. Butcher (Eds.), Handbook of research methods in clinical psychology (pp. 491-534). New York: Wiley. Nguyen, T, D., Attkisson, C. C., & Stegner, B. L. ( 1983). Assessmentof patient satisfaction:Developmentand refinement of a service evaluation questionnaire. Evaluation and Program Planning, 6, 299-314. Pekarik, G. (1992). Relationship of clients' reasons for dropping out of treatment to outcome and satisfaction. Journal of Clinical Psychology, 48, 91-98.

tion. This strongly suggests that satisfaction is not meaningfully related to traditional client measures of outcome. The high satisfaction scores may indicate that client satisfaction occurs without client regard to degree of symptom change. Clients may be able to discern a difference between satisfaction and success, as suggested by Larsen et al. (1979). It is conceivable that clients have rated their degree of satisfaction on the basis of something other than their satisfaction with the symptom or presenting problem change. General likability of the therapist, perceived earnest efforts of the therapist or agency, or simply the availability of services are but a few reasons why clients may rate their therapy experience as satisfactory independent of degree of problem change (Sloane, Staples, Cristol, Yorkston, & Whipple, 1975 ). The high satisfaction scores, however, may be due to low validity of the instrument and client response: Clients may be unwilling or unable to identify dissatisfaction. This prospect is contradicted by extensive research on the satisfaction questions used in this study (Nguyen et al., 1983). As in other satisfaction research, ratings were skewed toward high satisfaction. At present, 15 years after the development of the Client Satisfaction Questionnaire (CSQ) by Larsen et al. ( 1979; which served as the source of items used in the present study), the CSQ continues as the most widely used and highly regarded satisfaction measure reported in the literature. It may be that clients do tend to be highly satisfied with mental health services, as suggested by Nguyen et al. (1983), but other measures could and should be developed to attempt to further differentiate degrees of satisfaction. Future research could address this by generating satisfaction items that facilitate the reporting of dissatisfaction. One way to do this would be to use items that assess a wider range of clinical phenomena. For example, rather than simply ask about satisfaction with the therapist, items could assess specific aspects of therapist behavior (e.g., therapist advice on how to cope with problems outside the session). Although not the primary goal of this study, it does provide interesting data on the proportion of clients who achieved clinically significant improvement at community clinics. Although there were more failures than successes with all measures, it should be noted that the study did not restrict the sample to treatment completers, that is, it included those who terminated prematurely. This research has confirmed the findings of much previous research: Correlations between client satisfaction and therapeutic outcome measures were low. Satisfaction surveys should continue, therefore, to be considered a distinct contribution to overall assessment of quality of services provided but not a substitute for traditional measures of outcome.

References
American Psychiatric Association. ( 1987). Diagnostic and statistical manual of mental disorders ( 3rd ed., rev.). Washington,DC: Author. Ankuta, G. Y., & Abeles, N. (1993). Client satisfaction, clinical significance,and meaningfulchange in psychotherapy.Professional Psychology."Research and Practice, 24, 70-74.

SATISFACTION AND OUTCOME Public Health Service Act ammendments of 1975, Pub. L. No. 94-63. Sleek, S. ( 1994, February). Psychologists, lawyers facing similar changes: Rise of corporate entities brings changes. TheAPA Monitor, pp. !, 28. Sloane, R. B., Staples, E E, Cristol, A. H., Yorston, N. J., & Whipple, K. ( 1975 ). Short-term analytically oriented psychotherapy versus behavior therapy. American Journal of Psychiatry, 132, 373-377. Stiles, W. B., Shapiro, D. A., & Elliott, R. K, (1986). "Are all psychotherapies equivalent?" American Psychologist, 41, 165-180.

207

Weisz, J. R., & Weiss, B. (1989). Assessing the effects of clinic-based psychotherapy with children and adolescents. Journal of Consulting and Clinical Psychology, 57, 741-746. Wilier, R. D., & Miller, G. H. (1978). On the relationship of client satisfaction to client characteristics and outcome of treatment. Journal of Clinical Psychology, 34, 157-160. Winegar, N. (1992). The clinician's guide to managed mental health care. Binghamton, NY: Haworth Press.

Appendix Determination of Criterion Scores for Reliable Change (RC) and Clinical Significance (CS) BSI All the data needed to calculate RC using the formulas shown in the Method section were available for the BSI, so RC was calculated as described in that section. Jacobson and Truax ( 1991 ) identified two methods for finding the cutoff between functional and dysfunctional populations when data from both populations are available. The first method involves use of the formula devised. Other researchers (Weisz & Weiss, 1989) have claimed that early dropouts receive virtually no treatment and therefore can be considered as a substitute for a no-treatment control group in outcome research. Using similar logic, intake and 10-week follow-up ratings for clients who attended one or two sessions could serve as an estimate of test-retest reliability. Using the formula for RC as described earlier, we determined that a pre-post test change of 5 points or more would be required for a client to be considered reliably changed on this measure. The cutoff point separating functional from dysfunctional populations was determined by calculating a point two standard deviations below the mean of the dysfunctional population (i.e., more functional) as outlined by Jacobson and Truax (1991). This resulted in a cutoff point of 5, meaning that a client was required to have had a pretreatment rating of 6 or higher (i.e., in the dysfunctional range) and a posttreatment rating of 5 or lower to have crossed over from the dysfunctional population range into the functional population range on this measure. This change pattern, along with an RC of 5 or greater, was the criterion for achieving clinical significance. T h e r a p i s t Ratings As with client ratings, a traditional measure of test-retest reliability is virtually impossible to secure for therapist ratings. An estimate of this was available, however: Therapist ratings at intake and termination for single-session clients were both based on the therapist's assessment of the client at the intake interview. The therapist rated the same behavior on two occasions (intake and when asked to recall intake 10 weeks later at the first follow-up). This appears to be a reasonable estimate of testretest reliability and was so used in the RC formula for therapist ratings. Using the formula for RC then, it was determined that a pre-post test change of 4 or more points would be required for a client to be considered reliably changed on this measure. The cutoff point separating functional from dysfunctional populations was determined in the same manner as used for client ratings. This resulted in a cutoff point of 6, meaning that a client was required to have had a pretreatment (intake) score of 7 or higher (i.e., dysfunctional) and a posttreatment ( 10-week follow-up) score of 6 or lower to have crossed over from the dysfunctional population range into the functional population range. This change pattern, along with an RC of 4 points or greater, was the criterion for achieving clinical significance for this measure. Failure The first criterion for categorization as a "failure" was the failure to achieve RC. Some clients began treatment with an intake score or rating

c = (soMa + slMo)/So + sl,

where c = the cutoffscore, M0 = the mean of the functional population, M~ = the mean of the dysfunctional population, So = the standard deviation of the functional population, and s~ = the standard deviation of the dysfunctional population. Nonpatient data supplied by Derogatis and Spencer (1982) and patient data from this study provide the following: Mo = 16, So = 16, M, = 70.4, s~ = 39.7. Using this data in the formula provided a cutoff score (c) of 32. Their second method uses the score that is two standard deviations above the nonpatient mean as the cutoff, which is 48. The cutoff scores provided by these two methods are quite far apart (32 vs. 4 8 ) - - a full standard deviation (of the nonpatient group). Jacobson and Truax ( 1991 ) advised use of the first method when the normal and dysfunctional populations overlap and the second method when they do not. In this study, the two populations overlap, but barely, making neither method ideal: The first method is too stringent, requiring that clients be within one standard deviation of the nonpatient mean (rather than the 2 standard deviations Jacobson and Truax prescribed as the cutoffwhen nonpatient data are relied on); the second method is too lenient because it identifies as functional a score that is only approximately one-half a standard deviation (of the patient population intake score) below the mean intake score of the patient group. The solution to this problem was to use as cutoffa score of 40, which is the midpoint between the cutoffscores provided by the two methods. The use of this cutoffis supported by a number of statistical justifications. This score is one-half standard deviation (of the nonpatient population ) higher (i.e., more lenient) than the overly strict cutoff provided by the first method and one-half standard deviation (of the nonpatient population) lower (i.e., more strict) than the too-lenient cutoff provided by the second method. It is quite conservative, requiring a follow-up score within one and a half standard deviations of the nonpatient population, which is.5 standard deviation lower (better adjusted) than prescribed by Jacobson and Truax. Client Ratings Because test-retest reliability, required by the RC formula, is virtually impossible to secure for client ratings, some estimate of it had to be

208

PEKARIK

AND WOLFF w o u l d n o t b e classified as failures. S i m i l a r p r o c e d u r e s w e r e u s e d f o r t h e r a p i s t r a t i n g s , a n d BSI scores, r e s u l t i n g in m i n i m a l i n t a k e d y s f u n c t i o n s c o r e o f 8, 7, a n d 57 f o r client r a t i n g s , t h e r a p i s t r a t i n g s , a n d BSI scores, respectively. C l i e n t s w i t h o u t these m i n i m u m scores a t i n t a k e were excluded from the failure category. C l i e n t s w e r e classified as f a i l u r e s f o r e a c h m e a s u r e i f t h e y failed t o a c h i e v e R C , h a d t h e identified m i n i m u m i n t a k e score, a n d failed t o c r o s s over t h e c u t o f f p o i n t s e p a r a t i n g f u n c t i o n a l a n d d y s f u n c t i o n a l s c o r e a n d r a t i n g ranges.

in o r n e a r the f u n c t i o n a l r a n g e ; however, it w o u l d be i n a p p r o p r i a t e t o c o n s i d e r s u c h cases failures. To a d d r e s s this, m i n i m u m i n t a k e s c o r e s were established for each measure on the basis of the number of change u n i t s r e q u i r e d t o a c h i e v e R C a n d t h e scores f o r d e f i n i n g t h e d y s f u n c t i o n a l a n d f u n c t i o n a l ranges. T h e g o a l w a s to e x c l u d e as f a i l u r e s clients w h o b e g a n t r e a t m e n t in t h e n o r m a l r a n g e o r so close to the n o r m a l range that achievement of RC would entail a "supernormal" follow-up. F o r client ratings, t h e m i d p o i n t o f the n o r m a l r a n g e w a s s u b t r a c t e d f r o m the p r e v i o u s l y e s t a b l i s h e d c r o s s o v e r p o i n t . T h e e s t a b l i s h e d m i n i m u m c h a n g e r e q u i r e d f o r R C w a s t h e n a d d e d to t h e r e s u l t t o a r r i v e at a m i n i m u m d y s f u n c t i o n s c o r e r e q u i r e d b y a c l i e n t in o r d e r to b e c o n s i d ered a p o t e n t i a l failure. T h i s p r o c e d u r e e n s u r e d t h a t clients w h o w o u l d have to achieve a b e t t e r - t h a n - n o r m a l s c o r e at f o l l o w - u p (i.e., b e t t e r t h a n the m i d p o i n t o f t h e n o r m a l p o p u l a t i o n r a n g e ) in o r d e r t o a c h i e v e R C

Received November Accepted

8, 1 9 9 4

R e v i s i o n r e c e i v e d J u n e 6, 1 9 9 5 October 20, 1995

New Editor Sought for New

Contemporary Psychology

The Publications and Communications (P&C) Board has opened nominations for the editorship of ContemporaryPsychologyfor the years 1999-2004. John H. Harvey, PhD, is the incumbent editor. Beginning with the new editorship, the P&C Board intends to increase the timeliness of reviews and to implement a new, more selective coverage policy, indicated in the following statement:

Contemporary Psychologyis a journal of review and evaluation. Although it is sustained by a varied intellectual diet, its main staple is ideas as they are reflected in books. Not all books nourish the review process, but books that provoke thought about the broad arena of psychology, monographs that integrate new empirical work, works that enhance clinical practice, treatises that are likely to influence public thinking, and textbooks that will be used in colleges and universities are grist for the evaluative process. ContemporaryPsychologyendeavors to provide a meaty fare of sophisticated, even opinionated and controversial reviews that emphasize evaluation rather than mere pr6cis and summary. (As protection against occasional gross bnas, there may be competing reviews and occasional responsa.) Often a book will serve as a springboard for weighing competing positions. Just as often, it will plumb the full depth and implications of an idea. In each assessment, the reviewer will be encouraged to bring the full panoply of scholarship into play.
Beyond reviewing individual books and groups of related books, Contemporary Psychologywill provide occasional essays on the "state of the media" in psychology, including comment on new information technologies (Intemet and the World Wide Web, for example) and new production modes for books and journals (electronic journals, for example). In sum, ContemporaryPsychology endeavors to take psychological ideas seriously and to provide a forum in which they can be examined with spice, wit, and intelligence. Candidates should be members of APA and should be available to work with the P&C Board on developing new procedures in 1997. The successful candidate will start receiving books on January 1, 1998. Please note that the P&C Board encourages participation by members of underrepresented groups in the publication process and would particularly welcome such nominees. To nominate candidates, prepare a statement of one page or less in support of each candidate and send to: Donna M. Gelfand, PhD, Search Committee Chair c/o Lee Cron, P&C Board Search Liaison Room 2004 American Psychological Association 750 First Street, NE Washington, DC 20002-4242 Members of the search committee are Norman Abeles, PhD; Joe L. Martinez, Jr., PhD; Richard M. Suinn, PhD; and Judith P. Worell, PhD. First review of nominations will begin June 1, 1996.