Welcome!

TAY Performance Improvement

Project (PIP) Toolkits

TEQI Summit Day 2

October 16, 2015

Introductions/Acknowledgements

Todd Gilmer, Ph.D.

Sarah Hiller, MPIA
Andrew Sarkin, Ph.D.
Steven Tally, Ph.D.

This presentation was funded by the Mental

Health Services Oversight and Accountability
Commission (Contract #13MHSOAC014)
2

Presentation Objectives
Overall: Present the proposed TAY PIP Toolkit concepts and gather
feedback
Introduction: What is a PIP, and why have we chosen to develop PIP
Toolkits?
Describe the proposed TAY PIP Toolkits: content, format, and
resources
TAY PIP Toolkit Topics and rationale
Clinical PIP- Implementing a TAY-specific outcome measure across TAY
programs
Program PIP- Conducting a consumer-oriented quality improvement
project with peer support programs for TAY

Keep in mind for discussion

Tailoring the PIP toolkits to TAY- what makes
something TAY-specific?
Potential challenges to using the PIP toolkits- for
example, the needs of smaller counties
What other resources would be useful for
inclusion in these PIP toolkits?
How should we approach the
delivery/dissemination of the PIP toolkits?
4

Background: UCSD California TAY

Evaluation Project (2014-2016)

Aims:
1.
2.
3.

Describe the current state of MHSA-funded TAY programs and services in California
Identify shortcomings or gaps in existing evaluation/QI efforts in TAY programs and services
and to develop recommendations for improvements in these efforts
Develop Technical Assistance Resources

Evaluation Advisory Group: county administrators, TAY providers, TAY with lived
experience, family members of TAY, and other TAY stakeholders

Mixed-methods study design: drawing on multiple sources of data:

Phone interviews with 39 county representatives
Online survey completed by 180 TAY service providers
Literature review of TAY evidence base

Study findings cited in this presentation will be marked with this stamp

What is a PIP?
Performance Improvement Project
Part of annual quality improvement activities since 2003
Regulations from Centers for Medicare and Medicaid Services

Each county must conduct two PIPs per year:

Clinical PIPs target the assessment and improvement of clinical services or
environments. Examples include analysis of hospitalization patterns or highrisk therapeutic procedures.
Nonclinical (or Program) PIPs can cover such areas as operational efficiencies
or administrative processes.

PIPs are audited by an External Quality Review Organization (Behavioral

Health Concepts, Inc.)
www.caleqro.com
www.bhcinfo.com
6

What does a PIP involve?

Documented process which follows these
steps:
1. Review and select a study topic(s)
2. Select and define the study or research question(s)
3. Select a representative and generalizable study population
4. Select the study outcomes and variable(s)
5. Review sampling techniques (if a sample is being used)
6. Review and conduct reliable data collection techniques
7. Analyze data and interpret study results
8. Assess and implement intervention/improvement strategies
9. Assess for real improvement
10.Assess factors for sustained improvement
7

Why create TAY PIP Toolkits?

Counties are required to conduct two PIPs per year:
Smaller counties may struggle to conceptualize and plan PIPs
Increase county/provider quality improvement capacity to conduct more
complex/meaningful PIPs

County evaluation and quality improvement activities are

rarely TAY-specific
Only 44% of interviewed County respondents described TAY-specific
quality improvement activities

Toolkits can address a variety of technical assistance needs

8

Why create TAY PIP Toolkits?

(continued)
Preferred Types of TAY-Specific Technical Assistance as reported by 180
California Providers, 2014
Type of Needed Technical Assistance

% of
Providers

Help tracking client outcomes after leaving your program

58%

Help identifying effective programs for TAY

47%

Help coordinating or networking with other programs in your county

46%

Help recruiting or engaging specific populations (e.g. LGBTQ youth)

44%

Help identifying measures and/or survey tools

38%

Help involving TAY in program development and/or evaluation

34%

Help designing and/or conducting evaluations of specific programs

34%

Help with quality improvement

27%

Rationale- Clinical PIP Toolkit: Measuring

outcomes across TAY programs
46% of interviewed county administrators mentioned
wanting a TAY-specific standardized measure to
compare programs.
Specific instruments used with TAY for
reporting/evaluation/quality improvement
~20 different instruments mentioned by administrators and providers
Most popular:
Child and Adolescent Needs and Strengths measure (CANS)
Level of Care/Child and Adolescent Level of Care (LOCUS/CALOCUS)
Neither are TAY specific

10

Rationale- Clinical PIP Toolkit

(continued)
Six outcome domains mentioned by study
Outcome domains currently measured for TAY programs as mentioned by
participants:
California
Counties and Providers
Outcome Domain

Counties
(n=39)

Providers
(n=180)

Access/use/outcome of mental health services (e.g.,

symptoms, intensity of care, hospitalizations)

22%

14%

Education/employment

13%

15%

Housing (e.g., homelessness)

11%

3%

Psychosocial outcomes (e.g., self-esteem, functioning,

quality of life)

11%

9%

Criminal justice (e.g., incarcerations)

6%

2%

TAY consumer goal achievement

3%

24%

11

Rationale- Program PIP Toolkit: TAY

Peer Support Programs
Nationwide push to incorporate peer support into behavioral health
services
Mental Health Services Act in 2003; Presidents New Freedom
Commission Report in 2003
Part of larger recovery and consumer orientation movement

Peer support programs are common; 54% of providers had peer-led

programs for TAY.
Support groups (44%), education (23%), counseling (19%)

However, little has been done to measure peer-led program

outcomes
Literature review- little to no evidence for effectiveness, especially for
TAY
Strong quality improvement activities are necessary to ensure local program
effectiveness
12

Content, Format, Delivery/Dissemination

WHAT WILL THE TAY PIP

TOOLKITS INCLUDE?
13

Content
Each toolkit will have a pre-determined (but
adaptable) study objective and
implementation/analysis plan.
Step-by-step guide:
Each toolkit will include a detailed 10-step CMS Procedure guide
Will include troubleshooting advice to overcome anticipated challenges and meet
EQRO requirements
Emphasis on:
Consumer-oriented study objective/design
Interpretation of findings and feasible/actionable recommendations/conclusions

Supplementary tools to facilitate the PIP

implementation:
Implementation Checklist and Feasibility Analysis
Annotated EQRO PIP Forms: Validation, Submission, and Scoring Tools
14

Implementation Checklist and

Feasibility Analysis
Checklist with graduated responses
Example: How many staff members do you have with intermediate to
advanced statistical analysis skills:
0, 1-2, 3-4, 4+

Scores overall and for each section, adapted for each toolkit:
Information about local TAY Programs, TAY consumers, and providers
serving TAY
Existing data
Data collection
Data entry, management, and IT support
Data analysis and reporting
Quality improvement experience/capacity

Use checklist scores to check feasibility of project

Toolkit will include suggestions for alternative strategies/solutions
15

Implementation Checklist Example

LOCAL TAY PROGRAMS AND CONSUMERS
Questions
1.

How many programs that include TAY does your county

directly run? These can be programs exclusively for TAY, or
those that include TAY with adult or child populations.
a. How many of these county-run programs are
exclusively for TAY?

Circle your answer

0

1-2

3-4

5+

1-2

3-4

5+

1.

How many contracted providers do you have who work with

TAY?

5+

3-4

1-2

1.

How many programs that include TAY do your contracted

providers run? These can be programs exclusively for TAY,
or those that include TAY with adult or child populations.
a. How many of these provider-run programs are
exclusively for TAY?

1-2

3-4

5+

1-2

3-4

5+

Write in your answer

Actual number of county-run
programs that include TAY:
_______
Actual number of county-run
programs exclusively for TAY:
_______
Actual number of contracted
providers who work with
TAY:________
Actual number of provider-run
programs: _______

Actual number of county-run

programs exclusively for TAY:
_______
100+ Actual number of TAY served:
______

In total, how many TAY consumers do your county and your 0-30 30-50 50-100
contracted providers work with in TAY-exclusive programs
or programs where TAY are included in child/adult
programs?
4+
3
2
1
1. In how many languages (including English) do you typically
List languages:
have to provide TAY programs/materials?
Responses
Write in number of circled responses in each column
Scoring
x0
x1
x2
x3
Totals
Sub-score Total: _____/21
=____ =____ =____ =____ Add the numbers in the four cells
Multiply the two numbers above each of the following cells
to the left together, and write the
together and write the product
16 above.
sum on the line
+
+
+
+
1.

Feasibility Analysis Example

Checklist: Local TAY Programs and Consumers
% SCORE
RANGE

75%-100%

50%-75%

INTERPRETATION ISSUES AND ALTERNATE STRATEGIES

Proceed; alternate
strategies are
optional
Proceed with
caution, and
incorporate some
alternate strategies

Not enough TAY or TAY-specific programs to include?

If you have fewer than 50 TAY consumers in your programs, consider
teaming up with another nearby county to increase your sample size,
and conduct the PIP jointly. Remember to include results and
recommendations specific to your own county in your report.
Few or no TAY-exclusive programs?
If you only have TAY that are involved in programs with adult and child
populations, and no programs just for TAY, consider collecting data
from just these TAY, or TAY and adults for comparative purposes.
Ensure that in your data collection, you can distinguish which
participants are TAY and which are adults by age for analysis
purposes, and include this comparison in your report.

Proceed after
Too many languages in use for TAY programming?
25%-50% incorporating several Translating data collection instruments can take up valuable time and
alternate strategies staff effort, so consider limiting your PIP to one or two language
groups (e.g., English and Spanish). Acknowledge this limitation in
your reporting, and consider future projects with other language
groups in the coming years.

0%-25%

Use alternate
strategies or do not
proceed

Difficulty coordinating with many providers? Consider limiting your

PIP to a few providers who serve the most TAY. These larger providers may
also have greater capacity and more staff to help with data 17
collection and/or
analysis.

EQRO PIP Forms

Completed template forms with text that can be
adapted
Submission
Validation
Scoring

Options for background information, study

design, and analytic plan.
Based on CA EQRO forms (Behavioral Health
Concepts, Inc.)
18

Additional Supplementary Resources

Summary of PIP Topic and Proposed Plan for stakeholders
PIPs require provider/consumer/other stakeholder support and input

Data Collector/Data Entry Training Resources

Links/files for video tutorials/powerpoint presentations

Low-tech Double Data Entry MS Excel Workbook Template

Spreadsheet 1: Instructions
Spreadsheets 2 & 3: data entry
Spreadsheet 4: Automatically highlights cells that show discrepancies between
spreadsheets 2 & 3
Refer to original paper surveys to correct entries and produce final dataset

Using Excel for Hypothesis Testing resources

Links/files for video tutorials/powerpoint presentations

19

PIP Format
Primary booklet & supplementary
documents/forms
Easy to read, user friendly PDF with linked table of contents
Avoid academic jargon

Adaptable: Options to adjust PIP for different

practical/resource constraints & variations across
counties/TAY programs
Examples:
Alternate data analysis plans to account for statistical analysis capacities
across counties.
Accommodating variations in TAY programs using adaptable measures to
assess different outcomes.

20

Delivery/Dissemination
Booklet PDF
Training webinars
2-3 sessions
Recorded and posted to a website

Outreach to relevant county staff via email

Focus on smaller counties

Additional Resources/Support for using the PIP Toolkits

FAQ sheets from webinars/meetings/consultations
Support from UCSD Faculty
Additional external PIP-related resources
For example, PIP Technical Assistance Manual by Wieman et al., 2004

Toolkit feedback form

21

Next Steps
Develop a TAY-specific Clinical PIP and Program
PIP, approved by the EQRO
Make PIP Toolkits available on the TEQI
Summit website
Host a TAY PIP webinar (also downloadable)

22

DISCUSSION

23