DESIGNING, MANAGING, AND ANALYZING MULTISITE EVALUATION

DEBRA J.ROG

SUMMARIZED BY

MUNAWIR RAZAK
NIM : 1106038513

MULTISITE EVALUATIONS (MSEs)

WHAT...? MSEs involve examining a program or policy in two or more sites.

a. The intervention is intended to be exactly the same across the different sites b. The programs studied are of the same type but may vary from one another

EXAMPLES OF MULTISITE EVALUATIONS (MSEs)

MULTISITE EVALUATIONS (MSEs)

Example 1
Evaluation of 4 community coalitions aimes at preventing family violence

a. Each of the Sites was located in a different county of one Midwestern State b. The sites are funded by local foundation, received twoyear planing grantsm followed by 5 years of implementation funding c. The initiative had a cross-site technical assistance provider as well as a cross-site evaluation team. d. Evaluation was modest in funding and scope e. Relied primarily on the collection data through interviews, f. Led by a cross-site evaluators, with local evaluators involved only in the planning stage. g. The design involved longitudinal case studies, with an emphasis on using the same lens in each site h. The focus was initially on the implementation of the coalitions, followed by an assessment to their ability to achieve outcomes.

MULTISITE EVALUATIONS (MSEs)

Example 2
Evaluation of 4 providers of supportive housing for homeles family

a. This SME is relatively small and in progress in Washington state, it may ultimately involve up to ten providers b. Most of them serving small numbers of families (thirty or fewer) c. The Study involves both qualitative and quantitive data on the implementation and fidelity of the housing programs, and on the experiences and outcomes of families d. The evaluation funding is limited, data are being collected largely through telephone interviews with the providers, and family data are being collected by providers staff. e. This MSE provides a good illustration on MSE data collection, training, and quality control on a small budget

MULTISITE EVALUATIONS (MSEs)

Advantages

MSEs offer opportunities to accelerate the development of knowledge in a program area

a. Offer the power of results that can rise above any single investigation b. We are likely to learn more about the generalizability of an intervention and its effect c. Opportunity to understand the trade-offs and relative advantages and disadvantages of the different types of program strategies d. Can create an invisible college

MULTISITE EVALUATIONS (MSEs)

Disadvantages
a. Take more funding b. Take more time to design, implement and analyze c. Error that occur in MSE are likely to be more serious and long lasting

MULTISITE EVALUATIONS (MSEs)

Multisite Approaches and Designs

a. Laying the foundation for an MSE b. Determining the MSE Design c. Sampling Sites

MULTISITE EVALUATIONS (MSEs)

Multisite Approaches and Designs

a. Laying the foundation for an MSE
- Determining who id the lead for the evaluation and identifying the mechanism by which cross-site decisions are made - Determining the nature of the relationship between the evaluation team and the evaluation sponsor - Communication is the most distinct for MSEs compared to single-site evaluations - The More decentralized the decision making in in the MSE, the more important it is that decisions and other information be communicated regularly and frequently. - Communication should include a mix of in-person, telephone and electronic communications.

MULTISITE EVALUATIONS (MSEs)

Multisite Approaches and Designs

b. Determining the MSE Design
- The Only distinction that MSEs share is that evaluation examines an intervention in two or more sites. - Factors that shape the MSE design nature of evaluation questions; readiness of the field; the nature of problem; the reason of evaluation; nature, diversity, and number of sites; and resources. - A range of design often required to meet the variety of evaluation questions and purposes for the information - The larger the number of sites, the more important are standards for data collection

MULTISITE EVALUATIONS (MSEs)

Multisite Approaches and Designs

c. Sampling Sites
- Some MSE design may examine representative sites from clusters of sites sharing similiar charactheristic. - Sampling may be limited to sites that meet a treshold level of fidelity of implementation

MULTISITE EVALUATIONS (MSEs)

Strategies for Multisite Data Collection

a. Collecting Common Versus Specific Site Data b. Developing a Common Protocol c. Maximizing Existing Data

d. Developing a Common Data Collection Tool

MULTISITE EVALUATIONS (MSEs)

Quality Control in MSEs

a. Selecting and Hiring Data Collectors b. Common Training and Booster Sessions c. Readiness of Interviewers

d. Communications, Supervision and Ongoing Review

MULTISITE EVALUATIONS (MSEs)

Tips for the MSE evaluator

a. MSEs require interpersonal acumen b. Should control and accomodate ego, attending to equity, and being sensitive c. Maintaing communication, keep everyone informed, deliver on promise made

MULTISITE EVALUATIONS (MSEs)

Data Management
a. Computerizing and Managing Qualitative Data b. Computerizing and Managing Quantative Data c. IDs and Confidentiality

MULTISITE EVALUATIONS (MSEs)

Assessing Multisite Interventions

a. Monitoring Fidelity b. Assessing Common Ingredients c. Studying Implementation d. Measuring Program Participation e. Assessing Comparison as well as Treatment Sites

MULTISITE EVALUATIONS (MSEs)

Monitoring Multisite Implementation

a. Design Features to Monitor b. Monitoring Methods

Thank YOU