Evaluation & Assessment Final Project

Summer Enrichment at Loyola (SEaL)
Social Skills Development Evaluation

Loyola University Chicago
December 2014
Antoaneta Topalova and Laura Roman
SEaL EVALUATION PLAN
SEaL Evaluation Plan

Loyola University Chicago
ELPS 431
TABLE OF CONTENTS
EVALUATION PLAN INTRODUCTION .................................................................................. 5
SEAL PROGRAM OVERVIEW ..................................................................................................... 5
SIGNIFICANCE OF PROBLEM ............................................................................................................. 6
CONTEXT AND HISTORY OF THE PROGRAM .............................................................................. 6
RICH DESCRIPTION OF THE PROGRAM ........................................................................................ 7
STAKEHOLDERS ................................................................................................................................... 8
REVIEW OF LITERATURE .......................................................................................................... 9
LOGIC MODEL .................................................................................................................................. 10
EVALUATION APPROACH ........................................................................................................ 11
QUANTITATIVE APPROACH ...................................................................................................12
SURVEY INSTRUMENT ....................................................................................................................... 13
SURVEY METHOD & DESIGN ......................................................................................................... 14
SURVEY POPULATION AND SAMPLING PROCEDURE ................................................................ 15
STATISTICAL ANALYSIS ................................................................................................................... 16
QUANTITATIVE RESULTS PRESENTATION ................................................................................. 17
QUALITATIVE APPROACH .................................................................................... 17
QUALITATIVE QUESTIONS .............................................................................................................. 18
FOCUS GROUP PARTICIPANTS ....................................................................................................... 18
FOCUS GROUP PROCEDURE & IMPLEMENTATION ................................................................. 19
MODERATOR ....................................................................................................................................... 22
QUALITATIVE RESULTS PRESENTATION ................................................................................... 22
ANALYSIS PLAN ............................................................................................................................... 22
LIMITATIONS ................................................................................................................................... 23
TIMELINE ............................................................................................................................................. 24
BUDGET .................................................................................................................................................. 25
NEXT STEPS ....................................................................................................................................... 25
REFERENCES ..................................................................................................................................... 27
APPENDICES
APPENDIX A: MANUAL &STUDENT ASSESSMENT REPORTS 2014 ...................................... 29
APPENDIX B: MAP OF PARTNERING SCHOOLS .......................................................................... 62
APPENDIX C: LOGIC MODEL.......................................................................................................... 63
APPENDIX D1: PRE-SURVEY EMAIL INVITATION .................................................................... 64
APPENDIX D2: POST-SURVEY EMAIL INVITATION ................................................................. 65
APPENDIX E1: PRE-SURVEY PERMISSION FORM .................................................................... 66
APPENDIX E2: POST-SURVEY PERMISSION FORM .................................................................. 67
APPENDIX F: SUMMER 2014 SEAL ASSESSMENT .................................................................... 68

APPENDIX G: SURVEY CONSTRUCT MAP ................................................................................... 77
APPENDIX H: SURVEY QUESTIONNAIRE- EVALUATION PLAN ........................................... 80
APPENDIX I: FOCUS GROUP INTERVIEW QUESTIONS ............................................................ 85
APPENDIX J: FOCUS GROUP PROTOCOL .................................................................................... 86
APPENDIX K: FOCUS GROUP RECRUITMENT MATRIX .......................................................... 88
APPENDIX L: PARENTAL OR LEGAL GUARDIAN PERMISSION FORM ............................... 89
APPENDIX M: CODING RUBRIC ..................................................................................................... 91
APPENDIX N: TERMS OF PARTICIPATION................................................................................... 92
APPENDIX O: TIMELINE ................................................................................................................... 93
APPENDIX P: BUDGET ...................................................................................................................... 94
APPENDIX Q: PRESENTATION ........................................................................................................ 95
Evaluation Plan Introduction

The objective of this evaluation plan is to assess to what extent the SEaL program is
meeting its goal to further develop students social skills. As the reader endeavors in this
evaluation plan, they will encounter research findings supporting the limited research on social
skills development among youth afterschool programs. Durlak and Weissburg (2007) emphasize
how more time and resources seem to be invested in research pertaining to students academic
development, and this is why we have decided to orient our focus to social skills development
among youth. According to Jones and Bouffard (2012) academic development is as important as
social skills development.
SEaL Program Overview
The Summer Enrichment at Loyola (SEaL) program was started in 2011 at Loyola
University Chicago as a service to the Chicago Area community. The program was intended to
increase college access among mid-performing high school students identified as low-income,
first-generation, and/or students of color (Manual & Student Assessment Reports 2014; see
Appendix A). The SEaL program offers three weeks of skill development in key academic
subjects such as math, science, reading, and writing. Along with training on college related
topics such as financial aid and college applications; additionally, the SEaL program offers all
students LifeSkills workshops led by Loyola college students where the students learn about
college life and college preparation. Because this program is intended to serve families of lowincome, it is a free program to all students and families (Manual & Student Assessment Reports
2014; see Appendix A). This evaluation plan will focus on assessing the social skills
development of the students in the program. We have defined social skills to be in the form of
these activities and interactions: peer to peer interaction, public presentations, team work,
cultural competencies, interpersonal skills (i.e. engaging in activities with others),

communicating with others/networking, and reaching out for help.
The general evaluation approach of this assessment plan will be outcomes based with a
formative judgment. The intention is to determine how the social component of the program can
be improved. The focus of this evaluation plan is to assess the short, medium and long-term
social outcomes of the SEaL program, which are highlighted in our logic model (see Appendix
C). The use of Fitzpatricks, Sanders and Worthens (2003) formative, with an outcomes based,
question will be taken into account; how can we revise our curricula to better achieve desired
outcomes? (p. 27).
Significance of Problem
Our intention is to see how this program influences the students social skills development and
whether the program meets its social development objectives. In previous years SEaL program
administrators have gathered survey data primarily on academic skills development, yet limited
data on social skills development. We are constructing an evaluation of the social aspect of the
SEaL program because research pertaining to academic skills development seems to dominate
over research related to social skills development (Durlak & Weissburg, 2007).
Context and History of the Program
The SEaL program, which is designed for mid-performing students, low-income, and students of
color, is held at Loyola University Chicago (LUC) Lakeshore Campus (LSC). Mid-performing
students are defined by the high school teachers as students who need extra academic guidance
and support (K. Chavez, personal communication, October 3, 2014). The SEaL program at LUC
began in 2011 with nine students and has grown tremendously, from 60 students in 2013 to 135
students in 2014 (K. Chavez, personal communication, October 3, 2014). As a disclaimer, we
would like to note that throughout this evaluation plan we will be referencing data and
information from the 2014 summer cohort. However, the assessment part of this plan is
designed to be taken by future summer cohorts, since we anticipate that the enrollment numbers
will not exceed 160.
Summer 2014 was the first year that the SEaL program included rising 9th grade
students. Students were recruited from all parts of the city in order to increase diversity and
build a sense of community amongst them. The participating high schools from the Chicago
land area were selected because they were located in low-income urban communities (see
Appendix B).
Rich Description of the Program
High school counselors were asked to nominate students for the program in early March of
2014 and the students were asked to apply by early May of 2014. The nomination process
consisted of the counselors providing students contact information to SEaL. Students who fit
the earlier mentioned admission requirements were invited to apply for the program by SEaL
staff. Admitted students were responsible for taking public transportation to and from the
campus with the public transportation fare cards provided by the program. The participants
commuted to campus Monday thru Friday from 8:30AM- 5:00PM. Since this is a voluntary
program the students were asked to sign a Terms of Participation form stating that they
commit to attend every day the program is in session (see Appendix N).
The program was in session during the summer for three weeks, during which students
engage in daily activities purposefully designed to develop their academic and social skills. The
students participated in recreational activities along with academic workshops. They were also
allotted a generous amount of time for breakfast and lunch, during which they could engage in
relationship building dialogues (see Appendix A).

Stakeholders
The program was governed by several stakeholders, some were direct and others were indirect.
Direct stakeholders included the assistant director of the Summer Scholars program, the graduate
program coordinators (GPCs), the college coaches, and the instructors. These direct stakeholders
were the people who oversaw the program. They worked together to create the schedule and the
curriculum for all the students. The graduate program coordinators were important to the
success of the program because they, together with the college coaches, were the ones who
recruited the students. The GPCs were responsible for contacting school counselors and
community organizations when it was time for them to nominate eligible students. They were
also in constant contact with the instructors, campus partners, and the program directors to make
sure that all the materials and facilities were prepared for the duration of the program. All direct
stakeholders were in constant contact with the students; they made sure that the program ran
smoothly.
The Director of the Office of First Year Experience, the high school counselors and the
families/supporters of the students are important, because these individuals create the support
system that is needed for the students, in addition to assisting in the programming efforts for
SEaL. The SEaL program in particular is a program that is free to all students, it requires
funding from the institution, grants, and donations. It is with the help and support from the
indirect stakeholders that the program is able to run every summer. The school administrators,
counselors, and parents/supporters of the students are crucial to the success of the program,
without their trust and support; there would be no students in the program.
9
Review of Literature
A significant body of literature reviewed supports the notion that after school programs,
along with summer programs, can significantly aid a students social skills development.
After-school program implementation has increased drastically in the United States over the
past 15 years and there is widespread support in the education literature that such programs can
be a positive asset for children and communities (Miller, 2012, p. 36). In an assessment of the
Richardson Community Center entrepreneurial program, Miller (2012) concluded that the
programs success was highly due to its ability to foster and cultivate productive relationships.
Those relationships seem to be positively correlated with attaining social capital networks.
Lauvers (2012) article on student success and after school programs also emphasizes the
importance of students social development through expanded learning programs. Lauver (2012)
describes work done by Durlak and Weissburg (2007), who analyzed 73 after school programs
servicing students between the ages of five and 19. The authors found that after school programs
aiming to support social skills development were successful in doing so. That success was
mainly due to the programs effectively enhancing confidence among the students, promoting
positive behaviors toward peers and adults, reduced aggression, noncompliance and conduct
problems along with drug use (Lauver, 2012, p. 42).
The findings of Wright et al. (2009) supported an increase of prosocial behaviors and
bonding with peers for youth from low-income communities (p. 74). Through conducting
research centered on prosocial settings and youth, along with observing various after school
programs, Wright et al. found peer-relationships a contributing factor to a positive social
adjustment and outlook. The authors also found that as peer social support increased, prosocial
behavior and self-esteem increased as well (p. 87).
10
These studies insinuate that well structured and intentional after school programs support
students social development. The social development aspect is central to the mission of the
SEaL program and for that reason it will be the main focus of this evaluation plan.
Logic Model
In order for this evaluation plan to follow some sort of a sequence/order, we created a
logic model (see Appendix C). This logic model will guide our evaluation plan through its
clearly defined inputs, outputs and outcomes. Since this evaluation plan is outcomes focused, we
have identified the short, medium, and long-term social skills development outcomes we will be
assessing.
The logic model first outlines the inputs of the SEaL program that are divided into three
categories: stakeholders, Loyola University personnel and university resources. Next, the model
describes outputs, which were generated by inputs. The outputs pertain to activities and
participation. The activities section under outputs depicts the academic and social activity
involvement of the SEaL students. The activities component provides the reader with a sense of
what the program entails and how the students engaged in the program. The participation section
under outputs solely highlights the students as the main participants and identifies their
characteristics. Following the outputs, the logic model describes the programs outcomes. The
outcomes section is divided into three categories: short, medium, and long-term outcomes. The
intentionality behind the three categories is to showcase the immediacy of some outcomes versus
the more progressional development of others.
All of the highlighted elements from the short, medium, and long-term outcomes are
social skills that the SEaL program aims for its students to develop. The short-term outcomes
centered on developing professional behavior, building relationships and developing
11
interpersonal skills, are outcomes that the students are able to accomplish throughout the SEaL
program. The medium and long term outcomes related to sharing knowledge with peers/family,
enhancing social skills, fostering cultural competencies and cultivating networking skills, are
skills that can take a longer time to develop. Through this evaluation, we would be able to assess
whether the program is meeting its objective of developing students social skills; skills that the
SEaL program truly anticipates for its students to continue to practice and implement throughout
their lives.
The SEaL program strived to accomplish its learning goals/ objectives by incorporating
all of the elements of a logic model (inputs, activities, outputs, and outcomes). The inputs of the
SEaL program were the driving forces of the program as well as its main contributors.
Evaluation Approach
The nature of the SEaL program requires us to utilize a formal outcomes oriented approach.
Fitzpatrick, Sanders and Worthen (2003) described a formal evaluation approach as a structured
and reliable way of gathering systemic data. Additionally, the SEaL programs outcome we
choose to evaluate is the students social skills progress at the end of the program. Fitzpatrick
et.al explain that, outcome or impact studies are concerned with describing, exploring or
determining changes that occur in program recipients as a result of a program (p. 26). In the
case of the SEaL program, our outcomes evaluation approach is geared towards assessing the
development of the students social skills, which we defined earlier. Since the SEaL program is
inclusive of multiple grade levels (9-12 grade), we will invite all 135 students to participate in a
pre- and post self-administered survey.
The evaluation plan will utilize a mixed-methods approach to evaluate the students
social outcomes related to participating in the SEaL program. We will follow an explanatory
12
sequential design where we will first collect and analyze quantitative data in the form of pre- and
post-surveys. In the second phase of this explanatory sequential design, we will collect and
analyze the qualitative data from focus groups. Since our quantitative data will be more
dominant, the qualitative data will be utilized to support and explain the quantitative data.
Research and assessment pertaining to students social development in such programs,
according to Durlak and Weissberg (2007) has been overlooked, in terms of formal evaluation
(p. 5). Hence, the reason why we want to assess to what extent the SEaL program is meeting its
objective to further develop students social skills.
Some of the weaknesses pertaining to the outcomes approach include environmental
variables that are out of our control, which can impact the students social skills development.
Some of those variables may include social encounters that students might experience during
their commutes to and from the program, along with external social encounters that take place at
the students residence. Again, we cannot account for such experiences, because this evaluation
plan does not encompass what happens to students outside of the SEaL program. Some other
weaknesses pertain to students preconceived notions of the identities of others (e.g. gender, age,
sexual orientation, and race) and how those notions can impact their social skills development
(e.g. being respectful of others cultural backgrounds) during the program.
Quantitative Approach
Initially, students and parents will be notified of the opportunity for the student to
participate in a pre-survey through an email announcement (see Appendix D1). The students and
parents will receive a consent form titled Pre-survey Permission Form (see Appendix E1).
Within this form the students will be invited to participate in the pre-survey, as well as be
13
informed of the benefits linked to their participation. Additionally, parents and students will be
informed that students personal information will also be released to the evaluators of the SEaL
program. Incentives for the pre-survey will be in the form of school supplies (e.g. water bottles,
Loyola T-shirts, and Loyola pens).
Prior to students taking the post-survey, they will be asked to obtain a signed permission
form from their parents or legal guardians, allowing them to participate in the post-survey (see
Appendix E2). Students will receive an incentive for taking the post-survey as well. That
incentive will comprise of students being entered in a raffle where five students have the chance
of winning a $20 Visa gift card. Each student will be assigned a random number on two sheets
of small paper. Each paper will be stamped with an LUC signature mark for authentication.
After which, one of the papers will be placed into a large bin, from which a graduate program
coordinator will randomly select five winners. The other stamped paper will be left with the
student as a way of identifying the winners, and trying to limit fraud.
Survey instrument
As a disclaimer, prior to the implementation of this evaluation plan, we will consult with
the Institutional Review Board (IRB) since the majority of our student population is comprised
of minors. The survey instrument we created for the purposes of this evaluation project is a
modified version of the current pre- and post- SEaL summer 2014 survey (see Appendix F). The
survey content and style (i.e. questions) are adapted from the Social Skills Assessment
Adolescents, which was found on the Adapt Behavioral Services website (http://www.adaptfl.com/files/Social%20Skills%20Assmt-Prepare.doc.). We have tailored the questions to the
outcome we are trying to evaluate, which is the development of students social skills. The
survey will contain 29 questions pertaining to the social skills categories mentioned previously.
14
The survey construct map we have developed assists the reader in navigating and intricately
deconstructing each question, along with connecting the questions back to the Logic Model and
purpose behind this evaluation plan (see Appendix G). All students will be eligible to answer
every question on both pre- and post-surveys. All survey questions, with the exception of the
demographic questions and one multiple-choice question, follow a Likert scale format.
In order for us to gain a more concrete understanding of the survey and its logistics (e.g.
duration of survey) we will conduct a pilot study. The pilot study is intentional because we
would like to determine whether the participants easily understand the questions on both surveys.
We will ask three former students from the 2014 cohort to complete both pre- and post-surveys
and share any verbal feedback they might have after taking the surveys. This pilot study will
provide us with a better understanding of the time it takes to complete the surveys, and it will
also support our validity.
Survey Method & Design
The primary purpose of this quantitative survey is to evaluate whether participating in the
SEaL program helped students improve social skills (i.e. peer-to-peer interaction, public
presentation, team work, interpersonal skills, communicating with others/networking, reaching
out for help). We have chosen to implement a one group longitudinal pre-test post-test design,
because we would like to compare students level of social skills prior to entering the program
and once again after completing the program.
The pre-survey instrument (i.e. pre-test) will be administered to the students on the first
day of the program, where students will be taken to a computer lab to take the online Google
Forms survey (see Appendix H). The pre-survey will be in the form of an online self-
15
administered survey, which the students will receive via email. The pre-survey incentives will
be handed out to students after all students have completed their pre-survey. As mentioned
earlier in this evaluation plan, each cohort of students will have a college coach present in the
computer lab while students take the pre-survey. The college coach will be there to answer any
questions students may have regarding the technology or the pre-survey. The same principle will
apply when administering the post-survey. Each college coach will take their cohort into a
computer lab on the last day of the program where students will access their emails and take the
post-survey (see Appendix D2). Once all students have completed their post-surveys, their
names will be entered into the raffle that was mentioned earlier.
This pre-experimental design would allow us to explore if the SEaL program was
successful at meeting its objective to further enhance and develop students social skills. This
way of collecting the data will be economically efficient and will also yield a fast turnaround in
regards to the results (Creswell, 2009). Administering the survey during the last day of the
program is feasible and convenient since all students will be aware of the incentive that is to
come, if they choose to take the survey.
Survey population and sampling procedure
Our survey population will comprise of the students participating in the SEaL program of
summer 2015. Our population is a non-random stratified convenience sample. The sub-group
categories we would like to compare will be students grade level, gender, and race. Creswell
(2009) defined a stratified sample as specific characteristics of individuals are represented in the
sample (p. 148). For the intent and purposes of this evaluation plan, our survey will follow a
single-stage sampling procedure. According to Creswell (2009) a single-stage sampling
procedure is one in which the researcher has access to names in the population and can sample
16
the people directly (p. 148). Students confidentiality will be protected by them entering their
student ID numbers instead of their full names. As mentioned earlier, survey information will
only be used by SEaL administrators.
Statistical analysis
For the data analysis part of this evaluation plan we will incorporate a t- Test design,
which tests the statistical significance of the difference of means. The t- Test design will allow
us to better grasp and compare the relationship between our variables, which are the pre- and
post- surveys. Newcomer and Conger (2010) emphasized that t-Test utility is appropriate when
the research question at hand has a null and an alternative hypothesis along with multiple means
to test. Since our research question has a null and an alternative hypothesis, along with a pre-test
and post-test means, we will be utilizing a t-Test for our statistical analysis. The H0 (null)
hypothesis is that the [SEaL] program has no discernable impact on developing social skills
among low-income, mid-academically performing youth. The alternative, Ha hypothesis states
that there is a statistically significant relationship between participating in the [SEaL] program
and the social skills outcomes of program participants.
Our independent and dependent variables will be measured at an ordinal level. Ordinal
level measurements are appropriate for Likert scale surveys and fit with our purpose of
measuring the difference between our two means (Sauro, 2004; Newcomer & Conger, 2010).
The independent variable in this case is the matched pair. Matched pair is indicative of the
students before and after survey responses in relation to participating in SEaL. On the contrary,
the dependent variable will be the dimensions of social skills being measured (e.g. friendship
making skills, cultural competence, etc.).
17
Quantitative Results Presentation

The findings of this assessment will be presented in a written report utilizing a few
different methods to visually depict the information. The responses from the pre- and postsurveys will be summarized and written in a narrative form. To bring more transparency to some
of the responses, a few students direct answer selections will be quoted. The identity of the
students will be kept confidential by not releasing any demographic information about them.
Additionally, pie charts will be employed to provide a visual representation of participants
demographic information. Bar graphs will also be employed to visually represent answers from
the Likert-scale questions. We will present the findings to the director of the SEaL program,
Karladora Chavez, with whom we will go over the results and share our thoughts on what, if
anything, needs to be done next.
Qualitative Approach
Our assessment plan also contains a qualitative component, which will further explore
students thoughts on the SEaL program, along with their experiences. The qualitative part of the
assessment will be facilitated after the students complete the post-survey. This way the students
will be predisposed to the notion of thinking about their experience during the SEaL program,
and during the qualitative component, they will have the opportunity to vocalize their thoughts
through open-ended questions. The qualitative component of this assessment plan will be in the
form of focus groups. According to Krueger and Casey (2010), a focus group is a great tool for
gathering information for formative evaluation plans. Focus group interviewing can elicit
information about what is currently working, not working, or needs improvement in a program.
They have been used to listen to participants and staff members perceptions of program
outcomes (p. 380). Since our evaluation plan is outcomes focused, employing focus group
18
interviewing to collect qualitative data is most appropriate. Additionally, focus group interviews
are time efficient and cost effective. The data collected from focus group interviews can later be
applied towards making improvements to a program (Krueger & Casey, 2010).
As alluded to earlier, the elements of our evaluation plan that the qualitative component
will address will be whether the SEaL program has met its goal of further developing students
social skills. The focus group platform will enable students to contribute their honest thoughts
and opinions on their experience throughout the SEaL program.
Qualitative Questions
The objective of our focus group interviews is to probe deeper into the students experiences and
explore the students thoughts on how/if the program further developed their social skills. The
focus group interviewing does not contain set parameters such as a survey questionnaire, where a
person has a limited answer selection. The open-ended questioning method contains no right and
wrong answers, but rather deeper exploration of a topic. The questions will follow a funnel type
sequencing, where the initial question will address students overall experience in the SEaL
program (Krueger & Casey, 2010). The following questions will probe at classroom
environment, projects and group activities (see Appendix I). All of these concepts are connected
to the overarching theme of the evaluation: social skills development. Additionally, several of
the questions will address the objectives of the survey questionnaire, such as communication
skills, networking skills, and friendship building skills.
Focus Group Participants
The focus groups will be comprised of students who participated in the pre-survey and
post-survey parts of the evaluation. We anticipate having a total of four focus groups. A focus
group of four freshmen students, a focus group of eight sophomores, a focus group of eight
19
juniors and a focus group of eight seniors. Focus group participation will be voluntary. At the
start of the SEaL program, all students will receive a consent form stating the terms and
conditions of the focus group along with the benefits of their participation to the overall program
improvements (see Appendix L). Although all students who submit a signed consent form will
have the chance to participate in our focus groups, we only want a select number of students in
order to obtain a sample of the SEaL student population. Additionally, the size of our focus
group is suitable in this case because, for some students, the topic being discussed might be
perceived as sensitive and personal (Krueger & Casey, 2010).
The focus group participants will be homogenous in nature. What homogeneity means
in this context is that the participants have something in common that relates to the topic of
conversation (Krueger & Casey, 2010, p. 382). The basis of homogeneity in the context of our
evaluation plan is that all focus group participants are SEaL students, all of whom identify as
people of color, and belonging to at risk inner city schools. At risk schools are likely to serve
a high proportion of minority and low-income students, have poor student achievement, andif
they are high schoolshave lower graduation rates (Qualified Teachers for At-Risk Schools,
2005, p.6). A strength to having a homogenous sample is the comfort that having something in
common brings to individuals, this fosters thoughtful listening (Krueger & Casey, 2010, p.
382).
Focus Group Procedure & Implementation
A focus group protocol will be used to lead the focus group discussions and ensure that
there is structure (see Appendix J). Our focus group protocol is inspired by the sample focus
group protocol by Schuh (2009) and we have adapted the protocol to fit the purposes of our
study and our sample. Being cognizant of our four grade levels (freshmen, sophomore, junior,
20
senior), we have decided to request four volunteers from each class. As mentioned earlier, the
number of volunteers will not be consistent due to the grade levels ranging in size. With this
being said, we anticipate to have a total of 28 focus group volunteers. According to Krueger and
Casey (2010), the size of a focus group can range from as few as four or five to as many as a
dozen people (p. 382). We want to implement a consistent focus group division and therefore,
we will try to recruit four students from each class (see Appendix K). This equates to having
four focus groups in total, where students from each of the two sophomore classes will
participate in the same focus group. The same principle will be applied to the junior and senior
level classes (see Appendix K). In the end, the size of each focus group will range anywhere
from four to eight participants. We, the authors of this evaluation plan, will serve as the
moderators of the focus group interviews. We believe this is most appropriate because, we are
not directly associated with the SEaL participants. Thus, it is our assumption that students will
not feel pressured to produce specific answers, nor will they feel afraid to speak up and share
their experiences. The power dynamic between the students and us will be very minimal. After
the students have finished their post-test survey, the instructors will notify them that it is time for
the focus group interviews. The selected 28 students, who have a signed by a legal guardian
consent form (see Appendix L), will be invited to voluntarily participate in the focus group. It
will be restated that even though they have a signed consent form, they are not obligated to
participate if they choose not to.
We have allotted 60 minutes for the focus group interviews, during which the students
will be answering six open-ended questions pertaining to our overarching research question (e.g.
social skills development). We believe the duration of the interviews will enable students from
the smallest focus group (4 students), to the largest focus group (8 students) to candidly
21
participate and address the six questions we have prepared. We deem 60 minutes to be an
appropriate set of time, during which all students from the largest focus group will have an equal
chance to participate more than once.
The rationale behind our questions is to address the objective of our study, as well as to
be conversational and information generating (Krueger & Casey, 2010). The moderators will
follow a question sequence, which starts with a question that opens up the conversation, after
which, the following questions target the topic at hand: social skills development. There will be
a total of six questions: four overarching questions, and two probing questions. The probing
questions purpose will be to route the conversation.
The focus group interviews will take place at Loyola Universitys Lake Shore Campus,
where available classrooms will serve as the setting for the focus groups. We will have the focus
groups audio taped so that we, the evaluators, can later transcribe the facilitations and input the
required information into coding software (Microsoft Word). There are confidentiality risks
associated with employing this practice, such as exposing the identity of the participants. We
have decided that we will be the only ones who will have access to the audio recordings.
Furthermore, while transcribing the audio recordings, whenever we come across a student name
that was exposed/ addressed, we will assign a pseudonym for that student. Once we have
finished with all of the transcriptions, all four of the audio recordings will be permanently erased.
Through implementing such tactics we are minimizing the risks associated with students
confidentiality. Parents/ legal guardians of the students will be notified of our intentions and
plans in the form of a focus group consent form, which will require their signature (see Appendix
L). Focus group consent forms will be distributed to students to take home at the start of the
program.
22
Moderator
As mentioned earlier, we will serve as the moderators of the focus groups in order to
make students feel comfortable enough to share their honest thoughts and experiences as they
relate to the SEaL program. Since we are the creators of this evaluation plan, we will be
prepared to keep the conversation focused on the questions at hand. We have dialogue
facilitation experience and training, which will certainly be applicable to the focus group
interviews. Additionally, we plan to prepare for the focus group interviews a few days before we
execute them, as well as follow a focus group protocol, which was mentioned previously (see
Appendix J).
We will also know when to ask follow- up questions and when to give the group some
time to reflect on a question. For every focus group, one of us will serve as the moderator while
the other as the assistant moderator, who will be responsible for taking notes and audio taping
the conversation.
Qualitative Results Presentation
The final results of the qualitative section of this evaluation will be displayed in a written report.
The report will be organized by themes such as engagement, site process, safety, cultural
experience/exposure and socializing, to name a few (Krueger & Casey, 2010) (see Appendix M
for themes/codes). The qualitative report will be comprised of a bulleted style summary list
where key findings and themes from the coding are highlighted. To further support each theme,
direct quotes from the students will be incorporated in the report as well.
Analysis Plan
The objective of a qualitative analysis is to find out what is meaningful to the purpose of
the study (Krueger & Casey, 2010, p. 396). A systematic analysis process will be implemented,
where we, as mediators, will listen for key points and reoccurring themes during the dialogue.
23
We will employ a descriptive coding analysis (Rogers & Goodrick, 2010). Descriptive coding
assists you in managing the volume of data by making it easier to retrieve and aggregate data
relating to a particular issue (p. 440). According to Rogers and Goodrick (2010) categories or
evaluator-generated codes can be created either before or after evaluators review all of the data.
Other decisions pertaining to the coding process can also be made either before or after data is
reviewed. For the purposes of our evaluation plan, we will develop a few initial codes that will
be tentative (see Appendix M), and as we start to apply them, we will modify them (Rogers &
Goodrick, 2010). As we are explicitly coding the transcriptions, we will employ Microsoft Word
to assist us in identifying reoccurring themes among students answers. The reasoning behind
our decision to employ Microsoft Word is to save time and accelerate the coding process.
Since all of the transcriptions will be written on Microsoft Word, we will take advantage of that
convenience and start to tag and highlight text containing some of our identified codes, as well as
new, potential codes/themes.
Limitations
Quantitative Approach
We understand that there are some limitations to keep in mind pertaining to the quantitative
approach. The number of students taking the pre-survey and essentially the post-survey should
be the same. However, we also understand that not all students who show up on the first day of
the program and take the pre-survey will be present on the final day of the program in order to
take the post-survey. The SEaL program has attempted to encourage the daily participation of
students throughout the program and in particular, by abiding by the programs Terms of
Participation (see Appendix N). Another potential limitation is the format of the survey, which
follows a Likert-scale format. In some instances the answer choices available may not be
24
representative of a students authentic response to a question. The way a question is worded

might also mislead a student, especially if they do not understand some of the terminology being
used.
Qualitative Approach
In addition to quantitative limitations, we also recognize potential qualitative limitations. A few
of those limitations include the environment in which the focus group interviews will be held, the
time allotted for conducting the interviews, the questions being asked and lastly, the moderators
approach to the focus group interviews. For an example, some of the focus group questions may
be better comprehended by upper classmen versus the rising 9th grade students. This could
potentially skew the results and impact validity. If students request for an explanation, this
potential limitation can be avoided by the moderator with additional clarification.
Timeline
A timeline was created with the intention to successfully implement this evaluation plan
in a timely manner (see Appendix O). It is our hope that the timeline is employed each year for
every cohort. For the purposes of this specific evaluation, we have taken into account the
incoming summer 2015 SEaL cohort and have constructed the timeline accordingly. We
intentionally backtracked our timeline to August, 2014 because, we decided that it is best to start
preparing our assessment materials for the next cohort early on. We will administer the presurvey and post-survey to a few volunteers from the 2014 SEaL cohort with the intention of
tailoring and adjusting the survey based on the pilot tests. We anticipate that there will be a
sufficient amount of time to prepare for the incoming 2015 SEaL cohort, which includes
obtaining an approval from the Institutional Review Board (IRB). Since the 2015 SEaL program
will begin in late June 2015 and end in mid-July, 2015 most of the work will be done during the
25
duration of the program (e.g. administering the pre- and post-surveys, analyze data from pre- and
post-surveys, conduct focus group interviews).
It is integral that everyone follow the designated timeline and everything be completed on
the assigned month. This way we can avoid any delays, which could potentially jeopardize the
delivery of the final report. As a disclaimer, we do understand that in some instances delays are
inevitable, and therefore, we will modify the timeline if need be.
Budget
Having a budget for a project, which in this case is in the form of an assessment, is of
high significance in order for the project to be implemented. The budget we have created for this
evaluation plan (see Appendix P) covers the projected costs we have accounted for as needed in
order to implement our evaluation. The budget can be applied to the evaluation of the 2015
SEaL program.
Since we would like to have a large number of participants for both the pre- and postsurveys, we will allocate a larger sum of our budget towards incentives (e.g. Visa gift cards).
Furthermore, to compensate this cost, some of our evaluation-associated expenses will be either
free of charge or very cost efficient (e.g. survey administration, consent forms, college coaches,
incentives provided by SEaL program, etc.).
Next Steps
As this evaluation plan development comes to an end, there are several next steps to
consider for continuing to move forward with development of the SEaL program.
First, as the intent of this evaluation is to understand and assess the social development of the
student population in the SEaL program. Our goal is to create an understanding of the
importance of social development in summer programing in order to create innovative ways to
26
restructure and potentially shift the program objectives. The student experiences and
testimonials are the primary sources of information being used to move forward with the
development of the SEaL program. It is fundamental to create clear, actionable, and purposeful
plans to move forward with improvement, with the analyzed data. As the SEaL program is a
rather new program, working to implement new changes, as a result of this evaluation, will assist
in its continued growth.
27
References
Creswell, J.W. (2009). Research design: Qualitative, quantitative, and mixed method approaches
(3rd Edition). Thousand Oaks, CA: Sage Publications.
Durlak, J. A., & Weissberg, R. P. (2007). The Impact of After-School Programs that Promote
Personal and Social Skills. Collaborative for Academic, Social, and Emotional Learning
(NJ1).
Fitzpatrick, J., Sanders, J., & Worthen, B. (2003). Program evaluation: Alternative approaches
and practical guidelines (3rd Ed.) New York: Longman.
Jones, S. M., & Bouffard, S. M. (2012). Social and Emotional Learning in Schools From
Programs to Strategies. Social Policy Report, 26, 1-33.
Krueger, R. A., & Casey, M. A. (2010). Focus Group Interviewing. In J. S. Wholey, H. P. Hatry,
& K. E. Newcomer (Authors.), Handbook of practical program evaluation (3rd ed.) (pp.
378-403). San Francisco, CA: Jossey-Bass.
Lauver, S. (2012). Supporting Student Success Through After-school and Expanded Learning
Programs. District Administration, 48(3), 40-43.
Miller, P. M. (2012). Community-Based Education and Social Capital in an Urban After-School
Program. Education & Urban Society, 44(1), 35-60. doi:10.1177/0013124510380910
Newcomer, K. E., & Conger, D. (2010). Using Statistics in Evaluation. In J. S. Wholey, H. P.
Hatry, & K. E. Newcomer (Authors.), Handbook of practical program evaluation (3rd
ed.) (pp. 454-492). San Francisco, CA: Jossey-Bass.
Qualified Teachers for At-Risk Schools. (2005). National Partnership for Teaching in At-Risk
Schools, 2-22.
Sauro, J. (2004). Fundamentals of Statistics 1. Retrieved October 13, 2014, from Usable Stats
28
website: http://www.usablestats.com/about.php
Rogers, P. J., & Goodrick, D. (2010). Qualitative Data Analysis. In J. S. Wholey, H. P. Hatry, &
K. E. Newcomer (Authors.), Handbook of practical program evaluation (3rd ed.) (pp.
429-453). San Francisco, CA: Jossey-Bass.
Wright, R., John, L., Duku, E., Burgos, G., Krygsman, A., & Esposto, C. (2009). After-School
Programs as a Prosocial Setting for Bonding Between Peers. Child & Youth Services,
31(3/4), 74-91. doi:10.1080/0145935X.2009.524461
29
Appendix A: Manual & Student Assessment Reports 2014
Summer Enrichment at Loyola

SEaL
Manual and Student
Assessment Reports 2014
30
Table of Contents
Executive Summary .....................................................................................................................................3
Mission ..........................................................................................................................................4
Program Components
Pre-Program Logistics .................................................................................................................................5
Participating Schools
Application Process
Timelines
Summary
Pre-program Preparations ................................................................................................................. 10
Spreadsheets
Classrooms
CTA Passes
Schedules
Instructors
Swag
Family Orientation
Day one
Program Interactions ............................................................................................................................... 14
Classes
Programs
Assessment Summary .............................................................................................................................. 16
Measurement Tools
Demographics
Pre-Assessment Data ...............................................................................................................................18
Post-Assessment Data ...............................................................................................................................21
Assessment Results
Student Testimony.....................................................................................................29
Recommendations ....................................................................................................................................31
SEaL Flyer .................................................................................................................................................. 33
31
Executive Summary
The Summer Scholars program, under the Office of First Year Experience, had over 130 students enroll in
the program this year. The program seeks to provide prospective students a micro-college experience
during the summer. Students had the opportunity to participate in a 3-week summer program. The
program ran from Monday through Friday from 8:30am 5:00pm. They took classes designed for their
academic level and their demographics.
Staff worked hard to create an intentional learning experience for participants, leading to the
creation of three major learning outcomes. As a result of participating in the Summer Scholars program,
students will:
1. Enhance academic skills and preparation for future goals
2. Understand and develop their identity as it relates to Loyola University Chicago and the
surrounding community
3. Create a long lasting bond by mentoring
Significant Outcomes
Students were asked to complete a pre- and post-assessment of their experiences. Major outcomes of
the post-assessment are listed below:
100% of the respondents indicated a level of confidence in their assigned academic program:
Reading, Science, Math, or Writing.
100% of respondents indicated a high level confidence in the plan to attend a college or
university after they completed high school.
95% of the respondents rated Activity Period a 5 or higher on the scale of 1-10.
94% of respondents indicated that they had established new relationships during their time with
Summer Scholars, whether that was their peers, the staff, and/or their faculty members.
75% of respondents indicated that they were confident in completing FAFSA.
32
Mission
The Summer Enrichment at Loyola (SEaL) program has joined the Summer Scholars program and will run
under the Office of First Year Experience for years to come. The SEaL program exposes local high school
students to a college environment and education in the diverse urban environment of Chicago at no
cost. The majority of students in the program are of low income, first generation, foster youth and/or
people of color. The program offers the opportunity to grow both intellectually and socially by:
Building community with peers, college student leaders, and faculty and staff;
Learning the life skills and knowledge essential for college success;
Immersing themselves in their academic passions both in and outside the college classroom.
Enhance academic skills and preparation for future goals
Understand and develop their identity as it relates to Loyola University Chicago and the
surrounding community
Ultimately, the SEaL Program is committed to helping students put their best foot forward as a college
applicant and future college student.
Program Components
In order to achieve the program learning outcomes, the Summer Scholars program has recurring
programs and curriculum aimed at achieving student development in these areas. These programs
include:
Participating in a 3-week academic enrichment program
Participating in College Coach-sponsored programs
Academic preparation in either Mathematics, Reading, Writing, Science, and general study skills
Preparation for college selection, application, and admission processes
Committing to complete to program
33
Pre-Program Logistics
This section describes the details in planning for the SEaL program. It includes the links to the emails,
flyers, and spreadsheets.
Participating Schools
For the summer of 2014, the goal was to increase the number of SEaL participants radically. The
objective was to have close to 160 rising 9th -10th grade students participating this summer. Due to some
changes in the SEaL program, recruiting for the freshmen class was a task added a couple of months
before the program began. Recruitment for rising 10th 12th grade students began in early March by
contacting the schools that part took in the program in previous years. The participating high schools in
2013 are listed below:
o Summer 2013 High schools:
Southland College Prep
Senn High School
Noble Street College Prep
Kelly High School
John Marshall Metro High School
Gwendolyn Brooks High School
Curie Metro High School
Cristo Rey High School
Christ the King Jesuit Prep High School
As a Graduate Program Coordinator, it is important to begin communication with school administrators,
and their staff, early in the year. All the emails sent to the counselors and students have been put into a
word document which can be found in the N: Drive, the document is called SEaL emails 2014:
N:\FirstYearExperience\FirstYearExperience\FYE\Pre-College Summer Scholars\SEaL\SEaL emails 2014docx.docx
Each school and organization was asked to submit an online nomination form with general information
of their interested students who fell under our criteria. We asked that each school submit a minimum of
4 nominations per grade level- rising 9th 12th grade students. Asking for student nominations is a way
gathering student contact information; however, it should not be mandatory. The counselors are
encouraged to submit a nomination by a suggested date in order to begin mass emailing. The
nomination form is called SEaL Nominations 2014 can be found in the Google drive: 2014 Summer
Documents>SEaL> SEaL Pre-Program> SEaL Nominations 2014.
After several planning meetings, the Leadership Team determined that expanding the invitation to other
High Schools in the Chicagoland area would be a great way to increase the number of participants. A
counselor list was created with the most updated phone numbers and emails of the
counselors/principals to contact for student recruitment. Over time the list grew dramatically. As of
now, the list includes elementary schools, middle schools, high schools, and community partners or
organizations. The list was last updated in July 2014 with phone numbers, fax numbers and emails. The
counselor contact list can be found in the Google drive: 2014 Summer Documents>SEaL> SEaL Pre-Program>
SEaL Counselor Contact List.
The summer of 2014 was the first year the SEaL program had a freshmen class. First Star is a program
which Summer Scholars is going to adopt. Funding for the program was not where is needed to be at the
time; however, we wanted to continue with this pilot program, so recruitment for the freshmen cohort
began in mid-May. Something we were not conscious about was the differences in academic schedules
8th graders have in comparison to high school students. For the most part Chicago Public Schools have a
34
freshmen orientation that conflicted with the duration of the SEaL program. Phone calls were made in
order to see which of the participating schools would be able to help recruit for the rising 9th grade class
as well. We realized that the best way to recruit for the freshmen class was by contacting Charter
Schools, Noble Network, and Private schools as well as the public schools. It is highly recommended to
contact 8th grade teachers and principals through email and by phone. It is much more difficult for
elementary and middle schools to release student information; for that reason, delivering flyers or
students applications in person is a great way to build trust.
Application Process
The application process this summer was changed to make things easier for all parties involved. In
previous years, the counselors would receive a packet which included the applications for the students,
the information packets for the parents, and flyers. This year we wanted to be more involved in the
communication with the students about the program. The online application became live as of early
March and the link was provided with all the recruitment materials. A paper application was created to
mimic the online application, which was sent out to schools per request.
Student Recruitment
During recruitment period, several school visits were scheduled per request. The schools and
organizations were made aware of this option during the early stages of recruitment. School visits were
normally during school hours: Monday-Friday, 8:00AM- 4:00PM. Some schools would ask that
recruitment be done during college days. Occasionally community organizations requested a speaker to
present during an event on weekends. During these events, the recruiter would pass out flyers, pens,
and applications.
Students should be contacted as soon as nominations are submitted. Flyers and mass emails should
continue to be sent out to community partners, parent groups and other organizations. Some
organizations to consider contacting early in the year are as follows:
Boys and Girls Club
YMCA
Local Churches
Community Centers
35
College Coach Recruitment

The college coaches play a huge role in this program. They work directly with the students every day
for the duration of the program. They assisted in the pre-program logistics by participating in a call
center in which they called all the students and reminded them about attending the program. It was
very important to recruit of team that would put in the time and effort to make this summer a
memorable one. The college coaches were recruited under the office of First Year Experience along with
the orientation leaders. Summer Scholars became the umbrella for five different programs: Academy,
Institutes, Seminars, SPELL, and SEaL. For that reason, over 30 college coaches were hired to work with
all the programs. All the college coaches lived and worked at Messina Hall where they got to know one
another and the students. They went through an intensive three week training which included a retreat
component at LUREC.
Instructor Recruitment (Bridget Wesley can Insert more info)

The SEaL program is unlike any other program. We try to provide students with a college experience yet
cater to their academic needs. Instructors are carefully chosen to provide a holistic experience to the
students. This summer the increase of participants allowed us to invite previous SEaL instructors back
and open up the opportunity to others. Through referral based recruitment, instructors went through an
interview process with the director of the Office of First Year Experience, Bridget Wesley. Along with all
other summer scholars instructors, they had a one day orientation where they received their class
schedules and had the opportunity to meet the college coaches who would be working with them for
the three weeks.
Admissions
Enrollment into the Pre-College Summer Scholars SEaL program is decided by Graduate Program
Coordinator. The direct involvement with the admissions processes and procedures allow for an easier
facilitation in the disbursement of information to students and ease registration issues. Applications
were accepted on a rolling basis. The students would complete the application which had a short answer
component which was the most important part. This was the only way to see the type of student who
was applying. Transcripts where submitted by the counselors after the students were accepted into the
program.
The last day to submit applications was two weeks prior to the start of the program. Student application
deadlines should be enforced in the future. Having more clear information on the website with set
timelines of communication will help in having a more concrete list of participants.
In theory this process is easy to work with; however, students would attend family orientation or the
first day without having ever applied. College coaches had laptops and tablets on hand for those
students to apply upon arrival. We allowed students to join the program because students would drop
out of the program, allowing a seat for others. This became somewhat of an issue with the junior class.
Ideally we would want no more than 20 students per classroom, but in the end the two junior cohorts
had over 25 students.
Counselor Contact Timeline

Counselor communication flow should be very natural. Counselors will build a sense of trust in the GPC
and in the program. They will begin to feel comfortable contacting you at any point in time with all sorts
of questions. It is important to send out recurring emails but also leave room for them to feel a sense of
36
connection. Although for summer 2014 there was a good communication flow, in the future it would be
a good idea to update the instructors on who completed the program and how their students did. The
following schedule of communication was used for summer 2014 for the most important information:
Nomination instructions (1-2 Days after nomination) Early March

Next Steps (1-2 Days after nomination letter) Mid March
Nomination Deadline Reminder (Once a week-Final day to apply) Early April
Admitted student list (1 week after nominations deadline) Mid April
Transcript request (included with admitted list)
Thank you for supporting (1 week prior to start of program)- soon after all information
has been submitted
Student Contact Timeline

Student contact should include set deadlines and dates when information will be sent. For example,
student communication happened sporadically with some students receiving information only one week
or a few days before orientation. The following schedule of communication was used for summer 2014:
Nomination Letter (1-2 Days after nomination) Mid March

Next Steps (1-2 Days after nomination letter) Mid March
Application Deadline Reminder (Once a week-Final day to apply) Throughout April
Admissions Letter (1-2 Days after admittance)
Forms and Program Benefits (included with admission letter)
Student Welcome/Orientation Virtual Packet (One Month Prior to Orientation)
o Orientation Information
o Campus Maps
o Activities/Programs Info
o Dress Code
o Class Syllabus/Book Purchase/Supplies
Call Center ( 1 week after Family Orientation)
First Day of the Program Reminder (1 week prior to start of program)
Follow-up and student confirmation should occur as deadlines are missed. Students should be
informed that they will be dropped from the program if they do not confirm their attendance one week
prior to the start of the session.
37
Summary
Overall, administration should move to a more clear process of communication and outreach to
students. In the future working closely with instructors, schools counselors, school administrators and
community partners to develop clear procedures which will ensure that students are processed in a
timely and effective manner should be initiated. Presenting the mission of the program and the learning
outcomes while recruiting, to those involved, can allow for increase in student and parent interest.
Below is a table with clear learning outcomes that students present after completion of the program.
Academic Identity
Social Identity
Understand classroom dynamics in a university

setting
Develop skills and confidence to pursue future
academic goals
Develop educational interests and pursuits
Navigate between their personal values and those

of the community
Develop public speaking skills
Practice critical thinking skills
Establish relationships with faculty, staff, and

peers
Understand the concept of shared space
Increase their self-efficacy in utilizing resources
Become independent thinkers and responsible
38
Pre-Program Preparations
This section describes the details in planning for the SEaL program. It includes examples of emails,
timelines, flyers, and spreadsheets.
Spreadsheets
When working with a large group of people, keeping things organized can become a hard task. For the
SEaL program, several spreadsheets were created in order to keep things organized. A list of completed
student applications was created along with on for those who were nominated but never applied. These
lists were used when we needed to send mass emails. They allowed us to send the proper emails to
those students and to know what type of emails needed to be sent out. Out of all the spreadsheets
created, the most important one was the complete student roster which included check lists of the
forms the students had turned in. This list was used to create the individual cohort rosters and it was
also used to give to Damen Dining for the lunch period. This list can be found in the N: Drive or the
Google drive: N:\FirstYearExperience\FirstYearExperience\FYE\Pre-College Summer Scholars\SEaL\2014\SEaLimportant files\SEaL Pre- Program\Seal Final Roster 2014.xlsx
Gathering materials from students

Soon after admitted, they are sent a list of materials that need to be turned in prior to the
program. This year we asked the students to request from their instructors to submit their most
recent standardize test scores and transcripts. These pieces of papers were not utilized as it was
intended this year. Ideally the instructors would be given this information ahead of time to see
where their groups of students are in terms of their academics.
The sheet that the students were responsible for submitting prior to the start of the program
was the terms of participation form. This paper was a check list of all the things they were
agreeing to; it also included a section where a parents/supporter gave permission for the
students to participate in this program.
All other materials or forms that were important were given during the family orientation and
during the first day of the program. This included the emergency contact card, the parent
agreement form and the photo release form.
Classrooms
Preparing for the program also includes preparing for the instructors. When reserving
classrooms or auditoriums, it is good to know ahead of time the preferences of the instructors.
Some instructors preferred lecture style classrooms others would request a science lab.
Whatever the case maybe, campus reservations needs a couple of months in advance if large or
popular venues need to be reserved; places like computer labs, smart classrooms, the east
quad, Damen mpr, or auditoriums. Reservations can be made through 25live or directly with
the department- if they do not show up in the reservation website.
39
CTA passes
The SEaL program has always provided public transportation passes to its students. One issue which was
come across this year was ordering one week passes. The city of Chicago has changed it form of
payment for transportation. CTA is now using the Ventra pass system. They do not sell seven day passes
at retail stores nor at the train stations. Ordering in bulk directly from the company was difficult. Loyola
needs 7-10 business days to create a check, soon after the check is mailed; it takes over a month for the
check to be processed and 7-10 business days to receive them. Overall, this process took too long and
we had to buy over 100 reloadable passes all throughout the city. Most retailers only sell a couple at a
time so figuring out who has the most and who would sell most of their stock was a tough task.
Schedules
The goal of this summer was to enrich the students with knowledge and experience. When creating
their daily schedules, we kept in mind what we wanted them to take away from this summer. The
students took classes in different classroom and different buildings. This allowed for them to explore the
campus and to see what it feels like to walk around a college campus. Below is a sample of the daily
schedule for two of our cohorts:
40
Instructors
During the preparations for the program, the instructors were asked to submit a syllabus. This
was to gain a better understanding on what the classes would consist of. At this time we asked
them to create a supply list. However, because the SEaL program does not have set
funds/budget, this supply list was more of a wish list. We would only get what we could and
what was essential; for example, ACT books, novels, pencils, paper, markers, etc. The
instructors were also asked to send in any printing materials that they would need. They were
informed not to purchase anything on their own because the reimbursement process was
difficult.
Emails with program updates were sent a couple weeks prior to the start of the program.
Instructors were asked to complete the mandated reporter exam online and to submit their
New Hire Paper work soon after the instructor orientation. Aside for that, most of these emails
contained information about class rosters and important dates.
Identification Badges
The increase in students admits made us realize that the students safety was a priority. We decided to
make Summer Scholars Identification Badges. The IDs were created after the family orientation because
we had a better idea of who would be attending the program. The IDs consisted of the students first and
last name, grade, and the Pre-College Summer Scholars logo. We handed these out during the first day
check-in. The students had to wear the badge with the Loyola lanyards provided on a daily basis.
Ordering Swag
Ordering ahead of time can be difficult if we do not want to purchase a surplus of materials. As stable
program, we are able to make a rough estimate on how many students will apply for the summer
scholars program year after year. This year we made a large order for the entire program. We ordered tshirts in a variety of sizes. The t-shirts were designed by one of the graduate program coordinators and
scheduled to arrive two weeks prior to the beginning of the summer. Lunch wristbands were ordered for
the SEaL students in order to make lunch a Damen dining an easier process. More lanyards were not
ordered because a large order was made from the previous years. New ramblers, pad folios, and water
bottles were ordered for this year with the new logo, enough for all five programs for this year and the
next.
Family Orientation
Ordering ahead of time can be difficult if we do not want to purchase a surplus of materials. As stable
program, we are able to make a rough estimate on how many students will apply for the summer
scholars program year after year. This year we made a large order for the entire program. We ordered tshirts in a variety of sizes. The t-shirts were designed by one of the graduate program coordinators and
scheduled to arrive two weeks prior to the beginning of the summer. Lunch wristbands were ordered for
the SEaL students in order to make lunch a Damen dining an easier process. More lanyards were not
ordered because a large order was made from the previous years. New ramblers, pad folios, and water
bottles were ordered for this year with the new logo, enough for all five programs for this year and the
next.
Day one
The first day of the SEaL program was designed to get the students familiar with their peers,
their instructors, their college coaches and the campus. The students were explained in detail
what the mission of the program was. They participated in various ice-breakers and team
building activities which were led by the college coaches. They met with their instructors for
45min, took the post assessment and took a tour of the campus. On this day they were also
give their swag, their IDs, and their Ventra passes. Below is the letter that was sent out
electronically to all the students who had applied.
N:\FirstYearExperience\FirstYearExperience\FYE\Pre-College Summer Scholars\SEaL\2014\Pre-program\SEaL
Orientation.doc
41
42
Program Interactions
This section describes the program interactions amongst the students and staff and how they achieved
the various intended learning outcomes.
Classes
Each SEaL student took two academic courses and one college prep course. These courses were created
for the SEaL Summer Scholars students only. The academic and college prep courses were taught by
current high school teachers or college professors. The students had two 90 min academic periods, one
in the morning and one in the afternoon and one 90 min college prep period. The instructors taught
their classes during their scheduled periods while the college coaches were there to assist in any way.
With classroom liaison hours to complete, the college coaches spent a lot of time with their SEaL
cohorts. Throughout the three weeks, the students attended LifeSkills workshops two times a week and
Reflection period three times a week. They had an Activity period after lunch every day where they had
the chance to do get some exercise.
43
College Coach Programs

Throughout the entire program, the College Coaches prepared various evening programs which were
offered to the SEaL students. The programs were held at the first floor lounge of Messina Hall. The
programs were initially proposed to the Graduate Program Coordinators and executed by the College
Coaches. The majority of the programs were designed to be fun, interactive, engaging and educational;
the topics varied day to day. The students were given a schedule off all the programs during their first
day of classes. They were permitted to stay at the Hall after scheduled program hours if they turned in a
parent agreement form. The students were asked to head home by 8:00PM because of Chicago Curfew
hours and their safety. If students were going to get picked up, they would make their own
arrangements and notify a college coach. For the safety of all the residential students and to respect
their space; SEaL students were not allowed on the residential floors unless the evening program
required it. A sample of the calendar and the parent agreement form can be found below:
44
Assessment Summary
Measurement Tools
In order to measure student growth and achievement of the program outcomes, the following
measurement tools were incorporated into the Summer Scholars program.
Pre-Assessment Survey
Prior to participation in the program, students took a survey consisting of four questions, two of which
were general and two of which were grade-specific. The grade-specific questions were regarding their
skill level in the various areas that they would be studying.
Post-Assessment Survey
On the final day of the program, the students took another survey that included questions about their
activity periods, college coaches, Reflection/LifeSkills, and classes. These questions were aimed at
assessing how prepared the students felt to enter college after the program was over. The survey
included multiple choice and short answer responses. They were also asked various questions regarding
general program feedback.
Student Demographics
The racial identities of the students were as follows: 28 African American, 3 American Indian or Alaska
Native, 6 Asian/Pacific Islander, 53 Latino/Hispanic, 2 White/Caucasian, and 7 identified under other.
The students were all commuters and are from various schools in Chicago.
45
School Demographics
A chart of high schools they attend are listed below. Of the 135 participants, there were 19 freshmen,
25 sophomores, 49 juniors, and 40 seniors. These numbers do not accurately reflect the demographics
of all the students, since not all of them completed the post assessment.
High School (65 respondents)
Percentage
High School
Percentage
Curie- 4
6%
Oak Lawn- 1
1.5%
UNO Garcia- 2
3%
Kelly- 4
6%
Noble College Prep- 6
9%
Lincoln Park- 1
1.5%
Ace Technical Charter- 1
1.5%
Westinghouse- 1
1.5%
Southland College Prep- 2
3%
Northside College Prep- 1
1.5%
Gwendolyn Brooks- 7
10.8%
Amundsen- 1
1.5%
Percy L Julian- 1
1.5%
Evanston Township- 1
1.5%
Senn- 5
7.7%
Young Womens Leadership
1.5%
Carl Schurz- 2
3%
Oak Park- 1
1.5%
Cristo Rey- 4
6%
Northtown Academy- 3
4.6%
Illiana Christian- 1
1.5%
Roosevelt High School- 1
1.5%
CMSA- 5
7.7%
Rauner College Prep- 1
1.5%
Von Steuben- 2
3%
Riverside Brookfield- 1
1.5%
Lane Tech- 1
1.5%
Hinsdale Central- 1
1.5%
Whitney Young- 1
1.5%
Morton East- 1
1.5%
Providence St. Mel- 1
1.5%
46
Pre-Assessment Data
On the orientation day (July 24) of the program, students took a 4-item questionnaire. The assessment
sought to determine where students were at developmentally and in their learning prior to participating
in the program. 124 students completed the questionnaire. Of the 124 students, there were 40 rising
seniors, 40 rising juniors, 27 rising sophomores, and 17 rising freshmen. Around 20% of the students
who responded were returners from prior years.
Skills
Freshmen
Almost three-fifths of freshmen felt that their reading comprehension skills were good to excellent;
whereas around two-fifths felt that their skills were ok/average. Only one student felt that their ability
in this area was below average. Around 45% of the students felt that their writing composition skills
were good to excellent. Another half of the students felt that their skills were ok/average. Only one
student thought that they were below average in this area.
Sophomores
Almost three-fifths of the sophomores felt that their reading comprehension skills were good to
excellent; whereas around two-fifths felt that their skills were ok/average. Only one student felt that
47
their ability in this area was below average. Results for the science lab class were similar, except there
were no students who felt that their skills were below average in this area.
Juniors
Around a quarter of the students felt that their logic and problem solving skills were good/above
average. A majority of students (71%) were ok/average, and another 5% felt like their abilities in this
area were below average. The results for the Math ACT section were similar.
Seniors
Three-fifths of the senior class rated their research writing ability as average, and one third of the
students rated their ability as good to above average. Two other students rated themselves as below
average and no students rated themselves as poor in the subject of research writing.
Around one-third of the students rated themselves as above average to excellent in their personal
48
writing abilities. Nearly half of the students felt that they were average at creative writing, and another
15% of students felt that they were below average.
49
Post-Assessment Data
On the final day of the program, students were given a post-assessment to fill out. The assessment
asked questions regarding their classes, activity periods, and LifeSkills/Reflection periods. Of the 135
students who were in the program, 92 students took the post-assessment. 34 students were rising
seniors, 21 were rising juniors, 19 were rising sophomores, and 18 were rising freshmen.
Plans after High School

All freshmen, juniors, and seniors plan on attending college or university after high school. Of the
sophomores, over four-fifths plan to go to college/university. Additionally, one student said that they
would join the military, and two had other unspecified plans. In addition to attending a college
preparation class, the students were constantly exposed to the idea of attending college through the
activities and reflections that were led by the college coaches.
Classes
Reading Comprehension/Writing Composition - Freshmen
The freshmen class focused on developing their reading skills and confidence. The students read Perks of
being a Wallflower by Stephen Chbosky during their three weeks. The results for both the reading
comprehension and writing comprehension classes for the freshmen were similar. Overall, most of the
freshmen felt confident in their reading comprehension and writing composition skills. Only a few of
them still felt a little concerned about their ability to perform well in the classes.
Science Lab/Science Reasoning Sophomores

The science lab class consisted of experiments, projects, and presentations that focused mainly on
environmental issues. Overall, there was less lecturing and more hands-on work that the students
completed. The science reasoning class spent a lot of focus on taking and improving their scores on the
ACT practice tests. This class also included lectures, projects, and group presentations.
Around three-fourths of the students felt confident in their science lab skills they accumulated
over the course of the program. Only a few students said they felt only a little confident. There were no
students who felt a complete lack of confidence in their lab skills. Around 70% of students felt confident
in their gained science reasoning skills. It is important to note that both groups, Sophomore A and B,
had the same professors for both their science lab and science reasoning classes.
50
Logic and Problem Solving/ACT Math Juniors

The Logic/Problem Solving class focused less on the application and more on the skills needed to
succeed on the ACT test. The students found this class to be less engaging than the ACT Math Prep class
but they felt that they learned a lot in the class.
62% students said that they were confident in their logic/problem solving skills. One-fifth of the
students said they were only a little confident, and one-fifth said they were very confident. Similarly, the
majority of the students 57%, said they were confident in their ACT Math skills, 19% said they were only
a little confident, and 14% said they were very confident.
Research Writing/Creative Writing Seniors

The research class that the seniors took consisted of brainstorming, researching, and writing a four page
research paper that they would be able to use for their upcoming class. The creative writing class
consisted of students brainstorming personal statement ideas about their own lives followed by
continuously writing their essays.
Over three-fourths of the seniors felt confident or very confident in their research writing skills
by the end of the class. While almost one-fourth of the students said that they feel only a little confident
with their research writing skills after they finished SEaL. Almost all of the students 91% reported high
levels confidence in their creative writing skills for personal statements. While nearly one-tenth of the
students said that they felt only a little confident in their personal statement writing skills.
51
Challenging Assignments
Freshmen
Of the challenging assignments listed in these responses, general reading and writing assignments
seemed to be the most challenging for the freshmen. More specifically, analyzing text and writing
persuasive letters was difficult for some students. Around one-third of the students did not find any
challenges with completing their classroom assignments.
Sophomores
Of the thirteen students who responded, nearly forty percent felt that the ACT practice tests were
challenging. The science projects followed closely behind with 30% of students who found them
challenging. About twenty-three percent of the students thought the science reasoning assignments
were difficult. Only one student felt that the science experiments were challenging.
Juniors
The Problem Solving and Logic portion of the curriculum was definitely the most challenging for the
junior students, but the majority of the students said that these challenges they faced made them think
outside the box and learn new things.
Seniors
One-third of the students said that the research papers were the most challenging. These students
explained that the actual research aspect and length of the four page essay was the most challenging
part about it. However, most of them stated that by the end of their time here, their challenges were
overcome with help of the instructors and college coaches. Over half of the students said that the
personal essay was the most challenging assignment.
Establishing Friendship
In general, the students felt that the Summer Scholars program is a great environment to build new
friendships and learn how to interact well with new people. The vast majority of the freshmen,
sophomores, juniors, and seniors said that they had established new friendships. There were few
students who felt that they hadnt established new friendships in the program. The students spent the
entire day with their cohort, including all activity and reflection periods and lunch. This allowed the
students to form close connections with each person in their individual cohorts.
Personal Gains
From the responses from the students, they all gained something important from participating in the
SEaL program. Many gained the skills that are applicable to both high school and college. Several of the
students said that the college application process was no longer intimidating, and they felt prepared to
enter college. The students also gained new friendships throughout the program, as well as confidence
and leadership skills. The exact number of students who felt a personal gain during the SEaL program is
presented below, followed by some of their comments.
New Friends - 18
College Experience/Skills - 34
Life Skills/Experience - 30
Confidence - 20
Improved Academic Skills (Math, Science, Reading, Writing) 15
Leadership 10
52
Student Comments
The SEaL program gave me the key to my future, it made me think of what I really want to be in my
life and how I have the chance to accomplish what I want if I stay dedicated to my goal. I learned
that we are all here for a reason and that we are all worth it.
I have gained new friends and more knowledge about math, and what to expect when going into
college.
I gained a lot of things from participating in the SEaL program. I learned more about college and
what it takes to make it there. I also learned some things that I can use in my daily life, like managing
stress.
I gained confidence because it actually feels like you are in college.
The most important thing I learned was to NEVER GIVE UP!
I have gained more support. I know that people are supporting my idea to go to college, and that
really helps me work harder.
Activity Periods
The activity periods were 50 minute long periods that began directly after lunch every day. The
activities were broad and ranged from physically active to inactive and were planned by the College
Coaches. The grade levels were separated during these periods every day except for every Thursday,
which was designated as the large-scale activity day.
Favorite Activities
In this portion of the post assessment the students were asked to give a short answer. They were asked
to indicate what their favorite activity/program they liked during their Activity Period. Their responses
were tallied and from the 110 responses, the favorite activity of the SEAL students was Hunger Games
activity. Other student favorites included the Field Day, going to the beach, playing capture the
flag/capture the Rambler, and Mafia. From the information we received, activities that were the least
favorites included Wii, Vision Boards, and Throwing Your Fears Away.
53
Overall Experience
The information we collected from this question encompassed responses laying on a 1-10 scale.
Students that responded with a 1 rated the overall activity experience as inadequate while those that
responded with a 10 felt the experience was exemplary. The 92 responses we collected ranged from 410. Nearly 70% of the student responses rated their activity
period experience as a 9 or 10. Conversely, less than 9% of the
students deemed their activity period experience a 4, 5, 6, or
7.
54
College Prep
The students all took a college preparation course that focused on different topics, including finding
their talents, the college admissions process, and financial planning for college. The students also had a
talent show in the second week of the program. The majority of students, 52%, hadnt taken a college
preparation class before this program. Around 40% had.
The students were asked whether or not they felt more prepared to enter high school (freshman) or
college (sophomores). The seniors were not polled. 84% of freshmen felt more prepared to enter high
school. Around 40% of the sophomores felt much more prepared to go onto college, whereas around
60% felt only somewhat prepared. 55% of the juniors felt a lot more prepared to attend college and
45% felt somewhat prepared.
Perceived Differences between High School and College
In the assessment, there was a question regarding what the students felt were the differences between
high school and college. The top two responses to this short answer question were independence and
freedom. They also mentioned the difference between the professor and student relationship,
classroom and campus size, and the workload. The idea that college is different than high school was
reinforced in their college prep class and by the college coaches.
Outcomes
The SEaL daily schedule allowed the students to see
various buildings and utilize various resources on
campus. The students were also able to easily find their
way to classes. 96% of the students felt confident in
navigating a college campus, and 2% were not
confident.
Around three-fourths of the students also felt confident
completing the Free Application for Federal Student Aid
(FAFSA.) Only around one-fifth felt that they did not
have the knowledge to complete it.
The students are confident enough in asking for quality
recommendation letters from mentors, teachers, and
employers. 96% of the students felt that they could
obtain these letters. Only 4% were not confident
55
LifeSkills and Reflection Periods

LifeSkills and Reflection Periods occurred at different times of the day for the grade levels. Each period
was an hour and a half long and consisted of mostly discussions with the students. The LifeSkills period
included workshops on college goals, financial planning, stress management, wellness, and leadership.
Reflection periods werent as physically active as the activity periods and focused on serious subject
matter. These periods focused on forming trusting relationships between the college coaches and the
students. Almost three-fourths (78%) of the students said that these sessions adequate length, while
only a small number of students (22%) said that they were too long.
LifeSkills Workshops
One-third of the students said that Leadership was their favorite and another third said that stress
management was their favorite. Wellness was also favored in the previous question, and 20% of the
students enjoyed it. College Goals (11%) and Financial Planning (4%) were last, however, they tend to be
the more serious issues. For next year, we should consider making sure each presentation is tailored to
the age group of the audience
Below is a chart describing what topics the students felt they have learned the most about during these
periods:
Meals
Breakfast
Breakfast usually consisted of granola/cereal bars, pop tarts, and fruit and
was served every day from 8-9 AM. According to the survey, the majority
of students were not satisfied with the breakfast served during the SEaL
program. Based on a 1-5 scale, 71% of the students ranked the breakfast
a 3 or under.
Many would like to see more drinks and healthier options present in the
morning. They also expressed the need for a rotation in the food.
Lunch
The students ate at Damen Dining Hall from 12-12:50 every day. All of the students in the program
rated lunch at Damen Dining Hall a 3 or above.
Overall, the students really liked the food from Damen. The lunch
schedule would sometimes run into conflicts due the other programs, like
Envision, that ate in the dining hall during the same time. Many of the
students wanted a longer lunch period to have more down-time. Some of
the students suggested that Damen Dining should make breakfast.
56
57
Student Testimony
Activities/Programs
Students were provided with activities on a daily basis that were planned for SEaL. The students were
overall satisfied with the activity periods. The College Coaches dedicated to SEaL spent time planning
these activities on a daily basis.
The students rarely attended the programs that the college coaches organized for all of the
Summer Scholars in the evenings. This was due to various conflicts. Many of the students had a long
commute home after classes and were therefore unable to attend. The students expressed that they
would also like to see more programs and activities that were off of the Loyola campus. They wanted to
have the opportunity to take field trips in their classes as well.
My favorite activity was throwing a rock into the lake which had a fear that each
of us had written on it. This activity was my favorite because it really made me
think about life, and it helped me reassure myself that I can be successful if I really
want to be.
- Yesenia Carrera, Junior
My favorite activity was beach day. It was amazing because many people would
think we would be stuck in class all day, but really we got be able to explore
campus. I can honestly say it made me feel like a college student.
- Neomy Gutierrez, Junior
Staff Interactions
The students were asked to rate how approachable the college coaches were on a one to five scale.
99% of respondents felt that the college coaches were a three or above, meaning that they were
approachable. Student feedback was mixed in the short answer section of the survey. Many students
mentioned an inconsistency in the staff members in interactions with students. Some students stated
that the staff was very approachable while others felt that staff members were rude or crossed
boundaries.
I enjoyed the college coaches a lot more last year. They seemed to care more
about you and wanted to help you with any problems you had in school or at
home. I could trust them more. The activities could have been more fun; the ones
we did this year didnt really bring us close together like last year. I didnt even
know everybody in the senior class this year, and last year everybody was ONE big
family.
- Mercedes Cerda, Senior
Anonymous
The students also rated the college coaches on how well they handled and planned the activity periods.
On a one to ten scale, half of the students gave the college coaches a 10, meaning that the activity
periods were very well led. The other half of the students responded with a 4-9, with the majority
58
responding with an 8 or 9.
the college coaches were all so helpful and nice. Each and every one of them
was so dedicated to this program. All my questions got answered, and for me this
is a good thing. They helped me experience what college is like.
- Itzahmara Pantoja, Senior
Classes/Coursework
The SEaL students enjoyed their professors and classes. Several of the students thought that their
classes and professors helped them to achieve the college experience. The College Coaches also acted
as TA's and mentors in the classes. This provided the support that some of the students needed.
Students generally felt prepared for the majority of the classes but still felt challenged.
The class assignments went really well and I didn't have a problem because of the
professor's help. I mean it was difficult for me to answer high-level thinking
questions in reading but I was supported by the professor. I changed by thinking
of new meanings in books and movies.
- Zoely Rodriguez, Freshman
Taking the ACT Math portion was the most challenging part of the program.
Having to solve and answer several different mat questions is a challenge for me
because I am not a big fan of math. I have changed because I learned quick and
easy methods to solving the problems on the ACT in a decent amount of time.
- Jaleya Bell, Junior
59
Recommendations
This section has recommendations of ways to modify next years program. These recommendations will
allow the program to operate better overall. The responses are mainly regarding classroom selections,
modifications to the Activity, LifeSkills, and Reflection periods, and modifying the program times.
LifeSkills/Reflection
The LifeSkills and Reflection periods could be half an hour shorter. It was difficult for the college
coaches to plan workshops/reflections that would keep the attention of the students for an hour and a
half. Additionally, the people who sign up to lead these workshops and reflections should be good at
public speaking and knowledgeable on the topics that they are presenting. Having public speakers from
Loyola, like from the Wellness Center, would be beneficial in LifeSkills.
Activity Periods
The activity periods should be lengthened. Each period was 50 minutes, but would get cut down to 40
minutes to include travel time. The students were often late to class because of the short amount of
time these periods were. The students also gave a lot of positive feedback on these periods. It is
important that each activity period is well planned out but not overly rigid. Taking suggestions for
different activities from students was also an important part of keeping them engaged during this time.
Assessment
The pre-assessment should be taken after the class sizes are set in order to get responses from all of the
students who are participating in the program. There should also be more questions included in the
pre-assessment that correlate with the questions asked in the post-assessment. It is easier to analyze
the data outcomes by doing this.
Class Attendance
Students need to be held accountable for attending class. This year, many students skipped class often
or were excused from class for several days. If a student is falling behind in their work due to this, the
student should be held accountable. While in class, it was also important that each student be mentally
present. Many students had issues concentrating. It was difficult for the college coaches to uphold
these standards.
Programs
Programs should be scheduled earlier in the evening (around 5:30-6 pm). This summer, most programs
started at 7 pm, which was too late for the SEaL students. These programs should be more inclusive of
the SEaL students in general as there was a disconnect between SEaL and the residential students.
Classrooms
Classroom sizes were often not large enough to accommodate all students, especially during reflection
and LifeSkills periods. Access to larger computer labs is also a must. There were issues coordinating
these spaces. Students also requested that they have their classes in different buildings for each class.
They wanted to see more of campus in order to get a better feel of what college is like.
60
Supplies
The professors must either contact a college coach or email loyolasummerscholars@gmail.com 72 hours
in advance in order to get supplies that were not already purchased at the beginning of the summer.
For printed materials, professors must contact the staff at least 24 hours in advance.
Modified Training
Many of the college coaches felt that training could have been condensed this summer. One of the
suggested modifications was more training from SDMA. The combined training with the orientation
leader staff was also not very helpful for the college coaches, as most of the material was directed at the
orientation leaders.
Communication
There were problems with miscommunication among the leadership team and the college coaches. The
college coaches also had a fair amount of miscommunication among themselves. It is important that
the leadership team is clear with their expectations of the college coaches, and it is equally important
that the college coaches are communicating their needs to the leadership team.
Final Day
Have a large ticket event planned for the students in addition to the family picnic/graduation. One
suggestion would be six-flags.
61
62
Appendix B: Map of Partnering Schools
Summer Enrichment at Loyola

Partner Schools 2014
Chicago Math and Science Academy

Senn High School
7212 N. Clark St., Chicago IL 60626
5900 N Glenwood Ave Chicago, IL 60660
Noble Street College Prep

1010 N Noble St Chicago, IL 60642
Christ the King Jesuit High School

5088 W Jackson Blvd Chicago, IL 60644
Cristo Rey Jesuit High School

1852 W 22nd Pl Chicago, IL 60608
Kelly High School

4136 S California Ave Chicago, IL 60632
Curie High School

4959 S Archer Ave Chicago, IL 60632
Gwendolyn Brooks College Prep

250 E 111th St Chicago, IL 60628
63
Appendix C: Logic Model
Program: SEaL Summer Program

Outputs
Inputs
Stakeholders
o Parents
o Program
Director & higher
level
administrators
Personnel
o Counselors
o Teachers
o College
Coaches
o Graduate
Program
Coordinators
University
Recourses
o Technology
o Rooms
o Equipment
o Materials
o Incentives
Activities
Activity Period
o Dodge Ball
o Scavenger
Hunt
o Hunger
Games
o Beach Day
o Sports
Meals
o Continental
Breakfast
o Dining Hall
Academics
Program
o College Prep
o Study Skills
Workshops
o Reading,
Science,
Math, Writing
Res-life
afterhours
programing
Flyers
Trainings
Participation
Students
o 100-200
students
o High School
o Chicago Area
o Low income
o People of
Color
o Midperforming
Short
Developing a habit
of punctuality
Completing the
assigned work
Develop
professional
behavior
o Public speaking
Building
Relationships
o Communicatin
g with peers &
college coaches
& instructors
o People Skills
Personal
development
o Independent
Develop
Interpersonal skills
o engaging in
activities with
others
Navigating
Resources
o Knowing how
to use the
computer/print
er/internet
Outcomes -- Impact
Medium
Safety
Awareness
Gain an
understanding of
public transit,
safe routes
throughout the
city and
navigating your
way around.
Community
Building
Sharing
Knowledge with
peers/family
o Bring what
they learned
in the
program to
their families
and
communities
. Educating
them on
various
topics; i.e
college,
health,
financial
fitness.
Long
Enhance Social
Skills
o Listening
Foster Cultural
Competencies
o Being able to
socialize with
people from
various
background
while being
confident in
your identity
and
ability/comp
etence.
Cultivate
Networking
Skills
o Getting to
know college
coaches,
instructors
and being
comfortable
asking
questions
o Resourceful
and selfreliant
Improve
Academic Skills
Create College
Bound outlook
64
Appendix D1:
Pre-survey e-mail Invitation
Dear SEaL student,

Thank you for your participation in the SEaL program at Loyola University Chicago. We are
writing to invite you to take a short survey to help us understand the impact that the SEaL
program has had on your social skills development. The survey is designed to assess your social
skills, and we would like to hear about your experience. Here is a link to the
survey: https://docs.google.com/forms/d/1G6p7wjYKalC9D8CeSWdXRIbmYT8M3J082v98i1DF7M/viewform
It should take about 10-15 minutes to complete and includes a space for additional
comments. We appreciate your honest responses to the questions in the survey. This research
has been approved by the Office of First Year Experience at Loyola University Chicago.
All responses are completely confidential and will be destroyed after we have gathered all of the
data that we need. Your choice of whether or not you participate will not have any impact on
you, or your eligibility to continue in the SEaL program. If you have any additional questions,
please contact us at atopalova@luc.edu and/or lroman3@luc.edu .
Your participation in this survey is voluntary. You may withdraw your consent and discontinue
participation from the survey at any time. Your refusal to participate will not result in any
penalty.

Program Evaluators
65
Appendix D2:
Post-survey e-mail Invitation
Dear SEaL student,

writing to invite you to take a short survey to help us understand the impact that the SEaL
program has had on your social skills development. The survey is designed to assess your social
skills, and we would like to hear about your experience. Here is a link to the
survey: https://docs.google.com/forms/d/1G6p7wjYKalC9D8CeSWdXRIbmYT8M3J082v98i1DF7M/viewform
It should take about 10-15 minutes to complete and includes a space for additional
comments. We appreciate your honest responses to the questions in the survey. The Office of
First Year Experience at Loyola University Chicago has approved this research.
data that we need. Your choice of whether or not you participate will not have any impact on
you, or your eligibility to continue in the SEaL program. If you have any additional questions,
please contact us at atopalova@luc.edu and/or lroman3@luc.edu .
Your participation in this survey is voluntary. You may withdraw your consent and discontinue
participation from the survey at any time. Your refusal to participate will not result in any
penalty.

Program Evaluators
66
Appendix E1:
Pre-survey Permission Form
Dear SEaL parent and student,

writing to invite your child to take a short pre-survey during program hours. The survey is
designed to assess their social skills development.
The survey should take about 10-15 minutes to complete and it includes a space for additional
comments. We appreciate students honest responses to the questions in the survey. This
research has been approved by the Office of First Year Experience at Loyola University
Chicago.
data needed. Your choice of whether or not you participate will not have any impact on the
student, or their eligibility to continue in the SEaL program. If you or your child have any
additional questions, please contact us at atopalova@luc.edu and/or lroman3@luc.edu .
By circling a letter below you acknowledge that you and your child have read this letter and
understand that, their participation in this survey is voluntary. If at any point during the survey
you or your child wishes to opt out of the survey, they my do so. Your refusal to participate will
not result in any penalty.
A.
B.
I give my consent to participate: My child may take the survey if he/she wishes to
I do not consent to participate: My child will not take the survey
____________________________
Parent/ Legal Guardian Signature
_____________
Date
____________________________
Student Signature
_____________
Date
67
Appendix E2:
Post-survey Permission Form
Dear SEaL parent and student,

writing to invite your child to take a short post-survey during program hours. The survey is
designed to assess their social skills development to help us understand the impact that the SEaL
program has had.
The survey should take about 10-15 minutes to complete and it includes a space for additional
comments. We appreciate students honest responses to the questions in the survey. The Office
of First Year Experience at Loyola University Chicago has approved this research.
data needed. Your choice of whether or not you participate will not have any impact on the
student, or their eligibility to continue in the SEaL program. If you or your child has any
additional questions, please contact us at atopalova@luc.edu and/or lroman3@luc.edu.
By circling a letter below you acknowledge that you and your child have read this letter and
understand that, their participation in this survey is voluntary. If at any point during the survey
you or your child wishes to opt out of the survey, they my do so. Your refusal to participate will
not result in any penalty.
C.
D.
I give my consent to participate: My child may take the survey if he/she wishes to
I do not consent to participate: My child will not take the survey
____________________________
Parent/ Legal Guardian Signature
_____________
Date
____________________________
Student Signature
_____________
Date

Appendix F: Summer 2014 SEaL Assessment
68
69
70
71
72
73
74
75
76
77
Appendix G: Survey Construct Map
Summer Enrichment at Loyola (SEaL)

Survey Construct Map and Item Matrix
Pre-Survey and Post-Survey
Logic Model
Component
SQ
#
Question
Answer
Type
Answer Options
General
Information
Student ID Number
Text box
Not Applicable
Multiple
Choice
Text box
Not Applicable
Please select the grade you will be in the upcoming school

year
What is your personal email?
What High School do you go to?
Text box
Not Applicable
What is your gender identity?
Check List
Not Applicable
What is your racial identity?
Check List
Not Applicable
Check box/
Likert Scale
Check box/
Likert Scale
Check box/
Likert Scale
Check box/
Likert Scale
Check box/
Likert Scale
5-point
Strongly Disagree to Strongly Agree
5-point
5-point
5-point
5-point
Not Applicable
Communicating and Networking

Short Term
Building
Relationship
Develop
professional
behavior
Long Term
Enhance
Social Skills
I listen to others
I can ask for help
I am polite
10
I feel comfortable introducing myself to adults
11
I enjoy contributing to discussions
Cultivate
Networking
Skills
12
I offer to help my peers
13
I am comfortable asking my college counselor about anything
14
I can obtain recommendation letters from mentors, teachers,

or employers
78
Check box/
Likert Scale
Check box/
Likert Scale
Check box/
Likert Scale
5-point
5-point
5-point
Check box/
Likert Scale
Check box/
Likert Scale
Multiple
Choice
Check box/
Likert Scale
Check box/
Likert Scale
Check box/
Likert Scale
Check box/
Likert Scale
Check box/
Likert Scale
5-point
5-point
3-point
0-1, 2-3, 4-5 times a week
5-point
5-point
5-point
5-point
5-point
Check box/
Likert Scale
Check box/
Likert Scale
Check box/
Likert Scale
5-point
5-point
5-point
Friendship-making Skills
Short Term
Building
Relationship
s
Develop
Interpersona
l skills
Medium Term
Sharing
Knowledge
with
peers/family
1
5
1
6
1
7
1
8
1
9
2
0
2
1
2
2
I can begin a conversation
2
3
2
4
2
5
I enjoy working with a partner in class assignments
I feel comfortable introducing myself to new peers

Outside of the classroom, how often do you engage in extracurricular team building activities?
I partake in games and activities
I can ask my peers for favors
I feel comfortable offering help to a classmate
I am happy to share
I apologize when necessary
In the Classroom
Short Term
Develop
Interpersona
l skills
Medium Term
I like group projects

I often engage conversation
Sharing
Knowledge
with
peers/family
Long term
Enhance
Social Skills
2
6
2
7
2
8
2
9
I get nervous in a group setting

I enjoy giving presentations to others
I learn from my peers
Attending college is my goal
79
Check box/
Likert Scale
Check box/
Likert Scale
Check box/
Likert Scale
Check box/
Likert Scale
5-point
5-point
5-point
5-point
Check box/
Likert Scale
Check box/
Likert Scale
Check box/
Likert Scale
Check box/
Likert Scale
Check box/
Likert Scale
Check box/
Likert Scale
5-point
5-point
5-point
5-point
5-point
5-point
Cultural Competencies
Short Term
Building
Relationship
s
Long Term
Foster
Cultural
Competencie
s
3
0
3
1
3
2
3
3
3
4
3
5
I accept cultural differences

I treat all of my peers with respect for their culture
I do not inflict my beliefs onto my peers
I believe that it is acceptable to use a language other than
English in the U.S.
I dislike when others have insensitive comments or behaviors
I respect non-traditional family structures

Appendix H: Survey Questionnaire- Evaluation Plan
80
81
82
83
84
85
Appendix I:
Focus Group Interview Questions
1. What has your experience been at the SEaL program?

2. Based on your experience, what projects and/or activities helped you develop
your public speaking skills?
a) In what way(s) has the SEaL program given you the opportunity to engage
in activities with others?
3. If this applies to you, can you describe your experience in making friendships
with your peers?
a) Do you think there might be a way for the SEaL program to enhance and
support your experience in making friends?
4. If, during your time at SEaL, you had the opportunity to interact with a peer who
identifies with a different ethnic background than your own, can you please
describe your interaction with them.
86
Appendix J:
Focus Group Protocol
Introduction
Hello everyone and welcome to todays focus group discussion. My name is
and I
along with my assistant moderator,
,will be leading todays discussion.
will
be taking notes throughout our discussion. Over the next hour, together, we will be discussing
your experiences at the SEaL program this summer. The SEaL program is being assessed to
determine if it is meeting its objectives. This is part of the reason why you are all here today.
We would like to hear about your experiences in the program and your thoughts on the program
as a whole. Your candid feedback is greatly appreciated and will be utilized to make
improvements to the program, if need be.
I have questions that I will be asking over the next hour. These questions will help us navigate
the conversation, and please keep in mind that this is an informal discussion. You are welcome
to respond to as many or as few of the questions as you wish. There is no need for you to raise
your hand. We simply ask that you are all respectful of each other and let each other finish
talking before someone else begins to contribute to the conversation. Per the focus group
consent form you and your legal guardians signed, all of the information you shared will be kept
confidential, and the audio tape of the discussion will be erased once it has been transcribed.
Any questions?
I will read over the consent form which you and your legal guardians signed at the beginning of
the program. (Read consent form from Appendix ??) If at this point anyone feels uncomfortable
and would like to withdraw their participation from this focus group, you may do so now.
Begin audio recording and state the reason for this focus group, name of moderator and
assistant moderator, the location, the date and the time.
Focus Group Questions
1. Lets all introduce ourselves my going around the room and stating our name, year in school,
and what class you were assigned to for the SEaL program.
2. Now, can you all share what your experience has been at the SEaL program?
(Overarching/general question)
This is a general question and all answers are welcome.
3. Based on your experience, what projects and/or group activities helped you develop your
public speaking skills? (Develop professional behavior- public speaking)
Wait for response. If it takes them a long time to answer, ask: For example what
were some of the projects or activities that made you feel confident and think, I
can do this work. I am good at this.
3a. As a follow up to the previous question, in what way(s) has the SEaL program given you the
opportunity to engage in activities with others? (Building relationships & developing
interpersonal skills)
Meaning, reflect on when and how you were able to connect and engage with
fellow peers.
4. If this applies to you, can you describe your experience in making friendships with your peers?
87
(Relationship building)
Were you able to make friends? What was that like? If you were not able to make
friends, do you mind sharing why you think that occurred?
4a. Do you think there might be a way for the SEaL program to enhance and support your
experience in making friends? (Relationship building & cultivating relationships)
Are there certain projects, activities or outings you think might help you in
making friends? Does anything else come to mind in that respect?
5. If, during your time at SEaL, you had the opportunity to interact with a peer who identifies
with a different ethnic background than your own, can you please describe your interaction with
them. (Foster cultural competencies)
How did you feel about being able to interact with someone who identifies
differently than you? Do you see yourself interacting with someone from a
different ethnic background when you go back to school? Elaborate on your
answer.
Conclusion
It looks like I have no further questions for you all. Before we conclude our discussion, would
anyone like to add anything we did not discuss, or does anyone have any questions.
Thank you all for participating in this focus group. Your participation and question responses are
greatly appreciated. I hope this discussion was insightful and reflective for you. Please know
that your participation in this focus group will help improve the SEaL program, and subsequently
the experience of other students in the program. If further questions arise, you all have our
contact information as a method of communication. Once again, thank you.
88
Appendix K: Focus Group Recruitment Matrix
Freshmen
Sophomores
Juniors
Seniors
1 class = 19 students
2 classes =25 students

total
2 classes= 49 students
total
2 classes= 42 students
total
Focus group = 4
students
Focus group = 4
students from 1 class +
4 students from 2nd
class (8 students total)
Focus group = 4
4 students from 2nd
Focus group = 4
4 students from 2nd
89
Appendix L:
Parental or Legal Guardian Permission Form for Minor

Office of First Year Experience
Title of Project: SEaL Social Skills Program Evaluation
Evaluators(s): Antoaneta Topalova and Laura Roman
Your permission is being sought to have your child participate in a Focus Group. Please read the
following information carefully before you decide whether or not to give your permission.
Purpose of the research: The purpose of this study is to help us determine whether the SEaL
program met its goal to enhance the development of students social skills.
Procedure to be followed: During the focus group, your child will be asked a series of questions
describing their own experience during the program, while being audiotaped. The audiotaping is
for the sole purpose of transcribing their responses, and in no way will be used to examine or test
the intellectual level of your child.
Discomforts/risks: The risks in this study are minimal. There are no foreseeable discomforts or
dangers to either you or your child in this study.
Benefits for participation: There are no direct benefits to your child. The results of this study,
however, will increase our ability to improve the program.
Time duration of participation: Participation in the focus group will not exceed 1 hour.
Statement of confidentiality: All records are kept confidential and will be available only to
professional researchers and staff. Only the evaluators of this program will have access to the
audiotaping(s). It is our priority to keep the students information confidential, and therefore,
while transcribing the audiotaping, pseudonyms will be assigned. Once the evaluators are
finished with the transcriptions, the audiotaping(s) will be discarded. If the results of this study
are published, the data will be presented in-group form and individual subjects will not be
identified.
Voluntary participation: Your childs participation is voluntary. If you feel your child has in
any way been coerced into participation, please inform the program administrator. We also ask
that you read this letter to your child and inform your child that participation is voluntary. At the
time of the focus group, your child will once again be reminded of this by the evaluators.
Termination of participation: If at any point during the study you or your child wishes to
terminate the session, we will do so.
90
Questions regarding the evaluation should be directed to:

Antoaneta Topalova (Speaks English and Bulgarian) atopalova@luc.edu
Laura Roman (Speaks English and Spanish lroman3@luc.edu
Questions or concerns regarding participation in this evaluation should be directed to:
Karladora Chavez (Program Director) kchave1@luc.edu
SIGNING THE FORM BELOW WILL ALLOW YOUR CHILD TO PARTICIPATE IN THE
FOCUS GROUP DURING PROGRAM HOURS WITHOUT YOUR PRESENCE. Please return
by Monday, June 10th. If you do not sign and return this form, the researchers will understand
that you do not wish to allow your child to participate.
Parent or Legal Guardian Signature Box

I, the parent or legal guardian of _______________________________, a minor ______ years
of age, permit his/her participation in a program of research named above and being conducted
by Antoaneta Topalova and Laura Roman.
_________________________________ _____________
Signature of Parent or Guardian
Date
________________________________________________
Please print your name here.
Student Signature Box

I, _______________________________, agree to participate in focus group named above and
understand that my participation is voluntary.
_________________________________ _____________
Signature of Student Date
________________________________________________
Please print your name here.
91
Appendix M: Coding Rubric

Construct Code
Meaning
Theme
ERESP
Emotional Response
Comments related to feelings

towards program experience.
SITE
Site process
Comments related to
geographic location, classroom
setting.
SAFE
Safety
Comments related to a
students safety concerns
whether positive or negative.
ENGAGE
Engagement
Comments related to
experience in peer, faculty and
staff interactions.
PUB SPEAK
Public Speaking
Comments related to students

experience in building public
speaking skills.
SOCLZ
Socializing
Comments related to social

interactions with peers, and/or
during program activities.
ACADEM
Academic
Comments related to academic

experience throughout SEaL.
CULTRL
Cultural Experience/Exposure
Comments related to
exposure/experience with
different cultural groups.

Appendix N: Terms of Participation
92
93
Appendix O: Timeline
2014
Month July
Task
Pilot of pre-survey
Pilot of post- survey
Look at findings of pilot tests &

implement needed changes
Send evaluation proposal to IRB
for review
Send consent forms & all other
invitations to 2015 participants
Email links to pre-survey
Administer pre-survey
Analyze pre-survey data from
2015 cohort
Email post-survey link (same as
pre-survey)
Administer post-survey
Analyze post-survey data
Conduct focus group interviews
Transcribe focus group audio
recordings
Analyze data from focus groups
Prepare final report
Share findings with Karladora
Chavez
Aug
X
X
Sept
Oct
Nov
Dec
2015
Jan
Feb
Mar
Apr
May
June
July
Aug
Sept
X
X
X
X
* The program will be in session late June mid July (Seal Website: http://www.luc.edu/summerscholars/seal/)
X
X
X
X
X
X
X
X
X
X
X
94
Appendix P: Budget
Activity
Survey
Administration
Item
Google Forms
Cost Per Item

$0.00
Quantity
N/A
Total
$0.00
Computer Lab
Pre- Survey
Consent Form
Post- Survey
Consent Form
College Coaches
$0.00
$0.07
4 rooms
140
$0.00
$9.80
$0.07
140
$9.80
$0.00 Provided by
the SEaL program
$0.00 Provided by
the SEaL program
$0.00 Rent from
LUC
$20.00
$0.07
$0.00
T-shirt, Water
Bottles, Pens
Audio Recording
Device
Visa Gift Card
Focus Group
Consent Forms
Room Reservations $0.00
140 of each item
$0.00
$0.00
5
140
$100.00
$9.80
$0.00
Statistical Analysis
Microsoft Word
$0.00
N/A
$0.00
Reporting
Spiral Bound
Reports
$5.00
$5.00
Incentive
Focus Groups
Incentive
Total Cost of Evaluation: $ 134.40
95
Appendix Q: PowerPoint Presentation
SEaL
96
97
98
99
100
101
102
103
104

Evaluation &amp; Assessment Final Project

Загружено:

Сведения о документе

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Evaluation &amp; Assessment Final Project

Загружено:

Авторское право:

Доступные форматы

Summer Enrichment at Loyola (SEaL)

Social Skills Development Evaluation

Antoaneta Topalova and Laura Roman

SEaL EVALUATION PLAN

SEaL Evaluation Plan

SEaL EVALUATION PLAN

SEaL EVALUATION PLAN

APPENDIX F: SUMMER 2014 SEAL ASSESSMENT .................................................................... 68

SEaL EVALUATION PLAN

Evaluation Plan Introduction

SEaL EVALUATION PLAN

cultural competencies, interpersonal skills (i.e. engaging in activities with others),

SEaL EVALUATION PLAN

SEaL EVALUATION PLAN

relationship building dialogues (see Appendix A).

SEaL EVALUATION PLAN

SEaL EVALUATION PLAN

SEaL EVALUATION PLAN

SEaL EVALUATION PLAN

SEaL EVALUATION PLAN

SEaL EVALUATION PLAN

SEaL EVALUATION PLAN

SEaL EVALUATION PLAN

SEaL EVALUATION PLAN

Quantitative Results Presentation

SEaL EVALUATION PLAN

SEaL EVALUATION PLAN

SEaL EVALUATION PLAN

SEaL EVALUATION PLAN

SEaL EVALUATION PLAN

SEaL EVALUATION PLAN

SEaL EVALUATION PLAN

representative of a students authentic response to a question. The way a question is worded

SEaL EVALUATION PLAN

SEaL EVALUATION PLAN

SEaL EVALUATION PLAN

SEaL EVALUATION PLAN

SEaL EVALUATION PLAN

Appendix A: Manual & Student Assessment Reports 2014

Summer Enrichment at Loyola

SEaL EVALUATION PLAN

SEaL EVALUATION PLAN

SEaL EVALUATION PLAN

SEaL EVALUATION PLAN

SEaL EVALUATION PLAN

SEaL EVALUATION PLAN

College Coach Recruitment

Instructor Recruitment (Bridget Wesley can Insert more info)

Counselor Contact Timeline

SEaL EVALUATION PLAN

Nomination instructions (1-2 Days after nomination) Early March

Student Contact Timeline

Nomination Letter (1-2 Days after nomination) Mid March

SEaL EVALUATION PLAN

Understand classroom dynamics in a university

Navigate between their personal values and those

Practice critical thinking skills

Establish relationships with faculty, staff, and

Increase their self-efficacy in utilizing resources

Become independent thinkers and responsible

SEaL EVALUATION PLAN

Gathering materials from students

SEaL EVALUATION PLAN

SEaL EVALUATION PLAN

Evaluation & Assessment Final Project

Evaluation & Assessment Final Project