Running head: PROOF OF CONCEPTS PROGRAM EVALUATION 1

Assessing the Proof of Concepts Program Evaluation

Shakoiya Aiken

University of West Georgia

PROOF OF CONCEPTS PROGRAM EVALUATION
2

Table of Contents

Title page……………………………………….1

Table of Contents……………………………………….2

Description of Program and Major Stakeholders……………………………………….3-5

Purpose of Evaluation……………………………………….6-8

Proposed Methodology/Data Collection …………………………9-10

Procedures/Analysis………………………………………10-11

Evaluation Budget……………………………………….12

Evaluator’s Qualifications………………………………………13

Dissemination Plan……………………………………….14-15

References…………………………………………………16

Appendices……………………………………….17-24
PROOF OF CONCEPTS PROGRAM EVALUATION
3

Description of the Proof of Concepts Initiative

The Proof of Concepts is a program designed to help students with multiplication and

division fact fluency. The instruction happens outside of the designated math block time. The

lessons are scripted in which the time it takes to complete them can vary by teacher and

students. There are guided lessons, problem sets, and exit tickets, along with outside

assessments. This program is taking place in all third grade classrooms at a local Cobb County

School, due to the fact their Georgia Milestones scores from 2017-2018 displayed lack of

proficiency for third grade students in math. This school is considered to be the Pilot school in

which the evaluation and results of the initiative will be used to make informed decisions for

next school year. Decisions will be made for this school, as well as other Cobb County

Schools that demonstrated a lack of proficiency with growth in math test scores. The initiative

will last the entire school year. Teachers are given scripted lessons that are expected to be

done in 20-25 minutes outside of the normal math block for the first unit, while teaching their

current standards. The second unit of 3 rd grade is multiplication and division, so Proof of

Concepts will become their whole math block. Following the second unit, they will return to

working on it for 20-25 minutes outside of the normal block time. Students were given an

addition and subtraction fact pre-test in the beginning of the first unit. Those students who

scored below 60 were given the Gloss assessment. The Gloss allows us to see which stage

each individual student falls in with their math skills. There is an assessment being given in

the middle of every unit and at the end of the unit to assess student mastery.
PROOF OF CONCEPTS PROGRAM EVALUATION
4

The Proof of Concepts Initiative is new to the Cobb County School District. The original

target schools of the Proof of Concepts were three local schools in the District. The three schools

failed to demonstrate growth with students and their mathematical concepts pertaining to

division and multiplication fluency. The initiative strives to prepare students to show growth on

the assessments within the program, as well as the Georgia Milestones. They are not expecting

anything from this evaluation, but my academic coaches want to see whether or not this initiative

is effective by analyzing the data to see whether or not students demonstrated growth with their

fluency.

The overall objective of the Proof of Concepts is to demonstrate fact fluency with

multiplication and division. It is not primarily about memorizing the facts, but more so being

accurate, efficient, and flexible. The focus is accuracy in attending to precision. Efficiency is

selecting appropriate strategies to solve problems. Lastly, flexibility in using strategies such as

making 10 or decomposing numbers.

Stakeholders

The stakeholders for this program include the students in third grade, teachers, academic

coaches, and district personnel. We have not seen a benefit for teachers as of yet. District

personnel and superintendents benefit by being able to push their agenda and test a theory.

Students are confused, due to the fact of two different math concepts being done at the same

time. But, the overall benefit will be them being able to show growth with mathematical

concepts, specifically with multiplication and division fluency. The teachers are expected to

work with all students in a given time, which makes it difficult to reach all students in the way in

which they prefer. The teachers weren’t given much training on how to implement this program

within their classrooms. All of these students are expected to pass the Georgia Milestones and
PROOF OF CONCEPTS PROGRAM EVALUATION
5

score in the proficient or advanced range, so this program is designed to better assist them with

the math skills in which they need in order to be successful.

District personnel working with the Proof of Concepts Initiative are essential

stakeholders in the program. They are responsible for coordinating the entire program and

communicating with academic coaches, teachers, and administration. In addition, they are

responsible for scheduling skype meetings and conducting pop up visits to observe the third

grade classrooms.

The focus of this evaluation will be on the growth of students with multiplication and

division and to see if they can apply those concepts in the math setting. Administrators and

academic coaches suggested that this be a focus because they don’t believe it is transforming

teaching and learning. The teachers believe they are spending a lot of time on Proof of Concepts,

but still have other standards that need to be taught and covered, so they are eager to know if this

initiative will demonstrate the growth in which the students need.

Logic Model

See Appendix A.
PROOF OF CONCEPTS PROGRAM EVALUATION
6

Purpose of the Evaluation

The idea for this evaluation came from the academic coaches working directly with the

Proof of Concepts Initiative. I am analyzing and evaluating this program to determine whether

or not it is helping us with meeting our school improvement goal. Since we are the pilot

school, we are looking at whether or not the program should be used for the whole county. In

addition, it was stated by our principal that when Georgia Milestones scores post for the 2017-

2018 school year, it is possible that fifth grade maybe required to take part with this program

if we didn’t show progress. If that is the case, I will be directly connected with the program.

In addition, I will be involved as an external evaluator. Due to the fact, I am not directly

involved with the initiative, I am able to look at the program from a different perspective. I

will more than likely be freed of the bias that may come with evaluating this program.

Although the teachers are unable to currently see the benefits, being the external evaluator

will allow me to see the changes that have occurred during this pilot run that may have gone

unnoticed. I am also able to look at the data and analyze it to determine whether or not growth

is being shown on each individual assessment.

The program administrators specifically wanted to assess the instruction the students are

receiving and the growth in which they are demonstrating throughout the program with the hopes

of identifying areas of improvement.

After talking with the program administrators, it was decided that a formative evaluation

would be conducted. The county basically modified New York Engage Lessons and converted

them to Proof of Concepts, with a few adjustments. They are using this program as a pilot at my

school to determine whether or not it should be used for the whole county. This type of

evaluation seems to be more beneficial because it allows me to evaluate whether or not it’s
PROOF OF CONCEPTS PROGRAM EVALUATION
7

appropriate and feasible to be used in other grade levels or as a whole county initiative. This

program is being implemented in a new elementary setting, with a new population. Process

evaluation also fits the focus because we are determining whether or not the program activities

are being implemented as intended based on following the Proof of Concepts notebook with the

lessons in order. Lastly, the goal is to see whether or not the program is effective and meeting

the ultimate goal of demonstrating growth with each assessment given.

Evaluation Questions

This process, formative, and impact evaluation is important to conduct with the Proof of

Concepts Initiative because being a fifth grade teacher and not a third grade teacher, I have the

responsibility of making decisions pertaining to our School Improvement Plan based on different

initiatives. It is most certainly a difficult process. The goal of this evaluation would be to monitor

the ability for students to be able to apply multiplication and division concepts to show growth

with each assessment given, observe teacher implementation of program, and assess the

effectiveness of the program.

Evaluation questions include the following:

1) Does the Proof of Concepts (coordinators) provide adequate training for the teachers who

are required to deliver and implement the lessons?

2) Based on the exit tickets given, does it demonstrate student growth for each lesson?

(spreadsheet)

3) What are teachers’ attitudes regarding the process of Proof of Concepts?

4) Based on the formative assessments given, does it demonstrate growth with each

assessment for each student?

Constructs
PROOF OF CONCEPTS PROGRAM EVALUATION
8

The main constructs of interest are teacher attitudes, training, and data. Teacher attitudes

are defined as the way in which the teachers view the program in regards to the training and

observations they receive as they implement Proof of Concepts. Training is defined as the skills

necessary to learn in order to implement the initiative.

Data is defined as the information needed to assess the effectiveness of the program.
PROOF OF CONCEPTS PROGRAM EVALUATION
9

Proposed Methodology

Participants

Participants will include students in third grade that are a part of the Proof of Concepts

Initiative. In addition, the evaluation will include third grade teachers. There are about 110 third

grade students and 6 third grade teachers. Since the program administrators plan to utilize the

findings from the evaluation to enhance the program for future schools in the county, all students

are expected to play a role in it. There shouldn’t be any students exempted from it.

Unfortunately, there will not be any incentives. Ethics were most certainly thought of. This is the

first year of this initiative, so we are looking for ways to enhance the program to better suit the

needs of students before implementing in all schools within the county.

Design and Methodology

The designs of this evaluation will be a formative, process and outcome evaluation with a

focus on the instruction/objectives of the initiative and teacher attitudes. The process evaluation

is being conducted because the focus is to determine whether or not program elements have been

implemented accordingly. This process evaluation focus is on whether or not the lessons and

instruction within the initiative are well designed and delivered to students in an efficient manner

that allows them to develop and demonstrate growth with each assessment. I will be using Mixed

Methods. I chose this approach because I am using quantitative and qualitative data in order to

answer all of my research questions. They are blended together to allow me to do so.

The data for the evaluation will come from a survey, data spreadsheet of exit tickets and

formative assessment. A survey (see Appendix C) will be administered to all teachers expected to

implement initiative in their classrooms in the 2018-2019 school year. The survey was created

by Shakoiya Aiken and specifically designed for the Proof of Concepts Initiative (see Appendix
PROOF OF CONCEPTS PROGRAM EVALUATION
10

C). This survey will answer the evaluation questions. The survey will also contain open-ended

questions for teachers to communicate their perceptions about the Proof of Concepts.

Structured formal interviews will be used as the second source of data collection.

Teachers will be asked about their attitudes and experiences of becoming a Proof of Concepts

Instructional teacher. Interviews will allow for understanding and useful to answer one of the

evaluation questions, “What are teachers’ attitudes regarding the process of becoming a teacher

required to implement Proof of Concepts?”

Finally, program exit tickets and assessments will be used as the third source of data.

Teachers are required to keep a spreadsheet of all of the completed assessments within the

program. This information will be used to answer the first evaluation question, “Does the

program help with student growth on each assessment?” These spreadsheets can be monitored

throughout the program because they are stored on our Proof of Concepts data folder on the staff

drive.

Procedure

Prior to data collection, approval from the Teaching and Learning Department in Cobb

County School District must be obtained. A form will be completed and submitted when the

program administrators approve the program evaluation plan. Data collection will begin after

approval is obtained.

Data collection will begin in September of 2018. The instructor and the evaluator will

introduce the evaluation plan to the teachers during a staff meeting in the beginning of

September. Teachers will receive copies of the consent form (see Appendix D) for them to sign

and return to the evaluator, as well as a copy for their own individual records.
PROOF OF CONCEPTS PROGRAM EVALUATION
11

During the course of the semester, spreadsheets will be monitored to ensure teachers are

inputting data during the last day of each week. Information from the spreadsheets will be used

to track time spent on each Proof of Concept lessons, observations, and additional training.

Individual interviews will be conducted with each teacher during the evaluation process

(see Appendix B). Teachers will contact the evaluator via email to arrange a time to conduct their

interview and share their concerns and perceptions. Towards the last session for Proof of

Concepts, teachers will be asked to complete the survey (see Appendix C). Teachers will be

asked to complete their surveys on google forms, so that their responses remain anonymous and

information can’t be lost. Completing online will protect themselves and their responses.

Proposed Analysis

Mixed method analysis will be used in this evaluation incorporating both quantitative and

qualitative analysis. Teacher interviews will be transcript and analyzed. Data spreadsheets will be

used to analyze student growth. Mixing multiple methods can provide a more holistic

understanding (Bond, Mayoh, & Todres, 2011).

The post surveys will be analyzed using both qualitative and quantitative methods. The

closed-ended items will be analyzed using descriptive statistics and category construction.
PROOF OF CONCEPTS PROGRAM EVALUATION
12

Budget

The evaluation for the program will be conducted free of charge. The evaluation will be

conducted free of charge, due to this being a part of the evaluator assignments requirements for

her program evaluation course in her Educational Specialist program. If funding is available, the

fees may be paid by the Proof of Concepts Initiative. The complete budget is outlined in Table 2

that follows.

Table 2

Evaluation Budget

Budget Category Amount Description

Evaluation Fee $858 10 hours of interviews (1 hour
x 6) at $33 per hour = $198
Hourly rate
5 hours of document and
survey data collection at $33
per hour = $165

15 hours of data analysis &

report writing at $33 per hour
= $495

Office Supplies $75 Surveys done online.

Interview transcripts, and
course documents

Transcription Services $1,000 Flat rate for 6 transcripts

Total $1, 933

Evaluator’s Qualifications
PROOF OF CONCEPTS PROGRAM EVALUATION
13

The evaluator holds a Bachelor of Science in Elementary Education from Jacksonville

State University and a Master of Education in Early Childhood Education from the University of

West Georgia. She is currently in her final semester of her Educational Specialist degree in

Instructional Technology at the University of West Georgia. She has experience analyzing data

and using it to make informed decisions. During the duration of this evaluation, she will be

supervised by academic coaches at her present school. She is a fifth grade teacher, so she is not

directly connected to the program. She serves on the Principal’s Advisory Committee at the

same school in which the program is being implemented. She is expected to work with

administrators, teachers, and coaches to analyze programs, initiatives, and data as a whole.

Her role is to make sure they are aligning with the school improvement plan. In addition, it is

important to know and understand the third grade standards, and how the curriculum connects

together. This leads to being able to reflect and analyze whether or not the instruction and

assessment within the Proof of Concepts Initiative are successful and effective.

Dissemination Plan
PROOF OF CONCEPTS PROGRAM EVALUATION
14

After data collection and analysis is complete, a report of findings will be written and

submitted to the Proof of Concepts program coordinators. A PowerPoint presentation with charts

and graphs of the findings will be created and presented to the coordinators and the third grade

teachers. The final report will also be used to complete the requirements for the Program

Evaluation course in the Fall of 2018 at the University of West Georgia. Findings also have the

potential to be submitted as a presentation for the upcoming schools that will be required to

implement Proof of Concepts in their classroom upon approval from the Proof of Concepts

program coordinators. A complete timeline of the dissemination plan follows in Table 3 below.

Evaluation Timeline

Table 3

Evaluation Timeline

Meeting with stakeholders to review the

evaluation plan
August 2018
IRB form B submission to Teaching and
Learning Cobb County School District
Department
IRB Approval from The Teaching and Learning
Department

Introduce the evaluation plan to teachers apart

September 2018 of the Proof of Concepts Initiative

Collect informed consent (if necessary)

Monitor spreadsheets (end of the week)

October 2018 Monitor spreadsheets (end of week)

October 2018 Begin individual teacher interviews

November 2018 Monitor spreadsheets (end of the week) (end of

semester)
PROOF OF CONCEPTS PROGRAM EVALUATION
15

Conclude conducting teacher interviews

Conduct survey
PROOF OF CONCEPTS PROGRAM EVALUATION
16

References

Bond, C., Mayoh, J., Todres, L., (2011). Innovative Mixed Methods Approach to Studying the

Online Health Information Seeking Experiences of Adults With Chronic Health Conditions.

Journal of Mixed Methods Research 6 (1), 21-33. DOI: 10.1177/1558689811416942

Running head: ASSESSING TRAINING AND SUPERVISION OF MENTORS 17

Appendix A

The Proof of Concepts Initiative

The Proof of Concepts Initiative is a fairly new program for the Cobb County School District that prepares students by giving them the

lessons and teaching in which they need to become fluent with multiplication, division, addition, and subtraction fluency. The overall

objective of the Proof of Concepts Initiative is to provide a rigorous and relevant curriculum that will give students the ability to show

growth with their mathematical skills, specifically with multiplication and division fluency. The objective of the course is to guide

teachers through the initiative to allow them to prepare their students, so they are able to demonstrate the growth in which they need.

INPUTS ACTIVITIES OUTPUTS OUTCOMES*

Initial Intermediate Longer-Term
1 Counselor Education Proof of Concepts Trained teachers Positive Positive Positive
Professor/Program notebook Increased Acquisition of Empowered to
Coordinator development Children knowledge with skills required to deliver training to
receiving Proof Proof of Concepts deliver other teachers at
2 Academic Coaches Proof of Concepts of Concepts instruction to other schools
Training Increased students
1 Educational 3 hours a week of knowledge New strategies to
Specialist student Data documentation instruction with analyzing and using Experience with help students
training all students data struggling demonstrate
1 Classroom children growth
15-20 minutes of Understanding of
1 Proof Of Concepts individual data documentation Increased Negative
Scripted Lessons instruction per knowledge on Inability to cope
student Completing the how to model with arising
12 Weeks initiative and guide difficulties with
students through Proof of Concepts
25 Class meetings Negative the Proof of implementation
PROOF OF CONCEPTS PROGRAM EVALUATION 18

Increased Concepts (adjusting to the

discomfort with the changes
initiative (Proof of Re-enrollment consistently being
Concepts) in the program made)
for following
Time commitment year
with implementing
Proof of Concepts Negative
and regular Inability to
standards based properly
instruction implement Proof
of Concepts into
Frustration with classroom in a
lack of growth with separate math
individual students. block

Failing the
assessments within
the initiative
Running head: ASSESSING TRAINING AND SUPERVISION OF MENTORS 19

Appendix B

Interview Questions

1. What is your name (first name only required or last name) and job title here?
2. How long have you been at this school?
3. What are your thoughts about Proof of Concepts? How do you feel about Proof of

Concepts?
4. How effective are you with implementing Proof of concepts lessons in your classroom?

On a scale 1-10, rate yourself and explain why.

5. Do you feel as if you are preparing your students to be successful with math and its

concepts by utilizing Proof of Concepts in your classroom? If so, how?

6. Do you feel that you are being supported by being provided with the training necessary to

implement the initiative in your classroom? Please explain.

7. Do you feel administrators, school personnel, and district personnel effectively

communicates the vision/expectations for Proof of Concepts? Please explain.

8. What would you say are the weaknesses at implementing Proof of Concepts?
9. What would you say are the strengths at implementing Proof of Concepts?
10. What are your personal barriers of using Proof of Concepts?
11. How can your school help you or what things can be done to help you overcome the

barriers in which you face?

12. Do you feel as if using Proof of Concepts is beneficial to student learning and success?
PROOF OF CONCEPTS PROGRAM EVALUATION
20

Appendix C

Dear teachers,

I want to first say thank you for taking the time to complete this survey. I appreciate it.

We would like to learn about your experiences being the teachers required to implement Proof of

Concepts within your third grade classrooms. Your responses are valued and will help with the

evaluation of this program and the training needed for future teachers expected to implement this

initiative in their classrooms. All responses received will remain anonymous and will not be

reported to the coordinators of the program/initiative. The survey will be completed on Google

Forms .The survey contains 12 questions and the time in which it takes to complete can vary

depending on the way in which you think and respond to each question. If you have any

questions regarding the survey or this evaluation process, please contact Shakoiya Aiken at

saiken2@my.westga.edu .If you have questions about your rights as a participant, contact The

Cobb County School District Teaching and Learning Department at

michelle.mikes@cobbk12.org or 770.426.3562.
PROOF OF CONCEPTS PROGRAM EVALUATION
21

Link to Google Forms Survey:

https://forms.office.com/Pages/ResponsePage.aspx?id=-x3OL5-

ROEmquMR_D8kYLU8jnGLQqwBAt2-

266rhUndUM1FMTlIwWjlSWExMTVRZN1ZKQ1cyNFdTMC4u

Just in case link doesn’t work

1. Do you feel as if our school has Proof of Concepts materials to support student learning?

2. I am consistently using Proof of Concepts lessons. 1=Strongly Do not Agree 2= Do Not

Agree 3=Neutral 4= Agree 5= Strongly Agree

3. More than 90 % of students are demonstrating growth with each assessment 1=Strongly

Do not Agree 2= Do Not Agree 3=Neutral 4= Agree 5= Strongly Agree

4. I have support and training with Proof of Concepts to meet my teaching needs.

1=Strongly Do not Agree 2= Do Not Agree 3=Neutral 4= Agree 5= Strongly Agree

5. How would you like to see Proof of Concepts being used at your school in the next 3-5

years?

6. How can you support your colleagues and school to improve Proof of Concepts

implementation?

7. I have the knowledge needed to implement Proof of Concepts in my classroom?

1=Strongly Do not Agree 2= Do Not Agree 3=Neutral 4= Agree 5= Strongly Agree

8. I am comfortable using and implementing Proof of Concepts? 1=Strongly Do not Agree

2= Do Not Agree 3=Neutral 4= Agree 5= Strongly Agree

9. I understand the importance of Proof of Concepts.1=Strongly Do not Agree 2= Do Not

Agree 3=Neutral 4= Agree 5= Strongly Agree

PROOF OF CONCEPTS PROGRAM EVALUATION
22

10. I can adequately assess the growth of each child on each assessment?1=Strongly Do not

Agree 2= Do Not Agree 3=Neutral 4= Agree 5= Strongly Agree

11. What are your thoughts or feelings towards or about Proof of Concepts?

12. Please provide any specific comments or suggestions you have for this program:
PROOF OF CONCEPTS PROGRAM EVALUATION
23

Appendix D

Assessing the Proof of Concepts Initiative

Participant Information and Consent Form

The Cobb County School District Teaching and Learning Department

Proof of Concepts Initiative

During the fall 2018 semester, you were required to implement Proof of Concepts into your
classroom. We are interested in learning more about your development, experiences, and
attitudes as a teacher required to implement Proof of Concepts.

What does your involvement entail?

Your involvement entails participating in taking an anonymous survey, permission to having
your data monitored and analyzed, and individual interviews with the evaluator, Shakoiya Aiken.
This involvement is a requirement for the course. At any time, you may opt out and contact
Shakoiya Aiken as soon as possible. Your participation will begin once you have reviewed and
signed this Participation Information and Consent Form. If you have any questions or concerns,
they will be answered before participation begins.

Will you benefit from your participation?

Your participation will aid in enhancing the development and implementation of Proof of
Concepts for the Cobb County School District.

All research remains confidential.

I can ensure you that the information provided from you will remain confidential. Information
will be store on a flash drive in which Shakoiya Aiken only has access to. The informed consent
forms will be stored in a locked filed cabinet, as well as interview responses. The data received
from survey will only be viewed by Shakoiya Aiken. No other individual will have access to
your responses. In order to access survey on google forms, password is needed. Shakoiya Aiken
password is unique in which no other individual has access to.

Risks to you during research.

There is minimal risk involved in this study. Based on structured interviews and observations
feedback, you may experience distress or frustrations. This can be caused by the data or
information in which you have shared. Remember that you do have the option to opt out at any
given time.

CONTACT INFORMATION
If you have questions or concerns at any time about the research, procedures, or anything else in
regards to the study, please contact me:
PROOF OF CONCEPTS PROGRAM EVALUATION
24

Shakoiya Aiken, M. Ed, Educational Specialist Student & Teacher

Cobb County School District
514 Glover Street
Marietta, Ga. 30060
Phone (770) 426-3300
Email: saiken2@my.westga.edu

Michelle Mikes, M. Ed, Program Coordinator/Supervisor

Cobb County School District
514 Glover Street
Marietta, Ga. 30060
Phone (770) 426-3300
Email: michelle.mikes@cobbk12.org

If you have questions about your rights as a participant, contact The Cobb County School
District at (770) 426-3300.

_____________________________________________________________________________

CONSENT

I have received a copy of the information above and I have read and understood the information

given. I agree to participate in this study and have all my input and feedback included in any

publication or presentation in this study.

Participant's signature ______________________________ Date __________

Investigator's signature _____________________________ Date __________

PROOF OF CONCEPTS PROGRAM EVALUATION
25