The A~aba rr a Reading

hlltlative Program EvahJJation

Report

The Alabama Reading Initiative (ARI) Program Evaluation 2015 was commissioned by the
Alabama State Department of Education (ALSDE) as an intra-agency program evaluation.
The purpose of the program evaluation was to examine if the ARI has been effective in
supporting statewide literacy efforts as envisioned at its inception in FY1 999. To address
the objective , the program evaluation compared reading test data during the following ARI
phases , Phase 1: FY1999 to FY2006 , Phase II: FY2007 to FY2011 , and Phase Ill : FY2012
to FY2015 . The program evaluation revealed changes in the structure of the program ,
including changes that occurred in its configurations , funding , delivery modalities , and
support to educators in Alabama . The program evaluation involved the use of pre-existing
quantitative reading data and qualitative data from a sample of Alabama teachers and
reading coaches .

RESEARCH
DEVELOPMENT
ALABAMA STATE DEPARTMENT OF EDUC AT I ON
 Executive Summary
Introduction

The Alabama State Department of Education (ALSDE) charged the Research and Development
section with conducting a comprehensive evaluation the Alabama Reading Initiative (ARI) . Work
on the program evaluation began in August 2015 and continued through January 2016. The
ARI is a statewide literacy initiative piloted in FY1999, as a K-12 program. Since that time the
ARI has gone through three dramatic changes in its configuration. The three phases addressed
in this study are Phase 1: FY1999 to FY2006, Phase II: FY2007 to FY2011, and Phase Ill:
FY2012 to FY2015. In Phase I, the ARI was a K-12 program . During Phase II, at the urging of
then Governor Bob Riley, the decision was made by State Superintendent of Education , Dr. Joe
Morton , to focus ARI efforts on the K-3 grade span and reduce support to secondary schools. In
Phase Ill , beginning in FY2012 , the ARI completed its expansion back to a K-12 program that
was begun in FY201 0.

During its initial phase of development the ARI succeeded in becoming a robust, comprehensive ,
and effective reading reform initiative. Initially, the vision of the ARI was to implement research-
based instructional practices through the delivery of professional development supported and
sustained by full-time reading coaches working side-by-side with educators. The deployment of
the intensive, systematic professional development was initially geared towards increasing
teacher capacity leading to improved reading proficiency of Alabama students.

The purpose of the current ARI program evaluation was to examine whether or not the ARI in its
multiple configurations has continued to be effective in supporting statewide literacy efforts. The
evaluation examined statewide reading proficiency trends from FY1999 to 2015 utilizing data
gleaned from tests including the SAT 9, SAT 10, Alabama Reading and Mathematics Test
(ARMT), National Assessment of Education Progress (NAEP), ACT Aspire , and ACT Plan .

Historical Overview of Statewide Reading Assessments

The test used to measure literacy in Alabama has undergone numerous changes during the
ARI 's existence. The ARI was developed in part to address persistently low reading proficiency
levels on the SAT 9, a norm-referenced assessment. The nationally normed SAT 9 was
superseded by the SAT 10 in 2002-2003. For the purposes of this paper the SAT 9 and SAT 10
will be referred to simply as the SAT. In 2003-2004, a criterion-referenced test called the
Alabama Reading and Mathematics Test (ARMT) was piloted in Grades 4, 6 and 8. Beginning
in 2004-2005 , and continuing through 2012-2013 , the ARMT was administered statewide to all
students in Grades 3 through 8. The ARMT was replaced in 2013-2014 with the ACT battery of
norm-referenced assessments, specifically, the ACT Aspire given in Grades 3 through 8, the
ACT Plan that was administered in Grade1 0, and the ACT given in Grade 11. The only test that
has been in continuous use since the ARI's inception is the National Assessment of Educational

11
Progress (NAEP). Noticeably absent has been a statewide test used for measuring K-2 reading
proficiency.

The ARI Program Evaluation Questions

The evaluation examined the ARI non-negotiables as they related to school-level

implementation; the grade-level bands supported by the ARI ; school-level roles and
responsibilities , and the human resources provided to accomplish each. The evaluation also
reviewed levels of funding during each of the three phases of the ARI. Finally, the qualitative
component of the program evaluation was conducted utilizing data from two electronic survey
instruments that were open for responses from January 1-15, 2016; the ARt Program Evaluation
Teacher Survey and the ARt Program Evaluation Coaches Survey. There were 5,110 responses
to the teacher survey and 681 responses to the coach survey.

The overarching question for the ARI program evaluation was, "To what extent has the ARI
achieved the goal of 100% literacy for Alabama students since the inception of the initiative in
1998 through 2015?" The overarching question was further explored through use of the eight
probing questions that follow:

1. What differences exist in Phase I, Phase II, and Phase Ill statewide reading proficiency
levels?
2. Are there any differences in Phase I, Phase II , and Phase Ill reading proficiency levels
based on a comparison of the ARMT?
3. Do differences exist in Phase I, Phase II , and Phase Ill reading proficiency levels based
on the NAEP reading proficiency levels?
4. Are there differences in Phase Ill reading proficiency levels based on the ACT Aspire
reading proficiency levels?
5. Do differences exist in Phase Ill reading proficiency levels based on the ACT Plan reading
proficiency levels?
6. Do differences exist in Phase I, Phase II , and Phase Ill based on the ARI implementation,
scope, structure, and funding?
7. Are there differences in teachers' perceptions of how the changes in the ARI Phase I,
Phase II , and Phase Ill impacted the effectiveness of the program?
8. Are there differences in coaches' perceptions of how the changes in the ARI Phase I,
Phase II , and Phase Ill ARI impacted the effectiveness of the program?

The scope, structure, and funding components of the ARI were divided into three distinct
categories based on implementation .

Program Evaluation Methodology

This evaluation was designed as a mixed-methods, causal-comparative program evaluation.

The ARI program evaluation was conducted utilizing both quantitative and qualitative methods.

iii
The quantitative aspect of the evaluation was conducted by analyzing pre-existing reading
assessment data. Descriptive and inferential statistical calculations were conducted in order to
determine if differences existed between the three ARI phases. Inferential statistics involving
the use of Independent Samples t-test and Analysis of Variance (ANOVA) were used to
determine the degree of significance between variables (Gay, 2003). Statistical calculations
were conducted using SPSS statistical package . The qualitative component of the program
evaluation consisted of an analysis of data from the two, previously mentioned, surveys .

Summary of Findings

The summary of findings is presented by addressing each of the eight program evaluation
questions with regards to each phase of the ARI implementation. The overarching question for
the ARI program evaluation sought to determine the extent to which the ARI has maintained its
effectiveness in its current configuration . Variables examined in order to address the key
objective included comparisons in Phase I, Phase II , and Phase Ill reading proficiency levels
determined by state level SAT, ARMT, ACT Aspire, ACT Plan , and NAEP reading proficiency
levels. The data collected from both electronic survey instruments were also presented as it
relates to each ARI phase .

Phase 1: FY1999-FY2006

The ARI was effective based on improved reading proficiency scores as measured by the SAT,
ARMT, and the NAEP assessments. An analysis of the SAT data from FY1999-2003 indicated
proficiency rates increased for ARI schools by 8.8% over a five-year period , whereas non-ARI
schools proficiency rates increased 3.1%. More specifically, the analysis revealed that in 2003
alone ARI schools increased reading proficiency by 4.7% while non-ARI schools increased
reading proficiency by only 2.2%. Also , from FY1999-2004, there was a 13 percent greater
number of minority students proficient in ARI schools (49 .6%) than non-ARI schools (36.4%).
Overall , the SAT data for Phase I indicated that ARI schools outperformed non-ARI schools in 6
out of 8 years the test was administered .

When ARMT was piloted in 2003-2004, the ARI was in its fifth year of implementation . The
students who were in kindergarten in FY1999 were the first group of students who piloted the
ARMT. The reading proficiency for fourth-grade students the pilot year was 76 .95%. The
reading proficiency for the same students the following year in fifth grade was 80.93%; and by
FY2006 , the reading proficiency for these students in sixth grade was 83.08% . Therefore ,
evidence indicates that the initial cohort of students taking the ARMT continued to benefit from
the literacy foundation provided by the ARI in kindergarten through third grade. Further analysis
of all ARMT data from 2004-2006 indicated a consistently positive trend in the number of
students proficient in reading.

iv
The NAEP reading results in 2003 indicated the state's scaled reading score of 207 was nine
points below the national average of 216. In response to the 2003 NAEP reading scores the
ARI developed extensive reading comprehension resources and professional development. In
the fall of 2006 the training was provided across the state to all schools with additional support
provided to 41h and 81h grade teachers. This work was seen as a primary driver of the tremendous
gains recorded on NAEP reading scores during Phase II of the program .

It must also be noted that the ARI funding increased nine-fold during this period, from six million
dollars in FY2000 to fifty-five million dollars in FY2006 , while the number of ARI schools
increased similarly from 81 to 752 . Therefore, ARI's effectiveness was supported by appropriate
levels of funding during the rapid expansion that occurred in Phase I.

The ARt Program Evaluation Teachers ' Survey instrument, one source of the qualitative data
used for this study, was completed by 2,161 teachers who self-identified as beginning their
tenure with ARI during Phase I. Fifty-seven percent of these teachers reported receiving
daily/weekly support from ARI reading coaches while only 7% reported never receiving support
from a reading coach. The importance of these percentages is driven home by several
comments from the survey indicating the effectiveness of early ARI support. Three such
comments from teachers who received support from the building reading coach were as follows :

"ARI support helps me to be able to teach reading effectively;"

"My building coach is all about the success of each student in our building. She is a team
player who collaborates effectively to mirror what is done in the classroom ;' and
"The initial ARI training I attended was very focused and well-developed . It changed the
way I viewed the teaching of reading , and its importance."
Both the quality of the ARI support, and the initiative's inability to continue comprehensive
support throughout its existence was made clear when one teacher stated, "When we had ARI
support, it was VERY beneficial. I miss it! "

Further analysis of the survey data indicated all the teachers in Phase I completed one or more
of the required ARI professional development experiences needed for schools to become an
ARI school. Those teachers reported the training support from ARI has declined over time.
Some of the comments were :

"They are no longer in our rooms as much doing one-on-one support. Now it is mostly by
grade level,"
"I feel that ARI is a natural process that ALL teachers should be using ."
"The only change I see , is schools are not offering more ARI training for new teachers
and they are not getting the benefit of ARI ,"
"We have received less training in the actual teaching of reading during recent years. We
received many updates on how to manage newly adopted reading series. More recently,
we have been given permission to use the reading series as a resource , rather than a
taskmaster. This has allowed me to incorporate more ARI strategies in teaching reading .

v
 We have not had as much professional development devoted to ARI in recent years, but
I still use the concepts. I would welcome more professional development in this area ."
Overall, teachers reported the changes over time have negatively impacted ARI 's ability to
provide focused support for implementation and development of a comprehensive literacy
program.

The ARI Program Evaluation Coaches Survey instrument was completed by 31 reading coaches
who had been reading coaches since Phase I. Similar to the teachers, the reading coaches
indicated that ARI , over time , has decreased the support provided to reading coaches. Some
comments from Phase I reading coaches follow.

"When we first began , we had lots of support from regional coaches and principal
coaches , over the years that support faded , there have been too many changes and some
of the good things we used to do have been tossed away. We have become a mile wide
and an inch deep, one can do many things ... but one cannot do many things well ; it has
caused new coaches to have little or no training as a coach;"
"As a coach of three schools, my effectiveness has been compromised ...the data shows
the negative impact of pulling coaches out of elementary schools hence the gap we are
seeing in middle schools, which will transfer to the high schools, a narrowed focus is
needed."
Reading coaches are particularly sensitive to the how detrimental to ARI's effectiveness the
continued expansion of focus without an accompanying expansion resources has been.
Building-level reading coaches were originally tasked with providing on-site support to teachers
in a specific school along with coaching support in the delivery of skillful reading instruction . In
addition , the job description of building reading coaches in Phase I included working with
struggling readers to overcome reading difficulties. With reduced budgets resulting in most
reading coaches being assigned to multiple schools, that level of support is no longer realistic.

Phase II: FY2007-FY2011

During Phase II , the ARI continued to build upon the reading gains made in Phase I. An analysis
of the ARMT reading data revealed a steady growth in reading proficiency levels for Grades 3
through 8. The reading proficiency levels Grades 3, 4, 5, and 6 remained at or above 84% . The
reading proficiency for Grade 7 ranged from 71% to 83% and the Grade 8 reading proficiency
remained above 70% .

In 2007 , Alabama broke NAEP records by increasing eight points in one test cycle, more than
any other state in the nation . The 2009 Grade 4 NAEP reading scores remained stable but 2011
Grade 4 NAEP reading scores increased to the national average of 220 for the first time.
Alabama was commended nationally for once again showing the most growth on the Grade 4
NAEP reading assessment. The Alabama Grade 8 NAEP reading scores showed a five-point

vi
increase from 2003 to 2011 , a level of growth that was 61h highest in the nation. Governor Bob
Riley and state education leaders celebrated the NAEP results and traced them to the statewide
literacy efforts led by the ARI.

Dr. Joe Morton, former Alabama State Superintendent of Education , and former Alabama
Governor Bob Riley presenting the 2007 NAEP data.

In a September 25, 2007 , press release, then Governor Bob Riley said ,

"Alabama is number one in the nation in reading improvements at the

fourth grade level. This didn't happen by accident. This is part of our game
plan to improve education in Alabama , and this is proof positive that our
game plan is working . The progress we have made in just a few years is
truly remarkable, and if we keep our focus on funding the programs, we will
continue to record these types of amazing gains. I want to thank all the
teachers because they are doing exactly what we asked them to do. They
know how incredible our Alabama Reading Initiative is and that one of the
reasons it is so successful is because it provides support for our
teachers. They know we must continue to expand it to all grade levels, and
we will. " (ALSDE, 2007).

In addition , former State Superintendent of Education Dr. Joe Morton observed: "We're number
one on the list of fourth grade Reading improvement in the nation, and it is not because the
states are listed alphabetically." According to Sherry Calvert, Principal of FE Burleson
Elementary in Hartselle, "The impact ARI has had on our school has been unbelievable. When
I asked my faculty what they attribute our positive change in performance to , the answer was,
'The training ARI provides in reading comprehension."'

vii
Although state officials seemed to be in agreement that full funding of the ARI enabled the
program to deliver services effectively to schools, as the number of ARI schools increased in
Phase II, from 899 to 920 schools, the ARI budget did not increase at the same rate . On the
surface, it did not immediately appear that the lack of funding increases had a major negative
impact on the ARI support to schools. However, the reimbursement provided to schools for
hiring reading coaches was reduced from $64,167 to $55,000. Because they were now
shouldering more of the cost of the coaches, systems were given permission to hire and use
reading coaches as they deemed fit. Therefore, some schools and systems expanded the roles
and responsibilities beyond the job description of ARI reading coaches. The funding decisions
made in Phase II had far reaching implications as captured in the survey data gathered from
teachers and building reading coaches in this phase.

Based on the results of the teacher survey, there were 987 Phase II teachers who completed
the survey. Fifty percent of these teachers reported receiving daily or weekly support from ARI
reading coaches, down from 57% of Phase I teachers. Fully 9% of teachers reported never
receiving support from a reading coach, up from 7% in Phase I. Some comments regarding ARI
support received by these teachers were "I received more support from the reading coach right
after going through ARI training . Over the past 7 years, I received less than weekly support."
One teacher compared the ARI support received when each school had its own building based
reading coach to what happens when reading coaches are rotated between schools. That
teacher stated , "I received a lot more support (daily/weekly) when the reading coach was specific
to the school. The new rotating coaches are good , but are not there enough for that face-to-face
support. " Another teacher provided additional insight into the many roles and responsibilities of
the reading coach , which were not part of the original job description, "Our Reading Coach is a
very good teacher and will do/help when needed . The problem is that our reading coach is pulled
in so many directions (writing school-wide reports , teaching math , going "here and there" for
"this and that"), that the "help" is not much ." This represents a stark departure from the ARI
recommended roles and responsibilities of the building reading coach .

Additional analysis of the survey revealed a decline in the percentage of teachers who indicated
that they had participated in one or more ARI-facilitated professional development offerings .
One of the teachers recounted that "For the first several years , after our faculty was initially
trained , we received ARI support on a continuous basis. The ARI helped us look at our data very
closely ... which was sometimes a bit painful but very necessary in identifying interferences to
student learning. For the past three of four years, we have received limited help, as I understand
school systems are now required to "ask" for help." Another teacher questioned the wisdom of
relying mostly on building reading coaches with ever expand ing job responsibilities to provide
professional development for faculties . That teacher stated , "The coaches are spread too thin.
Often times they service more than one school. They have a lot of other roles that often take
them away from being able to help teachers frequently. ARI-related to special education
students .... no materials. What additional materials are there to help our students succeed in

viii
reading? Programs? Online programs? Etc ... " As the ARI-facilitated support decreased, there
appeared to be a corresponding decrease in school-level professional development. As one
teacher noted, "I feel like over time the support has decreased. Sadly, I feel like we need that
support and even retraining more now! Many schools (mine) are in dire need of those explicit
and intentional lessons." Phase II teachers clearly suggested the need for more ARI-facilitated
professional development.

Based on the building reading coach/partner survey results , there appeared to be very few
reading coaches that were hired in this phase . The number of survey respondents who self-
identified as building reading coaches Phase II was 49 reading coaches. However, even the
limited survey data from building reading coaches hired in Phase II yielded important insights
into the impact of the changes in ARI over time and the effect of those changes on the success
of the program.

The data from the building reading coaches seemed to indicate that the workload of the building
coaches over time negatively impacted their effectiveness. The coaches expressed concern
over the requirement that they serve more than one school and become experts at all grade
levels. A coach pointed out that "The effectiveness has diminished due to the fact that I am
required to serve a high school in addition to my elementary school , it is too much for one
person ." Another coach while appreciative of the support from the ARI, by the same token ,
revealed that "Support I have received from ARI has been good . However, I feel that some of
the effectiveness was lost when coaches began assisting numerous grade levels, rather than
just the early grade levels." The coaches also regretted the lack of accountability in some cases
because , as one noted , "Over time teachers are no longer held accountable for small group
instruction because of the watering down of the role of reading coaches. The effectiveness of
the ARI has slowly declined because coaches are not able to complete the coaching cycle due
to the fact that they have to support more than one school." Furthermore, the building reading
coaches expressed professional nostalgia, as stated by one of the coaches, "I miss the on-site
support ... as the number of regional coaches declined their support for coaches is almost non-
existent, coaching communities are non-existent. There are workshops offered , but none that
are school specific."

Alarmingly, some survey responses revealed that some highly skilled building reading coaches
have had to revert back to being classroom teachers because of decisions made by school
leadership, which adversely impacted their effectiveness. As one former coach stated , "Because
of coaching support from ARI during the 2007-2010, the data at my school was unbelievable.
Our DIBELS data was very high and our population of students in Tier 2 and Tier 3 decreased
significantly. As a reading coach , the expectation was set that I was to coach daily. My last year
as a building reading coach 2011-2012 , we did not receive any support from the ARI. Our
system made the decision that they could provide the support we needed ; however, we were
asked to coach all grades in our building and all content areas. We desperately needed the
ARI's support but our system did not allow that support to be provided . I am no longer a building

ix
reading coach." Based on these comments there appears to be a consensus of opinion by Phase
I and Phase II coaches regarding the negative impact of changes in the funding, focus,
implementation, and support of the ARI in its current configuration.

Phase Ill: FY2012- FY2015

In Phase Ill , Alabama continued to administer the NAEP, however, in 2013-2014, the ARMT was
phased out and was replaced by the ACT battery of assessments as the measure of reading
proficiency for Grades 3 to 11. In 2015, the Grade 4 NAEP reading proficiency declined from a
220 state average scaled score in 2011 to a 217 state scale score. This meant that from 2011
to 2015 Alabama's NAEP 41h grade reading proficiency decreased from the national average to
4 points below the national average. The 2015 Grade 8 NAEP reading proficiency scaled score
saw a 1 point increase from 258 to 259 . Although this represents a 1 point increase in the state
reading proficiency, the scores are still 5 points lower than the Grade 8 national reading average .

Data from the ACT Aspire seemed to suggest that the increases in reading proficiency attained
in the ARI Phase I and Phase II are now trending downwards. Analysis of the ACT Aspire data
also showed a decline in reading proficiency for Grades 3 to 8. A comparison of the 2014 and
the 2015 ACT Aspire reading data showed that proficiency remained under 50% in Grades 3
through 8 for both years. The reading proficiency data showed no growth from 2014 to 2015 in
Grades 3, 4, and 5. There was a slight increase in the Grade 6 reading proficiency from 42% in
2014 to 43% in 2015 . Conversely, from 2014 to 2015, Grade 7 reading proficiency decreased
from 35% to 34%; the Grade 8 reading proficiency also dropped from 48% in 2014 to 44% in
2015.

The Grade1 0 ACT Plan data appeared to confirm the declining trends in reading proficiency
levels since students in Alabama started taking the test in 2013. A look at the ACT Plan reading
proficiency data reveals a declined from 39% in 2013, to 31% in 2014, and 32% in 2015 .
Declining reading scores on the 1Qth grade Plan test (soon to be the 1Qth grade Aspire test) do
not bode well for the 11th grade ACT that follows .

During Phase Ill deep funding cuts were made in ARI. The ARI budget was cut from almost
sixty million dollars in FY2011 to $48 ,212,043 in FY2015. At the same time , ARI schools
increased in scope and number from 920 schools in FY201 0 to providing as-needed support to
all schools as a part of regional teams. Drastic cuts were also made in the ARI personnel at state
and school levels. The number of field-based regional coaches decreased from 103 regional
coaches in FY2007 to 68 regional coaches today. The ARI allocation provided to schools for
hiring reading coaches was also cut from a little over $55 , 000 in 2010 to $51 ,000 during Phase
Ill.

The survey data from teachers who self-identified as Phase Ill teachers provides some insight
into the ARI Phase Ill experience. Six hundred teachers who completed the survey self-identified
in Phase Ill teachers. Of this number, 47% indicated that they received daily/weekly support

X
from the building reading coach and 16%, more than twice the percentage reported in Phase I,
indicated that they had never received support from the building reading coach . Although these
teachers appreciated the work done by the building reading coach as one of the teachers put it,
"Coaches have been instrumental in teaching me different strategies, but I don't get to see them
enough ." Another teacher stated "My reading coach's time is spread among six grade levels
and the ridiculous number of students. It is virtually impossible to give regular support for the
entire staff in a normal school day. She does her job well but is challenged by an overflowing
schedule . Sad!!"

The survey data regarding ARI-facilitated professional development became increasingly

negative in Phase Ill. Teachers revealed that they received a preponderance of their
professional development at the school level. However, the response from one teacher seemed
to suggest that at her school , there may not be on-going professional development provided by
the reading coach , "We used to have a full-time ARI coach that would help supplement our
classes with reading materials and provide professional development sessions throughout the
year. She was a fantastic part of our staff, but her position has since been moved from our high
school to our junior high school. " Because the ARI regional coaches are stretched so thin , some
have had to provide support to teachers by means other than by side-by-side coaching methods.
One teacher stated that "The ARI regional coaches do not visit my building as much as they did
in the past. They still provide support through professional development delivered to my staff at
the request of the principal or the instructional coach . They have been available for support via
text or phone when needed ." Because face-to-face , in-classroom coaching was a cornerstone
of the original success of the ARI , these comments underscore the fact that changes in the
funding , implementation , focus , and structure of the ARI have negatively impacted the
effectiveness of the program .

There was a total of 111 building reading coaches who self-identified as Phase Ill reading
coaches. Survey responses from this group of coaches seemed to be in line with responses
presented by building reading coaches in Phase I and Phase II. In spite of the fact that these
coaches have only been coaching for three years or less and have not been privy to the
expansive support provided in Phase I, they are still appreciative of what support the ARI can
provide. One coach, a transplant from another state, suggests a need for more support and
professional development, "I have come from teaching in a state who does not have ARI , and I
know that any support is more effective than no support at all." Another coach revealed that
administrators do not give the coaches the support needed to successfully fulfill their job
description. That coach stated , "We are used for "other" duties, not for the purpose of
reading/instructional coaches." The coaches also decried the lack of coaching emphasis in the
lower grades, but overall they were appreciative of the interactions with ARI regional staff. For
that reason , as one respondent said "The ARI would be more effective if additional regional
coaches were available. Support is spread so thin amongst several schools." All of which points

xi
to the need for more resources and a keener focus on literacy if Phase Ill of ARI is ever to match
the success of Phase I.

Conclusions

The current configuration of the ARI has profoundly changed from its original status as a
statewide flagship literacy specific initiative to its current configuration as a component of a
general support program .
The ARI was originally designed as a school improvement initiative requiring , at least, an
85% commitment to 100% literacy by school leaders and teachers through intensive on-going
professional development to teacher practice thus affecting student reading outcomes.
It appears reduced literacy focus in the early grades resulted in a downward trend in reading
proficiency levels in the upper grades.
The differences in the ARI then and now are evident in the focus , scope , implementation ,
and funding ; and the ARI has been asked to expand its reach well beyond its original focus .
It appears that the ARI has lost its specialized autonomy and advocacy for research-based
reading instruction and became just another member of an on-demand support system.
Reductions in the numbers of ARI field staff came at a time when the program could least
afford to lose personnel. The ARI regional staff was cut from 103 at the end of Phase I to
the current level of 68 regional staff. The few remaining field staff have been required to
provide support to individual schools and/or regional teams that may or may not have
anything to do with literacy.
Teachers, as well as reading coaches who are affected by changes in the ARI , clearly
suggest that the changes have negatively impacted the effectiveness of the ARI.

Discussion

It appears that the ARI was stretched so thin that it created an atmosphere in which the initiative
lost a substantial amount of its once nationally lauded effectiveness. In order to recapture the
effectiveness and relevance of ARI , it needs to return to its primary focus: 100% literacy for
Alabama students. At the school level, teacher attrition rates may also be a factor in the
implementation and sustainability of "best practice" instructional delivery methods embraced by
ARI-trained teachers and administrators. It is difficult to determine the percentage of ARI-trained
teachers and administrators from Cohort I and Cohort II that are still actively practicing in
Alabama classrooms. Clearly, the expansion of focus coupled with a reduction in funds available
to address the expanded focus has had a tremendously negative impact on program outcomes.

xii
Recommendations for Future Practice

In order to recapture the effectiveness and relevance of ARI as a nationally renowned reading
program aligned with the goals of PLAN 2020 , the following recommendations are made:

The ARI should return to its primary focus: 100% literacy for Alabama students, especially at
the elementary level.
Elementary schools should be staffed with highly trained reading coaches whose primary
duties include providing coaching support to teachers as well as working with struggling
readers .
School staff and leaders should once again be requ ired to recommit to participate in year-
round professional development provided by the ARIon best practices in reading instruction .
The partnerships between the ARI and colleges of education should be strengthened to
ensure a seamless transition from college to the classroom.
Prospective educational leaders at the elementary level should be required to complete a
two-week intensive ARI training .
The Alabama Legislature should fully fund the ARI. Full funding will enable the ARI to be
effective in the delivery of support to schools statewide, which will yield a return on investment
through improved reading outcomes for Alabama students.

xiii
 The Alabama Reading Initiative Program
Evaluation Report

A report submitted by the Alabama State Department of

Education, Research and Development Section

Principal Evaluator: Patience N. Oranika, Ed.D.

Abstract

The Alabama Reading Initiative (ARI) Program Evaluation 2015 was commissioned
by the Alabama State Department of Education (ALSDE) as an intra-agency
program evaluation . The purpose of the program evaluation was to examine if the
ARI has been effective in supporting statewide literacy efforts as envisioned at its
inception in FY1999. To address the objective , the program evaluation compared
reading test data during the following ARI phases, Phase 1: FY1999 to FY2006 ,
Phase II : FY2007 to FY2011 , and Phase Ill : FY2012 to FY2015 . The program
evaluation revealed changes in the structure of the program , including changes that
occurred in its configurations, funding , delivery modalities, and support to
educators in Alabama . The program evaluation involved the use of pre-existing
quantitative reading data and qualitative data from a sample of Alabama teachers
and reading coaches.

1
 Table of Contents

Table of Contents 2

Introduction 4

ARI Program Evaluation Questions 5

ARI Program Evaluation Methodology 5

ARI Phase I Implementation: FY1999-FY2006 6

ARI Phase II Implementation : FY2007-FY2011 9

ARI Implementation Phase Ill : FY2012- FY2015 11

Summary of Findings 23
Conclusions 26

Discussion 26

Recommendations for Future Practice 27

References 28

2
 Key Elements of ARI

Commit to a 100%
literacy rate among
students.

Commit at least 85% of

the faculty and
administration to attend
a two-week intensive
summer institute
focused on reading
improvement and
ongoing year-round
professional
development.

Employ at least one full-

time reading coach ,
whose job was to work
with teachers as well as
with struggling readers .

Collaborate between
schools and higher
education faculty
partners, who served as
mentors, provided
access to research and
helped solve
instructional problems
related to literacy and
learning .

Engage in partnerships
with local businesses.

Protect the read ing

block in the master
schedule.

3
 Introduction

In August 2015 , the Alabama State Department of Education (ALSDE) directed the Research
and Development (R&D) Section to conduct a program evaluation of the Alabama Reading
Initiative (ARI). The purpose of the current ARI program evaluation was to examine whether or
not ARI in its current configuration has been effective in supporting statewide literacy efforts as
originally envisioned. The evaluation analyzed statewide reading proficiency trends from FY1999
to FY2015 utilizing data garnered from tests including the Stanford Achievement Tests (SAT),
the Alabama Reading and Mathematics Test (ARMT), the National Assessment of Education
Progress (NAEP), The ACT Aspire, and the ACT Plan. During subsequent discussions of
reading assessments both the SAT9 and the SAT10 will be referred to as simply the SAT.

For the purpose of this evaluation the span of time that the ARI has been in existence, FY1999
to FY2015 was divided into three distinct categories based on the changes in the
implementation, scope, structure and funding of the initiative. The three phases addressed in
this study are: Phase 1: FY1999 to FY2006, Phase II: FY2007 to FY2011, and Phase Ill: FY2012
to FY2015. The evaluation examined the ARI non-negotiables as they relate to school-level
implementation, grade levels supported by the ARI , and the levels of funding during each of the
three phases.

The ARI as a statewide reading initiative was created out of the need to improve the literacy
rates of Alabama students. Reading data in 1997 revealed approximately 92,000 Alabama
students were scoring at the lowest levels (stanines 1, 2, and 3) on the Stanford Achievement
Test 9 (SAT9). Scoring in stanines 1, 2, and 3 equated to having below grade level reading ability
and the large number of Alabama students in that category prompted the creation of the ARI. To
address the need , ARI required participating schools to agree to certain conditions in order to
receive best practice teacher and leadership training coupled with ongoing implementation
support. The ARI required schools to commit to the following:

1. Commit to a 100% literacy rate among students.

2. Commit at least 85% of the faculty and administration to attend a two-week intensive
summer institute focused on reading improvement and ongoing , year-round professional
development.
3. Employ at least one full-time reading coach , whose job was to work with teachers as
well as with struggling readers.
4. Collaborate between schools and higher education faculty partners, who served as
mentors, provide access to research and help solve instructional problems related to
literacy and learning.
5. Engage in partnerships with local businesses.
6. Protect the reading block in the master schedule

4
With these prerequisites in place the ARI began utilizing research-based best practices to
support the implementation of skillful reading instruction in Alabama schools.

ARI Program Evaluation Questions

The purpose of the current ARI program evaluation was to examine if ARI in its current
configuration has been effective in supporting statewide literacy efforts as originally envisioned.
The overarching question of the ARI program evaluation sought to determine, "To what extent
has ARI achieved the goal of 100% literacy for Alabama students since the inception of the
initiative from FY1999 to FY2015?" The overarching question was addressed by seeking
answers to the following questions:

1. What differences exist in Phase I, Phase II , and Phase Ill statewide reading proficiency
levels?
2. Are there any differences in Phase I, Phase II , and Phase Ill reading proficiency levels
based on a comparison of the ARMT?
3. Do differences exist in Phase I, Phase II , and Phase Ill reading proficiency levels based
on NAEP reading proficiency levels?
4. Are there differences in Phase Ill reading proficiency levels based on ACT Aspire reading
proficiency levels?
5. Do differences exist in Phase Ill reading proficiency levels based on Plan reading
proficiency levels?
6. Do differences exist in Phase I, Phase II , and Phase Ill based on ARI implementation ,
scope, structure, and funding?
7. Are there differences in teachers' perceptions of how the changes in the ARI Phase I,
Phase II , and Phase Ill impacted the effectiveness of the program?
8. Are there differences in coaches' perceptions of how the changes in the ARI Phase I,
Phase II , and Phase Ill ARI impacted the effectiveness of the program?

ARI Program Evaluation Methodology

The ARI program evaluation was conducted utilizing quantitative and qualitative methods. The
quantitative aspect of the evaluation was conducted by analyzing preexisting reading
assessment data. Descriptive and inferential calculations were employed in the current program
evaluation . Descriptive and inferential statistical calculations were conducted in order to
determine if differences exist between the three ARI phases evaluated . Inferential statistics
involving the use of Independent Samples t-test, and Analysis of Variance (ANOVA) were used
to determine the degree of significance between one variable and another (Gay and Airasian ,
2003).

Creswell (2012), recommended the use of descriptive statistics to determine general tendencies
in the data. Gay and Airasian (2003) ind icated that t-tests are appropriate for determining if the

5
means of two groups are significantly different from each other. Furthermore , Cohen 's d
computations were employed to determine practical significance between one variable and
another. Data utilized for the evaluation were garnered from the ALSDE public data domain.
Statistical calculations were conducted using SPSS statistical package . The evaluation
questions are addressed in the next section .

The qualitative component of the program evaluation was conducted utilizing data from two
electronic survey instruments that were open for repsonses from January 1-15, 2016 . The
surveys were entitled ARt Program Evaluation Teacher Survey and the other was the ARt
Program Evaluation Coaches Survey.

ARI Program Evaluation Question 1. What differences exist in Phase I, Phase II , and Phase
Ill statewide reading proficiency levels?

To address this question , it was necessary to present qualitative and quantitative data regarding
state level read ing proficiency levels for each phase .

ARI Phase I Implementation

The ARI Phase I implementation spans the FY1999 to FY2006 . The focus of Phase I included
three points of emphasis: beginning reading , expansion of reading power, and effective
intervention . In its first year of implementation, approximately 625 teachers, administrators ,
counselors, librarians, and central office staff participated in initial training to support literacy
efforts for 8,662 students in 16 demonstration sites. From FY1999 to FY2006 , 900 schools were
added to ARI. Schools were added to ARI by increments as cohorts every year. Tab le 1 shows
the ARI growth by year, cohort and number of schools .

Descriptive Table 1. The ARI Growth , by Year and Cohort

Year Cohort Number of ARI Schools

1998-1999 Cohort 1 16 Schools

1999-2000 Cohort 2 81 Schools
2000-2001 Cohort 3 267 Schools
2001 -2002 Cohort 4 423 Schools
2002-2003 Cohort 5 450 Schools
2003-2004 Cohort 6 485 Schools
2004-2005 Cohort 7 511 Schools
2005-2006* Cohort 8 753 Schools
2006-2007 Cohort 9 900 Schools
*Note. All K-3 Schools statewide were trained by the end of FY2006

A breakdown of the tota l number of ARI schools at the end of Phase I showed : 79 midd le, 86
high schools, and 735 elementary schools. During Phase I, Alabama was one of three states to

6
receive a federal Reading First grant with an initial allocation of $16,105,592 in 2002. In 2003,
parallel to the work to the ARI , the Alabama Reading First Initiative (ARFI) began . Through this
initiative low-performing Alabama schools with grades K-3 competed for sub-grants of $200,000
or more to support research-based reading programs, substantial professional learning, and
coaching for teachers to improve student outcomes. In 2003-2004 74 schools were named ARFI
schools. The ARFI federal allocations continued from FY 2003 through FY 2009, and the total
number of ARFI schools grew to 93 . Two prominent requirements of ARFI schools included the
purchasing of a scientifically based comprehensive reading program, from a short list of ALSDE
approved programs, and the hiring of a full time reading coach.

The SAT data analysis from 2000 to 2003 revealed that ARI schools increased the percent of
proficient students on the SAT by 4.5 percentage points. The comparable gain for non-ARI
schools over this period was 2.2%. The Cohort 1 schools raised proficiency rates by 8.8% over
their five years in the ARI , compared to 3.1% for non-ARI schools. The ARI schools showed
greater gains than non-ARI schools for Black and Hispanic students as well as for White and
Asian students (Table 2).

Cohort 1
Cohort 2 44.1%
Cohort 3 45.3%
Cohort 4 45.1% 35.3%
Cohort 5 45.1% 36.3%
Cohort 6 44.1% 37.0%
Cohort 7 44.4% 33.9%

A comparison of the SAT data for Grades 3 and 4 reading proficiency levels from FY1999 to
FY2006 revealed that ARI schools outperformed non-ARI schools 6 out of 8 years, 75% of the
time. The greatest difference in the performance of ARI schools over non-ARI schools occurred
in 2003. In that year, ARI schools posted a 4.7% increase in the SAT over a 2.9% increase by
non-ARI schools in the same year. The only year that the non-ARI schools outperformed ARI
schools was 2004, non-ARI schools posted a 1% increase in reading proficiency over a -0 .2
decrease in proficiency in ARI schools. Table 3 presents a descriptive representations of the
comparative data.

7
Table 3. Descriptive Statistics Grades 3 and 4 SAT Test Gains ARI and Non-ARI Schools

Alabama Reading Initiative

Stanford Achievement Test Gains
Reading Comprehension (3rd and 4th Grades)
Non ARI Schools vs. ARI Schools

Q)
b.O 3
"'s::
+J
Not ARI
Q)
u
.... 2 ARI
Q)
Q.
>
cc 1

-1
1999 2000 2001 2002 2003 2004 2005 2006

The next set of data analyzed was the ARMT reading proficiency data from FY2003 to FY2006 .
The ARMT was a criterion-referenced assessment administered statewide to gauge student
proficiency levels in mathematics and reading . The test was also used to determine annual
measurable objectives (AMOs) for schools statewide . Proficiency levels were based on the
percentage of students scoring on Levels 3 and 4 on the ARMT. Descriptive data for the ARMT
in Phase I indicated increased proficiency levels in each year that the test was given beginning
with the 2003-2004 pilot year. Based on descriptive data , the sixth grade yielded the highest
reading proficiency level in 2003-2004, while the eighth grade posted the lowest proficiency level
at 57 .34% . The next year 2004-2005, the fourth grade read ing proficiency ranked highest at
83.34%. In the same year eighth grade reading proficiency was 69.51%, an increase of 12.21%
over the previous year. In year 3 of the ARMT, fourth grade proficiency remained high and
showed a 1% increase from the previous year while eighth grade proficiency showed a 2.24%
increase. Table 4 shows Grades 3 to 8 reading proficiency levels from 2003 to FY2006 based
on the ARMT

8
Table 4. Descriptive Statistics , Grades 3 to 8 ARMT Reading Proficiency Levels 2003 to 2006
- --~ -----
--~ ~-- -- -- - - - - - - - - - - - ----
-~ - - - - - -- -- - - -- -- ---- ----
School Year Grade Pet Proficient
2003-2004 4 76.95
6 82.43
8 57.34
2004-2005 3 95 .92
4 83.34
5 80.93
6 81.65
7 75.15
8 69.51
2005-2006 3 83 .65
4 84.04
5 80 .71
6 83.08
7 74.34
8 71 .75
2006-2007 3 84.09
4 85.01
5 84.06
6 84.87
7 77 .00
8 71.94

ARI Phase II Implementation

ARI Phase II implementation began in FY2007 and continued through FY2011 . In 2007,
Governor Bob Riley and the state legislature provided funding to support a reading program for
every K-3 school in the state. With that commitment from the governor and the legislature, Dr.
Joe Morton, then State Superintendent of Education , announced in a September 18, 2007 memo
that "All state-funded ARI reading coaches should contain their efforts in Grades K-3 ." The ARI
had been in existence for nine years when 2007 National Assessment of Educational Progress
(NAEP) revealed that Alabama students in Grade 4 made more gains than any other state in the
history of the assessment.

During Phase II , a secondary pilot project involving 14 schools with combinations of Grades 4-9
began. The name of the pilot was the Alabama Read ing Initiative Project for Adolescent Literacy
(ARI-PAL) . The ARI-PAL schools were all locally funded and they committed to implementing a
whole school reform model, including a full-time literacy coach and an intensive intervention
program for struggling readers. The ARI-PAL did not expand in its second year of
implementation . According to the American Institute for Research (2006) , this was due in
part to a recognit ion of the need to provide additional tra ining and professional development
focused on integrating and implementing reading across all content area courses .

9
The second cohort of locally funded ARI-PAL schools was added in 2008-2009.The ARI
secondary regional coaches provided weekly support to Cohort 1 and Cohort 2 ARI-PAL
schools. A third cohort consisting of low-performing schools that qualified for the federal School
Improvement Grant (SIG) funds was added in 2009-2010. The SIG specifications included
contracts between schools and external professional development providers which limited the
extent of ARI support in SIG schools. Additionally, the ARI state and regional staff were divided
into 11 regional teams corresponding with the 11 regional in-service centers. The ARI state staff
members became team leaders and coordinated the K-12 literacy work in the regions . The
primary delivery of professional development geared toward school leadership teams was titled ,
"Literacy and Justice for ALL. " The ARI Phase II ARMT descriptive data revealed a steady growth
in reading proficiency levels for Grades 3 to 8. Reading proficiency levels for Grades 3, 4, 5 and
6 remained at or above the 84% , Grade 7 proficiency levels ranged from 71% to 83%, Grade 8
reading proficiency levels remained above 70% (Table 5).

Table 5. Descriptive Statistics, Grades 3 to 8 ARMT Reading Proficiency Levels 2007- 2011
School Year Grade Pet. Proficient
2007-2008 3 85.35
4 86 .09
5 84.37
6 86.61
7 79.13
8 73.87
2008-2009 3 85.83
4 86 .5
5 84.65
6 86 .02
7 81.28
8 74.89
2009-2010 3 86 .59
4 86.69
5 85.64
6 86 .13
7 83.02
8 73.51
2010-2011 3 86.82
4 87.93
5 88.59
6 88 .19
7 84.09
8 76 .95
2011-2012 3 88.16
4 88.16
5 88.88
6 88.35
7 86.25
8 79.33

10
 ARI Implementation Phase Ill: FY2012- FY2015

ARI Phase Ill implementation spans FY2012 to present. Phase Ill introduced changes to the role
and focus of ARI which was modified to align with the new goals of the ALSDE. The newly
appointed State Superintendent of Education, Dr. Tommy Bice, in a June 12, 2012 memo to
schools and systems stated 'With the ARI FY 2013 allocation , you have the opportunity to
exercise flexibility to provide coaching support in the schools that have the greatest needs in
Grades K-12." Dr. Bice further stated "Therefore, SDE support in your system and schools may
look different from previous years in that it will be differentiated and customized based on YOUR
needs." Subsequently, the ALSDE introduced the concept of Regina! Planning Teams (RPT)
and Regional Support Teams (RSS). The goal was to provide a "one voice" support approach
from the ALSDE to the local level. Although, ALSDE provides differentiated support to all public
schools in the state of Alabama, the level and intensity of that support is currently dependent on
the unique needs as determined and requested by each school. Thus, the support structure of
ARI involves collaboration with representatives from other sections within the department.
ALSDE also adopted the use of ACT battery of assessments for Grades 3 to 11 . In addition, Dr.
Bice stated, "The SDE recognizes that some systems have comprehensive assessments in
place that monitor the progress of young learners. Should you have such an assessment in
place, your system may request to opt out of using the state-funded DIBELS." This essentially
ended the previous practice requiring the use of the DIBELS for monitoring K-2 reading growth.

ARI Program Evaluation Question 2. Are there any differences in Phase I and Phase II
reading proficiency levels based on a comparison of the ARMT?

An independent samples t-test was conducted to compare the means of ARMT reading
proficiency data from Phases I and II. The assumption of the homogeneity of variances was
assessed by the Levene test, F = 6.353, p = .040; this indicated that equal variances could not
be assumed ; therefore the unpooled variances (equal variances not assumed) version of the t
test was used. There was no statistically significant difference between the means of Phase I
and Phase II, t(6)= -2.772 , p = .094, two-tailed (Tables 6 and 7).

Table 6. Independent Samples t- Test Group Statistics, ARMT Reading Phase I and Phase II
ARI N Mean Std. Deviation Std. Error Mean

Percent Proficient 3 I, .7398 .05175 .02988

6 .8255 .01958 .00799

11
Table 7. Independent Samples t-Test ARMT Reading , Phase I and Phase II
I
t-test for Equality of Means

F
1
Levene's Test for
Equali!Y_of Variances I
F Sig . It df Sig . (2-
I I I tailed)
I Percent Equal variances 1 6.353 ! .040 I -3.762 7 .007
Proficient assumed I I
I Equal variances not I -2.772 2.292 .094
I assumed I I I

Further calculations using Cohen's d was conducted , d =2.19. Although there were no
statistically significant differences, the Cohen's d calculation revealed the presence of practical
differences between Phase I and Phase II reading proficiency levels, indicating that continued
ARI instruction will have a convincingly positive impact on reading proficiency.

ARI Program Evaluation Question 3. Do differences exist in reading proficiency levels with
regards to Phase I, Phase II , and Phase Ill based on NAEP reading proficiency levels?

The NAEP is administered every two years to only fourth and eighth students. NAEP results are
reported comparatively by presenting state and national side-by-side. The test proficiency levels
are reported as basic, proficient, and advanced. For the purpose of the current program
evaluation , proficiency level was determined by adding the percentage of students at or above
the proficient to percentage at advanced levels. Proficient levels for the three phases were then
compared to see if any trends emerge. Alabama made significant gains in reading , increasing
13 points from a scaled score of 207 in 2003 to a scaled score of 220 in 2011. In 2007, Alabama
broke NAEP records , increasing eight points in one year, more than any other state in the nation.

In a 2007 press release Governor Riley recounted comments from the U.S. Secretary of
Education Margaret Spellings who said, "I want to be the first to congratulate the state of
Alabama for leading the nation in 4th grade Reading gains." The Secretary of Education added
"Combining proven methods of instruction with hard-working students and dedicated teachers
has paid off with reading scores rising eight points in just two year. It's phenomenal. " (ALSDE ,
2007). The pictures below show Dr. Joe Morton , the Alabama State Superintendent of Education
and Governor Bob Riley presenting the 2007 NAEP Reading Report.

12
In a September 25 , 2007 , press release , Governor Bob Riley said ,

Alabama is number one in the nation in reading improvements at the fourth grade
level. This didn't happen by accident. This is part of our game plan to improve
education in Alabama , and this is proof positive that our game plan is working. The
progress we have made in just a few years is truly remarkable , and if we keep our
focus on funding the programs , we will continue to record these types of amazing
gains. I want to thank all the teachers because they are doing exactly what we asked
them to do. They know how incredible our Alabama Reading Initiative is and that one
of the reasons it is so successful is because it provides support for our teachers. They
know we must continue to expand it to all grade levels, and we will. (ALSDE , 2007).

The director of the National Center for Education Statistics, Mark Schneider, stated "Alabama's
gain in fourth grade scores is higher than any other state between 2005 and 2007 ." This gain
stands out for this year's assessment, and in the history of NAEP's state-level Reading
Assessment. " This sentiment was shared by State Superintendent of Education Joe Morton who
said "We're number one on the list of fourth grade Reading improvement in the nation, and it is
not because the states are listed alphabetically." Ms. Sherry Calvert, Principal of F.E. Burleson
Elementary in Hartselle said "The impact ARI has had on our school has been unbelievable.
When I asked my faculty what they attribute our positive change in performance, the answer
was the training ARI provides in reading comprehension ." (ALSDE , 2007).

Although the 2009 Grade 4 NAEP reading scores remained the same , it was not a setback
because of the increased rigor on the NAEP that year. The Alabama 2011 Grade 4 NAEP
reading scores increased to 220, equaling the national average for the first time. Alabama was
commended nationally as one of four states in the nation to show significant gains in reading
scores. Alabama's growth was second only to Maryland and was ahead of Hawaii and
Massachusetts in this group of high performing states. The Alabama Grade 8 NAEP reading
showed a five-point increase from 2003 to 2011 , the sixth largest growth in the nation. The
governor and state education leaders once again celebrated the NAEP results , which they
attributed to statewide efforts led by ARI .

By 2015 the NAEP reading were not as positive , showing a 3 point decline in the fourth grade
from a 220 state average in 2011 to a 217 . This left Alabama's Grade 4 NAEP scores 4 points
below the 2015 national average of 221. However, the eighth grade proficient levels for the
same time period held steady at 28%. The data seem to indicate that as the grade spans for
which the ARI was asked to provide support while the funding for doing so decreased , the NAEP
reading proficiency scores declined .(Tables 8 and 9).

13
 Avg. Avg. Pet. Pet.
2015 217 221 65 29
2013 219 221 65 31
2011 220 220 67 31 7
2009 216 220 62 28 6
2007 216 220 62 29 7
2005 208 217 53 22 4
2003 207 216 52 22 5
2002 207 217 52 22 4
1998 211 213 56 24 4

Avg. Avg. Pet.

2015 259 264 71
2013 Ill 257 266 68 25
2011 258 264 69 26
2009 255 262 66 24
2007 II 252 261 62 21
2005 252 260 63 22
2003 253 261 65 22
2002 253 263 64 21
1998 255 261 67 22

Inferential calculations using a one-way ANOVA comparing Phase I, Phase II and Phase Ill state
level NAEP reading scores was computed. No significant difference was found in the NAEP
reading scores f(2.,6) = 4.234, p =.07 . Tables 10 and 11 present the result of the ANOVA.

Table 10. ANOVA on NAEP Readin Percent Proficient, Phase I, Phase II , Phase Ill
Sum of Squares df MeanS uare F ----1--- S....,
ig._.- - - 1
Between Groups I
19.639 2 9.819 4.234 .071
Within Groups I 13.917 6 r- 2.319 ____,,__

Q otal 133.556 ~8 : --- :

--

14
Table 11 Descri tive Statistics NAEP Readin
J J Percent Proficient Phase I, Phase II, Phase Ill -
'Phase N Mean Std . Deviation Std . Error
I 4 21 .7500 .50000 .25000
II 3 23.6667 2.51661 1.45297
Ill 2 25.5000 .70711 I .50000

Total 9 23.2222 2.04803 .68268

ARI Program Evaluation Question 4. Are there differences in reading proficiency levels based
on Phase Ill ACT Aspire reading proficiency levels?

Table 12 shows a descriptive comparison of the ACT Aspire 2013-2014 and the 2014-2015
reading data for Grades 3 to 8. The data revealed statewide reading proficiency levels for both
years stayed under 50%. Grades 3, 4, and 5 showed no improvements, sixth grade showed a
1% increase whereas Grades 7 and 8 reading proficiencies declined from 2014 to 2015 .

Table 12. Descriptive Comparison ACT Aspire 2014 to 2015 Reading

ACT Aspi r e 2014 to 2015 Reading C omparison

50% - %

43%
42%

40% 38% 38%

35% 35% 35%
c:- 34% 34% 34%

...
c:::
<=> 30%
=e
......
20%

10%

0 %
3rd 4th 5th 6th 7th 8th

- 2014 2015

ARI Program Evaluation Question 5. Do differences exist in Phase Ill reading proficiency
levels as measured by ACT Plan reading proficiency percentages?

An analysis of reading proficiency using the Grade 10 ACT PLAN data revealed a decline from
39% proficient in 2013 to 31% in 2014, representing an 8% decline. The ACT PLAN is a norm-
referenced ACT test generally viewed as preparation for the ACT Test. The ACT provided by
the state to all students in the 11 th grade is also widely accepted by most colleges for college

15
admission purposes. Table 13 is a display of ACT Plan descriptive data for 2013 , 2014 , and
2015 .

Table 13. Descriptive Statics ACT Plan 1Oth Grade Reading Proficiency 2013, 2014, and 2015
Reading Proficiency Pet.
2013 39
2014 31
2015 32

ARI Program Evaluation Question 6. Do differences exist in ARI funding implementation,

scope , and structure linked to Phases I, Phase II , and Phase Ill?

Funding analysis from ARI Phase I, Phase II , and Phase Ill seem to suggest changes in the
funding , implementation, scope and structure have negatively impacted ARI. In FY1999 , theARI
was not funded , but the initiative was present in 16 literacy sites . The LEAs paid for their reading
specialists. Schools had to apply to become literacy demonstration sites. Commitment from
schools included maintaining a demonstration site with faculty commitment and interest,
committing to 10 day initial training , adjusting reading instruction to meet best practice research ,
and participating in a 10 day initia l professional development with additional professional
development to follow.

By FY2000 , the ARI budget was $6 million. There were 81 ARI schools with funding for 81
reading specialists in the literacy demonstration sites. Regional coach trainers provided state-
wide professional development as support for literacy demonstration sites. Schools continued
to apply to become ARI schools , agreeing to commit to 100% literacy by participating in
extensive professional development, adjusting reading instruction, and training teachers and
coaches in foundations of beginning reading .

From FY2001 to FY2004, the ARI remained a K-12 initiative and the ARI budget increased yearly
as new schools joined the initiative. By FY 2003 , the ARI budget was $12 .5 million and there
were 450 ARI schools. The commitment agreement from LEAs and schools rema ined in place.
The state fund ing provided one reading specialist for each participating LEA, with incentives to
LEA's with multiple participating schools. The ARI budget increased further from $40 million in
FY 2005 to fund 511 ARI schools to $55 million to fund 752 ARI schools by FY2006 . ARI tra ined
schools received $64,167 to hire a school-based reading coach . During that period , 32 regional
reading coaches and principal coaches provided support to teachers, coaches, and
administrators through professional development, on-site visits to schools, coaching orientation/
internship, beginning reading research , and additional coaching commun ities for coaches to
develop and refine their coaching practices. The ARI schools from 1998 to 2002 underwent a
thirty-hour recertification professional development. Additionally, schools that were not making
sufficient progress participated in additional retooling to refocus and hone the instructional skills

16
and practices. DIBELS testing with progress monitoring was required for all K-2 students and
paid for by the ARI so that schools had a way to assess growth in reading.

At the beginning of Phase II , FY2007 , as the number of trained ARI schools grew, the ARI budget
was fifty-six million dollars. There were 899 ARI schools (all K-3 schools), however, funding for
hiring school coaches dropped from $64,167 to $55 ,000 . The school-based coaches were
supported by 96 Regional Principal/Reading coaches. School-based coaches continued to
follow the job description developed by the Alabama Reading Initiative, attend required
professional development, and spend 50% of their time working with teachers to improve reading
instruction. School-based coaches were also required to provided principals, as well as regional
coaches, with schedules and coaching plans as part of their responsibilities. They worked with
grades K-5. DIBELS progress monitoring provided teachers and coaches with data to determine
student needs. New Teachers were trained every year in Foundations of Reading. Students
who began in an ARI school as first graders in FY2004 were fourth graders in FY2007. The
Alabama Reading Initiative Project for Adolescent Literacy (ARI-PAL) began and 6 secondary
coaches were hired to work in 14 middle schools .

The ARI budget fluctuated from over sixty-four million dollars in FY 2008 to $59 ,609 ,850 by the
close of FY 2010. There were also fluctuations in the numbers of school based coaches,
Regional Staff, operational structure and scope and funding allocations for hiring school based
reading coaches. The state increased funding allocation for hiring school-based coaches from
$55 ,000 to $66,581. School-based coaches were directed to concentrate their efforts in K-3 ,
continue to follow the job description, attend professional development with travel and per diem
paid , and support teachers in the classrooms and with data driven progress monitoring. The
ARI budget for FY 2009 dropped slightly from $66 ,581 million the previous year to sixty million
dollars with proration. The allocation for hiring reading coaches was also cut by one thousand
dollars from the previous year, but the Grade 4 NAEP reading proficiency did not show any
growth rather it remained the same. Alabama students scored four points below the National
Average . Again , in FY 2010, the state budget was reduced further to $59 ,609 ,850 due to
proration. The allocation for hiring school-based coaches was once again cut from $65 ,581 to
$61 ,587. Regional Coach Staff was cut from 90 to 80. The structure of ARI was again changed
and support went to back to K-12. This was the last year for concentrated support for K-3 and
ARI-PAL.

By Phase Ill, the ARI budget was cut from $65 million in FY2011 , down to $48,212,043 in
FY2015. There were 920 ARI schools, 788 of them were K-3 schools. Schools continued to
receive $61 ,587 per coach allocation . LEAs were asked to develop a plan for possible expansion
of coaches beyond third grade per State Superintendent's directive. The number of Reg ional
Coaches dropped from 80 in Phase II to 55 in Phase Ill. The school and regional coach job
descriptions became less specific. LEAs and schools were still asked to adhere to the coach's
job description , although with the drop in coaching allocations, schools had to supplement the
salaries of coaches, locally coaches were increasingly required to take on additional duties

17
beyond their job description. LEAs chose to opt out of using state-funded DIBELS for monitoring
student progress. Allocations available for hiring coaches were further slashed from $61 ,587 to
$57,325. LEAs had the option to determine where to place coaches K-12. Regional support
coordinators were hired to coordinate efforts among the various regional support staff in each
region . The work of the 61 ARI regional support staff was directed primarily by the Regional
Support Coordinators with support from the ARI. Despite cuts to its budget and manpower, ARI
was asked to expand its roles and responsibilities from a 100% K-3 literacy focus in 920 schools
to a K-12 literacy in more than 1400 schools. In addition , reading coach support was no longer
restricted to reading teachers and reading classrooms. Many reading coaches were reclassified
as instructional coaches and expected to support other content area classrooms/courses. Table
14 is a descriptive representation of ARI budgets from FY2000 to FY2005 .

Table 14. ARI Funding 2000 to 2015

ALABAl\V\. R_.<U>l1"G ThTIL~ Tl'\""E FUl'-"D:DTG

FY 1999- FY 1015

YEAR STATE APPROP:RIA.TION AJUSCHOOLS ARFl FEDERAL ARJST.AFF

FUNDS
FY 1999 $ 0 16 .uti 1 State Sa.ff
.:c:e.iv ed S 1,.500, 000 from
p ri'i.-ai:e fundin!!!:)
n.1ooo $6,000,000 81 ..uti 3 State Sa.ff

n. 1001 $10,.Z40,000 167 ARJ 5 State St4

n. 1002 $11,300,000 4Z3 ..uti 1 1 Re-,g t ona.l

FY 1003 s1:::,soo,ooo 460 ..uti 516,106,.59:! 2 6 Re-gi c na.l

F\.'1004 $12,.500,000 485 ARI $18,0S:Z, 502 31 R.e!Pon.a.l

74 .'\.RFI S<:bools
n.2oo~ $40,000,000 511.-uli $19,040,035 40 Regional
75 ARFI S<:bools
FY 2006 $55,000,000 752.-uli $19,.373,141
93 ARFI S<:bools
FY 2007 $56,000,000 899 (all K -3) ARI $19,.301,943 103 Region.al
94 .-\RFI S<:bools
FY 2008 $64,404,704 902 (all K - 3) .-'UU $17,969,131
95 .-\RFI S<:bools
FY2009 $69,443,081 914 (793 K-3) ARI s 7.,308..264
93 .-\RFI S<:bacols
FY 2009 $61,804,.3 4::: 914 (793 K-3) ARI s 7,.308..254
Prorat ..d 93 .-\RFI S<:bools
(11% )
FY 2010 $64,443,081 no (788 K-3) ..uu S 2,000,000
(<:a.rry forward,
approxilnate)
FY 1010 $59,609,850 910 (788 K-3) ..uti s 2.000,000
Prorated (<:a.rry forw:~d,
(7.5 %) aooroxiln.a.te
FY lOll $59,95Z,.360
Current ARI Staff
S~:ate.Staff
FY 201Z $56,847,:257
1 Pr~m c.oordmaror
3 Ad:min.istrators
n. 101.3 $58,458,789 11 Spe<iafuts
3 ClHicals
F\.' 2014 $48,153,789
cS8 Regioaal Staff
F\.'2015 $48,.Z1:!,043
$.4-S llliltion to LE.As for approrinuotely 78~ loocal coaches

Note. In FY 1999 ARI received $0 state allocation , but there was a $1 ,500,000 received from private funding .

18
In order to address evaluation questions 7 and 8, it was necessary to gather qualitative input
from teachers and coaches charged with the implementation of ARI. Two electronic survey
instruments were open for responses from January 1-15, 2016 . The surveys were entitled ARI
Program Evaluation Teacher Survey and the other was the ARI Program Evaluation Coaches
Survey.

ARI Program Evaluation Question 7. Are there differences in teachers' perceptions of how
the changes in ARI Phase I, Phase II , and Phase Ill impacted the effectiveness of the program?

A total of 5,11 0 teachers completed the ARI Program Evaluation Teachers ' Survey instrument.
Teachers who completed the survey had classroom experiences ranging from 0 to 17+ years of
experience. The demographic breakdown based on experience were as follows : a total of 1 ,940
(38 .2%) had 17+ years; 1,042 (20.4%) had 12 to 16 years ; 902 (17.68%) had 8 to 11 years of
experience.

There were 2,161 teaches who self-identified as Phase I teachers. These teachers indicated
that they had participated in one or more of the different types of ARI facilitated professional
development. In response to the question item on the survey, a total of 1,142 (58%) out of 1,898
respondents indicated that they participated in ARI Phase I training. Additional breakdown of
Cohort I teachers showed that fifty seven received daily/weekly support from a building reading
coach , but seven reported they never received coaching support. The number of teachers who
self-identified as Phase II teachers was 987. In this group , fifty percent reported receiving support
from the building reading coach whereas 9% reportedly never received building reading coach
support. Survey results also showed 600 self-identified teachers completed the survey in Phase
Ill. Forty seven percent of the teachers received coaching support whereas sixteen percent
never received building reading coach support.

Teachers who received daily/weekly coaching support thought it was wonderful. Looking back
on the type of support received , one teacher stated "My reading coach made herself available
for consultation RE : better ways to meet student needs. She did frequent walk-throughs and
offered helpful constructive criticism and suggestions for improvement to me, as a new teacher.
She taught me how to teach reading to Title I children who lacked background knowledge and
limited experience with text. She was invaluable ."

By Phase Ill the percentage of teachers who reported that they received support from the
building reading coach had declined and the percentage who stated that they never received
support from the building reading coach was double that of respondents from Phase I. When
comparing the reading coach support teachers received in the past to the reading coach support
teachers currently receive , one teacher revealed , "I am not aware that we have had one SINCE
2006 , but I would have loved the support. I went to a wonderful workshop at MEGA 2015 and
just had a workshop at our school through the school system . I want to be trained in ARI. I had
a bldg . coach prior to 2006 which was great" The fact that th is teacher and others in her building

19
have not had building reading coach support should be of great concern to decision makers
whose actions or inactions created th is environment.

Table 15. Percentage of Teachers Who Received Support from the Building Reading Coach.
Frequency of Building Cohort I Cohort II Cohort II
Reading Coach
Support
Daily/Weekly 57 50 47
Never 7 9 16

Further analysis of the survey revealed the number of teachers who participated in different
types of ARI facilitated professional development such as the Alabama Reading Academy, ARI
recertification, the two-week initial training for new teachers , and the one-week initial training
has dropped astronomically while incidences of school level professional development provided
by the building level reading coach has become the norm.

Table 16. Percentages of Teachers Who Participated in ARI Facilitated Professional Development.
Type of Professional Cohort I Cohort II Cohort Ill
Development
Two Week Initial Training 58.68 20.34 6.64
One Week Initial Training 25.24 43 .31 41.69
School Level Training by 52 .29 60.02 59.14
Building Reading Coach
Alabama Reading Academy 17.98 8.07 3.65
ARI Recertification 17.93 5.23 4.49

The problem with this trend seems to be summed up in teachers' comments such as, "I feel that
ARI is a natural process that ALL teachers should be using. The only change I see, is schools
are not offering more ARI training for new teachers and they are not getting the benefit of ARI. "
This reduction of training opportunities has negatively impacted the percentage of teachers who
have participated in ARI facilitated professional development, "We used to have a full-time ARI
coach that would help supplement our classes with reading materials and provide professional
development sessions throughout the year. She was a fantastic part of our staff, but her position
has since been moved from our high school to our junior high school. "

Building level professional development facilitated by building level reading coaches appears to
happen so rarely that one teacher stated , "I feel it became virtually extinct way too quickly. We
were observing our students making great gains in comprehension and were celebrating as
more and more of our students developed a love for reading that had not been there before.
Then the whole initiative, support, and funding seemed to stop , sadly."

20
Teachers identified as Cohort II teachers, 525 (60%) of 875 used the following negative
descriptors in response to the same question: dwindled and lessened overall support, diminished
coaching support and fewer available resources. About 16% of the respondents indicated that
they no longer receive coaching support. Cohort Ill respondents indicated that they had no
frame of reference pertaining to whether the changes to ARI had any effect on the program
effectiveness.

ARI Program Evaluation Question 8. Are there differences in coaches' perceptions of how
the changes in the ARI phases has impacted the effectiveness of the program?

Demographic data from the ARt Program Evaluation Coaches Survey yielded the following data:
681 (88%) of 770 ARI coaches statewide participated in the survey. A breakdown of the
coaching experience indicated 344(53.25%) had one to five years, 161 (24.92%) had six to ten
years, 141 (18.82%) had 10+ years of experience as reading coaches . Among the reading
coaches, 449 (66%) reported they were able to provide support to teachers as reading coaches,
literacy coaches, instructional partners, and intervention specialists. Conversely, 232 (44%)
reported they were used in "other" capacities. It was very important to get the input of the reading
coaches because building reading coaches were hired to provide on-going, on-site coaching
support to teachers and struggling readers . Historically, the effectiveness of building reading
coaches directly impacted the success of teachers and students at the school level. This support
helped to undergird the effectiveness of the ARI as an initiative.

An analysis of the demographic composition of survey respondents revealed that there were
312 (51%) building reading coaches who self-identified as Phase I Coaches. Phase I building
reading coaches indicated that they had participated in one or more ARI facilitated professional
development. The input gathered from building reading coaches as it relates to the changes in
ARI were mostly negative as captured by some of the comments: "When we first began we had
lots of support from regional coaches and principal coaches; over the years that support faded ,
there have been too many changes and some of the good things we used to do have been
tossed away." The coaches seemed to suggest that compromises made in the structure of the
ARI have had far reaching and ongoing consequences for reading proficiency in the state. One
coach pointed out, "The data shows the negative impact of pulling coaches out of elementary
school, hence the gap we are seeing in middle schools which will transfer to the high school. "
Another building reading coach seemed to summarize the sentiments of other Phase I building
reading coaches stating , "When we first began , we had lots of support from regional coaches
and principal coaches, over the years that support faded , there have been too many changes
and some of the good things we used to do have been tossed away." These coaches strongly
indicate that they believe the cost savings measures and consequent changes to ARI will not
serve Alabama teachers and students well in the future.

Survey data from building reading coaches who self-identified as Phase II coaches had positive
things to say about ARI as an initiative, but there seemed to be negative perceptions of the

21
changes in ARI over time. One of the building reading coaches stated "ARI has enhanced my
knowledge of best practice for the purpose of increasing reading instruction ." However, another
reading coach pointed out that the lessons learn from ARI pertaining to best practices in
delivering instruction were gradually being eroded . "When all teachers were being held
accountable for delivering the strategic teaching strategies with small groups, we saw a positive
impact on student learning according to data. I feel that over time, many teachers have gotten
away from small group instruction and strategic teaching strategies used in teaching foundation
skill."

One of the coaches stated , "I do not believe that one (or two) individuals can effectively support
more than one school. In order for all schools to benefit from having an instructional coach , a
full-time coach needs to be present and available to assist at all times at each school. I believe
that since I have been pulled to work at the middle and high school , my elementary school has
suffered . I believe this is due to my rotating schedule . I have just enough time to begin to
support a teacher when I have to leave and go to another school. As good as the concept of
using elementary coaches (who have a strong ARI foundation) and who know the kids is, it is
not realistic to support that many schools effectively." The implication is that although the
concept of using coaches as needed might appear to make practical sense, it does not mean
that it will be effective in the long run .

Some comments made by the building reading coaches yielded valuable insight into the
coaching reflection and sense of their professional efficacy. A reflection from one of the coaches
indicated that there is a lack of accountability on the part of coaches and principals, which
hinders the support that teachers and students receive from the building reading coach . The
coach who offered this candid insight stated , "I don't feel as accountable as I once did and can
tell that I am not as quick to carry through with one of our plans ... I actually liked being held
accountable as I found that myself, my principal and my supporting ARI coach was on the same
page. My intentions are there but it's difficult to follow through unless the principal feels a sense
of accountability as well. " It is evident that the loss of coaching accountability has produced
consequences for the students , teachers, reading coaches, and principals.

The number of building reading coaches who self-identified as Phase Ill reading coaches was
130. These coaches were hired between 2011 and 2015 , thus some had fewer than three years
of experience as reading coaches. Based on the number of years that they have been reading
coaches, it appeared that this group of coaches might not be able to compare the quality of the
ARI support in the past to the present, as well as comment on changes that have occurred in
the ARI over the years. An analysis of the survey data from Phase Ill coaches yielded mixed
results . One of the reading coaches stated "In some respects I love the changes to ARI! " But
others disagreed. "I think some of the best teaching I saw was when teachers were going through
the rigorous professional development and certification process of the ARI. The best practices
learned, applied , and monitored were helpful not only as a professional community, but most
importantly on the impact of our students. New teachers and veteran teachers would benefit

22
from having more support and more ongoing professional development from ARI on best
practices and the implementation of effective reading strategies. I wish there was a stronger
presence of this in our systems and our schools." Another building coach seemed to appreciate
the work done by ARI , but suggested that more support from ARI would be beneficial. That
teacher noted, "I believe ARI has had a great impact in our schools. I would love to have more
support from them on-site. I do feel, however, that trying to serve a K-12 school is very
challenging. I would love to see more funding for additional coaches in each school." Altogether,
building reading coaches in Phase I, Phase II , and Phase Ill agree that the support from ARI has
been invaluable; however, the building reading coaches appeared to suggest that the changes
over time have negatively impacted the effectiveness of the program .

Cohort Ill coaches indicated that they did not have any frame of reference to compare the impact
of the changes in ARI due to the fact that they have only been coaches during the Phase Ill time
frame . Overall , the coaches were appreciative of ARI and the interactions with reg ional staff
and ARI personnel. Nonetheless, the coaches revealed that administrators did not give them the
support needed to effectively execute their job description. As one coach stated , "We are used
for other duties, not for the purpose of reading/instructiona l coaches. " Again , the coaches
negatively reported on the lack of coaching emphasis in the lower grades. In the words of one
respondent , "ARI will be more effective if additional regional coaches were available. Support is
spread too thin amongst several schools ."

Summary of Findings

The overarching question for the current ARI program evaluation sought to determine the extent
to which ARI has maintained its effectiveness in its current configuration . Variables examined in
order to address the key objective included comparisons in Phase I, Phase II , and Phase Ill
reading proficiency levels determined by state level SAT, ARMT, ACT Aspire , ACT Plan , and
NAEP reading proficiency levels. This evaluation was designed as a quantitative, causal-
comparative program evaluation . This section of the evaluation presents a summary of the
findings , conclusions, discussion, and recommendations for practice and future ARI program ~
evaluations. The overarching question sought to determine if ARI has met its goal of 100 %
reading proficiency. State level 2015 reading proficiency data from ACT Aspire, ACT Plan and
NAEP suggest the mission has not been achieved . The fact that the mission has not been
accomplished is by no means an indictment of the ARI. Rather, it appears likely that the ARI is
perhaps a victim of factors beyond its control.

The next question addressed was "What differences exist in the ARI Phase I, Phase II , and
Phase Ill statewide reading proficiency levels?" Several changes have occurred in the statewide
reading assessments utilized over the course of the ARI's existence. The only test administered
in all three phases has been the NAEP. For that reason reading proficiency trend comparisons
could only be made util izing the ARMT for Phases I and II while NAEP proficiency levels were

23
used to compare all three phases. However, each of the assessments yielded snapshots of
reading proficiency levels for the given phase. Additional details regarding reading proficiency
levels were addressed in subsequent program evaluation questions.

Additionally, the program evaluation sought answers to "Are there any differences in ARI phase
I, Phase II , and Phase Ill reading proficiency levels based on a comparison of the ARMT?"
Results of the Independent Samples t-test indicated that there was a significant difference
between the Phase I and Phase II ARMT reading scores. Phase I ARMT reading mean scores
were significantly lower than Phase II ARMT reading mean scores. Additional computations of
Cohen's d also indicated the presence of practical differences in reading proficiency levels
traceable to Phases I and II.

The next question examined "Do differences exist in ARI phase I, Phase II , and Phase Ill reading
proficiency levels based NAEP reading proficiency levels?" A one-way ANOVA comparing
Phase I, Phase II and Phase Ill NAEP reading state level scores failed to yield statistically
significant difference was found in the NAEP reading scores in all three phases. Following that,
the evaluation sought to determine answers to the following , "Are there differences in reading
proficiency levels based on Phase Ill ACT Aspire reading proficiency levels?" State level
comparisons of the 2014 and 2015 Aspire reading proficiency for Grades 3 through 8 have
shown no improvement. The ACT Aspire reading proficiency for Grades 3 to 8 have remained
under fifty percent since 2014. Reading proficiency levels remained stagnant in Grades 3, 4,
and 5. There was a 1% increase in sixth grade, a 1% drop in seventh grade and a 4% drop in
eighth grade.

Further investigation sought to answer the following question , "Do differences exist in Phase Ill
based on 101h grade ACT Plan reading proficiency levels?" An analysis of the data revealed state
level Grade 10 ACT Plan reading proficiency levels fell from 39% in 2013 down to 32% in 2015 .

Finally, the evaluation sought answers to the following question, "Do differences exist in Phase
I, Phase II , and Phase Ill ARI implementation , scope, structure, and funding?" Evidence
uncovered far-reaching differences in the implementation, scope , structure , and funding in all
three phases of the ARI. In Phase I, the ARI implemented rules that schools were required to
abide by in order to receive ARI implementation support. The ARI also had a K-3 focus in 752
schools, and its structure was driven by job-embedded professional development as well as
side-by-side coaching support provided by highly trained reading coaches. By the end of Phase
I, the ARI budget was more than seventy four million combined state and federal Reading First
funding .

Phase II brought about an expansion of scope and structure from elementary to secondary
schools. The expansion brought about a major shift in the implementation guidelines, scope,
structure and funding of the ARI. At the end of Phase II , ARI was charged with supporting 920
K-12 schools with a shrinking budget of under sixty million dollars. Also , due to funding issues,

24
school districts were given the flexibility to use reading coaches as deemed appropriate. That
flexibility in practice meant that coaches were no longer solely focused on coaching teachers
and working with struggling readers . Hiring , retention, and placement of coaches was left to
LEAs. Phase Ill ushered in even more changes in the scope , implementation , structure, and
funding of the ARI. The ARI support to LEAs was subsumed within two bigger ALSDE initiatives,
the Regional Planning Support Teams (RPT) and the Regional Support Teams (RSS). This
further diluted the role of reading coaches . In ARI phase I, reading coaches were required to
provide daily/weekly support and professional development to teachers ; in Phase II, the
language from ALSDE changed from "required " to "may." Implicitly LEAs were permitted to
cherry-pick what aspects of ARI guidelines they were comfortable with implementing. Reading
coaches, for example, were used to various degrees as quasi-administrators, a far-cry from the
original intent of providing side-by-side to teachers and working with struggling readers.

Furthermore, the ARI changed from its intensive focused systematic support role to an on-
demand supportive role to more than 1400 schools. The problem was further complicated by
cuts in the operating budget of the ARI , down from more than seventy-five million to a little over
forty-eight million. The ARI regional staff experienced significant cuts from 103 staff persons in
2007 to 68 regional staff in 2015. In spite of declining revenue , the number of schools supported
by ARI ballooned from a K-3 focus in 899 schools to a K-12 focus in more than 1400 schools.
Statewide, schools/LEAs were permitted to pick and choose what, when , and how to implement
professional development opportunities offered by the ARI. This all occurred at the same time
that the ALSDE stopped monitoring K-2 reading proficiency levels. There is currently no reliable
reading assessment used to measure K-2 reading proficiency levels statewide.

Funding cuts to ARI seem to have negatively impacted the effectiveness of the program . The
cuts have restricted the ARI 's ability to be systemic, systematic, and intensive in its focus ,
implementation, scope and support. The unity of focus and collegiality experienced by teachers
and coaches during ARI Phase I, and to some degree during Phase II , seems to have been
diluted over the years due to the continual changes in ARI. In the words of one teacher "I was
supported in the classroom during my first two years. When the reading coach moved to K-2
grades only, my former school still included us in updates. The new school level ARI coach has
never been in my classroom for reading ." As reading coaches are pulled away for other duties
teachers are growing increasingly concerned about declining reading proficiency scores. A
classroom teacher stated 'We get very little support from our coach . Most of their time is spent
off campus and working on responsibilities other than being a Reading Coach . There's not much
coaching or support right now for us. I am very concerned about our declining reading scores."
Declining reading proficiency is an unintended consequence of very little support for teachers at
the classroom level.

One of the building level reading coaches recalled a time when "I felt the ARI was effective when
there was constant contact with all school based coaches, a very explicit focus on our
responsibilities , the requirement to turn in our calendars, the support with our principals and the

25
coaching communities that allowed coaches from several systems to collaborate and learn from
each other." Teachers and coaches alike revealed that "The current mode of regional coaches
supporting whatever schools are in their assigned areas is not as effective as receiving support
from a regional coach who is trained in either elementary or middle/high school. " The major
regret expressed by the teachers and reading coaches was that the ARI as a program was in
their words "a mile wide and an inch deep," due to all that has been required of the program in
the last three years.

Conclusions

The current configuration of the ARI has profoundly changed from its original status as a
statewide K-12 reading initiative to its current configuration as a K-12 support program .
The ARI was originally designed as a school improvement initiative requiring at least an 85%
commitment to 100% literacy by school leaders and teachers through intensive on-going
professional development.
It appears reduced literacy focus in the early grades resulted in a downward trend in reading
proficiency levels in the upper grades.
The focus of the two initiatives that subsumed the ARI is wider than literacy, therefore , the
ARI focus on literacy is not receiving the same intensity as in the past.
The differences in the ARI then and now are evident in its focus, scope, implementation, and
funding.
The ARI has been asked to expand its reach well beyond its original focus and , because of
that, has lost much of its ability to impact literacy because it has become just another member
of an on-demand support system.
Reductions in the numbers of the ARI field staff came at a time when the program cou ld least
afford to lose personnel. The ARI reg ional staff was cut from 103 at the end of Phase I to
the current level of 68 regional staff. The few remaining field staff have been required provide
support to individual schools and/or regional team needs that may or may not have anything
to do with literacy.
Teachers as well as reading coaches who are affected by changes in the ARI clearly
indicated that the changes have negatively impacted the effectiveness of the ARI.

Discussion

It appears that the ARI has been overextended and that it has lost a substantial amount of its
once nationally lauded effectiveness. In order to recapture the effectiveness and relevance of
the initiative, the ARI needs to be returned to its primary focus: 100% literacy for Alabama
Students. At the state level educational initiatives are often changed due to budgetary
expediencies and/or the departure or promotion of individuals associated with the program. At
the school level, teacher attrition rates may also be a factor in the implementation and
sustainability of "best practice" instructional delivery methods embraced by the ARI trained
teachers and administrators. It is difficult to determine the percentage of ARI trained teachers

26
and administrators from Cohort I, and to some degree Cohort II , that are still actively practicing
in Alabama classrooms. There seems to be a need for a refocusing of the ARI objectives at
both the state and local level.

There are several pathways to receiving educator certification in the state , but none of them
includes a requirement that incoming or prospective teachers receive the intensive, two-week
long, "best practices" training originally mandated by the ARI during Phase I of its existence.
The original ARI delivery model was effective in the development of teachers capable of leading
high quality literacy-based instruction. The continual widening of the focus of the ARI has
produced unintended consequences, the most important of which is declining literacy rates
among Alabama students . Alabama students, parents, and community stakeholders alike desire
graduates who are college and career ready. That goal is laudable, but the ALSDE is not
appropriately supporting it if we provide a haphazardly executed literacy loosely addressed
across all grade levels.

Recommendations for Future Practice

In order to recapture the effectiveness and relevance of the initiative that is aligned with the goals
of PLAN 2020 , the following recommendations are made:

It is recommended that the ARI return to its primary focus : 100% literacy for Alabama
students especially at the elementary level.
It is recommended that elementary schools be staffed with highly trained reading coaches
whose primary duties include providing coaching support to teachers as well as working
with struggling readers.
It is recommended that school staff and leaders be required to recommit to participate in
year-round professional development provided by the ARI on best practices in reading
instruction.
It is recommended that the ARI returns to its school-wide support, rather than the current
on-demand support.
It is recommended that the partnerships between the ARI and colleges of education be
strengthened to ensure a seamless transition from prospective teacher to effective
classroom teacher of reading .
It is recommended that prospective educational leaders at the elementary level be
required to complete a two-week intensive ARI training .
It is recommended that the Alabama Legislature fully fund the ARI. Full funding will
enable the ARI to be effective in the delivery of support to schools statewide , which will
yield a return in investment through improved reading outcomes for Alabama students.
In conclusion, changes in program implementation including budgetary decisions in education
do not happen in a vacuum , they impact educators and students. Decision makers are therefore,
encouraged to pay careful attention to the scalability and funding of programs such as the ARI
in order to maintain program effectiveness .

27
 References

American Institute for Research (2006). Lessons & Recommendations from the Alabama Reading
Initiative: Sustaining focus on secondary initiative . Retrieved from
http://www.air.org/sites/default/files/downloads/report/ARI_Popular_Report_final1_0.pdf

Center for Educational Accountability ( 1999-2000). Evaluation of Year 2 of the Alabama Reading
Initiative. Retrieved from http://www.uab.edu/education/cea/content-areas/literacy/15-content-
areas/content-areas/63-evaluation-of-year-2-of-the-alabama-reading-initiative

Creswell , J. (2012) . Educational research: Planning, conducting and evaluating quantitative and
qualitative research (4th ed .). Boston, MA: Pearson Education Inc.

Gay, L., & Airasian , P. (2003) . Educational research : Competencies for analysis and application.
(7th ed .). Upper Saddle River, NJ : Prentice-Hall , Inc.

Institute of Education Services. National Assessment of Education Progress (NAEP). Retrieved from
http://nces.ed .gov/nationsreportcard/glossary.aspx#percentile

Moscovitch , E. (2001 ). Evaluation of the Alabama Reading Initiative (Final Report). Retrieved from
http://files.eric.ed .gov/fulltext/ED464366 .pdf

28