Using the International Classroom Observation Tool to Evaluate the Effectiveness of Integrating

Technology into Staff Development Trainings

By
Donna Neal
Phone: (386) 801-4200
Montra Rogers
Phone: (281) 703-1535
Email: mr1690@nova.edu
To be submitted to:
ASCD Education Leadership
Curriculum Teaching and Technology
EDD 7914
Submitted to:
Dr Shirley Walrod

Nova Southeastern University

September 21, 2014

2
Abstract
School districts around the country are employing a number of initiatives with the hopes of
remedying gaps in student achievement. Districts must begin the task of providing stakeholders
opportunities to connect the myriad of district efforts. Although traditionally used to evaluate the
integration of technology in classrooms, this article uses the International Society for
Technology in Education/International Classroom Observation Tool to evaluate the effectiveness
of integrating technology into staff development trainings. Furthermore, the presentation
highlighted within this work demonstrates the connection between 21st century and literacy
skills.
Introduction
In an effort to support those who will be responsible for planning, implementing, and
assessing district initiatives such as a one to one student laptop program, the integration of
technology into curriculum and instruction, as well as a comprehensive literacy program, it is
important to model the integration of digital content as well as digital tools and applications
during professional development trainings. In the past, such programs have focused on
developing the teachers ability to use specific hardware and software applications. These
approaches did not focus on the expansion of teachers knowledge, skill, and confidence in using
multiple digital tools and applications in the classroom with hopes of improving teaching and
learning. There is no one single approach to working with educators. However, those who are
responsible for creating staff development trainings will increase their chances for success and
ultimately the transfer of strategies and skill into instructional practices if they ground their work
in the solid theoretical background of adult learning theory and organizational development
(Matzen & Edmunds, 2007).
The critical issue that is addressed is if integrating technology into professional
development transfers to learning. Whether technology should be used in schools is no longer the
issue in education. Instead, the current emphasis is ensuring that technology is used effectively to
create new learning opportunities and promote student achievement. However, educational

3
technology is not transformative on its own. It requires the assistance of educators who
effectively integrate technology into curriculum and align it with student learning goals.
Teacher quality is the number one factor affecting student achievement and surpasses
other classroom barriers such as previous achievement levels, class size, homogeneity of the
class, and socio-economic factors (U.S. Department of Education, 2010; Kirkpatrick, 2009).
Therefore, professional development for teachers is vital as educators work to integrate
technology into instruction practices with hopes of improving the quality of learning in
classrooms. According to Fatemi (2013), one of the most serious obstacles to integrating
technology into curriculum and instruction is the lack of professional development for educators.
It is important to note that traditional one-time-only trainings are not effective in helping teachers
to become comfortable with integrating digital content, digital tools or applications, or
technology into instruction (Mancabelli, 2012). Instead, a well-planned, ongoing professional
development program that is tied to the districts curriculum goals, designed with built-in
evaluation, and sustained by adequate financial and staff support is essential if teachers are to use
technology appropriately to promote learning for all students in the classroom Fatemi, 2013).
The ISTE/ICOT. As integrating technology in todays classroom is a current initiative in
education, observation is necessary to ensure appropriate inclusion. An effective resource for
monitoring technology use in the classroom is the International Classroom Observation Tool
(ICOT). The ICOT uses a set of questions to guide observations on a number of key components
of technology integration (ISTE.org). The tool itself is divided into multiple sections that
address the setting of the observation, participant characteristics, student grouping variations,
teacher role, and learning activities. In addition, the tool questions the necessity of technology in
the lesson or presentation as it relates to teaching and learning. This observation tool also

4
includes a list of technology standards that addresses a range of learning outcomes. For
example, there is emphasis on operating systems and hardware/software problems, content
specific uses of technology, and technology integration as a teaching tool. Finally, the tool
monitors the use of technology through both the teacher and students lens. Using a Three
Minute Chart Matrix, the observer can monitor if technology was used for teaching and learning
as opposed to recreation or routine tasks such as log-on (ISTE.org). Although the ISTE/ICOT is
principally used observe the integration of technology in a classroom setting, the tool was
considered and modified to address the integration of technology in a professional development
setting.
Setting
The observation took place in the Houston Independent School District in Houston,
Texas. The district encompasses 301 square miles within a large metropolitan city incorporating
approximately 54% of the geographic area, of the city. It serves an enrollment of 203,354
students in prekindergarten through twelfth grade programs. There are 276 schools. Three chief
schools officers oversee the districts elementary, middle, high, and alternative/charter schools
(respectively). A total of 21 school-support officers and two alternative/charter school
compliance officers report to the three chiefs. These individuals provide leadership to principals
and align resources and support for teachers. There are 11,417 teachers. The ethnic composition
of the districts teachers is 37% African-American, 31% White, 26% Hispanic, and 4% Asian.
The district serves a diverse student population. The ethnic composition of the student body is
63% Hispanic, 25% African-American, and 11% White and Asian/Pacific Islander. Eighty
percent are economically disadvantaged and 93% qualify for Title I funds. LEP/ESL students
comprise approximately 30% of the entire student body.

National Education Technology Standards

The observed presentation incorporated a variety of standards that guide best practices in
technology integration. A list of those standards is included below.

1A.6. interactive communication tools

1A.7. curriculum-based presentations/publications

1A.8. curriculum-based collaborations

1A.9. appropriate technology selected

2A.2. technology-enhanced instructional strategies

3A.l. learning experiences address content standards

3B.l. technology supports learner-centered

6D.l. safe and healthy use of technology strategies

3D.2. technology integrated as a teacher tool

3D.3. technology integrated as a student tool

A debriefing held at the end of the presentation revealed that while both presenters understood
and included both content specific and process standards; they had not considered the inclusion
of national or state technology standards. The observer was able to note evidence of the National
Education Technology Standards in the presentation, the inclusion of such standards was not the
intent of those responsible for creating and presenting the training. The observer suggested that
technology standards are incorporated along with content objectives into future presentations.
The Presentation
The district in which the presentation took place has a number of initiatives in place to
support student achievement. Two initiatives, the Power Up/One to One Laptop Computer

6
Initiative and the Literacy By 3 Comprehensive Literacy Initiative, are deemed the most
important. To support the districts goals, the Secondary Social Studies Department provided a
professional development training for district and school leaders. The goals of the training were
two-fold. First, the Secondary Social Studies Department wanted participants to discover the link
between 21st century and literacy skills. Furthermore, the presenters modeled effective strategies
that demonstrated the link between 21st century and literacy skills in order to improve lesson
design and instruction.
The presentation began with introductions and the team provided an overview of the
research supporting the assertion that 21st century and literacy skills are connected. According to
Larson and Miller, (2011), students will need digital age proficiencies [to prepare] for the world
beyond the classroom; the educational system must understand and embrace 21st century skills
within the context of rigorous academic standards. Furthermore, digital age proficiencies are
literacy proficiencies and include reading, writing, listening, speaking, and thinking; currently,
these traditional literacy functions are accomplished with the aid of digital tools rather than the
more traditional paper and pencil (Larson & Miller, 2011). In short, digital literacy is literacy for
the 21st century. Using personal technology devices such as smart phones and tablets, along with
the web tool or digital application Kahoot, a gamed based classroom response system,
participants were asked to respond to the following question: Which two instructional areas do
you most want to develop with your teachers as they begin to incorporate digital tools into
instruction? Overwhelmingly, a number of school leaders responded that they would like to
develop teachers in the area of student centered instruction. Next, school leaders were introduced
to the Technology Integration Matrix. The Technology Integration Matrix provides a framework
for defining and evaluating technology integration and helps to set a clear vision for effective

7
teaching with technology (Gorder, 2008). After an examination of this matrix, the presenters
engaged the participants in a Think, Write, Pair Share strategy, and school leaders were asked to
respond to the following prompt: As a whole, where do the teachers in my school fit on the
Technology Integration Matrix? During the presentation debriefing, it was revealed that the
presentation team struggled with whether or not to use a digital tool or application such as Padlet
to capture participant responses. It was decided that having participants share with one other
school leader versus the entire group was less threatening. As a wrap up, participants were
invited to share interesting thoughts garnered from conversation with others.
In an effort to allow participants to experience a technology integrated social studies
lesson, the team used Blend Space, a web tool or digital application that allows teachers to
integrate a variety of digital content. The lesson focus was the Causes of World War I. The
learning target was: Identify Militarism, Alliances, Imperialism, and Nationalism as the MAIN
causes of World War I beginning in 1914. Participants were asked to examine maps, documents,
images, and secondary sources to answer the following big question: How did each of the
MAIN causes of World War I contribute to the conflict becoming a world war?
The observer was able to use the ICOT to evaluate the effectiveness of technology
integration in the presentation and model lesson. The ICOT solicited evidence of how the teacher
might integrate technology into instruction, and how students might use technology in their
learning. The ICOT probes for evidence of whole class, individual, and grouping opportunities;
the observer witnessed opportunities for whole class discussion, individual reflection or work, as
well as student (participant) pairs and small groups during the presentation and model lesson.
The team employed the following instructional strategies during the presentation and model

8
lesson, lecturing, interactive directions, and discussion. The observer was able to note evidence
of these activities on the ICOT.
Finally, the observer witnessed learning activities such as information analysis and
writing. Using the ICOT, the observer was able to note evidence of student-focused strategies
and activities. Regarding the model lesson, the team discussed assessment opportunities with
participants. In a classroom setting students might use a variety of web tools or digital
applications such as Thinglink, Animoto, or Glougster to create a product that explains how each
of the long term causes of World War I contributed to the conflict becoming a world war. As the
ICOT questions evidence of how students might integrate technology into learning, the observer
noted the discussion of digital applications in relation to assessment opportunities as evidence of
how students might integrate technology into their new learning.
Finally school and district leaders were shown a video that provided a model of
technology integration within a social studies classroom. With this, participants were asked to
evaluate this lesson using the districts Instructional Practices Evaluation rubric. As an
evaluation or reflective tool, participants were asked to use the digital tool Padlet to post in one,
two, or three words something they learned today, a new realization or an aha moment. It is
important to note that the presentation team provided a number of web resources that will help
facilitate technology integration.
Surprisingly, leaders struggled to evaluate the video lesson using the Instructional
Practices Rubric. This was of concern as a major part of a school leaders role is to evaluate and
provide feedback to staff using this Instructional Practices rubric. This segment of the
presentation would have also provided for the integration and evaluation of the National
Education Technology standards. Unfortunately, the observer was not able to record evidence of

9
technology standards as indicated on the ICOT. Although passionate about the potential impact
this work could have on both teachers and students, the presentation team admitted being a little
discouraged as they were not confident that such the message would transfer. The team discussed
the notion that only about 10% of what is learned in professional development sessions ever is
incorporated into practice (Joyce & Showers, 1982; Knight, 2007).
Conclusion
Our evaluation of the presentation and model lesson using the ISTE/ICOT was beneficial.
The observer determined that the presentation and model lesson revealed evidence of the
integration of digital applications and web tools from both the trainer and participant perspective.
However, a careful examination of the presentation and model lesson using the ISTE/ICOT
revealed that although the observer was able to align the various strategies included in the
training to technology standards, the presenters were not aware of these standards and did not
consider these learning objectives during the initial planning and creation of the training. In any
good lesson or presentation, standards, whether content or technology, drive the demonstration.
As districts implement initiatives to address woes in education, it is important to help teachers
understand how these efforts are connected. Finally, it was determined that providing for an
opportunity for debriefing and reflective discussion is necessary in the observation process.

10
References
Fatemi Jahromi, S. A. (2013). Exploring the human element of computer-assisted language
learning: An Iranian context. Computer Assisted Language Learning, 26(2), 158-176.
Retrieved from http://search.proquest.com.ezproxylocal.library.nova.edu/doc
view/1373085304?accountid=6579.
Gordner, L. (2008). A Study of Teacher perceptions of instructional technology integration in the
classroom. Delta Pi Epsilon, 50:2, 63-76.
Guhlin, M., Ornelas, L., & Diem, R. (2002). Methods that work: Educator competencies for
technology in public schools, 19(3), 142-146. Retrieved from http://search.proquest.com.
ezproxylocal.library.nova.edu/docview/62196948?accountid=6579.
Joyce B., & Showers, B. (1982). The coaching of teaching. Education Leadership, 40, 4-10.
Kirkpatrick, C. L. (2009). Engaging second-stage teachers in their work: The role of
professional culture in schools. (p.140). Retrieved fromhttp://search.proquest
.com.ezproxy local.library.nova.edu /docview /1031152029?accountid=6579.
Knight, J. (2007). Instructional coaching: a partnership approach to improving instruction.
Thousand Oaks: Corwin Press.
Larson, L. C., & Miller, T. N. (2011). 21st century skills: Prepare students for the future. Kappa
Delta Pi Record, 121-123.
Mancabelli, R. (2012). Five ways to turn pd into "personal transformation": How we can shift
our view of learning in a more meaningful way? District Administrator, 48(2), 72.
Matzen, N. J., & Edmunds, J. A. (2007). Technology as a catalyst for change: The role of
professional development. Journal of Research on Technology in Education, 39(4), 417430. Retrieved from http://search.proquest.com.ezproxylocal.library.nova.edu/docview
/62042877?accountid=6579
U.S. Department of Education (2010) A blueprint for reform: The reauthorization of the
elementary and secondary education act. Retrieved November 3, 2014, from
http://www2.ed.gov/policy/elsec/leg/blueprint/index.html

11