Use Of Electronic Media To Promote

Acquisition of Action Verbs

and Prepositions
Created in:

Autism Language Program,

Children’s Hospital Boston

Howard Shane, Ph.D.

howard.shane@childrens.harvard.edu
 www.aac-rerc.com

This work is funded by the National Institute on Disability and

Rehabilitation Research of the U.S. Department of Education, under grant
number H133E030018. The opinions contained in this presentation are
those of the grantee and do not necessarily reflect those of the U.S.
Department of Education.
 Acknowledgement
Contributors to this presentation

• Patients and parents in the Autism Language Program

• Monarch School students, parents, and staff
• Model Autism Program, Boston Public Schools
• Sharon Shaham, James Sorce, Meghan O’Brien, Marie
Duggan & John Costello (Center for Communication
Enhancement, Children’s Hospital Boston)
 Today’s Agenda
• Overview of Teaching Language Concepts Program (TLC)
• Why is learning language concepts difficult for some persons with
ASD?
• Why are visuals needed?

• Three Phases of Instruction

• Dynamic Scene Cues
• Static Scene Cues
• Element Cues

• Developing Effective and Efficient Implementation

Procedures for Teaching Element Cues
• Using typically developing children
• Adapting and applying findings to individuals with autism
 Overview of VIP
Goal
Utilize visual supports to teach learners on the autism spectrum how to
better communicate during everyday home, school and community
interactions.

Key ingredients
• Visual language for comprehension and expression
• A ‘closed language’ with restricted pragmatic opportunity -- targeting
basic communicative operations
• Mentors and learners use the same visual symbols
• Use of advanced computer and video technologies
 What is Visual Immersion?
A symbol-rich environment
An environment where visual symbols clarify spoken
language, support expressive communication, assist
with transitions, and facilitate learning.

• Used in home, school, and community

• Easy access to
– Non-electronic symbols
– Electronic Screen Media (Monitors and TVs)
 What Does VIP Target?
• Seven concrete Communicative Operations :
– Protests/Refusals
– Requests
– Directives
– Comments
– Questions
– Social conventions
– Transitions
• VIP is not intended to teach advanced communicative operations:
– Abstract concepts: “…with liberty and justice for all.”
– Passive voice: “The book was read by the boy.”
– Complex syntactic structures: “If he hadn’t checked the weather in the
morning, then he would have forgotten to bring his umbrella.”
– Figurative language: “She flew to the bookstore.”
– Humor : “Why did the chicken cross the road?”

• VIP provides foundation for more advanced language-based

intervention programs
 VIP: Three Instructional Settings
• Natural setting:
Familiar, everyday people,objects, actions, locations and events as
opportunities to overlay language. Takes advantage of adding
language to already learned behavioral skills:
• Daily living/self-help routines
• Operational knowledge
• Preferred play routines

• Tabletop setting:
A controlled environment involving one-to-one instruction

• Virtual setting:
Electronic (television or computer) display for presentation of materials

Setting selection varies as a function of child and the skill to be learned.

Overview of Teaching Language
Concepts Program (TLC)
 Why Teach Language Concepts?
Autism Language Program (ALP):
• Population: Moderate to severe autism (from initial diagnosis to adulthood)

• Core deficit: Difficulties with language comprehension and expression, in

addition to pragmatic language skills

– Comprehension:
• Children on the spectrum often experience difficulty comprehending
spoken language and/or auditory processing impairment (Novick et al.,
1980; Boddaert et al., 2003).
• Many children are able to follow routine-based, familiar directives, but
have not yet demonstrated comprehension of more abstract linguistic
concepts (e.g., verbs, prepositions, attributes)

– Expression:
• 50% of individuals with autism spectrum disorders do not use speech
functionally (Wetherby and Prizant, 2005; Lord & Paul, 1997; Rutter, 1978)
• Many children are able to reliably use PECS to request and label, but have
not yet developed the ability to describe, comment, or ask questions
 Model of Language Comprehension
(Typical Development)

Spoken Short-Term Language Comprehension

Language Memory Processor
 Model of Language Comprehension
(Moderate-Severe ASD)

Verbal Short-Term Language Comprehension

Information Memory Processor

= Clinical insight suggests these impairments are due to:

•Attention to auditory stimuli
•Not understanding language as meaningful/symbolic
•Fleeting nature of spoken language
•Language processor ‘broken’
•Combination of these

Language comprehension is often compromised

• Occasional comprehension of noun labels and familiar directives (in context)
• Impaired comprehension of relational linguistic concepts (e.g., verbs, prepositions,
attributes)
 ‘Teaching Language Concepts’ (TLC)
Program
• A visual instruction system for hard to learn language
concepts (e.g., verbs, prepositions and attributes).
• TLC is a closed visual language
• Instruction in the virtual environment (video clips), tabletop
environment (photographs, toy figurines and miniature
objects), and natural environments (home, school and
community)
Three Phases of TLC Instruction
 TLC Instructional Phases:
• Learner progresses through all/some of three phases of
visual language symbols, from concrete to abstract
representations:

• Dynamic Scene Cues: full-motion video clips of

action scenes
Bypassing Spoken Language Comprehension
Using Dynamic Scene Cues
(Moderate-Severe ASD)

Verbal
Information

Short-Term Imitation
Memory (as inferred from
dynamic scene cues)

Visual
Information
(dynamic scene cues)

*Note: No involvement of language processor required

for comprehension of dynamic scene cues
 TLC Instructional Phases:
• Learner progresses through all/some of three phases of
visual language symbols, from concrete to abstract
representations:

• Dynamic Scene Cues: full-motion video clips of

action scenes

• Static Scene Cues: photographs that capture a

prototypical moment in the action scene
 Bypassing Spoken Language
Comprehension Using Static Scene Cues
(Moderate-Severe ASD)

Verbal
Information

Short-Term Imitation
Memory (as inferred from
static scene cues)

Visual
Information
(static scene cues)

*Note: No involvement of language processor required

for comprehension of static scene cues
 Implications of Static Scene Cue Mastery
Use of dynamic and static cues bypasses language
processor
– Due to load on language processor when using
elements, some children may use static scenes as their
communication system

Mastery of static scene cues is a significant

accomplishment
– Can be used to promote general understanding and
communication related to:
• Daily Living Skills
• Play
• Transitions
• Requesting
• Commenting
• Clarifying
• Directives
 TLC Instructional Phases:
• Learner progresses through all/some of three phases of
visual language symbols, from concrete to abstract
representations:

• Dynamic Scene Cues: full-motion video clips of

action scenes

• Static Scene Cues: photographs that capture a

prototypical moment in the action scene

• Language Element Cues: graphic icons

representing each of the individual linguistic
components that comprise an action scene (e.g.,
subject, object, verb, preposition, adjective, etc.)
TLC Phase 3: Element Cues
 Symbolate:
Language Comprehension?

• Stringing symbols together does not automatically result in

comprehension.
• Learners must first have a knowledge of language
elements and semantic relationships, as provided in VIP.
• Paradoxically, stringing symbols together may actually
interfere with comprehension.
 Model of Visual and Spoken Language
Comprehension
(Moderate-Severe ASD)

Comprehension of
Verbal
Verbal
Information Representation

Short-Term Language
Memory Processor

Comprehension of
Visual Visual
Information Representation
(element cues) (element cues)
 Clinical Observations About
Element Understanding
• Some label elements without meaning attached (echolalic-like)
• Some comprehend agent and object elements -- struggle with relational
elements
• Some comprehend isolated elements -- difficulty interpreting element
strings

What improves element comprehension?

• Immersion in symbol-rich environment
• Mass trials across multiple settings and communication partners
• Computer-based instruction
• Persistence over extended period

Bottom line: A more effective, efficient way of teaching language

elements must be developed.
TLC 3 Phases: Clinical Observations
• Progressing from
– Dynamic Scene Cues to Static Scene Cues -- is easy
• No involvement of language processor required

– Static Scene Cues to Element Cues -- is difficult

• Requires involvement of language processor

But
• The payoff is worth it -- foundation for generative
language
– Goal: Combine language element cue vocabulary with
their understanding of semantic relationships to generate
novel sentences for expressing requests, comments,
replies, etc. in natural settings.
 The Empirical Challenge

• Can we better understand the process?

• Can we expedite the process?

Developing Effective and Efficient
Implementation Procedures for
Teaching Element Cues
Top Down Approach
Example of Mixed Display - with Referencing
 The TLC Approach
Strategy: First study typical children, then apply findings to children with
autism
Rationale:
– Not yet literate
– Able to verbalize thought process
– Compliance
Subjects: Ages 3 yrs, 11 mos to 6 yrs, 4 mos

Procedure: General task - sequence three linguistic elements: left-to-

right reading order
– agent + action + object
– agent + preposition + object
– Experimenter provides no spoken language to label elements
– Learner demonstrates comprehension of element string by acting
out directives
Multiple design-prototype-test cycles to refine TLC instructional
approach
Iteration 1: Top-Down Approach - Sequentially
Through Three Instructional Phases

Iteration 1 Procedure
• Learner presented with
– Dynamic scene cues; imitate action
– Static scene cues; imitate action
– Element cues; demonstrate comprehension by
acting out directives

• For each instructional phase, learner viewed the

visual cues and was provided with spoken
prompt, “Now you do it!”
 Iteration 1: Top-Down Approach Sequentially
Through Three Instructional Phases
Outcome:
– Able to accurately imitate dynamic and static cues
– Difficulty understanding task when presented with
element cues (often attempted to match objects to
elements rather than carry out directive)
– Difficulty comprehending left-to-right sequence of
‘reading’ elements
– Difficulty understanding meaning of verb/preposition
(abstract) symbols
– Difficulty comprehending de-contextualized element
strings.
– Tendency to act on objects in a familiar manner (e.g.,
‘man on ladder’ rather than ‘man climbs ladder’)
 Iteration 2: Modified
Top-Down Approach
Modifications to Iteration 1 Procedure:
– Introduced visual template to focus attention on left-
to-right order of elements, and presented elements
in left-to-right sequence
– Introduced Mixed Display (Static scene cue along
with Element cues) and demonstrated association
between scene cue and its elements
– To enhance understanding of task expectation:
• Presented task as ‘game’ to discover meaning of
the relational (verb or preposition) symbol
• Experimenter modeled the task
Iteration 2: Visual Template
 Iteration 2: Modified
Top-Down Approach
Outcome:
– Initially learners did not appear to attend to the
relational element -- verb or preposition (e.g., “You
don’t need that one”)
– Still initially attempted to match the agent and
object elements to their physical objects
– Given repeated trials, demonstrated
comprehension of the individual elements, but
continued to experience difficulty with the left-to-
right ‘reading’ order for elements
 Iteration 3: Bottom-Up Approach
Modifications to Iteration 2 Procedure
Instruction started at element rather than dynamic phase
• Presented elements on computer monitor
– Experimenter modeled the task
– Guided discovery to select combination of element
strings to play corresponding video clip

• Then moved to tabletop

– Presented non-electronic task with identical element
cues and physical objects;
– Experimenter modeled the task
– Learner encouraged to create element strings to direct
the actions of communication partners
 Software Applications

Learning Together with Music Puddingstone Place

Iteration 3: Bottom-Up Approach
 Iteration 3: Bottom-Up Approach
Outcome:
– Older learners were able to execute each directive
presented and label each element

– Younger learners had difficulty attending to screen and

task; more likely to engage in their own independent
play with the materials
 Iteration 4: Modified Bottom-Up
Approach with Compelling Scenes
• Modifications to Iteration 3 Procedure:
– Created ‘fun’ scenes depicting engaging, motivating
activities using human models and life-sized objects
– Clicked on message window to activate video clip of
corresponding element string (in hopes of enhancing
attention to relational element)
– Presented learner with an ‘agent’ element cue
representing him/herself
– Encouraged learner to use elements to direct his/her
own play and actions of communication partners
Iteration 4: Modified
Bottom-Up Approach
 Iteration 4: Modified Bottom-Up
Approach with Compelling Scenes
Outcome:
– Actively engaged

– Successfully combined elements - although still initial

tendency not to attend to relational element (verb or
preposition)

– Prepositions more difficult than verbs

 Iteration 5
Modifications to Iteration 4 Procedure:

– Focus attention to relational element (verb or

preposition):

• Animated relational element

• Click on relational element to play video

 Key Ingredients to Teaching
Element Cues
• Present elements electronically
• Guided discovery with element combinations (to play
video)
• Provide children with symbols representing themselves
and familiar communication partners to allow them to
direct the activity
• Begin instruction with fun, motivating tasks. Gradually
expand to language-arts activities.
• Animate center elements to direct attention to element and
enhance likelihood of comprehension
 Directions for future TLC research
• Improve symbols
– Animated symbols (verbs, preposition, attributes)

• Generalization of language skills targeted in TLC to natural

setting

• Facilitating generative language