Use of Ontologies for Dialogue

about Audiovisual Data

Behrang QasemiZadeh, Ian O’Neill, Phillip Hanna, Darryl Stewart
 What is Data Modelling?

• A data model is an abstract model to say how

data is represented and accessed

• Video is a challenging source of information to be

modelled:
– Very rich
– Unstructured
– Temporal
 Video Data Modelling
• Machine understanding of video is still an
unsolved problem; and text based annotations
are still the common means of video modelling.

• Video modelling can be defined as the process of

en-/decoding raw video data to an internal
representation (textual information).
=URI

• Note: Any abstraction implies a metaphorical

representation
 Dialogue system and video modelling
• Goal: Dialogue System (DS) should be able to
map utterances to/from video data
annotations
– This means new requirements for the video data
model:
• Representing multiple levels of abstraction
• Or, multiple levels of information granularity
 if ‘Language as a knowledge grid’
• Then metaphors are carried via
– words
– structures
–…
• Using ontologies
– Words are covered by vocabulary (URIs)
– Structures are ontological relations
» Note that ontological does not necessarily mean logical

– Metaphors vs Literals
 More on DM and video modelling
• Semantics is the study of meaning in
communication
– Communication between collaborative agents
– Communication between user and DS
• Our Video Data Model does not try to capture
semantics
– it tries to say what is in a video in a language that
collaborative agents understand
– semantics grow when communication occurs
 Proposed Video Data Model
• A layered representation of video content not
video semantics!
– Video content is what can be seen/sensed by sensors
• Three Basic Elements:
– Properties: what we want to measure
– Objects: entities comprising a collection of
properties
– Events: changes over time
• Video Contents are represented using
properties, objects and events
 Proposed Video Data Model
• Moving from the core layer toward the nth
provides higher levels of abstraction with lower
levels of information granularity
• The heart of the core layer:
– Atomic Event:
• An event between what are
considered as successive video frames
• How to break a continuous space into a discrete one, and
how to represent this
– mathematicians have no choice other than to speak in
mathematical language
 Proposed Video Data Model
• Sketchy, approximate model
 Proposed Video Data Model, big picture
• Sketchy, approximate model
 Proposed Video Data Model, big picture
• Sketchy, approximate model

Physically Stored Data

Ontology of Events

Ontologies of Properties

Ontology of Time Ontology of Objects

 Proposed Video Data Model, Ontologies of Properties

• Sketchy, approximate model

 Proposed Video Data Model, Ontologies of Properties
property( location ).
property_dsc( `A place where something is or could be located` ).
location( Location ):- value( Location, location).

• Sketchy, approximate modelspace( officeyard ).

space( office ).
space( oos ).
space( office, office ).
space( office, 'red brick building' ).
space( officeyard, yard ).
space( officeyard, backyard ).
space( officeyard, `office's back yard` ).

space_location_mapping( office, [office, doorway] ).

space_location_mapping( officeyard, [lo, l, o1, o2, d, c1, c2, c3, c4, c5, c6, q1, q2, r] ).

value( LocationValue, location ):- member( LocationValue, [lo, l, o1, o2, d, c1, c2, c3, c4, c5, c6,
q1, q2, r, office, oos] ).

neighbour( lo, Location ) :- member( Location, [oos, l, c1, o1 ] ).

neighbour( l, Location ) :- member( Location, [oos, lo, o1, c1, c4] ).
neighbour( o1, Location ) :- member( Location, [lo, o1, c1, o2, d, l] ).
neighbour( c4, Location ) :- member( Location, [oos, l, c1, c5] ).

pvalue( [l,lo], left, location, adjectival).

pvalue( [o1, o2, lo] , pathway, location, nominal ).
pvalue( [r, q1], parking, location, nominal ).
pvalue( [lo, l, o1, o2, d, c1, c2, c3, c4, c5, c6, q1, q2, r], backyard, location, nominal ).
pvalue( [l, c1, c2, c3,c4, c5, c6], grass, location, nominal_on ).

property_description( location, LocationValue, 1, 'Pp', space ) -->

[SpaceNames],
{space(SpaceVlaue, SpaceNames), space_location_mapping(SpaceVlaue,
LocationValue)}.

property_description( location, Value, 2, 'Pp', side )  [Location, side, of, Space],

{space(Value, Space),
pvalue( Value, Location, location, adjectival )}.
 Proposed Video Data Model, Time Ontology

• Sketchy, approximate model

 Proposed Video Data Model, Time Ontology
• It may describe an order!
•• Indicates
Sketchy,temporal entities:
approximate instants and intervals
model
• Indicates time formats and values
• Temporal relations: Allen Interval Relations
• Current implementation supports a few formats
• Time Ontology by J. Hobbs and F. Pan
 Proposed Video Data Model, Time Ontology
• It may describe an order!
•• Indicates
Sketchy,temporal entities:
approximate instants and intervals
model
• Indicates time formats and values
• Temporal relations: Allen Interval Relations
• Current implementation supports a few formats
• Time Ontology by J. Hobbs and F. Pan
 Proposed Video Data Model, Time Ontology
• It may describe an order!
•• Indicates
Sketchy,temporal entities:
approximate instants and intervals
model
• Indicates time formats and values
• Temporal relations: Allen Interval Relations
• Current implementation supports a few formats
• Time Ontology by J. Hobbs and F. Pan
 Proposed Video Data Model,
Ontology of Objects

• Sketchy, approximate model

 Proposed Video Data Model,
Ontology of Objects

• Sketchy,
• Providesapproximate model
descriptions of objects
• Provides hierarchical views of objects
• Natural Language Descriptions
 Proposed Video Data Model,
Ontology of Objects

• Sketchy,
• Providesapproximate model
descriptions of objects
• Provides hierarchical views of objects
• Natural Language Descriptions
object_type(person).
object_type(vehicle).

object_def( object( object_type(person), uri(object_id),

object_properties( [ age(value), gender(value),
height(value), upper_garment_colour(value),
lower_garment_colour(value), hair_colour(value) ] ),
location(value),
time(value) ),
`an individual human being` ).
object_subclass (person, SubClasses):-
memeber( SubClasses, [ man, woman, youth, baby, child, girl,
individual, lad, lassie, male, man, teenager ]).
object( man, ID, Properties, Locaiton, Time ) :-
obejct( person, ID, Properties, Locaiton, Time ),
member( [gender(male)], Properties ).
object_descn( object( Type, URI , 1, 'N1s') -->
a, [ObjectType], property_description( age, Value, 1, 'Pp' ),
{ object(ObjectType , URI, Properties, Time, Location),
member(age(Value), Properties) }.
Proposed Video Data Model, ontology of events

• Sketchy, approximate model

 Proposed Video Data Model,
Ontology of Events

• layered
• The Sketchy, approximate
architecture model
is used in definitions of events
• Atomic Events vs. Composite Events
• Temporal Recipes + Event Definitions
• Events have different semantic valences 
• Semantic Roles are important 
 Proposed Video Data Model,
Ontology of Events
event( move ).
event_desc( event( event_type(move), uri(event_id),
event_objects( [agent(object_id)] ),
event_details( [from_location(location), to_location(location)] ),
( start_time(time), end_time(time) ) ),
`when an object change its physical place during the time according to a specified
• layered
• The Sketchy, approximate
architecture model
is used
location and time ontology, physical move` ).
in definitions of events
• Atomic Events vs. Compositeinherited_detail(
Events 'from_location', location ).
inherited_detail( 'to_location', location ).
• Temporal Recipes + Event Definitions
inherited_detail( 'start_time', time ).
inherited_detail( 'end_time', time ).
• Events have different semantic valences 
• Semantic Roles are importantdescribe(
 EventDetail( Value), [A, B, C] ) -->
[DetailPreposition ], property_description( PropertyType , Value, A, B, C ),
{ detail_prepostion( EventDetail, DetailPreposition ),
inherited_detail( EventDetail, PropertyType )}.

atomic_event( move, ID, *agent(ObjectID)+, *'from_location‘(FLocation), to_location(ToLocation) +, (

STime, Etime) ) :-
after( time_instant(STime),time_instant(ETime) ), neighbour(FLocation, ToLocation),
objects_time( ObjectID, STime, SObjectTimeID ),
objects_properties( SObjectTimeID, PropertyID1 ),
properties( FLocation, location, PropertyID1 ),
objects_time( ObjectID, ETime, EObjectTimeID ),
objects_properties( EObjectTimeID, PropertyID2 ),
properties( ToLocation, location, PropertyID2 ),
generate_event_id(ID).

composite_event( move, IDList, Objects, Details , (STime, ETime) ) :-

atomic_event( move, ID1, Objects, Detail1, (Stime, InterTime) ),
composite_event( move, IDList2, Objects, Details2 , (InterTime, ETime) ) ,
temporal_relation( (Stime, InterTime), (InterTime, ETime) , fi),
IDList = [ ID1| IDLlist2 ],
update_details( Detail1, Detail2, Detail).
The Advantage of Using Ontologies
• Without ontologies there cannot be a vocabulary for
representing knowledge
– ontologies clarify the terminology
• Ontologies enable coherent and cohesive reasoning
– in our special case, ontologies provide a means of
translating utterances into Description Logic and vice
versa.
• Ontologies provide a generic framework for applying
different dialogue strategies
• Combining a speech act ontology with objects, events,
and properties will make for a more comprehensive
model of dialogue
 Challenges
• More theoretical:
– Temporal ontologies as an alternative
– Properties as interactive non-independent entities
(complex metaphors)
– Finding a suitable level of abstraction in a multi-level
model
– No corpus!

– Dynamic knowledge management: what portion of

knowledge should be considered and how!
– Synchronisation of system observations and inferences
– Ontology management
 Future Work
• Short term goal
– Representing physically stored data as RDF resources
– Use of TimeML for annotation of utterances in a dialogue,
mapping between a relative time space to an absolute
representation
– Designing a proper, generic interlingua between task query
analyser agents and the broader dialogue system, a specialized
speech act ontology
• Long term goal
– Studying the semantics within a dialogue system
• how semantics evolve
• how to reformulate semantics
– Applying mining methods for extracting and learning behaviour
patterns, or studying mining through semantics
• DMS and the semantic web
Thanks for your patience.
Questions?