A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :

F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T A N D L A U R E L A L L E N D E R

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T AND LAUREL ALLENDER

This book, produced by the Partners of the Army Research Laboratory Advanced Decision Architectures Collaborative Technology Alliance, also serves as the Final Report for DAAD19-01-2-0009.

SECTIONI.COLLECTING,PROCESSING,ANDDISTRIBUTINGBATTLEFIELD INFORMATION SECTIONI COLLECTING,PROCESSING,ANDDISTRIBUTINGBATTLEFIELD INFORMATION

14SECTIONI

Chapter

1. DYNAMICNETWORKANALYSISAPPLIEDTOEXPERIMENTS FROMTHEDECISIONARCHITECTURESRESEARCHENVIRONMENT

KathleenM.Carley,Ph.D. CarnegieMellonUniversity,Pittsburgh,PA MichaelK.Martin,Ph.D. CarnegieMellonUniversity,Pittsburgh,PA JohnP.Hancock ArtisTech,Inc.,Fairfax,VA

INTRODUCTION ADACTAresearchisproducingexperimentalresultsfromsimulations ofintermeshednetworksofwarfightersandbattlefieldsurveillance assets.Thesenetworksformacomplexsystemwithbehaviorsthat emergefrompatternsofinteractionamongconstituententities.Thesim ulatedinteractionsarespatiallysituated,temporallydistributedcommu nicationsamongpeople,robots,andsoftwareagents.Ingeneralterms, thecomplexsystempartsofwhichweaddressinthispapercanbe conceptualizedasatwolevelmetanetworkthatincludesinteractions amonghumanagentsatonelevel,interactionsamongartificialagentsat anotherlevel,andcrosslevelinteractionsbetweenhumanandartificial agents. TheissueaddressedinthispaperisDynamicNetworkAnalysis(DNA) ofsystembehavior.Forthepurposesofthischapter,wearenotinter estedinexaminingtheperformanceofonesystemrelativetoanother. Althoughthistypeofcomparativeanalysiscanbeusefulforresearchers orsystemdesigners,itisofquestionableusetowarfighters.Instead,we areinterestedinanalysesthatproducetacticallyrelevant,actionable resultsthathighlightthestrengthsandweaknessesinthesystembeing
15

16CARLEY,MARTIN,ANDHANCOCK observed.Tobeuseful,theanalyticalresultsmustfostertacticalinsight andstimulatebattlefielddecisionsthatprudentlyinfluencefuturesys tembehavior. Tothisend,wedescribetwocasestudiesthatapplyDNAtothesim ulatedbattlefielddatabeinggeneratedbyexperimentsintheDecision ArchitecturesResearchEnvironment(DARE).Thefirstcaseinvolvesinter ceptsofsimulatedcommunicationsamonghumanagents,whichwe frameasanexerciseinadversarialreasoning.Thesecondcaseinvolves simulatedcommunicationsamongsurveillanceassets(i.e.,software agentsandrobots),whichweframeasanexerciseinunderstandingthe automatedcontrolofaPersistentCoordinatedVideoSurveillance(PCVS) system.Together,webelievethecasestudiesdemonstratehowDNA (Carley,2002)canfostertacticalinsightincomplexmultientityscenar ios.TheyalsodemonstratethatthecombinationofDNAtechniques requiredfortacticalinsightmayvaryaccordingtothetypeofnetwork beinganalyzed.Finally,theyshowhowDNAassistsinthedevelopment, understandingandtuningofsoftwareagentsystems. ANALYTICALAPPROACH Oneapproachtoanalyzingsystembehavioremploysmainstream statisticaltechniquesonsummarymeasuresofperformance(e.g.,per centoftargetstracked,percentofprioritytargetstrackedwithaccept ableaccuracy,trackcorrelation,etc.).Thistypeofanalysis,however, doesnotfullyexploittheinformationgeneratedinDAREexperiments orbybattlefieldsurveillanceingeneral.Moretothepoint,itprovideslit tleinsightintotheidentificationofleversinthenetworksunderlyingsys tembehavior.Onceidentified,leverscanbeusedtodetermineactionsto taketoinfluencefuturesystembehavior. DNAprovidesanalternativeanalyticalapproachthatcompliments mainstreamstatistics.WithDNA,thefocusshiftsfromaggregatemea suresofperformanceforacollectionofbattlefieldentitiestotheperfor manceimpliedbythestructureofrelationsamongbattlefieldentities. Thisshiftistheessenceofwhatitmeanstoviewthebattlefieldfroma networkscienceperspective.Thatis,fromanetworkscienceperspective wearelessconcernedwithwhatisnormal(e.g.,averages,dispersions) aboutentitiesinthebattlefield(e.g.,people,places)andmorecon cernedaboutdetectingsubstantivepatternsintheobservedrelations amongentities.TheemphasisonthestructureofrelationsinDNAmakes itparticularlywellsuitedtothedetectionofanomaliesandexceptions (e.g.,centralities,exclusivities)theleverswithpotentiallylargeinflu

DynamicNetworkAnalysis17 encesonsystembehavior.DNA,therefore,fostersscrutinyofstrengths andvulnerabilitiesintherelationsamongbattlefieldentities(i.e.,the observedsystem).WithaDNAmodel,wecanidentify(amongother things)implicitgroupsofentities,keypeopleandlocations,andopera tionallysignificanttimeframes.Wecanevenbegintoinferrelations amongentitieswherenonehavebeenobserved. Networksciencehasbeenhamperedhistoricallybyadominantly socialperspectivefocusingonwhointeractswithwhom.However,Carley (2002)arguedthatthesesocialnetworksexistwithinanecologyofnet worksthatcanusefullybecharacterizedintermsofthedynamicsofthe relationsamongthewho,what,where,how,andwhy.Thisisknownas themetanetworkperspective.*ORA(e.g.,Carley,Columbus,DeReno, Reminga&Moon,2008)isadynamicnetworkanalysispackagethatcan beusedtoassessmultimode,multiplexnetworks;identifykeyplayers, groupsandvulnerabilities;enablecomparisonoftwoormorenetworks; andfacilitatereasoningaboutspatiotemporalnetworks. *ORAsupportsanalysisofdynamicnetworksinmanyways:(1)com parisonsoftemporallyorderedsnapshotsofstaticnetworks,(2)statisti calchangedetectiononsequencesofnetworks,(3)trailanalysisfortrail dataandconversionoftraildatatonetworks,(4)simulationofchangein networks,and(5)comparativestaticsforimmediateimpactassessment. Herein,wemakeuseofstatisticalchangedetectionandtrailanalysis, alongwith*ORAsvisualizationcapabilities. CASESTUDY1:TERRORISTSINADELPHI Thescenarioforcasestudy1wasframedasinterceptsofsimulated communicationsamongagentsrepresentingterroristsandnoncomba tants(e.g.,pizzadeliveryguys).Theagentscommunicatedviaphoneand emailastheymovedabouttheAdelphiregion. DataGeneration ThedataweregeneratedbyArtisTechsAlgoLinksimulator.The AlgoLinksimulatorwasoriginallydevelopedtotestthecapabilitiesof messageanalysistoolstosupportintelligenceanalysisrequirementsin battlefieldcommunications.AlgoLink(seeFigure1.1)facilitatescustom constructionofentitynetworksthatfollowspecifiedcommunication structures,times,places,durations,andbehaviors.Itgeneratesboththe foreground(networkofinterest)andthebackgroundcommunicationas specifiedandstitchesthemtogetherintoasinglecommunicationrecord.

18CARLEY,MARTIN,ANDHANCOCK Backgroundcommunicationisbothstructured(e.g.intermingledhierar chicalandsocialorganizations)andrandomasspecifiedbythehuman simulationoperator.AlgoLinkusesrealisticcommunicationandorganiza tionaldata,timing,andmorphologiesbutcontainsnoinformationabout anyrealpersonororganization.

Figure1.1 TheAlgoLinkmessagesimulatorinterface ThedatasetcreatedforthisexperimentwasgeneratedbyArtisTech stafflookingforwardtosystembasedIntelligentAgentcommunication behavioranalysis.Theentitiesofinterestwereorganizedintoasmall numberofcellsthatwereuniquelyconnectedandstitchedintoa largerbackgroundcommunity.Thedatasetwasgeographicallycen teredontheARLAdelphicampusbecausetheARTEMISprojectiscoordi natedwiththeARLComputationandInformationSciencesDirectorate researchsystems.Thesimulatedsocialstructurewasmorphologically similartoamediumsizedcommunityofintelligentagentsactingwitha specificpurposeinabattlefieldCommand,Control,andCommunications (C3)system.Tosimulateaneventthatstimulatesthecommunication network,aspikeincommunicationvolumewasinsertedataselected time.TheAlgoLinkgenerateddatasetwasarapidwaytoassessthefea sibilityofcollaborationbetweentheARTEMISPCVSandCASOSteams.

DynamicNetworkAnalysis19 Analysis TheAlgoLinkoutputwasdeliveredasanXMLfilecontaininga sequenceofcommunicationsrecords.Eachsimulatedcommunication recordidentifiedthesender,thereceiver,thetimethecommunication occurred,itsduration,andwhetheritscontentwasoperationallyrele vant,irrelevant,orambiguous.Italsocontainedlatitudeandlongitude forthepositionofthemobilesendersandreceiversduringeachcommu nication. Ouranalysisstrategyinthiscasecanbegenerallydescribedasan overviewandzoom.Thatis,wefirstexaminedthegeneralcontextof communicationsactivities,andthendrilleddowntodetermineimpor tantagents,timeframes,andlocations.Thesubsetof*ORAcapabilities thatprovedparticularlyusefulhereincludedgeospatialvisualization,key playeridentification,changedetectionanalysis,andthecorrelationof standardnetworkandgeospatialvisualizations. Using*ORAsgeospatialvisualizationcapabilities(e.g.,Davis,Olson, &Carley,2008)weexploitedthepresenceoftimevarying,geolocated attributesoftheinterceptedcommunicationstodiscoverthescenario involvedsuspiciousentitiesfleeingtheAdelphiarea(seeFigure1.2).

Figure1.2 AgentsfleeingAdelphioverthetimecourseofthescenario Usingafuzzygroupclusteringtechnique,FOG(e.g.,Davis&Carley, 2008),wefoundthatthesuspiciousentitieswereorganizedintofive groupswithsharedmembers(seeFigure1.3).Theinterstitialmembers arelikelytocontainthecoordinatorsandleaders.

20CARLEY,MARTIN,ANDHANCOCK

Figure1.3 Agentsorganizedin5groupsthatsharemembers Todrilldown,wefirstused*ORAsKeyEntityReporttoidentifythe threeagentsmostcriticaltooperations(seeFigure1.4).Becausethe datawereaboutcommunications,twodifferentcentralitymeasures wereuseddegreecentralityandbetweennesscentrality.Degreecen tralitymeasureswhoisconnectedtomostothers(i.e.,theactormost likelytobeintheknow).Betweennesscentralitymeasureswhois mostlikelytobeonallthepathsbywhichinformationflows(i.e.,the actormostlikelytobeinfluential).Thisenablednarrowingourfocustoa smallgroupofleadersinsteadoffocusingonthesetofinterstitialmem bers.

DynamicNetworkAnalysis21

Figure1.4 KeyEntityReportidentifies3importantagents Wenextasked,issomethinghappening?Onewayofansweringthis istoseewhetherthereisachangeinstandardbehavior.Usingthe ChangeDetectionReport(seeFigure1.5),weidentifiedperiod3asthe timeframeinwhichoperationsmostlikelyoccurred.

Figure1.5 ChangeDetectionReportsignalstimeperiod3isdifferent

22CARLEY,MARTIN,ANDHANCOCK

Thenetworkchangedetectionanalysis(McCulloh&Carley,2008; 2009;McCulloh,Webb,Graham,Carley&Horn,2008)extendschange detectionfromoperationsresearch,whereithasbeenusedonvariable leveldata,torelationaldata.Thisisastatisticalapproachfordetecting smallpersistentchangesinorganizationalbehaviorovertimeusingsta tisticalprocesscontroltechniquesappliedtonetworksummarystatistics. Period2wasthepointatwhichorganizationalbehaviorchanged,leading toradicaldifferencebyperiod3.Thisappearstohavebeenaplanning executionphaseshift. Examinationofindividuallevelmetricsforthethreekeyplayersand networklevelmetrics(e.g.,centrality,betweenness,efficiency,connect edness)corroboratedourinterpretationthatperiod3wasoperationally significant.Individuallevelmetricsindicatedthatactor286engagedin extensivecoordinationatperiod2,passingthereignsofcontroltoactor 652atperiod4(seeFigure1.6).

Figure1.6 Individuallevelmetricsconvergeonperiod3 Examinationofnetworklevelmetrics(seeFigure1.7)showedthatthe groupwasgenerallyaverydistributedstructurethatcoordinatedintoa centrallycontrolled,moreefficient,unitatperiod3.Then,itwentback toits

DynamicNetworkAnalysis23

Figure1.7 Networklevelmetricsconvergeonperiod3 standardformandbecamemorehiddeninthesociocommunication environment. Havingidentifiedthekeyplayersandtimeperiod,wefocusedour analysesondiscoveringwhatmayhavehappened.Examinationofthe agentxlocationnetworkforperiod3(seeFigure1.8)indicatedalarge clusterofsuspiciousentitiesintheAdelphiarea(includingkeyplayer, Agent286),afairlylargeclusterofsuspiciousentities(includinganother

Figure1.8 AgentxLocationNetworkforperiod3 keyplayer,Agent97)inwhatappearstobeastagingarea,anapparent waypointbetweenthestagingareaandtheclusterofsuspiciousentities

24CARLEY,MARTIN,ANDHANCOCK inAdelphi(alsovisitedbyAgent97,whoseemstobealiaison),anda runner(keyplayer,Agent652)whovisitsmanylocationswithfewsuspi ciousentitiespresent. Anotherkeyadvance,developedaspartoftheCTA,wasthecapabil itytomovebetweentraildatawhowaswherewhenandnetworks. Whenweexaminedthetrailsvisualizationforperiod4(i.e.,theperiod immediatelyfollowingtheapparentoperation,perhapsaperiodofinitial surveillance),wesawthatthethreekeyplayerswereneverinthesame placeatthetime;Agent652wasagainrunning,whereastheactivitiesof Agents97and286wererestrictedtooneortwoareas.Theapparent coordinationhandofffromAgent286toAgent652isrelatedto652s increaseinspatialmovementandcoordinationneededduetoincreased movement.Inthetrailsvisualization(seeFigure1.9),timeprogresses downtheyaxis.Geographicregionsformverticalbinsalongthexaxis. Arrowsareplottedasagentsmovefromregiontoregion(orwithin regions),andinthiscasearecolorcodedtothethreekeyplayers.

Figure1.9 *ORALoomvisualizationoftrailsfor3keyplayersduringperiod4 Finally,correlatingastandardagentxlocationnetworkvisualization withageospatialvisualizationfortheendofthescenariowefoundthat Agent286wasalonewithasinglemovementbetweentwolocationsin Adelphi,Agent97washoledupwithasizablegroupofsuspiciousenti tiesnorthofAdelphi,andAgent652wasalonebutontherun(seeFigure 1.10).

DynamicNetworkAnalysis25

Figure1.10 Correlationofagentxlocationnetworkandgeographiclocationatsce narioend Giventhepatternofcommunicationsandmovementduringthesce nario,twocoursesofactionsappearreasonable:(1)scourAdelphifora bomb,IED,etc.plantedduringoperationsinperiod3,or(2)goafterdis persedsuspiciousentities.Withrespecttoaction2,Agent286maybean easytargetwithdirectknowledgeoftheoperationsthatoccurredduring period3.TargetingthelocationwhereAgent97ishiding,however,will yieldmoresuspiciousentities.Notethatthesimulateddatadoesnot includecommunicationscontentsospecificationoftheeventisnotpos sible,buttheappliedDNAanalysisaccuratelyidentifiedthetime,place, andleadentitiesinthesimulation. CASESTUDY2:MOVEMENTINTHEPERIMETER Thescenarioforthiscasecenteredonanautomatedsurveillance system(i.e.,theARTEMISPCVSsystem)thatisresponsibleforidentifying movingentitieswithintheperimeterofaBlueForceresearchcom pound.Thesystemdividestheperimeterintofourareasofresponsibility, whereeachareaisassignedtoaTaskingAgentthatisresponsiblefor surveillance(seeFigure1.11).TaskingAgentsgetsimulatedmovement reportsfromsimplevideoanalysisalgorithms.TaskingAgentsthenprior itizetargetsandassignmobileroboticassetstopursueandidentifythe targets.

26CARLEY,MARTIN,ANDHANCOCK

Figure1.11 TaskingAgentAreas DataGeneration ThedataweregeneratedbyArtisTechsARTEMISPCVSsystemproto type(seeFigure1.12).Thisprototypeuseshundredsofsmallreasoning

Figure1.12 3DViewofARTEMISPCVSscenarioontheGISARLAdelphitestbed model

DynamicNetworkAnalysis27 algorithmsencapsulatedinsoftwareagents.Theagentscommunicate withhumans,agents,andothersystemelementsandevencreateand deleteotheragentswithfrequenciesthataredependentontheirreac tionstothesensedenvironment(simulation). Theparticularexperimentalrunanalyzedherewasconductedto determinewhetherTaskingAgentreasoningandcommunicationabout sharingmobileroboticassetswasfunctioningasexpected.Thescenario includedashorttimeofquiescenceinthecompound,followedbythe injectionofarelativelylargenumberofmovingtargetsthatmoved aboutthecompoundusingreasonablepathsbutfreelycrossingareasof surveillanceresponsibility.Whentargetscrossareasofresponsibilitythe complexityofreasoningincreasesandrequiresthatTaskingAgentstrans fer(handoff)trackingandevenpossiblylendroboticassetstoother TaskingAgents.Theactoflendingtheassetinvolvescommunicationto notifyanadjacentTaskingAgentofanincomingunidentifiedentity, andamultimessageTaskerGlobalTaskerhandshaketotransfercon trol.Thereweretwoofthesehandoffeventsinthescenario.Thissce nariowasasimpleonetofacilitateearlycollaborationbetweenARTEMIS andCASOSstaff. Analysis Aswithcasestudy1,thedataweredeliveredasanXMLfilecom prisedofasequenceofcommunications.Thecommunicationswere betweensoftwareagentsorsoftwareagentsandrobots. Althoughthedatasetsweresuperficiallysimilar(i.e.,logsofcommu nicationsrecords),casestudy2presentedseveralchallengesnotpresent incasestudy1.Thesechallengesaroseprimarilybecausethedatafor thiscasewererelativelyimpoverished.DNArequireslargedatasetsand wedidreceivemoredataforcasestudy2thanforcasestudy1.However, theextradatawereoflittlebenefitbecausetheyprovidedlittleinforma tionregardingthestructureofthesystemwewereanalyzing.Intermsof structuralinformation,theextradataweremostlyredundant. Theanalysiswascomplicatedbythefactthatthescenarioincluded onlytwoinstancesofthetargethandoffeventthesignalwewereto detect.Instructuralterms,thismeansthatwewerelookingforachange innetworkstructurethatinvolved(perArtisTechsdescriptionofthe handshake)threelinksatmost.Thus,thestatisticalchangedetection analysisusedincasestudy1wasofnouse.Suchasmallchangeinstruc turewouldnotbedetectedasbeingstatisticallysignificant.

28CARLEY,MARTIN,ANDHANCOCK Theanalysiswasfurthercomplicatedbytheabsenceofgeocoordi natesforthemobilerobots.Therefore,*ORAsgeospatialvisualization capabilitiescouldnotbeemployed.Theinsightthatcanbegleanedfrom visualizationsofagentsxlocations,asusedincasestudy1,wasalsolost. Withoutlocation,wealsolackedanymeansforconstructingtrailsdata toexaminewhowaswherewhen. Finally,thegoalofanalysisincasestudy2differedfromcasestudy1. Incasestudy1,weemployedDNAtechniquesdesignedtoidentify importantentities,locations,andtimes.Thepurposeoftheanalysis, therefore,wastoidentifycentralities.Incontrast,thepurposeofthe analysisforcasestudy2wastoidentifyexclusivities(i.e.,thetwoTasker GlobalTaskerhandshakes). Giventheimpoverisheddatasetandourgoalofdetectingonlytwo instancesofthehandoffamongasmallsetofagentsandrobots,the analysisstrategyweadoptedforthiscasewasoneofconvergingopera tions(tokickstartaDNAofricherdatainthefuture).ArtisTechper sonnel(withtheirknowledgeofthesystem)manuallyanalyzedthedata. CASOSpersonnelappliedDNAtechniquestothedata.Thefollowingdis cussesonlytwooftheissuesweaddressed. TofindthehandoffwhereoneTaskingAgentloanedarobotto anotherTaskingAgent,wereliedonArtisTechsidentificationofmes sagetypesthatindicatesuchahandoffoccurred.Threemessagetypes wererelatedtothehandoff:RemoveIdentity,ReportPositionToSelected Tasker,andReportPositionToSelectedTaskerReturn.Findingthehandoff wasthensimplyamatterofusing*ORAsSphereofInfluencecapabilities tovisualizewhichagentswereinvolved.Figure1.13showsthethree importantmessages,alongwiththeagentsthatsentandreceivedthem. ItcanbeseenthatTaskingAgent3ispositionedinthenetworkdiffer entlythanTaskingAgents1,2,and4.Furthermore,weseethatTasking Agent3wastheonlyagenttosendtheReportPositionToSelectedTasker Returnmessage(asindicatedbythearrowonthelinkbetweentheagent nodeandthemessagenode).GlobalTaskingAgent7receivedthismes sageandsentaRemoveIdentitymessagetoTaskingAgent3.Thesphere ofinfluencealsoindicatesthattheReportPositionToSelectedTaskermes sagewasprobablynotuniquelyrelatedtothehandshake.

DynamicNetworkAnalysis29

Figure1.13 SphereofInfluencevisualizationshowingrobothandoff Inanticipationofricherdatasets,thesecondissueweexaminedwas whetherwecoulduseDNAtechniquestopartitionagentsintofore groundandbackgroundagents.Tothisend,theArtisTechteamproduced ameticulousmessagetraceanalysisandeventidentificationasground truthfortheCASOSteam.Evenasfullyknowledgeabledesignersofthe communicationlogicthisanalysisanddocumentationtookmorethan4 hoursusingsimpletextsearchandanumericmessagefrequencyanalysis providedbyCASOS.Thecomplexityofthisanalysisunderscorestheneed fornetworkanalysistechniquesforamorecompletesystemexperiment analysis. PerArtisTechsanalysis,agentscouldbedividedintoforeground agentsthatweresubstantivelyrelatedtothescenarioandbackground agentsthatexistedsimplytomakethesimulationrun.Ourtaskat CASOS,therefore,wastoemployoneoranotherDNAtechniquetosepa rateforegroundfrombackgroundagents.WefoundthattheNewman groupingalgorithmworkedwell,separatingforegroundandbackground agentsintogroupsthatcloselyapproximatedthemanualanalysis.Seven oftenbackgroundagentswerecorrectlyclassified.Buttherewasdis agreementbetweenArtisTechjudgesregardingwhetheroneofthethree misclassifiedagentswasabackgroundoraforegroundagent.Onefore groundagentoutof29undisputedforegroundagentswasmisclassified. ThattheNewmangroupingalgorithmcorrespondsfairlywellwiththe judgmentsofdomainexpertsispromising,andsuggestsalineof researchconcerningthepsychologicalvalidityofNewmangrouping.

30CARLEY,MARTIN,ANDHANCOCK SYSTEMUNDERSTANDINGANDTUNINGACTIVITIES ArtisTechpostulatedtheadditionalbenefitofperformingDNAon theoutputlogsfromtheinternalsystemcommunications;theanalysis supportedunderstandingofhowtheteamandsystemachievedthemea suredperformance.TheARTEMISresearchteamwasintheprocessof settingupsystemmodelexperimentstostudytheadvantagesthatauto matedreasoningcouldaddtowidelyusednumericimageandvideopro cessingforthepurposeofPCVS.Aswemodeledthereasoningandsetit intothesimulationwehaveconductedmanytestruns.Wecaneasilysee whenthemacrosystembehaviorisasdesignedandwhenitdeviates fromintentions.However,todeterminewhythedeviationsoccurand howthereactiveagentnetworksachieveintendedsystemperformance requiresmessagelevelanalysis.ArtisTechsharedmixedresulttestlogs withCASOSspecificallytofacilitatetheidentificationofexpectedand unexpectedresults,andhowtheyarise. Resultsfrominitialanalysesinbothcasestudiesdivergedfrom expectedresults.Inhindsightthisishardlysurprising.Designofacom plex,multiagentsystemwithemergentreactivesystembehaviormay notbepossiblewithoutthesupportofnetworkanalytics. Inbothcaseswefoundevidenceofpragmatic,technicallycorrect, programmingpracticesthatinterferedwithsubstantiveDNAofthecom municationslogs.ArtisTechconfirmedthattheexperimentrunsthat wereanalyzedwerenotconsideredfinalorexpectedtobeentirelycor rect,merelytypicalincontentandform.Whetherweviewthisasaveri ficationissue(i.e.,buildingthethingright)oravalidationissue(i.e., buildingtherightthing)dependsonperspective.Giventhatverification isprimarilyaninternalactivity;thedegreetowhichprogrammers achievedtheintentbehindspecificationsisamatterofinterpretation. Fromtheperspectiveofnetworkanalysis(i.e.,usersoftheoutputlogs), however,wecannoteaminorshortfallinvalidation.Specifically,com municationrecordsthatarenotrelevanttothescenariobeinganalyzed shouldbefilteredfromthelogtoincreaseusability. Intheinitialanalysis,weexperimentallyillustratedthatDNA approachesprovideacapabilitytoanalyzecomplexmessagesetstofind particularbehaviorpatternsofinterest.Inthesecondanalysiswebegan forgingacollaborativeresearchapproachdesignedtounearththeana lyticstepsthecombinationsofDNAtechniquesrequiredtoanalyze differenttypesofmessagingbehavior.

DynamicNetworkAnalysis31 LESSONSLEARNEDANDFUTUREDIRECTIONS Opencollaborationbetweendataprovidersandnetworkanalysts createdabeneficialgapbetweenexpectedandobservedsystembehav ior.Asdataprovider,ArtisTechdevelopedthemultiagentsystemand environmentsimulations,designedthescenarios,andconductedthe simulationbasedexperimentswithexpectedsystembehaviorsinmind. Thenonlinearitiesinherenttocomplexsystemscomprisedofinteracting agents,however,makeitnotoriouslydifficulttopredictemergentand reactivebehavior,andareindeedthereasoncomputersimulationisnec essary.Asnetworkanalyst,CASOSreceivedmilitarilyrelevantbattlefield simulationdatawithoutpriorknowledgeofexpectedsystembehavior. ThechallengetoCASOS,therefore,wastouseDNAtocharacterizewhat happenedinthemysteriousscenariosreceivedandCASOSobserva tionsinitiallydivergedfromArtisTechdefaultexpectations.Toresolvethe discrepancy,weusedDNAtoexaminewhythesimulationsdidnot behaveasexpected.Thus,thepostulatedbenefitoftheArtisTechCASOS collaborationwastheuseofnetworkanalyticstogaininsightintothe performanceofmultiagentsystems.ArtisTech,withCASOS,nowplans tousethisapproachandDNAtoolstobuildasetofreusablesystem experimentanalysismethodsthatwillbeappliedtounderstandhow variantHuman/AutomationPCVSexperimentsachieveobservedresults usingnetworksofcommunicationandbehaviors. WithregardtodemonstratingthetacticalrelevanceofDNA,the availabilityofmilitaryscenariodataisinvaluable.Itprovidesopportuni tiestocombineextantDNAtechniquesintoanalyticstrategiesthatpro duceresultswarfighterscanuse.Generally,wefoundthatthefamilyof DNAmetricsandvisualizationsthathavebeenimplementedin*ORA overthepast10orsoyearsprovidesanamplebasisforconductingtacti callyrelevant,actionableanalysesonbattlefielddata. Two*ORAcapabilitieswereparticularlyhelpfulduringthiseffort: geovisualizationandnetworkchangedetection.The(notsosimple)act ofplacingnetworksonamapputsabstractsocialnetworksintoacon crete,spatialcontext.Itprovidesexplicitinformationaboutwherethe actionistakingplace.Thecapacitytodetectstructuralchangesamong temporallyorderednetworksprovidesexplicitinformationaboutwhen theactionistakingplace. Thecasestudiesalsohelpedtoidentifyseveralareaswherefuture developmentcouldimprove*ORAstacticalrelevance.Theimprove mentswouldgenerallysupportdeliveryofoneormoreTacticalInsight Reports.Asenvisioned,thesereportswouldcontaintheoutputsofall

32CARLEY,MARTIN,ANDHANCOCK DNAtechniquesthatcontributetoaparticularanalyticstrategy.From thetwocasestudiesdescribedabove,itappearsthatcorrespondences amongnetworksoftherelationsamongentities,networksofgeospa tiallyanchoredentities,andnetworksdistributedovertimewillplaya centralroleinsuchreports. ForArtisTechthegoalofDNAapplicationstosystemanalysisand monitoringistoinvestigatethecreationofageneralizeddatastructure andDNAmethodcombinationapproachthatwillallowtheencodingof expectedordetectedmessageandbehaviorpatterns.Theinitialapplica tionofthisconcernspostexperimentreview.Themorefarreaching implicationsofsuchDNAmechanismsextendtorealtimeembedded systemmonitoringtoincreasedistributedsystemsecurityandusertrust. ACKNOWLEDGEMENTS ThisworkispartoftheDynamicsNetworksprojectatthecenterfor ComputationalAnalysisofSocialandOrganizationalSystems(CASOS)of theSchoolofComputerScience(SCS)atCarnegieMellonUniversity (CMU).For*ORA,forthepartsusedinthisanalysis,thegeospatialcom ponentsweresupportedbyAROandERDCTECW911NF0710317,the *ORALoombyONRN000140610104,theovertimechangedetection basicanalysisbyARIW91WAW07C0063andtheORAimplementationof changedetectionbyARLDAAD190120009,thefourieranalysisbyONR N000140610104,thekeyentityanalysiswassupportedbyONR N000140610104,extensionstohandlecommunicationsnetworksby ARLthroughtheCommunicationsandNetworks(CN)Collaborative TechnologyAlliance(CTA)20002504andtheimmediateimpactassess mentbyONRN000140610104.Thebasicresearchconductedhereto explorehowtousethesetechnologiesforadversarialbehaviorandto linktotheArtisTechARTEMISPCVSsystem,AlgoLinksimulator,DARE wassupportedbytheArmyResearchLabthroughtheAdvancedDecision Architecture(ADA)CollaborativeTechnologyAlliance(CTA)DAAD1901 20009.Theviewsandproposalscontainedinthisdocumentarethoseof theauthorandshouldnotbeinterpretedasrepresentingtheofficialpol icies,eitherexpressedorimplied,oftheOfficeofNavalResearch,the ArmyResearchOffice,theArmyResearchInstitute,theArmyResearch LabortheU.S.government.

DynamicNetworkAnalysis33 REFERENCES
Carley,K.M.(2002).Smartagentsandorganizationsofthefuture.InL.Lievrouw& S.Livingstone(Eds.),Thehandbookofnewmedia(pp.206220).ThousandOaks,CA: Sage. Carley,K.M.,Columbus,D.,DeReno,M.,Reminga,J.,&Moon,I.C.(2008).ORA UsersGuide2008(Tech.Rep.No.CMUISR08125).Pittsburgh,PA:Institutefor SoftwareResearch,SchoolofComputerScience,CarnegieMellonUniversity. Davis,G.B.,&Carley,K.M.(2008).ClearingtheFOG:Fuzzy,overlappinggroupsfor socialnetworks.SocialNetworks,30(3),201212. Davis,G.B.,Olson,J.,&Carley,K.M.(2008).OraGISandloom:Spatialandtemporal extensionstotheORAanalysisplatform(Tech.Rep.No.CMUISR08121).Pittsburgh, PA:InstituteforSoftwareResearch,SchoolofComputerScience,CarnegieMellon University. McCulloh,I.,&Carley,K.M.(2008).Detectingchangeinhumansocialbehavior simulation(Tech.Rep.No.CMUISR08135).Pittsburgh,PA:InstituteforSoftware Research,SchoolofComputerScience,CarnegieMellonUniversity.

34CARLEY,MARTIN,ANDHANCOCK

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T A N D L A U R E L A L L E N D E R

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T AND LAUREL ALLENDER

This book, produced by the Partners of the Army Research Laboratory Advanced Decision Architectures Collaborative Technology Alliance, also serves as the Final Report for DAAD19-01-2-0009.

SECTIONI.COLLECTING,PROCESSING,ANDDISTRIBUTINGBATTLEFIELD INFORMATION SECTIONI COLLECTING,PROCESSING,ANDDISTRIBUTINGBATTLEFIELD INFORMATION

14SECTIONI

Chapter

2. FROMBACKGROUNDSUBTRACTIONTOTHREATDETECTIONIN AUTOMATEDVIDEOSURVEILLANCE

JoshuaEckroth TheOhioStateUniversity,Columbus,OH DikpalReddy UniversityofMaryland,UMIACS,CollegePark,MD JohnR.Josephson,Ph.D. TheOhioStateUniversity,Columbus,OH RamaChellappa,Ph.D. UniversityofMaryland,UMIACS,CollegePark,MD TimothyN.Miller TheOhioStateUniversity,Columbus,OH

INTRODUCTION:PERSISTENTVIDEOSURVEILLANCE Asvideocamerasandothersensorsbecomecheaperandeasierto network,sensornetworksbecomeincreasinglyattractivemeansto acquireusefulinformationformilitaryoperations,suchasvideosurveil lanceforfacilitiesprotection,andpersistentsurveillancetomaintain sensorycontactwithtargetsofinterest(Pendal,2005).However,without assistancefromautomation,humanswillbeoverloadedbyinformation, andunabletouseiteffectively. Whatinformationconsumesisratherobvious:itconsumesthe attentionofitsrecipients.Henceawealthofinformationcreatesapov ertyofattention,andaneedtoallocatethatattentionefficientlyamong theoverabundanceofinformationsourcesthatmightconsumeit. (Simon,1971) 35

36ECKROTHETAL. Itisdifficultforhumanstovigilantlymonitoralargenumberofvideo feedsforextendedperiodswithoutfatigue,orcomplacency,especiallyif theyareaskedtorecognizethesignificanceofrareeventsinacomplex streamofevents(Warm,et.al.,1996;Grier,et.al.2003;Pattyn,et.al., 2008).Moreover,sometimeseventsofinterestcannotberecognized simplyfromthevideowithoutknowingwhereaspecificcameraispoint ing,atarestrictedarea,forexample.Keepingtrackofthelocationand significanceofacamerasfieldofviewimposesanadditionalcognitive burden.Thecognitiveburdenisevengreaterifrecognizingeventsof interestrequirestrackingentitiesastheybecomevisibleindifferentcam eras,andaccessingmentalmapstounderstandthesignificanceof motionpaths. Automationcanhelp.Itcanpotentiallyprovideuserswithalerts basedonrecognizingindicatorsofthreateningbehavior,includingbehav iorthatcannotbedetectedwithasinglecamera.Forexample,themove mentofanentityfromplacetoplaceinrelationtothemapmightshowa patternindicativeofscoutingtheperimeterofafacility,wherenosingle cameraviewshowsanythingsuspicious.Manyunsolvedtechnicalprob lemsremain,however,includingproblemsabouthowtoextractneeded informationfromvideoimagery,andproblemsofhowtoprocess acquiredinformationtoclimbthelevelsofabstractionfromindividual cameracenteredframeworkstoaworldcenteredframework,andfrom motiondescription(ObjectO5604movedalongpathP52fromlocation L12attimeT55toL13atT66.)tobehaviordescription(O5604proceeded slowlynorthonroadR3totheintersectionwithR7,turnedEastonR7 .)torecognitionofindicatorsofthreat(O5604slowlycircledthefacil ity.). Abundantvideoalsothreatenstooverloadcommunicationsand storagesystems.Typically,thesensedbitsarehighlyredundantforthe tasktobeperformed.Forexample,whenitisimportanttotrackamov ingobject,theimportantinformationiswhichpartoftheimageischang ingcontinuously.Tounderstandwhichregionisrelevant,subtractionof theunchangingbackgroundisneeded,andifitcanbeperformedatthe sensoritself,significantlyfewerbitsneedtobetransmitted.Accordingly, anewframeworkcanbeenvisagedinwhichthetraditionalwaysofsens inganimage(inrectangularpixels)arereplacedbysensingtheimage informationinacompactform(compressedsensing),andthenrecon structingtherequiredimage,asneeded,fromthiscompactinformation. Thispaperdescribessomeelementsofrecentprogressintechnology forvideosurveillancethatspecificallyaddressmethodsforbackground subtractiontodetectchangesfromframetoframeinvideo,trackingof

AutomatedVideoSurveillance37 viewedobjectsincameracenteredimagespace,trackingofobjectsin worldorientedobjectspaceusinginformationfrommultiplecameras, andmethodsforclimbinglevelsofabstractionindescriptionsof behavior. BACKGROUNDSUBTRACTIONANDTRACKING USINGCOMPRESSIVECAMERAS Introduction Mostcamerasinstalledforsurveillanceapplicationsareusedforsim plevisiontaskssuchasdetectionandtracking,poseestimation,and3D reconstructionfromsilhouettes,andveryfewforhigherleveltaskssuch asactivityrecognition.Inmanyofthesetasks,theamountofdatacol lectedishugeforthepurposeoftheapplication.Forexample,atypical backgroundsubtractionalgorithmcomparesasingleframe,fullresolu tionimageobtainedfromacameratoanotherfullresolutionimagethat hasbeenlabeledthebackground.Pixelsthatdifferbetweenthetwo imagesareconsideredforeground.Theresultisblobsinthefore groundthatrepresentobjectsthatarebelievedtohavemovedbetween thetimethebackgroundimagewasacquiredandthetimethefore groundimagewasacquired(usually,thebackgroundimageisupdated everytimeanewframeisobtainedfromthecamera,sothatmovements fromoneframetothenextaredetectedandplacedintheforeground). Theresultingforegroundimageisoftenrepresentedasablackandwhite fullresolutionimage,whereblackrepresentsbackgroundandwhiterep resentsforeground;iftheseimagesareobtainedfrom,say,parkinglot surveillance,thentheforegroundblobswilllikelyresemblesilhouettes ofmovingcarsandwalkingpeople.Yet,theseblobsoccupyonlysmall regionsoftheimage,since,inmostsurveillancevideo,therearefew movingobjectsatanyonetime.Similarly,whentracking,weareinter estedinonlyasmallregionoftheimagethatismoving,anddonotcare abouttheotherpartsoftheimage.Neverthelesstheentireimageis commonlysensedandtransmittedtoacentrallocationforprocessing. Thismeansthatahugeamountofdata,whichisultimatelyuseless,is collectedandtransmitted,thuswastingsensingandbandwidth resources.Inthispaper,wepresentanapproachtoalleviatingthisprob lem.Weshowourapproachontwovisionapplications:backgroundsub tractionandtracking.Inbothoftheseapplications,weusea compressivecameratoobservetheimagesandthenprocessingis doneonthesemeasurements.

38ECKROTHETAL. Acompressivecameraisadevicethathasbeenbuiltontheprinci plesofcompressedsensing(Candes,2006).Suchacamerameasures,not theimagepixelslikeaconventionalcamera,butasmallnumberofran domlinearprojectionsoftheimage.Thenumberofrequiredmeasure mentsissignificantlyfewerthanthenumberofpixelsinaconventional camera,andthisinherentlyreducesthespaceneededtostorethe image.Thefullimage(intheformofpixels)canbereconstructedfrom theselinearcombinations. Wepresentaninnovativeapproachwhereinvisiontasksareper formed,notonthefullimages,butonthecompressedmeasurements directly,withouttheneedtoreconstructthefullimages.Becausethefull imageisnotneededforthetasks,asignificantreductionisachievedin theamountofdatathatneedstobesensed.Inthenextsection,wepro videabriefintroductiontocompressedsensingtheoryandcompressive cameras.Wethenpresentourapproachtobackgroundsubtractionand trackingoncompressedmeasurements. CompressiveCamerasandComputerVisionApplications CompressedSensing.Supposewehaveanimage x ofsize N 1 (i.e.,vectorized),thenwecanrepresenttheimageinsomebasis as x = (1) where isasparsecoefficientvector( sparse).Thevector has veryfewlargenonzerocomponentsindicatingthattheimagecanbe compressed.Waveletsareanexampleofsuchabasis. IntheCSframework(Candes,2006)wedonotmeasurethe larg estelementsof ,butweinsteadmeasure M < N linearprojectionsof theimage x ontoanotherbasis . (2) where arethecompressedmeasurements.Since M < N , the
y

y = x =

systemofequationsareunderdetermined,bututilizingthesparsityof wecanrecoverthesignalbysolvingthefollowing l1 optimization problemcalledBasisPursuit.

= arg min s.t.y = (3) 1

Compressivecameras.Basedoncompressivesensing(CS)theory,a prototypesinglepixelcamera(SPC)wasproposedin(Wakin,2006).The

AutomatedVideoSurveillance39 SPChardwareisspecificallydesignedtoexploittheCStheory,anddiffers fromaconventionalcamerabyusingonlyasingleopticalphotodiode (infrared,hyperspectral,orvisual)alongwithadigitalmicromirror device(DMD).Italsocombinesthesamplingandcompressionprocess, unlikeaconventionalcamera.TheDMDisusedtogeneratetherandom linearprojectionsdescribedinthetheory.Acompressivecamera,suchas SPC,provides,notthepixels,butthecompressedmeasurements.Thisis especiallyusefulinbandwidthconstrainednetworks,wherethecom pressedmeasurementscanbetransmittedtoacentrallocationforpro cessing,andwheretheimagecanberecovered.Furthermore,the cameracanbechangedfromonespectralmodetoanotherbyreplacing onlyonephotodiode. BackgroundSubtraction.Backgroundsubtractionisfundamentalin detectingandtrackingobjects.Indetectionandtrackingwearetypically interestedinmovingobjects.Toidentifywhichobjectsaremovinginan imagewesimplysubtractthestaticbackgroundfromtheimage.Fre quently,however,thebackgroundisnotreallystatic.Typicalchangesin backgroundincludechangeinilluminationandhighfrequencychanges suchasleavesandwaves.Intheprevailingbackgroundsubtractionalgo rithms,thebackgroundandforegroundarefullysampledimagesfrom conventionalcameras.Afterthedifferenceoperation,thebackground imagesarediscardedorusedinfuturebackgroundmodels.This approachofdiscardingfullysampledimagescanbeveryexpensivein normalimaging,andisparticularlysoinhyperspectralimaging.Usinga compressivecamera,suchasanSPC,canalleviatetheproblem.Butsens ingtheforegroundandbackgroundimagesinacompressedform,and thenreconstructingtheimagestoperformbackgroundsubtraction,can becomputationallyexpensive.Instead,in(Cevher,2008)weshowedthat backgroundsubtractedsilhouettescanbeobtaineddirectlyfromthe compressedmeasurements.Whilebackgroundsubtractiononcom pressedimagesisnotnew(Aggarwal,2006),unlikepreviousapproaches, wesensetheimagesdirectlyinacompressedform.Wealsoshowthat, sincethedifferenceimageswhichwedesirearesparseinthespatial domain(Fig.1.inCevher,2008),theyshouldbesparserthancomplete imagesintheappropriatebasis.Thispermitssensingtheimageswith evenfewermeasurementsthanthosethatcompressivecamerasalready permit. Theresultsofthisbackgroundsubtractionapproachcanbeseenin Figure2.1.Thebackgroundimage(toprow)andthetestimage(second row)arereconstructedfromthecompressedmeasurements.Thediffer enceoftheseimagesisshowninthethirdrow.Thefourthrowshowsthe

40ECKROTHETAL. differenceimagereconstructeddirectlyfromthecompressedmeasure ments.Thecolumnscorrespondtomeasurementrates, 2%,1%and0.5%respectively.

M N

,of50%,5%,

Figure2.1 BackgroundSubtractionusingaSinglePixelCamera Weadapttothechangesinbackgroundbyupdatingthebackground model,notbyreconstructingthebackgroundimages,butbyusingthe compressedmeasurementsthemselves(Fig.2.ofCevher,2008).This allowsustoadapttovariouschangingbackgroundswithouttheneedto reconstructthefullimagesfromcompressivemeasurements.Weapplied ouralgorithmondatacollectedunderchangingillumination.Weshowed thatouradaptivebackgroundsubtractionalgorithm,overcompressive measurements,underchangingillumination,worksaswellasworking withnormalfullimages(Fig.6.ofCevher,2008). Itshouldbenotedthatouralgorithmcanalsorecovertheappear anceoftheobjectsbyreconstructingasingleauxiliaryimage.Hence, dependingontheapplication,wecaneitherrecoversilhouettesonly,or withminimaladditionalcomputation,recovertheappearanceofthe object.Webelievethatthisisasignificantimprovementoverprevious sensingandbackgroundsubtractionmethods. Insummary,ourbackgroundsubtractionalgorithm,notonlyworks oncompressedmeasurements,butalsotakesadvantageofthelimited

AutomatedVideoSurveillance41 imageinformationrequiredforthetasktodecreasetheamountofsens ingrequired,andhencetherequirementsforstorageandbandwidth. Further,thealgorithmcanadapttochangesinbackground. Tracking.Manytrackingalgorithmsusebackgroundsubtractedsil houettestotrackobjects.Inmanyscenariosthebackgroundsubtracted silhouetteissparseintheimagedomain.Thissparsitytranslatesdirectly intosparsityoftheobjectparameter(suchaslocation,volume,etc.)in space.Sincewewouldliketoestimateasparseparameterwhichcorre spondstoasparseregioninanimage,itiswasteofsensingresourcesto observecompleteimagesusingconventionalcameras,performback groundsubtractiontothrowawaymostoftheimage,andthentrack.In (Reddy,2008)weshowedtrackingasasparseapproximationproblem, andwearguedthattodosoweneedtomeasure,nottheentireimage, butafewrandomprojectionsoftheimagevectors,appropriatelycho sen. Toillustratetheidea,weshowtheresultsonthesimplercaseof backgroundsubtractedsilhouetteimages.Theobjectparameter(e.g., location)andtheimagepixelintensityarerelatedusingEquations7and 8of(Reddy,2008).ThisrelationcanbevisualizedinFigure2.2.

Figure2.2 Groundplanetrackingscenario Wetestedthealgorithmonanoutdoorscenewithwalkingpeople.A fewimagesofthetestsceneareshowninFigure2.3.

42ECKROTHETAL.

Figure2.3 Movingpeopletrackedusingbackgroundsubtractedsilhouetteimages Weresolvethepositionsofoccludingobjectsbyfusingthesilhouette imagesfrommultipleviewsinamannersimilarto(Khan,2006).Thesil houetteblobsindifferentcamerasaresimplymappedontotheground planeusingthehomographyofeachcameraasshowninFigure2.2.In suchamappingtheimageblobscorrespondingtoanobjectoverlapon thegroundplaneandtheregionofoverlapistreatedasthelocationof theobject. Oneoftheprincipaladvantagesofthisapproachisthatiteasily scalestoalargenumberofcamerassincethecomplexityofthealgo rithmdependsonlyonthegridsizethatisusedforlocalization.The speedupofthisapproachisshowninFigure2.4.

Figure2.4 Speedupachievedasthenumberofcamerasincreases

AutomatedVideoSurveillance43 TheresultsoftrackingpeopleinanoutdoorsceneareshowninFig ure2.5.

Tracking Results in an Outdoor Scene 0 -5 -10 -15 -20 -25 -30 -35 -40 -45 -50 0 10 20 30 40 50 Person 1 track Person 1 ground truth Person 2 track Person 2 ground truth

Figure2.5 Trackingresultsover500framesusingdifferenceofcompressedfore groundandbackgroundmeasurements Insummary,wehavedescribedmethodsforbackgroundsubtraction andtrackingusingthecompressedmeasurementsofacompressivecam era. USEOFABDUCTIVEINFERENCINGFORTRACKINGINOBJECTSPACEAND THREATDETECTION Introduction Astechnologyadvances,andvideocamerasbecomecheaperand easiertodeploy,videosurveillancewilluseincreasingnumbersofcam eras,oftenwithoverlappingviews,whilegapsincoveragewillbecome smaller.Inprinciple,smallergapswillmakeitharderforobjectsofinter esttoescapedetection,andenableobjectstobetrackedforlongerperi odsoftime.Longertracksenablemorecomplexbehaviortobe

44ECKROTHETAL. recognized,andincreasethelikelihoodthatthelocationofanobjectof interestwillbeknownwhenthetimecomestoact.Inprinciple,overlap pingviewsenable3Dfromstereo,reducedpositionaluncertainty,and improvedrecognitionofbehaviorandobjectcharacteristics.However, forthesepotentialstoberealized,anumberoftechnicalproblemsmust besolved.Inthissectionwedescribemethodsfortrackinginworldori entedcoordinates(objectspace)usinginformationfrommultiplecam eras,andmethodsforascendinglevelsofabstractionindescriptionsof behavior.Theinputsareassumedtobetheoutputsofsinglecamera backgroundsubtractionandtrackingalgorithmsintheformofreported detections(detectedmovements)andtrackupdates. TrackinginObjectSpace Whentrackingisdoneinimagespaceusingsinglecameradata, theproblemremainsofdeterminingwhichreportedtracksrepresentthe sameexternalobjects.Theinformationformakingthedeterminations consistsofinformationaboutthefieldsofviewofthecamerasinworld orientedcoordinates,thelocationsofthedetectionsortrackinthe images,thevisiblecharacteristicsoftheentitiesdetectedortracked,and knowledgeofentitiestypesandterrainconditionsthatconstrainmove ments,andtherebysometimesenablereidentificationofentitiesafter theypassthroughgapsinsensorcoverage. Sincetrackupdatereportsarejustentitydetectionreportswiththe additionalinformationassociatinganewdetectionwithasequenceof earlierones,thepossibilityarisesofdoingthetrackinginobjectspace, assumingthatdetectionscanbemappedtoworldorientedlocations, e.g.,whenentitiescanbeassumedtobeonthegroundplane.Benefits oftrackinginobjectspaceinclude: Benefitsfromusingworldunits(e.g.,meters)whenmeasuring trackvelocities.Thisenableskinematicconstraintsfromphysics andobjecttypestobereadilyusedtoconstrainexpectedand possiblemovements. Processinginobjectspacecandeterminewhichcamerasshould seecertainentities,andthenthosecameras(orthedatafrom thosecameras)canbequeriedtodetermineifanentitywas seenasexpected.Ifitwasnot,thenaputativeentitymaybe noise,ortheremightbeocclusion,orerrorinestimatedsensor location,alignmentorsensitivity,orsensorfailure.Dependingon theoperationaldemandsoftheapplication,thesepossibilities canbeevaluatedexplicitly,andsystemsadaptedaccordingly.

AutomatedVideoSurveillance45 Focusofattentioncanbedirectedtolocationsofinterest, expressedinworldcoordinates,sothatcamerascanbedirected topan/zoom/focustocorroboratedetections,anticipatethe arrivalofentitiesinthefieldofview,refinelocationestimates,or pickupadditionaldetails. However,adrawbackofmappingdetectionstoobjectspacebefore trackingisthatimagedetailsthatmightbeusefulfortracking(e.g., shapes,color)needtobetransmittedtotheprocessinglocation,orleft outofconsiderationbythetrackingalgorithm. TrackingUsingAbductiveInferencing Abduction.Abductiveinferencing(inferencetothebestexplanation) isadistinctivepatternofreasoningthatisubiquitousinordinarylife,and inthetrainedreasoningofintelligenceanalysts,diagnosticians,accident investigators,andscientists(Josephson&Josephson,1996).Itcanbe viewedaspartofcommonsenselogic.Researchershavedevelopeda varietyofcomputationalmodelsofabductiveinferencing,althoughthey havenotalwaysreferredtoitassuch.Ourworktreatsabductionasa typeofknowledgebasedproblemsolving,wherethechoiceofamethod (andknowledgerepresentation)forataskdependsonsuchconsider ationsastheformsinwhichknowledgeisavailable,andtheoperational demandsonqualityandtimelinessofsolutions(Chandrasekaran& Johnson,1993). Inusingabductiveinferencingfortrackingfromdetectionreports, thereportsaretheinputsthatneedtobeexplained.Hypothesesavail ableforexplainingthemareofthreetypes:noise(e.g.,spuriousdetec tions,lightreflections,camerajitters),movementsofknownentities,or initialdetectionsofanentity. Noise.Severaldifferenttypesofnoiseoccur,andmustbeeitherfil teredoutinimageprocessing,oraccountedfor.Sometypesofnoiseare highlytransient.Theseincludeglintsofsolarreflectionandpixellevel noisefromphysicalsensors.Anothertypeofnoiseconsistsofreal objectsthatarefixedinoverallposition,butmoveinresponsetowind (e.g.,vegetation,flags).Anothertypeisaresultofshadowsofobjects visibleinthescene(andmovingwiththem)orshadowsofobjectsthat areoutofview,suchasaircraftandclouds.Anothertypeconsistsofreal object,butoftypesthatarenotofinterestfortheapplication,suchas birds,smallanimals,andtumbleweed. Smallandtransientdifferencesbetweenvideoframescanbefiltered outasprobablenoise,andnotreportedasdetections.Thresholdscanbe

46ECKROTHETAL. setempiricallytominimizebothfalsepositivesandfalsenegatives,tak ingintoaccountestimatedcostsforeithertypeofmistake.Withfeed back,thresholdsmightbesetadaptivelybasedonsuccessesin interpretingdetectionsintotracks,andfailurestodetectentitiesthatare otherwiseindicatedtohavebeeninthefieldofview,eitherbyothersen sors,orbyfillinginthegapsindashedlines. Noiseconsistingoftheeffectsofwindonfixedobjectswillprobably bestbetreatedasexplicithypothesesaboutvegetation,flags,andsuch, oratleastasbelongingtotheclassoffixedobjectsthatstayinplace,but occasionallywiggle.Anadvantageofdoingsoisthathypothesesabout suchobjectscanbemaintainedintheworldestimate,sosubsequent detectionswithsimilarcharacteristicsinthesamelocationwillnotdraw additionalcomputationalorattentionalresources,andthelikelihoodcan bereducedofconfoundingsuchobjectswithnearbymovingobjectsof interest. Shadowsofvisibleobjectsdeservespecialtreatment.Shadowstend tolookverymuchlikethebackground,butdimmer.Thissimilaritycanbe exploitedtohelpfilterthemout,treatingthemaspartofthebackground (SextonandZhang,1993).However,sometimesshadowsaretoodarkfor thebackgroundtoshowthrough,andothermethodsareneeded.One canimagineanabductiveapproachthattreatsshadowsaspartsofa complete3Dscenereconstructionthatincludesanilluminationmodel. Shadowsarethenpredictedbytheillumination,togetherwiththeopac ityoftheshadowcausingobjects,andexplainwhycertainsegmentsof surfacesaredarker,shapedastheyare,andmovewiththeobjects;these propertiesofsurfacesegments,inturn,explainpropertiesof2Dimage segments.However,toourknowledge,atreatmentofshadowsofthis sorthasnotbeenachievedtechnically. Similartotheeffectsofwindonfixedobjects,smallobjectsthatare notofinterestfortheapplicationwillprobablybestbetreatedasexplicit hypothesesaboutbirds,smallanimals,windblownobjects,andsoon. Explicitrecognitionofsuchphenomenacansparecomputationaland attentionalresources,andenablebetterdiscriminationofobjectsthat reallyareofinterest. Noneofthesemethodsforhandlingnoisehavebeenimplemented yetinourexperiments,exceptforthefilteringoutofsmallandtransient differencesbetweenvideoframessotheydonotcausedetections,and thefilteringoutofdetectionsthatdonotfindsubsequentinterpretation asbelongingtotracksthatmovesignificantlyfromtheirinitialpositions. MappingDetectionsfromImageSpacetoObjectSpace.Weassume thatcameralocations,alignments,angularfieldsofview,andimagedis

AutomatedVideoSurveillance47 tortioncharacteristicshavebeenpredeterminedaccuratelyenoughfor practicalpurposes.Thisinformationaboutthecamerasisthenavailable foruseinmappingdetectionstoobjectspace.Wherethefieldsofview ofcamerasoverlap,3Dsurfacescanbereconstructedfromstereopro cessing,andthese3Dsurfacescanbetransformedtosurfacesinobject spaceusingthecamerainformation.However,3Dsurfacereconstruction iscomputationallyexpensive,andsometimestherewillbenooverlap. Yet,sometimesweareonlyinterestedintrackingobjectsmovingonthe terrainsurface,whenviewedfromrelativelyhighup.Wehavebeen experimentingwithdatasetsofthissort.Underthesecircumstances,the locationsofdetectionscanbemappedfromimagespacetoobjectspace byapplyingawarp,thatis,aperspectivetransformation,thatmaps imagelocationstolocationsinthegroundplane.Wehaveimplemented thismethodforourexperiments.Trackingisthendoneintheobject space(inthegroundplane).Wherefieldsofviewoverlap,detectionsare intermingledinthegroundplain,andtrackingusesdetectionswhose sourcesaredifferentcameras.Wherefieldsofviewdonotoverlap,track ingisalsodoneintheobjectspace,butusingdetectionsfromsingle sources.Awarpisinvertible,sotracksinobjectspacecanbemapped backtoimagespacefordisplay. TrackingExperiments Wetreateddetections,withlocationsmappedintoobjectspace,as theitemsthatneedtobeexplainedbyabductiveprocessing.Aswe described,hypothesesavailableforexplainingthesedetectionsareof threetypes:noise,movementsofknownentities(aknowntrackcontin uestothere),orfirstdetectionsofanentity.Afirstdetectionwilltypi callycorrespondtoanobjectenteringthefieldofview,butitmightbea previouslyunmovingobjectthatbeginsmoving(e.g.,anautointhepark inglot). Tracksmaysplitorjoin.Suchphenomenaareseeninthedataused inourexperiments.Peoplewalkinagroup,whichistrackedasasingle objectuntiltheygotheirseparateways.Peoplewalkingseparatelycon vergeandwalktogether.Peoplegetoutofautos,orenterthem.Explic itlynotingsplittingandjoiningeventsisimportantforclassifyingsuch eventsexplicitly,asmultiplepeopleexitingavehicleforexample.Classi fyingsucheventsmaybeusefulforrecognizingsignificantbehaviorand mayhelpwithdecomposingtracksintosegmentsoverwhichthetracked objecthasaconsistenttype(e.g.,person,vehicle,groupofpeople). Thus,thedesiredoutputatthefirststageofabductiveprocessingcon

48ECKROTHETAL. sistsoftracksegments,whereatracksegmentgoesfromatrackorigi nationpoint,orfromasplittingorjoiningevent,toanothersplittingor joiningevent,ortoatrackendingpoint.

Figure2.6 Trackingmultipleentities Ourexperimentsusedvideodatafromtwofixedcamerasmounted onabuildingattheARLfacilityinAdelphiMaryland,andlookingdown wardataparkinglot.Thefieldsofviewpartiallyoverlap.Individualper sons,groupsofpeople,andvehiclesareseenmovingabout(seeFigure 2.6).Entitiesmayinitiallyappearbyenteringthefieldofview,orfrom withinthefieldofview,suchaswhenavehiclebeginsmovingafter remainingparkedforsometime.(Detectionscomefromabackground subtractionprocess,sounmovingobjectsarenotdetected.) Figure2.7showstheresultsoftrackingavehiclebasedondetections fromtwodifferentcameras.Thedetectionsweremappedtoobject spacebywarpingtheimagestothegroundplane.Thetrackingwasdone inobjectspace,withtheresultsmappedbackintothecameraimage spacesfordisplayusingtheinversewarps.Theresultswerealsomapped totheimagespaceofanoverheadphoto,whichisshowninthefigure. Thestraightlinesintheoverheadimagecorrespondtotheedgesofthe viewsfromthetwocameras.

AutomatedVideoSurveillance49

Figure2.7 Vehicletrackinthreeviews Limitations.Themoregeneraltaskofautomatedvideosurveillance involvesmanymoreelementsthanthoseinvestigatedinourexperiments todate.Thenumberofvideocamerasemployedmaybemany,andsome orallofthemmaybemobile.Additionally,camerasmaybeactivelycon trolled,panned,zoomed,orfocusedtomaximizeinformationpickup fromobjectsorregionsofinterest.Toconservepower,acameramight beturnedononlywhenneeded,e.g.,whenanobjectisexpectedto comeintothefieldofregard,ortocorroborateuncertaininformation fromanothercamera.Thegeneralproblemalsoincludestheneedto ascendlevelsofabstractionindescriptionsofbehavior. ClimbingLevelsofAbstractioninDescriptionsofBehavior Ourapproachtoprocessingatmultiplelevelsofabstractionisbased onviewingtheproblemasoneoflayeredabduction,wheretheconclu sionsofonelayerbecomedatatobeexplainedatthenexthigherlayer (Josephson&Josephson,1996,Ch.10;Schenk,1995;Banerjee,2006). Ourcurrentdesignforautomatedvideosurveillancepostulatesthefol lowinglayers,characterizedbythetypesofhypothesesevaluated: Detectedmovements(JDLfusionLevel0) Tracksegments(JDLfusionLevel1) Classifiedtracksegmentstrackedobjectsareclassified(person, groupofpeople,vehicle,etc.)andpossiblyidentifiedindividually

50ECKROTHETAL. (JohnSmith,UPStruck,vehiclewithlicenseplateWOOF).Split tingandjoiningeventsareclassifiedbythetypesoftheirinputs andoutputs(groupofpeoplesplitsintothreeseparatepeople). (AlsoJDLfusionLevel1) Behaviorclassifiedtracksegmentsaredescribedinrelationto terrainfeaturesandentitystates(vehicleslowlyexitedthepark inglot,pedestrianwalkedonthesidewalk),andsplit/joinevents aredescribedintermsofenteringandemergingfromvehicles, andfromgroupsofpeople.(Vehiclecametoastopinthepark inglot,twopeopleemergedandwalkedtowardthebuilding entrance.)(JDLfusionLevel2) Scriptsstereotypicalpatternsofactivity(Schank&Abelson, 1977)(personparkingavehicleinthevisitorsectionofthepark inglotandproceedingonfoottothebuildingentrance;vehicle cruisingtheparkinglotlookingforanemptyspot).Thescripts areprestored,andtaggedwiththeirthreatsignificance,e.g.,as benignorpossiblythreatening.Behaviorthatcannotbeinter pretedaspartofaknownscriptisconsideredtobeanomalous, especiallywhenarelativelycompletelibraryofscriptshasbeen builtup.Anomalousbehaviortriggersanalert.(JDLfusionLevel 3) AdescriptionofJDLfusionlevelscanbefoundinSteinberg,Bowman& White(1998).Wehavenotyetimplementedanyofthelayersabovethat oftracksegments,althoughwehavepreviouslyexperimentedwithlay eredabductionmodelsforseveralotherapplications(op.cit.). SUMMARY Wehavedescribedrecentprogressintechnologyforvideosurveil lancethataddresses:methodsforbackgroundsubtractiontodetect changesfromframetoframeinvideo,trackingofviewedobjectsincam eracenteredimagespace,trackingofobjectsinworldorientedobject space,andmethodsforclimbinglevelsofabstractionindescriptionsof behavior. IMPLICATIONSFORFUTUREARMY Althoughadvancesinvideohardwareandnetworkingofferenor mouspotentialforsurveillanceapplications,theoverloadingofcommu nications,storage,andhumanattentionwillseverelylimitthepotential, unlessinformationcanberepresentedcompactlywhilepreservingits

AutomatedVideoSurveillance51 significanceandunlessahighdegreeofautomationcanbeachievedin processingincomingvideoinformationtodetectitsrelevancefordeci sionmaking.Thenhumanswillonlyneedtoviewanextremelysmallsub setoftheimagesavailable.Manyunsolvedtechnicalproblemsremain, includingproblemsabouthowtoextractneededinformationfromvideo imagery,andproblemsofhowtoprocessacquiredinformationtoclimb thelevelsofabstractionfromindividualcameracenteredframeworks,to aworldcenteredframework,andfrommotiondescription,tobehavior description,torecognitionofindicatorsofthreat.However,significant progressisbeingmade,asillustratedbythispaper,andhardpartsofthe problemsarebeingtackledwithsomesuccess.Continuingprogresscan beexpected,andimperfectbutusefulsystemsshouldbeavailableover thenextfewyears. COLLABORATIONS Theworkonbackgroundsubtractionandtrackingusingcompressed imageswasdoneinclosecollaborationwithVolkanCevher,Marco DuarteandProf.RichardBaraniukofRiceUniversity.Theworkonabduc tivetrackinginobjectspacebenefitedfromcollaborationwithSean McGinnisandJohnHancockofArtisTech,Inc.andRobertWinklerofARL. REFERENCES
Aggarwal,A.,Biswas,S.,Singh,S.,Sural,S.,Majumdar,A.K.:ObjectTrackingUsing BackgroundSubtractionandMotionEstimationinMPEGVideos.In:ACCV,Springer (2006)121130. Banerjee,B.(2006).ALayeredAbductiveInferenceFrameworkforDiagramming GroupMotions,LogicJournaloftheIGPL14(2)363378,2006. Bulich,C.,Klein,A.,Watson,R.,andKitts,C.2004.CharacterizationofDelayInduced PilotingInstabilityfortheTritonUnderseaRobot.IntheProceedingsofthe2004IEEE AerospaceConference,BigSkyMT. Candes,E.:Compressivesampling.In:ProceedingsoftheInternationalCongressof Mathematicians.(2006). Casper,J.andMurphy,R.R.(2003).Humanrobotinteractionsduringtherobot assistedurbansearchandrescueresponseattheWorldTradeCenter.IEEETransaction onSystems,ManandCyberneticsPartB:Cybernetics,Vol33,No.3,pp367385. Cevher,V.,Sankaranarayanan,A.,Duarte,M.F.,Reddy,D.,Baraniuk,R.G., Chellappa,R.:CompressiveSensingforBackgroundSubtraction.In:ECCV,Marseilles(Oct 2008). Chen,J,Y.C.andThropp,J.E.2007.Reviewoflowframerateeffectsonhuman performance.IEEETransactiononSystems,ManandCyberneticsPartA:Systemsand Humans,Vol.37,pgs10631076.

52ECKROTHETAL.
Chandrasekaran,B.&Johnson,T.R.(1993).GenericTasksandTaskStructures: History,CritiqueandNewDirection.In:David,J.M.,Krivine,J.P.&Simmons,R.,(Eds), SecondGenerationExpertSystems.NewYork:SpringerVerlag. Endsley,MicaandGarland,Daniel2000.SituationAwarenessAnalysisand Measurement.Mahwah,NJ:LawerenceEarlbaumAssociates. Fiorini,P.andOboe,R.1997.InternetBasedTelerobotics:Problemsand Approaches.IntheProceedingsoftheInternationalConferenceonAdvancedRobotics, Monterey. Grier,R.A.,Warm,J.S.,Dember,W.N.,Matthews,G.,Galinsky,T.L.,Szalma,J.L.& Parasuraman,R.(2003).TheVigilanceDecrementReflectsLimitationsinEffortful Attention,NotMindlessness.HumanFactors:TheJournaloftheHumanFactorsand ErgonomicsSociety45:349359 Hewish,M.2000.PilotlessprogressreportUAVshavemadeexceptionalstrides recently.JanesInternationalDefenseReviewOctober01,2000.http:// search.janes.com Josephson,J.R.&Josephson,S.G.,Eds.(1996).AbductiveInference:Computation, Philosophy,TechnologyNewYork:CambridgeUniversityPress. Khan,S.M.,ShahM.,2006.Amultiviewapproachtotrackingpeopleincrowded scenesusingaplanarhomographyconstraint.In:ECCV,2006,vol.4,133146. Krotkov,E.,Simmons,R.,Cozman,F.,andKoenig,S.1996.Safeguarded TeleoperationforLunarRovers:FromHumanFactorstoFieldTrials.InProceedingsofthe IEEEPlanetaryRoverTechnologyandSystemsWorkshop,Minn.,MN. PattynN,NeytX,HenderickxD,SoetensE.,2008.Psychophysiologicalinvestigation ofvigilancedecrement:Boredomorcognitivefatigue?Physiology&Behavior[serial online].January28,2008;93(1/2):369378.Elsevier.Availablefrom:AcademicSearch Complete,Ipswich,MA.AccessedApril15,2009. Pendall,MajorDavidW.,2005.PersistentSurveillanceanditsImplicationsforthe CommonOperatingPicture.MilitaryReview:4150. Reddy,D.,Sankaranarayanan,A.,Cevher,V.,Chellappa,R.:Compressedsensingfor multiviewtrackingand3DvoxelreconstructionIn:ICIP,SanDiego,CA(Oct.2008). Schank,R.&Abelson,R.(1977).Scripts,Plans,Goals,andUnderstanding,Hillsdale, NJ:LawrenceErlbaum. Schenk,T.(1995).ALayeredAbductionModelofBuildingRecognition,inAutomatic ExtractionofManMadeObjectsinAerialandSpaceImages,Gruen,A.;Kuebler,O.; Agouris,P.(Eds.),Birkhuser,1995. Sexton,G.G.;Zhang,X.,Suppressionofshadowsforimprovedobjectdiscrimination, IEEEColloquiumonImageProcessingforTransportApplications,Volume,Issue,9Dec 1993,pp.9/19/6. Simon,Herbert,1971.InMartinGreenberger(ed.)Computers,Communicationsand thePublicInterest,pp.4041,JohnsHopkinsPress. Steinberg,A.N.,Bowman,C.L.andWhite,Jr.,F.E.(1998)RevisionstotheJDLData FusionModel,Proc.3rdNATO/IRISConf.,QuebecCity,Canada. Voshell,M.,Woods,D.D.,andPhillips,F.2005.Overcomingthekeyholeinhuman robotcoordination:SimulationandEvaluation.ProceedingsoftheHumanFactorsand ErgonomicsSociety49thAnnualMeeting,2630September,Orlando,FL. Wakin,M.B.,Laska,J.N.,Duarte,M.F.,Baron,D.,Sarvotham,S.,Takhar,D.,Kelly, K.F.,Baraniuk,R.G.:Anarchitectureforcompressiveimaging.In:ICIP,Atlanta,GA(Oct. 2006)12731276.

AutomatedVideoSurveillance53
Warm,JoelS.,Dember,WilliamN.,Hancock,PeterA.Vigilanceandworkloadin automatedsystems.Automationandhumanperformance:Theoryandapplications (1996).

54ECKROTHETAL.

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T A N D L A U R E L A L L E N D E R

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T AND LAUREL ALLENDER

This book, produced by the Partners of the Army Research Laboratory Advanced Decision Architectures Collaborative Technology Alliance, also serves as the Final Report for DAAD19-01-2-0009.

SECTIONI.COLLECTING,PROCESSING,ANDDISTRIBUTINGBATTLEFIELD INFORMATION SECTIONI COLLECTING,PROCESSING,ANDDISTRIBUTINGBATTLEFIELD INFORMATION

14SECTIONI

Chapter

3. AGILECOMPUTINGANDITSAPPLICATIONSTOTACTICALMILITARY ENVIRONMENTS

NiranjanSuri,Ph.D.1,MarcoCarvalho,Ph.D.1,DaniloAnsaloni1, MarcoArguedas1,GiacomoBenincasa1,ErikaBevegn1, JeffreyM.Bradshaw,Ph.D.1,MaggieBreedy1,SteveChoy2, JesseKovach2,MassimilianoMarcon1,RaffaeleQuitadamo,Ph.D.1, MatteoRebeschini1,LarryTokarcik2,MauroTortonesi,Ph.D.1, RobertWinkler2

1

InstituteforHumanandMachineCognition,Pensacola,FL 2 ArmyResearchLaboratory,Adelphi,MD

INTRODUCTION Agilecomputingisaninnovativemetaphorfordistributedcomputing systems,whichprescribesanewapproachtotheirdesignandimplemen tation.Thischapterdescribestheoverallagilecomputingmetaphoras wellasoneconcreterealizationthroughamiddlewareinfrastructure. Agilecomputingmaybedefinedastheopportunisticdiscovery, manipulation,andexploitationofavailablecomputingandcommunica tionresourcesinordertoimprovecapability,performance,efficiency, faulttolerance,andsurvivability.Thetermagileisusedtohighlightthe desiretobothquicklyreacttochangesintheenvironmentaswellasto takeadvantageoftransientresourcesonlyavailableforshortperiodsof time.Agilecomputingthrivesinthepresenceofhighlydynamicenviron mentsandresources,wherenodesareconstantlybeingadded,removed, andmoved,resultinginintermittentavailabilityofresourcesand changesinnetworkreachability,bandwidth,andlatency.

55

56SURIETAL. Fromahighlevelperspective,thegoalofagilecomputingistofacili tateresourcesharingamongdistributedcomputingsystems.Atthis broadlevelofdescription,agilecomputingoverlapswithseveralother areasofresearchincludingdistributedprocessing,peertopeerresource sharing,gridcomputing,andclustercomputing.Thefollowingfactorsdif ferentiateagilecomputingfromtheotherareasofresearch. TransientResources Theresourcesinatacticalenvironmentareexpectedtobehighly transient.Indeed,oneoftheperformancemetricsforagilecomputingis definedasafunctionoftheminimumlengthoftimethataresource mustbeavailableinordertobeutilizedproductively.Theexpectationis tosupportenvironmentswhereresourcesareavailableontheorderof secondsorminutes,asopposedtohours,days,orlonger.Therefore, agilecomputingdiffersfromgridcomputingandclustercomputing, whichtargetsenvironmentswhereresourcesaremorestable. LimitedCommunications Thenetworksusedtointerconnectresourcesintacticalenviron mentsareexpectedtobewirelessandadhoc.Theimplicationforagile computingisthatthemiddlewaremustbeabletosupportandoperate inlowbandwidth,highandvariablelatency,andunreliablenetworks. Again,thisdifferentiatesagilecomputingfromgridcomputingandclus tercomputing,wherethenetworklinkstendtobehighperformance andreliable. OpportunisticResourceExploitation Anothergoalofagilecomputingistotakeadvantageofunexpected resourcesthathappentobeavailable.Inparticular,thegoalistoexploit newresourcesdynamicallyjoiningthenetworkenvironmentaswellas existingresourcesthathaveunexpectedsparecapacityfortransientperi odsoftime.Whilesomepeertopeersystemsprovidelimitedformsof thesamecapabilities,agilecomputingextendsthiscapabilitytoproac tivelymanipulatetheavailableresourcesinordertosatisfytheapplica tionrequirements.Manipulationcanincludephysicallymovingresources (forexample,robotsorotherautonomousvehicles)inordertoprovide communicationsorprocessingcapabilitieswhererequired.Theseaddi tionalresourcesmaybeuncommittedresourcesassignedforusebythe

AgileComputing57 middlewareorotherresourceswhichcanbemanipulatedwithoutinter feringwiththeiroriginaltaskassignments.Thisaspectalsodifferentiates agilecomputingfromgridcomputingandclustercomputing. AGILECOMPUTINGMIDDLEWARE TheAgileComputingMiddlewareisaconcreteimplementationthat realizesthegoalsofagilecomputing.Themiddlewareaddressesseveral problemswithinthetacticalmilitaryenvironment.Atafundamental level,themiddlewareprovidesdiscoveryofavailableresourceswithin theenvironment.Themiddlewarealsosupportstaskingresourcesfor computation,communication,orstoragepurposes.Policiesmaybeused toconstrainboththediscoveryandtheresourcetasking(intheAgile ComputingMiddleware,policiesarerepresentedinKAoS,whichisaset ofservicesthatsupportspolicybasedmanagement).Atahigherlevel, themiddlewaresupportsresourceallocationalgorithmsthatmapappli cationrequeststotheavailableresourcesintheenvironment.Thealloca tionmechanismcanadapttochangesinboththeapplicationdemands aswellastheavailabilityofresources.Iftheresourceallocationpolicy changes,themiddlewareiscapableofmigratingcomputationstonew resourcesaswellastransparentlyredirectinganynetworkcommunica tions.Inaddition,themiddlewareproactivelymanipulatestheenviron mentinordertodeploycapabilitieswhereneeded. Theprimarycomponentsthatcomprisethemiddlewareareshown inFigure3.1.ThemiddlewareinterfaceswiththeKAoSPolicyand DomainServicessystemaswellastheXLayerCrossLayersubstrate.The followingsubsectionsdescribeeachofthecomponentsofthemiddle ware. Mockets TheMocketslibraryistheagilecomputingcomponentwhichpro videsthetransportcapabilitytoapplications,aswellastotheothercom ponentsofthemiddleware.ThedesignandcapabilitiesofMocketswere motivatedbyobservationsofproblemsexperiencedintacticalnetwork environments,whicharetypicallywirelessandadhocwithlowband width,intermittentconnectivity,andvariablelatency.TheMockets libraryaddressesspecificchallengesincludingtheneedtooperateona mobileadhocnetwork(whereTCPdoesnotperformoptimally),pro videsamechanismtodetectconnectionloss,allowsapplicationstomon

58SURIETAL. itornetworkperformance,providesflexiblebuffering,andsupports policybasedcontroloverapplicationbandwidthutilization.

Application Application

KAoS Policy And Domain Services

Agile Computing Middleware Mockets Group Manager (Discovery) AgServe FlexFeed Dissemination Service

Cross-Layer Interface

Datagram (UDP) Wireless Network Routing MAC Physical

Figure3.1 ComponentsoftheAgileComputingMiddleware Mocketssupportbothstreamandmessagebasedabstractions, therebyworkingasafeaturecompletereplacementforTCPandUDP sockets.Thestreambasedabstractionsupportsexchangeofareliable andsequencedstreamofbyteslikeTCP,whichsimplifiesadaptingexist ingapplicationsthatuseTCPtouseMockets. Themessagebasedabstractionprovidesmanyenhancedcapabilities thatareonlypossiblewhentransmittingindividual,selfcontainedmes sagesasopposedtoacontinuousstream.Thesecapabilitiesare: Differentclassesofserviceunreliable/unsequenced,reliable/ unsequenced,unreliable/sequenced,andreliable/sequenced. Taggingofmessagestogroupmessagesintodifferentcategories (forexample,fordifferenttypesofdatasuchasvideo,voiceover IP,andcontrol. Prioritizationofmessages,eitherindividuallyorbycategory (basedonthetagsassignedtomessages). Replacementofmessageswhereoldandoutdatedmessages canbereplacedbynewermessagesinordertoreducenetwork traffic. OtherkeyfeaturesofMocketsinclude:

AgileComputing59 1. Applicationlevelimplementationofthecommunicationslibrary inordertoprovideflexibility,easeofdistribution,andbetter integrationbetweentheapplicationandthecommunications layer. 2. Transparentmobilityofcommunicationendpointsfromonehost toanotherinordertosupportmigrationofliveprocesseswith activenetworkconnections. 3. Interfacetoapolicymanagementsysteminordertoallow dynamic,externalcontrolovercommunicationsresourcesused byapplications. 4. Detailedstatisticsregardingtheconnectionanddatatransfer, whichallowstheapplicationandthemiddlewaretoobservethe statusofaconnectionandadaptasnecessarytoobservedprob lemssuchasconnectionloss,dataaccumulation,orsignificant retransmissions. TheMocketslibraryisfurtherdescribedin(Suriet.al.2005),(Torto nesiet.al.2006),and(Stefanelliet.al.2008). GroupManager TheGroupManagercomponentsupportsresourceandservicedis covery.Itenablestheagileandopportunisticexploitationofresourcesby optimizingqueriestofindnodesinnetworkproximityand/ornodesthat areresourcerichorhaveexcesscapacity. TheGroupManagersupportsproactiveadvertisement,reactive search,oracombinationofthetwo.ThesearchisrealizedusingaGnu tellalike(Clip22001)probabilisticsearchmechanism.Inaddition,the radius(intermsofnetworkhops)oftheadvertisementorsearchcanbe controlledonaperrequestbasis,providingapowerfulmechanismto controlhowstronglyorweaklyaservicemaybeadvertisedandhowfara searchrequestmaytravel.Inthecurrentlyrealizedimplementation,dis tanceisdefinedasthenumberofhopsinaMANETenvironment,butcan beadifferent,scenariorelevantparametersuchasbandwidthorlatency. Propagationoftheadvertisementandsearchmessagesoccursvia oneofthreemechanismsUDPbroadcast(thesimplestcase),UDPmul ticast,orviatheXLayerSubstratethatprovidesbandwidthefficient flooding.Inaddition,tunnellingviaTCPsupportsbridgingmultiplenet works.Inallofthesefourcases,eachnodemayselectivelyrebroadcast anincomingmessagetoprovidecontroloverthedistributionradius.

60SURIETAL. Thesecapabilitiesenableapplicationstomaketradeoffsbetween discoverability,bandwidth,andlatency.Proactiveadvertisementuses morebandwidthbutreducesthelatencywhenaclientneedstofinda serviceandviceversa.Onanotherdimension,aservicethatiswidely presentinanetworkdoesnotneedtoadvertisestrongly(or,conse quently,aclientlookingforsuchaservicedoesnotneedtosearch widely)asopposedtoservicesthatarescarce. Groupsmaybeusedtopartitionnetworknodesintodifferentsets therebyrestrictingadvertisementsandqueries.TheGroupManagerpro videssupportfortwodifferentgrouptypes:peergroupsandmanaged groups.Peergroupsarecompletelydecentralized:theydonothavean ownerormanager,butinsteadmaintainnodemembershipindepen dentlyfromtheperspectiveofeachnode.Thisdesignchoicealsoimplies thatthereisnoattemptatmaintainingaconsistentgroupviewforall nodesthataremembersofapeergroup.Inaddition,nospecialmecha nismisrequiredinordertojoinapeergroup.Nodescansimplyqueryor registerresourcesandservicesinthecontextofaspecificpeergroupand willimplicitlybetreatedbytheGroupManagerasbeingmembersofthe samepeergroup. Thedecentralizedanddynamicnatureofpeergroupsmakethem verywellsuitedforresourceandservicesharinginMANETenviron ments.However,foradditionalflexibility,theGroupManageralsosup portscentralizedresourceandservicemanagementbymeansof managedgroups.Amanagedgroupiscreatedbyaparticularnodeandis ownedbythatnode.Othernodesneedtoexplicitlyjoinamanaged groupbysendingamembershiprequesttothegroupownernode. Accesstogroups(ofbothpeerandmanagedtypes)mayoptionally berestrictedusingapasswordorotherauthorizationmechanism, therebypreventingnodesthatdonotpossessthenecessaryauthoriza tioncredentialsfromjoiningaspecificgroup.Suchrestrictedaccess groupsarecalledprivategroupsandareoneofthemechanismsthatcan protectresources. FurtherinformationabouttheGroupManagerisavailablein(Suriet. al.2008)(Suriet.al.2006). AgServe TheAgServecomponentsupportsaServiceorientedArchitecture (SoA)(Estefanet.al.2008)ontopoftheAgileComputingMiddleware. AgServesupportsdynamicdefinition,instantiation,invocation,reloca tion,andterminationofservices.Theunderlyingmiddlewaremonitors

AgileComputing61 serviceresourceutilization,invocationpatterns,andnetworkandnode resourceavailabilityandusesthatinformationtodetermineoptimalini tialplacementandsubsequentmigrationofservices. AgServeoperatesinconjunctionwithakernelcomponentthatexe cutesoneachnodethatissharingresources.Thekernelcomponent, calledtheAgileComputingKernel,providescontainerstohostservices. AgServesupportsdynamicdeploymentofservicesbyexploiting mobilecode.Clientsmaydefinenewservicesdynamically,thereby injectingnewcapabilitiesintothenetwork.Servicesarepackagedasself containedarchivesthatincludeallthenecessaryJavaclassfilesand otherrelatedJARfiles.Archivesaredynamicallydistributed(pushed)to othernodesusingtheACIKernelfunctionsforremoteserviceinstalla tion.Policiesmaybeusedtocontrolandmanageserviceimplementa tionsasrequiredforsecuritypurposes. AgServealsoallowsactivatingservicesonremotenodes.Service activationisnormallytriggeredbyapplicationdrivenserviceinvocation requests.AgServetakesadvantageoftheACIKernelfunctionstoinstanti atethespecifiedserviceonthetargetnode.Onceactivated,theservice isassignedauniqueidentifierthatisreturnedtotheclientstub,whichis subsequentlyusedforserviceinvocation. Inaddition,AgServeleveragestheACIKernelservicecontainerfunc tionstoprovidetransparentservicemigration.Aservicerunninginside theservicecontainercanbeasynchronouslystopped,itsexecutionstate captured,movedtoanewservicecontainer(usuallyonadifferentnode), andthenrestarted.Thismigrationprocessistransparenttoboththeser viceitselfandtheclientutilizingtheservice.AgServetakesadvantageof thetwoJavacompatibleACIKernelservicecontainers,respectively basedontheAromaVMandtheMobileJikesRVM,whichsupporttrans parentservicemigrationforservicesimplementedinJava. ServicemigrationdifferentiatesAgServefromotherSoAs,asitallows toreacttoenvironmentalchangesandisthereforecrucialtorealizingthe goalofagility.Forexample,asnodeswithfreeresourcesbecomeavail able,serviceinstancesmightbemigratedfromheavilyloadedserver nodestoothernodes,therebyimprovingtheoverallperformanceofthe system(selfoptimizingbehaviour).Servicemigrationalsoallowsthesys temtoreacttoaccidentalevents,suchasapowerlossoranincoming attack(survivabilitybehaviour). AgServeusestheGroupManagerfunctionstosupportdynamicser vicedefinition,registration,lookup,andinvocation.Thedynamicactiva tionandmigrationofserviceinstancesraisestheneedtoaugment traditionalservicedescriptionswithmetainformationsuchaslocationof

62SURIETAL. serviceimplementationcode,resourceutilizationprofile,andthecom municationsprofile.Currently,theResourceTrackerinsideeachACIKer nelmaintainsadatabaseofservicesandtheirresourceutilization information.Whilethisaspecthasnotyetbeenrealized,theresource utilizationprofileandothermetainformationcouldbeembeddedinto theWSDL(Christensenet.al.2001)descriptionfortheserviceandpack agedaspartoftheservicearchive.Inaddition,serviceregistration, lookup,andinvocationoperationshavebeenmodifiedinordertosup portdynamicchangesinservicelocations. FurtherinformationaboutAgServeisavailablein(Suriet.al.2008). FlexFeed FlexFeedisapublishsubscribesystemforefficientstreamingofdata fromproducerstoconsumers.FlexFeedprovidesthreemaincapabilities: 1. HierarchicalDataDistributionsupportsonedatasourceeffi cientlysendingdatatomultipledatasinks. 2. InStreamDataProcessingexploitsintermediatenodesinthe networktotransformthedataasitflowsthroughthenetwork. 3. PolicyEnforcementprovidestheabilitytodynamicallytrans formthedatabeingdeliveredtosinksbasedonpoliciesinplace. FlexFeedrealizesthenotionofagilecomputingbydynamicallysens ingthenetworkandtheresourcesavailableatnodesandallocatingpro cessingelementsatnodesthathavethenecessarycapacity.These processingelementsmaygeneratenewdataforsinksforoneofthree possiblereasons: 1. Generatingdataforasinknodebasedonincomingdata,ifthe incomingdatadoesnotmatchtherequesteddataandthe requesteddatacanbederivedfromtheincomingdata. 2. Modifyingthedatabeingsenttoasinknodeinordertosatisfya policyinforce,whichmaylimitthenatureofthedatathatthe sinknodeisallowedtoobtain. 3. Modifyingthedatatodynamicallyadapttoaconstraintonthe networklink,suchaslimitedbandwidth. Ingeneral,FlexFeedcanbeviewedastakingadataflowgraph,as showninFigure3.2,anddynamicallymappingittotheunderlyingnet worktopology,basedontheintermediatenodesavailable,theresources availableattheintermediatenodes,andonthecharacteristicsofthe networklinksbetweentheintermediatenodes.

AgileComputing63
Sensor B Sensor A Processing Element 1 Database Processing Element 2 User X Processing Element 3 User Z

User Y

Figure3.2 FlexFeedDataFlowGraph Theshadedboxesrepresentfixednodesinthenetworkthatarepro ducersandconsumersofdata.Theclearboxes,ontheotherhand,are processingelements(realizedasservices)thatdynamicallytransformthe dataasitflowsthroughthem.WithFlexFeed,theseprocessingelements aredynamicallyallocatedtoavailableintermediatenodes.Intheexam pleabove,ProcessingElement1isperformingsensordataprocessingby comparingrawincomingdataagainstanotherdatasourceaDatabase. ProcessingElement2isperformingsensorfusionbycombiningtheout putofProcessingElement1withthedatafromSensorB.Thisdataisfed toUserX.However,UserYandUserZrequirevariationsofthedatabeing generatedbyProcessingElement2,whichishandledbyProcessingEle ment3. OneinterestingfeatureofFlexFeedistheabilitytodynamicallymod ifythedataflowgraphtoadapttotheunderlyingnetworkconditions. Forexample,supposethelinktoUserXbecomescongested.FlexFeed mayintroduceanewProcessingElement4thatwilldownsamplethe databeingfedtoUserXinordertoreducenetworktraffic. FlexFeedisfurtherdiscussedin(Carvalho&Breedy2002)(Carvalho et.al.2005). DisService DisServiceisapeertopeerinformationdisseminationsystem.Dis ServiceisintegratedintotheAgileComputingMiddleware,muchlike MocketsandAgServe,andprovidesathirdentrypointintothemiddle wareforapplications.DisServiceimplementsstoreandforwarddelivery ofdataandcachesdatathroughoutthenetwork,therebyenablingdis ruptiontolerantcommunicationsandimprovingdataavailability.In keepingwiththephilosophyofagilecomputing,DisServiceopportunisti callydiscoversandexploitsexcesscommunicationsandstoragecapacity

64SURIETAL. inthenetworktoimprovetheperformanceofinformationdissemina tion. DisServicealsosupportsthenotionofhierarchicalgroupstoorganize theinformationbeingdisseminatedandtobeefficientaboutdeliveryof information.Subscriptionsallowclientstoexpressinterestinparticular groups.Informationispublishedinthecontextofagroup,andmayalso betaggedtodifferentiatebetweenmultipletypesofdata(forexample, blueforcetracking,sensordata,logistics,orotherruntimeinformation). EachnodeinthenetworkrunningDisServiceoperatesinadistrib uted,peertopeermannerwhileprocessingandcommunicatingthe publishedinformationandrequestedsubscriptionsfromneighbouring nodes.Informationisdisseminatedusinganefficientcombinationof pushandpull,dependingonthenumberofsubscribers,thecapacityof thenetwork,andthestabilityofnodesinthenetwork. Oneofthegoalsofthesystemistodynamicallyadapttodissemina tionpatternsthatarecombinationsof{one|few|many}nodesto{one |few|many}nodes.Observationsofinformationneedsintacticalenvi ronmentsidentifiedthreeprimarymodesofdissemination.Situation Awareness(SA)data,suchasblueforcetrackinginformation,isoneto manysinceitneedstobedisseminatedtomost,ifnotall,nodesinthe network.Directeddata,ontheotherhand,isbynatureofinterestonly toasmallsubsetofnodes.Hence,itisonetooneoronetofew.Exam plesofthistypeofdataincludesensordatabeingtransferredtoanode forprocessingorfusion.Finally,thelastmodeisforondemanddata, whichincludeslargeobjectssuchasmaps,pictures,videos,andother multimediaobjects.Giventhelimitedbandwidthofthenetwork,this typeofdatashouldnotbedelivereduntilexplicitlyrequestedbysome nodeinthenetwork. FurtherinformationaboutDisServiceisavailablein(Suriet.al. 2008b). METRICSANDEXPERIMENTATION Anovelapproachsuchasagilecomputingalsodemandsnovelmet ricstomeasurethesuccessofanyparticularimplementation.Athree partmetricisproposed: StartTime.Thisfirstmetricmeasurestheoverheadwithrespectto discovery,allocation,andsetupofanewlydiscoveredresource.Itisthe lengthoftimebetweenanodebecomingavailableandthenodes resourcesstartingtobeutilized.

AgileComputing65 MinimumPresenceTime.Thissecondmetricmeasurestheminimum lengthoftimethataresource(thatistobeopportunisticallyexploited) mustbeavailabletobreakevenintermsofperformance.Itmeasuresthe overheadcausedbyadaptationaspect(e.g.,reallocatingtaskstousethe resourcesonthenewnode)ofthemiddleware. ReleaseTime.Thisthirdandfinalmetricmeasuresthelengthoftime requiredtoletaresourcereturntoitsstatepriortoexploitation.Itis importantintermsofmeasuringtheimpactofthemiddlewareonthe resourcesbeingexploited. Thesethreemeasures,takentogether,provideametrictoevaluate andcomparetheagilityofanimplementation:

wheretheorderedtriple(P1,P2,P3)representstheagilityofthesystem. Itisalsopossibletocomputeasinglevalueusingaweightedaverageas follows:

whereW1,W2,andW3indicatetherelativeweightsofthethreeagility measures. MocketsPerformance TodemonstratetheeffectivenessofMocketsinmitigatingtheimpact ofabruptnetworkchangesonendtoendlatency,wepresentanexperi mentalevaluationofthelibrary.Morespecifically,theexperimentreal izesaperformancecomparisonofMocketswithSCPSTP(CCSDS Secretariat1999)andSCTP(Stewart2007)inthecontextofaBlueForce Tracking(BFT)application. TheBFTapplicationtransmits1KBupdatesforeachnodeatarateof 1Hz.Witha7nodeconfiguration,clientsreceiveupdatesatanaverage rateofoneevery142ms,foratotalthroughputof7KBps.Toreduce latency,theMocketsbasedversionwasconfiguredtoexploitmessage replacement.Morespecifically,newpositionmessagesreplacedprevi ousmessagesinthetransmissionqueue.ThisBFTapplicationoperated onanemulatednetworkthatprovideda30Kbpslink.Periodically(every 45seconds),thequalityofthelinkdroppedtoa5Kbpslinkfor15sec

66SURIETAL. onds.Theexperimentmeasuredthemessagedeliverylatency.FigureX.3 showsthelatencyforthefirst1,300messagestransmitted.

Figure3.3 LatencyComparisonBetweenMockets,SCTP,andSCPSTP Whenthebandwidthdrops,messagesaccumulateinthetransport queue.WithMockets,messagereplacementremovesoldenqueued messageswhenanewmessagebecomesavailable.Therefore,Mockets transmitsfewerandmorerecentmessages.Notethatthislargediffer enceinperformancedoesnotmeanthatMocketsprovideshigher throughputthanSCTPorSCPSTP.Theimprovementismadepossibleby Mocketshavingadditionalinformationfromtheapplicationonhowto handlethedata,delineatingthetrafficintoindividualmessages,andhav ingsufficientinformationtodeterminewhenanewmessageobsoletes previouslyenqueuedmessages. GroupManagerPerformance TheGroupManagercomponentsupportsresourceandservicedis coverywithintheAgileComputingMiddleware.Therefore,theevalua tionoftheGroupManagerwasdonebycomparingittoanotherpopular servicediscoverymechanismJXTAfromSunMicrosystems(SunMicro systems2007)(Traversatet.al.2003).JXTAisamiddlewareforpeerto peercomputingandprovidesservicediscoveryinapeertopeerenvi ronmentliketheGroupManager.JXTAwaschosenbecausethediscovery

AgileComputing67 mechanismfromJXTAisutilizedbySOSCOETheSystemofSystems CommonOperatingEnvironmentamiddlewaretoolkitfortheU.S. ArmyFutureCombatSystemsandotherprogrammes.Thecomparisonis appropriatesincetheAgileComputingMiddlewaretargetsasimilartac ticalmilitaryenvironmenttoSOSCOE. TheexperimentevaluatedtheperformanceoftheGroupManager andJXTAinanemulatedMANETnetworkwiththenumberofnodes beingthree,four,five,six,orten.Ineachcase,boththeGroupManager andJXTAsuccessfullydiscoveredtheservice.Infact,therewasnomea surabledifferenceintheabilityorthetimerequiredtodiscovertheser vice.However,measuringthebandwidthutilizedshowedthattheGroup ManagerperformedsignificantlybetterthanJXTA.Hence,theresults belowfocusonbandwidthmeasurementsofthesetwoimplementa tions. Figure3.4showsthescenariofortheGroupManagerandJXTA experiment.ThescenarioconsistsoftwoormoreUAVsandoneormore GroundVehicles.Inthethree,four,five,andsixnodetests,therewere twoUAVsandonetofourGroundVehiclesrespectively.

UAV 1

UAV n

Ground Vehicle 1

Ground Vehicle 2

Ground Vehicle n

Figure3.4 ScenarioforGroupManagerandJXTAPerformanceComparisonExperi ment Inthetennodetest,therewerefourUAVsandsixGroundVehicles.The connectivitybetweentheUAVsandthegroundvehicleschangesasthe

68SURIETAL. UAVsorbitandmoveinrelationtothegroundvehicles.Tosimplifythe problem,theUAVsareassumedtoflyacircularpathataconstantalti tudeinsuchamannerthattheycompleteoneloopevery60seconds. TheirmovementisalsosynchronizedsothatwhenoneUAViscloseto thegroundvehiclesandhasconnectivity,theotherUAV(s)arefarther awayanddonothaveanyconnectivity. TheresultsshowthattheGroupManagerperformsconsistentlybet terthanJXTAinalloftheseconfigurations.Moreover,therateofgrowth ofthebandwidthasafunctionofthenumberofnodesissmallerwith theGroupManagerthanwithJXTA.TheseresultsarecapturedinTable 3.1andthegraphinFigure3.5. Table3.1OverallComparisonofBandwidthUtilizationbetweenGroup ManagerandJXTA

Numberof Nodes 3 4 5 6 10

GroupManager JXTA BandwidthUtilization(Bytes/sec) 310 1441 452 1612 526 2531 751 3565 1817 6213

Figure3.5 BandwidthUtilizationTrendComparisonbetweenGroupManagerand JXTA

AgileComputing69 AgilityMeasurements Thefinalexperimentmeasuredtheagilityofthemiddlewareusing themetricsdescribedearlier.Thebenefitsofopportunisticresourcedis coveryandexploitationweremeasuredusingtheAgServelibrary.CPU andmemoryresourcesofanopportunisticnodewereexploitedby meansofexecutingaserviceonanopportunisticallydiscoverednode. Figure3.6showsthescenariofortheexperiment.Thereareanum berofclientsthatareinvokingacomputeintensiveprimenumberfac torizationservice.Theseservicesareexecutingonanumberof continuouslyavailableservers.Theonlyrequirementisthatthenumber ofclientsexceedsthenumberofservers,whichprovidesthemotivation forthesystemtoexploitotherresources.Periodically,opportunistic nodesbecomeavailable,atwhichpointserversthatareexecutingmore thanoneservicemigrateaserviceontotheopportunisticnode.Ifthe opportunisticnodeweretonolongerbeavailable,thentheserviceis migratedbacktotheoriginalserver(oranotherserver).TheRelease Timemetricismeasuredbytheimpactonauserprocessononeofthe opportunisticnodes.

Figure3.6 ScenarioForMeasuringAgility

70SURIETAL. Inthecontextofopportunisticresourceexploitationforserviceinvo cation,theresultsaresummarizedinTable3.2.Whiletheseresultshave beenobtainedinonlyonescenarioandinthecontextofserviceinvoca tion,theyarestillpromising.Thecurrentimplementationofthemiddle warestillneedstobeoptimized,whichshouldresultinevenbetter performanceoverall.Theorderofmagnitudeestimatesareaguideto selectingenvironmentsandscenarioswhereopportunisticexploitation ofresourcesispossible. Table3.2SummaryofAgilityMeasurements
Measure Start Time Minimum Presence Time Release Time Result 775 milliseconds 8 seconds 1981 milliseconds Order of Magnitude < 1 second < 10 seconds < 2 seconds

IMPLICATIONSFORFUTUREARMY CurrentandfutureArmymissionsincreasinglydemandmoreflexibil ityandadaptabilitytorapidlychangingsituations.Urbanoperationsand asymmetricwarfarerequirethattheArmybemoredecentralizedand agile.C4systemsthatsupporttheArmymustdynamicallyadaptandbest supporttheinformationandcommunicationsrequirementsofthewarf ighter.Agilecomputing,asdescribedinthischapter,laysthefoundation forsuchaC4infrastructure.DesigningC4systemstobeopportunisticin discoveringandexploitingresourcesisanovelideathathasthepotential forsignificantlyimprovingtheperformanceoffuturesystemsandfully realizingthegoalsofNetCentricOperations. CONCLUSIONS Thisresearchbeganwithasimpleideathatitshouldbepossibleto extractbetterperformanceoutofmiddlewareoperatinginahighly dynamicenvironmentifthemiddlewarecouldreactquicklyandopportu nisticallyexploitresources,giventheobservationthatnodesoftenhave someexcesscapacitysomewiggleroom.Intheprocessofanswering thisquestion,thenotionofagilecomputinghasbeendefinedandacom prehensivemiddlewaresolutiondesignedandimplemented.Themid dlewareincludescomponentsforcommunications(Mockets),resource andservicediscovery(GroupManager),peertopeerinformationdis semination(DisService),andserviceorientedarchitectures(AgServe).

AgileComputing71 Experimentalanalysishasshownthatthesecomponentsperformbetter thanexistingstandardsorpopularsolutions. However,thenovelcontributionofthisresearchisincombiningthe abovecomponentsintoanintegratedmiddlewareandleveragingtheir capabilitiestobeopportunistic.Experimentalanalysishasalsoanswered thequestionofextractingimprovedperformancefromthemiddleware bybeingopportunistic.Whilethisanalysiswasdoneinalimitedcontext, theresultsarepromising.Thedegreeofagilityhasbeendeterminedto beontheorderofafewsecondsforthecurrentmiddlewareimplemen tation,andthiscancertainlybeimprovedwithfurtheroptimization. REFERENCES
Suri,N.,Tortonesi,M.,Arguedas,M.,Breedy,M.,Carvalho,M.,Winkler,R.2005. Mockets:AComprehensiveApplicationLevelCommunicationsLibrary.InProceedingsof the2005IEEEMilitaryCommunicationsConference(MILCOM2005).October2005, AtlanticCity,NewJersey. Tortonesi,M.,Stefanelli,C.,Suri,N.,Arguedas,M.,andBreedy,M.2006.Mockets:A NovelMessageorientedCommunicationMiddlewarefortheWirelessInternet,in ProceedingsofInternationalConferenceonWirelessInformationNetworksandSystems (WINSYS2006).Setbal,Portugal,August2006. Stefanelli,C.,Tortonesi,M.,Benvegnu,E.,andSuri,N.2008.SessionMobilityinthe MocketsCommunicationMiddleware.InProceedingsoftheIEEESymposiumon ComputersandCommunications(ISCC2008).July2008,Marrakech,Morocco. Clip2.2001.TheGnutellaProtocolSpecificationVersion0.4.http://rfc gnutella.sourceforge.net/developer/stable/index.html. Suri,N,Marcon,M.,Quitadamo,R.,Rebeschini,M.,Arguedas,M.,Stabellini,S., Tortonesi,M.,Stefanelli,C.2008.AnAdaptiveandEfficientPeertoPeerService orientedArchitectureforMANETEnvironmentswithAgileComputing.InProceedingsof theSecondIEEEWorkshoponAutonomicComputingandNetworkManagement (ACNM08). Suri,N.,Rebeschini,M.,Breedy,M.,Carvalho,M.,andArguedas,M.2006.Resource andServiceDiscoveryinWirelessAdHocNetworkswithAgileComputing.InProceedings ofthe2006IEEEMilitaryCommunicationsConference(MILCOM2006),October2006, Washington,D.C. Estefan,J.A.,Laskey,K.,McCabe,F.G.,andThornton,D.2008.OASISReference ArchitectureforServiceOrientedArchitectureVersion1.0.Onlinereference:http:// docs.oasisopen.org/soarm/soara/v1.0/soarapr01.pdf. Christensen,E.,Curbera,F.,Meredith,G.,Weerawarana,S.2001.WebServices DescriptionLanguage(WSDL)1.1.Onlinereference:http://www.w3.org/TR/wsdl. Carvalho,M.,andBreedyM.2002.SupportingFlexibleDataFeedsinDynamic SensorGridsthroughMobileAgents.InProceedingsofthe6thInternationalConference onMobileAgents(MA2002).Suri,N.(Ed.).LectureNotesinComputerScience.Vol. 2535,pp.171185. Carvalho,M.,Suri,N.,Arguedas,M.2005.MobileAgentbasedCommunications MiddlewareforDataStreamingintheBattlefield.InProceedingsofthe2005IEEE

72SURIETAL.
MilitaryCommunicationsConference(MILCOM2005).October2005,AtlanticCity,New Jersey. Suri,N.,Benincasa,G.,Formaggi,S.,Winkler,R.,Choy,S.,Kovach,J.,andTokarcik,L. 2008.DisService:APeertoPeerInformationDisseminationServiceforTactical Environments.InProceedingsofthe2008MeetingoftheMilitarySensingSymposia (MSS)SpecialityGrouponBattlespaceAccousticandSeismicSensing(BAMS2008). CCSDSSecretariat.1999.SpaceCommunicationsProtocolSpecification(SCPS) TransportProtocol(SCPSTP)BlueBook.NASA,Washington,May1999. Stewart,R.(Ed.)2007.StreamControlTransmissionProtocol.RFC4960,September 2007. SunMicrosystems,Inc.2007.JXTAv2.0ProtocolSpecification.Onlinereference: https://jxtaspec.dev.java.net/JXTAProtocols.pdf. Traversat,B.,AhkilA.,MohamedA.,MikeD.,CarlH.,Hugly,J.,Pouyoul,E.,and Yeager,B.2003.ProjectJXTA2.0SuperPeerVirtualNetwork.SunMicrosystems,Inc. TechnicalReport.

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T A N D L A U R E L A L L E N D E R

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T AND LAUREL ALLENDER

This book, produced by the Partners of the Army Research Laboratory Advanced Decision Architectures Collaborative Technology Alliance, also serves as the Final Report for DAAD19-01-2-0009.

SECTIONI.COLLECTING,PROCESSING,ANDDISTRIBUTINGBATTLEFIELD INFORMATION SECTIONI COLLECTING,PROCESSING,ANDDISTRIBUTINGBATTLEFIELD INFORMATION

14SECTIONI

Chapter

4. REGULATINGTHEEXCHANGEOFTACTICALMILITARYINFORMATION USINGTHEKAOSPOLICYSERVICESFRAMEWORK

LarryBunch,JeffreyM.Bradshaw,Ph.D., MattJohnson,JamesLott, PaulJ.Feltovich,Ph.D.,NiranjanSuri,Ph.D., MarcoCarvalho,Ph.D. InstituteforHumanandMachineCognition,Pensacola,FL LarryTokarcik,RobertWinkler,SomiyaMetu ArmyResearchLaboratory,Adelphi,MD

INTRODUCTION Sharingtacticalinformationamongjointforces,coalitionpartners, andnongovernmentorganizations(NGOs)canbecriticaltosuccessful andsafeoperations.Unfortunately,theneedtoshareinformationis sometimesatoddswiththesensitivityofbattlefieldinformation (McNaugher,1989;Feltovich,etal.,2009).Frequently,theflowofdata acrossheterogeneouschannelsandamongdiversegroupsmustbe tightlyregulated.Thispresentssignificantoperationalchallenges,includ ingsuccessfulnegotiationofthemultilevel,interdependent,andsome timesconflictingagencyandinteragencyaims,governedformallyby policyandchainsofcommand,andinformallybyculturalandorganiza tionalnorms. IHMCandARLarecollaboratingtodevelopsoftwaresystemsin whichinformationispassedseamlesslyacrossmultipleorganizations, networks,andindividualswhilerespectingcomplexandpossiblyconflict ingsetsofpolicies.Insupportofthisobjective,wehavecreatedcapabili tiesthatallowustoautomaticallyanddynamicallyidentify,inacontext sensitiveway,whenrequisiteoperationalandsituationalimperatives warranttheautomated(orsemiautomated)declassificationanddissem 73

74BUNCHETAL. inationofsensitivebutperishablecombatdata.Moreover,weare exploringmechanismsforabstractingandtransforminginformationto enableittofulfillrequirementsforsharing,forexample,byredactingthe contentormetadataofmessagesorbydelayingtheirdeliveryuntilacrit icalsituationisover.Byprovidingautomatedmechanismstohelpiden tifyrelevantsensitivebutperishabledataintheCollateralSpacethatcan bereleased,transformed,ordelayedwithappropriatelevelsofhuman oversightandapproval,livescanbesavedandtimewastedinperforming thesetedioustasksmanuallycanbesignificantlyreducedoreliminated. Ourapproachincludestheuseofanextensiblestandardsbasedpol icyrepresentationtospecifyformalpolicystatementsaboutthetactical domain.TheKAoSPolicyandDomainServicesframeworkprovidesa graphicalinterfaceforquicklyconstructingpoliciesinreadableEnglishin apointandclickfashionatruntime.KAoSalsoprovidesinnovativefor malontologybasedpolicyanalysisanddeconflictionmechanismsand highlyefficientmechanismsforpolicymonitoringandenforcement (Uszok,etal.,2008).Italsoincludesmanypowerfulfeaturesthat enhanceitsusefulnessinrealbattlefieldsituations(e.g.,supportfor standaloneordisconnectedoperations,spatialandtemporalreasoning). Weintegratethispolicyreasoningcapabilitywithdistributedmiddleware componentscapableofenforcingsuchpoliciesinaMANET(mobilead hocnetwork)environment(Suri,etal.,2003;Carvalho,etal.,2007). BACKGROUND Informationexchangeinatacticalenvironmentisimportanttothe netcentricvisionoftheDepartmentofDefense(DoDReport,2001). Warfightersmustshareinformationacrossorganizational,national,and spatialboundariestoachieveoperationalgoals.Criticalinformation comesfromsuchdisparatesourcesasmannedandunmannedsensors, humanintelligencenetworks,theInternet,andGlobalPositioningSys tems.Thetacticalenvironmentdemandsasolutionthatkeepsinforma tionprotected,isnearrealtime,canbewidelydistributed,and accommodatesmultiplerichinformationsources.Suchmechanisms wouldallowthemovementofcertaindataincertainsituations(e.g., sensitivebutperishable)acrosstheboundariesbetweenvarious domains.Ideally,thesemechanismsshouldbeautomatedsuchthatmes sages,objects,orfilescanflowsecurelyacrosstheboundarywithmini malhumanintervention.However,itisimportantthatsuchsystems supporttheappropriatedegreeofhumanoversightandapprovalofthe informationexchangeprocess.

RegulatingTacticalInformationExchange75 Aprimeexampleoftheneedtoexchangetacticalinformationisthe sharingofblueforcetrackingdata.Thisinformationconcernstheloca tion,nature,andmovementoffriendlyforcesandispreciselythetypeof perishabletacticaldatathatneedstobesharedamongmultiplecooper atingorganizationssuchasUSforces,UKforces,NGOs,andlocalpolice. Knowingthelocationoffriendlyforcesandnoncombatantsiscriticalto avoidingpotentialfratricidesituationsandminimizingciviliancausalities. Whilealloftheparticipatingagenciesarethereforemotivatedtoshare suchtrackinginformation,eachorganizationhasregulationsgoverning thekindsofdatathatmaybesharedwithmembersoftheother domains.Theinformationsharingpoliciesrequiredcanbecomplexand requirearichlanguagethatcandrawuponanyaspectofthedataaswell asthecontextinwhichthedataisbeingshared.Wehavebeenespecially concernedwithsituationswhereinflexible,contextinsensitiveapplica tionofpoliciesmayendangerlives.Forexample,whenaspecialopera tionsgroupunexpectedlymovesintocloseproximityofanothergroup, theirnormallyundisclosedlocationmayneedtobetemporarilyrevealed toreducetheriskoffriendlyfire. CurrentInformationExchangeSolutions Oneapproachtoprovidinginformationexchangeacrossdomainsis foreachdomaintohaveanindependentnetworkthatcannotbe accessedbymembersoftheotherdomains.Thedomainsthenestablish aminimalnumberofconnectionpointsbetweenthenetworks(US,UK, Police)andshareallinformationacrossdomainsthroughthesepoints wherethedatacanbeinspectedandfiltered.ThisisDoDscurrent approachwhichimplementscrossdomaininformationexchangeviaa securityguardorgatekeeper,calledaCrossDomainSolution(CDS) (Crocker,2007).ACDSconfirmsthatinformationhasbeencorrectly downgradedwhentravelingfromahigherclassificationdomaintoa lowerone.ACDSappliespreestablishedrulesetstoperformthisfunc tion,usesstandardInternetprotocols,andisdesignedforspecificdata formats.ExamplesofsecurityguardsaretheRadiantMercurysystem developedfortheNavyanddeployedaroundtheworldat150sites,Ora cleVaultselectedbyINSCOMaspartofitsfilingsolution,andtheDia mondMatrixCDSdevelopedbytheUSAirForce.CDSsolutionsare capableofsecurelycontrollinginformationexchange,butatthecostof prohibitingpeertopeerdataexchange.Thisgatewayapproachcanlead tocircumstanceswhereSoldiersfromdifferentdomainsarecollocated, capableofconnectingandsharinginformationpeertopeer,andautho

76BUNCHETAL. rizedtosharetheinformation,butforcedtosendtheinformation throughthegatewayjusttoensuretheinformationsharingispermitted. Thiscanquicklybecomeunacceptableintacticalenvironmentswherea connectiontothegatewayisnotalwaysavailableandtimelinessis imperative. ResearchDirection Newtechnologyandcertificationadvancesarecreatingopportuni tiesforsignificantimprovementintacticalcrossdomainsolutions.Certi ficationandaccreditationsystemsbasedontheoldparadigmofneedto knowledtothetechnologysolutionofgatekeepers.Newcertification andaccreditationprocessesarebeingadvancedtoreflectthenetwork centric,Webtechnologybased,newparadigmofneedtoshare.This shiftistowardprovidingmoreinformationsharingcapabilitiestothe edgealongwiththeresponsibilityforsecuringtheinformationandcom plyingwiththepoliciesandregulationsthatgoverninformation exchange(Alberts,2003).Thisprojectdrawsuponrecentadvancesin AgileComputing(Surietal.,2003,2008)andcrosslayersubstratework (Carvalho,etal.,2007)torealizethesecapabilitiesintacticalMANET environments. CurrentCDSstypicallyuseaccesscontroltechniquesthatarebased onstatic,aprioriassignmentofroles,whichislimitinginthedynamic battlefield.Currentinformationsharingregulationsoperatingprocedures dependonmuchmorecontextualinformationthantheuserscreden tials,networkaddresses,andpayload.Forexample,someregulations refertotheperishabilityoftheinformation(FieldManual1008,1997) whichmayrequiretemporalreasoningandothersdependuponthe proximityofunitsorotherspatialrelationships.Partofourobjectiveisto showhowadvancesinsemantictechnologiesforreasoningandrepre sentationcancomplementandenhancecurrentsolutionsforinforma tionsharingincomplextacticalenvironmentswherecontextsensitive policyenabledinformationsharingisavitalchallenge. INNOVATIONS Theprimaryinnovationofthisprojectisapolicygovernedinforma tionexchangesoftwarecapabilityforWarfightersoperatinginatactical environment.Thissoftwareenablespoliciestobeeasilydevelopedthat willcontroltheabilitiesforanynodeinanetworktoshareauthorized informationwithitsneighbors.Thissecuredynamicdecentralizationof

RegulatingTacticalInformationExchange77 policyspecification,deconfliction,analysis,andenforcementisacritical steptowardmakingtacticalcrossdomaininformationpractical,cost effective,andflexibleenoughtomeettheneedsofnewsituationsas theyarise. Asecondinnovationisastandardsbasedformalandextensiblepol icyrepresentationfordescribinginformationsharingconditionsandcon straints.Thisrepresentationenablesustoprovidearichsetofconstructs forpolicyadministratorstodefinethecontextsinwhichinformation sharingispermitted.Theseconceptsincludeattributesoftheactors involved(e.g.,domain,echelon,mission),theinformationbeingshared (e.g.,type,location,perishability,specificcontent),andstates(e.g., threatlevel).Thesesemanticallyrichpoliciescanpermitorforbidinfor mationsharingbasedonanycombinationofthesecontextattributes (authorizationpolicies). Thethirdinnovationisanewformofinformationexchangepolicy thatcanrequire(orwaivearequirementfor)certainactionstobetaken bythepolicyenforcementmechanism(obligationpolicies).Forexample, ratherthanjustprohibitingcertaininformationfrombeingexchanged, theseobligationpoliciescanrequirethemodificationoftheinformation ortheprocessofexchangeinawaythatmakesitpermissible.This includes,forexample,automateddowngrading(e.g.,redactionofcon tent,metadata;reductionofresolutionofimages)ofinformationforcer tainexchanges.Inaddition,obligationpoliciescanspecifytheconditions underwhichhumansmustgivespecificapprovalforinformationrelease. Theycanalsorequirecertainexchangestobeloggedandappropriate personneltobenotifiedinthecaseofpolicyviolations. POLICYREPRESENTATION Onechallengeistospecifyinformationsharingpoliciesinawaythat isstraightforwardforuserstounderstandyetsufficientlyexpressiveto capturethecomplexityofregulatinginformationexchangeinthetactical domain. TheKAoSpolicyservicesframework(Uszok,etal.,2008)employsthe standardW3COWL(WebOntologyLanguage)(McGuinness,etal.,2004) torepresentpolicies.OWLisbuiltontopofXML(andRDF),andsoitcan takeadvantageofthemanythirdpartytoolsandapplicationsthathave appearedinthelastseveralyears.WehavefoundOWLtobeaveryflexi bleandefficientapproachtopolicyreasoningandrepresentation,and haveovercomeanumberoftechnicalchallenges(e.g.,nonmonotonic reasoning,increasedexpressiveness,easeofuse,dynamicchanges,effi

78BUNCHETAL. ciency,scalability)thathavelimitedtheuseofOWLinpreviousapplica tionsbyresearchers(Bradshaw,2008).KAoSdefinesthestructureofa policyasamappingfromanActionClassDescription(ACD)describing thecontextinwhichthepolicyappliestoanauthorizationorobligation. Thecontextincludesanextensiblesetofpropertiesdescribingthe Action,Actors,andStateinwhichthepolicyapplies.

KAoSPolicy:ACD(Action,Actors,State){Authorization,Obligation}

TheACDisdefinedusingrestrictionsintheOWLlanguagetospecify therangeofvaluesforeachpropertyoftheAction,Actors,andStatethat mustbesatisfiedtotriggerthepolicy.Tocreateavocabularyfortactical informationsharingpolicies,domainspecificconceptswereaddedto theKAoScoreontology.Fromtheseconcepts,sharingspecificpolicies canbedefined:

SharingPolicy:ACD(SharingAction,Groups/Roles,Content,State) {Authorization,{Transform,Delay,Notify,Approve}}

Domain Concepts and Relationships

Units Coalitions Missions MIL-STD 2525B

OWL-DL Policy Representation

Filter (CDIX Action) {Authorize, Prohibit} {Transform, Delay, Notify, Approve}

KAoS Hypertext Policy Representation

US Forces are obligated to Obtain Approval with Approver of type S2 When US Forces perform Send Tracking Data and the Recipient is of type Local Police and the Symbol is not of type Signals Intelligence

Figure4.1 Anontologyofdomainspecificconceptsareassembledintoaformal OWLpolicyrepresentationwhichistheninterpretedforusersbythe mediatinghypertextpolicyrepresentation. PolicyanalysisanddeconflictionalgorithmsinKAoScanrunveryeffi cientlyusingOWL.Policydecisionandenforcementalgorithmsareeven moreefficient,duetospecialOWLcompilationandcachingmecha

RegulatingTacticalInformationExchange79 nismsusedintheGuard,meaningthatmostpolicydecisionscanbe madeinatimeframethatapproximatestablelookup.However,theOWL representationisnotoptimalforhumanunderstanding.Forthispurpose, wedevelopedtheKAoSPolicyAdministrationTool(KPAT)thatallowspol iciestobespecifiedinarestrictedformthatisclosertonaturallanguage. Whentheusercreatesapolicyusingthegenericpolicyeditorin KAoS,hefirstseesabasicpolicystatementwithavariablenumberof conditionslikethefollowing:

Actorisauthorizedtoperformactionwithproperties

Underlinedphraseshavearangeofpossiblevaluesdefinedinthe ontologyofinformationsharingconceptsandcanbereplacedwithany valuefromthatrange.Forexample,thetermauthorizedcouldbe replacedwithnotauthorized,required,ornotrequired.Actor couldlikewisebereplacedbyaspecificindividual,group,orrolename, andsoforth.KPATusesthefamiliarhyperlinkmetaphorenablingoneto selecttheselinkedphrasestobrowsetherangeofpossiblevaluesdrawn directlyformtheontologyinordertoformthedesiredpolicystatement asdepictedinFigure4.1.Agenericformatforaninformationexchange authorizationpolicymightlooksomethinglikethis:
ActororGroupisauthorizedtoSendData whentheRecipienthasvalueActororGroup andContextAttributehasvalueContextValue

Thecontextattributescurrentlyavailableforgoverningtheactionof SendTrackingDataincludethecontextofthesenderandreceiverinclud ingtheirlocationsandthedomainstowhichtheybelongasinthefollow ingexample:

USForcesareauthorizedtoSendTrackingData whentheRecipientisoftypeUKForces andtheRecipientLocationis<5kmfromtheSender

WedevelopedanextensiveOWLdomainmodeltodescribethe informationcontentforblueforcetrackingincludingtheDoDstandard warfightingsymbologyMILSTD2525B(DISA,2007).Thisstandard definesahierarchyofapproximately1500classesofsymbolsranging fromtypesofunits,installations,andequipmenttotacticalgraphicsrep resentingmaneuvers,obstacles,andfiresupport.Thetopofthehierar

80BUNCHETAL. chydefinesusefulhighlevelabstractionstodescribetheinformation suchasallGroundUnitsorObstacles.

USForcesareauthorizedtoSendTrackingData whentheSymbolisoftypeCombatGroundUnit andtheEchelonhasvalueCompanyorAbove

Inaddition,somepoliciesrequireanactiontobeperformedasa conditionofinformationsharing.Suchobligationpoliciescanrequire obtainingaparticularlevelofhumanapprovalaswellasnotificationtoa humananalystoranothersystem.Otherobligationactionsincludetrans forming,abstracting,ordelayingtheinformationtomeettherequire mentsofinformationsharingconstraints:

USForcesareobligatedtoObtainApproval withApproveroftypeS2 WhenUSForcesperformSendTrackingData andtheRecipientisoftypeUKForces andtheSymbolisnotoftypeSignalsIntelligence

Finally,assigningrelativepriorityamongpoliciesenablesdefining abstractdefaultpolicieswithlayersof(typicallymorespecific)excep tions.Forexample,alowprioritypolicymayauthorizesendingalltrack ingdatafromUStoUKunits,butbeoverriddenwithahigherpriority policythatforbidssharingmaneuvertacticalgraphics.Exceptionpolicies ofhigherprioritymaysimilarlybeusedtooverrideorwaiveobligations. KAoSalsoprovidesanAPIanduserinterfaceformanagingpolicies,apol icyrepositoryforstoringandqueryingpolicies,andaservicetodistrib utepolicies. POLICYREASONING Policyreasoningcapabilitiesarerequiredateachnodethat exchangestacticalinformationtodeterminetheapplicablepolicies, assesswhethertheactionisallowed,andprovidealistofobligedactions toperform.WerelyuponofftheshelfOWLDescriptionLogic(OWLDL) reasoningsoftwaresuchastheJavaTheoremProver(Fikes,2003)and Pellet(Sirin,2007)toperformtheclassificationofinstancesintotypes andsubtypesaswellasotherrelationshipssuchasechelonsandforce structures.Thisenablesthepolicysystemtoderivefactsfromthe domainmodelsuchasthataTankisanarmoredgroundvehicleandthat agivenclientisamemberoftheUKForcesdomain.

RegulatingTacticalInformationExchange81 Specializedreasoningcomponentshavebeencreatedtosupportspa tialandtemporalpolicycontexts.Thespatialreasoningcomponentis abletoclassifypoliciesbasedonspatialreferencessuchasGPScoordi nates,areas,anddistances(Uszok,etal.,2008).Thisenablesevaluating policystatementssuchasRecipientLocationis<5kmfromtheSender orthatablueforcetrackingsymboliswithinanareasuchasaunitsarea ofoperations. Additionalpolicyreasoningcapabilitiesincludeenablingclientsto queryfortherangeofpossibleactionsthatwouldbeauthorizedbythe currentpolicysetaswellasrecognizingthepotentialforpolicyconflicts atpolicycreationtime.Thelatteralsoincludestoolstohelpusersresolve suchconflictsthroughpolicyprecedencerelationshipsorfurtherrefine mentanddifferentiationoftheconflictingcontexts. POLICYENFORCEMENT KAoSpolicyenforcementcanbeperformedinseveraldifferentways. Inthecaseoftheworkunderdiscussions,itisperformedbytheapplica tionmiddlewarethatcoordinatesthepeertopeerdisseminationofdata amongclients.WeusetheAgileComputingmiddlewareDissemination Service(Suri,2008)toprovideandregulatedecentralizedinformation distribution.TheDisseminationServicesupportsstoreandforwarddeliv eryofinformationbycachingdatathroughoutthenetwork,thereby makingittolerantofconnectiondisruptions.Subscriptionstoinforma tionareorganizedbyhierarchicalgroups.Wheninformationispublished inthecontextofagroup,itmayalsobetaggedtoidentifythetypeof data(e.g.,blueforcetracking,spotreports).Eachnodeinthenetwork runningthisserviceprocessesandcommunicatessubscriptionrequests andpublisheddatafromneighboringnodesinadistributed,peerto peermanner.Thedisseminationisperformedthroughacombinationof pushandpull,dependingonthenumberofsubscribers,thecapacityof thenetwork,andthestabilityofnodesinthenetwork. Thedisseminationmiddlewarehasbeenextendedtointeractwith KAoSpolicyservicestoenforceinformationsharingregulations.Thepol icydecisionpointimplementation,theKAoSguard,providespolicydeci sioncapabilitiesthatenabledistributedcomponentstogetfastpolicy decisionslocallysoenforcementcanbemaintainedinenvironmentswith unreliableconnectivityasshowninFigure4.2.Inthisway,information sharingpoliciescanbeenforcedbythemiddlewarewithoutthecoopera tionoreventheknowledgeoftheclientapplicationsbeingregulated.

82BUNCHETAL.

Figure4.2 PolicychangesaredistributedthroughanadhocnetworkbytheDissem inationServicetoaKAoSguardthatservesasthepolicydecisionpoint foreachnode. TheAgileComputingmiddlewarecomponentsactivelyinspectthe databeingexchangedandinteractwiththeKAoSpolicyservicestodeter minewhetheraninformationexchangeisauthorizedandwhetherany obligationsmustbefulfilledasdepictedinFigure4.3.Inthisway,the middlewarecanalloworblockanytrafficbasedontheauthorizationpol icies.Inthecaseofobligationpolicies,themiddlewaredynamically instantiatescomponentsdesignedtofulfilleachoftherequiredactions. Thetypesofobligationactionssupportedincludemodifyingthemessage contents,keepingthehumanintheloopthroughnotificationand approval,andaffectingthehandlingofmessagesbythemiddleware suchasintroducingdelay.

RegulatingTacticalInformationExchange83

Ad-hoc Network Node Applications Send Authorization Filter Transformation Filters Encrypt Decrypt Receive authorization to send/receive KAoS Policy Guard required data transformations required interactions

Dissemination Service

Notify/Approve Enforcers Agile Computing Middleware

Mobile Ad-hoc Network (MANET)

Figure4.3 EachnodeintheMANETisequippedwiththeAgileComputingmiddle warethatmanagesinformationaccessthroughencryptionandenforcing authorization,transformation,andapprovalpolicies. Policyenforcerscanautomaticallyredactspecifiedinformationfrom messagesbasedontheirXMLstructureandabstractthedetailsconcern ingunitandothersymboltypestoaspecifiedlevel(e.g.,showingonly thataUnitisaCombatUnitandhidingthedetailsaboutwhetheritis Artillery,Infantry,orArmored).Anenforcerisalsoavailabletotransform informationincludingreplacingapointlocatedwarfightingsymbolwith anareatacticalgraphicsuchasreplacingaunitsidentifierandlocation withaNoFireZoneorreplacingobservationsaboutenemyforcesand targetswithObstaclesandRestrictedAreas. Additionalenforcershavebeendevelopedspecificallytomaintain humanoversightandapproval.Policiesthatobligeapprovaldeploya componentthatidentifiesauserofthespecifiedtypeorroleandinterac tivelypromptstheusertoallowordenytheinteraction. IMPLICATIONSFORFUTUREARMYWORK Severaltechnicaltransitionsofthisworkareunderway.Withtraining andsupportfromIHMC,ourARLcollaboratorsinAdelphi,MDhave adoptedthepolicylanguageandservicesinnovationsfromthisproject foraninhouseresearcheffortconcerningtheautomatedregulationof informationfromunattendedsensornetworks.Theframeworkisbeing

84BUNCHETAL. usedinspecificationofsensorstatesandactivitiesaswellasdomainspe cificstatetransitionruleswhichentailinformationfrommultipleco locatedsensors.TheSensorstransmitdifferenttypesofalerts.Asingle alertorasequenceofalertswithinagiventimeframecancombineto triggerapolicy.Thesepoliciescanbesimpleorcomplexdependingon thenumberofalertsinthesequenceandthetimeintervalbetween them.Theframeworkisusedtospecify,deconflict,andenforcepolicies basedonthesealerts.KAoSisthenusedtorecognizeandtriggerthe highestprioritypolicywhichmatchesanypermutationofanycombina tionofalertsfromtheSensors.Theresultisaframeworkenablingauton omouscoordinationamongsensorstoprovidesmarteralerts. ARLandIHMCarealsoextendingthepolicylanguageandenforce mentcapabilitiestodatagatheringprocessessuchastheharvestingof informationfromunattendedsensors.Webelievethatpolicygover nancecombinedwithservicestoestablishreputationandtrustcan enhancedatagatheringmissioneffectivenessbymanagingthelimited computationandnetworkresourcestohelpensurethemostimportant andreliableinformationisgatheredandshared.Tacticalmilitaryenviron mentsareoftenhighlydynamicandconsistoffixedandmobilenodes suchasunattendedgroundsensors,robots,dismountedsoldiers,ground vehicles,andairbornenodessuchasunmannedaerialvehicles.Eachof thesesourceshasstrengthsandweaknessesandthevalueoftheinfor mationcanvarybasedoncontext.Thesenodesareofteninterconnected withwirelessnetworksthatresultinunreliableandbandwidthcon strainedlinks.Duetothelargevolumeofdataavailablewithtodays technology,itisimportanttoprovideautomatedfilteringofdatainorder tomakebestuseofthelimitednetworkresourcesandalsotoassistthe peopleprocessingthisdata.Therearenumerousfactorsthatneedtobe consideredwhenassessingtheusabilityofagivenpieceofdataincluding thetypeofsensor,itscondition,itshistory,anditsapplicability.Itis importanttomakeuseofthistypeofknowledgenotjustatthedecision point,butalsointhegatheringpoint.Whichpiecesofdataaregivenpri orityinthelimitednetworkcapacityisequivalenttomakingthedecision onwhatdatayouwillconsiderandwhatyouwillignore.Whenproviding suchafilteringmechanism,itisequallyimportanttoallowtheusersto havesomeinputintothatsystem.Theinputshouldcomedirectly,inthe formofaccessingasensorsreliabilityandapplicability,andindirectly thoughthegoverningpolicies.Inthisdatagatheringcontext,policycan beusedtospecifytheprecedenceofthedatabasedontherequestor, objective,and/orthedatatag.Wesupportbothpolicybasedprioritiza tion(setbypolicyadministratorsatahigherlevel),anduserrequested

RegulatingTacticalInformationExchange85 priority(basedontheuser'scurrentneeds).Thetwoaspectswork togethertocoordinatetheflowofdata. ThepolicycapabilityhasalsobeenappliedbyARLresearchersin Aberdeen,MDtoaidArmyintelligenceanalystsinrecognizingandnavi gatingtheinformationsharingregulationsastheyprepareintelligence packages(Mittrick,2008;Bunch,2008).Thisscenariodefinedaclassof DocumentswithpropertiessuchasClassificationLevelandPerishability aswellasalistofDocumentFeaturessuchasTopography,Routes,Enemy Activity,andsoforth.TypesofMilitaryMissionssuchasPeacekeeping andAssaultmissionswerealsodefined.EachtypeofMissionisthen associatedwithalistofDocumentFeaturesthatareappropriateforthe mission.Theontologywasalsopopulatedwithinformationabouteach instanceofadocument,actor,andmission.Fortextdocumentsrepre sentedinHTML,theRDFasyntaxenablesembeddingtheontologyinfor mationwithinthedocument.Forotherdocumenttypes,themetadatais definedexternallyinanOWLfilethatcontainsURLreferencestoeach documentdefinedtherein.Themetadataabouteachdocument,the semanticmissionfeaturedocumentrelationshipsdefinedintheontol ogy,andthepolicieswerealljoinedtogetherthroughaqueryinthe SPARQLsemanticquerylanguageasdepictedinFigure4.4. Finally,wehaveworkedcloselywiththeUSArmyCommunications ElectronicsResearch,DevelopmentandEngineeringCenter(CERDEC)to identifypotentialapplicationsforourinformationsharingpolicyinnova tionsintotworecentprojects:CollaborativeBattlespaceReasoningand Awareness(COBRA)andTacticalHumanIntegrationofNetworked Knowledge(THINK).Bycombininginformationwiththepoliciesand requirementsthatgoverndatasharing,andbyapplyingtheunderlying semanticallyrichrepresentationandreasoningstrategiestomeaning fullydescribeandrelateinformationtomissioncontexts,webelieve thesetechnologiescansignificantlyimprovetheretrievalandsharingof informationvitaltotheWarfighter.

86BUNCHETAL.

Figure4.4 ARLandIHMCarecombiningnoveltechnologiesfromthesemanticweb tohelpanalystsidentifymissionrelatedintelligencedocumentsaswell asrecognizeandcomplywiththeapplicableinformationsharingpolicies. CONCLUSIONS Thisresearchhasdemonstratedtheabilitytoovercomeseveralbar rierstoautomatedinformationsharingacrossdomainsintacticalenvi ronments.AkeycontributionistheextensionofanOWLbasedpolicy representationtoexpressmeaningfulanddetailedregulationsgoverning informationsharingbasedonthecontentaswellasthecontextofthe exchange.TheKAoSPolicyandDomainServicesframeworkprovidesa graphicalinterfacefordefiningpoliciesinanaturallanguageformat, soundontologybasedreasoningmechanismsforpolicyanalysisand deconfliction,andefficientmechanismsforenforcement(Uszok,etal., 2008).Italsoincludesadvancedpolicyservicesforspatialandtemporal reasoning.Finally,wedemonstratedaneffectiveapproachtoautomated policyenforcementinadistributedenvironmentbyincorporatingpolicy checkingandobligationfulfillmentcapabilitiesintotheapplicationmid dleware.Throughtheapplicationofthesescientificadvances,wehave demonstratedtheabilitytoregulateinformationsharingatthetactical edge.

RegulatingTacticalInformationExchange87 REFERENCES
Alberts,D.S.,Hayes,R.E.2003.PowertotheEdge:CommandandControlinthe InformationAge.CommandandControlResearchProgram. Bradshaw,J.M.2008.HowtodowithOWLwhatpeoplesayyoucan't.Invited keynoteforthe2008IEEEWorkshoponPoliciesforDistributedSystemsandNetworks. 24June2008.Palisades,NY. Broome,B.D.2005.ActionableIntelligencefortheWarfighter.InvitedTalkatthe 2005NettedSensorsWorkshop.2426October.McLean,VA.http://www.mitre.org/ nettedsensors/2005 Bunch,L.,Bradshaw,J.M.,Young,C.O.2008.PolicyGovernedInformationExchange inaU.S.ArmyOperationalScenario.InProceedingsofthe2008IEEEWorkshopon PoliciesforDistributedSystemsandNetworks.24June2008.Palisades,NY. Carvalho,M.,Suri,N.,Shurbanov,V.&Lloyd,E.2007.Acrosslayersubstrateforthe battlefield.Pensacola,FL:IHMCPositionPaper. Crocker,M.2007.CrossDomainInformationSharinginaTacticalEnvironment. CrosstalkJournalofDefenseSoftwareEngineering.March2007.http:// www.stsc.hill.af.mil/crosstalk/2007/03/0703Crocker.html MILSTD2525BCommonWarfightingSymbology.2007.DefenseInformation SystemsAgency(DISA). DoDReporttoCongress.2001.NetworkCentricWarfare.27July2001.http:// www.defenselink.mil/cionii/docs/pt2_ncw_main.pdf Feltovich,P.J.,Bradshaw,J.M.&Bunch.2009.Policyandsocialbarrierstonew militaryinformationtechnologies.Pensacola,FL:IHMCTechnicalReport. FieldManual1008.1997.U.S.DepartmentoftheArmyHeadquarters. Fikes,R.,Jenkins,J.,&Frank,G.JTP:ASystemArchitectureandComponentLibrary forHybridReasoning.ProceedingsoftheSeventhWorldMulticonferenceonSystemics, Cybernetics,andInformatics.Orlando,Florida,USA.July2730,2003. Kean,T.H.,Hamilton,L.H.,etal.2004.The9/11CommissionReport.National CommissiononTerroristAttacksUpontheUnitedStates. D.McGuinness,F.vanHarmelen,OWLWebOntologyLanguageOverview.http:// www.w3.org/TR/2004/RECowlfeatures20040210/. McNaugher,T.L.1989.Newweapons,oldpolitics.Washington,DC:Brookings Institute. Mittrick,M.R.,Richardson,J.T.,Kaste,R.C.2008.APolicyDrivenInformation ExchangeNetwork.ArmyResearchLabsTechnicalReportARLMR704. Sirin,E.,Parsia,B.,CuencaGrau,B.,Kalyanpur,A.,Katz,Y.,Pellet:ApracticalOWL DLreasoner.WebSemantics:Science,ServicesandAgentsontheWorldWideWeb,v.5 n.2,p.5153,June,2007. Suri,Niranjan,JeffreyM.Bradshaw,MarcoCarvalho,MaggieR.Breedy,ThomasB. Cowin,RaulSaavendra,andShriniwasKulkarni.2003.Applyingagilecomputingto supportefficientandpolicycontrolledsensorinformationfeedsintheArmyFuture CombatSystemsenvironment.PresentedattheAnnualU.S.ArmyCollaborative TechnologyAlliance(CTA)Symposium,April,2003. Suri,N.,Benincasa,G.,Formaggi,S.,Winkler,R.,Choy,S.,Kovach,J.,andTokarcik,L. 2008.DisService:APeertoPeerInformationDisseminationServiceforTactical Environments.InProceedingsofthe2008MeetingoftheMilitarySensingSymposia (MSS)SpecialityGrouponBattlespaceAccousticandSeismicSensing(BAMS2008).

88BUNCHETAL.
Uszok,A.,Bradshaw,J.,Lott,J.Breedy,M.,Bunch,L.,Feltovich,P.,Johnson,M.and Jung,H.2008.NewDevelopmentsinOntologyBasedPolicyManagement:Increasingthe PracticalityandComprehensivenessofKAoS.InProceedingsofthe2008IEEEWorkshop onPoliciesforDistributedSystemsandNetworks.24June2008.Palisades,NY.

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T A N D L A U R E L A L L E N D E R

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T AND LAUREL ALLENDER

This book, produced by the Partners of the Army Research Laboratory Advanced Decision Architectures Collaborative Technology Alliance, also serves as the Final Report for DAAD19-01-2-0009.

SECTIONI.COLLECTING,PROCESSING,ANDDISTRIBUTINGBATTLEFIELD INFORMATION SECTIONI COLLECTING,PROCESSING,ANDDISTRIBUTINGBATTLEFIELD INFORMATION

14SECTIONI

Chapter

5. XLAYER:ACROSSLAYERCOMMUNICATIONSSUBSTRATEFORTACTICAL ENVIRONMENTS

MarcoCarvalho,Ph.D. InstituteforHumanandMachineCognition,Pensacola,FL AdrianGranados InstituteforHumanandMachineCognition,Pensacola,FL CarlosPerez InstituteforHumanandMachineCognition,Pensacola,FL MarcoArguedas InstituteforHumanandMachineCognition,Pensacola,FL RobertWinkler ArmyResearchLaboratory,Adelphi,MD JesseKovach ArmyResearchLaboratory,Adelphi,MD SteveChoy ArmyResearchLaboratory,Adelphi,MD

INTRODUCTION Tacticalandmobileadhocnetworks(MANETs)arekeytechnology enablersforfuturemissioncriticalcommunicationinfrastructuressuch asthoseenvisionedfordisasterreliefoperationsandmilitarymissions. TheselforganizingnatureofMANETsenablestheondemandforma tionoftemporarycommunicationscapabilitiestosupporttheoperation

89

90CARVALHOETAL. ofmissioncriticalapplicationsintheedgesofmaincommunications infrastructure,acriticalcapabilityformilitaryoperations. However,thisunprecedentedcapabilitycomesataprice.Thestabil ityandreliabilitygenerallyobservedinfixedcommunicationsinfrastruc tures,whichareoftenleveragedandassumedbytacticalapplications anddecisionarchitectures,arenonecessarilyavailableinMANETs(at leastatsimilarlevels).Theflexibilitythatenablestheondemandforma tionofMANETsalsocreatesanewtypeofcommunicationsenvironment, onethatisdynamicandsignificantlymoreresourceconstrained. Whiletherehasbeenasignificantamountofresearchfocusedpri marilyonbuildingnewMANETtechnologiesandprotocolsthatwould providebetterpredictabilityandreliability,itiswidelyacceptedtoday thatapracticalsolutionwilllikelyrequireabetterinteractionbetween applicationandcommunicationsinfrastructure.Thetraditionallayered approachthathasshieldedapplicationsfromtheunderlyingnetworksis nolongerapplicable,andnewchallengehasemergedhowtoprovide suchinterface(ormutualawareness)whileavoidingcustomizedstove pipedsolutions. Thenotionofacrosslayercommunicationssubstratefortactical battlefieldenvironmentsdescribedinthisworkproposestoaddressthe problembyenablingatwowayinterfacebetweenhigherlevelapplica tions,middlewareordecisionarchitectures,andtheunderlyingcommu nicationsinfrastructure. Inthisworkwesummarizeourresearchforthedevelopmentof XLayer,acrosslayercommunicationssubstratedesignedtoprovideand supportcommunicationinfrastructuresforthebattlefield. Thegoalistoprovideamodulararchitectureandasetofinterfaces that1)allowsapplicationsanddecisionarchitecturesystemstobetter adapttoandleveragethecharacteristicsofthedynamiccommunications environmentand,2)enablestheunderlyingcommunicationsinfrastruc turetobettersupportapplicationrequirementsandconstraints. Insupportoftheapplicationanddecisionarchitecturesystems, XLayermonitors,abstracts,andrepresentsthecharacteristicsandcapa bilitiesoftheunderlyingcommunicationsinfrastructuresoapplications canbetteradapttochangesintheunderlyingcommunicationsenviron ment(byreallocatingresources,forexample). Insupportofthecommunicationsinfrastructure,applicationscan provideinformationaboutresourcerequirements(bothcomputational andcommunications)orutilizationpatterns.Thisinformationcanthen beusedbytheunderlyingcommunicationsinfrastructuretobetterallo cateresourcesandcapabilitiesinresponsetochangesthatmayoccur

TheXLayerCommunicationsSubstrate91 eitherondemandorproactively,basedonexplicitapplicationrequire mentpatterns. RELATEDWORK Traditionally,crosslayerstrategiesfortacticalandmobileadhocnet workshaveprimarilyfocusedonshorttermadaptation(andstate reporting)betweenneighboringprotocollayers,generallyfortheoptimi zationofasingleprotocolsuchasTransmissionControlProtocol(TCP) (Kunniyur&Srikant,2000;Srikant,2004). Ingeneral,mostimplementationsarebasedonvariationsofQuality ofService(QoS)protocolsinheritedfromthewirednetworksandstilluti lizesomeofthenotionsofsignalingandcoordinationofneighboring protocollayersforresourcereservation.Thegoalofmosttraditional crosslayerstrategiesistomonitoranddetectshorttermchangesin channelconditions(orcompetingtraffic)tonotifyupperlayersabout newQoSconditions.Inmostcases,applicationsaregenerallyexpected toadjustdataratesaccordinglywhennotifiedbyaneighboringlayerthat currentserviceexpectationsarenolongeravailable. AsillustratedbyGoldsmith&Wicker(2002),theactualadaptation andreportingbetweenlayersisgenerallydoneafterlocallayeradapta tionsarenolongerpossible(orcosteffective).Thedifferenttimescales ateachlayerusuallyimplythatlocaladaptationwithineachlayergener allyoccursfirst(andmorefrequently)thanadaptationbetweenlayers. ProtocolslikedRSVP(Mirhakkak,Schult,&Thomson,2001),for instance,provideperflowendtoendbandwidthguaranteesforarange ofrequirements(asopposedtoaspecificrequirementlikeinRSVP).In thiscase,dRSVProutersexchangebandwidthreservationdetails throughasignalingprotocolandtheflowiseitherdeniedaccessor droppedifchannelavailabilitybecomesinsufficient. Signalingforshorttermresourcereservationisalsousedbythe SWANProtocol(Ahn,Campbell,Veres,&Sun,2002).SWAN,likedRSVP,is fullydecentralized,butitisbesteffortonlyandmakesnoassumptions aboutunderlyingQoScapabilitiesfromtheMediumAccessControl (MAC)layer.ThesignalinginSWANisintendedforflowadmissionand thecrosslayernatureoftheprotocolliesinthefactthatMAClevel packetdelayinformationissharedandusedforestimatingmedium accesscontention.AfteraflowisadmittedinSWAN,theprotocoluses thepacketsexplicitcongestionnotificationflag(ECN)tonotifythat requestedservicesarenolongersupportedforthatflow.

92CARVALHOETAL. TIMELY(Bharghavanetal.,1998)isanothercrosslayerarchitecture thatprovideslinklayerscheduling,resourcereservationandadaptation, aswellaspriorityawaretransportprotocolthatselfregulatesflowbased onfeedbackfromthelowerlayers.TIMELYwasinitiallyproposedforcell basedwirelessnetworks,andhelpedcreatethebasisforsubsequentad hocspecificarchitecturesandprotocolswithsimilarcapabilitieslike Spine(Sivakumar,Das,&Bharghavan,1998)andCEDAR(Sivakumar, Sinha,&Bharghavan,1999). THEXLAYERCOMMUNICATIONSSUBSTRATE XLayerisspecificallydesignedtoprovideandsupportacommunica tioninfrastructurethatallowsapplicationsanddecisionarchitecturesys temstobetteradaptandmeetQoSrequirementsandconstraints.A modulararchitecturemakesXLayerflexibleenoughtosupportdifferent scenariosthatrequireofveryspecificcapabilities. IntheXLayerarchitecture(Figure5.1),eachservicemoduleprovides asetofcapabilitiesthatcanbedirectlyutilizedbyothermodels,byload ablesubcontrollers,orevenbyoverlayingapplications.Furthermore, stateinformationcreatedormaintainedbydifferentmodulescanbereg isteredandmadeavailabletoothercomponentswithinthearchitecture. Thegoalistofacilitatethesharingofstateinformation,avoidingthe redundantrediscoverycostsandoverhead.Forexample,neighborhood andlinkdetectionarecriticaltasksforseveraltypesofapplicationsand services(e.g.routing,discovery,datadissemination,etc.).Mostoften, eachoftheseservicesprovideitsowndetectionmechanism,whichcould beavoidedifinformationobtainedbyoneservicecouldbeeasilyshared andusedbyothers.TheXLayercommunicationssubstrateenablesthis capability,amongstmanyothers.

TheXLayerCommunicationsSubstrate93

Figure5.1 TheXLayerArchitecture XLayersupportstwotypesofmodules:servicesandcontrollers. XLayerservicesprovideessentialcapabilitiestootherXLayercompo nentsandaretypicallystarteduponinstantiationoftheXLayer.XLayer controllersallowdeveloperstoextendthefunctionalityoftheXLayer andcanbeenabledordisableddependingontherequirementsofclient applications. XLayerServices XLayerservicesaremodulesthatprovidetheessentialcapabilities requiredbyotherXLayercomponentstoperformmorecomplextasks.In theXLayer,asetofcoreservicesprovidesupportfornetworkinterface detectionandconfiguration,transportofmessages,flooding,neighbor discovery,andmonitoringofthedifferentnodeandnetworkproperties, suchasresourceutilization,linkquality,topology,androuteinformation. TheNetworkManagementService.TheNetworkManagementSer vicedetectsandmanagesthephysicalnetworkinterfacesthatareavail abletotheXLayer.Unlesspredefinedinterfacesarelistedinthe configurationfile,thisservicewillenableallactiveEthernetdevices, includingwiredandwirelessnetworkinterfaces.

94CARVALHOETAL. TheNetworkManagementServicealsoallowsforpseudointerfaces thataretreatedbytheXLayerasregularnetworkinterfacesandcanbe usedtransparentlybyothercomponents.Oneexampleofapseudo interfaceistheonethatgetsinstantiatedwhenaslaveXLayerattaches itselftoanotherXLayeractingasamaster.Thishappensthroughawired connectionbetweentwoplatforms,eachindependentlyrunningthe XLayer.Uponconnection,oneXLayerbecomesthemasterandtakes controlovertheinterfacesofthesecondXLayer(theslave),forwhichall controllersandservicesareautomaticallyshutdownandsupersededby thecorrespondingcontrollersandservicesrunningatthemaster XLayer.Theconnectionbetweenmasterandslavehappensthrougha pseudointerface(whichisjustadirect,localTCPconnectionbetween them).Whileconnected,asinglenodeisvisibleatthenetworklevel. Upondisconnection,theslaveXLayerimmediatelyreinstatesitsmain servicesandcontrollers,becomingagainanindependentnode. Inadditiontothesecapabilities,theNetworkManagementService gathersstatisticsonthenumberofpackets(andbytes)transmittedand receivedoneachofthenetworkinterfaces.ItalsomakesavailableanAPI toactivateanddeactivateinterfaces,andtonotifyothermodulesabout theirstatus(up,down). TheMessagePropagationService.TheMessagePropagationService provides1hopcommunicationwithotherXLayerenablednodes.Ituses unicastorbroadcastUDPpacketstoencapsulateoneormoreXLayer messages.MessagesintheXLayerhaveauniquetypeidentifierthatis associatedwithapredefinedmessageformat.XLayercontrollersand servicesmayimplementnewmessagetypestosupportnewcapabilities. Inthiscase,theMessagePropagationServicewilldelegatemessage parsingandhandlingtothecomponentsregisteredforthenewmessage type. TheMessagePropagationServicealsoenablesbasicneighbordiscov erythroughtheexchangeoftwotypesofperiodicmessagescalledbea consandRTTs(roundtriptime).Thesemessagesareusedbythenodeto advertiseitspresenceinthenetworkandtoletothernodesmonitorthe qualityofexistinglinks,includingroundtriptimes,packetdelaysand packetlosses.Theintervalsatwhichtheseperiodicmessagesaresent canbeindividuallytunedforeachoftheavailablenetworkinterfaces. Forinterfaceswithlimitedcapacity,increasingtheintervaldecreasesthe effecttheseperiodicmessageshaveonlinkquality.Ontheotherhand, decreasingthisintervalcanincreasetheaccuracyoftheinformationrela tivetothestateofthelinkatanygiventime.

TheXLayerCommunicationsSubstrate95 Althoughtheexchangeofdiscoveryandlinksensingmessagesorigi natesattheMessageProcessingService,theactualprocessingofthese messagesisdelegatedtootherXLayerservicesthatmakethelinksensing and1hopneighborinformationavailabletootherXLayercomponentsin acentralrepository.Thelinksensingisperformedusingbroadcastin mostofthecases,althoughmulticastcanalsobeutilizedforspecificdis coverytasksifsupportedbytheunderlyinghardware. IntheMessageProcessingService,outgoingmessagesarequeued untilacertaintimehaselapsed(delaytolerance)andthentransmitted togetherinasinglepackettoreducebandwidthutilization.Alternatively, timesensitivemessagescanbesentoutimmediately.Inthiscase,mes sagesmaybesentoutinasinglepacketwithothermessagespendingfor transmission.Incomingmessagesarealwaysqueuedforprocessingby thecorrespondingregisteredmessagehandler(s).Amessagehandlercan beanotherXLayerservice,anXLayercontroller,oraclientapplication. XLayercontrollersandservicescanalsoregisterextendedpackethead ers,alongwithpacketfiltersthatcanbeusedtoaccept,reject,modifyor simplyinspectthecontentofamessage,henceenablingtopologyemu lationcapabilities,instrumentation,monitoringandotherimportantfea turesrequiredbycontrollersandmiddlewareapplications. TheTransportService.ThemaingoaloftheTransportServiceisto provideendtoendcommunicationforXLayerenablednodes.The receptionandforwardingofmessagesisdonethroughtheMessage PropagationService,however,theTransportServiceisresponsiblefor determiningwhichinterfaceorinterfaceswillbeusedtoforwardthe message.ThismakespossiblefortheTransportServicetosupportdiffer entmodesofoperationtoaccommodatetheQoSrequirementsofother componentsandapplications.TheTransportServicesupportsfourbasic modesofoperation: Best:inthismode,messagesareforwardedusingthebestavail ablelinktoreachanexthoptothedestination,accordingtothe routinginformationprovidedbytheInformationService. Capacity:inthismode,theTransportServiceinterleaves,if possible,messagesamongthedifferentlinksthatcanreacha nexthoptothedestinationinordertosharebandwidth utilizationanddecreasethepossibilitytooverloadasingle(best) link. Reliability:inthismode,theTransportServicewillduplicate,if possible,messagesamongthedifferentlinksthatcanreacha nexthoptothedestinationinordertominimizepacketlosses.

96CARVALHOETAL. Adaptive:inthismode,theTransportServiceusesthelink qualityinformationcollectedbytheMessageProcessingService ofalloutgoinglinkstodeterminetheforwardingstrategythat bestsatisfiestheQoSrequirementsoftheapplication. AnotherimportantkeyfeatureoftheTransportServiceistheability totrackandsplitflowsinordertosupportdualpathalgorithms.Thisis achievedbymarkingeachmessagewithaflowidentifierwhichisusedto implementdifferentforwardingandflowcontrolstrategies. TheDisseminationService.Whilefloodingisoftenusedbynumerous routinganddistributedcoordinationalgorithmsinMANETs,itgenerally constitutesaveryexpensiveoperationinmobileadhocnetworks(Will iams&Camp,2002).ThegoaloftheDisseminationServiceistoprovide acommonAPIforaconfigurablesetofmessagefloodingmechanismsfor MANETs.TheDisseminationServiceprovidesamechanismforregister ingmessagesthatneedtoberepeatedlyandperiodicallydisseminatedat specifictimeintervals,effectivelyenablingapowerfulAPIforproactive messagepropagationinthenetwork. TheDisseminationServiceisresponsiblefordisseminatingames sagethroughoutthenetworkusingthemostsuitablefloodingalgorithm giventhecurrentnetworkconditions.Itusesinformationcollectedby otherXLayercomponentstodeterminethebeststrategy(orcombina tionofstrategies)forflooding.Furthermore,eachfloodingalgorithm mayusetopologyandneighborhoodinformationalreadyavailablefrom otherXLayercomponentstodeterminetheforwardingroleofeach node.Havingthisinformationalreadyinplacehelpssimplifyandspeed uptheimplementationoffloodingalgorithms. TheInformationService.TheInformationServiceisthecentralrepos itoryofsharedinformationintheXLayer.Itholdsinformationaboutthe localandremotenodes,aswellastheexistinglinksbetweenthem.This serviceisalsoresponsibleforcomputingtheroutingtableusingthe availabletopologyinformation,andisresponsibleforadvertisingtothe othernodesitslocalproperties,suchasCPU,memoryanddiskspaceuti lization,networkinterfaces(IPaddressandnetworkmask),nodeposi tion(UTMandGPScoordinates),andothers. InformationintheInformationServicethatvarieswithtimeisusu allystoredasametric.Ametricissimplyaslidingwindowstructurethat keepstrackofashorthistoryofvaluesforavariable.Forexample,the CPUutilizationisstoredasametricthatkeepsashorthistoryofvalues foreachtimetheCPUutilizationwasmeasured.Ametricprovidesinfor mationabouttheaveragevalue,variance,andsimplestatisticaltestsfor trendanalysis.

TheXLayerCommunicationsSubstrate97 Althoughsomemetricsandpropertiesseemtobecommonamong heterogeneousnodes,sometimesnotallofthemareaccessible.Others areonlyavailabledependingontheparticularcharacteristicsofthe underlyinghardware,suchastransmissionpower,ESSID,channel,etc. Therefore,XLayerservicesandcontrollersdonotcompletelydependon theinformationgatheredbytheInformationService,butwhenavailable, theyuseittosimplifytheirtasks. TheInformationServicegathersitsinformationfrommultiple sources.Informationaboutresourcesinthelocalnodeisobtainedfroma specializedcomponentcalledtheSystemResourceMonitor,whichmoni torsCPUutilization,freememory,freediskspaceandnetworkIOutiliza tion. Informationabout1hoplinksisinferredfromthelinksensingper formedbytheMessagePropagationService.Everypacketreceivedby theMessageProcessingServiceisreportedtotheInformationService. Dependingonthetypeofpacketreported,differentinformationis inferred.DatapacketsallowtheInformationServicetodeterminethata linkisstillactiveandthataneighbornodeisstillaliveandimmediately reachable.BeaconsandRTTsallowtheInformationServicetocompute droprateandlinkdelaymetrics. Thedropratemetriciscomputedoveraslidingwindowbydividing thenumberofbeaconsreceivedbytheexpectednumberforthattime interval.Thenumberofexpectedbeaconsiscomputedfromthebeacon ingratereportedinsidethebeacon. Thelinkdelaymetriciscomputedbyaddingtheexpectedbeacon arrivaltime(basedontheirsequencenumbersandreportedbeaconing rate)withonehalfoftheRTT. Informationaboutremotelinksisgatheredfromothermodulesand controllers.Forexample,theHazySightedLinkState(HSLS)XLayer controller,whichadvertisestheincominglinksofthenode,alsocollects theincominglinksreportedbyotherHSLScontrollersrunningonother nodesandfeedsthisinformationtotheInformationService.This1hop neighborhoodinformationisusedbothtocomputetheroutesandby otherXLayermodules. TheInformationServicealsocollectsinformationadvertisedbyother InformationServicesrunningonothernodes.Fromthisitbuildsaview oftheworldthatcanbeusedbyotherXLayermodulestoinferthestate ofthenodesandthenetwork.Additionally,anyXLayermodulecanuse theInformationServicetostoreanyinformationtheywanttosharewith others.Forexample,acontrollerthatcollects2hopneighborhood informationcansharethisdatathroughtheInformationServicesothat

98CARVALHOETAL. theDisseminationServicecancomputemultipointrelaystoefficiently floodpacketsinthenetwork. Ingeneral,XLayerservicesandcontrollersrequestinformationfrom theInformationServicewhenneeded.ButtheInformationServicealso providescallbackmechanismsforasynchronouschangenotification.For linkchanges,forexample,callbackscanberegisteredtoreceivenotifica tionwhenalinkisadded,removed,oritscostchanges.Formetrics,the InformationServiceprovidestwoformsofasynchronousnotification. Thefirstallowssubscriberstoreceivenotificationwheneverametricis updated.Thesecondallowssubscriberstospecifyarangeforthemetric andgetnotifiedassoonasarangeviolationisdetected. TheBasicService.TheBasicServicemakesavailabletwokeyfunc tionalitiestootherXLayermodules:1)amechanismtoload/startand unload/stopXLayercontrollers,and2)amechanismtoaddorremove bridgeconnections.AbridgeconnectionisapermanentTCPconnection thatisestablishedwithanotherXLayerservice.Thisconnectionisseen asanadditionalnetworkinterface;allpacketswrittentothisinterface areforwardedtotheotherXLayerservice.Thisenablesatransparent bridgebetweentwoormoreXLayerawarenetworkswithdifferent addressspaces,transportprotocols,orroutingalgorithms. TheLoggingService.ToallowforremoteloggingoftheXLayerser vice,theLoggingServiceprovidesamechanismwherelogmessagesare interceptedandsentoveraUDPsocket.Thisfacilitatesmonitoringand debuggingXLayermodulesexecutedonplatformswhichcanonlybe accessedremotelyorhaveverylimitedstoragespace. XLayerControllers ThefunctionalityoftheXLayercanbeextendedthroughXLayercon trollers.AnXLayercontrollerisamodulethatimplementsoneormore algorithmstoperformspecifictasks.Inmanycases,thesealgorithmscan bemadeavailabletootherXLayermodulesandcomposedtofurther extendtheXLayerfunctionality. InthesamemannerasXLayerservices,XLayercontrollersimplement theXLayermoduleinterface.ThisinterfaceisusedbytheXLayertoini tialize,start,andstopacontroller.XLayercontrollerscanalsobedynami callyloadedbytheXLayer.DynamicloadingofXLayercontrollershas severaladvantages.First,itmakespossibletoextendthefunctionalityof theXLayerwithoutchangingtheXLayerexecutable.Second,itencour agesthereutilizationofcodebysharingcommonfunctionalityamong controllers.Andthird,itfacilitatestheintegrationXLayercontrollersby

TheXLayerCommunicationsSubstrate99 thirdparties.SomeofthecurrentXLayercontrollerscanbecategorized asfollows. AdaptiveandPredictiveRouting.XLayercontrollerssuchastheOLSR (OptimizedLinkStateRouting)andHSLS(HazySightedLinkState)provide routingcapabilitiestotheXLayer.Inparticular,theHSLScontrollerallows theXLayertoshareinformationaboutincominglinkswithotherXLayer enablednodes.Theinformationreceivedfromothernodesispassedto theInformationServicetolatercomputeaninternalroutingtablethatis usedtosupporttransportcapabilitiesintheXLayer.TheHSLScontroller usestheDisseminationServicetosharetheincominglinkinformation withallthenodesinthenetworkusingincrementalTTLsasspecifiedin SantivanezandRamanathan(2001). UsingtheDisseminationServicetosendthelinkstateinformation allowstheHSLScontrollertoleveragedifferentdisseminationmecha nismsbeyondsimpleflood,suchasmultipointrelay.TheHSLScontroller doesnotdolinksensing.Forthis,itusesinformationalreadyprovidedby theInformationandMessagePropagationServices. ApredictiveversionoftheOLSRprotocolwasalsoimplementedas anXLayercontroller.AsillustratedinFigure5.2,thepredictiverouting controllerwasresponsibleforcorrectingtheoffsetofhellomessagesin OLSRtoconstructaprojectionofthetopologyfornexthopselection. Topologyestimationcanbeobtainedthroughashorttermprojec tionofthelinkstatetoeachofitsimmediateneighborstoweightthe selectionofthenexthop.Inourpreviousresearch(Carvalho,etal., 2008),wehaveshownsignificantimprovementsinbothpacketlossand averagedelayforpredictiveroutingbysimplyadjustingthelocaltopol ogybasedonshorttermmobilityandlinkqualitytrendsforneighbor nodes.

100CARVALHOETAL.

Figure5.2 ACrossLayerPredictiveRoutingProtocol VirtualTopologyControl.TheVirtualTopologyControllerregistersan extendedheaderandapacketfiltertoinspectallpacketsreceivedbythe MessagePropagationService.TheVirtualTopologyControllerthen acceptsorrejectspacketsbasedonthesenderandreceivernodesvir tualpositionandtransmissionpower,constructingavirtualtopology thatcanbeusedtosetupsimulationsandexperiments. TopologyAdaptationandControl.Supportfortopologyadaptation andcontrolalgorithmsisprovidedbycontrollerssuchastheCDS(Con nectedDominatingSet)andFlowcontrollers. TheCDScontrollerallowscomputingaConnectedDominatingSet (Wu&Li,1999)fortheXLayernetwork.Thecomputationisperformedin adistributedmanner,whereeachXLayerenablednodedetermines whethertobepartoftheConnectedDominatingSetbasedoninforma tionaboutits1hopand2hopneighbors. Sometopologycontrolalgorithms,(e.g.,thedualpathmechanism supportedbytheTransportService),makeuseofinformationaboutthe flowsofdatagoingthroughtheneighborsofthelocalnode.Particularly, thedualpathmechanismcanusethisinformationtooptimizethedual nexthopselectionalgorithm.Theinformationaboutflowsgoingthrough thelocalnodeisgeneratedbytheTransportServiceandstoredinthe InformationService.TheXLayerFlowcontrollerthentakesthisinforma

TheXLayerCommunicationsSubstrate101 tionfromtheInformationServiceandsharesitwithothernodesthatare runningtheXLayerFlowcontroller.Theinformationreceivedfromother nodesisthenalsostoredintheirrespectiveInformationService,soitcan belaterusedbytheTransportServiceandbyothertopologycontrol algorithms. MulticastForwarding.TheXLayerMulticastcontrollerenableseffi cientmulticastpacketforwardinginMANETsbasedontheapproachfol lowed bytheOLSRBasicMulticastForwarding(BMF)plugin(Tromp, 2006).Itcaptureslocaltrafficandencapsulatesmulticastpacketsinto XLayermessagesthatarethenfloodedthroughtheDisseminationSer vice.Additionally,thecontrollermaylistentoICMPgroupmembership messagestobuildmulticasttreesasinJacquetetal.(2001),whichhelps toreducethenumberofdisseminatedpacketsonlargenetworkswhere onlyasmallnumberofnodesaremembersofthesamemulticastgroup. GroupManagementandDiscovery.Asetofcontrollersprovides basicgroupmanagementanddiscovery.ThesecontrollersusetheDis seminationServicetopermanentlyadvertisegroupmembershipand performpeersearcheswithinacertainscope(i.e.,hopdistance).By usingtheDisseminationService,differenttypesoffloodingalgorithms canbeusedaccordingtothenetworkconditions.Forexample,Figure5.3 showstwotypesofflooding:ontheleft,asimplefloodmechanism;on theright,anadaptivefloodwhichusestopologyinformationtodeter minethebestfloodstrategyaimingtoreducethenumberofretransmis sionsthatareneededtoreachallthenodesinthenetwork.

Figure5.3 FloodbasedDiscoveryusingtheAdaptiveDisseminationService

102CARVALHOETAL. PlatformEmulation.ThemodulardesignoftheXLayerallowsthe incorporationofdifferentimplementationsofcontrollersthatperforma seriesoftasksaccordingtothescenariowheretheXLayerisdeployed. ThisenablesusersoftheXLayertotransparentlyinterfacewithanode withoutmakinganyassumptionsabouttheenvironmentwherethe XLayerisrunning.Forexample,thePlatformcontrollersupportsthe queryandmanipulationofplatformrelatedfunctionalitysuchasposi tion,direction,andspeedofthenode.

Figure5.4 TheMANETEmulationTestbedsupportedtheXLayer ForourMANETemulationtestbed,showninFigure5.4,theXLayer providesanimplementationofaplatformcontrollerthatallowsinterfac ingwitheachofthenodes.Similarly,otherimplementationsoftheplat formcontrollercanbeprovided,forexample,tocontrolarobotic platform. DynamicGatewaySelection.Havingacommoninfrastructurefor sharinginformationandfunctionalityallowstheXLayertousedifferent routingstrategiesthatcouldhelponreducingroutingrelatedtrafficon ultradensenetworks.Forexample,Figure5.5showsaroutingstrategyin asegmentednetworkthatassignsdifferentroutingprotocolstodifferent portionsofthenetwork.Then,nodesoneachsegmentautonomously decidetoserveasgatewaysbetweenthevariousnetworksegments. Ingeneral,clientapplicationscommunicatewiththeXLayerservice usingTCPsocketsandaproprietarybinaryprotocol.Inordertosimplify thedevelopmentoftheseclientapplications,aproxylibraryenables remoteprocedurecallsbyhandlingthemarshalingandunmarshalingof functionparametersandresults.Italsohandlestheregistrationprocess ofseveralcallbackmechanisms,includingthestateofthechannel

TheXLayerCommunicationsSubstrate103 betweentheproxyandtheXLayerservice(connected,disconnected), receptionofmessagestargetedtotheclientapplication,andmonitoring eventssuchasmetricorpropertyupdates.

Figure5.5 SimulationoftheAdaptiveGatewaySelectionalgorithm TheproxylibraryisimplementedinC++,butaJavaNativeInterface (JNI)wrapperallowsJavaapplicationstointeractwiththeXLayeraswell. ThishasenabledanimportantfeatureoftheJavaproxy:animplementa tionofadatagramandstreamorientedsocketsthatmaybeusedtrans parentlybyneworexistingJavaapplicationstotakeadvantageofthe XLayertransportcapabilitiessuchasdualpath,multipathrouting,and adaptivetransport. INTERACTINGWITHTHEXLAYER TheXLayerDatagramSocketenablesendtoend,unreliabledelivery ofpacketsoverUDPthroughtheXLayerTransportService.TheXLayer Socket,ontheotherhand,implementstheReliableUDPprotocol(Bova &Krivoruchka,1999)overXLayerDatagramSocketstosupportareliable, inorderdeliveryofpacketsthroughtheTransportService,providinga streamorientedtransportprotocol,similartoTCP. AnotherwayofinteractingwiththeXLayerserviceisachieved throughtheuseofproviders.Providersarespecializedsmallclient applicationsthatcanmodifyorupdatethestateoftheXLayerorthe informationthatitmaintainsattheleveloftheInformationService.They canalsobeusedtoextendthecapabilitiesoftheXLayerbyproviding queryandmanipulationoperationsonexternalhardwareorsoftware

104CARVALHOETAL. entities,suchasGPSdevicesoraroboticplatform.Essentially,aprovider applicationinterfaceswiththeunderlyinghardwareorsoftwareentityto enableverybasicoperationssuchassettingandgettingthevaluesof particularproperties. CommunicationbetweentheXLayerandproviderscanbesynchro nous:theXLayerservicebasicallyqueriestheprovidertoupdateitsinter nalstate.Communicationcanalternativelybeasynchronous:XLayer registersitselfwiththeprovidertogetupdatesonaspecificproperty throughacallbackmechanism.Forexample,inthecaseoftheGPS device,theXLayermayqueryaGPSprovidertoperiodicallyupdatethe nodesgeographicalposition,oritcantelltheprovidertonotifyitwhen everthenodemovestoaspecificlocationoroutofacertaindistance range. CONCLUSIONS ThemodulararchitectureoftheXLayerprovidesthenecessary mechanismstoenabletheinteractionbetweenapplicationsormiddle wareandthedifferentlevelsofthecommunicationssubstrate,which makesthedesignoftheXLayerflexibleenoughtosupportalargerange ofdifferentscenariosandcapabilities. Amodulardesignalsofacilitatesthedevelopmentofmorecomplex functionalitybyleveragingontheinformationandfunctionalityprovided bycoreservicesthatabstractessentialcapabilitiesrealizedacrossthe variouslayersofthecommunicationsstack. ArichsetofAPIsallowapplicationstobetteradaptandallocate resourcesforagiventask,whileatthesametimecontributingtobetter definetheinformationandresourceneedsthatcanbeusedbythe XLayerservicetoallocatethenecessarycapabilitiessoastomeetthe QoSrequirementsofapplications. ThedesignoftheXLayeralsomakesitanextensibleplatformsuit ableforresearchandexperimentationofnewprotocolsandalgorithms fortacticalandmobileadhocnetworksforthebattlefield. TheXLayerwasimplemented,testedanddemonstratedinmultiple exercisesandexperimentscarriedincollaborationwiththeArmy ResearchLaboratory,theAirForceResearchLaboratory,andcommercial companieslikeRaytheonCo.andRockwellCollins. Duringthelifeoftheproject,theXLayerhasbeenportedtoseveral operatingsystems(Linux,OpenWRT,Pyramid,Windows,MacOSX),mul tiplecomputingplatforms,andtacticalradios.Differentconfigurations havebeenutilizedtodemonstrateandhighlightspecificcapabilitiesof

TheXLayerCommunicationsSubstrate105 theXLayer,suchasdiscovery,linksensing,adaptiveandpredictiverout ing,topologycontrolandadaptation,enhancedforwardingofbroadcast andmulticasttraffic,andothers.Whenpossible,theXLayerwasexe cutedaspartofthecommunicationsdevice(forinstance,inthecaseof theLinksysbased803.11radios,theiPodTouchplatforms,andthe Soekrisbasedradios).Inothersituations,suchasthecasewherethe XLayerneededtosupporttacticalradios,thecommunicationbetween theradiosandtheXLayerwasrealizedthroughtheimplementationof customizedinterfaceadaptors(orpseudointerfaces)loadedbytheNet workManagementService. AmultisystemdemonstrationincludingatacticalInformationMan agementSystemforanairbornegroundsupportoperationscenariowas executedattheAirForceResearchLaboratoryinearly2009.Duringsuch demonstration,theXLayerenabledseamlessoperationacrossthemulti plenetworkandcommunicationsenvironments.Thekeyfeaturesofthe XLayerthatmadepossiblesuchintegrationincludetheutilizationofmul tipleredundantlinkstoprovideenhanceddiscovery,nodeandnetwork resourcemonitoring,seamlesscrossdomainroutingandondemand multipathtransport. TheXLayerhasalsobeenappliedinsupportofkeycapabilitiesfor tacticalInformationManagementSystemsforAirborneNetworkenvi ronments(Carvalhoetal.,2009),qualityofserviceenableddatadissem ination(Loyalletal.,2009),andseveralotherapplicationsinsecurityand networkmanagement. REFERENCES
Ahn,G.S.,Campbell,A.,Veres,A.,&SunL.(2002).Swan:Servicedifferentiationin statelesswirelessadhocnetworks.INFOCOM02:Proceedingsofthe21stAnnualJoint ConferenceoftheIEEEComputerandCommunicationSocieties,2,457466. Bharghavan,V.,Lee,K.,Lu,S.,Ha,S.,Li,J.,&Dwyer,D.(1998).TheTIMELYadaptive resourcemanagementarchitecture.IEEEPersonalComm.Magazine,5(4),2031. Bova,T.,&Krivoruchka,T.(1999).ReliableUDPprotocol.URL:http://www.ietf.org/ proceedings/99mar/ID/draftietfsigtranreliableudp00.txt Goldsmith,A.,&Wicker,S.(2002).Designchallengesforenergyconstrainedadhoc wirelessnetworks.IEEEWirelessCommunicationsMagazine,9(4),827. Carvalho,M.,Winkler,R.,Perez,C.,Kovach,J.,&Choy,S.(2008).Acrosslayer predictiveroutingprotocolformobileadhocnetworks.DefenseTransformationand NetCentricSystems2008,EditedbySureshR.,ProceedingsoftheSPIE,6981,69810O 69810O10. Carvalho,M.,Granados,A.,Naqvi,W.,Brothers,A.,Hanna,J.&Turck,K.(2008).A crosslayercommunicationssubstratefortacticalinformationmanagementsystems. MILCOM08:ProceedingsoftheIEEEMilitaryCommunicationsConference,17.

106CARVALHOETAL.
Carvalho,M.,Ceccio,P.,Hanna,P.,Suri,N.,Uszok,A.,Bradshaw,J.,&Sinclair,A. (2009).Enablinginformationmanagementsystemsintacticalnetworkenvironments. SPIEDefense,SecurityandSensingConference,DefenseTransformationandNetCentric Systems. Jacquet,P.,Pascale,M.,Laouiti,A.,Viennot,L.,Clausen,T.,&Adjih,C.(2001). MulticastOptimizedLinkStateRouting.URL:http://tools.ietf.org/html/draftjacquet olsrmolsr00 Kunniyur,S.,&Srikant,R.(2000).Endtoendcongestioncontrolschemes:Utility functions,randomlossesandECNmarks.ProceedingsoftheIEEEINFOCOM. Loyall,J.,Gillen,M.,Bunch,L.,Carvalho,M.,Edmondson,J.,Schmidt,D.,Corman,D., Martignoni,A.,&Sinclair,A.(2009).QoSenableddisseminationofmanagedinformation objectsinapublishsubscribequeryinformationbroker.SPIEDefense,Securityand SensingConference,DefenseTransformationandNetCentricSystems. Mirhakkak,M.,SchultH.,&Thomson,D.(2001).Dynamicbandwidthmanagement andadaptiveapplicationsforavariablebandwidthwirelessenvironment.IEEEJournalon SelectedAreasinCommunications,19(10),19841997. Santivanez,C.,&Ramanathan,R.(2001).HazySightedLinkState(HSLS)Routing:A scalablelinkstatealgorithm.BBNTechnicalMemorandumNo.1301.URL:http:// www.ir.bbn.com/documents/techmemos/TM1301.pdf Sivakumar,R.,Das,B.,&Bharghavan,V.(1998).Spineroutinginadhocnetworks. ACMClusterComputingJournal,1(2),237248. Sivakumar,R.,Sinha,P.,&Bharghavan,V.(1999).CEDAR:Acoreextraction distributedadhocroutingalgorithm.INFOCOM99:Proceedingsofthe18thAnnualJoint ConferenceoftheIEEEComputerandCommunicationsSocieties,17,202209. Tromp,E.(2006).OLSRMulticastForwardingplugin.URL:http://olsr bmf.sourceforge.net Williams,B.,&Camp,T.(2002).Comparisonofbroadcastingtechniquesformobile adhocnetworks.MOBIHOC02:Proceedingsofthe3rdACMInternationalSymposiumon MobileAdHocNetworkingandComputing,Lausanne,Switzerland,194205. Wu,J.,&Li,H.(1999).Oncalculatingconnecteddominatingsetforefficientrouting inadhocwirelessnetworks.DIALM'99:Proceedingsofthe3rdinternationalworkshopon Discretealgorithmsandmethodsformobilecomputingandcommunications,Seattle, Washington,714.

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T A N D L A U R E L A L L E N D E R

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T AND LAUREL ALLENDER

This book, produced by the Partners of the Army Research Laboratory Advanced Decision Architectures Collaborative Technology Alliance, also serves as the Final Report for DAAD19-01-2-0009.

SECTIONI.COLLECTING,PROCESSING,ANDDISTRIBUTINGBATTLEFIELD INFORMATION SECTIONI COLLECTING,PROCESSING,ANDDISTRIBUTINGBATTLEFIELD INFORMATION

14SECTIONI

Chapter

6. HOWGOODISTHATNEWSOFTWARETOOL?THEMATHEMATICAL MODELINGOFPERFORMANCEMETRICS RobertR.Hoffman,Ph.D. InstituteforHumanandMachineCognition,Pensacola,FL MorrisMarx,Ph.D. InstituteforHumanandMachineCognitionandUniversityofWest Florida,Pensacola,FL RaidAmin,Ph.D. UniversityofWestFlorida,Pensacola,FL PatriciaL.McDermott AlionScienceandTechnology,Boulder,CO

INTRODUCTION Thischapterpresentsamethodforevaluatingperformanceusing cognitivelyinspiredAdvancedDecisionArchitecturesanddecisionsup porttools.Themethodcanaidevaluatorsattestcentersandcognitive scientistswhoadoptasystemslevelapproachtoevaluation.Thewarf ightercanbenefitbyreceivingtoolsthatbettersupportcognitivefunc tioningandsituationawareness.Inparticular,weareinterestedherein measuringthelearnabilityofnewsoftwaresupportedworkprocesses andtheresilienceoftheworktodisruption. REVIEW:WHYASYSTEMSLEVELANALYSIS? Whennewtechnologyisintroduced,itrepresentshypothesesabout howtheworkisexpectedtochange:thatperformancewillbemoreeffi cientanddecisionswillbeimproved(Dekker,NyceandHoffman,2003; Woods,1998).Newsoftwaretoolstypicallycomewithpromisesof 107

108HOFFMANETAL. heightenedfunctionalitiesthatwillexpandthecapabilitiesofwarfight ers.However,experienceshowsthatchangesintechnologyorwork methodscommonlyresultinnegativeeffects,suchasunintendedcas cadingfailure,workerfrustrationduetouserhostileaspectsoftheinter face,andsoforth(HollnagelandWoods,2006). Duringtheprocessofdevelopinginformationtechnology,many eventsoccurthatresultinuserhostiletechnologies.Forinstance,end userinvolvementmightbelimited,ortheusabilityevaluationmightrely onweakmethods.Theresultcanbeasoftwaresystemorinterfacethat seemgoodfromthedesigners'perspectivebutinthefinalanalysisare userhostile,andforceuserstocreateanynumberofpostimplementa tionklugesandworkarounds(KoopmanandHoffman,2003;Laplante, HoffmanandKlein,2007). Thenotoriousfrustrationsandcostlyfailurestriggeredbysoftware interventionshaveledtoasignificantconcernwithevaluationinthe softwareengineeringandhumanfactorscommunities(e.g.,Clegg,2000; Grudin,1992;Neville,etal.,2007;NormanandKuras,2004;O'Neill, 2007;RossonandCarroll,2002;Schaffer,2004).Softwareengineersare enjoinedtoassurethattheirgraphicalinterfacesareeasytolearn,effi cienttouse,easytoremember,leadtolowerrorrate,andaresubjec tivelypleasing.Suchfactorsarecontrastedwithtraditional considerationsofcost,reliability,efficiency,andsoforth. TheMetricsProblem:AnExampleScenario Supposethatanemergencyresponseteamneedsbettersoftwareto supportthemanagementandcoordinationofactivitiesandtheutiliza tionofresources.Aprototypeiscreatedandthenevaluatedbyagroup ofexperiencedemergencyresponders.Onegroupofparticipants receivessomeinitialpracticetrialsinordertolearnaboutthetask(the displays,thecommands,andtheoperationalproceduresofthenew technology).Usingheadphones,thegroupsimulatesa"distributed team"andconductsanemergencyresponsemanagementoperation basedonasimulatedscenarioderivedfromcasestudies.Acomparison groupdoesthesame,butinthiscasealloftheparticipantsaresituated inasingleroomandcancommunicateopenlyanddirectlywitheach other,forminga"colocated"team. Throughsuchanexperiment,onecouldevaluatehypothesesabouta variableofthecognitivework(distributedversuscolocated).Measures couldbemadeofsuchthingsastimetofirstresponseandnumberofcit izensevacuatedperhour.Onecouldprobetheemergencyresponders'

MathematicalModelingofPerformanceMetrics109 momenttomomentsharedsituationalawareness,evaluatepatternsin theteammembers'communicationswithoneanother,andtakeperiodic ratingsofmentalworkload(Endsley,1995;HancockandMeshkati,1988; Moray,1977). Nowsupposethattheuseofthesoftwareresultsinworkthatseems tobeimprovedrelativetothecurrentcapabilities,butonlyfortheco locatedteam.Isthisoutcomebecauseofsomethinghavingtodowith colocation,orisitsomethingthatisattributabletothenewtechnology? Itcouldbeeither,both,orneither.Itmightalsobebecausetheteamthat wascolocatedsimplyhappenedtolearnhowtousethetechnologybet terthanthedistributedteaminthefirstplace. Toresolvethis,wecanimaginescoresofcontrolledfactorialexperi mentsinthetraditionoflaboratoryresearch.Butbythetimetherele vantfactorshavebeencontrolled,keyvariablesisolated,andeffectsizes estimated,thetooldesignhasbeenchangedandreevaluated,etc.,the cognitiveworkwillalmostcertainlyhavepassedontootherincarna tions. Thus,weneedtoescapethefundamentaldisconnectbetweenthe timeframeforlaboratoryexperimentationandthetimeframeforeffec tivechangeinbattlefieldoperations.Thechallengeistocreateafast trackfortheevaluationofperformancesupporttoolsthatdoesnotpre cludecontrolledexperimentationforthestudyofthevariablesofcogni tivework. NEWCONTRIBUTION Thecognitiveworkofwarfightersiscomplex,demanding,andcon stantlychanging,aslegacytechnologiesandworkmethodsmixwithnew ones,addressingbotholdandemergingproblemsandchallenges.New softwaretoolsforthewarfighterneedtocomewithempiricalproofthat theyareusable,useful,andunderstandable.Howcanwecreatemetrics toinsurethatnewsoftwaretoolsaregood?ThisARLADAtaskhasgener atedasimplemethodfortheevaluationofsoftwarepriortoitsfielding, amethodthatmeasuresthelearnabilityandresilienceoftheworkthat issupportedbyanewsoftwaretool. Wepresentaframeworkforcreatingafasttrackfortheevaluation ofthetechnologythatshapestheworkmethods.Thegeneralapproach involvesevaluatingthenumberoftrialsittakesfortheusertoreach somecriterionlevelofperformance.Fromsuchdata,wederiveascale forthelearnabilityofworkmethods.Weshowhowtomathematically modelsuchdata,andwederivemethodsforestimatinglikelihoods(e.g.,

110HOFFMANETAL. trialstocriterionforbestandworstperformingparticipants).Themeth odologyalsoincludesameasureoftheresilienceoftheworksystem whenitisdisruptedorwhenitisforcedtoadapttochangingorunex pectedcircumstances. MeasurementFoundation Theprincipalperformancemeasureusedintheevaluationofper formanceisoftenareflectionofefficiency(numberofprimarytaskgoals achievedperunittime).Itcouldalsobeameasureofavoidance(i.e., numberoferrorspertrial),oritcouldrepresentaperformancethreshold thatplotsthetradeoffbetweendecreasingcycletimeandincreasing error.Thechoiceofaprincipalperformancemeasuredependsonthe domain,thespecifictasks,andotherfeaturesofthetasks.Aconceptual genericdefinitionofaprincipalperformancemeasureisthenumberof principaltaskgoalssuccessfullyaccomplishedperunittimeatwork.For example,performanceusinganewinterfaceforthecontrolof unmannedaerialvehiclesmightinvolveaprincipalperformancemeasure forthesensorpayloadoperatorof"numberoftargetsphotographedper mission"(seeHancock,etal.,2005). Aprincipalperformancemeasurecanbeusedtoformthecriterion fortraining. CriterionforTraining Inmuchoftheresearchonhumanperformance,anydatathatmight becollectedduringtheinitialinstructionandtaskpracticetrialswould notbeofanyparticularinterest,withregardtothemajorhypotheses beingstudiedinthemaintrials(anexperimentinvolvingcontrolledand manipulatedvariables).Butfortheevaluationofthelearnabilityofatask method,practicetrialsareaneglectedresource,andanumberofmea surestakenduringpracticetrialscouldbeinformative. Ratherthanhaveallparticipantsrunthroughafixednumberoftrain ingtrialspriortothemaintesttrials,eachparticipantcouldberun throughhowevermanytrialsittakestoachievesomeprespecifiedlevel ofproficiency.Practicecaninvolvetrainingtosomecriterionlevelofper formance,howevermanytrialsthatmighttake,sothattheparticipants areequatedfordegreeoforiginalbaselineachievement.Thisstrategy servestoaddressthepossibleconfoundintheinterpretationofmain effectsandinteractionsoftheindependentvariablesthatareformative ofthemaintesttrials(seePoulton,1965).

MathematicalModelingofPerformanceMetrics111 Anyvaluecanbeusedtosetacriterion,andwhetheritisaliberalor aconservativeonewilldependuponthesystemathandandthenature oftheindividualswhoareexpectedtooperateit.Fortaskshavinghistor icalperformanceprecedents,thecriterionmightbebasedonarchived data,traditionalbaselinesofperformance,legacytrainingstandards,or otherpriorempiricalsources.Fortaskshavingnolegacycounterpart, andhencenopreviousbaselinedata,acriterioncanbesetbasedon informedjudgmentandcanbeallowedtomatureasevaluationsarecon ducted.Ifthetrainingcriterionissettoolow(e.g.,65%correctperfor mance)itcanbeaconcernthatthemaintesttrialswouldthenbe conductedusingundertrainedparticipants.Conversely,ifitissettoo high(85%orhigher)theremaybeceilingeffects(i.e.,performanceon maintesttrialscanonlygodown).Clearly,itisimportanttobewaryof ceilingandflooreffects(e.g.,iftheparticipantsaretrainedtoperfect performancethereisnoroomleftforimprovement).Ideally,onehasa measurementscalethatsignalswhensucheffectsmaybeinoperation. Ifafterthemaintesttrialsthereisahiatus(perhapsoneortwo days),andparticipantsarethenbroughtbackforanotherpartdayseries oftrials,onecanmeasure"trialstoreachievecriterion."Thiscanbe takentoreflectthestrengthoftheoriginallearning(seeWoodworth, 1938,Ch.2).Ingeneral,trialstocriterionshowsthat"theparticipants gotit,"andtrialstoreachievecriterion(thesavings)showsthat"the participantsretainedit."Forinstance,ifittakesfewertrialstoreachieve thecriterionlevelofperformancefollowingadelayaftertheinitialtrain ingandpractice,thenthetaskworkwasbetterlearnedduringthatorigi naltrainingandtesting,andbydefinitionwasmorelearnable. RangeStatistics Inthestandardviewofhypothesistesting,realworldvariabilitymust berestrictedinsomefashionandtosomeextent,eitherbybeingcon trolled,manipulated,orheldconstant.However,allofthevariabilityof theworldisineffectwhenthesoftwaresystemsaretobeusedincon ductingtheactualwork.Intelligence,motivation,alertness,problemdif ficulty,andmanyothervariablesareinplaywhenanynewtechnologyis beingusedinachangedworksystem.Table6.1expressestheextremes.

112HOFFMANETAL. Table6.1Someofthevariabilitythataffectsperformanceduring softwareevaluation Bestperform ing participant CAPABILITY Highintelli gence Highlyprofi cient(expert) READINESS Highalertness Highprepared ness Highcapacity forrecognition primeddeci sionmaking Lowarousal Insufficient experience Hasaheadache MOTIVATION Highintrinsic motivation Achieveratti tude

Worstperform ing participant

Inadequate training Lowintelli gence

Lowmotivation Proceduralist attitude

Ameasureofthelearnabilityofaworkmethodistheinverseofthe midrangefortrialstocriteriondata.LettingBstandfortrialstocrite rion(ortrialstoreachievecriterion)forthebestperformingparticipant andWstandfortrialstocriterion(ortrialstoreachievecriterion)for theworstperformingparticipant,themidrangeis(B+W)/2.Inthiscase ofonlytwonumbers,themidrangeisaformofaverage.Wearenot interestedinthemidrangebecauseitmaycarrywithitsomeofthe propertiesofthemathematicalaverage,asinterestingandusefulas thosepropertiesmightbe.Rather,weareinterestedinitbecauseit expressesvariabilityandsuggestsameasurementscale. Theparticularfunctionofthemidrangethatisofinterestisthe inverseofthemidrange,anewstatisticthatwedenotewiththesymbol . i (pronounced"ibar").Theinversemidrangeequals -------------------------(B + W) 2 Thus, i =2/(B+W).If i islarge,thatis,closerto1.00,onecancon cludethattheworkmethodiseasilylearnedandalsothatitisrelatively insensitivetothevariabilityoftheworld,includingindividualdifferences, scenario(ortrials)differences,etc.If i issmall,thatis,closertozero, onecanconcludethattheworkmethodisofrelativelylowlearnability andmayalsobeinfluenced(negatively)morebythevariabilityofthe world.ThisscaleisillustratedinTable6.2.Thiscanbeausefulmeasure ofthelearnabilityofaworkmethod,especiallybecauseitcanbespeci fiedindependentlyofthenatureoftheworkmethod,theoperational scenarios,etc.

MathematicalModelingofPerformanceMetrics113

Table6.2Aninterpretationoftheibarscaleforthelearnabilityofa workmethod

i SCALERANGE

EXAMPLE VALUES (B,W) i (1,1,)1.00 (1,2)0.66

DESIRED DISCRIMINA
TION

between

RangeofTrivialCognitiveWork

1.00to0.66 suggeststhat thecognitive workmaybe trivial.On the assumption thatitisnot,

>.66

RangeofNontrivialCognitiveWork

(1,3)0.50

indicates thatthe performance criterion needstobe raised. Edgeofthe range. Criterion maystillbe settoolow.

114HOFFMANETAL. Rangeof(re)learnability (1,4)0.40 (2,3)0.40 (2,4)0.33 (2,5)0.29 (1,6)0.29 (2,5)0.26 (3,5)0.25 (3,6)0.22 (4,6)0.20 (2,9)0.18 (3,8)0.18 (5,7)0.16 (4,9)0.15
Fine discrimina bilityis desired.

Rangeof Stretch

Finest discrimina biltyis desired.

AsTable6.2shows,the i scaleisreallythreescalesinone.The RangeofTrivialCognitiveWorkissonamedbecauseitshowsthatthe i scaleprovidesareadymademechanismsignalinganeedforfinetuning whenthetrainingcriterionhasbeensettoolow,orforselectingtraining trialscases(orproblemtasks)ofanappropriatedegreeofdifficultyto mitigateceilingeffects.IntheRangeofStretch,thecognitiveworkmight beextremelydifficult,theworkmethodmightbeverylowinlearnability, thecriterionhasbeensettoohigh,orsomecombinationofthesemaybe thecase.Inthecaseofextremelydifficultcognitivework,differencesin i attheseconddecimalplacemightbemeaningful.Anexamplemight behelicoptertraining,wheretraineesreceivehoursofpracticeatthe taskofhoveringahelicopter,takinganaverageofabout20hoursto receiveapprovaltoattemptsoloflight(StillandTemme,2006).Insuch cases,valuesof i of0.05andsmallerwouldbeconceivableanddiffer encesbetween i valuesattheseconddecimalplacemightbemeaning ful. ResilienceMeasure Resilienceistheabilitytorecognizeandadapttounanticipatedper turbationsthatmightstretchtheworkers'competenceanddemanda shiftofprocessesandstrategies(Hollnagel,etal.,2006).Wecanadapta methodusedinclinicaltrials.Onceitisseeninclinicaltrialsthatsome

MathematicalModelingofPerformanceMetrics115 treatmentiseffective,onecanceasetheexperimentaltrialsandstart givingthetreatmenteventothosewhohadpreviouslybeenreceiving eitherthecontrolortheplacebotreatment.Inourcase,oncetherehad accruedsufficientdatatowarrantconclusionsaboutthelearnabilityofa workmethod,therecouldbeasessionofresiliencetrials(ratherthanre trainingtrials)inwhichthesystemisstretched.Thiscanbeperformed inavarietyofways.Voiceloopstooneteammembermightbedelayed (simulatingcommunicationdelaysduetobandwidthissues).Onemight simulateacommunicationloss,oralossofteamfunctionality.Perfor mancecouldthenbeevaluatedbytrialstoreachievecriterion.One assumessomereasonableleveloflearnabilitybutnowinterpretsthe rangeasareflectionoftheresilienceoftheworksystem,andlikewise interprets i asascaleofresilience. TeamPerformanceMeasure Althoughateamiscertainlymorethanalinearaggregationofindi viduals,withslightreinterpretation,themeasuresandscalethatwehave describedcanbeappliedintheanalysisofworkmethodsandtechnolo giesforteamsandteamcognitivework.Forinstance,ratherthanevalu atingtrialstoreachievecriteriononthepartofthebest(andworst) performingparticipants,onecanevaluate i forthebest(andworst)per formingteam.Measurescanbeoflearnabilityandresiliencewith respecttotheteamcognitivework.Alsoaswiththesimplercase,the rangestatistics(theBandWnumbersfortrialstoreachievecriterion) arepulledoutfromthetrainingandretrainingtrials,toevaluatehypoth esesconcerningthelearnabilityorgoodnessofthecognitiveworkwhile allofthemainexperimentdatacanbeusedintheevaluationofhypoth esesconcerningthecognitivework(e.g.,theeffectofsomevariableon teamsensemaking). MathematicalModeling Countsoftrialstoachieve(orreachieve)criterionarehighlyunlikely tohaveasymmetricdistribution(seeBook,1908).Measuresoftrialsto criterionareaninstanceofaprocesswherevaluesarebeingconstrained bysomesortofstoppingrule.Toconcretizethis,onewouldexpectrela tivelyfewparticipantstoachievecriteriononthefirst,second,oreven perhapsthirdtrial.However,onewouldexpecttoseemanyparticipants achievecriterionafterjustafewtrials.Asmallerpercentageofpartici

116HOFFMANETAL. pantswouldtakemoretrialstoachieve(orreachieve)criterion.Sucha distributiontypicallywouldbehighlyskewedbuthaveafattail,which characterizesdistributionssuchasthenegativebinomial(seeFigure6.1). Forcasessuchasthese,anorderstatistic(rangeormedian)ispreferred toanaveragebecausewearedealingwithdistributionswheretheaver agevaluetendstobemisleadingandunrepresentative(i.e.,thereisa considerabledifferencebetweenthemeanandthemedian)(Newelland Hancock,1984).

Figure6.1 Astylizeddistributionfortrialstoreachieveorreachievecriterion Inclassicalstatistics,whereoneassumestheunderlyingdataare normalorapproximatelysowithvariance 2,thenthestatisticusedfor inference X hasvariance / n,wherenisthesamplesizeallnice andneat.Evenifoneassumesthatthisdistributionisnegativebinomial, itisnotimmediatelyclearwhataclosedformexpressionorapproxima tionforthevarianceof2/(B+W)wouldbe. i involvesthemaximumandminimumvaluesandisthereforean orderstatistic,aclassthatincludesthemedianandtherankorder.The orderstatisticapproachofnonparametricinferenceistofindcumulative
2

probabilitydistributions.Thebasicdatumunderlyingthe i measureis thenumberoftrialsforeachindividual(orteam)toachievethedesig

MathematicalModelingofPerformanceMetrics117 natedcriterion.Asetofsuchdataisassumedtobearandomsample, X 1, , X n ,thatis,asetofnindependentdiscreterandomvariables,each possessingthesameprobabilitydistribution.Wedenotetheprobability densityfunctionofthe X iby p x ( k ) .Since2/(B+W)isaonetoone transformationofB+W,itsufficestofindtheprobabilitydistributionfor B+W.TheprobabilitydistributionofB+Wiscompletelydeterminedby

p x ( k ) .Threeclassesofdistributionswouldbepreferredformodeling
rangestatisticsmanifestingarapidrisetothemode,withalongtail(see Figure6.1)(LarsenandMarx,2006).ThesearedescribedinTable6.3. vFromtheprobabilitydensityfunctions,onecanconductstatistical evaluationssuchasthedeterminationofconfidenceintervals,aswellas estimatethelikelihoodthatanygivenparticipant(orteam)would achievecriteriononanygiventrial.Thisisillustratedinthescreenshotof theibarModelingTool,showninFigure6.2. Table6.3Equationsformodelingrangestatistics NegativeBinomial(ProbabilityDensityFunction)

- p ( 1 p ) ,0<p<1,0<r,k=0,1,2,... px(k;r,p)= ------------------Whatisit? Itisadiscrete distributionhavingtwo parameters:p= probabilityofsuccess; r=numberofsuccesses beforeexperimentis terminated Whatsitgoodfor? Describesdistributions thathavehighskew andfattails. Whatsitsweakness? Assumespisconstant acrosstrials.

(r + k) k! ( r )

ContinuousLogisticFunction(CumulativeDistributionFunction)

1 - ,wherea>0andb>0 G ( x ) = ---------------------a bx 1+e

Whatisit? Afamilyofdistributions thatmodelthe probabilityofsome eventthatis determinedbyoneor moreparameters. Whatsitgoodfor? Assumeslearning acrosstrials.ModelsS curvegrowth.Precise modelscanbederived fromsmallnumbersof smallnumbers. Whatsitsweakness? Itcandampenthe effectofoutliers.The scenariosforlearning arenontrivial.

118HOFFMANETAL. CONDITIONALPROBABILITIES(PROBABILITYDENSITYFUNCTION) Theprobabilitythatachievingcriterionoccursonthekthtrial p(k)=[1q(1)][1q(2)]...[1q(k1)]q(k),k=2,3,... q(k)=theconditionalprobabilityofsuccessonthekthtrial,given failuresontrials1,2,...,k1;q(1)=probabilityofsuccessonfirsttrial

Whatisit? Describesthe probabilityofsuccess giventhatsuccessive failureshaveoccurred. Whatsitgoodfor? Assumeslearning acrosstrials. Canfittodistributions thathavehighskew. Whatsitsweakness? Requiresassumptions onhowfailedtrials affecttheprobabilityof success.

Figure6.2 AscreenshotoftheibarModelingTool TheexampledepictedinFigure6.2showsthehistobarsforrelative frequencies(yaxis)oftrialstocriterion(xaxis).Byusingrelativefre quencies,theyaxisshowsprobabilitiesthatcanbedirectlyreadoffof thegraph,andusedtodeterminecumulativeprobabilities.Theexample isbasedondatathatshowboththefastrampandthefattail.Thelogistic

MathematicalModelingofPerformanceMetrics119 growthfunctionseemstobestcapturetheramp(greenline)whereas thenegativebinomialseemstobestcapturethetail(brownline).Using theslidersthatdeterminetheparametersofeachofthefunctions,one canactivelyexplorethemodelingspaces. IMPLICATIONSFORFUTUREARMYWORK: VALUEADDEDFORTHEWARFIGHTERINTRUSTINGTHETECHNOLOGY Theexperimentaldesignandmeasurementmethodologydeveloped underthisARLADAtaskprovidessoftwaredevelopersandcognitivesci entistsasimpleandrobustmethodfordeterminingwhethernewsoft waretoolsarelearnable.Useofthisapproachduringthesoftware developmentstagewouldassurethewarfighterthatnewsoftwareis proventobeusable,usefulandunderstandable.Themeasurementof trialstoachieve(andreachieve)criterion,andderivedrangestatistics, permitstheevaluationofthelearnabilityoftheworkmethods.ibar modelingisasuiteofequationsandassociatedjointcumulativedistribu tionsfunctionsthatenableresearcherstomodel"smallnumbersofsmall numbers"inawaythatpermitstheordinarycourseofcontrolledexperi mentation.Atthesametime,italsoprovidesafasttrackintotheanaly sisofthegoodnessofnewworkmethodsandtheassociatedsoftware toolsforconductingworkincomplexdecisionmakingenvironments. Asthemilitarystechnologicalsophisticationandcomplexitygrow, wemustpossessvalid,reliable,andmeaningfulmeasuresofsystemlevel performancetocreateascienceofcomplexandnetcentricsystems.The formalpairwiseorbaselinecrosscomparisonofanycomplextechnology mighthavebeenadesiredvisionyearsago,butthatvisionisreceding duetotherapidpaceoftechnologyandtheconsequencethatthebat tlespaceisaconstantlymovingtarget.Weneedinnovativeandflexible approachestothequestionofassessment.Ifwearetomakeconsistent progress,andifwearetobeabletodocumentandmeasurethis progress,wenowrequireaquantumimprovementinthewayweassess suchcomplexities. Whatweofferhereisareadytouseandspecificmethodologythat isnotaclosedendsolution.Furthermathematicalmodelingisunder way,usingdatasetsgeneratedintheARLADAefforts.Thecharacterand capacitiesoflargescaleinteractinghumanmachinesystemsremain openforexplorationinreferencetothecognitiveworksystemsinwhich weareinterested.

120HOFFMANETAL. REFERENCES
Book,W.F.(1908).Thepsychologyofskillwithspecialreferencetoitsacquisitionin typewriting.StudiesinPsychology,Vol1.Missoula,MT:UniversityofMontanaPress. Clegg,C.W.(2000).Sociotechnicalprinciplesforsystemdesign.AppliedErgonomics, 31,463477. Dekker,S.W.A.,Nyce,J.M.andHoffman,R.R.(MarchApril2003).Fromcontextual inquirytodesignablefutures:Whatdoweneedtogetthere?IEEEIntelligentSystems, pp.7477. Endsley,M.R.(1995).Measurementofsituationawarenessindynamicsystems. HumanFactors,37,6584. Grudin,J.(1992).Utilityandusability:Researchissuesanddevelopmentconcepts. InteractingwithComputers,4,209217. Hancock,P.A.andMeshkati,N.(Eds.).(1988).HumanMentalWorkload. Amsterdam:NorthHolland. Hancock,P.A.,Pepe,A.A.andMurphy,L.L.(2005,Winter).Hedonomics:Thepower ofpositiveandpleasurableergonomics.ErgonomicsinDesign,pp.814. Hollnagel,E.andWoods,D.D.(2006).Jointcognitivesystems:Foundationsof cognitivesystemsengineering.BocaRaton,FL:TaylorandFrancis. Hollnagel,E.,Woods,D.D.andLeveson,N.(Eds.)(2006).Resilienceengineering: Conceptsandprecepts.Hampshire,England:Ashgate. Koopman,P.andHoffman,R.R.,(November/December2003).WorkArounds, MakeWork,andKludges.IEEE:IntelligentSystems,pp.7075. Laplante,P.,Hoffman,R.andKlein,G.(2007,January/February).Antipatternsinthe creationofintelligentsystems.IEEEIntelligentSystems,pp.9195. Larsen,R.J.andMarx,M.L.(2001).Anintroductiontomathematicalstatisticsand itsapplications(3rd.ed.).EnglewoodCliffs,NJ:PrenticeHall. Moray,N.(Ed.)(1977).Mentalworkload:Itstheoryandmeasurement.NewYork: PlenumPress. Neville,K.,Hoffman,R.R.,Linde,C.,Elm,W.C.andFowlkes,J.(2007,January/ February).Theprocurementwoesrevisited.IEEEIntelligentSystems,pp.7275. Newell,K.M.andHancock,P.A.(1984).Forgottenmoments:Skewnessandkurtosis areinfluentialfactorsininferencesextrapolatedfromresponsedistributions.Journalof MotorBehavior,16,320335. Norman,D.O.andKuras,M.L.(2004)."EngineeringComplexSystems."Technical ReportNo.040043.MITRECorporation,Bedford,MA. O'Neill,M.J.(2007).Measuringworkplaceperformance(2nd.ed.).NewYork:Taylor andFrancis. Poulton,E.C.(1965).Onincreasingthesensitivityofmeasuresofperformance. Ergonomics,8,6976. Rosson,M.B.andCarroll,J.M.(2002).Usabilityengineering:Scenariobased developmentofhumancomputerinteraction.SanFrancisco,CA:MorganKaufman. Schaffer,E.(2004).Institutionalizationofusability.Boston:AddisonWesley. Still,D.L.andTemme,L.A.(2006).Configuringdesktophelicoptersimulationfor research.AviationSpaceandEnvironmentalMedicine,77,323. Woods,D.D.(1998).Designsarehypothesesabouthowartifactsshapecognition andcollaboration.Ergonomics,41,168173. Woodworth,R.S.(1938).Experimentalpsychology.NewYork:HenryHoltand Company.

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T A N D L A U R E L A L L E N D E R

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T AND LAUREL ALLENDER

This book, produced by the Partners of the Army Research Laboratory Advanced Decision Architectures Collaborative Technology Alliance, also serves as the Final Report for DAAD19-01-2-0009.

SECTIONII.PRESENTINGBATTLEFIELDINFORMATIONTOWARFIGHTERS

SECTIONII PRESENTINGBATTLEFIELDINFORMATIONTOWARFIGHTERS

122SECTIONII

Chapter

7. ADVANCINGTHESTATEOFTHEARTINMILITARYCOMMAND& CONTROLINTERFACES:ECOLOGICALINTERFACEDESIGN

KEVINB.BENNETT,PH.D. WrightStateUniversity,Dayton,OH LAWRENCEG.SHATTUCK,PH.D. NavalPostgraduateSchool,Monterey,CA

INTRODUCTION

The discipline of human factors and ergonomics emerged as a direct result of problems encountered in military systems during World War II. Advances in technology (e.g., radar and sonar) created new capabilities and opportunities; it also created the potential for new kinds of failures (some subtle, some spectacular). Military contexts continue to be a proving ground for technological innovation. For example, a wide variety of new forms of data have become available for todays military decision makers (e.g., fuel and ammunition levels in individual vehicles, sensor data, sattelite and UAV imagery, etc.). Similarly, advances in graphical display and interface technology (e.g., high-resolution, bit-mapped screens) provide new options in the presentation of these data. Together, these new technological advances provide the potential to improve the efficiency and effectiveness of military operations.

123

124BENNETTANDSHATTUCK
Unfortunately, this potential is not being realized. Existing Army systems grossly under-utilize these display and interface resources. To be fair, it should be noted that it is not a problem that is germane to just the Army; the same criticisms can be leveled at many other work domains as well. The general question is how can we leverage the potential that is provided by todays hardware and software technology in the design of our socio-technical systems? More specifically, in the context of Army command and control: How do we go about designing interfaces that commanders will find both powerful and easy to use? This chapter reviews a research program that was conducted under the ARL ADA CTA program with these questions in mind. An ecological interface was designed to support mobile battalion-level commanders during tactical operations. The particulars of this interface, the framework which guided its development, and the implications for display and interface design in general will be discussed.

GRATUITOUSLYGRAPHICALUSERINTERFACES(GGUI)

The technological advances referred to earlier are capable of producing very fast and sophisticated graphics in the interface. In fact, the term graphical user interfaces (GUI) was coined early on to describe this capability. This term is certainly apropos. The problem is that, far too often, the graphical objects in these interfaces do NOT represent meaningful aspects of the underlying domain. The powerful graphics are wasted on vacuous eye candy while the meaningful information in the domain is portrayed via relatively ineffective alpha-numeric formats. The definition of the word gratuitous is uncalled for, lacking good reason, or unwarranted. Since this is a perfect description of how the powerful graphics technology is squandered in these interfaces, we propose the modified term of gratuitously graphical user interfaces (GGUI) to describe them. In the following section we will describe a representative example of a GGUI from the domain of military command and control.

ExperimentalFBCB2Interface Gettinginformationfromatableislikeextractingsunlightfrom acucumber.(Farquhar&Farquhar,1891).

EcologicalInterfaceDesignforMilitaryC2125
The overall goal of our ARL ADA CTA research program was to design graphical interfaces that provided decision support for mobile, battalion-level commanders during tactical operations. The Army interface that most closely matched this purpose at the onset of the research program (2000) was the Force XXI Battle Command Brigade and Below (FBCB2) interface. At that time the FBCB2 interface was just beginning to be placed in the field for selected Army tactical units. A recent visit to the Armys National Training Center suggests that this interface has become a staple in todays tactical operations centers. We developed experimental versions of this interface to replicate its visual appearance and selected functionality (e.g., Talcott, Bennett, Martinez, Shattuck, & Stansifer, 2007); the original version is illustrated in Figure 7.1. The FBCB2 interface will be analyzed with regard to three general types of representations that might be used in an interface: analogical, metaphorical, and alpha-numeric (Bennett & Flach, In Preparation).

Analogical. Analogical representations provide a one-to-one mapping between

an object in the interface and an object in the work domain. The primary analogical representation in the FBCB2 interface is the contour map: there is a one-to-one correspondence between changes in the elevation of the real-world battlefield terrain and changes in the density and orientation of the contours on the map.

Figure7.1 AnexperimentalversionoftheFBCB2interface.Reprintedwithpermis sionfromHumanFactors,Vol.49No.12007.Copyright2007bythe HumanFactorsandErgonomicsSociety.Allrightsreserved.

Metaphorical. Metaphorical representations lack this one-to-one

correspondence; the information that they provide is more symbolic in nature. Metaphorical representations found in the FBCB2 interface include the rectangular shapes that appear on the contour map. These symbols represent tactical units (e.g., a company) and convey information about its physical makeup (e.g., the ovals symbolically represent the treads of a tank and indicate their presence in the unit).

126BENNETTANDSHATTUCK Alphanumeric. Alpha-numeric representations are numbers and letters (e.g.,

the numbers and labels on both the contour map and the buttons found on the right hand side of the FBCB2 interface). The interpretation of this representational format is quite different from the previous two. In analogical and metaphorical representations there is a resemblance between the visual appearance of an object in the domain and an object in the interface. In contrast, the interpretation of an alpha-numeric representation depends totally upon a culturally-defined meaning: the mappings between objects in the interface and objects in the domain (i.e., numbers and letters) are completely arbitrary. The primary reason that the FBCB2 interface qualifies as a GGUI is due to the relative proportion of meaningful information that is presented using alphanumeric representations. For example, the status of friendly combat resources (e.g., tanks, Bradleys, fuel, ammunition, personnel) is important information. In the FBCB2 it appears as alpha-numeric text inside a window that covers the contour map, as illustrated in Figure 7.1 (right panel). This is but one example of how the meaningful information is presented alpha-numerically, rather than analogically. This is an unfortunate design choice. Consider the difference between two numbers that are represented as digital values. The size of the difference can be mentally calculated, but it cannot be seen directly in the representation. Now consider the same two numbers represented as bar graphs (an analogical representation). In this case the size of the difference can be seen directly. This is exactly what the quotation at the beginning of this section is alluding to (as cited in Wainer, 1981, p. 236). The difference between these two types of representations can have a large impact on performance. For example, Bennett and Walters (2001) compared performance between analogical and digital displays for a complicated process control task. The analogical display was clearly superior; difficulties with the alpha-numeric representations are summarized as follows (p. 430):

Incontrast,thedigitaldisplayimposedatrulyseveresetof constraints.Theroutetounderlyingmeaningwasmuchless direct:thedomainsemantics(relationships,properties,goals, andconstraints)werenotdirectlyvisible.Instead,the participantswereforcedtoderivethisinformationmentally, usingthedigitalvaluesinconjunctionwiththeirknowledge aboutthesystem(seeBennett&Flach,1992andBennettetal., 1997foramoredetaileddiscussionofsimilarconsiderations). Thus,theconstraintsintroducedbythedigitaldisplaymadeit muchmoredifficulttoassesssystemstate,todetermine

EcologicalInterfaceDesignforMilitaryC2127 appropriatecontrolinput,andtogaugetheappropriatenessof thesystemdynamics.Asaresult,performancesuffered.

Thus, the FBCB2 interface provides one example of a GGUI. The vast majority of the graphics are gratuitous: there are graphical tabs, buttons and colored boxes. However, most of the critical information in FBCB2 is represented alphanumerically. These representations are excellent for adding detail and clarification. However, information that is analogical in nature (e.g., combat resources) should be represented analogically in the interface. Otherwise the inherent power of graphics technology to support direct perception is not realized. In the next section we will describe a theoretical framework that can be used to develop and evaluate graphical user interfaces that are not gratuitous.

COGNITIVESYSTEMSENGINEERING(CSE)

The FBCB2 interface is not an isolated example, either in the Army or in general: good interfaces are surprisingly difficult to design and implement. Building a good interface is a difficult process that requires knowledge and multi-disciplinary expertise. Cognitive systems engineering provides a comprehensive framework for design and evaluation that can guide these efforts. The CSE approach maintains that interface and display design is ultimately a form of decision making/problem solving support for agents completing work in a domain. From this perspective there are three fundamental components: the domain (i.e., the work to be done), the cognitive agents (including both humans and machines) responsible for doing the work, and the interface (i.e., the displays and controls). These three components are illustrated in Figure 7.2. Each component contributes a set of independent, but mutually-interacting, set of constraints on the performance of the overall system. Each component will be described in greater detail.

128BENNETTANDSHATTUCK

Figure7.2 Threefundamentalcomponentsofasociotechnicalsystem DomainConstraints

An analysis and description of the underlying constraints in a domain (i.e., a work domain analysis) is absolutely essential; it is the starting point for interface design. The CSE approach provides analytical tools for discovering and modeling these work domains. Two primary tools are the abstraction (meansends relations) and the aggregation (part-whole relations) hierarchies (Rasmussen, 1979, 1983). Conducting a domain analysis using these tools is essentially the process of building a model of the domain in terms of the critical categories of information (e.g., physical, functional, and goal-related aspects) and the relationships between these categories.

AgentConstraints

The analytical tool for modeling human constraints is the SRK classification scheme developed by Rasmussen (1983). Skill-, rule-, and knowledge-based behaviors represent modes of behavior (or control) that are independent of the underlying psychological mechanisms.

Skillbasedbehavior. These are the high-capacity, sensory-motor activities that are executed without conscious control. Consider an agent driving a car. The road provides a reference signal that is perceived by the agent; the agent produces motor activity (e.g., turning the wheel) that minimizes deviation from this signal. The agent coordinates and adjusts his/her actions based on feedback obtained from the continuous, space-time signals that arise from (and are specific to) the combination of structure in the environment and observer action.

EcologicalInterfaceDesignforMilitaryC2129 Rulebasedbehavior. The agent has developed effective solutions and

associated actions based on prior experience or knowledge. The agent reacts to cues or signs that determine when a stored procedure should be executed. In the car-driving example, the agent is following directions to get to a destination. The route is monitored for key aspects of the environment (e.g., the street sign for Elm Street) that will trigger the associated behavior (e.g., turn right).

Knowledgebasedbehavior. The agent is faced with situations which have not

been encountered before and is essentially engaged in problem solving. The agent is forced to engage in model-based analysis to derive potential solutions. In the car-driving example this might occur when a familiar route is blocked (e.g., construction) and an alternative route must be derived (based on a map or internal knowledge of adjacent streets).

ECOLOGICALINTERFACEDESIGN(EID) Theaimofecologicalinterfacedesigncanbedescribedastrying tomaketheinterfacetransparent,thatis,tosupportdirect perceptiondirectlyattheleveloftheuser'sdiscretionarychoice, andtosupportthelevelofcognitivecontrolatwhichtheuser choosestoperform.Themappingacrossinterfaces,whichwill supporttheuser'sdynamicswitchingamonglevelsoffocusand control,mustsupportcontrolofmovements,acts,andplans simultaneously.Todothis,thedesignermustcreateavirtual ecologicalenvironmentthatmapstheintendedinvariantsofthe functionalsystemdesignontotheinterface.(Rasmussen& Vicente,1990,p.102,emphasisadded)

As previous sections have alluded to, the interface also contributes a set of constraints that have an impact on performance. There are two fundamental principles of effective interface design: direct perception and direct manipulation. The extent to which these two desirable characteristics are achieved in an interface will be determined by the quality of very specific sets of mappings between the interface, the domain and the agents (see Figure 7.2). A brief summary of each principle will be provided here; additional details are provided in Bennett and Flach (In Preparation).

130BENNETTANDSHATTUCK Directperception. In describing direct perception, Rasmussen et al. (1994, p. 129) observed that In Gibsons terms, the designer must create a virtual ecology, which maps the relational invariants of the work system onto the interface in such a way that the user can read the relevant affordances for actions. Successfully achieving direct perception involves 1) the incorporation of the categories of information that were identified in the work domain analysis, and 2) building representations of these constraints in the interface that allow that information to be obtained, or picked-up, easily by the agent. Directmanipulation. Norman (1986, p. 53) described direct manipulation as ... the qualitative feeling of control that can develop when one perceives that manipulation is directly operating upon the objects of concern to the user. The actions and the results occur instantaneously upon the same object.
When direct perception and direct manipulation have been achieved the interface will produce space-time signals that can be perceived and acted upon. More specifically, the displays and controls in an effective interface will maintain an intact perception-action loop thereby supporting skill-based behavior. This feature has an intuitive link to Gibsons work (e.g., Gibson, 1966). Effective displays and controls should also provide a rich set of signs for action as well. These cues will suggest the appropriate action to be taken to an expert agent, thereby supporting recognition-primed decision making (Klein, 1989) and rule-based behavior. Finally, to support performance during novel situations the displays should contain symbolic representations from the categories identified in the work domain analyses (i.e., the abstraction and aggregation hierarchies). These representations will serve as external models of the domain to support problem solving behavior (i.e., knowledge based behavior).

THERAPTORINTERFACE

The ecological interface that was designed from the CSE/EID perspective during the ARL ADA CTA will now be described. An overview illustration of the Representation Aiding Portrayal of Tactical Operations Resources (RAPTOR) interface is provided in Figure 7.3. Only a brief synopsis of this interface will be provided here. Please see Bennett, Posey, and Shattuck (2008) for more details on the domain work analyses and the rationale for each of the displays.

EcologicalInterfaceDesignforMilitaryC2131 FriendlyCombatResourcesDisplay

Friendly combat resources and their representation in the FBCB2 interface were discussed earlier. The corresponding RAPTOR display is illustrated in Figure 7.4. The primary format is analogical in nature: there is a bar graph for each combat resource. The horizontal extent of each bar graph varies dynamically as a function of the percentage of the associated resource. The color of each bar graph represents the categorical status of the combat resource. The static categorical boundaries are also represented analogically. Precision is provided via the alpha-numeric labels and digital values. The background color of the rectangular shape represents its overall combat readiness, as determined by the combat resource with the lowest category.

Figure7.3 OverviewofRAPTORinterface. ReprintedwithpermissionfromtheJournalofCognitiveEngineeringand DecisionMaking,Vol.2No.4Winter2008. Copyright2008bytheHumanFactorsandErgonomicsSociety. Allrightsreserved.

132BENNETTANDSHATTUCK

Figure7.4 Friendlycombatresourcesdisplay. ReprintedwithpermissionfromtheJournalofCognitiveEngineeringand DecisionMaking,Vol.2No.4Winter2008. Copyright2008bytheHumanFactorsandErgonomicsSociety. Allrightsreserved. EnemyCombatResourcesDisplay

Displays for enemy combat resources are also needed. There is, of course, a higher degree of uncertainty in these estimates, relative to friendly combat resources. Army intelligence officers indicated that the primary concern was the status of enemy combat vehicles (i.e., tanks and personnel carriers) in terms of three categories: 1) those that have been observed and verified as being alive (A) and dangerous, 2) those that have been engaged and disabled (D), and 3) and those that are templated (T): likely to be in the area of engagement, but not yet observed). The primary representation format for this information in RAPTOR is a horizontal contribution (or stacked) bar graph (see Figure 7.5). Each segment of a bar graph represents alive, templated, and disabled vehicles, respectively. They are also color-coded (bright red, dull red, gray, respectively). The analog graphics are also annotated with digital values that provide exact values of the number of vehicles in each category (and assorted other information). The bottom contribution bar graph represents tanks. The lack of red segments indicates that all tanks have been disabled (or that the unit had none initially).

EcologicalInterfaceDesignforMilitaryC2133

Figure7.5 Enemycombatresourcesdisplay. ReprintedwithpermissionfromtheJournalofCognitiveEngineeringand DecisionMaking,Vol.2No.4Winter2008. Copyright2008bytheHumanFactorsandErgonomicsSociety. Allrightsreserved.

ForceRatioDisplay

The domain analyses revealed that a critical consideration in tactical operations is force ratio: the relative amount of combat power that exists between two opposing forces at any point in time. A simplified estimate of force ratio was devised based on the relative number of friendly and enemy tanks and armored personnel carriers. The force ratio display is illustrated in Figure 7.6, on the right. The primary graphical format is the contribution bar graph. There are two of these, aligned on the left hand side with the origin of the axes; one is for friendly force equivalence (top) and one is for enemy force equivalence (bottom). The force ratio reflecting line is connected to these two bar graphs; changes in their horizontal extent push (or pull) the endpoint of the line, thereby changing its orientation and the value of force ratio that is determined by the displays geometrical properties.

134BENNETTANDSHATTUCK

Figure7.6 Forceratioandforceratiotrenddisplays. ReprintedwithpermissionfromtheJournalofCognitiveEngineeringand DecisionMaking,Vol.2No.4Winter2008. Copyright2008bytheHumanFactorsandErgonomicsSociety. Allrightsreserved. ForceRatioTrendDisplay

The force ratio trend display (left side of Figure 7.6) illustrates the actual and planned values of force ratio over time. The display is scaled using the laws of perspective geometry (e.g., Bennett, Payne, & Walters, 2005; Bennett & Zimmerman, 2001; Hansen, 1995). Trend lines for both actual values and planned values of force ratio are plotted on the display grids. The degree of spatial separation between the planned and actual trend lines visually specifies discrepancy from plan (an important consideration identified in the domain analyses).

SpatialSynchronizationMatrixDisplay

EcologicalInterfaceDesignforMilitaryC2135
The spatial synchronization matrix display illustrates a number of spatial constraints (see Figure 7.7) that are critical to land-based tactical operations. The primary component is a contour map providing an analog spatial metaphor of the physical characteristics of the battlefield terrain. Synchronization points (i.e., a physical location on the battlefield that a friendly unit must occupy at a particular point in time) are represented by labeled circles. The planned spatial route for each unit in the mission is represented by the activity segments (lines) that connect the synchronization points. Additional analog spatial information in the display includes transparent icons representing friendly unit locations, arcs representing primary weapons envelopes, obstacles, and enemy units and fortifications.

Figure7.7 Spatialsynchronizationmatrixdisplay. ReprintedwithpermissionfromtheJournalofCognitiveEngineeringand DecisionMaking,Vol.2No.4Winter2008. Copyright2008bytheHumanFactorsandErgonomicsSociety. Allrightsreserved. TemporalSynchronizationMatrixDisplay

136BENNETTANDSHATTUCK
There is also a need to coordinate the activities of the various units across time, which is supported by the temporal synchronization matrix display (see Figure 7.8). Time is represented in the X axis of the matrix, ranging from the initiation of the engagement (0) to a point three hours later in time (3). The various units involved in the tactical operation are represented along the Y axis of the matrix (e.g., Company B). A row in the matrix graphically illustrates the sequence of activities that are planned for each unit ( e.g., Breach North) and an analog representation of the amount of time that each activity should take (the horizontal size of the cell). In addition, temporal synchronization points (i.e., the points in time where there is a requirement to coordinate the activities of these units) are illustrated graphically by the thick gray lines that run vertically through the display. The present time in the engagement is illustrated by the thin vertical line.

Figure7.8 Temporalsynchronizationmatrixdisplay. ReprintedwithpermissionfromtheJournalofCognitiveEngineeringand DecisionMaking,Vol.2No.4Winter2008. Copyright2008bytheHumanFactorsandErgonomicsSociety. Allrightsreserved.

EcologicalInterfaceDesignforMilitaryC2137
Visual changes in the display indicate the status of various activities with regard to their associated temporal synchronization requirements. For example, in Figure 7.8, Team Cs activity of Breach South has been highlighted by a change of color (yellow in the actual display). This indicates that this unit is in danger of not completing the activity on time. Similarly, the next activity for the same unit (Move to Enemy) has also been highlighted (red in the actual display). This indicates that the unit will not complete the activity on time (e.g., the unit cannot travel fast enough to reach the destination by the designated time).

EVALUATIONS

The RAPTOR interface was evaluated continually throughout the entire time course of the ARL ADA CTA. Informal evaluations were obtained on a regular basis during two annual meetings. One of the meetings was attended by internal CTA researchers (including Army and civilian researchers working in government labs); the second was an exhibition format open to all Army personnel. In both cases invaluable feedback on the interface was obtained from researchers and practitioners who were very familiar with the domain. Several formal empirical evaluations were also completed. All of these studies incorporated realistic tactical scenarios that were developed in cooperation with experienced Army practitioners. All of these studies also used experienced Army officers as participants. The majority of the studies compared performance with the RAPTOR interface to experimental versions of existing Army interfaces. The primary evaluations will be described in chronological order.

DirectPerception

The first round of evaluation utilized controlled laboratory experiments that were designed to assess the effectiveness of selected displays and controls in the RAPTOR interface. For example, Talcott, Bennett, Martinez, Shattuck, and Stansifer (2007) evaluated the capability of the RAPTOR interface to support a critical activity: obtaining the status of friendly combat resources. A qualitative simulation of a traditional offensive engagement (i.e., force-on-force) was developed; the status of combat resources (e.g., ammunition) was summarized at three different points in time and at three different echelon levels (battalion, company, platoon). An experimental, baseline interface was designed to replicate the visual appearance and selected functionality of an existing Army interface (FBCB2, see Figure 7.1).

138BENNETTANDSHATTUCK
The participants were asked to report on the status of particular resources (e.g. the number of tanks) at particular echelon levels (e.g., company) using the two different interfaces. The accuracy and latency of their responses were recorded. Talcott et al. (2007, p. 131) summarize their results: The superior performance of the RAPTOR interface was present in all assessment categories (quantitative, categorical, and needs), dependent variables (accuracy, latency), and echelon levels (battalion, company, platoon). These results clearly indicate that the RAPTOR interface provided better support for obtaining friendly combat resources than did the FBCB2 interface. Similar experiments were conducted to isolate functional characteristics of the FBCB2 interface and to evaluate other RAPTOR displays (e.g., enemy combat resources, see Figure 7.5). The results of one of these evaluations are illustrated in Figure 7.9. In this study three variations of the FBCB2 interface were developed to identify specific features of the FBCB2 interface responsible for the poor performance. The basic findings of Talcott et al. (2007) were replicated: performance with the RAPTOR interface was found to be significantly better than performance with all versions of the FBCB2 interface.

Figure7.9 AccuracyandlatencymeansforFBCB2andRAPTORdisplays

EcologicalInterfaceDesignforMilitaryC2139 SituationAssessment;DecisionMaking

Subsequent empirical studies expanded the scope of evaluation, moving away from the study of simple information extraction and toward the study of more complicated activities that need to be supported in tactical operations. More varied and more realistic scenarios were developed to accomodate this change in focus. Most importantly, these scenarios were continuous (i.e., not discrete snapshots) and interactive (i.e., dynamically evolving, not canned). One study examined two versions of RAPTOR: an enhanced version and a baseline version (without the force ratio, force ratio trend, and review mode). Experienced Army officers assumed the role of battalion commander and viewed a dynamic, authentic scenario. The scenario was paused at six different points that coincided with critical events and the participants were required to complete two questions (Please verbally describe the situation as you understand it and What actions, if any, would you take at this time?). Participants who used the enhanced RAPTOR interface exhibited a greater tendency to produce references to plans and operations orders. Participants were also extremely enthusiastic about RAPTOR. One participant stated I could see that decision...the display made that decision abundantly clear to me... No more eloquent description of direct perception as a goal of interface design is possible. A second participant expressed the opinion that RAPTOR was Much more useful than some of the other interfaces Ive ... used ... in Iraq. A final capstone evaluation was recently completed (Hall, In Preparation). The RAPTOR interface and a baseline interface (modeled on the functionality of the FBCB2 interface) were evaluated. Sixteen experienced officers participated. One dynamic, interactive scenario was used during training (defensive) and two (offensive, counter-insurgency) were used during testing. Performance on the following dependent measures were obtained.

SituationReports(SITREP). Participants filled out 22-item status reports on the

current state of the engagement (e.g., physical locations, combat resources, historical activities of friendly and enemy elements).

CommandersCriticalInformationRequirements(CCIR). Participants monitored for (and then reported) sets of events that were designated as being particularly important to mission success (e.g., friendly activities, enemy activities, environment). DecisionPoints. Participants chose between alternative courses of action that
were appropriate under different sets of critical events.

140BENNETTANDSHATTUCK RequestsForInformation. The number of times that a participant accessed

information contained in the written operations order.

CSWAT. Participants reported their subjective workload.

The data analyses are on-going and the results must be considered preliminary. However, the general trend is quite clear. Performance with the RAPTOR interface was dramatically better than performance with the Baseline interface for all dependent measures. Consider the results for situation reports, as illustrated in Figure 7.10.

Figure7.10 SITREPperformanceforFBCB2andBaselinedisplays

Participants using the RAPTOR interface completed these reports in approximately half the time (means = 3.30 min. vs. 6.05 min.) and with approximately twice the accuracy (means = 98% vs. 54%) than those participants using the Baseline interface. Overall, RAPTOR participants produced significantly better performance on almost all dependent measures: better SITREP reports, more effective CCIR monitoring, less need to access the written operations orders, and lower perceived workload.

EcologicalInterfaceDesignforMilitaryC2141 SUMMARY

Cognitive systems engineering (Rasmussen, et al., 1994) and ecological interface design (Bennett & Flach, In Preparation; Rasmussen & Vicente, 1990) provided the theoretical framework that was used to guide the development of the RAPTOR interface. The goal in interface design is to bring the highly efficient perceptual-motor skills of the human to bear on the problem of humancomputer interaction. Providing representations that support direct perception is the first step, ensuring that the relational invariants of the work system are present in the interface. This, in turn, provides affordances: these invariants will suggest control actions that are appropriate, given the current context. Direct manipulation has been achieved when these control actions can be executed through the selection and manipulation of the objects that are present in the interface. Effective perception-action couplings need to be established: the agent should be able to use highly efficient perceptual-motor routines that allow effective synchronization with the space-time signals that are being presented in the interface. These ideas are embodied in the RAPTOR interface. In combination with domain expertise supplied over an extended period of time by Army coinvestigators, experienced officers, and laboratory researchers, a clearly superior ecological interface has been developed to provide decision support for tactical operations. There is a tremendous need for the injection of EID-inspired ideas, such as those embodied in the RAPTOR interface, into other military command and control interfaces. Military personnel are subjected to work conditions that can have a serious impact on their capability to perform in an effective manner. They experience sleep deprivation and extreme physical fatigue. They also experience high degrees of stress, where the stakes can involve life or death. It is under these conditions that the benefits of the CSE and EID approach (the leveraging of powerful perception-action skills during decision making and problem solving) are likely to be magnified.

REFERENCES
Bennett,K.B.,&Flach,J.M.(InPreparation).Display and Interface Design: Subtle Science, Exact Art.:Taylor&Francis. Bennett,K.B.,Payne,M.,&Walters,B.(2005).Anevaluationofa"timetunnel"' displayformatforthepresentationoftemporalinformation.Human Factors, 47(2),342 359.

142BENNETTANDSHATTUCK
Bennett,K.B.,Posey,S.M.,&Shattuck,L.G.(2008).Ecologicalinterfacedesignfor militarycommandandcontrol.Journal of Cognitive Engineering and Decision Making, 2(4),349385. Bennett,K.B.,&Walters,B.(2001).Configuraldisplaydesigntechniquesconsidered atmultiplelevelsofevaluation.Human Factors, 43(3),415434. Bennett,K.B.,&Zimmerman,J.(2001).Apreliminaryinvestigationofthetime tunnelsdisplaydesigntechnique.Displays, 22(5),183199. Farquhar,A.B.,&Farquhar,H.(1891).Economic and industrial delusions: A discourse of the case for protection.NewYork:Putnam. Gibson,J.J.(1966).The senses considered as perceptual systems.Boston,MA: HoughtonMifflin. Hall,D.S.(InPreparation).RAPTOR: An empirical evaluation of an ecological interface designed to increase user cognitive performance.NavalPostgraduateSchool, Monterey,CA. Hansen,J.P.(1995).Anexperimentalinvestigationofconfigural,digital,and temporalinformationonprocessdisplays.Human Factors, 37,539552. Klein,G.A.(1989).Recognitionprimeddecisions.Advances in Man-Machine Systems Research, 5,4792. Norman,D.A.(1986).Cognitiveengineering.InD.A.Norman&S.W.Draper(Eds.), User centered system design(pp.3161).Hillsdale,NJ:LawrenceEarlbaumAssociates. Rasmussen,J.(1979).On the structure of knowledge- A morphology of mental models in a man-machine context(No.M2192).Roskilde,Denmark:RisoNational Laboratory. Rasmussen,J.(1983).Skills,rules,andknowledge;signals,signs,andsymbols,and otherdistinctionsinhumanperformancemodels.IEEE Transactions on Systems, Man, and Cybernetics, SMC-13,257266. Rasmussen,J.,Pejtersen,A.M.,&Goodstein,L.P.(1994).Cognitive systems engineering.NewYork,NY:Wiley. Rasmussen,J.,&Vicente,K.J.(1990).Ecologicalinterfaces:Atechnological imperativeinhightechsystems?International Journal of Human-Computer Interaction, 2(2),93111. Talcott,C.P.,Bennett,K.B.,Martinez,S.G.,Shattuck,L.,&Stansifer,C.(2007). Perceptionactionicons:Aninterfacedesignstrategyforintermediatedomains.Human Factors, 49(1),120135. Wainer,H.(1981).Graphicaldataanalysis.Annual Review of Psychology, 32,191 241.

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T A N D L A U R E L A L L E N D E R

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T AND LAUREL ALLENDER

This book, produced by the Partners of the Army Research Laboratory Advanced Decision Architectures Collaborative Technology Alliance, also serves as the Final Report for DAAD19-01-2-0009.

SECTIONII.PRESENTINGBATTLEFIELDINFORMATIONTOWARFIGHTERS

SECTIONII PRESENTINGBATTLEFIELDINFORMATIONTOWARFIGHTERS

122SECTIONII

Chapter

8. COMMUNICATIONVIATHESKIN:THECHALLENGEOFTACTILEDISPLAYS

LYNETTEA.JONES,PH.D. DepartmentofMechanicalEngineering, MassachusettsInstituteofTechnology,Cambridge,MA

INTRODUCTION

Tactile communication systems represent a promising technology that can be used to present information to soldiers in a variety of contexts by utilizing a relatively underused sensory channel to convey information that is both private and discreet. Applications of this technology include delivering vibrotactile cues to soldiers to assist in navigation or threat location in the battlefield, providing tactile feedback to increase situational awareness in virtual environments used for training, and employing tactile cues to enhance the representation of information in multi-sensory displays used for planning and decision making. The opportunity to use the sense of touch in these diverse application domains arose as a result of tactile display technologies becoming more sophisticated due to the reduced power requirements of actuators used in the displays, and the option of wireless communication for mobile users (Jones & Sarter, 2008). This in turn made the displays less intrusive and thus more acceptable to users. Tactile displays have achieved success as communication systems in a variety of contexts in which it has been shown that vibrotactile cues delivered to the skin can provide spatial information about the environment to pilots (Rupert, 2000), or alert the operator of a vehicle of an impending collision (Spence & Ho, 2008). The displays are typically composed of a matrix of electromechanical actuators (known as tactors) that are mounted in a vest or waistband and are sequentially

143

144JONES
activated to provide information about a persons spatial orientation or the position or movement of a vehicle (McGraith et al., 2004). In most of these applications, the tactors are activated at a fixed frequency and amplitude and the number and location of the tactors simultaneously active is used to convey information. Tactile displays have also been employed in environments in which the visual and auditory communication channels are heavily taxed or in which visual displays are inappropriate because the operator is involved in other activities that require attention. In these latter situations, the sense of touch provides a communication channel that is direct, salient, and private. The goal of the research conducted for the Advanced Decision Architectures Collaborative Technology Alliance (CTA) was to design and build wirelesscontrolled wearable tactile displays that could be used to evaluate the contexts and conditions in which tactile communication facilitated soldier performance in the battlefield, in vehicle or robot control, and in information exchange during planning for future combat missions. The domains in which it was envisaged that the displays would be used were those that required that the hands were free for other activities (e.g. holding a weapon, driving a vehicle). The display was therefore designed so that it could be worn on the arms, legs, or torso. The research at MIT focused on five issues: Can tactile signals be used to provide spatial cues about the environment that are accurately localized? How does the location and configuration of the tactile display influence the ability of the user to identify tactile patterns? What is the maximum size of a tactile vocabulary that could be used for communication? Which characteristics of vibrotactile signals are optimal for generating a tactile vocabulary? Can a set of Army Hand and Arm Signals be translated into tactile signals that are accurately identified when the user is involved in concurrent tasks? Within this framework, a series of laboratory and field studies was conducted. The tactile displays fabricated at MIT were made available to other partners in the CTA and assistance was provided to help partners conduct experiments. Studies conducted at ARL that have used the MIT tactile display include those that have done the following: investigated the efficacy of tactile and multimodal alerts on decision making by Army Platoon Leaders (Krausman et al., 2005, 2007); analyzed the effectiveness of tactile cues in target search and localization tasks and when controlling robotic swarms (Hass, 2009); evaluated Soldiers abilities to interpret and respond to tactile cues while they navigated an Individual Movement Techniques (IMT) course (Redden et al., 2006); and measured the effects of tactile cues on target acquisition and workload of Commanders and Gunners and determined the detectability of vibrotactile cues while combat assault maneuvers were being performed (Krausman & White, 2006; White et al., in press). The MIT tactile displays have also been

TactileDisplays145
incorporated into multi-modal platforms developed by the University of Michigan, ArtisTech in the CTA test bed, and Alion MA&D for a robotics control environment. Finally, a comprehensive review of tactile displays was written in collaboration with Nadine Sarter at the University of Michigan (Jones & Sarter, 2008). The organization of this chapter is as follows. The first section provides a brief overview of the basic design features of tactile displays. The next three sections summarize the major findings of a series of experiments that evaluated the use of tactile displays for communicating spatial information, simple instructional cues, and arm and hand signals. The chapter concludes with a discussion of the implications of tactile communication systems for the military.

TACTILEDISPLAY ActuatorsandWirelessTactileControlUnit

The development of tactile displays entails research in a number of areas, each of which impacts the decisions made in related domains. The initial task is the selection and characterization of the actuators used in the display. Different types of actuators have been used to create tactile displays, with small electromechanical motors (Figure 8.1) being the most common due to their size, availability, and low power requirements (Jones et al., 2006). The latter is an important issue for applications involving mobile users of tactile communication systems, such as soldiers in the battlefield. Other factors considered when choosing an actuator are its durability and safety. Pancake motors (see Figure 8.1) were selected for the tactile displays based on the results from experiments in which the performance of several types of small actuators was compared (Jones et al., 2006). Pancake motors vibrate by rotating a mass in a plane parallel to the surface on which the motor is mounted. The motors are encased in plastic (Smooth-On, Inc) to make them more robust and increase the contact area between the skin and tactor. A wireless tactile control unit (WTCU) was designed and fabricated to control the motors (see Figure 8.1). The WTCU has two main components, a wireless transceiver module for communication with a notebook computer and a microcontroller that receives commands from the wireless module (Bluetooth) and translates them into sequences of motor actuation. The transceiver was required to ensure that the remote unit actually received commands and to determine whether the command was recognized as valid. This feedback is essential because the communication range for Bluetooth devices is variable depending on the type of operating environment. Dense urban environments tend to degrade the radio signal more

146JONES
than open terrain. Two motor driver integrated circuits were also required in the WTCU to drive the 16 motors in the display. The circuit was designed to make the most efficient use of the energy available to it. The power consumption of the WTCU with the Bluetooth connected and three motors vibrating is 848 mW (Lockyer & Jones, 2006).

Figure8.1 (Left)Actuatorsusedintactiledisplays,clockwisefromtopoffigure:C2 tactor(EngineeringAcousticsInc.),encasedpancakemotor,Tactaidtac tor(AudiologicalEngineeringCorp.),rototactor(SteadfastTechnologies). FromJones&Sarter(2008),reproducedwithpermissionoftheHuman FactorsandErgonomicsSociety.(Right)MITwirelesstactilecontrolunit.

The software that operates the tactile display includes a graphical user interface (GUI) for the notebook computer as well as an assembly program for the microcontroller. GUIs were written in Microsoft Visual Basic .NET to run on a notebook computer. They provide the user with a list of commands to transmit to the motors and display the data returning from the display. The software interfaces with the computers COM (serial) port, which communicates using the RS-232 protocol.

CharacterizationofActuators

The properties of the pancake motors were characterized in a series of experiments in which the forces and accelerations of the motors were measured while they were activated. These studies are essential to determining which stimulus variables (e.g. amplitude, frequency, waveform) can be controlled in tactors being considered for a specific tactile display. It was surprising to find in these experiments that the peak frequency of vibration varied considerably across tactors, ranging from 90 to 174 Hz, as illustrated in Figure 8.2, but that the frequency of vibration of an individual motor was consistent (Jones & Held, 2008).

TactileDisplays147

180

Frequency (Hz)

160 140 120 100 80 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Motor number

Figure8.2 Oscillationfrequenciesmeasuredforeachencasedpancakemotor.Each lineonthexaxisrepresentsadifferentmotorandeachshapeonthey axisshowsthefrequencyofoscillationonasingletrial.Themotorsare rankorderedbyfrequency.FromJones&Held(2008),reproducedwith permissionoftheASME.

This range of frequencies means that with the judicious selection of tactors it would be possible to create a display with pancake motors in which frequency encoded information (e.g. urgency of a message, proximity to a target). Measurements of the traveling waves caused by vibrating motors on the skin provide information about the optimal spacing of tactors in a display used for spatial cueing. For these motors, the surface wave is markedly attenuated at 40 mm from the point of activation and by 60 mm there is very little motion on the skin. A spacing of at least 50 mm would therefore be optimal for this class of tactor if the display is designed to provide precise spatial cues.

COMMUNICATIONOFSPATIALINFORMATION

Tactile displays hold particular promise for presenting spatial information about the environment, such as the location of a threat or the intended direction of navigation. The spatial coordinates of a vibrotactile stimulus delivered to the skin are accurately represented in the central nervous system, and so tactile cues on the body can be used to represent the location of an external event. Van Erp (2005) has shown that a one-dimensional array of tactors worn around the waist

148JONES
is very effective for representing spatial information about the environment, and that it is quite intuitive to perceive an external direction emanating from a single point of stimulation. The ability to localize a point of stimulation varies across the body and is best when tactile stimuli are presented near anatomical landmarks such as the navel or spine (Cholewiak & Collins, 2003; Cholewiak et al., 2004).

ExperimentalStudies

A series of experiments was conducted to determine the accuracy of vibrotactile localization around the waist and on the back using the WTCU and tactors configured for the two sites tested (Jones & Ray, 2008). Ten normal healthy participants (five women, five men) aged between 19-24 years took part in each experiment. None of these individuals participated in both experiments. Apparatus. Two tactile displays were created using the encased tactors. The waist display comprised eight tactors that were mounted on a waist band that had a strip of Velcro sewn along its length. The positions of the tactors were adjusted so that tactors were positioned over the navel, spine, right side above the hip and left side above the hip and at the mid-points between these locations as shown in Figure 8.3. The spacing between tactors ranged from 80-100 mm across participants. The display used on the lower back comprised a four-by-four array of tactors mounted on a spandex waist band (see Figure 8.3). The spacing between the center points of the tactors was 40 mm in the vertical direction and 60 mm horizontally. This inter-tactor spacing is considerably greater than the threshold of 11 mm for distinguishing two sites of vibrotactile stimulation on the back (Eskildsen et al., 1969). Procedure. Participants were first familiarized with the sensations associated with activating each tactor and during the experiment they indicated verbally which tactor had been activated, using a visual representation of the display. Each tactor was randomly activated for 500 ms on each trial and was presented five times, giving a total of 40 trials with the waist display and 80 trials with the back display.

TactileDisplays149

Figure8.3 Tactiledisplayaroundthewaist(left)andontheback(right).FromJones andRay(2008)withpermissionofIEEEpress.

Results. Participants experienced little difficulty in locating a point of stimulation around the waist, with an overall response rate of 98% correct. This performance is similar to the 92% correct reported by Cholewiak et al. (2004), who also used an eight-tactor belt but with a different tactor (C2 tactor, see Figure 8.1) which vibrates at a higher frequency (250 Hz) and amplitude than the pancake motors. Localization of a point of stimulation was more difficult with 16 tactors in the array, and on this task, participants were able to identify the correct location of stimulation on only 59% of the trials (range: 40-82%). The locations that were most accurately identified were the two corners in the upper row and those least accurately localized were in the middle of the second row as shown in Figure 8.4. An analysis of variance indicated that there was a significant main effect of tactor (F(15, 135)=2.12, p=0.01) and that activation of tactors in the first row (most superior position) resulted in more accurate localization than activation of any other row (F(3,27)=4.56, p=0.01). Further analysis of the results revealed that most errors involved mislocalization by a single tactor. When responses were coded in terms of localizing the site of stimulation to within one tactor location, the overall response rate increased to 95% correct. Participants were also more likely to identify the correct column of tactor activation (87% correct) than the correct row (68%), which presumably resulted from the smaller difference in inter-tactor spacing in the vertical (40 mm) as compared to the horizontal (60 mm) direction.

150JONES
100 90 80

Percent correct

70 60 50 40 30 20 10 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Tactor number

Figure8.4 Thepercentofcorrectresponsesinlocalizingwhichtactorwasactivated ina16tactorarray.Tactorswerenumberedsequentiallyfromleftto rightbeginningwith1intheupperlefthandcornerofthedisplay(see Figure8.3).AdaptedfromJones&Ray(2008).

Discussion. The results from these two experiments indicate that the ability to localize a point of stimulation on the body is affected by the number of tactors in the array and the inter-tactor spacing. When the number of tactors increases and the distance between the tactors decreases, spatial localization becomes more difficult. Several authors (e.g. Ho et al., 2005; Tan et al., 2003) have proposed that tactile cues delivered to different spatial locations on the back could be used to direct visual attention to the location of potential hazards when driving or to highlight on-screen information in safety critical tasks. The results from the present experiments suggest that a 16-tactor array on the back is unable to support precise spatial mapping between a tactile stimulus on the skin and the location of an external visual target. However, it is clear that an array with fewer tactors mounted over the same surface area would function effectively in providing spatial cues.

TACTILECOMMUNICATIONCREATINGTACTONS

TactileDisplays151
Tactile displays can also be used to communicate more abstract information, such as instructions (e.g. stop, attention, rally), and for this application it is important to evaluate the most effective way of presenting tactile cues to the user. Tactile warning signals have been implemented in a number of devices and vehicles to alert the user to an impending danger, but information that is more complex requires a tactile vocabulary. There are five basic dimensions of vibrotactile signals frequency, intensity, duration, waveform and location available for creating tactile patterns (Jones & Sarter, 2008). Among these, frequency and duration have most often been used to generate tactile stimuli (MacLean, 2008). For example, by grouping vibrotactile pulses of varying duration together, rhythms can be created that can encode cues such as the urgency of a message or the proximity of a vehicle. The development of a tactile vocabulary that can be used to communicate involves identifying what types of information can be recognized with minimal training and determining how messages that are usually conveyed visually or aurally can be converted into tactile patterns. Tactile signals that represent abstract messages are often called tactile icons or tactons, by analogy to visual icons and auditory earcons.

TactilePatternsFactorsInfluencingRecognition

A series of eleven experiments has been conducted using the WTCU and MIT tactile displays to determine the characteristics of tactile patterns that can be readily learned and identified (Jones et al., 2007, 2009; Lam, 2006; Margossian, 2007; Piateski, 2005; Piateski & Jones, 2005). In this research, the tactile displays were mounted on the arms, around the waist, and on the back as these sites are readily accessible and the displays do not impede hand or body movements. The site on the body on which a display is mounted depends on the application domain (i.e. mobile users in a battlefield, mobile or sedentary users interacting computer-based simulations) and factors such as privacy of communication, non-intrusiveness, and display robustness (Jones & Sarter, 2008). An advantage of using the torso is that the displays are discreet, and the communication is private. One initial concern in selecting the torso as a site to mount displays was whether the added mass associated with carrying backpacks and wearing body armor would negatively affect the ability to detect tactile signals. An experiment was therefore conducted to measure tactile pattern recognition while stationary participants wore the Interceptor Body Armor (IBA vest). A set of eight tactile patterns was programmed into the WTCU and participants were required to identify which pattern was presented by the tactile display on the back (Figure 8.3) using the visual template of the patterns (see Figure 8.5). The results from eight participants indicated that the IBA vest had no effect on performance. For

152JONES
the 320 stimuli that participants identified, not a single error in identification was made (Lam, 2006). For this type of display and tactor, the additional mass (6.85-7.79 kg) did not affect performance. It appears that the gap between the IBA vest and the tactile display was sufficient to allow users to perceive the tactile patterns.

Figure8.5 Visualrepresentationofthepatternsusedinseveralexperimentswith thefourbyfourtactorarray.Thenumbers,colorsandarrowsallrepre sentthesequenceoftactoractivation.Allpatternsareofthesamedura tion,butvaryinthenumber,locationandtemporalsequenceof activation.

Other experiments revealed that these tactile patterns created by varying the spatial location and temporal sequence of tactor activation, and the number of tactors concurrently active can be accurately identified and used as cues for navigation and to provide simple instructions. For example in one field study (Jones et al., 2006), participants had no difficulty in learning that pattern A in Figure 8.5 meant go straight ahead and that pattern G signaled stop at the

TactileDisplays153
next waypoint. These studies were illustrative in demonstrating the importance of evaluating tactile pattern recognition in the context of the other patterns within a set that could be presented. The ability to identify a particular pattern varied dramatically (e.g. from 34% to 76% correct), depending on what other patterns were presented concurrently (Piateski, 2005). Other experiments have been conducted using tactile displays mounted on the forearm, which may be preferred site for mounting displays that are used when interacting with virtual environments and when interpreting information presented on a computer screen. For tactile displays mounted on the arm, Jones et al. (2009) found that tactile patterns that involved a pattern of tactor activation in two directions, for example across and up the arm were misidentified more frequently than patterns in which the course of tactor activation was limited to a single direction (e.g. left/right or up-down). These findings indicate that tactile patterns that limit the course of tactor activation within a pattern to a single direction will be more accurately identified. In addition to the spatial sequence of tactor activation, changes in the intensity of a vibrotactile signal can be used to create tactile messages. Such cues could convey information about the proximity of a hazard or obstacle when navigating, or the presence of a restricted area during flight. The intensity of a vibratory stimulus can be modulated by increasing the amplitude of a single vibratory stimulus or by varying the number of tactors activated simultaneously. Cholewiak (1979) has shown that the perceived intensity of a vibratory stimulus increases as the number of tactors activated increases from 1 to 64. This suggests that by varying the number of tactors concurrently active in the display, the perceived intensity of the stimulus will vary. Amplitude discrimination. An experiment was conducted with ten participants in which their ability to discriminate between two vibratory stimuli of varying intensity was measured. The intensity of the vibrotactile stimulus was controlled by varying the number of tactors simultaneously activated. The 16-tactor display was mounted on the back (see Figure 8.3) and participants were required to indicate which of two stimuli presented sequentially was stronger. There were eleven combinations of tactors and each was presented six times. The maximum number of tactors that can be activated simultaneously with the WTCU is four, and so the greatest intensity difference between two patterns was the one in which one and then four tactors were activated. The results indicated that when there was a difference of at least two activated tactors between two signals (1 vs 3 or 2 vs 4), participants almost always chose the signal with more tactors activated as the stronger stimulus. When the difference in signal intensity was due to a single tactor, the ability to discriminate between two stimuli depended on the number of tactors activated in each stimulus, as can be seen in Figure 8.6. When comparing 1 versus 2 tactors,

154JONES
participants were correct on 88% of the trials, but their performance fell to 73% correct when comparing 2 versus 3 tactors. These findings provide guidance as to how intensity can be encoded reliably in a tactile display, and indicate that for this type of tactor a difference of at least two tactors activated should be used as an intensity cue.

Figure8.6 Thepercentofcorrectresponseswhenchoosingthemoreintensestimu lusinpairofvibrotactilestimuli.Thenumberofconcurrentlyactivated tactorsisshownaboveeachbar. TACTILECOMMUNICATIONARMANDHANDSIGNALS

The Army Hand and Arm Signals represent a code that is presented visually to convey simple commands (e.g. take cover, advance to left) for which tactile analogs could readily be developed. A set of eight arm-and-hand signals was converted into tactile patterns that retained some of the iconic information of the hand signal. For example, the hand signal for take cover involves lowering the outstretched arm from above the head to the side and so the tactile analog involved activating the four rows of tactors sequentially down the back as illustrated in Figure 8.7.

TactileDisplays155

Figure8.7 Schematicrepresentationoftactilepatterns,associatedarmandhand signalsandcommands.AdaptedfromJonesetal.,2009. ExperimentalStudies

156JONES
In the contexts in which these displays will be used for communication (i.e., mobile soldiers in the battlefield), users must be able to recognize and respond to the meaning of the tactile signal as it is presented while they are engaged in other activities. A series of experiments was therefore conducted to examine tactile pattern recognition while the user was involved in a concurrent task, two of which were physical, the other cognitive. If participants can identify these tactile analogs of the arm-and-hand signals accurately, then this indicates that more abstract information, such as instructions or warnings, can be presented tactually and used in situations in which visual and auditory communication is restricted due to either safety or privacy concerns, or ambient levels of light or noise. The eight tactile patterns (Figure 8.7) were presented on the four-by-four tactor display mounted on the back (seeFigure 8.3). Each pattern was presented three times. Ten adults (five men and five women) aged between 18 and 38 years participated in the experiments. Procedure. Participants were first trained to associate the tactile pattern with its meaning. During this phase, which lasted approximately five minutes, participants viewed the schematic of the patterns and associated hand signals as illustrated in Figure 8.7. During the experiments, participants did not view the visual template and had to respond to the tactile signal by verbally identifying the pattern. In one set of experiments, participants were either walking or jogging while the tactile stimuli were delivered, and in the other they were engaged in a word-generation task on a computer. Results. The results from the three experiments are shown in Figure 8.8. The overall mean correct response rates were 91% (walking), 91% (jogging) and 93% (computer task), which demonstrates that participants were able to identify tactile patterns despite the performance of a concurrent task and that the body movements associated with walking and jogging did not impair perception. Moreover, in these experiments participants were not alerted about the upcoming tactile signal, and were still able to process the tactile cues accurately. Most participants reported that they remembered the tactile cues by visualizing the pattern and attended to the location at which the signal began. Tactile communication systems are often evaluated in terms of their information transmission capabilities. Information transfer (IT) measures the increase in information about a signal transmitted that results from knowledge of the received signal (Tan et al., 1999). For tactile communication systems involving the hand, the static IT varies with the dimensionality of the tactile stimuli and the number of fingers involved in perceiving the stimuli (Tan et al. 1999). The IT measured on the back during this experiment ranged from 2 to 3 bits for stimuli that are considered to be two-dimensional. These values are comparable to those measured for other tactile communication systems presenting stimuli with similar dimensionality.

TactileDisplays157

100 80

Percent correct

60 40 20 0 1 2 3 4 5 6 7 8

Hand signal

Figure8.8 Groupmeanpercentageofcorrectresponsesinidentifyingtactilepat ternsrepresentingarmandhandsignalswhilewalking(whitebars),jog ging(graybars)andperformingacognitivetask(blackbars). CONCLUSIONS

The results from this research on tactile displays clearly demonstrate that simple navigational and instructional commands presented tactually on the arm or back can be identified quickly and accurately. Much of the previous research on tactile communication systems has focused on the more sensitive skin on the palmar surface of the hand and fingers and so the present results are important in demonstrating the capabilities of other areas on the body for communication. These sites are non-intrusive, do not impede movements, and have the added advantage that the displays can be worn under clothing and so are protected and concealed.

158JONES
The configuration of the tactile displays used in this research (i.e., belt, sleeve, vest) varied as a function of the site on the body on which the display was mounted. This in turn influenced the ability to identify tactile patterns. For tactile pattern recognition, the forearm was determined to be inferior to the back. This probably reflects the smaller inter-tactor distance on the arm (24 mm as compared to 40-60 mm on the back), the reduced intensity of the tactile stimulus with three rather than four tactors activated for most patterns, and the shorter duration of most of the patterns presented on the arm (Jones et al., 2009). Nevertheless, with appropriate selection of patterns, participants were able to identify accurately up to eight patterns displayed on the arm. An important question emerges in the development of tactile communication systems which concerns the potential size of a vocabulary that could be used for communication. A number of experiments was conducted using the four-by-four tactile display mounted on the back in which the accuracy of pattern recognition was evaluated as a function of the number of patterns presented. To date, the maximum vocabulary tested had 15 unique elements and participants were able to identify these with an accuracy of 96% correct (Jones et al., 2007). Further experiments with larger vocabularies will need to be undertaken in order to determine the upper limit of a tactile communication system. It was also important to establish in this research that participants did not need extensive training to become familiar with the tactile patterns presented. Clearly, for this technology to be useful for soldiers, learning a tactile vocabulary should not require a prolonged period of training. With the exception of a tactile alert, tactile patterns do not have intrinsic meaning and so users will always need to become familiarized with the instruction or cue associated with a particular pattern. In all of the experiments conducted to date, the time required to become familiar with the patterns was less than 5 minutes and participants had no difficulty in relating a tactile pattern to a simple instruction or command. The tactile displays used in these experiments stimulated the skin at a fixed frequency and amplitude and the location and number of tactors concurrently active was varied to convey information. These two variables were used to create tactile patterns as they represent dimensions that are reliably perceived. Other dimensions of vibrotactile signals such as frequency and the temporal pattern of activation hold promise as additional cues that could be used to create tactile vocabularies in future research. This research has demonstrated the potential of tactile communication systems for the Army, has highlighted the application areas that hold most promise, namely spatial cueing and navigation, and has shown that a simple tactile vocabulary can be understood and responded to as people are engaged in other activities.

TactileDisplays159 REFERENCES
Cholewiak,R.W.(1979).Spatialfactorsintheperceivedintensityofvibrotactile patterns.SensoryProcesses,3,141156. Cholewiak,R.W.,&Collins,A.A.(2003).Vibrotactilelocalizationonthearm:Effects ofplace,space,andage.Perception&Psychophysics,65,10581077. Cholewiak,R.W.,Brill,J.C.,&Schwab,A.(2004).Vibrotactilelocalizationonthe abdomen:Effectsofplaceandspace.Perception&Psychophysics,66,970987. Eskildsen,P.,Morris,A.,Collins,C.C.,&BachYRita,P.(1969).Simultaneousand successivecutaneoustwopointthresholdforvibration.PsychonomicScience,14,146 147. Hass,E.(2009,February).Reviewofmultimodal(audio,tactileandvisual)display research.PaperpresentedattheTactileWorkshop,HumanResearchandEngineering Directorate,U.S.ArmyResearchLaboratory,Aberdeen,MD. Ho,C.,Tan,H.Z.,&Spence,C.(2005).Usingspatialvibrotactilecuestodirectvisual attentionindrivingscenes.TransportationResearchPartF:TrafficPsychologyand Behaviour,8,397412. Jones,L.A.,&Held,D.A.(2008).Characterizationoftactorsusedinvibrotactile displays.JournalofComputingandInformationSciencesinEngineering,8,04450115. Jones,L.A.,&Ray,K.(2008).Localizationandpatternrecognitionwithtactile displays.InProceedingsoftheSymposiumonHapticInterfacesforVirtualEnvironment andTeleoperatorSystems,(pp.3339).LosAlamitos,CA:IEEEComputerSociety. Jones,L.A.,&Sarter,N.(2008).Tactiledisplays:Guidancefortheirdesignand application.HumanFactors,50,90111. Jones,L.A.,Kunkel,J.&Piateski,E.(2009).Vibrotactilepatternrecognitiononthe armandback.Perception,38,5268. Jones,L.A.,Kunkel,J.,&Torres,E.(2007).Tactilevocabularyfortactiledisplays.In ProceedingsofthesecondjointEurohapticsConferenceandSymposiumonHaptic InterfacesforVirtualEnvironmentandTeleoperatorSystems,(pp.574575).Los Alamitos,CA:IEEEComputerSociety. Jones,L.A.,Lockyer,B.,&Piateski,E.(2006).Tactiledisplayandvibrotactilepattern recognitiononthetorso.AdvancedRobotics,20,13591374. Krausman,A.S.,&White,T.L.(2006).Tactiledisplaysanddetectabilityofvibrotactile patternsascombatassaultmaneuversarebeingperformed(ARLTR3998).Aberdeen ProvingGround,MD:U.S.ArmyResearchLaboratory. Krausman,A.S.,Elliot,L.R.,&Pettitt,R.A.(2005).Effectsofvisual,auditory,and tactilealertsonplatoonleaderperformanceanddecisionmaking(ARLTR3633). AberdeenProvingGround,MD:U.S.ArmyResearchLaboratory. Krausman,A.S.,Pettitt,R.A.,&Elliot,L.R.(2007).Effectsofredundantalertson platoonleaderperformanceanddecisionmaking(ARLTR3999).AberdeenProving Ground,MD:U.S.ArmyResearchLaboratory. Lam,A.(2006).Vibrotactilepatternrecognitiononthetorsowithoneandtwo dimensionaldisplays.Unpublishedbachelorsthesis,MassachusettsInstituteof Technology,Cambridge,MA. Lockyer,B.,&Jones,L.A.(2006).OperationmanualfortheMITWirelessTactile ControlUnit(WTCU).Cambridge,MA:MassachusettsInstituteofTechnology. MacLean,K.E.(2008).Foundationsoftransparencyintactileinformationdesign. IEEETransactionsonHaptics,1,8495.

160JONES
Margossian,C.(2007).Vibrotactilepatternrecognitiononthetorso:Effectsof concurrentactivities.Unpublishedbachelorsthesis,MassachusettsInstituteof Technology,Cambridge,MA. McGraith,B.J.,Estrada,A.,Braithwaite,M.G.,Raj,A.K.,&Rupert,A.H.(2004). Tactilesituationawarenesssystem.Flightdemonstrationfinalreport.USAARLReportNo. 200410. Piateski,E.(2005).Atactilecommunicationsystemfornavigation.Unpublished mastersthesis,MassachusettsInstituteofTechnology,Cambridge,MA. Piateski,E.,&Jones,L.(2005).Vibrotactilepatternrecognitiononthearmand torso.InProceedingsofthefirstjointEurohapticsConferenceandSymposiumonHaptic InterfacesforVirtualEnvironmentandTeleoperatorSystems,(pp.9095).LosAlamitos, CA:IEEEComputerSociety. Redden,E.S.,Carstens,C.B.,Turner,D.D.,&Elliott,L.R.(2006).Localizationoftactile signalsasafunctionoftactoroperatingcharacteristics(ARLTR3971).AberdeenProving Ground,MD:U.S.ArmyResearchLaboratory. Rupert,A.H.(2000).Aninstrumentationsolutionforreducingspatialdisorientation mishaps.IEEEEngineeringinMedicineandBiologyMagazine,March/April,7180. Spence,C.,&Ho,C.(2008).Tactileandmultisensoryspatialwarningsignalsfor drivers.IEEETransactionsonHaptics,1,121129. Tan,H.Z.,Durlach,N.I.,Reed,C.M.,&Rabinowitz,W.M.(1999).Information transmissionwithamultifingertactualdisplay.Perception&Psychophysics,619931008. Tan,H.,Gray,R.,Young,J.J.,&Traylor,R.(2003).Ahapticbackdisplayfor attentionalanddirectionalcueing.Hapticse,3,Article1.RetrievedNovember22007 fromhttp://hapticse.org/Vol_3/hev3n1.pdf. VanErp,J.B.F.(2005).Presentingdirectionswithavibrotactiletorsodisplay. Ergonomics,48,302313. White,T.L.,Kehring,K.L.,&Glumm,M.M.(inpress).Effectsofunimodaland multimodalcuesaboutthreatlocationsontargetacquisitionandworkload.Military Psychology.

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T A N D L A U R E L A L L E N D E R

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T AND LAUREL ALLENDER

This book, produced by the Partners of the Army Research Laboratory Advanced Decision Architectures Collaborative Technology Alliance, also serves as the Final Report for DAAD19-01-2-0009.

SECTIONII.PRESENTINGBATTLEFIELDINFORMATIONTOWARFIGHTERS

SECTIONII PRESENTINGBATTLEFIELDINFORMATIONTOWARFIGHTERS

122SECTIONII

Chapter

9. MULTIMODALINFORMATIONEXCHANGEANDDYNAMICADAPTATION

NADINESARTER,PH.D.,THOMASFERRIS,ANDSHAMEEMHAMEED DepartmentofIndustrialandSystemsEngineering CenterforErgonomics UniversityofMichigan,AnnArbor,MI

INTRODUCTION

High levels of complexity and dynamism will characterize the battlefield of the future. It will involve a large number of human and machine agents in various locations who need to collaborate on planning and problem-solving tasks, sometimes under considerable time pressure, high risk, and uncertainty. This implies that effective information systems, i.e., the ability to collect, store, distribute, fuse, and share information, will become increasingly important. To support this need, a line of research was conducted that focused on the distribution of information across multiple media and modalities, including vision, audition, and touch. First, a computer-based simulation platform was created that supports collocated and remote synchronous multimodal communication and coordination among multiple human and machine agents. The creation of this rather unique platform was necessary to conduct controlled studies of a wide range of issues related to multimodal information processing, including modality preferences, crossmodal interactions, and context-sensitive modality selection and combination. This chapter provides an overview of the main findings from this line of research and provides guidance for the design of future multimodal displays.

161

162SARTER,FERRIS,ANDHAMEED
The battlefield of the future will be highly complex and dynamic. It will involve a large number of human and machine agents some of them collocated, others distributed across various locations who need to collaborate on planning and problem-solving tasks, sometimes under considerable time pressure, high risk, and uncertainty. To support these tasks, effective information systems are needed that make it easy to collect, store, distribute, fuse, and share information. With this goal in mind, a line of research was conducted that focused on the distribution of information across multiple media and modalities, including vision, audition, and touch. Multimodal information presentation was originally suggested by Multiple Resource Theory (MRT; Wickens, 1980) as a promising means to support operators in coping with large amounts of data. MRT assumes that different sensory channels are associated with separate attentional resources. Thus, by distributing information across modalities, resource competition should be minimized and overall processing performance should improve. In order to design effective multimodal displays, a number of important questions related to multimodal information processing were investigated as part of this line of work, including (a) users modality preferences, (b) crossmodal spatial and temporal links between vision, hearing, and touch, and (c) the design of a context-sensitive multimodal interface that is both adaptive and adaptable. The following sections will highlight some of the major activities and findings from this effort.

MULTIMODALSIMULATIONPLATFORM

As a first step in this line of research, a computer-based multimodal simulation platform was developed in our laboratory. This platform supports collocated and remote synchronous communication and coordination among multiple agents in the context of simulated battlefield operations. It can present multiple participants with a shared dynamic top-down view of a battlefield and enable them to exchange information with other human and machine agents via the visual, auditory, and tactile channel. This platform was used to conduct three complementary studies on various aspects of multimodal information presentation: (1) the first longitudinal study of natural patterns and preferences of modality usage and integration in combat teams, (2) an experiment on the use of peripheral visual and tactile cues for monitoring mission health in a preattentive manner, and (3) a study on the performance effects of crossmodal spatial links in attention (i.e., the effects of ipsi- and contralateral presentation of cues and targets in different modalities on detection rates and response times).

ContextSensitiveMultimodalDisplayDesign163
The simulation displays information and controls similar to those in a Stryker Vehicle Commander (VC) station (see Figure 9.1). Centered on the screen, a top-down map of the surrounding area is shown with icons indicating the location of convoy vehicles along the route, enemy vehicles, areas of suspected weapons caches, and other points of interest. In addition, at the top left and right corners of the screen, small rectangular displays show the terrain in front of the vehicle as if viewed through periscopes projecting forward from the top of the vehicle. Animations of distant explosions or traveling vehicle headlights can be presented at times on these periscope displays. Various indications of threats or failures can be presented. For example, immediately below each periscope display, a thermal detection system display indicates whenever a heat source is detected near the Stryker vehicle. Also, below each thermal detection system display, a temporary loss of satellite communications can be indicated by a change in the backlighting of a communication display from faint green to bright yellow. The rest of the display is filled with static images of gauges and switches to add visual noise and represent some of the other controls visible to a Stryker VC.

Figure9.1 MultimodalSimulationPlatform

In addition to the visual presentation of information, the simulation includes auditory and tactile displays. Lateralized scenario-related sounds (gunfire and explosions, radio communications, and squawk noises that represented radio short-circuit problems) can be played from speakers located immediately to the left and right of the monitor at approximately 25 degrees of visual angle. Vibrating devices, so-called tactors, are used to present participants with tactile signals. These tactors are commercially available light-

164SARTER,FERRIS,ANDHAMEED
weight devices that present vibrations to the skin, usually at an intensity no greater than that of a personal pager and at varying levels of intensity, in various body locations (e.g., wrist, back) and, in some cases, as a sequence of vibrations that creates a particular pattern or signature. Vibrations on the left and right side can be used, for example, to alert participants to potential danger approaching from the left or right side of the vehicle, respectively, or to provide navigation guidance.

NATURALMODALITYUSAGEPATTERNS

One important decision in the design of multimodal interfaces is the assignment of modalities to tasks or types of information. This decision can be based on a number of considerations, including appropriateness and compatibility (e.g., information that needs to be referred to at a later time should be presented in visual, rather than transient auditory, form). Another driver for modality assignments can be users natural tendencies for, and the role of context in, modality usage. To gain better insight into these issues, we conducted an experiment that involved three groups of three ROTC cadets/ officers each (Ho and Sarter, 2004). They each completed a set of 30-minute scenarios in the context of simulated battlefield operations after a three-hour training session. First, participants received a briefing and their operational order. They discussed their mission as well as plans for utilizing the various communication channels. Then, they completed one or two scenarios, depending on the particular session. The order of scenarios was randomized. Debriefings, including an after-action review led by a senior ROTC cadet, and discussions of observed modality usage were conducted after each scenario. Two collocated participants played the roles of unit commander (CO) and military intelligence officer (S2). The distributed participant played a platoon leader (PL) and a senior ROTC cadet was hired to play the role of the battalion commander (BC). He also served as a confederate who controlled the enemy units throughout all experimental sessions. Participants were presented with a dynamically updated shared map of the battlefield. They could send input to the system by using the touch screen, a mouse, or voice commands (voice input was available only for firing at vehicles). They could also set up visual, auditory, and tactile alerts for various events (such as enemy sighted, low fuel, or low ammunition). Communication between participants was achieved through visual (text message, drawing/referring), auditory (two way radio, face-to-face

ContextSensitiveMultimodalDisplayDesign165
conversation), and tactile (tactile messages) means. Tactile information was carried by two tactors that applied vibrations to the participants left and right wrists. The choice of modality assignments was based solely on individual and group preferences and experiences; they were not determined in advance by the experimenter. Our findings show that participants did not necessarily interact multimodally just because a multimodal interface was made available. Instead, they were highly selective in their use and combination of different channels (for similar findings, see Oviatt, 1999; Oviatt et al., 1997). In this study, participants employed multimodal interaction primarily in the context of spatial tasks and to support complementarity. They tended to switch modalities mostly for the purpose of recovering from communication breakdowns. In the context of human-human interaction, most modality combinations were sequential, rather than simultaneous, in nature. For example, one modality was used first for attention capture (such as a tactile buzz) or for pointing out an object, then a different modality was used to expand on, or disambiguate the meaning of, the first cue. This confirms and illustrates the use of multiple modalities for the purpose of contrastive functionality, i.e., users tendency to employ different modalities in sequence to achieve or indicate a shift in either content or functionality of communication (Oviatt & Olsen, 1994). The preference for sequential modality usage was observed even though 10 out of 27 modality combinations in human-human interaction involved drawing and radio - two channels that, in principle, could be used simultaneously. Our findings also illustrate that modality usage patterns evolve as a function of team coordination and change in response to factors such as scenario demands, the mission phase, and group dynamics. Individual and group modality preferences are only one factor that needs to be considered in the design of multimodal interfaces. In addition, possible interactions and interference between sensory channels need to be taken into account. To this end, we conducted another study employing the above multimodal simulation that examined crossmodal spatial and temporal links in attention.

CROSSMODALSPATIALANDTEMPORALLINKSINATTENTION

The assumption that modalities are associated with separate attentional resources (see Wickens, 1980) was challenged, to some extent, by recent studies uncovering both spatial and temporal links in attention between vision, audition, and touch (e.g., Spence and Driver, 1997). These studies have shown that shifts in attention to a particular location in one sensory modality lead to concurrent shifts in other modalities (for an overview, see

166SARTER,FERRIS,ANDHAMEED
Driver & Spence, 2004). As long as two signals occur within a particular range of stimulus-onset asynchronies (SOAs), crossmodal links lead to improved target detection when the preceding cue originates from the same or similar location as a subsequent target in a different modality (valid/ipsilateral cuing). Performance suffers when cues and targets are appear in different locations. Crossmodal spatial links have been demonstrated for all cue-target pairings involving vision, audition, and touch (e.g., Spence et al., 2004). However, a small number of studies have reported various forms of crossmodal asymmetry. For example, Ward (1994) found that spatially uninformative visual cues (i.e., cues not predicting the likely location of a subsequent target) orient auditory attention but spatially uninformative auditory cues do not affect visual attention. Since most of the above research was conducted in highly controlled laboratory environments using fairly few and simple tasks and signals, we conducted an experiment to determine whether the performance effects of crossmodal spatial links that were observed in those earlier studies scale to more complex environments and to examine more systematically and in depth the issue of crossmodal cuing asymmetries between vision, audition, and touch. Twelve cadets, two graduates from the University of Michigan (UM) Army Reserve Officers Training Corps (ROTC) program, and one civilian UM graduate student employed by the US Army participated in this study (7 females and 8 males). Each participant played the role of a vehicle commander (VC) for the first of a convoy of vehicles in a simulated night-time rendezvous mission. Throughout the mission, participants were presented with visual, auditory, and tactile stimuli, some of which were designated as targets (requiring a speeded response) while others served as cues or distracting events. This experiment again employed the simulation platform described earlier (see Figure 9.1), including the top-down map of the surrounding area indicating the location of convoy vehicles along the planned route, enemy vehicles, and areas of suspected weapons caches. Animations of distant explosions or traveling vehicle headlights were shown on the periscope displays that were described earlier. And thermal detection system and satellite communication displays were used to indicate the presence of a nearby heat source and the temporary loss of satellite communications, respectively. The visual events presented on the lateralized displays (i.e., periscope, thermal detection, and satellite communication displays) appeared at approximately 20 degrees of visual angle from the center of the map display.

ContextSensitiveMultimodalDisplayDesign167
Scenario-related sounds included gunfire and explosions, radio communications, and chirp noises that represented radio short-circuit problems. Tactors were attached to wristbands worn by participants on both arms. Vibrations on the left or right side served to alert the VC to potential danger approaching from the corresponding side of the vehicle. In addition, four tactors were secured to the outside of each thigh in a randomized pattern to represent incidental vibrations similar to those experienced when riding over terrain. Each participant was responsible for three tasks. The highest-priority task was to respond as quickly as possible to the visual, auditory, or tactile stimuli designated as targets (see Table 1:) by pressing a button. The target stimuli represented events that posed potential danger to the Stryker convoy, and the button press served to alert the other convoy vehicles. 48 targets were presented either in isolation (uncued trials, n=24) or preceded by an ipsilateral (same-side, n=12) or contralateral (opposite-side, n=12) peripheral cue in a different modality (see Table 1:). Each cue event was presented in isolation an additional 12 times throughout the mission. The participants secondary task was to monitor radio communications for reports of mission status and to respond to orders and inquiries. The task with the lowest priority was to manually control (using a joystick) an unmanned aerial vehicle (UAV) to search for and mark weapons caches in areas displayed on the map. This task encouraged, in a natural way, a central fixation of visual attention. Additionally, a remote eyetracker was used to delay cue presentation (or target presentation, in uncued trials) until the gaze direction was determined to be near the center of the screen. Target detection rates for this study were extremely high, and therefore the primary dependent measure was response time, measured as the duration between the target onset and the corresponding button press.

Table 1: Targets and peripheral cues in three modalities

Visual Auditory Tactile

168SARTER,FERRIS,ANDHAMEED

Distant explosionin periscope Gunfiresounds Vehicle Target headlightsin periscope Thermal detection systemalert Satelliteradio backlightcolor change (temporary commloss) Chirpnoise (radioshortcir cuit) Vibrationon thighs (incidentalvehicle vibrations) Vibrationonwrist, indicatingpoten tialthreat

Explosion sounds

Cues

The findings from this experiment confirmed that crossmodal spatial links affect performance in rather complex settings and that these performance effects differ for various modality pairs. Cuing effects were larger, and response times in this experiment were longer and varied to a higher degree, than in earlier studies. These findings can be explained, in part, by the competing attentional demands that were imposed by mission-related tasks in the present experiment. Ipsilateral (presentation on the same side) crossmodal cuing was beneficial only for auditory cuing of visual targets but not vice versa, which mirrors the asymmetry between auditory and visual cues and targets found in earlier studies. Significantly faster responses were found for contralateral (presentation on opposite side) tactile cuing of auditory targets, but not vice versa. Prior to this study, significant benefits for ipsilateral cuing had been reported for all cue-target combinations involving vision, audition, and touch (Spence et al., 2004). The surprising finding from this experiment was that visually-cued tactile targets, tactile-cued visual targets, and tactile-cued auditory targets showed a trend towards the opposite effect. Participants were faster to respond in case of contralateral presentations. One possible explanation for these findings is crossmodal Inhibition-Of-Return (IOR). IOR refers to the fact that,

ContextSensitiveMultimodalDisplayDesign169
for a certain period of time, our attentional focus does not return to an already visited location to avoid wasting attentional resources (Klein, 2000). The fact that the IOR effect was not found for auditorally-cued targets is consistent with work by Reuter-Lorenz and Rosenquist (1996; however, see Spence & Driver, 1998, for different observations). The crossmodal cuing effects and the modality asymmetries observed in this study suggest that fixed assignments of modalities to tasks and types of information are not desirable. Instead, an adaptive multimodal interface is needed where the location, modality, and timing of information presentation is varied as a function of surrounding stimuli and task context. Creating such an interface was therefore the next step in this line of research.

CONTEXTSENSITIVEINFORMATIONPRESENTATION

The goal of this activity is to develop a multimodal interface that conveys information via vision, audition, and touch and adjusts the timing and nature of information presentation to accommodate changing battlefield contexts (for more details on this activity see Hameed and Sarter, 2009). To this end, we first had to identify and examine appropriate drivers for the adaptation of modalities. A review of the literature on adaptive interfaces suggested several candidates. For example, Hollnagel and Woods (2005) listed personal preference, experience, norms/standards of the work environment, individual versus collaborative work, temporal and task demands as factors to be considered in selecting the appropriate form of information presentation. Findings from our own earlier research suggest that factors such as crossmodal spatial and temporal links in attention may also need to be considered. For example, visual targets that follow an auditory cue should preferably be presented ipsilaterally whereas visual targets that follow a tactile cue best appear on the opposite side to avoid an increase in response time (see Ferris and Sarter, 2006). Given the nature of the military environment, ambient conditions and channel availability (as determined by proper functioning of the equipment and/ or the human sensory system) also need to be considered in the design of an effective interface but were not examined in detail in the above studies, except for the issue of substitution of an unavailable modality.

170SARTER,FERRIS,ANDHAMEED
The set of candidate drivers was reduced based on the following considerations. As reported earlier, one important finding from our studies was that personal preferences are not appropriate because of the collaborative and interactive nature of combat operations. Soldier teams, rather than individual users, will need to explore, and agree on, preferred modality usage and combination in advance of operations (see Sarter and Ho, 2004). In our experiments, these group preferences were based primarily on observed performance and effectiveness (rather than (dis)likes). Type of task was confirmed to be an important determinant of modality usage. For example, spatial information was exchanged preferably using the visual and tactile channel. Spatial (but not verbal) tasks benefited from modality combination where, frequently, verbal exchanges helped disambiguate visual and tactile information. Non-spatial tasks were most often conducted unimodally. The findings from our studies also highlight the need for careful consideration of spatial and temporal crossmodal links in attention (Driver and Spence, 2004; Spence and Driver, 1997) which have been ignored, for the most part, in multimodal interface design to date. Response times and accuracy to targets in various modalities were shown to be affected by the location and timing of preceding cues and distractors. Thus, the interface will need to track and, in some cases, adjust the timing and location of information presentation to prevent problems related to phenomena such as crossmodal attentional blink, inhibition of return, and crossmodal spatial links. Finally, operator state will be employed as yet another adaptation driver. In particular, heart rate variability will be assessed online as an indicator of variations in mental workload. The above drivers are being implemented in our current design to allow the system itself to adjust the timing and nature of information presentation the adaptive component. In addition, the user will have the opportunity to override the system and decide based on own needs and preferences how to set up display components the adaptable component of the interface. Such a hybrid approach to context-sensitive information presentation seems preferable to exploit the advantages, and guard against the shortcomings, of each individual strategy. Soldiers need to be supported in maintaining awareness of (changes in) the active configuration of communication channels without being overburdened with interface management and monitoring tasks. They need to know about, and be able to anticipate, their own interface settings and options as well as the active and available channels on the part of remote communication partners. Thus, we are developing feedback and controls that allow for at-aglance checking and rapid changes of the modality configuration.

ContextSensitiveMultimodalDisplayDesign171

MULTIMODALDESIGNGUIDANCE

One additional important component of our overall research effort consisted of a review of the literature on multimodal information processing and presentation and our own empirical findings with the goal to develop a guidance document for multimodal display design (for more details, see Sarter, 2006; Jones and Sarter, 2008). This guidance relates to 3 major steps: 1) the selection of the modalities, 2) the mapping of modalities to tasks and types of information, and 3) the combination, synchronization and integration of modalities.

SelectionofModalities

A designer first needs to decide whether the use of multiple modalities is necessary or desirable for a given task/task environment (Reeves et al., 2004). Possible reasons for including more than one modality include differences in user preferences, needs, and abilities (e.g., ETSI, 2003; Sutcliffe, 2003; but see our own findings in the context of collaborative domains). Multimodal system output may be desirable also to support functions such as synergy, redundancy, disambiguation, and an increased bandwidth of information transfer (see Oviatt, 2002; Sarter, 2002). If a multimodal interface seems necessary and appropriate, the required and appropriate channels need to be identified next. This decision is determined, in part, by environmental constraints, such as high levels of ambient noise, which may exclude or limit the use of certain channels (in this case, auditory information presentation). The selection of modalities and media also needs to consider the tasks and type of information that the user needs to handle (ISO, 1988).

MappingofModalitiestoTasks/TypesofInformation

Once the set of feasible and desirable media and modalities has been identified, the mapping of modalities to different tasks and types of information needs to be determined. For example, foveal vision allows for permanent presentation and thus delayed and prolonged attending. Thus, it is appropriate for presenting complex graphics and large amounts of detailed

172SARTER,FERRIS,ANDHAMEED
information which may need to be referred to at a later time. In contrast, peripheral vision is well suited for detecting motion, luminance changes, and the appearance of new objects, but it does not support the recognition of objects or details. Thus, it represents an early orientation mechanism that helps attract and guide operators attention to relevant information. The auditory channel differs from vision in that it is omnidirectional, i.e., it allows for information to be picked up from any direction. Also, auditory information presentation is transient, and, since it is impossible for us to close our ears, auditory displays are often reserved for alerting functions. One form of auditory information presentation that has recently received increased attention is called sonification. Sonification involves the transformation of data relations into perceived relations in an acoustic nonspeech signal for the purposes of facilitating communication or interpretation (Kramer, 1993). This form of auditory information presentation is being used in medical settings for the representation of continuous parameters, such as heart rate (Kramer et al., 1999), but could be employed also as an indicator of rate of closure of an approaching vehicle, for example. In contrast to vision and audition, touch represents an underutilized channel for information presentation. Touch refers to the sensations that result from mechanical, thermal, chemical, or electrical stimulation of the skin (Cholewiak and Collins, 1991). Cues presented via the tactile channel are transient. Like vision and hearing, touch allows for the concurrent presentation and extraction of several dimensions, such as frequency and amplitude, to create more complex and abstract stimuli. Tactile stimuli are particularly well suited for presenting spatial information. They also have been employed for notifications that range in salience between auditory alarms and visual messages.

Combination,Synchronization,andIntegrationofModalities

As mentioned earlier, the above discussed assignment of modalities to tasks and types of information cannot be made without consideration of the resulting spatial and temporal combination and synchronization of these channels (e.g., Oviatt, 2002; Raman, 2003). To date, very little is known to guide this step in multimodal design. Basing modality combinations on user preferences, needs, and abilities (e.g., ETSI, 2003; Sutcliffe, 2003) is problematic and unsatisfactory because it shifts the responsibility from the designer to the user. who is unlikely to have the knowledge or time to determine and implement appropriate and effective modality combinations in advance or during ongoing operations.

ContextSensitiveMultimodalDisplayDesign173
Reeves et al. (2004) provide some specific examples of effective modality combinations. For example, they propose that map-based displays should be combined with spoken interaction and that virtual displays benefit from being coupled with non-speech audio. In current systems, one often employed modality sequence is the use of an auditory alert, followed by the visual presentation of relevant information. For example, route guidance systems for cars use an auditory signal to notify the driver of an upcoming turn, and a visual display then provides more detailed information about the turn (e.g., Mollenhauer et al., 1997).

IMPLICATIONFORTHEFUTUREARMY

Some of the main findings from a line of research on multimodal information processing and display design in support of future military operations were presented. This work provides insight into user preferences for modality usage/combinations, examined the role of crossmodal spatial and temporal links in attention in the context of complex task environments, and highlights challenges and possible solutions for the contextsensitive use of multimodal displays. The reported research confirms the promise and benefits of multimodal information presentation but also highlights potential pitfalls and limitations that need to be considered to ensure the robustness of military displays. It emphasizes that merely distributing information across modalities is not sufficient to reap the benefits of this approach. Modalities are not entirely independent of one another, and not each modality is equally well suited for each task of type of information. For example, if soldiers are to be presented with spatial information or navigational guidance, then vision, touch, and possibly 3-dimensional auditory cues are the preferable sensory channels/ methods for this purpose. If the information will likely need to be referred to at a later point in time, vision is the best option. On the other hand, touch involves the benefit of allowing the soldier to process the information in parallel with all the other visual information that may be need to be presented and absorbed. It also allows for the private display of information. Environmental conditions, such as vibrations, may render the sense of touch less useful and effective, on the other hand. Thus, tradeoff decisions need to be made that consider factors such as appropriateness, feasibility, and acceptability. And a hybrid adaptive/ adaptable interface design is needed that tracks the state of various drivers for context-sensitive information presentation, adjusts modality assignments for the soldier to minimize interface management tasks, but allows him/her to track and override those adjustments if necessary.

174SARTER,FERRIS,ANDHAMEED
In summary, the findings from this line of work (along with the considerable and growing body of literature on multimodal displays in general) can inform the design of future military displays that support more effective information exchange and collaborative decision making on the battlefield. These findings also remind us, however, that the rather arbitrary and fixed assignments of tasks and types of information to various sensory channels that have dominated to date will not going to result in the most effective multimodal display systems.

ACKNOWLEDGMENT

The work reported in this manuscript was supported by a grant from the Army Research Laboratory (ARL), under the Advanced Decision Architecture (ADA) Collaborative Technology Alliance (CTA) (grant #DAAD 19-01-2-0009; CTA managers: Dr. Mike Strub and Dr. Laurel Allender; project manager: Sue Archer).

REFERENCES
Arnell,K.M.andJolicoeur,P.(1999).TheAttentionalBlinkAcrossStimulus Modalities:EvidenceforaCentralProcessingLimitation.JournalofExperimental Psychology:HumanPerceptionandPerformance,25,630648. Brewster,S.A.andBrown,L.M.(2004),"Tactons:StructuredTactileMessagesfor NonVisualInformationDisplay".ProceedingsAustralasianUserInterfaceConference (pp.1523).Dunedin,NewZealand. Cholewiak,R.andCollins,A.,1991.Sensoryandphysiologicalbasisoftouch.InM. HellerandW.Schiff(Eds.),Thepsychologyoftouch.LEA,Mahwah,NJ,pp.2360. Driver,J.andSpence,C.,2004.Crossmodalspatialattention:evidencefromhuman performance.InC.SpenceandJ.Driver(eds.),CrossmodalSpaceandCrossmodal Attention.OxfordUniversityPress:Oxford,UK,pp.179220. ETSI(EuropeanTelecommunicationsStandardsInstitute),2003.Multimodal interaction,communicationandnavigationguidelines(ReportETSIEG202191V1.1.1 (200308).ETSI,SophiaAntipolis,France. Ferris,T.K.andSarter,N.B.(2008).CrossmodalLinksBetweenVision,Audition,and TouchinComplexEnvironments.HumanFactors,50(1),1726. Ferris,T.,Penfold,R.,Hameed,S.,Sarter,N.B.(2006).CrossmodalLinksinAttention: TheirImplicationsForTheDesignofMultimodalInterfacesinComplexEnvironments. Proceedingsofthe50thAnnualMeetingoftheHumanFactorsandErgonomicsSociety. SanFrancisco,CA.October. Hameed,S.andSarter,N.(2009).ContextSensitiveInformationPresentation: IntegratingAdaptiveandAdaptableApproachestoDisplayDesign.Acceptedfor presentationatthe53rdAnnualMeetingoftheHumanFactorsandErgonomicsSociety. SanAntonio,Texas.October.

ContextSensitiveMultimodalDisplayDesign175
Ho,C.Y.andSarter,N.B.,2004.Supportingsynchronousdistributedcommunication andcoordinationthroughmultimodalinformationexchange.InProceedingsofthe48th AnnualMeetingoftheHumanFactorsandErgonomicsSociety,NewOrleans,LS. Hollnagel,E.andWoods,D.D.(2005).Jointcognitivesystems:Foundationsof cognitivesystemsengineering.BocaRaton,FL:CRCPress. ISO(InternationalOrganizationforStandardization),1998.ISO14915Multimedia userinterfacedesignsoftwareergonomicrequirements,Part3:Mediacombinationand selection,author. Jones,L.A.andSarter,N.(2008).TactileDisplays:CommunicatingThroughTheSkin. HumanFactors,50(1),90111. Klein,R.M.(2000).Inhibitionofreturn.TrendsinCognitiveSciences,4,138147. Kramer,G.,1993.Auditorydisplay:Sonification,audification,andauditory interfaces.PerseusPublishing. Kramer,G.,Walker,B.,Bonebright,T.,Cook,P.,Flowers,J.,Miner,N.,andNeuhoff, J.,1999.Sonificationreport:Statusofthefieldandresearchagenda.Reportpreparedfor theNationalScienceFoundationbymembersoftheInternationalCommunityfor AuditoryDisplay. Mollenhauer,M.A.,Hulse,M.C.,Dingus,T.A.,Jahns,S.K.,andCarney,C.,1997. DesigndecisionaidsandhumanfactorsguidelinesforATISdisplays.InY.I.Noy(Ed.), ErgonomicsandSafetyofIntelligentDriverInterfaces.LEA,Mahwah,NJ,pp.2361. Oviatt,S.L.(1999).Tenmythsofmultimodalinteraction.Communicationsofthe ACM,42(11),7481. Oviatt,S.,2002.Multimodalinterfaces.InJ.JackoandA.Sears(Eds.),Handbookof humancomputerinteraction.LEA,Hillsdale,NJ. Oviatt,S.L.,DeAngeli,A.&Kuhn,K.(1997).Integrationandsynchronizationofinput modesduringmultimodalhumancomputerinteraction.InProceedingsofConferenceon HumanFactorsinComputingSystems,CHI'97.NewYork:ACMPress. Raman,T.V.,2003.XFormsXMLpoweredwebforms.AddisonWesley. Oviatt,S.,&Olsen,E.(1994).Integrationthemesinmultimodalhumancomputer interaction.ProceedingsoftheInternationalConferenceonSpokenLanguage Processing,vol.2,551554. Reeves,L.M.,Lai,J.,Larson,J.A.,Oviatt,S.,Balaji,T.S.,Buisine,S.,Collings,P., Cohen,P.,Kraal,B.,Martin,J.C.,McTear,M.,Raman,T.V.,Stanney,K.M.,Su,H.,and Wang,Q.Y.,2004.Guidelinesformultimodaluserinterfacedesign.Communicationsof theACM,47(1):5759. ReuterLorenz,P.A.andRosenquist,J.N.(1996).Auditorycuesandinhibitionof return:Theimportanceofoculomotoractivation.ExperimentalBrainResearch,112,119 126. Sarter,N.B.(2007).MultipleResourceTheoryasaBasisforMultimodalInterface Design:SuccessStoriesandQualifications.InKramer,A.,Wiegmann,D.,andKirlik,A. (Eds.),Attention:FromTheorytoPractice(pp.187195).OxfordUniversityPress. Sarter,N.B.(2006).MultimodalInformationPresentation:DesignGuidanceand ResearchChallenges.InternationalJournalofIndustrialErgonomics(Invitedcontribution toSpecialIssueonNewInsightsinHumanPerformanceandDecisionMaking),36(5), 439445. Sarter,N.B.,2002.Multimodalinformationpresentationinsupportofhuman automationcommunicationandcoordination.InEduardoSalas(Ed.),Advancesinhuman performanceandcognitiveengineeringresearch.JAIPress,NewYork,pp.1336.

176SARTER,FERRIS,ANDHAMEED
Sarter,N.B.andHo,CY.(2004).IdentifyingNaturalPatternsofModalityUsageand Integration:ACriticalFirstSteptowardsSupportingSynchronousDistributedMultimodal CoordinationontheModernBattlefield.ReportpreparedfortheArmyResearch Laboratory(ARL)andCECOM.August. Spence,C.andDriver,J.(1998).Auditoryandaudiovisualinhibitionofreturn. Perception&Psychophysics,60(1),125139. Spence,C.andDriver,J.,1997.Crossmodallinksinattentionbetweenaudition, vision,andtouch:Implicationsforinterfacedesign.InternationalJournalofCognitive Ergonomics,1(4):351373. Spence,C.,McDonald,J.,andDriver,J.(2004).Exogenousspatialcuingstudiesof humancrossmodalattentionandmultisensoryintegration.InSpence,C.andDriver,J. (Eds.),CrossmodalSpaceandCrossmodalAttention.OxfordUniversityPress,NewYork, NY.277320. Sutcliffe,A.,2003.Multimediaandvirtualreality:Designingmultisensoryuser interfaces.LEA,Mahwah,NJ. Ward,L.M.(1994).Supramodalandmodalityspecificmechanismsforstimulus drivenshiftsofauditoryandvisualattention.CanadianJournalofExperimental Psychology,48,242259. Wickens,C.D.(1980).Thestructureofattentionalresources.InNickerson,R.(Ed.), AttentionandPerformanceVIII.LawrenceErlbaum,Hillsdale,NJ.239257.

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T A N D L A U R E L A L L E N D E R

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T AND LAUREL ALLENDER

This book, produced by the Partners of the Army Research Laboratory Advanced Decision Architectures Collaborative Technology Alliance, also serves as the Final Report for DAAD19-01-2-0009.

SECTIONII.PRESENTINGBATTLEFIELDINFORMATIONTOWARFIGHTERS

SECTIONII PRESENTINGBATTLEFIELDINFORMATIONTOWARFIGHTERS

122SECTIONII

Chapter

10

10. INTEGRATINGDIVERSEFEEDSTOEXTENDHUMANPERCEPTIONINTO DISTANTSCENES

ALEXMORISON,MARTINVOSHELL,PH.D.,AXELROSELER,PH.D., MAGNUSFEIL,JAMESTITTLE,PH.D.,DAVIDTINAPPLE,DAVIDD.WOODS, PH.D. TheOhioStateUniversity,Columbus,OH

INTRODUCTION

New technological capabilities create opportunities but also new challenges as people attempt to carry out even more demanding tasks in conflict and crisis situations. This chapter examines the impact of advances in sensing, networking, and robotic technologies. These capabilities allow people in remote locations to apprehend and explore aspects of the world as if they were standing in places that are too difficult, dangerous, or impossible to occupy. Investments in these technologies are predicated often on the belief that they will produce increased situation awareness for the remote human observer. However, as prototypes are rolled out, automation surprises occur which reveal the risk of unanticipated and unintended breakdowns in awareness (Woods and Hollnagel, 2006, McGuirl et al., 2009). For example, urban search and rescue workers using robots to search inaccessible areas of debris found that interpreting the video feeds was a slow deliberative process instead of automatic reliable perceptual experience (Casper and Murphy, 2003). It was difficult to accurately apprehend three-dimensional surface layout, the relative size/mass of objects, the robots ability to traverse obstacles, the robots speed, and the robots local orientation. The robots acted

177

178MORISONETAL.
as a stand-in for a remote human observer, mediating the observers ability to perceive and act in the scene of interest (in this case, the feed was a video image from a single, forward looking camera). As a result, the remote observer is partially decoupled from many of the information-rich properties of the scene that would be available if the observer were present in that scene. Similar findings have come from other recent cases of using robots in search tasks where remote observers experience serious difficulties to understand the distant scene being traversed by a robotic system (e.g., Darken et al., 2001). Based on such findings, we began a line of research to innovate new ways to bring distant feeds from robots and sensor nets into the range of human perceptual skills. The work began by exploring ways to break the tyranny of the flat framed screen as the default display medium in order to overcome the keyhole effect (Voshell et al., 2005). In exploring different possibilities, we were developing possible uses of 3D (three dimensional) visualizations. In this work we found that 3D technology actually represents control of point of view and shifts in perspective. This is in contrast to the usual approach which tries to compare the impact of 2D versus 3D displays (Tittle et al., 2002; Roesler et al., 2005). Other studies began to examine how the new sensing technology influenced coordination over multiple groups (connecting multiple perspectives) and commander decision making. In one study incident commanders viewed a video feed from a UAV as they responded, from a distance, to a simulated cascading chemical plant accident. The results showed that incident commanders overrelied on the video feed underutilizing other sources of information and missed important events embedded in the scenario (McGuirl et al., 2009). Emerging form this work was the realization that the key leverage point to achieve high performance human-technology systems was to explicitly coordinate multiple perspectives across multiple roles and scales (Woods et al., 2004; Klein et al., 2004). Multiple potential design solutions were developed based on this insight about how to extend perception (Woods et al., 2006) including concepts for

perspective folding for feeds from UGVs (Feil, 2004; Voshell, 2005) adaptive focus based on event recognition and spatial/temporal center-surround (Christoffersen et al., 2007; Davis et al, 2007; Morison et al., 2009) perspective control (Roesler, 2005; Morison, in preparation)

Seehttp://csel.eng.ohiostate.edu/productions/diversefeedsfordemon strationsoftheconcepts.

IntegratingDiverseFeedstoExtendHumanPerception179
In terms of human-robot interaction, extending perception concepts led to performance advantages arising from coordinating shared perspectives resulted in more efficient navigation, search, and rescue in a staged-world emergency response simulation (Voshell and Oomes, 2007). At a larger system scale, the challenge of synchronizing multiple perspectives over multiple roles and echelons in netcentric operations was examined using disaster response as an empirical test case (Klein et al., 2005; CSEL, 2007; Gunawan et al., 2007; Militello et al., 2007; Peffer et al., 2007; Voshell, 2007; Peffer et al., 2008; Prue et al., 2008; Trent et al., 2008; Voshell et al., 2008; McGuirl, 2009; Branlat et al., 2009). The chapter provides a roughly historical account of the research on the theme of extended perception conducted within the context of the ARL ADA collaborations. It begins with reframing the distinction between two-dimensional and three-dimensional displays based on past findings on human perception. These findings illustrate that in a three-dimensional environment the fundamental opportunity afforded by the new technology is control of point-ofview. With this insight in mind, the next section presents new design concepts to widen/overcome the keyhole typically found in HRI displays. Each of these designs leverage a three-dimensional virtual environment to coordinate perspective taking and even support multiple simultaneous perspectives. In assessing the advantage of the new display concepts in a navigation, search, and identify task, a novel metric for quantifying path complexity is introduced. A new display concept for adaptive focus is presented within a surveillance context. This design overcomes the keyhole of a single pan-tilt-zoom (PTZ) video camera with a narrow field-of-view (FOV). Another display concept expands control of point-of-view to enable multiple people to take advantage of a single feed from single robot. The design coordinates the perspectives of a robot handler, a rescue worker, and a robot in a search and rescue task. Included with this theme of coordinating shared perspectives, is the need to include synchronizing activity within netcentric operations. These netcentric operations include checkpoint operations in a military setting, urban fire fighting, and managing a convoy of walking wounded. The final section introduces a theory of perspective-taking in virtual and physical environments. Perspective-taking is the process of changing the point of observation relative to a scene in order to understand the properties, objects, actors, and activities that are of interest to the observer. Supporting perspectivetaking effectively is a key prerequisite to support people monitoring at a distance

180MORISONETAL.
via sensor networks. This theory defines properties of perspective in sensor mediated experience, necessary relationships between a human and a virtual point-of-view, and requirements for movement of the virtual point-of-view. In the subsequent section, these actionable constraints and requirements are then used to construct multiple instantiations of a perspective controller.

3DVIRTUALDISPLAYSAREVIEWPOINTDEPENDENT

New technology affords us the ability to produce three-dimensional displays, which led many to ask about possible performance advantages or costs relative to two-dimensional designs. In terms of visual perception and psychophysics, this question is more properly framed in terms of cues to depth and how well people can discriminate properties of scenes when the point of view on that scene is varied. A synthesis of studies of perceptual judgments related to threedimensionality including, distance, curvature, orientation, distance bisection, relative size, co-linearity or coplanarity, and discrimination of curved versus straight lines was carried out. Also included in the synthesis was studies of two judgment tasks for which there are no direct two-dimensional analog: local surface orientation and ordinal depth. These were included because they are potential sources of information in three-dimensional displays (Tittle et. al., 2001). One outcome of this review was a preliminary set of guidelines, illustrated in Table 1:, for new three-dimensional displays. The first important point of Table 1: is the large percentages for human perceptual judgments in the first three rows in the 3-D column. This indicates that three-dimensional figures should not be encoded as differences in magnitudes of depth, orientation, or curvature because discrimination of small changes will not be noticeable. Instead designers should try to encode important task dimensions within more qualitative threedimensional distinctions such as ordinal depth, deviations from co-planarity, and the presence of flat versus curved surfaces. This is illustrated by the small percentages in the last three rows of the 3-D column. But a more important outcome occurred when we tried to illustrate these perceptual judgment tasks by animating a continuous shift in perspective from a top down view (usually called 2D) and a line of sight view (usually called one type of 3D view). The label 2D versus 3D misrepresented the actual situation. The animations revealed (a) that 3D virtual displays are viewpoint dependent, and (b) providing control over the movement of perspective was a powerful cue (see Roesler et al., 2002, and http:// csel.eng.ohio-state.edu/productions/diversefeeds. Note the latter is basic for specialists in visual perception.). The fundamental principle is:

IntegratingDiverseFeedstoExtendHumanPerception181 Theviewfromanypointofobservationsimultaneously revealsandobscures.Woods,2006

Table 1: Comparison of two-dimensional vs. three-dimensional perceptual judgments

Moving the point of observationa viewtrackis critical to reveal what would otherwise be obscured from a single vantage point. No one point of observation will suffice for non-trivial scenes, tasks, and for moving observers. Rather, it is the ability to shift perspectives that provides humans with the power to understand a distant scene through the technology of sensor/robotic networks (an insight that goes back at least to Gibson and Carmichael, 1966), In other words, three-dimensional technology represents the opportunity for control of point of view (Roesler et al., 2002; Roesler, 2005; Morison, in preparation). To seize this opportunity first required a model of perspective taking for physical scenes of interest (see later section).

DESIGNINGTOWIDENTHEKEYHOLE

182MORISONETAL.
Field experience and naturalistic studies have shown that when access to the distant environment is mediated through sensors on robots, those in the robot handler role have difficulties comprehending the sensed environmentthey call their experience a soda straw view or what HCI would call a keyhole effect (Woods and Watts, 1997). The research team faced the challenge of how to overcome the keyhole to provide the remote observer a better way to apprehend the world that the robot is in.

PerspectiveFolding

New concepts were developed through an innovation process that generates design seeds based on a diagnosis of the challenges and opportunities (Roesler et al., 2001; Roesler et al., 2005). In this case, key generative ideas were escape the flat framed screen, use ideas about perspective taking, use techniques for overcoming keyhole effects in HCI. Dynamic demonstrations of the concepts are available at http://csel.eng.ohio-state.edu/productions/diversefeeds. The first display design seed, zoom stacking, integrates multiple views that differ by magnification. It is a different approach to provide a center-surround or focus+context display. As illustrated in Figure 10.1, the integration of multiple image scales through overlapping planes converts the conventional notion of zoom as approach towards or movement away into a shift in point of observation that reveals previously obscured structure.

Figure10.1 Zoomstacking,acentersurroundvariantforintegratingviewsbyusing multiplemagnifications.2009CognitiveSystemsEngineeringLabora tory,TheOhioStateUniversity,allrightsreserved.

IntegratingDiverseFeedstoExtendHumanPerception183

A second design seed was generated by searching for a means to provide optic flow cues for the remote observer as the robot moved through the scene of interest. This prototype used new image processing software to fold multiple views around a virtual point of observation represented by the robots position in the world. This is depicted in Figure 10.2. The multiple views were aligned by a central view pointing in the direction of motion and two pairs of side views angled at 45 and 60 degrees off of the axis of motion, analogous to a fovea, parafovea, and periphery (the implicit spherical coordinate system is part of the interface so that the observer could shift the orientation of the folded views as the robot moved through the space). By virtually folding the video panels around a virtual view-point and direction, an impression of the video feeds wrapping around the remote observer was created. This effect embedded the remote observer in the scene. A physical implementation on a pseudo-robotic platform showed that folding provided a variety of immediate advantages that helped the remote observer better understand the world the robot was moving through. These included optic flow cues to better apprehend the movement of the robot in the environment, less symptoms of the keyhole effect, and better ability to look ahead as direction of motion changes to support anticipation of future activities.

Figure10.2 Originalprototypeforperspectivefoldingtoovercomethekeyhole effect.2009CognitiveSystemsEngineeringLaboratory,TheOhioState University,allrightsreserved.

184MORISONETAL.
Further engineering trials were carried out within a game engine used to simulate tasks in urban operations (Voshell et al., 2005). This allowed manipulation of design variations in demanding tasks. As a result, the perspective folding concept evolved into a different configuration around the central viewa single pair of side views coupled with a vertically aligned pair of views. This is shown in Figure 10.3. The engineering trials showed that it was important to have the downward plane provide a visual cue to the relative height of the camera above the ground plane, and thus convey relative motion (Voshell, 2005). This version of perspective folding was compared with a single plane view with an equivalent field of view. The outcome indicated that simply enlarging the field of view of a single virtual camera does not overcome the keyhole effect. The folding mechanism provided a means to see the relationships between the robotic vehicle and the environment it is moving through. These cues helped to heighten spatial awareness.

Figure10.3 The5folddisplayprototypeusedintestinghumanrobotcoordinationin asimulatedsearch,andrescuetask.2005MartinVoshell,allrights reserved.

The multi-camera means to instantiate the folding concept introduced visual rifts across the multiple views, which could introduce perceptual burdens. As a result, the team explored means for producing continuous folding. This resulted in building and testing a physical working prototype of a new design seed, the Dynamic Perceptor Sphere, which provides a continuous physical folding from a single camera fixed in the direction of

IntegratingDiverseFeedstoExtendHumanPerception185
movement of the robot by means of a spherical coordinate system (see Figure 10.4). The concept is a spherical projection canvas in which the users viewpoint can shift off the camera axis, which is locked on the direction of robot movement. The surround display is created by projecting a single video feed from a fisheye lens located on the robotic platform onto a spherical canvas in order to reverse the circular, nonlinear, compressed image captured by the fisheye lens. The spherically projected fisheye imagery reproduces the complete panoramic image captured by the sensing device on the remote robotic platform. Independent of the exact implementation mechanism, the observer can shift point of view within the spherical projection even though the camera is fixed in the direction of robot movement. Since the processing to create the Dynamic Perceptor Sphere is all done at the remote observers display station, the same basic feed can be sent to different observers who can look offaxis independently as their role and task demands (the concept introduces no bandwidth constraints).

Figure10.4 TheDynamicPerceptorSphereproducedcontinuous"folding"bymeans ofasphericalcoordinatesystem(virtualillustrationoftheimplemented system).2004MagnusFeil,allrightsreserved.

186MORISONETAL. SimulationTestingandNewMetrics

In evaluating the 5-fold display, the research team used a game engine (Unreal) to allow an investigator to introduce various robotic platforms, sensors, and the ability to rapidly develop and test new interfaces, camera arrangements, and task situations in a virtual environment (Voshell, 2005). Participants navigated a virtual robot through a modified version of the National Institute of Standards and Technologies Orange reference test arena for autonomous mobile robots. A series of obstacles, ambiguities, and world geometry were added to make the navigation less trivial. Participants were inserted into a training environment and given one of three interfaces: single plane (45 FOV), 5-flat display (totaling 135 FOV), and 5-fold display (totaling 135 FOV). Each participant was familiarized with the interface, taught how to identify goal items (hazardous objects), and learned the joystick control. Upon completion they were placed within the modified NIST arena, and using the same interface from training, were instructed to find as many hazardous objects as quickly as possible. Subjects had one hour to complete the task before the robot lost all battery power. To overcome the deficiencies of traditional measures (e.g., mean number of items collected and mean completion time show significant inter-subject variance and insensitivity of the measures to the intricacy and complexity of the task environment) Phillips and Voshell (2006) developed a fractal scoring metric sensitive to differences in navigation efficiency. The fractal dimension is a score from 1 to 2 that captures the simplicity or complexity of a path path tortuosity. A score closer to 2 indicates a more complex path. Fractal score can reveal whether a handler/robots search remains efficient as task demands change. The simulation included re-tasking situations, i.e., after using the robot to explore a portion of the environment for targets, a robot handler is given a new task that requires the robot to move through previously explored spaces to move to a different part of the test arena. The question is how well can the robot handler re-navigate to find the their way to the physical area associated with the new tasking (tests how much the robot handler has learned about the space previously navigated). Figure 10.5 provides a sample of the performance difference in this task with a perspective folding display (left path and fractal score near 1) versus a single camera view (top and right panels with a larger fractal score). The fractal score concisely captures the navigation difficulties of one participant using the single camera fixed in the direction of robot movement and indicates that perspective folding reduced the keyhole effect.

IntegratingDiverseFeedstoExtendHumanPerception187

Figure10.5 Humanrobotnavigationfollowingretaskingwithdifferentdisplays;the fractalscorecapturesthepathtortuosityandconciselyindicatesperfor mancedifferences.2009CognitiveSystemsEngineeringLaboratory, TheOhioStateUniversity,allrightsreserved. OvercomingtheKeyholeEffectinVideoSurveillance

New sensing capabilities can take the form of surveillance networks not simply individual robotic vehicles. Current means for coupling people to large networks of spatially distributed sensors creates multiple keyhole views into the environment being surveilled. Although multiple sensors exist within the network, often the spatial relationships between sensors are unknown.

188MORISONETAL.

Figure10.6 Displayconceptforvideosurveillanceusingastitchedpanorama,overlay brightnessactivitymap,andalgorithmgeneratedscanpathway(From Morisonetal.,2009)2009CognitiveSystemsEngineeringLaboratory, TheOhioStateUniversity,allrightsreserved.

A new display concept was developed for supervisory visualizations of smart surveillance systems for a single smart PTZ camera (Morison et al., 2009). The surveillance task is monitoring a large viewable physical area with a PTZ camera (i.e., a 54 FOV). The viewable area for the sensor is 360 in pan (azimuth) and -90 in tilt (elevation). The visualization is based on a panoramic frame of reference that captures the entire viewable area for the PTZ camera. This is illustrated in Figure 10.6. The base panorama is overlaid with both a brightness-coded activity map and a algorithm generated scan-path. The activity map is a spatial representation of the output from a single smart algorithm monitoring for human activity patterns. In this instance, smart algorithm refers to any unsupervised algorithm that detects spatial and temporal properties of a scene from a sensor. The scan-path represents the virtual pathway of the smart PTZ camera across the entire viewable area.

IntegratingDiverseFeedstoExtendHumanPerception189
For the case of a smart PTZ camera in a surveillance task, we conceptualize the visualization opportunity created by computer vision algorithms as: (a) supporting the out-of-scene agents ability to take virtual patrols as if they were exploring a continuous space, and (b) integrating the structure of activity and events in the monitored physical scene. The visualization design, for the case of a single PTZ camera, requires a visible spatial frame of reference that surveillance personnel can modify, i.e., is directable (Woods and Hollnagel, 2006). This visualization redefines the unit of analysis for surveillance from a sequence of images from a camera to virtual pathways or patrols through the viewable space. The scan-path panoramic display support these virtual pathways through spatial-, temporal-, and activity-based frames of reference. The previous two examples, perspective folding for a robotic platform and panoramic visualization for surveillance, show overcoming the keyhole by coordinating perspectives between an operator or supervisor and a sensor platform. Both assume the interaction is between a single person and single sensor platform. However, in operational environments such as the military and emergency response interaction often occurs across multiple human roles and multiple sensor platforms.

CoordinatingSharedPerspectives

A key finding of work on HRI is that coordination between human and robot is critical to the success of any human-robot system (Woods et al., 2004; Woods and Hollnagel, 2006). At a minimum at least two human roles exist and must be planned for: the robot handler and the problem holder. Fieldwork experience indicates that robot handlers spend significantly more time gathering information about the state of the robot and the state of the environment than actually navigating the robot (Burke et al., 2004). The problem holder roles are fulfilled by the search and rescue personnel (task force leader, structural, search, canine, medical etc) whose goals lie in trying to characterize the search situation and achieve rescue goals (Casper and Murphy, 2003). The problem holders, handler, and robot must be able to work together as effective team members to coordinate their activities and achieve operational goals (as represented graphically in Figure 10.7). Designing for coordination over these roles has been shown to improve the ability of the human-robot system to be more resilient in demanding situations (Voshell and Oomes, 2006).

190MORISONETAL.

Figure10.7 CoordinatingsharedperspectivesforHRI(fromWoodsetal.,2006). 2009CognitiveSystemsEngineeringLaboratory,TheOhioStateUni versity,allrightsreserved. SynchronizingActivityOverMultipleEchelonsInNetcentricOperations

The coordination and sharing perspectives themes led to another line of work that scaled up these issues to the level of mission in netcentric operations that connect personnel and robots in the scene of action to other echelons of control. This research defined the conditions for coordinated joint activity among distributed agents within systems composed of humans and automation (Klein et al., 2004; Klein et al., 2005; CSEL, 2007). The results revealed that netcentric operations create dynamic interdependencies between different groups, roles, activities, and locations. Coordination encompasses the various mechanisms that allow parts of the system (sub-systems or agents) to manage interdependencies between their roles and tasks, and conflicts between their goals as the situation changes. As situations cascade, role interdependencies become difficult to manage and groups can work at cross purposes (Branlat et al., 2009). The research used three different kinds of situations in order to develop the concept of synchronization loops (based on collaborations with CSEL, West Point, and Resilient Cognitive Systems, Inc.). Through a walking wounded scenario in crisis management, (Gunawan et al. 2007) the concept of Synchronized Coordination Loops was developed as a means to improve evacuation of wounded persons from a hot zone (the idea is based on an analogy to the role of voice loops in space shuttle mission control, Patterson et al., 1999). The

IntegratingDiverseFeedstoExtendHumanPerception191
concept also was explored in timely coordination of information from sensors, robots and personnel at different echelons in the checkpoint operations (Tittle et al., 2006). Finally, coordination loops were investigated in the context urban fire fighting (Tittle et al., 2007; Peffer et al., 2007; Voshell et al., 2008; Prue et al., 2008; Peffer et al., 2008; Trent et al., 2008; Branlat et al., 2009). The concept of perspective recurs throughout the line of work. As a result, another project set out to develop a theory of perspective taking as new sensing and robotic capabilities allow remote observers the ability to monitor distant worlds (the scene of interest) as if they were present (Woods and Sarter, 2009).

FUNDAMENTALPRINCIPLESFORPERSPECTIVETAKING

The R&D on overcoming the keyhole in human-robot interaction, supporting surveillance in distributed sensor networks, coordinating shared perspectives, and challenges in synchronizing interdependent activities in netcentric operations all included concepts about perspective. In addition, the technology developments created new opportunities for remote observers to be able to observe scenes from points of observation that were difficult, dangerous, or impossible previously. As a result, a new model for perspective-taking in virtual and distant environments was developed based on well established relationships of perspective projection Only a brief overview of fundamental concepts related to perspective taking is possible here plus the relationships modeled are inherently spatial and interactive (i.e., beyond the capability of paper as a medium). To better appreciate the complex relationships, the interested reader should examine the videos provided at http://csel.eng.ohio-state.edu/diversefeeds. Past methods of interaction between a human and three-dimensional virtual environments were based on a static screen and view-point control hardware such as a joystick, mouse, or virtual reality display e.g. for gestures. More recently, increases in the number and kind of three-dimensional virtual environments, such as video games, google earth, and GIS systems, have created an explosion of view-point controllers that provide varying degrees of freedom, types of motions, and different physical realizations, such as video camera controllers, six degree-of-freedom controllers, and three-dimensional mice. Note that with 3D visualization and new sensor capabilities view-point or perspective control must be provided. However, none of the current controllers are founded on a model of the necessary relationships between a person as observer, a virtual view-point, and a virtual or distant physical scene of interest. In other words, the

192MORISONETAL.
forms of perspective control are accidental or implicit in both the choice of control device and the manner in which the degrees of freedom map onto movement of the virtual viewpoint (virtual camera position) relative to the scene. The results for performance and capability range broadly from cumbersome limited exploration ability to general confusion. To take advantage of the new opportunity, a model of perspective-taking is needed to define the appropriate mappings. Consider your current view-direction reading this chapter, if you move your head to the right or to the left a corresponding change in view-direction occurs (this is also true while moving your eyes however for now assume you are looking in the direction your head is pointing). If you imagine a point of rotation in the center of your head and a line that starts at that point and passes through your eyes; this is your line of sight or view-direction vector relative to your current body position (e.g., sitting in a chair, standing upright, or pacing). This is an in-scene perspective looking out from a sensor at a location to explore the world around that sensor position (e.g., the dynamic perceptor sphere for HRI; Figure 10.4). Now, imagine a second point of rotation exists, external to you, at the center of the current page or computer screen you are reading. If you rotate your head around this center point, but do not adjust your body or head orientation to compensate, the point of rotation will move in your field of view. For example, if you rotate around the external point of rotation (e.g. to your left or clockwise from a top-down view), you will be able to bring into view objects on your desk that were occluded from your view by the computer screen. The point on the screen that serves as a point of rotation exists in the world and moving the sensors (your eyes) around that point defines a sphere of possible viewpoints. Flat text on a flat computer screen is a very impoverished scene of interest so in the basic demonstrations of perspective taking we use a cube as the simplest form of a scene of interest in the world. As you rotate viewpoint around the cube (for most tracks), the resulting image shifts from showing a single face of the cube (one-point perspective) to seeing portions of 2 faces of the cube (two-point perspective) to seeing portions of 3 faces of the cube (three-point perspective) to seeing 2 faces (portions of a new face), and then to seeing a different single face of the cube (see the animations at http://csel.eng.ohio-state.edu/diversefeeds). Each of the above points-of-rotation create spherical coordinate systems that capture two distinct, but related, viewing types. When people are remote from the scene themselves and able to investigate the world through mobile or networks of sensors, both types are necessary for humans to explore and maintain comprehensibility of the distant environment. However, current view-point controllers do not explicitly support these two types of perspective: a viewer-centered perspective (reorientation about the imaginary

IntegratingDiverseFeedstoExtendHumanPerception193
point located within your head) and a perspective centered external to the viewer (centered at an object or in the scene of interest). The model of perspective taking specifies key principles and relationships that should guide explicit design of perspective control to enhance comprehensibility as remote observers explore distant worlds through sensor nets. Spherical Coordinate Systems. The perspective-taking model utilizes two spaces each defined by a spherical coordinate system. In general, a spherical coordinate system is defined by three parameters: a pan, a tilt, and a radius. In addition, all spaces have a distinct origin that can be described as either the intersection of the pan and tilt axes or the location of zero-radius. These two spherical coordinate systems define a view-direction space (sensor or observer centered space i.e., inside out) and a view-position space. The view direction sphere is best represented in Figure 10.4 which we refer to as the perceptor sphere. Note, however the continuous spherical folding in Figure 10.4 represents only half the sphere in the direction of motion. The view-position space, which we refer to as the perspective sphere, is depicted in rows two and four of Figure 10.8 The perspective sphere represents all of the positions (points of observation) a virtual camera or an observer could take in order to view an area of the physical world. The spheres also represent constraints about how the virtual camera can move smoothly and continuously with respect to a point of rotationeither the observers aim point in the scene or the position of the sensing mechanism relative to the scene. The spherical coordinate systems naturally represent the relationship between viewer and scene viewed. These concepts begin to provide the machinery to be able to harness of the power of perspective-taking. The relationships defined in the perspective-taking model specify what relationships an apparatus for perspective control must support for comprehension of a distant scene. Simultaneously Reveals and Obscures. The first relationship in perspective-taking is that moving the virtual camera relative to the scene of interest affects the perspective in the captured image. This is illustrated by rows one and two together in Figure 10.8. More precisely, the image in row two shows the current position of the virtual camera and the corresponding view of the scene (i.e., what is currently revealed) and in row one is the image of all structure currently obscured by the current view. During relocation of the camera along a viewtrack, the planar projection of the same scene is subject to continuously change. As the point of observation moves different portions of the scene are revealed or get obscured. Hence, it is through shifting perspective that observers come to comprehend activities and status of the scene of interest. This fundamental principle of perspective leads to the following.

194MORISONETAL.
Always Two Views in Parallel. In remote perception, the link is broken between observers location and the location of the point of observation (i.e., the sensor) in the world. The remote observer sits at a location in front of the display while the remote camera at a distance changes its location and orientation relative to the environment under observation. This disconnect can make it difficult for the remote observer to apprehend the relationship between camera and scene that is the subject of observation. The principle is to provide, in parallel, the view of the scene from the virtual camera position and the view of the virtual camera relative to the scene and relative to the other positions the virtual camera could assume. This is depicted in row two of Figure 10.8, and is contrasted with the view, only from the camera, represented in the image in row one. The perspective sphere (the view position sphere) fulfills this requirement for a representation of how to shift perspective relative to the scene. Moving Point of Observation. To overcome the inherent limit of the view from any single point of observation, the remote observer need to be able to smoothly and continuously shift point of observation. This point led to the original concept of viewtracks for 3D visualizations (Roesler et al., 2002). Again the perspective sphere captures regularities about shifting perspective so that the observer can simply explore the world of interest by looking at what is interesting or reorienting where they look as they search for interesting events. The general structure can be annotated with specific viewtracks to facilitate movements along paths that are frequently used or informative (e.g. the scan path notion used in the surveillance work; Morison et al., 2009). The model for perspective-taking also supports perspective shifts between inside-out (as if the remote observer were positioned at the sensor position in the distant world) and outside-in classes of perspective (the two spherical coordinate systems). The distant view affords the identification of prospective in-scene points of observation from where informative perspectives on the stage can be obtained.

IntegratingDiverseFeedstoExtendHumanPerception195

Figure10.8 Propertiesofperspectiverelativetothesceneofinterest. 2009CognitiveSystemsEngineeringLaboratory,TheOhioStateUni versity,allrightsreserved.

Center/Surround Structure. The last property we will address here is the principle of center/surroundprovide a high resolution central view with a larger (even if lower resolution) surround area. This principle has been found to be critical for exploration in virtual environments repeatedly. The surround area supports the need for re-orienting, i.e., to shift perspective in the search for what are interesting structures or activities across al of the scene of interest. This principle was in evidence in the work to break down the keyhole in HRI and in surveillance described earlier.

196MORISONETAL. MechanismsForPerspectiveControl

The model of perspective-taking outlines a set of relationships and principles for connecting controlling point-of-view of the remote observer to the control of the multiple sensors deployed into the physical scene of interest. Figure 10.9 contains pictures of three different realizations of the perspective-taking model, that is, three different devices based on the fundamental relationships of perspective taking in spherical coordinate systems. The controller in Figure 10.9a is a design concept for a viewpoint controller that creates a specific path transitioning from in-scene to a distant view and vice versa. The controllers in Figure 10.9b and Figure 10.9c are working devices that control multiple PTZ cameras (fixed and mobile, indoor and outdoor). Figure 10.9b shows a body-scaled realization where the monitor displays a centersurround view and moving the monitor around the sphere and changing the sphere diameter allow a person to view the scene from any perspective the sensor net can provide. Figure 10.9c shows a smaller scale realization in the form of a 3D joystick. Both of these two controllers were built to provide a degree of freedom that corresponds to view direction. This ability to adjust view direction provides two additional capabilities that are advantageous for interacting with a distributed sensor network, including the ability to look off axis from a selected perspective position and, more significantly, the ability to embody not only a perspective sphere, but the view direction sphere as well. Differences between the perspective controller in Figure 10.9b and Figure 10.9c include differing scales (i.e., body scale versus hand scale), the position of the screen, and the enhanced simplicity of shifting between outside-in and inside-out perspectives for the 3D joystick. Across each instantiation of controller several properties are maintained, including a fixed point of rotation that serves as the origin for the perspective sphere, a pan and tilt movement with respect to the origin, and the ability to translate a point away from or towards the origin. These constraints are dictated by the perspective-taking model for relating a local observers perspective to a distant point of observation with respect to any scene of interest. From a control theory perspective, the perspective-taking model requires the perspective sphere and view direction sphere be a position control.

IntegratingDiverseFeedstoExtendHumanPerception197

Figure10.9 (a)Multipleinstantiationsoftheconceptforperspectivecontrol:upper leftistheoriginaldesign;(b)rightisabodyscaledworkingprototype; (c)lowerleftisahandscaledworkingprototypeor3Djoystick.Thework ingprototypesinstantiatethekeyrelationshipsspecifiedintheperspec tivetakingmodeltolinkmovementsofthecontrollerinaspherical coordinatesystemtomovements/selectionofsensorsinthedistant world.2009CognitiveSystemsEngineeringLaboratory,TheOhioState University,allrightsreserved,allrightsreserved.

This chapter documents a journey of innovation that began with identifying the new complexities that accompanied new technology into difficult operational contexts. Studies of how these complexities occurred led to diagnoses and promising design directions applicable to human-robot interaction, smart surveillance systems, and coordinated activity in netcentric operations. The work to overcome the keyhole effect and related difficulties in using new technology led to insights about new opportunities to harness the power of new technology for extending human perception into distant scenes perspective control. Developing new devices for perspective control required progress on models of perspective taking as a fundamental capability of humantechnology systems.

198MORISONETAL. REFERENCES
Branlat,M.,Fern,L.,Voshell,M.andTrent,S.A.(2009).Adaptationand CoordinationinDistributedAnomalyResponse:AStudyofAccidentsinUrban Firefighting.Submitted. Burke,J.L.,Murphy,R.R.,Coovert,M.D.,&Riddle,D.L.(2004).Moonlightin Miami:Afieldstudyofhumanrobotinteractioninthecontextofanurbansearchand rescuedisasterresponsetrainingexercise.HumanComputerInteraction,19,85116. Casper,J.andMurphy,R.R.(2003).Humanrobotinteractionsduringtherobot assistedurbansearchandrescueresponseattheWorldTradeCenter.IEEETransaction onSystems,ManandCyberneticsPartB:Cybernetics,Vol33,No.3,pp367385. Christoffersen,K.,Woods,D.D.andBlike,G.T.2007.DiscoveringtheEventsExpert PractitionersExtractfromDynamicDataStreams:ThemUMPTechnique.Cognition, Technology,andWork,9,8198. CSEL2007.ConceptsinDistributedWorkandCoordination:CommonGround, ConvoyProblem,OpenWorkspace.Multimediaproduction,CognitiveSystems EngineeringLaboratory,TheOhioStateUniversity,http://csel.eng.ohiostate.edu/ productions/CoordinationProjects/Coordination%20Projects.html Davis,J.W.Morison,A.andWoods,D.D.2007.AnAdaptiveFocusofAttention ModelforVideoSurveillanceandMonitoring.MachineVisionandApplicationsJournal, 18(1),4164. Darken,R.,Kempster,K.andPeterson,B.2001.EffectsofStreamingVideoQuality ofServiceonSpatialComprehensioninaReconnaissanceTask.Proceedingsof Interservice/Industrytraining,Simulation&EducationConference(I/ITSEC),NTSA, Arlington,VA(http://www.iitsec.org/proceedings.cfm). Feil,M.2004.TheDynamicPerceptorSphere:ConceptsofFunctionalPresencein HumanRobotCoordination.MSThesis,TheOhioStateUniversity,August2004. Gunawan,L.T.,Voshell,M.,Oomes,A.H.J.andWoods,D.D.2007.Envisioning CollaborationataDistancefortheEvacuationofWalkingWounded.Proceedingsofthe 4thInternationalISCRAMConference(B.VandeWalle,P.BurghardtandC.Nieuwenhuis, eds.),Delft,theNetherlands,May2007. Klein,G.,Woods.D.D.,Bradshaw,J.,Hoffman,R.R.,andFeltovich,P.J.,2004.Ten ChallengesforMakingAutomationaTeamPlayerinJointHumanAgentActivity.IEEE IntelligentSystems,19(6),9195. Klein,G.,Feltovich,P.,Bradshaw,J.M.andWoods,D.D.2005.CommonGround andCoordinationinJointActivity.InW.RouseandK.Boff(Ed.).Organizational Simulation,Wiley,pp.139178. Hoffman,R.R.andWoods,D.D.2005.TowardaTheoryofComplexandCognitive Systems.IEEEIntelligentSystems,20(1),7679. McGuirl,J.M.,Sarter,N.B.andWoods,D.D.(2009).SeeisBelieving?Theeffectsof realtimeimagingonDecisionMakinginaSimulatedIncidentCommandTask. InternationalJournalofInformationSystemsforCrisisResponseandManagement,1(1), 5469. Militello,L.G.,Patterson,E.S.,Bowman,L.,andWears,R.(2007).InformationFlow DuringCrisisManagement:Challengestocoordinationintheemergencyoperations center.Cognition,TechnologyandWork,9(1),2531.

IntegratingDiverseFeedstoExtendHumanPerception199
Morison,A.,Davis,J.W.andWoods,D.D.2009.VisualizationstoSupportHuman SupervisionofIntelligentCamerasinSurveillance.ProceedingsoftheHumanFactorsand ErgonomicsSociety. Morison,A.inpreparation.MechanismstoEnablePerspectiveControlforExtending HumanPerceptionintoDistantScenesthroughSensorNetworks.Dissertation.TheOhio StateUniversity.ColumbusOH. Patterson,E.S.,WattsPerotti,J.C.andWoods,D.D.(1999).VoiceLoopsas CoordinationAidsinSpaceShuttleMissionControl.ComputerSupportedCooperative Work,8,353371. Peffer,J.,Tittle,J.andVoshell,M.2007.CoordinationLoopInsightsfroma FunctionalAnalysisofEmergencyManagementDomain.JointReportResilientCognitive Systems,PittsburghPAandCognitiveSystemsEngineeringLaboratory,TheOhioState University. Peffer,J.,Tittle,J.,Gualtieri,J.W.,Elm,W.C.,Voshell,M.,Prue,B.andWoods,D.D. 2008.HowcostlyisyourC2Coordination?Assessingthecoordinationrequirements withinCommandandControl.13thICCRTS:C2forComplexEndeavorsBellevue,WA., June2008. Phillips,F.andVoshell,M.2006.ANovelMetricforEvaluatingHumanRobot NavigationPerformance.RTA/HFMSymposium:HumanFactorsofUninhabitedMilitary VehiclesasForceMultipliers,Biarritz,France,911October2006. Prue,B.,Voshell,M.,Woods,D.D.,Peffer,J.,Tittle,J.andElm,W.2008. Synchronizedcoordinationloops:Amodelforassessingdistributedteamwork.NATO HFMSymposiumonAdaptabilityonCoalitionTeamwork,Copenhagen,April2123,2008, RTOMPIST999. Roesler,A.,Tittle,J.andWoods,D.D.2002.ViewTracks:3DVirtualDisplaysare ViewpointDependent.MediaPaperCD.CognitiveSystemsEngineeringLaboratory,The OhioStateUniversity,ColumbusOH,April2002. Roesler,A.2005.ANewModelforPerspective:TheRoleofPointofObservationin VirtualandRemotePerspectiveTaking.Dissertation.TheOhioStateUniversity, ColumbusOH. Roesler,A.andWoods,D.D.2007.DesigningforExpertise.InH.N.J.Schifferstein& P.Hekkert(Eds.),ProductExperience:AMultidisciplinaryApproach(pp.215237). Amsterdam:Elsevier. Schoenwald,J.,Trent,S.,Tittle,J.andWoods,D.D.2005.ScenariosAsAToolFor CollaborativeEnvisioning:UsingTheCaseofNewSensorTechnologiesforMilitaryUrban Operations.ProceedingsoftheHumanFactorsandErgonomicsSociety49thAnnual Meeting.2628September,OrlandoFL.url:http://csel.eng.ohiostate.edu/productions/ xcta Tittle,J.,Woods,D.D.,Roesler,A.,Howard,M.,Phillips,F.2001.TheRoleof2Dand 3DTaskPerformanceintheDesignandUseofVisualDisplays.Proceedingsofthe HumanFactorsandErgonomicsSociety45thannualmeeting.812October,Minneapolis, MN. Tittle,J.,Roesler,A.,andWoods,D.D.2002.TheRemotePerceptionProblem. ProceedingsoftheHumanFactorsandErgonomicsSociety46thAnnualMeeting.15 October,Baltimore,MD. Tittle,J.,Elm,W.andWoods,D.D.2006.TheLayeredCheckZone:ACognitive SystemsEngineeringApproachtoIntegratingSensorandRoboticSystemsintothe CheckpointMission.JointReport,ResilientCognitiveSystems,PittsburghPAand CognitiveSystemsEngineeringLaboratory,TheOhioStateUniversity.

200MORISONETAL.
Tittle,J.,Peffer,J.,Elm,W.,Voshell,M.,andPrue,B.2007.AnalysisofCoordination LoopsfortheEmergencyManagementDomain:MetricsInsights.JointReportResilient CognitiveSystems,PittsburghPAandCognitiveSystemsEngineeringLaboratory,The OhioStateUniversity. Trent,S.,Voshell,M.,Fern,L.andStephens,R.2008.DesigningtoSupport CommandandControlinUrbanFirefighting.Proceedingsof13thInternational CommandandControlResearchandTechnologySymposium:C2forComplexEndeavors. BellevueWA,June,2008 Voshell,M.G.,Woods,D.D.andPhillips,F.2005.OvercomingtheKeyholein HumanRobotInteraction:SimulationandEvaluation.ProceedingsoftheHumanFactors &ErgonomicsSociety49thAnnualMeeting.2628September,OrlandoFL. Voshell,M.G.2005.OvercomingtheKeyholeEffectinHumanRobotInteraction. MSThesis,TheOhioStateUniversity,ColumbusOH. Voshell,M.andOomes,A.H.J.2006.Coordinating(Shared)PerspectivesinRobot AssistedSearch&Rescue.Proceedingsofthe3rdInternationalISCRAMConference(B. VandeWalleandM.Turoff,eds.),Newark,NJ(USA),May2006. Voshell,M.,Trent,S.,Prue,B.,Fern,L.2008.CultivatingResilienceinUrbanFire Fighting:Supportingskillacquisitionthroughscenariodesign.Proceedingsofthe51st AnnualMeetingoftheHumanFactorsandErgonomicsSocietyNewYork,NY,Sept.22 26,2008. Voshell,M.andWoods,D.D.(2009).PlanningSupportforRunningLargeScale ExercisesasLearningLaboratories.CognitiveSystemsEngineeringLaboratory,Institute forErgonomics,TheOhioStateUniversity,ColumbusOH.March2009,Preparedfor ArmyResearchLaboratory. Woods,D.D.andWatts,J.C.(1997).HowNotToHaveToNavigateThroughToo ManyDisplays.InHelander,M.G.,Landauer,T.K.andPrabhu,P.(Eds.)Handbookof HumanComputerInteraction,2ndedition.Amsterdam,TheNetherlands:Elsevier Science. Woods,D.D.,Tittle,J.,Feil,M.andRoesler,A.2004.EnvisioningHumanRobot CoordinationforFutureOperations.IEEESMCPartC,34(2),210218. Woods,D.D.,Davis,J.W.andChristoffersen,K.2005.EventTemplateHierarchies asMeansforHumanAutomationCollaborationinSecuritySurveillance.Proceedingsof theHumanFactorsandErgonomicsSociety49thAnnualMeeting.2628September, OrlandoFL. Woods,D.D.,Voshell,M.,Roesler,A.,Phillips,F.,Feil,M.andTittle,J.2006.TheLaw ofStretchedSystemsinAction:ExploitingRobots.Podcastavailableathttp:// csel.eng.ohiostate.edu/podcasts/woods/ Woods,D.D.andHollnagel,E.2006.JointCognitiveSystems:PatternsinCognitive SystemsEngineering.BocaRatonFL:Taylor&Francis.

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T A N D L A U R E L A L L E N D E R

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T AND LAUREL ALLENDER

This book, produced by the Partners of the Army Research Laboratory Advanced Decision Architectures Collaborative Technology Alliance, also serves as the Final Report for DAAD19-01-2-0009.

SECTIONII.PRESENTINGBATTLEFIELDINFORMATIONTOWARFIGHTERS

SECTIONII PRESENTINGBATTLEFIELDINFORMATIONTOWARFIGHTERS

122SECTIONII

Chapter

11

11. ANINTERACTIVEDECISIONSUPPORTARCHITECTUREFORVISUALIZING ROBUSTSOLUTIONSINHIGHRISKMILITARYENVIRONMENTS

SHAUNHUTCHINS AlionScienceandTechnology,Boulder,CO JOHNR.JOSEPHSON,PH.D TheOhioStateUniversity,Columbus,OH MICHAELBARNES ArmyResearchLaboratory,FortHuachuca,AZ B.CHANDRASEKARAN,PH.D TheOhioStateUniversity,Columbus,OH PATRICIAL.MCDERMOTT AlionScienceandTechnology,Boulder,CO TIMOTHYN.MILLER TheOhioStateUniversity,Columbus,OH

INTRODUCTION

201

202HUTCHINSETAL.
The economic climate in 2009 is an important reminder of the crucial role that risk perception and risk misperception play in decision making that impacts everyday life. Many of the economic decisions such as percent of acceptable subprime mortgages were actually based on complex statistical models using overly optimistic assumptions (Krugman, 2009). Military decision making is also replete with examples of overconfidence in the assumptions of decisions, i.e., assumptions that are the product of risk perception and/or risk misperception by otherwise competent commanders that ultimately result in disaster. Examples include McArthurs refusal to believe the Chinese would attack across the Yalu River during the Korean conflict, Montcalms decision in 1759 not to defend his rear flank against Wolfe because of the supposedly impregnable cliffs of Quebec City, and the Maginot Line fiasco in World War II (Keegan, 1995). Our research focuses on instances where reality does not match the assumptions upon which decisions were made. We developed decision tools in the form of graphical visualizations and human-computer interaction methods that would help junior commanders understand the consequences of various courses of action (COAs). Decisions that result in successful outcomes under a variety of supportable assumptions operationally define our concept of robustness (Chandrasekaran, 2008). In Chandrasekaran (2008) it is argued that, in contrast to the traditional emphasis on optimality, planning should aim to produce plans that are robust, i.e., plans that have satisfactory performance even when reality fails to match the assumptions and models used in planning. Our decision tools do not lead to a single optimal solution but rather to a set of possible solutions. The final decision is left to the commander based on his/her instincts and weightings of various military criteria (e.g., speed versus sensor coverage). Our tools are designed to impart insight rather than be prescriptive in nature. Most traditional decision research approaches model uncertainty in the world probabilistically and offer solutions that are adequate on average. The traditional approaches also assume a defined probability space and a rational actor. Both assumptions can be misleading. First, much of the uncertainty in the world cannot be modeled in a fully defined problem space. Second, Kahneman and Tversky (1979) and other researchers in behavioral economics (Camerer and Lowenstein, 2002) conducted numerous studies indicating that humans do not necessarily follow normative models. In general, the results indicate humans use bounded rational solutions or heuristics that are expedient but also subject to bias. In particular, humans often misinterpret and misuse probabilistic estimates related to risky decisions (Johnson-Laird et.al., 1999; Wickens & Holland, 2000). The point is not that humans make poor decisions; rather, they usually do not make them based on defined probability spaces. The simple probability models that define the problem spaces may be inappropriate, or even incomplete, for complex dynamic systems. In many combat situations outcome

AnInteractiveDecisionSupportArchitecture203
probabilities change rapidly, requiring flexible real-time replanning as the situation unfolds rather than dependence on a single optimal solution (Bar-Yam, 1997). Robust decisions remain sufficient even when reality is not as expected because robust decision making acknowledges that models of the world are incomplete and human decision making is error-prone. The research described is an attempt to understand the interplay between robust solutions, normative models, and the ability of the military planner to understand the risk environment. Our research assumes the commander will base his/her decisions on multiple cues with relative importance that will change as the combat environment changes. Based on that assumption, we anticipate commanders will develop more robust solutions if they are able to understand various decision outcomes and their consequences. We additionally expect that planning time will be reduced with tools that allow the operator to filter unreasonable solutions, circumventing the computational explosion inherent in any decision tree. The uncertainty of the real world poses a variety of challenges in the design of decision support systems (DSS). Decision support systems attempt to mediate uncertainty through decision tools such, such as those examined, here toward robust decisions. Contingency planning against the most likely events or intentions is sometimes appropriate for dealing with uncertain situations. Other times, uncertainty may be addressed by simply exploring the robustness of selected plans and understanding how the plan may perform if reality does not quite match the planning assumptions. For example, a robust plan would provide good coverage even when the actual enemy avenue of approach turns out to be different from that assumed for planning. A plan is also robust if it provides good coverage when some of the sensors fail, either en route or during surveillance. The risks and uncertainties need to be presented to the planner in a way that is meaningful and protective against biases. While decision support is often couched in terms of making an optimal decision, helping the decision maker understand the decision space (i.e., develop a sense of relations between decision variables and outcomes in different parts of the decision space) is equally, if not more important. If after the selected plan is implemented external events call for rapid replanning, then the planner can exploit his/her understanding of the decision space to rapidly and effectively respond. To investigate the concept of robustness for risk management, we chose a sensor allocation problem. In addition to being a vehicle for exploring robust decision making, the problem is also intrinsically important militarily and relates to other problems such as persistent surveillance, border incursions, and force protection. The sensor allocation problem is a good application of one of our research goals: how to present interactive information to planners that allows them to understand the consequences of various courses of action while maintaining flexible options in cases of sudden combat reversals.

204HUTCHINSETAL.

TheSensorAllocationProblem

New technological developments, such as new types of sensors, unmanned ground vehicles (UGVs), and unmanned aerial vehicles (UAVs) that can be used to cover relatively large areas, have increased both the opportunities and challenges in persistent surveillance, a key contributor to information dominance (superiority in the generation, manipulation, and use of information sufficient to afford its possessors military dominance, Libicki, 1997). In a typical scenario, a commander might request an allocation plan for a newly named area of interest (NAI). The allocation plan will typically need to satisfy multiple, often conflicting, criteria. While coverage (the percent of NAI covered) is important, other outcomes of interest include timeliness (the speed of implementation), longevity (some of the assets will stop providing coverage as their energy storage is exhausted), and ability to provide continuing, though possibly reduced, coverage to some of the earlier-specified NAIs. In the remainder of this chapter we introduce the DSS prototype and discuss some design and implementation work performed in its development. The prototype is explained and we ground the need for research in some example allocation scenarios where a DSS of this sort would prove useful. Following the scenarios, we describe the results of some empirical research on the DSS graphical user interface (GUI) development. Lastly, we discuss implications of robust planning in future Army environments.

THEDSSPROTOTYPE

In order to study the issues of robustness, risk, and uncertainty in context, we developed a working prototype of an interactive system to help develop sensor allocation plans. The system was designed to help operators determine how to reallocate a limited number of robotic sensors to provide coverage for a newly identified NAI in addition to previously identified and covered NAIs in response to a changing threat situation. This prototype combined a decision support architecture developed by the The Ohio State University computer science researchers in our group with additional graphical visualizations and human-computer interactions developed at Alion Science and Technology.

TheArchitecture

AnInteractiveDecisionSupportArchitecture205
The architecture consists of a Generator-Evaluator, a Pareto Filter and a Viewer. The Generator-Evaluator generates allocation plans and, for each plan, evaluates its score on the criteria discussed in the sensor allocation problem, i.e. coverage, timeliness, longevity, etc. In the current implementation, the Generator is an evolutionary algorithm in which the set of survivors of each generation is composed of the Pareto-optimal set with respect to selected criteria. For example, if we have n plans, say Pi, P2 Pn, and each plan takes on values from several criteria, then we can say Pi dominates Pj if Pi is better than or equal to Pj in all the criteria. In that case, we can discard Pj. The Pareto-optimal subset is the subset of plans in which no plan dominates another plan. If a plan is better than another in one criterion, then it is worse in another criterion. The only way to choose between Pareto-optimal plans is to perform trade-offs. The Viewer has two major components: a geographic view and a data view. In the geographic view, specific plans can be viewed on a map of the area of interestthe NAIs, the locations and orientations of sensors for the new and old NAIs, and the coverages. In this view, the planner can move the sensors himself to create or tweak a plan and check the impact of changes on plan effectiveness. Using the second component, the planner can view and compare the properties of several plans in a table or scatter plot view. In the latter, trade-offs between pairs of criteria can be graphically visualized. A specific plan can be selected in the table or scatter plot view and visualized in the map view.

TheVisualizations

To explore issues of robustness, risk, and uncertainty in the sensor allocation scenarios, we built a tool that helps operators determine how to allocate a suite of robotic sensors along a border for persistent surveillance. The DSS takes the computational load off the planner and allows him/her to visualize alternate plans as shown in Figure 11.1, investigate the robustness of selected plans, and apply tacit preferences that are not encoded in the computational algorithms. The key for DSS success is pairing the computational framework for generating and evaluating plans with a GUI visualization scheme that supports Army planners both in making decisions that are robust under conditions of uncertainty and in understanding the decision space with respect to risks and opportunities.

206HUTCHINSETAL.

Figure11.1 Visualizationtosupporttheexplorationoftradeoffsbetweenplanalter natives EXPLORINGROBUSTNESSOFSENSORALLOCATIONPLANS

A plan that is optimized for a set of specific assumptions about the world can fail catastrophically when the assumptions dont hold exactly as anticipated. Given the potentially catastrophic consequence, it is more desirable to produce a plan that performs sufficiently, though not optimally, under the specific world model and produces satisfactory results even when the model is not quite right or missing relevant information. The importance of robustness becomes clear when we realize that even the best models of reality are in principle incomplete and often wrong in parts. Of course, robustness under all conditions is impossible to achieve. Some model assumptions are more problematic than others, so in practice a planner would seek to ensure that the plan is robust with respect to the most problematic assumptions of the model. Additionally, he or she could also focus on minimizing undesirable outcomes that may be especially sensitive to certain model assumptions. In the case of the latter minimizing strategy, the planner might spend additional resources either modeling the relevant aspects of reality better or simply adding contingency elements to the plan to avoid bad outcomes.

AnInteractiveDecisionSupportArchitecture207
The prototype DSS that we built helps a planner explore the robustness of a COA. In the scenarios that follow, a sensor allocation plan is defined in terms of specific sensors to be moved from their current locations to specified new locations. We discuss how a planner might use the DSS for sensor allocation planning to investigate the robustness of a COA. This discussion has multiple goals: (1) illustrating the concept of a robust plan in general and a sensor allocation plan in particular and (2) exemplifying the role of the DSS in robustness exploration.

Scenario1:ExploringtheSensitivityofaPlantoSpecificAssumptions

In this scenario, we explore how sensitive various COA outcome values are to the assumption about enemy presence, i.e. the NAI location. Sensitivity analysis is one of the oldest and most useful techniques available for a certain kind of robustness analysis. If sufficient data were available, one could calculate the correlation between a specific modeling assumption and any of the desired outcomes. Ideally, the correlation would provide an indicator of how much of a change in the outcome variable one might expect if the assumption changed. However, in many cases this indicator may be misleading. For example, if an outcome variable increases over a range of values of the assumption variable but decreases over the remaining range, then the correlation may be reported to be 0, giving a false indication of the sensitivity. In this first example scenario, the location of an NAI changes and the associated changes in an outcome measure of coverage are assessed. Here, we illustrate the use of our visual interface to get a qualitative sense of the sensitivity at different ranges of the assumptions values. (A more complete example of the use of a visual interface to explore the sensitivity of a plan is given in Chandrasekaran & Goldman [2007].). Figure 11.2 shows a sensor allocation plan: the old NAIs (NAI1 and NAI2), the new NAI (NAI3), the location of the sensors, and the resulting NAI coverages. Another part of the screen shows the values of the various performance, or outcome, criteria of interest for the COA. Among the many assumptions underlying this plan is one about the likely path of the enemy, which in turn led the commander to specify the location of NAI3 for coverage. Suppose the planner wishes to see how robust the plan is with respect to the assumption of NAI location. The planer moves NAI3 to the new nearby location indicated by the dashed green polygon reference with the white arrow in Figure 11.2. The system recalculates the coverage and other criteria values. The planner can visually note the changes in coverage as coverage regions on the terrain map, as bars in the bar graph, and as a table of values (not shown here). He or she can go back and forth between the two NAI

208HUTCHINSETAL.
locations, or change the location again. For each location change, the system can be arranged to show not only the new criteria values, but also the percentage changes in the values (this is not shown). In this example, the planner would likely conclude that the plan is not especially robust with respect to NAI3 coverage.

Figure11.2 TheredrawnNAI3isshownasadashedgreenpolygon.Thetopgraph depictsvaluesforthecandidateCOAwhoserobustnessisbeing explored.ThebottomgraphdepictsvaluesifNAI3ismovedtothewest. Scenario2:RobustnessExplorationofUndesirableStates

The next example illustrates a different type of robustness: identifying gaps in relevant modeling knowledge by working backward from undesirable states. Even when a planner knows that the model used to calculate the plan outcome measures is incomplete, he or she doesnt always know which kinds of incompleteness might matter. An approach that is often useful for identifying possible relevant missing knowledge is to examine the plan for ways in which it might fail. The planner usually has additional domain knowledge that can suggest specific kinds of failure possible for the specific plan. With additional reasoning, relevant knowledge that the model is missing might be identified.

AnInteractiveDecisionSupportArchitecture209
In this second scenario, the planner examines the locations of the sensors in the plan. Analysis of the terrain map reveals that some parts of the terrain appear more difficult for robot motion than others. Consequently, the planner further analyzes the map for the locations from which the sensors are to be moved. Most of them seem nonproblematic, except for one sensor (the final location and the path are indicated by the solid white arrow in Figure 11.3) which appears to require a path across or around a lake. Given the lake, it is possible that the sensor might never get to the intended destination or, if taking an alternate route, not reach the destination on time.

Figure11.3 Thesolidarrowindicatesapotentiallyproblematicsensorpath(Scenario 2).ThebottomgraphdisplaystheCOAvaluesifthatassetcannotachieve itsdestination.Thedottedarrowindicatesanalternativesensorthat couldbemovedtocoverNAI3(Scenario3).

The uncertainty concerning the sensors path and successful achievement of its intended destination suggests possibly relevant missing knowledge in the model, specifically, terrain data being taken into account for route estimation. At this point the planner may check if the route estimation part of the model contains terrain details, and if it doesnt, add a terrain model. However, a better option before making any model changes is to see whether the possible transport failure matters. The planner can check the sensitivity of the COA to the absence of the robotic sensor. The bar graphs in Figure 11.3 show the outcome values of the two possible end states. NAI3 coverage would be appreciably degraded in the absence of the robotic sensor. If the planner decides that the decrement is unacceptable, he or she can proceed to check, and if necessary augment, the model and redo the criteria calculations. Or, as shown in the next scenario, increase the robustness of the plan by modifying the plan.

210HUTCHINSETAL. Scenario3:PlanModificationToIncludeRobustness

Continuing with the example in the previous scenario, the planner has concluded that the plan is not robust with respect to a specific sensor, i.e., some outcome measures will be negatively impacted by its absence. He or she can replan to try to arrive at a more robust plan or modify the current plan to increase its robustness. For example, the planner might identify another sensor that can be moved at short notice and does not need to traverse difficult terrain if the other sensor fails to arrive at its destination. In Figure 11.3, the sensor indicated by the dotted white arrow meets these conditions, but the planner still needs to do additional checks to ensure satisfactory outcome values for this alternative plan, similar to what was done in Scenario 2 for the two end states. These examples explicate some of the advantages of investigating robustness for model assumptions, alternative assets, possible failures, and terrain assumptions. The visualization tools and the resulting criteria shifts encourage the commander to investigate multiple options in an expedient and intuitive manner. An important objective of our research is to investigate the efficacy of various visualization concepts to ensure that the tools in fact result in rapid and logically consistent decisions.

PROTOTYPEDSSSTUDIES FortLeavenworthDataCollection

Early GUI development focused on the representation of decision making cues in uncertain and risky environments. The initial work provided the necessary first steps toward developing a DSS interface supportive of robust planning. For example, initial prototype interfaces were evaluated with career military officers at the Battle Command Battle Laboratory at Ft. Leavenworth. The experts ranged in rank from captain to retired lieutenant colonel. They came from a variety of backgrounds including infantry, armor, military intelligence, aviation, and signals and communications. Their current posting focused on how to employ assets of the future to meet the threats of today and beyond. The experts were familiar with the types of robotic assets used in the scenarios. When running through the scenarios, they brought their particular expertise to bear on the situation. For example, the military intelligence operator would focus on the relative priority of the different NAIs and those who had served as company commander would discuss the best way to utilize terrain.

AnInteractiveDecisionSupportArchitecture211
A simulation testbed with the interchangeable graphical components was used to present scenarios and collect performance and situation awareness (SA) data (McDermott & Hautamaki, 2007). This approach was coupled with cognitive task analysis interview probes to understand critical cues, tradeoffs, reasoning, strategies, and alternative solutions. The testbed scenarios were presented to experts with three graphical versions of relevant decision information: (1) a vertical bar graph (not shown, but similar to Figure 11.3), (2) a petal display (Figure 11.4), and (3) a weighted vertical bar graph (Figure 11.5). Overall, the vertical bar graph resulted in the most accurate SA and the fastest SA response time. However, each display afforded different decision strategies. The vertical bar graph compares two assets on five criteria. In addition, a summary bar sums the five criteria and normalizes the value on a scale of 0 to 1. The vertical bar graph was deemed superior for making criteria-to-criteria comparisons. In other words, experts were easily able to compare how Asset A compared to Asset B on each of the criteria under evaluation, i.e., timeliness, survivability, etc. The comparative utility was particularly useful when SMEs wanted to maximize a specific criterion. Some experts always focused on the same criteria, and others would change their focus depending on the requirements of the situation. The bar graphs were a familiar format for viewing information and consequently required no learning. The petal decision aid (Ward, 2002) shown in Figure 11.4 is a more holistic display. Each criterion is represented by a slice of the pie. The more the slice is filled toward the outer edge, the higher the value. In Figure 11.4, the probability of detection shown in red (at 6 oclock on a clock face) is much higher for Asset UGV1-B (shown on the right) than for Asset UGV1-A (shown on the left). Gray circles inside the graph are reference lines for 25%, 50%, and 75% of maximum. Operators could also get an exact value by hovering over any slice. The width of the slice represented the relative weight of the criteria. In the following example, timeliness in yellow (at 4 oclock) is weighted the lowest. With the petal display, experts were quickly able to judge the overall strength of each asset. This was conveyed by how much of the polygon was colored as well as the overall score listed below each petal graph. Experts were also able to quickly see the relative strengths of each asset. UGV1-A excels in expected coverage and survivability while UGV1-B excels in expected coverage and probability of detection. It was more difficult to perform a criterion-by-criterion analysis in this display because the information was not spatially colocated. Experts had to look back and forth between the two petal graphs to determine which had higher timeliness, longevity, etc.

212HUTCHINSETAL.

Figure11.4 Petaldisplayaid

The third decision aid was a vertical weighed bar graph displayed in Figure 11.5. The weighed bar graph was similar to the vertical bar graph with the addition of the criteria weighting factor. The width of the bar represented the weight of the criteria. In Figure 11.5, survivability is weighted low and expected coverage is weighted high. Operators could use the arrows below each criterion to dynamically change the weight. The summary value took those weights into consideration. Therefore, in the following example, if an operator increased the weight of survivability, the summary value for Asset UGV2-B would increase at a higher rate than the summary value for Asset UGV3-B. The weighted bars were rarely preferred over the nonweighted bars. Most experts set the weights only once and kept them that way for the duration of scenarios. One expert made a tentative decision, then set the weights to reflect priorities and compared the weighted summary to check his decision. The drawback to this display is that it took additional time to set the weights. It also resulted in the slowest SA response time.

Figure11.5 Weightedbargraphdecisionaid

AnInteractiveDecisionSupportArchitecture213
The visualization study provided insight on how information presentation can impact the way the information is used. In a complex multi-dimensional decision such as this one, there are multiple factors upon which the operator can base his or her decision (i.e., plan attributes, characteristics of specific sensors, terrain). There are also multiple, and sometimes conflicting goalshaving significant coverage on each NAI, getting the sensors in place in time, and being able to conduct reconnaissance for the required duration. When an abundance of information is available, the information that is in focus can influence the decision outcome. The Ft. Leavenworth data collection effort showed that across participants, decision patterns were related to visualizations.

OngoingDSSResearch

Multiple visualization factors were investigated in the course of the research. Some examples are information frame (McDermott et al., 2003), frame consistency (Hutchins et al., 2008), probability presentation (McDermott et al., 2005), and risk presentation (McDermott et al., 2003). The results of prior data collection provided the building blocks for subsequent studies further focusing on visualization issues related to robustness of plans. Several things were done to help the operator perform a robustness analysis. First, a tradeoff analysis tool was developed to help operators understand the implications of different allocation schemes. The plan values for multiple criteria (i.e., timeliness, overall probability of detection, probability of detection for specific NAI) are displayed simultaneously. The operator can click on a value (i.e., the highest possible value for overall probability of detection) and see what the other values are for that plan. This allows operators to dynamically explore the problem space. The operator can discover trade-offs (i.e., plans that have good coverage of NAI2 tend to have low coverage of NAI1) and patterns (i.e., plans with high timeliness tend to have high coverage of NAI1). Map data is displayed simultaneously so as in the last example, the operator can visualize that NAI1 is in close proximity to the friendly base and the current location of several robotic assets. Therefore, plans that allocate multiple assets to NAI1 will be rewarded with high timeliness. The values are also tied to a risk timeline. The timeline shows the probability of detecting an enemy over time. The operator can view the timeline for the entire sector or for individual NAIs. Displayed in this manner, if there is a drop in the overall coverage, the operator can click between individual NAIs to see which NAI is vulnerable (seen by the decline in the timeline) and why (seen by the assets covering the NAI on the map).

214HUTCHINSETAL.
Second, the operator can use his or her expertise to tweak plans and see the implications of those changes. This allows the operator to perform what if queries. For example, what if an asset is moved from NAI1 to NAI3. It may decrease timeliness because the asset has farther to move but it provides double coverage with an asset that is likely to fail early, thus drastically increasing the coverage of NAI3 over time. In another example, the operator could move an asset so that it is midway between two close NAIs, providing some coverage on each. The new location may not be ideal for either single NAI but placing it in the middle allows it to quickly move into place if there is a problem on either NAI.

IMPLICATIONSFORTHEFUTUREARMY SensorAllocation

Even though the tactical and strategic utilization of unmanned sensors is becoming increasingly commonplace in the modern Army, integrative planning with the sensors remains a somewhat novel task. Their continued use will be important to the Army of the future. As evolving sensor technologies increase the possible sensor applications, our understanding of how the sensors will be integrated into future COAs must concurrently evolve. Our early work with the Battle Command Battle Laboratory at Ft. Leavenworth helped build a framework for that understanding. The focus of the Ft. Leavenworth study was on how information presentation can impact the way the information is used; however, since the context was set in sensor allocation, a critical early step was to identif potentially relevant decision criteria for the sensor allocation task. The military experts at the Battle Command Battle Laboratory helped us identify sensor characteristics such as sensor reliability, sensor sensitivity, the ability of the sensor to protect itself, the duration the sensor can operator unattended in the field, and the mobility and speed of the sensor, as well as other criteria such as terrain, enemy likelihood, etc. These characteristics may likely become part of the integrative planning picture in the future Army and therefore should benefit from close examination of the interaction between the representation of the sensor characteristics to planners and the usage of the information in the planning decision.

TheEscalatingRoleofDecisionSupportSystems

AnInteractiveDecisionSupportArchitecture215
One of three sources of information dominance is Command and Control. Command and Control can be defined as awareness of the location of all forces within the battlespace and the ability to decide what to do, where and when to do it, and the capacity to execute quickly (Libicki, 1997). As the complexity of information required to Observe, Orient, Decide, and Act grows, commanders will be forced to rely more heavily upon decision support systems in their Command and Control roles. It is therefore imperative that we look ahead to the decisions of the future Army and anticipate the types of decision support that commanders will need and do our best to understand decision complexity now. Our work with sensor planning takes a step in that direction by developing decision tools that combine the intuition of the commander with model-based tools to understand complexity rather than compute optimal COAs for unitary solutions. Our research has demonstrated that a DSS can serve two fundamental roles in the planning and decision-making process. The first is facilitation and the second is prevention. The facilitative role of a DSS is one of compaction and focus of information. Our work with the prototype DSS demonstrated that a complex problem space can be filtered down to an effective subset of reasonable alternatives. Decision strategies can be designed into the DSS interface to highlight pertinent information and overcome overconfidence in potentially false assumptions. The introduction lists several examples of military decisions with catastrophic consequences due to false assumptions in the model used for decision making. The preventative role of a DSS is achieved by enhancing the commanders analysis of alternatives, both alternative plans and alternative outcomes. A DSS that provides a commander with robustness analysis allows exploration of consequences under a variety of conditions, thereby reducing the misperception of the risk associated with their decisions. As future problem spaces begin to come into focus and commanders and policy makers determine which information is important to include in relevant decision support systems, system developers can choose the best architectures and visualizations to support the appropriate strategies.

SUMMARY

Planning often requires the use of models to evaluate the alternatives by predicting their performance. This reliance on models is a property of all prediction, whether by computer programs or by humans. One fundamental source of uncertainty in prediction is the fact that the world is stochastic in nature, so we can only get a probability distribution of the values for the various criteria of interest. A second source of uncertainty is that models are in principle

216HUTCHINSETAL.
incomplete, and even in the dimensions that are included, they may be inaccurate to various degrees. Given this inherent property of models, plans need to be robust in the following sense: if alternatives A and B have approximately similar expected outcomes, the one with outcomes that are less sensitive to model assumptions and uncertainties is the more robust one. Decision support systems ought to empower decision makers to evaluate how sensitive a decision alternative is with respect to their specific assumptions in the simulation. This issue is drawing attention in decision theory in general (Bankes, 2002; Lempert et al., 2002). Terms such as confronting surprises and planning in the presence of deep uncertainty are being used to characterize the problem. Robust adaptive planning (RAP) involves creating large ensembles of plausible future scenarios. A strategy is robust if it performs reasonably well, compared to the alternatives, over a wide range of plausible scenarios. This style of robustness analysis needs to be distinguished from the traditional approach of modeling uncertainties probabilistically and optimizing the expected values of performance criteria of interest. Optimization only accounts for performance under the most likely outcomes. The performance under less likely outcomes is not factored into planning. Furthermore, not all aspects of uncertainty can be modeled probabilistically, and even when available, planning based on expected values is not necessarily the best approach because the probability space itself may be variable (nonergodic). Two of the main sources of uncertainty in sensor allocation are the likelihood of enemy in certain areas and the reliability of sensors. A planner needs a way to explore the following as part of robustness analysis of a plan:

If assumptions about enemy approach or location change from the predicted approaches, what is the degradation in the coverage of the NAI of interest and what are the consequences? If assumptions about enemy approach or location change, does the plan provide the new NAI minimally acceptable coverage? Are there sensors that can be reallocated within an acceptable amount of time if information later becomes available indicating a different enemy location? Is there a backup asset that can be called in or allocated within a reasonable time to provide coverage to the new area? Can the current plan be modified so that reallocation is easier? If a sensor of a given type fails, what is the coverage degradation? What of the degradation in longevity of coverage? Is rapid reallocation possible in such case?

AnInteractiveDecisionSupportArchitecture217
Our work resulted in an integrated suite of technologies and visualizations for generating, simulating, and multicriterially comparing military plans, and assessing the robustness of a plan. A particular focus continues to be visualization of risks and uncertainties. Our approach combines cognitive research, which has the purpose to create effective visualizations based on multiple criteria and decision flexibility, with computer science research, which has the purpose to develop robust and efficient algorithms for generating and evaluating plan alternatives. The complementary approaches of The Ohio State University, LAIR, Alion Science and Technology, and the Army Research Laboratry ensure psychologically valid solutions that are based on mathematically tractable algorithms and grounded in experience with related military systems at the Battle Labs at Fort Huachuca and Fort Leavenworth. In our future research, we intend to evaluate our approaches during realistic scenarios using experienced Army officers. Our goal is to show that the visualizations will aid experienced officers to make intuitive decisions that are robust and timely under a variety of expected and unexpected outcomes.

REFERENCES
Bankes,S.C.(2002).Toolsandtechniquesfordevelopingpoliciesforcomplexand uncertainsystems.PNAS,2002.99(suppl.3):,pp.72637266. BarYam,Y.(1997).Dynamicsofcomplexsystems.Reading,MA:PerseusBooks. Camerer,C.F.,&Lowenstein,G.(2002).BehavioralEconomics:Past,Present& Future(draft).Pasadena,CA:Caltech. Chandrasekaran,B.,&Goldman,M.(2007).ExploringRobustnessofPlansfor SimulationBasedCourseofActionPlanning:AFrameworkandanExample.Proceedings ofthe2007IEEESymposiumonComputationalIntelligenceinMulticriteriaDecision Making(MCDM2007),15April2007,Honolulu,HI,InstitutionofElectricaland ElectronicEngineers,ISBN:1424406986,pp.185192. Chandrasekaran,B.(2008).DesigningDecisionSupportSystemstoHelpAvoid Biases&MakeRobustDecisions,WithExamplesFromArmyPlanning.Proceedingsof ArmyScienceConference,2008. Hutchins,S.,McDermott,P.,&Barnes,M.(2008).FrameConsistencyinMulti AttributeRiskPreferenceDecisions.ProceedingsoftheHumanFactorsandErgonomics Society52ndAnnualMeeting.2226September2008,NewYork,NY. JohnsonLaird,P.N.,Legrenzi,P.,Girotti,V.,Legrenzi,M.S.,&Caverni,J.(1999). Naveprobability:Amentalmodelofextensionalreasoning.PsychologicalReview, 106(1),6288. Kahneman,D.&Tversky,A.(1979).Prospecttheory:ananalysisofdecisionunder risk.Econometrica,47,363291. Keegan,J.(1995).FieldsofBattle.NewYork:VintagePress. Krugman,P.(2009).Thereturnofdepressioneconomicsandthecrisesof2008.New York:Norton. Lempert,R.,Popper,S.&Bankes,S.(2002).ConfrontingSurprise.SocialScience ComputerReview,20(4):pp.420440.

218HUTCHINSETAL.
Libicki,M.(1997).InformationDominance.RetrievedfromDTIC.milon5.30.09. http://www.dtic.mil/dtic/. McDermott,P.L.,Hutchins,S.,Barnes,M.,Koenecke,C.,Gillan,D.,&Rothrock,L. (2003).Thepresentationofriskanduncertaintyinthecontextofnationalmissile defensesimulations.ProceedingsoftheHumanFactorsandErgonomicsSociety46th AnnualMeeting.1317October2003,Denver,CO. McDermott,P,&Hautamaki,B.(2007).Resourceallocationdecisions:Theimpactof individualdifferencesonpreferencesandperformance.Posterpresentedatthe8th InternationalConferenceonNaturalisticDecisionMaking,Monterey:CA. McDermott,P.L..,Hutchins,S.,&Barnes,M.(2005).Toolittleortoomuch:The effectoffeedbackonriskassessmentinamissiledefensetask.Proceedingsofthe HumanFactorsandErgonomicsSociety49thAnnualMeeting.September,Orlando,FL. Ward,M.(2002).Ataxonomyofglyphplacementstrategiesformultidimensional datavisualization.InformationVisualization,Vol.1,pp.194210. Wickens,C.D.,&Hollands,J.G.(2000).Engineeringpsychologyandhuman performance.UpperSaddleRiver,NJ:PrenticeHall.

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T A N D L A U R E L A L L E N D E R

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T AND LAUREL ALLENDER

This book, produced by the Partners of the Army Research Laboratory Advanced Decision Architectures Collaborative Technology Alliance, also serves as the Final Report for DAAD19-01-2-0009.

SECTIONII.PRESENTINGBATTLEFIELDINFORMATIONTOWARFIGHTERS

SECTIONII PRESENTINGBATTLEFIELDINFORMATIONTOWARFIGHTERS

122SECTIONII

Chapter

12

12. COGNITIVELYINSPIREDAGENTSASTEAMMATESANDDECISIONAIDS

JOHNYEN,PH.D.1,LAURASTRATER2,MICHAELMCNEESE,PH.D.1, XIAOCONGFAN,PH.D.3,HAYDEEM.CUEVAS,PH.D.2,SOOYOUNGOH1, ANTHONYKIM1,DEVMINOTRA1,TIMHANRATTY4 PennsylvaniaStateUniversity,UniversityPark,PA 2SATechnologies,Marietta,GA 3PennsylvaniaStateUniversity,Erie,PA 4ArmyResearchLaboratory,AberdeenProvingGround,MD

1

INTRODUCTION

The advancement in sensor, communication, and information technologies have resulted in information-dense environments in which current and future warfighters must operate. In the face of dynamic, constantly changing events and conflicting reports, this vast amount of information makes it difficult for warfighters to develop and maintain a clear picture of the operational situation. This difficulty is further compounded by the challenges of asymmetric warfare where complex decisions that consider multi-dimensional factors (e.g., human, social, and cultural factors) are required not only for strategic planning, but also at the operational and tactical level, across all echelons.

219

220YENETAL.
To address these challenges, this research, supported by the Advanced Decision Architectures Collaborative Technology Alliance of the U.S. Army Research Lab, aims to achieve two complimentary research goals. Our first research objective is to develop a cognitively-inspired decision agent architecture that supports the delivery of relevant information to decision-makers in a timely fashion. These software agents act as both teammates and decision aids, enhance the humans capability in reasoning across multiple decision spaces, and enable dynamic team collaboration among decision makers. With its cognitive foundations, functions and components in such an architecture can be better related to the processes of human cognition. Such a relationship between computational decision aids and human decision process can contribute to the co-refinement of both the computational decision architecture and the humanagent team cognition theory. Our second research objective is to study factors that affect human-agent interactions such that warfighters can better calibrate their automation usage and maintain global situation awareness (SA). Despite the best intentions of automation designers, the current state of technology does not produce perfect automation. On the battlefield, for example, automation, like humans, must function in the face of uncertainty and in sub-optimal environmental conditions. Warfighters, trained to operate in teams, understand that these complexities may affect the judgments of their human teammates, but sometimes fail to realize that agent teammates can be similarly affected. When warfighters fail to adequately understand the factors influencing the performance of their agent teammates, they may make poor automation usage decisions (AUDs) and may fail to appropriately trust their agent teammates. Previous research has shown that trust in team members, whether human or agent, is critical in mediating team operations, particularly with increasing levels of cognitive complexity. The objective of our cognitively-inspired agent system (R-CAST) is to serve as decision aids and teammates of human decision makers. There are many normative, descriptive, and prescriptive decision theories that can be built within an agent system. We chose to adopt Kleins RPD model (Klein, 1997) in RCAST for several reasons. First, the RPD model is a holistic model of human decision making processes, including activities such as seeking relevant missing information, and monitoring expected outcomes of decisions to detect anomaly. Second, operators can more easily understand a decision aid based on a naturalistic decision process (e.g., macro-cognitive process described in RPD) due to his/her familiarity and experience with the process. This encourages active human-agent collaboration along the decision process. Such a decision aid would be intelligent not only in the evaluation and choice of options, but also in social interactions with human decision makers.

CognitiveAgentsasTeammatesandDecisionAids221
With a computational RPD model as the macro-cognitive process, the R-CAST system has been used in a series of experimental studies toward a better understanding of human-agent collaboration in time-stressed decision making situations. The first experiment was conducted to help us to understand whether future warfighters can benefit from R-CAST in handling complex multidimensional decision tasks. Being a teammate of a human decision maker, the R-CAST system ought to be trustable and understandable. The second and third experiments examined the issue of human-agent trust and raised the interesting question of how to facilitate a suitable level of trust between warfighters and decision aids to improve the overall performance of the human-agent team. The fourth experiment was conducted to evaluate the impact of providing a mental map visualization of the agents decision space to promote automation transparency on effective human automation-usage decisions. The rest of the chapter is organized as follows. In section 2, we describe related research regarding human-agent team cognition and its relationship to human trust in agents. In section 3, we describe the R-CAST agent architecture, which is empowered by a computational RPD model. In section 4 we give an overview of a synthetic task designed for studying multi-dimensional decision making. In section 5, we summarize a series of human-in-the-loop experiments where the R-CAST agents served as teammates and decision aids, investigating the issue of multi-context decision making, human-agent trust, and mental map visualization. Finally, we discuss the impacts of our work and point to some directions for future studies.

HUMANAGENTTEAMCOGNITIONANDTRUST

Team cognition is traditionally involved with studying and specifying the cognitive processes that impact team performance and that vary with certain constraints. Team cognition requires that team members maintain common ground, situation awareness, interdependence, and flexibility as they pursue joint objectives. Team cognition embodies the notion that teamwork is influenced by individual differences among team members as well as the contextual environment within which a team operates. Teamwork may also be influenced by many different social-cognitive factors (McNeese 2000) or social psychological states (Wellens & McNeese, 1987) that change how teams work and perform together. For example, social psychological variables such as trust, affiliation, and status may impact the specific nature of team cognition. In particular, this chapter focuses on investigating factors that influence the interactions between human and agent team members when agents are utilized to

222YENETAL.
support team cognition during a complex decision making task. In particular, we want to better understand how human operators trust in automation may potentially influence their subsequent automation usage decisions (AUDs). Many of the factors that influence human-human team collaboration also influence human-agent collaboration. Trust can operationally be defined not just as an emotional state or characteristic of the trustor, but rather by the behaviors and actions taken by the trustor as the result of that characteristic. This distinction is important in the current research effort as we focus on the calibration of AUDs, a behavior that reflects the users feeling of trust. Specifically, we seek to investigate the impact of providing users with information about agent process that impacts predictability, the characteristic of the agent trustee that most influences trust in the early stages of interaction with the agent. Human operators beliefs about and trust in automation mediate their subsequent reliance on automated systems (such as intelligent agents), ranging from the extremes of over-reliance and complacency to under-reliance and mistrust (Cuevas, Fiore, Caldwell, & Strater, 2007; Lee & See, 2004). This can lead to at least two potentially problematic situations, misuse and disuse (Parasuraman & Riley, 1997). In misuse (over-reliance), the operator blindly follows the judgments made by the automation, thereby abdicating their role of system supervisor; while with disuse (under-reliance), the operator either ignores the automations recommendations or delays action until system judgments can be verified, increasing decision time. While the source for these errors differs, either type of error can have critical consequences; thus, the goal of properly calibrating operators trust in automation and their AUDs seeks to reduce both misuse and disuse of automation. Given the broad range of factors influencing trust in automation and AUDs, it is hardly surprising that research findings have been mixed (see Beck, Dzindolet, & Pierce, 2002; Dzindolet et al. 2003; Lee & See, 2004). Despite the abundant research on the topic, little practical guidance is available to assist system designers in developing automated systems that support the operator in making appropriate AUDs, particularly in cases when optimal usage is a moving target, changing with environmental conditions, workload, and other situational parameters. Therefore, the long term goal of this research effort is to determine techniques for calibrating AUDs by enabling users to gain adequate understanding of agent functioning to allow for appropriate levels of trust in their automated teammate.

CognitiveAgentsasTeammatesandDecisionAids223 RPDENABLEDAGENTS:THERCASTSYSTEM

The R-CAST agent architecture (Fan et. al., 2005a; Fan & Yen 2007) is built on top of the concept of shared mental models (Cannon-Bowers, Salas, & Converse 1990), the theory of proactive information delivery in agent teamwork (Cohen and Levesque 1991; Fan, Yen, & Volz 2005), and the recognition-primed decision framework (Klein 1997). Figure 12.1 shows the major components of R-CAST, where the RPD process is the kernel of the other functional units. Kleins RPD model describes a holistic process of human decision making, including all activities related to human decision makings. These activities include identifying missing relevant information, seeking information, interpreting information, building hypothesis, situation monitoring for detecting anomalies, and decision adaption under unexpected situations. These activities are realized in the R-CAST agent architecture such that (1) each agent can perform these functions individually, and (2) a group of agents can collaborate on these functions as a team. Like the RPD model, R-CAST agents match the current situations with previous experience to determine their similarity. This matching process is implemented in two ways in R-CAST: (1) choose the first experience with a similarity higher than a threshold, and (2) choose the experience with the highest similarity. While the first approach is a computational realization of the satisficing criteria of the RPD model, the second approach offers an alternative that is closer to rationale decision-making. This flexibility enables the developer of agent applications to choose the suitable scheme, depending on characteristics of the application.

Figure12.1 TheRCASTAgentArchitecture(Fan&Yen,2007)

224YENETAL.
The R-CAST agents realize the seeking missing information component in the RPD model using a backward (Prolog-style) chaining inference, starting with high-level cues of the experience being matched, using rules in the agents knowledge base. When a piece of missing information is identified through this backward chaining inference, the agent contacts the information source associated with the missing information through agent communication protocols realized in Service Oriented Architecture (SOA). The information source can be another agent, a Web service, or a human operator. When the requested information arrives, the agent performs forward-chaining inference that fuses the newly arrived (previously missing) information with other information know to the agent. The outcomes of these fusions affects the degree a decision experience matches the current situation. Hence, this fusion within an agent typically corresponds to high-level information fusion (i.e., level 2 or above in the JDL data fusion model). The R-CAST agent architecture also implements decision progress monitoring and expectancy-based decision adaptation in the RPD model. After a decision is made (either directly by an agent or indirectly by a human operator based on the agent recommendation), the agent continues to monitors the expected outcomes (called expectancy in the RPD model) associated with the chosen decision. If the agent detects an anomaly (e.g., some expected outcomes not fulfilled), the agent adapts the previously made decisions to deal with the anomaly. Finally, a team of R-CAST agents realizes a collaborative RPD process (Fan, Sun, McNeese, & Yen, 2005), in which each agent anticipates information requirements of teammates and can proactively assist them by monitoring and/or seeking information relevant to their requirements. Together, the features described above form a cognitively-inspired agent architecture that not only supports warfighters in making decisions and adapting them to changing conditions, but also in augmenting them in their capabilities to sense, fuse, and interpret information in a multi-dimensional net-centric environment .

ASYNTHETICTASKFORMULTIDIMENSIONALDECISIONMAKING:THE THREEBLOCKCHALLENGE

CognitiveAgentsasTeammatesandDecisionAids225
To conduct experiments regarding multi-dimensional decision-making of warfighters, we have designed and implemented a simulation environment called The Three-Block Challenge. It can simulate command and control scenarios of urban operations where in close proximity (e.g., within three blocks) military operators need to quickly react to challenges related to three dimensions of mission: the humanitarian relief dimension, the peacemaking (i.e., policing) dimension, and the combat dimension (Fan et. al., 2005). The synthetic task environment contains objects of interest such as main supply routes (MSRs) and key buildings (religious buildings, schools, and hospitals). At run time, the environment can produce three types of threats: Improvised Explosive Device (IEDs), crowds, and insurgents, which represent the targets of humanitarian, peacekeeping, and combat operations, respectively. IEDs are motionless targets, and if exploded, can cause damage to the road (e.g., MSRs) and buildings nearby. A crowd represents a group of people which may contain activists that can be friends or foes. A crowd can be of medium (M) or large (L) size, and the group size of a crowd can change over time. Two crowds can merge together if they move close enough. Another type of movable targets is insurgents, each is associated with a threat level that can be L(low), M(medium), or H(high). A target may appear, stay on, and disappear from the battle field following certain temporal and spatial patterns unknown to human operators. There are also a limited number of friendly unitssquads and Explosive Ordnance Disposal (EOD) teamsunder the control of a C2 team. Each friendly unit has an associated property called combat readiness, which is represented by a percentage value, indicating current unit ability to handle threats. The readiness value decreases after a unit is tasked to a threat, and can recover incrementally as time passes. In our studies, a C2 team consists of an S2 suite (intelligence cell) and an S3 suite (operations cell). The roles of C2 operators have been simplified. S2 is responsible for processing incoming reports, called Spot reports; collecting relevant information from other sources; and alerting S3 of potential threats. S3 needs to process alerts from S2, and make decisions on which target to handle next and which resources (friendly units) to allocate toward that target.

226YENETAL.

Table 1: Credit value and resource requirements for handling targets

Targets Value Res.Req. Action 20 1U monitor M w/ofoe 40(+10)* 2U disperse M wfoe Crowd 2U disperse L w/ofoe 40(+10)* 50(+10)* 3U disperse L wfoe Insurgent n=1,2,3forL,M,H (n+1)U capture (3threatlevels:L,M,H) 50+50n 60(+20)* 1U+1E remove IED Ureferstosquadunit,EreferstoEODteam. *additionalcreditvaluewhenatargetisnearanMSR.

Decision making in target selection and resource allocation requires the S3 suite to consider trade-offs among multiple factors: target type, threat level, the combat readiness of the available units, the unit-target distance, and how long a target has been on the field. The type and threat level of a target determine how many friendly units will be needed to handle the target. Table 1: lists for each type of target the credit value (the reward points a C2 team can get if a target is handled successfully), the number of resources required to handle a target, and what action S3 should take. For example, the second entry says that dispersion of a medium-sized crowd with a foe needs two squad units, and 40 points can be credited if the crowd is dispersed successfully. The last entry indicates that one squad unit and one EOD team are required to remove an IED. If an IED is removed successful, 60 points can be credited if the IED is close to buildings only or MSRs only, 80 points if it is close to both.

HUMANAGENTEXPERIMENTS

Toward our goal of developing theory and technologies for supporting humanagent decision-making teams using cognitively-inspired agents as teammates and decision aids, we have conducted a series of human-in-the-loop experiments to investigate issues regarding multi-context decision making, human-agent trust, and mental map visualization. We next briefly describe an example setting for the experiments (other settings vary slightly, with or without R-CAST agents playing a role in S2 or S3 suites). For each study, we then summarize the research questions and our main findings.

CognitiveAgentsasTeammatesandDecisionAids227

ASettingwithRCASTAgentsinS3Suite

The role of S3 suite is played by an R-CAST agent (S3 agent) and a human operator. The human operator is equipped with two monitors: a map display for tracking situation development, and a tasking interface for collaborating with S3 agent to handle threats. The map display shows all the active entities on the field, as well as MSRs, buildings, and regional boundaries. It allows a human operator to highlight the target of interest, to figure out the spatial relationships between targets, MSRs, buildings, and friendly units, and to project forward the location of moving targets. The human operator needs to determine whether there are key buildings (e.g., hospital, mosque) near a target when assigning units to the target. This is one way the social dimension is introduced into the simulated C2 environment. The tasking interface consists of a threat table, a command panel, and a feedback display panel. The threats table shows consolidated information about the threats on the field: for each threat, it lists threat type, ID, status, crowd size, activists associated with a crowd, nearby buildings, priority, elapsed time, and the IDs of the tasks associated with prosecuting the threats. The command panel allows the S3 human operator to physically task units to the selected target. The feedback display panel shows some statistics about tasks issued, threats cleared, and reward points earned. In the experiment, the S3 agent offers decision aids and recommendations to the S3 operator regarding units to be assigned to a target selected by the user. It is, however, the S3 operator who has the final authority for decisions on target selection and resource allocation.

Experiment1:SupportingMultipleContextDecisionMaking

In this study (Fan et. al., 2006), we focused on the challenge of switching between multiple types of decision contexts. We chose this cognitive challenge for two reasons. First, this is an important characteristic of the multidimensionality of the warfighters mission. They need to deal with multiple missions, which involve different types of threats. For instance, a warfighter may need to deal with a combat mission (capturing key insurgents), police mission (controlling crowds), and humanitarian mission (making sure food, medicine, and supplies are delivered safely along logistic routes) at the same time. Second, decision-making involving different types of contexts introduces additional cognitive load, due to the need to change focus between these contexts.

228YENETAL.

Figure12.2 C2teamperformancevarieswithcontextswitchingfrequencies

Two sets of experiments, each with various settings of context switching frequencies and tasking complexities, were conducted. The participants were US Army Reserve Officer Training Corps (ROTC) students at Penn State. Figure 12.2 shows one of the experiment results. Overall, the study demonstrated that C2 team performance, while still limited by human cognitive capacity, could be largely improved when they were assisted by R-CAST agents capable of proactive information gathering/sharing and experience-based decision making. It also suggests that higher demand situations require more competent teammates. The experiments represent an important step forward in uncovering the nature of real-world problems that are highly relevant to the vision of the US Army.

Experiment2:HumanTrustonCognitiveAids

As we mentioned in previous sections, human trust in agents is an important factor that affects the calibration of his/her automation usage decisions (AUDs). To improve our understanding of this phenomenon, we introduced error into the agent recommendations: the agents did not consider unit combat readiness. In this experiment, we focused on the relationship between S3 operator and S3 agent, with the S2 function simulated by an R-CAST agent without an S2 operator. Our research question is whether knowledge about the source of agent error affects human trust in the agent. Sixty command and control teams, each consisting of a human operator and two intelligent agents, allocated resources to targets in simulated urban combat scenarios. We used a mixed 2 2 2 3 factorial treatment design (PG KAR TC RIT), where TC (task complexity) and RIT (ratio of insurgent threats) are within-subjects variables and KAR

CognitiveAgentsasTeammatesandDecisionAids229
(knowledge of agent reliability/error) and PG (participant group: ROTC students vs IST students) are between-subjects variables. The experiment group was informed about the agents source of error, whereas the control group was not informed about the source of errors. Both groups were informed about the same set of game rules and the agents reliability measure. The ANOVA output indicates that both the knowledge of agent reliability (KAR) and ratio of insurgent threats (RIT) had significant effects on C2 performance. Figure 12.3 shows that knowing the agent reliability helped the S3 operators rectify more inappropriate recommendations. Together with the analysis of the SAGAT and NASA-TLX measures, this study indicated that giving even a minimal basis (i.e., knowledge about the source of error from agents) for understanding conditions impacting agent reliability allowed operators to make better automation usage decisions, have better SA on the critical issues associated with automation error, and establish better trust in the intelligent agents.

Figure12.3 Incorrectagentrecommendationcorrected

This result is different from previous research regarding automation usage decisions (Dzindolet et. al. 2003) that involves randomly generated errors. This motivated us to investigate the impact of predictable versus random error on human AUDs.

Experiment3:AgentErrorPatternsandHumanTrustCalibration

230YENETAL.
Our research question in this study is, are AUDs different in interactions with random agent error as compared to predictable error (that is, when error patterns are present)? To investigate this, we designed a study with both a random error condition and a systematic or predictable error condition. To ensure a fair comparison, the error rate in the two conditions were comparable, though errors were actually lower in the random error condition. The results, shown in Figure 12.4, indicates that operators teamed with agents with systematic errors that the user can make sense of make more appropriate AUDs than operators teamed with agents that make random errors. Based on the findings of Experiment 2 and Experiment 3, we proposed a model about human-agent trust, in which the decision makers attempt to make sense of the error patterns play an important role in his/her decision on accepting an agent recommendation or not (Strater et al, in review). Using this model, we can explain the finding described above as follows. With agent errors that the decision maker cannot make sense of (e.g., random errors), the decision maker cannot distinguish reliable recommendations from erroneous ones and thus cannot properly calibrate AUDs. For errors that the decision maker can make sense of (e.g., errors due to ignorance about combat readiness criteria), understanding error patterns allows operators to distinguish reliable recommendations from those that are not reliable and helps calibrate AUDs.

Figure12.4 Acceptanceofcorrectagentrecommendationsfromagentswithsystem aticerrorsvsfromagentswithrandomerrors Experiment4:VisualizingMentalMapofRPDAgents

CognitiveAgentsasTeammatesandDecisionAids231
In order to serve as effective teammates, enhancing shared SA and stimulating the users active participation, R-CAST agents need the ability to expose their internal states and reasoning process to the human user. Towards this end, we implemented a visualization concept into R-CAST and conducted an experiment to examine the influence the visualization concept (information presentation) had on effective AUDs and situation awareness. Specifically, we designed a visualization concept that helps establish a mental map of the agents decision space while promoting transparency of the underlying agent recommendation process and higher-level SA (comprehension and projection). In this visualization, we reduce the dimension of the experience space of an RPD-agent onto a 2-D display and dynamically position the current agent-state onto the display based on shared similarity measures. Figure 12.5 shows an example of the R-CAST Visualization of the Agent Decision Space (VADS). The VADS maps a collection of past experiences or Common Historical Cases (CHCs) and current target icons based on their relative similarities. In addition to temporally repositioning active targets on the VADS, the conveyance of information about a target or a related CHC is augmented with the use of iconic symbols. To assess the effects of the visualization concept, we modified the Three-Block Challenge synthetic task environment to emphasize the importance of crowd control. The study employed a 2 (visualization mode) x 2 (work load) x 2 (task complexity) mixed design. The between-group factor was the mode of visualization (experimental group utilizing mental map visualization vs. control group utilizing tradition table visualization). The two within-group factors were the scenarios workload (5 crowds vs 10 crowds) and level of task complexity (ratio of fast-burning vs slow-burning crowd movement). The dependent variables included task performance, situation awareness, trust in automation, and subjective workload. Thirty two ROTC students from Penn State participated in the experiment, which included four 10-minute scenarios for each subject.

232YENETAL.

Figure12.5 RCASTVisualizationofAgentDecisionSpace(VADS)

Overall, the preliminary results have revealed improvement in both task performance and SA. One of the measures calculated for task performance was the product of Score and Real-time SA. The un-normalized result for this ANOVA measure showed the experimental group scored an average 20 percent improvement over the control group. Detailed results of the experiments will be reported in a separate paper.

CONCLUSIONANDIMPLICATIONSFORTHEFUTUREARMY

The ARL ADA CTA research initiated throughout the past five years shows significant results that inform and have impact within several key areas: cognitive science, agent architectures, human-computer interaction, and automation and trust. Looking at the area of cognitive science first, much of the results obtained are directly informative and applicable for how a command, control, and communications (C3) team addresses a very complex, evolving C3 mission to keep stability ongoing in the Iraqi warfront. The mission and scenarios incorporated into the experiments are important as it represents how team members must adaptively assign and adapt resources for Iraq multi-target, multi-

CognitiveAgentsasTeammatesandDecisionAids233
faceted C3 mission. As this mission and scenarios were developed through knowledge elicitation with experts, it is both contemporary and representative of dynamic cognitive processes wherein team members must analyze changes and adapt their resources accordingly under periods of time pressure. Many studies in cognitive science utilize more toy domains with static cognitive processes, and do not involve the levels of complexity and replanning while retaining necessitated balance across ongoing multi-layered mission objectives that often have to be traded off. The experimental studies conducted also incorporate two other integrated aspects of cognitive science: 1) adaptive use of automation given understanding of trust in the agent, and 2) measuring situation awareness in addition to human performance. By manipulating elements of trust we have been able to determine how trusts develops based on what human believe the automation is capable of doing (e.g., how reliable it is). This is extremely valuable for designing and coupling humans with intelligent agent architectures in a way wherein the joint interaction can address dynamic cognitive states that arise. Because previous studies typically only use agents that are not evaluated in terms of trust with human, and utilize agents that do not employ recognitionprimed decision making strategies, previous studies often only address static cognitive states. With the complexities in contemporary teamwork and advanced missions, our research is directly applicable. Because our studies look at situation awareness in addition to direct human performance measures, a deep understanding of how cognitive processes interact with automation is possible. The impact of these elements is significant because it provides a baseline for understanding how agents will need to be designed to fit human cognitive capacities, while at the same time providing a baseline on how human trust develops with differing information about the agents. Together these elements advance cognitive science and human-agent interaction in innovative ways that afford potential increases in the overall war-fighter mission. From the perspective of agent architecture, this research has resulted in a comprehensive cognitively-inspired agent architecture that is designed to serve as both a decision aid and a teammate for warfighters. Based on Kleins Recognition-Primed Decision (RPD) model, the RPD agent provides a computational framework regarding the holistic human decision making process, which can assist warfighters in making decisions across multiple contexts/ missions, collaborative sensing and seeking information relevant to the needs of warfighters, fusing information for situation understanding in a distributed netcentric environment, and detecting changes that require adaptation of decisions previously made. From human-computer interaction perspective, one of the principal challenges in supporting close human-agent collaboration is increasing automation transparency to align the decision makers understanding of the decision space (mental model) with that of the intelligent software agents working on their behalf. Enabling technologies that allow the conveyance of information

234YENETAL.
(complex insights, experiences, and high-level concepts) and its correct application is critical. One method examined in this research to improve humanagent understanding is information visualization. In this research we examined a visualization concept designed to enhance the decision makers perception, comprehension, and projection of the underlying knowledge space while improving shared human-agent SA. Allowing a holistic view of the agent decision space provides a transparent view not only of the agent decision process but equally important, encourages active participation from the user allowing for the adjustment of contextual constraint, the ability to prioritize missing information and ultimately, improved decision making. In summary, the research described in this chapter contributes to the technology for designing cognitively-inspired agents as well as our understanding about the principles regarding the macro-cognitive processes of human-agent team cognition and decision making, especially on the issue of trust in automation and the resultant automation usage decisions (AUDs). This understanding, in turn, can be adopted to enhance the design of intelligent agents. Hence, the design of cognitively-inspired agents, the findings of human-centric experiments, and the theory of human-agent team cognition form a cycle of synergistic research roadmap, which drive human/social science and computational/information technology forward toward the vision of equipping warfighters with cognitivelyinspired agents as teammates and decision aids. From this, suitable levels of trust can be developed to enable effective human-agent team performance for complex multi-facet decision making across strategic, operational, and tactical missions.

ACKNOWLEDGMENTS

We wish to thank Army Cadets and Officers at Penn State University for their participations in all of the experiments described in this chapter. This research has also benefited from many fruitful discussions with Dr. Laurel Allender at Army Research Lab, Dr. Linda Pierce at Army Research Institute, and Dr. Frank Ritter at Penn State University.

REFERENCES
Beck,H.P.,Dzindolet,M.T.,&Pierce,L.G.2002.Operator'sautomationusage decisionsandthesourcesofmisuseanddisuse.InE.Salas(Ed.),Advancesinhuman performanceandcognitiveengineeringresearch(Vol.2)(pp.3778).Bingley,UK: EmeraldGroup.

CognitiveAgentsasTeammatesandDecisionAids235
CannonBowers,J.A.;Salas,E.;andConverse,S.1990.Cognitivepsychologyand teamtraining:Trainingsharedmentalmodelsandcomplexsystems.HumanFactors SocietyBulletin33:14. Cohen,P.R.,andLevesque,H.J.1991.Teamwork.Nous25(4):487512. Cuevas,H.M.,Fiore,S.M.,Caldwell,B.S.,&Strater.L.2007.Augmentingteam cognitioninhumanautomationteamsperformingincomplexoperationalenvironments. Aviation,Space,andEnvironmentalMedicine,78(5,SuppSectionII),B6370. Deutsch,M.1960.Theeffectofmotivationalorientationupontrustandsuspicion. HumanRelations13:123139. Dzindolet,M.T.;Peterson,S.A.;Pomranky,R.A.;Pierce,L.G.;andBeck,H.P.2003. Theroleoftrustinautomationreliance.InternationalJournalofHumanComputer Studies58:697719. Endsley,M.1995.Towardatheoryofsituationawarenessindynamicsystems. HumanFactors37:3264. Fan,X;Oh,S.;McNeese,M.;Yen,J.;Cuevas,H.;Strater,L.;andEndsley,M.R.2008. TheInfluenceofAgentReliabilityonTrustinHumanAgentCollaboration.InProc.ofthe EuropeanConferenceonCognitiveErgonomics(ECCE'08),2008. Fan,X.,Sun,S.,McNeese,M.,andYen,J.,2005.ExtendingRecognitionPrimed DecisionModelForHumanAgentCollaboration,InProc.AAMAS05,pp.945952,The Netherlands. Fan,X.;Sun,B.;Sun,S.;McNeese,M.;Yen,J.;Jones,R.;Hanratty,T.;andAllender, L..2006.RPDEnabledAgentsTeamingwithHumansforMultiContextDecisionMaking. InProc.AAMAS06,pages3441,Japan. Fan,X.,andYen,J.2007.RCAST:Integratingteamintelligenceforhumancentered teamwork.InProceedingsoftheTwentySecondNationalConferenceonArtificial Intelligence(AAAI07),15351541. Fan,X.;Yen,J.;andVolz,R.A.2005.Atheoreticalframeworkonproactive informationexchangeinagentteamwork.ArtificialIntelligence169:2397. Kaber,D.B.,andEndsley,M.R.2004.Theeffectsoflevelofautomationand adaptiveautomationonhumanperformance,situationawarenessandworkloadina dynamiccontroltask.TheoreticalIssuesinErgonomicsScience5(2):113153. Klein,G.A.1997.Therecognitionprimeddecision(rpd)model:Lookingback, lookingforward.InNaturalisticdecisionmaking(Eds:C.E.ZsambokandG.Klein). Hillsdale,NJ:LawrenceErlbaumAssociates,Inc.285292. Lee,J.D.,andSee,K.A.2004.Trustinautomation:Designingforappropriate reliance.HumanFactors46(1):5080. McNeese,M.D.2000.Sociocognitivefactorsintheacquisitionandtransferof knowledge.Cognition,TechnologyandWork,2,164177. Norling,E.2004.Folkpsychologyforhumanmodelling:ExtendingtheBDIparadigm. InAAMAS04:InternationalConferenceonAutonomousAgentsandMultiAgent Systems,202209. Parasuraman,R.,andRiley,V.1997.Humansandautomation:Use,misuse,disuse, abuse.HumanFactors39(2):2302 Strater,L.,McNeese,M.,Cuevas,H.M.,Fan,X.,Oh,S.,andYen,J.MakingSenseof ErrorPatterns:TowardAppropriateHumanAgentTrust,inreview. Wellens,A.R.,&McNeese,M.D.1987.Aresearchagendaforthesocialpsychology ofintelligentmachines.ProceedingsoftheIEEENationalAerospaceandElectronics Conference(pp.944949).Dayton,OH:IEEEAerospaceandElectronicsSystemsSociety.

236YENETAL.

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T A N D L A U R E L A L L E N D E R

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T AND LAUREL ALLENDER

This book, produced by the Partners of the Army Research Laboratory Advanced Decision Architectures Collaborative Technology Alliance, also serves as the Final Report for DAAD19-01-2-0009.

SECTIONII.PRESENTINGBATTLEFIELDINFORMATIONTOWARFIGHTERS

SECTIONII PRESENTINGBATTLEFIELDINFORMATIONTOWARFIGHTERS

122SECTIONII

Chapter

13

13. GRBIL:ATOOLFORDYNAMICOPERATORANDENVIRONMENT MODELING

MICHAELMATESSA,PH.D. AlionScienceandTechnology,Pittsburgh,PA REBECCAMUI AlionScienceandTechnology,Boulder,CO

INTRODUCTION

Human performance models can be very useful in predicting behavior. However, for dynamic situations it may be difficult to model both a changing environment and the processes needed to react to that environment. This chapter describes the Graph-Based Interface Language (GRBIL) tool which allows the modeling of an environment with a task network model, the creation of an interface for observing that environment using drag-and-drop functionality, and the modeling of a human operator of that interface with a cognitive architecture. Using both modeling approaches capitalizes on the advantages of each (the intuitive graphical representation of task network modeling and the detail of the cognitive architecture). To enable our goal of making the system usable to nonmodeling experts, operator models are created by demonstrating procedures on the interface.

237

238MATESSAANDMUI ArmyRelevance

The Army is increasingly making use of the capabilities of unmanned ground vehicles (UGVs) and unmanned aerial vehicles (UAVs). These vehicles are controlled by Soldiers through the use of Operator Control Units (OCUs Figure 13.1). Operators set up plans for the vehicles which are then executed independently. Operators are also responsible for monitoring the status of the vehicles. The interface for the OCU is quite complex, with several modes of operation and control menus. Planning mission requirements for unmanned vehicles requires not only an understanding of vehicle capabilities but an understanding of the capabilities of the human operators. This is especially true in highly dynamic situations where limitations of human attention and motor abilities become relevant. A detailed understanding would provide predicted times to complete tasks, predicted errors on tasks, and the capability to predict changes due to factors such as different interface design and different number of operators.

Figure13.1 OperatorControlUnitusedinthefield

GRBILModelingTool239
In 2002, the Advanced Decision Architectures (ADA) Collaborative Technology Alliance (CTA), sponsored by the Army Research Laboratory (ARL), began an effort to develop valid computational models of human decision making in order to provide a detailed understanding of human performance. Early on, the researchers involved acknowledged the individual strengths of reductionist models in the form of task network models (e.g., IMPRINT - Archer & Adkins, 1999), and first principle models in the form of cognitive models (e.g., ACT-R Anderson, 2007). They undertook an effort to combine the two and to demonstrate how they could be integrated to assess the usability and effectiveness of a GUI. This work has resulted in a tool named GRBIL (Archer, Archer, & Matessa, 2007). The primary motivation of this work is to capitalize on the strengths and avoid the weaknesses of cognitive and task network models. This chapter will explain how our work resulted in advancements in cognitive modeling, describe the system architecture, give an example of interface evaluation, show innovations to human factors and artificial intelligence, note ADA collaborations, and provide implications for future Army work.

INNOVATION ModelingAdvancements

The GRBIL tool includes a number of advancements in cognitive modeling: affordable modeling through demonstration, GUI-based creation of visual monitors, and a visualization of operator cognition and environment interaction. GRBIL and other tools that provide modeling by demonstration have solved some affordability problems. For example, CogTool (John et al., 2004) is an application to assist in the creation of design mockups and the calculation of predictions of skilled performance time for tasks on those designs. However, CogTool provides no standard solution for spatial reasoning or integration with dynamic environments. Visual processing in CogTool is limited to scripted attention movement to static interface objects. The environment can not be monitored to detect a change. The interface used by CogTool is an HTML-based mockup that allows limited transitions between states with each state being represented by a HTML page (although models created with CogTool can later be integrated with dynamic simulators with some hand-coded modifications).

GRBILArchitecture

240MATESSAANDMUI
The architecture for GRBIL consists of four components, shown in Figure 13.2:

1. The GRBIL interface - used to draw and define the GUI that the operator will interact with and to specify the operators goals. 2. The cognitive modeling system - responsible for generating predictions of human performance. 3. A dynamic environment model - responsible for modeling environments external to the interface that may be changing and whose changes will affect users performance with the interface. 4. A software hub used to mediate communication between the other components and achieves the necessary level of component integration.

Figure13.2 TheGRBILarchitecture

GRBILModelingTool241
Component 1: GRBIL Interface. The first component of GRBIL is the user interface, through which the user can describe the physical look and feel of the GUI being analyzed. This is done in a similar fashion to many modern interface layout applications using WYSIWYG drag-and-drop functionality. An assortment of commonly used GUI widgets (e.g., radio button, text box, lists, and toggle buttons) can be dragged onto the GUI description. Once the control is placed on the GUI description, control properties can be used to adjust the labeling, size, shape, color and behavior of the control. The second step in describing a new GUI is to define the actions each control is capable of and what the desired effect of each action will be. This is done for each control in GRBIL via an Event Actions menu for each control. Using this process of adding GUI windows, placing controls on those GUIs and then describing the effects of using those controls on the state of the GUI and all its windows, a GRBIL user can describe the functionality of an entire new user interface. The next step in setting up an analysis is to define the tasks that an operator of the system might wish to perform. This is accomplished by stepping through and recording a series of actions, i.e. button clicks, moving of objects, etc. The recorded series of actions is referred to as a goal state in the GRBIL terminology. Many separate goal states can be defined for a given GUI, so that the GRBIL user can specify a robust set of potential user actions. Once recorded, this goal state will function as the goal for the cognitive model embedded in GRBIL and described later in this section. These recorded actions specify ideal human behavior and will not necessarily align to the behavior that GRBIL predicts. A byproduct of this work is that the GRBIL user has now developed a dynamic prototype of the new systems GUI. As a minimum, this prototype, once finalized, can be used as a look and feel specification for the project programming staff. In many cases, this prototype can actually be encapsulated and can function as the actual new systems software interface once connected to the appropriate action routines and system behaviors. Component 2: Cognitive Modeling System. The second component of GRBIL is an embedded cognitive model that predicts how the human might behave when confronted with the GUI described in Component 1. In GRBIL, the cognitive model interacting with the system is a representation of the knowledge that an average user would bring to bear on a new task. In particular, it represents as procedural knowledge the skills needed for manipulating the GUI controls defined in the GRBIL toolbox. It also has the necessary skills to navigate the new GUI. When retrieving a control step that is not immediately available, the cognitive model is able to reason about how to navigate the GUI using declarative knowledge to put itself in a position where

242MATESSAANDMUI
that control is available. By default, it will do that using the knowledge of the GUI that is automatically downloaded into the model before it is run. That knowledge takes the form of declarative memory chunks for each control that describes its position in the GUI (e.g. which window or sub-window that it is a part of) and which actions are associated with it. We have chosen Adaptive Control of Thought Rational (ACT-R) 6.0 (Anderson, Bothell and Byrne, 2004) as the cognitive modeling architecture in GRBIL. ACT-R is a computational cognitive architecture that accepts declarative and procedural knowledge about how to do the task, and after combining it with a computational description of the environment in which the knowledge will be applied, can generate a time-ordered series of behaviors. These behaviors include cognitive tasks such as attention shifts, memory retrievals, and decisions. They also include motor effects, such as button presses or movements of the mouse. In this way, ACT-R can manipulate the interface as it attempts to achieve its goal state. A number of characteristics of ACT-R are relevant to the performance of the model. First of all, ACT-R is limited in how fast it can perform its actions, especially external actions such as perceptual scanning and manipulation of the interface. In a dynamic, real-time environment such as robotic control, this can give rise to errors as the model is not able to keep pace with the demands of the task in the same way that a human operator would be unable to keep pace given the current system interface. Another source of errors is memory retrieval; ACTR can skip steps or retrieve them in the wrong order (Anderson, Bothell, Lebiere, & Matessa, 1998) in the same way that a human might. Finally, performance can vary as a function of individual differences in working memory, psychomotor speed, or individual strategies, all of which can be represented in a constrained manner in the cognitive model. Because models are assumed to represent a steady state of practiced behavior, GRBIL does not take advantage of ACT-R learning mechanisms. Component 3: A Dynamic Environment Model. The third component of GRBIL is the representation of the environment in which the new GUI will be used. In many cases, system interfaces are not only responsive to inputs from an operator, but also reflect changes in the external environment, such as in an airplane cockpit. For this reason, we needed to design GRBIL to allow for the easy incorporation of systems which model the environment external to the GUI and the system operator. Once incorporated into the GRBIL tool, this would enable changes in the environment model to be reflected on the GRBIL representation of the new GUI. This may affect the performance of the cognitive model, and visa versa. These requirements led us to select a task network modeling architecture for this component.

GRBILModelingTool243
In order to develop this component, we began by selecting a task network modeling environment, named the Improved Performance Research and Integration Tool (IMPRINT) developed by the ARL Human Research and Engineering Directorate (Archer & Adkins, 1999). We chose this environment primarily because the discrete event simulation techniques included in IMPRINT are very well suited for human performance modeling. Secondly, IMPRINT is a stable software tool, originally developed to support the assessment of human performance in the context of total system performance in complex environments. IMPRINT provides a mature architecture and database structure that can easily incorporate a modeling method for representing goaloriented behavior. The basic modeling capability is a classical reductionist method. IMPRINT requires the decomposition of a system mission into functions which, in turn, are decomposed into tasks. The tasks are linked together into a network describing the flow of events. The network can include various types of branching logic such as parallel branches, probabilistic branches, and repeating branches. At the task level, estimates of task performance time and accuracy means and standard deviations are entered along with the consequences of the failure to perform a task accurately enough. The data entered are assumed to be representative of performance under typical or baseline conditions. Also, standards of performance can be entered to provide benchmarks for performance adequacy at the mission, function, and task levels. IMPRINT is very well suited to describe the events that could occur in the environment that will affect how the new GUI being analyzed in GRBIL must be used. For example, changes in terrain, the appearance of enemy or friendly units, or the availability of new information (e.g., mission orders, contact reports, intelligence data) could all change the way in which the GUI described in GRBIL would be used. While GRBIL does not contain IMPRINT, it does allow a user to link to a model developed in IMPRINT. Component 4: Software Hub. This final component achieves integration and communication of the other three components at run time. This is a relatively complex problem and the progress of the resulting human behavior simulation is arbitrated through a software hub. As the model runs and time progresses, the task network model provides event triggers that represent changes in the environment. These events could trigger changes in the GUI that the first component is showing (perhaps symbology on a map display changes, or perhaps a target is identified). In order to respond to these changes in the environment, the cognitive model must not only be executing a series of predefined goal states, but must also be monitoring the GUI for changes in the environment and making the appropriate response at the appropriate time. This predicted response is driven in part from the goal states that the GRBIL user defined when describing the GUI (Component 1).

244MATESSAANDMUI
At run time, GRBILs second component (the ACT-R model) monitors for the conditions at the end of each goal state. If the condition is found to exist, then the model takes the appropriate action. In this way, the ACT-R model is responsible for maintaining the ordered series of goal states and for managing and predicting how the simulated human will process the available information. It is significant to mention that the GRBIL user does not develop ACT-R models in the traditional sense (through writing LISP code). Rather, GRBIL automatically generates executable ACT-R models using the descriptions of the GUI elements and the goal states. This is a significant advancement to the stateof-the-art in cognitive modeling and is the key to ensuring that GRBIL users do not need to be cognitive modelers or LISP programmers. The software hub enables the second and third components of GRBIL to run in parallel, so that changes to the environment can be happening at the same time that the modeled human is working on a current goal. This is necessary to provide realism, but also requires a fairly sophisticated time management capability within GRBIL

OCUInterface

In order to demonstrate the proof of concept of cognitive models generated with GRBIL, an interface for an unmanned vehicle OCU was selected. In our implementation (Figure 13.3), cognitive models performed the actions of human operators and task models performed the actions of unmanned vehicles. The operator models are able to monitor specific attribute values in the environment (such as text describing the status of a vehicle) and use spatial reasoning to detect more general conditions such as projected vehicle intersection. Models are capable of predicting the time to complete OCU tasks, predicting errors on tasks, and have the capability to predict changes due to factors such as different interface design and different number of operators. For example, interacting models of multiple operators were developed (one setting up and initiating vehicles, one monitoring) and predicted improved performance over a model of a single operator. This was due to the ability of the purely monitoring operator to react at the same time the single operator was busy with a procedure.

GRBILModelingTool245

Figure13.3 GRBILOperatorControlUnit ModelingThroughDemonstration

Creating cognitive models is accomplished by demonstrating a series of actions (e.g., button clicks, object movement). The recorded series of actions is referred to as a goal state procedure in the GRBIL terminology. Any number of goal state procedures can be demonstrated and saved for later execution by an ACTR model. In the model, action steps are represented as a list that is retrieved from declarative memory, and so can be used to predict errors that are dependent on sequence length and positional confusion (cf. Anderson & Matessa, 1997; Matessa & Polson, 2006). Model tracing is used so that potential errors are only noted in the output while the correct retrievals are actually made so the model runs correctly.

246MATESSAANDMUI GUIBasedVisualMonitors

Once goal state procedures are defined, it would be useful for operator models to choose procedures conditionally based on the state of the environment. Matessa, Archer, and Mui (2007) describe how monitors were developed for GRBIL to provide this functionality. Monitors can be created that designate the condition in which the procedure should apply. There are two types of monitors: object monitors that check the status of an interface object and spatial monitors that use diagrammatic reasoning to check the status of graphical objects. Figure 13.4 shows the interface used to create monitors. A monitor control is either the name of an interface object, name of a graphical object, or names of multiple graphical-objects. An attribute to monitor is an attribute of an interface object defined in GRBIL or a diagrammatic reasoning routine for a graphical object. An attribute trigger value must match the value of an attribute before the given goal state procedure can be executed. Any number of monitors can be demonstrated and saved for later execution by an ACT-R model.

Figure13.4 VisualMonitor

GRBILModelingTool247 Operator/EnvironmentVisualization

GRBIL has recently been enhanced with the ability to display a timeline view of human resource use and environmental events. The human resources include vision, decision-making, memory, and motor response. The environmental events are automatically determined from the definition of the monitoring events used by the model, and therefore the events are limited to those most relevant to human control operations. The resulting visualization can be used to understand the dynamic interaction between human perception/action and environmental consequences. Figure 13.5 shows an example of the timeline view with the first column showing vision resource use, the second column showing decision-making, the third showing motor, the fourth showing a summary of vision and decisionmaking representing visual monitors, and the fifth showing the intersection precondition of the monitor.

Figure13.5 Operator/Environmentvisualization

248MATESSAANDMUI

HumanFactorsandArtificialIntelligence

As the OCU task has shown, GRBIL can naturally combine the human factors of operator modeling with the artificial intelligence of autonomous vehicle agents. In order to develop these agents, we selected a task network modeling environment, named the Improved Performance Research and Integration Tool (IMPRINT) developed by the ARL Human Research and Engineering Directorate (Archer & Adkins, 1999). IMPRINT provides a mature architecture and database structure that can easily incorporate a modeling method for representing goal-oriented behavior. The basic modeling capability is a classical reductionist method. IMPRINT requires the decomposition of a system mission into functions which, in turn, are decomposed into tasks. The tasks are linked together into a network describing the flow of events. The network can include various types of branching logic such as parallel branches, probabilistic branches, and repeating branches.

Collaboration

The GRBIL tool was involved in a number of ARL collaborations, including diagrammatic reasoning development, the design of a Dismounted Tactical Control Unit, and integration with a span of control evaluation tool. Diagrammatic Reasoning System. Chandrasekaran et al. (2004) developed the diagrammatic reasoning theory DRS, which consists of a basic set of primitive objects, information gathering capabilities called Perceptual Routines, and creation/modification operations called Action Routines. A diagrammatic object can be one of three types: point, curve, and region. Perceptual Routines can be qualitative (e.g., LeftOf, On, InsideOf), quantitative (e.g., Distance, Angle, Length), or related to object recognition (e.g., ScanPath, Intersect). Action Routines can create or modify objects (e.g., Translate, Rotate, PathFinder). In order to support collaboration between DRS and GRBIL development in the ADA CTA, Matessa and Brockett (2007) describe how the perceptual capabilities of ACT-R can be enhanced by the addition of DRS. In order to use DRS-enhanced ACT-R models to evaluate dynamic interfaces, the code from Matessa and Brockett (2007) was integrated into the GRBIL evaluation tool. This gives visual monitors the ability to be triggered by diagrammatic conditions such as the intersection of two projected vehicle paths (Figure 13.6).

GRBILModelingTool249

Figure13.6 DRSEnhancedvisualprocessing

Tactical Control Unit Interface. Another collaboration involved the design of a Dismounted Tactical Control Unit interface (Figure 13.7). Using the GUI-based interface design capability of GRBIL, the spatial layout and functionality of the interface were able to be quickly generated without writing any code.

Figure13.7 DTCUdisplay

250MATESSAANDMUI
Span of Control Evaluation. The Autonomous Vehicle Operator Span of Control Evaluation Tool (AVOSCET Micro Analysis and Design, 2005) is intended to be a tradeoff analysis tool a tool (developed with an ARL SBIR) that analysts can use to create different scenarios involving unmanned vehicles and operators in order to evaluate the effects and the aptness of different operator-to-vehicle ratios. Although AVOSCET does include an operator model in its underlying simulation, the operator model does not represent the complexities of an operators interface nor the effect his interface, or control unit, might have on his ability to control unmanned systems. In order to begin to address this concern, AVOSCET has been integrated with GRBIL. In the integration, AVOSCET models the world, including the vehicles and the operators (to an extent), GRBIL models the operators control unit, and ACT-R models the operators actions on his control unit. Therefore, AVOSCET needs to be able to broadcast information regarding the state of the world, GRBIL needs to be able to broadcast information (to ACT-R only) with respect to interface design (how far apart are the various buttons, how many levels deep is a given action, etc.), and ACT-R needs to be able to broadcast information about what actions the operator is executing on the interface and how long it is taking. In order to facilitate all of the necessary communication between the three services, message transmissions are handled by the communication hub described above. The handoff point in AVOSCET is bound around the intervention execution tasks. All other operator tasks are still modeled in AVOSCET as usual but once an operator has received a call for help and has decided on the best intervention, AVOSCET makes a call to the communication hub notifying the other services that the operator needs to execute a particular intervention. At this point, GRBIL and ACT-R cooperate to facilitate the chosen intervention. Meanwhile, the given operator and vehicle are paused in AVOSCET until control is passed back and a task time is returned from the other services. Figure 13.8 illustrates where this handoff occurs inside the AVOSCET model.

GRBILModelingTool251

Figure13.8 DepictionofthehandoffinAVOSCET

Once a new task time for the given intervention is returned to AVOSCET, the simulation resumes as usual until another intervention has been chosen and is ready to be executed. The two screenshots below show the GRBIL representation of the OCU, and the runtime layout of GRBIL communicating with ACT-R respectively. The benefit in adopting the functionality of GRBIL and ACT-R in this effort lies in the increased fidelity with respect to the representation of the operators cognition and his user interface. In the normal AVOSCET model, interventions are assigned lump times to represent the collective time it takes, on average, for an operator to execute an intervention through his interface. However, these times do not take into account any differences in user interfaces. Given the variability in user interface complexities and architectures, it is important to see how such a component can affect the overall performance of an unmanned vehicle operator. Preliminary results from the integration of AVOSCET with GRBIL and ACT-R indicates much more realistic operator processing times than the ones that were originally assumed inside the AVOSCET simulation. Table 1: below summarizes the results from the runs exercised for this proof of concept effort.

252MATESSAANDMUI

Table 1: Results from the AVOSCET/GRBIL integration

Task RouteReplan (addingnewwaypoints) RouteReplan (editingexistingwaypoints) AVOSCET TaskTime (seconds) 116.91 121.62 GRBIL TaskTime (seconds) 32.11 23.38

Note that in this case, AVOSCET predicts fairly conservative intervention durations as compared to GRBIL and ACT-R. However, the most important thing to note here is perhaps not how the two compare to each other in this one case but rather how both would compare over several different cases combined. Obviously, given AVOSCETs current hard-coded intervention processing times, there would not be any variability in intervention durations given different interface complexities or architectures. However, if interventions were processed by GRBIL and ACT-R for a variety of user interface styles, we would likely see an appropriate variability in intervention durations. Human data were not collected for the procedures used in the AVOSCET/GRBIL comparison. In order to begin validating GRBILs timing predictions, performance data from human operators were collected. Two participants familiar with the OCU interface were asked to set up the routes for two vehicles, which involves using the interface to indicate waypoints on a map that each vehicle should pass through. The model predictions and human data are shown in Figure 13.9.

GRBILModelingTool253

60

50

time (seconds)

40 Model 30 Operator 1 Operator 2 20

10

0 Task 1 Task 2

Figure13.9 TaskcompletiontimesforGRBILmodelandhumanoperators

These results indicate that GRBIL is sensitive to the decreased time requirement of Task 2 and is capable of predicting the latency of actions by the two operators to some degree of accuracy.

FINDINGS

This ADA program has demonstrated that even complex interfaces can be duplicated by GRBIL. The creation of interfaces requires only drag-and-drop placement of controls and a menu-driven description of functionality. The project has also shown the ability to automatically generate cognitive models that perform dynamic spatial monitoring. The creation of these models requires only the demonstration of procedures and the creation of monitors with a simple interface. Models can predict time to complete tasks, errors, and the consequences of multiple operators. The dynamic interaction of these models and the environment is captured by a timeline visualization which assists in performance analysis.

254MATESSAANDMUI IMPLICATIONSFORFUTUREARMYWORK

Future challenges include the addition of learning trajectory and immediate feedback to interface design. Since GRBIL models retrieve interface procedure steps from memory, it will be possible to use ACT-R's theory of production compilation to predict the trajectory of learning from error-prone retrieval to efficiently compressed performance. Also, having operator models running in the background could provide immediate feedback during changes to the interface design.

REFERENCES
Anderson,J.R.2007.Howcanthehumanmindoccurinthephysicaluniverse?New York:OxfordUniversityPress. Anderson,J.R.,Bothell,D.,Lebiere,C.&Matessa,M.1998.Anintegratedtheoryof listmemory.JournalofMemoryandLanguage,38,pp.341380. Anderson,J.R.&Matessa,M.1997.Aproductionsystemtheoryofserialmemory. PsychologicalReview,104(4),728748. Archer,S.G.&Adkins,R.1999.IMPRINTUsersGuidepreparedforUSArmy ResearchLaboratory,HumanResearchandEngineeringDirectorate,April1999. Archer,S.,Archer,R.,&Matessa,M.2007.OurGRBILhasaSplitPersonality. ProceedingsoftheSAEDigitalModelingConference(pp.871876),June2007,Seattle, WA. Chandrasekaran,B.,Kurup,Banerjee,Josephson,&Winkler.2004.AnArchitecture forProblemSolvingwithDiagrams,inDiagrammaticRepresentationandInference,A. Blackwell,K.MarriottandA.Shomojima,Eds.,LectureNotesinArtificialIntelligence 2980,Berlin:SpringerVerlag,pp.151165. John,B.,Prevas,K.,Salvucci,D.,&Koedinger,K.2004.PredictiveHuman PerformanceModelingMadeEasy.InProceedingsofCHI,2004(Vienna,Austria,April24 29,2004)ACM,NewYork. Matessa,M.,Archer,R.,&Mui,R.2007.DynamicSpatialReasoningCapabilityina GraphicalInterfaceEvaluationTool.InProceedingsoftheEighthInternational ConferenceonCognitiveModeling(pp.5560),AnnArbor,MI. Matessa,M.&Brockett,A.2007.UsingaDiagramRepresentationSystemwithACT R.InProceedingsofBehaviorRepresentationinModelingandSimulation(BRIMS'07) Matessa,M.,&Polson,P.2006.ListModelsofProcedureLearning.InProceedings oftheInternationalConferenceonHumanComputerInteractioninAeronautics(pp. 116121),SanFrancisco,CA MicroAnalysisandDesign.2005.SoldierPerformanceAssessmentforControlof MultipleUnmannedSystems.TechnicalReportARLCR709.Boulder,CO.

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T A N D L A U R E L A L L E N D E R

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T AND LAUREL ALLENDER

This book, produced by the Partners of the Army Research Laboratory Advanced Decision Architectures Collaborative Technology Alliance, also serves as the Final Report for DAAD19-01-2-0009.

SECTIONIII.ACTINGONBATTLEFIELDINFORMATION

SECTIONIII ACTINGONBATTLEFIELDINFORMATION

256SECTIONIII

Chapter

14

14. MODELINGANDMEASURINGSITUATIONAWARENESSININDIVIDUALS ANDTEAMS

CLEOTILDEGONZALEZ,PH.D. DynamicDecisionMakingLaboratory CarnegieMellonUniversity,Pittsburgh,PA LELYNSANER,PH.D. DynamicDecisionMakingLaboratory CarnegieMellonUniversity,Pittsburgh,PA MICAENDSLEY,PH.D. SATechnologies,Marietta,GA CHERYLA.BOLSTAD,PH.D. SATechnologies,Marietta,GA HAYDEEM.CUEVAS,PH.D. SATechnologies,Marietta,GA

INTRODUCTION

257

258GONZALEZETAL.
Situation Awareness (SA) is a complex construct that cannot be fully understood from a single perspective. Rather, SA entails a multifaceted process in which individual and team factors, at both the micro and macro levels, need to be integrated. For example, research on team SA needs to take into account the factors that contribute to a given persons individual SA while integrating the factors that contribute to any two team members shared SA, as well as team or organizational SA. Also, the complexity of SA arises from multiple individual cognitive abilities including learning, working memory, different levels of perception, understanding, projection, etc. These individual factors interact with multiple team and organizational factors such as geographical distribution, collaborative tool usage, network proximity, similarity of the individuals' background experiences, and familiarity with each others' skills, among others. In this chapter we bring together different levels and factors of SA complexity into an integrated framework. Throughout the duration of the Army Research Laboratories Advanced Decisions Architectures (ARL-ADA) research program, we have addressed the complexity of SA by advancing the development and validity of measures at the individual and team levels. We have also created computational models of SA that have moved forward the descriptive nature of SA into more concrete and formal representations. This chapter summarizes all the past work we have done within this program and attempts to integrate all the previous findings into a research framework. The details of each of these research pieces can be found in the previous original publications. The integrated framework presented here came from a combination of scientific research methods at the micro and macro levels that included computational models, and laboratory and field studies. The framework integrates our past findings from experimental laboratory studies and ACT-R cognitive computational models on individual SA to field experiments and social network analysis of team and shared SA.

COMPUTATIONALMODELSOFSA

Computational models have been developed to address individual cognitive aspects of SA, such as recognition, perception, and learning; to address design aspects of SA, such as visual interfaces and goal directed task analysis; and to address organizational and team aspects of shared SA, such as communication network distance, physical proximity, and knowledge and background similarity. The existence of these different models illustrates the challenge that one faces in the study of SA generally. Efforts demonstrate a positive approach to advance our scientific understanding of SA because they address aspects of the problem that are salient from different perspectives. This research demonstrates that

ModelingandMeasuringSituationAwareness259
individual and organizational levels of SA are not incompatible, but rather they complement each other. As suggested recently (Lebiere, Gonzalez, & Warwick, 2009), because much of the history of computational cognitive modeling has been microcognitive, these psychological architectures are a good starting point to develop macrocognitive or organizational models of SA.

ComputationalModelsofIndividualAspectsofSA

Many computational models have been developed to address some important aspects of SA at the individual level, such as perception, attention, memory, and learning among others (Gonzalez, Juarez, & Graham, 2004; Juarez & Gonzalez, 2003, 2004; McCarley, Wickens, Goh, & Horrey, 2002). Some of these models have been framed in the context of military applications such as perceiving and understanding a situation in the battlefield, and many of these models involve leading cognitive architectures such as ACT-R (Anderson & Lebiere, 1998) and SOAR (Newell, 1990). Here we summarize our own development of cognitive models of situation awareness (CMSA) at the individual level by using the ACT-R architecture (Anderson & Lebiere, 1998). A SA meta-architecture was developed to propose a conceptual design for CMSA (Gonzalez et al., 2004; Juarez & Gonzalez, 2003). The meta-architecture presented a set of modules that encapsulate the most essential cognitive aspects of SA. A demonstration of the architecture was developed in a commanding decision making mission in the OneSAF Test Bed (OTB), which interacts in real-time with the ACT-R cognitive model. This architecture consists of a recognition, assessment, prediction, and control modules. The recognition module gathers and encodes visual sensory information from the environment (in OTB) and encodes the information into ACT-R representations. The assessment module represents and manipulates information and updates, and maintains information on the environment (e.g., updates information about location, status, and types of entities in the battlefield). The prediction module constructs and evaluates hypotheses about the function of an entity in a plan to determine the probability and possible success of a course of action. The control module then provides the capability to select actions that will change the state of the world. The SA meta-architecture is used as an experimental platform to study the relationships among the model parameters and SA performance (Juarez & Gonzalez, 2004). The architecture allows one to manipulate a number of parameters and evaluate their impact in SA, including parameters of the task such as scenario complexity and the 'human' (ACT-R model) parameters like experience.

260GONZALEZETAL.
A major question arising from these developments was which method to select for evaluating the SA of the model. One widely tested and validated approach to assessing SA in human subjects is the Situation Awareness Global Assessment Technique (SAGAT) (Endsley, 1995a). SAGAT allows for immediate assessment of SA by querying operators on their current perceptions, assessments, and projections of the situation (SA levels 1, 2, and 3 respectively). SAGAT has been empirically validated with regard to its utility to provide valid and reliable assessment of SA across a variety of domains (Endsley, 1990; Endsley, Sollenberger, & Stein, 1999; Hogg, Torralba, & Volden, 1993; Matthews, Pleban, Endsley, & Strater, 2000; Riley & Kaber, 2001). As with human SA measurement, we used SAGAT to measure SA for our computational models. The validation with SAGAT was implemented successfully (Juarez & Gonzalez, 2004); however, the cognitive aspects of the model still need to be validated with human data. A validation of the models predictions requires the comparison of SAGAT data from human participants to the models SAGAT data. Obtaining realistic human data in this context continues to be a challenge for SA modeling research. However, cognitive models have many other practical applications. One of them, presented next, is the use of computational models in conjunction with simulation tools in the evaluation of graphical interfaces.

ComputationalModelsofDesignAspectsofSA

Because of the relevance of SA in Army operations, methods and tools that help us predict and mitigate low levels of SA are valuable for researchers and interface designers alike. In fact, accurately and reliably assessing how well an interface design concept supports user SA is essential to any design process. Given the limited time and funding available for developing multiple high fidelity prototypes, designers need a tool that will allow them to evaluate multiple design concepts early in the design cycle to minimize expense while maximizing the ultimate utility of the user interface. The ability to quickly predict the SA afforded by a particular display design is essential for the success of the computer systems researchers develop. Our research described in Gonzalez, Juarez, Endsley, and Jones (2006) presents how the computational architecture and ACT-R model of SA were used in a new application for the prediction of the SA elicited through a graphical user interface (GUI), and how the CMSA could be used in a new tool for prototyping and evaluating the users SA through simple GUIs. This work is summarized next.

ModelingandMeasuringSituationAwareness261
The SA-oriented design process (Endsley, 2003; Endsley, Bolte, & Jones, 2003), provides a key methodology for developing user-centered displays by focusing on optimizing situation awareness. By creating designs that enhance an operators awareness of what is happening in a given situation, SA can improve dramatically. The design process starts with the gathering of SA requirements. These are determined through a cognitive task analysis technique called GoalDirected Cognitive Task Analysis (GDTA). A GDTA identifies the major goals and sub-goals for each job; the critical decisions the individual must make to achieve each goal and sub-goal; and the situation awareness requirements needed for making these decisions and carrying out each goal. These SA requirements focus not only on what data an individual needs, but also on how that information is integrated or combined to address each decision. This process forms the basis for determining the exact information (at all three levels of SA: perception, comprehension, and projection) that needs to be included in the display visualizations. A final step of the SA-oriented design process emphasizes the objective measurement of SA during man-in-the-loop simulation testing using SAGAT. SAGAT provides a sensitive and diagnostic measure of SA that can be used to evaluate new interface technologies, display concepts, sensor suites, and training programs (Endsley, 1995; Endsley & Garland, 2000). The Designers Situation Awareness Toolbox (DeSAT) was created to assist designers in carrying out the SA-oriented design process (Endsley, 2005). It includes (1) a software tool for easily creating, editing, and storing effective GDTAs, (2) A GDTA checklist tool, to aid designers in evaluating the degree to which a display design meets the SA requirements of the user, (3) A SAoriented design guidelines tool, which guides designers in determining how well a given design will support user SA, and (4) A SAGAT tool, which allows designers to rapidly customize SAGAT queries to the relevant user domain and to administer SAGAT during user testing to empirically evaluate display designs. The concept of DeSAT was used as a starting point to develop a new CMSA tool for predicting and evaluating how different GUI designs elicit SA. The CMSA tool (described in detail in Gonzalez et al., 2006) integrates the GDTA process through the DeSAT to generate a CMSA in a semi-automated way. The process of generating the CMSA from the GDTA is presented in Figure 14.1. The GDTA, which is the DeSAT output (an .xml file), and the image files that define a GUI are the two starting points of the process. The GDTA description consists of a list of goals, decisions, and information requirements to support situation awareness in a very specific domain (Endsley, 2005). The GUI is described in terms of its graphical components including widgets (e.g., buttons, graphical sections, icons) and their behaviors, as well as their organization in the graphical interface. The tool also allows one to define the domain knowledge and to identify the background knowledge needed to perform a task.

262GONZALEZETAL.

Figure14.1 AprocessfortheautomaticevaluationoftheSAelicitedbyaGraphical UserInterface,accordingtoapredefinedGDTA,fromGonzalezetal. (2006)

As a next step, the tool defines correspondence mappings between the GDTA components and specific widgets or graphical elements defined in the GUI. The mapping process helps produce a high-level definition language for SA from which an ACT-R model of SA is then generated and executed through the ACTR architecture (Anderson & Lebiere, 1998).

ModelingandMeasuringSituationAwareness263
The functioning of this CMSA tool was demonstrated with a realistic example of a GUI prototype design (Gonzalez et al., 2006). The example used a selected set of GUIs to perform the logistics tasks of a company unit (see example in Figure 14.2). The user of this system was expected to plan supplies, project the demand for a specific time period, and obtain the consumer trends for some items during specific time periods. The task involves high-level cognitive processes that sometimes are not necessarily expressed by a click of a button in a GUI. This logistics task consists of four sub-goals, each of which is represented in one of five screens in the application.

Figure14.2 ExampleofaGUIusedinthedemonstrationoftheevaluationoftheSA inGonzalezetal.(2006)

In this example, the GDTA defines information requirements, decisions, and goals. Although the GDTA can be interpreted by a human, the computer model needs domain knowledge to understand the GDTA components. This CMSA tool defines domain objects and constraints to support the model on making decisions. Knowledge about domain objects in our example consists of: roads, road condition and road type; terrain type, elevation, type of soil; type of

264GONZALEZETAL.
vehicle, strength, capabilities, location; enemy threats; fuel requirements; mission, supply and shipping schedule; history of consumption, requirements, and usage rate. In addition, the system requires the knowledge about the criteria to select alternatives. For example, the system must contain definitions for best path which is mapped to shortest distance, minimal threats, and road conditions. The CMSA tool produced a set of commands to generate a memory chunk for every element in the GDTA, for every element in the graphic interface, and for every domain object and domain constraint. Finally, the system generated ACTSA code with the procedural description of every task. The system then can be executed to collect the performance times and the number of operations needed to perform a task. This work, described in Gonzalez et al. (2006), demonstrates the practical applications of CMSA. Most of the tools currently known to automatically generate cognitive models from graphical user interfaces have been created to evaluate the interface itself. That is, most of the work done on this area generates a model to evaluate the usability or the human-computer interaction of the interface. In addition, most of the tools developed up to date deal with very simple interface actions, such as point, click, and move. Our tool focuses not on the evaluation of the interface elements, but rather on the evaluation of the SA afforded by an interface. As SA involves high level actions, such as prediction of future status, it is important to include high level operators in our models, beyond simple interface actions.

ComputationalModelsofTeamandOrganizationalAspectsofSA

Computational models of SA at the team and organizational levels have also been developed to represent organizational aspects of SA, such as communication, shared understanding, shared workload, etc. Successful models have been based on Social Network Analysis (SNA) theory developed under this ARL-ADA program (Graham, Gonzalez, & Schneider, 2007). This work is summarized here. SNA is a technique that seeks to quantify the relationships among people in an organization. People and organizations are represented as nodes in a network, and the relationships (e.g., information flows) between people are represented as lines drawn between these nodes. Thus, a social network is a graph consisting of individuals and connections among them, where each connection is associated with some form of communication or relationship between the nodes (Borgatti, 1994).

ModelingandMeasuringSituationAwareness265
Communication data among an organizations members can be gathered from shared e-mail headers, chat room traffic, instant messaging, and phone calls, or by surveying the individuals (Wasserman & Faust, 1994). While each of these communication media has different qualities, measure relevance is determined by the organizational context and collaborative tool characteristics. Graham, Gonzalez, and Schneider (2007) gathered communication data in a field experiment conducted at the Fort Leavenworth Battle Command Battle Laboratory. The US Army was in the opening phase of a ten-year organizational design process for a knowledge-centric command and control element. In support of this initial effort, the Fort Leavenworth Battle Command Battle Laboratory (BCBL) was conducting the first high fidelity experiment to determine organizational constructs that would support command and control in the Transformation Force. The experiment assumed a network-centric staff cell structure supported by a higher level of automation. Figure 14.3 from Graham et al. (2007) demonstrates a SNA representation of 90 minutes of communication relationships in a 10-person command-and-control cell of the network organization. The 10 nodes are members of a Command Integration Cell (CIC) designed to coordinate the activities of other functionally oriented cells, and constitute a subset of the 56-member prototype network organization that was the focus of this research.

Figure14.3

266GONZALEZETAL. CommandIntegrationCellofaprototypenetworkorganization(from Graham,Gonzalez,&Schneider,2007)

We used traditional SNA measures such as network density, network distance, physical distance, and self-forming teams to draw conclusions about the shared SA of this team, and to ultimately provide feedback to the Battle Lab on unit configuration.

ConclusionsonComputationalModelsofSA

As summarized above, our past research has produced computational representations of different aspects of SA including cognitive, design, and organizational levels. These research efforts demonstrate the complexity in representing and reproducing human SA at different levels of specificity. These efforts also suggested that the development of SA measures, at both the individual and organizational levels, is essential to make progress in assessing SA, both behaviorally and computationally. While the SA literature base is large, many of the metrics currently used to measure SA focus only on component aspects of SA and dont necessarily measure SA as an identifiable, integrated construct. This is partly due to the fact that the definitions of SA and shared SA are still under some debate. The need for better measures is particularly clear with respect to Shared SA. In order to improve the computational representations of SA, researchers need to develop valid approaches to assess both individual and shared SA.

MEASURESOFSA

Over the duration of this research program, we have investigated SA at both the individual and team levels. At the individual level we focused on the learning of SA and the influence of individual aspects of SA using laboratory dynamic decision-making tasks. At the team level, we focused on the organizational measures of SA through the data collected in several field experiments in the Army Laboratories such as Ft. Leavenworth and the Joint Personnel Recovery Agency (JPRA). These efforts are summarized next.

ModelingandMeasuringSituationAwareness267 MeasuresofIndividualSA

Task practice may allow people to develop the mental models, schemas, and goal-directed processing that are critical for SA in most domains (Endsley et al., 2003). An important first step in designing more effective displays and cognitive models of SA is to understand how individuals naturally improve their SA through practice. Similarly, cognitive mechanisms, such as working memory and learning or experience, are commonly assumed to influence SA (Endsley, 1995; Endsley & Robertson, 2000). Although there is some clear evidence regarding the influence of working memory on SA (Durso, Bleckley, & Dattel, 2006), we only have a weak understanding of the relationship between working memory and experience. A common method to measure SA described earlier in this chapter is SAGAT. In this method a task simulation is stopped at random points and a participant answers a set of queries about the situation. Queries may be answered while the simulation display is not visible or covered (Endsley, 1995) or while the display is visible and uncovered (Durso et al., 1995). Gonzalez and Wimisberg (2007) conducted a laboratory experiment in which these two individual SA measurement conditions were used while participants played a computer simulation over several days. SA was measured using two different query conditions methods, a covered condition in which queries were asked while the display was blanked out and an uncovered condition in which queries were asked while the display was shown. A working memory measure was collected from participants as well. Results, reported in detail in Gonzalez and Wimisberg (2007), showed that SA improves with practice when measured in a covered display condition, but not when measured in the uncovered condition. Furthermore, the effect of experience appears only in the covered condition for perception and comprehension queries, not for projection queries. Another main conclusion from this empirical study was that the moderating effect of working memory on SA changes with task practice, and depends on the conditions in which SA is measured. The participants level of working memory predicted SA scores only in the covered condition, such that, when the SAGAT requirement of blanking out the screen while an operator answers queries was implemented, working memory became a significant factor for SA. We also found that the effect of working memory was more relevant for SA at the perception level than for SA at the projection level. Because of the nature of perception-level queries, correct answers fully depend on the visibility of the elements on the display. In contrast, accurate answers to projection-level queries depend on a deeper understanding of the task.

268GONZALEZETAL.
Finally, we found a significant decrease in the correlation between SA and working memory with practice in the task. This result suggests that as experience accumulates, there is a decreasing need to use working memory to maintain SA, although working memory may still be necessary to task performance. Please refer to the original publication (Gonzalez and Wimisberg, 2007) for the details of the design and results.

MeasuresofSharedSA

Our work has extended from the individual aspects of SA to SA in teams and organizations (Graham, Gonzalez, & Doyle, 2003; Graham, Gonzalez, & Schneider, 2007; Saner, Bolstad, Gonzalez, & Cuevas, In Press). In order to function effectively as a team, operators need to develop an accurate understanding of the situation, that is, possess a high level of situation awareness (SA). If people are working in teams and any one of the team members has poor SA, it can undermine the success of the entire team. However, it is often the case that not all team members need to know all of the same information in a given situation (Endsley, 1995). In many situations, individuals in a team possess specialized knowledge and they rely on each other to perform particular tasks. As such, although each team member needs to have good individual SA on the information that is relevant to his/her job, the similarity of the individual SA among team members is only important with respect to their shared task requirements (Saner et al., In Press). Our approach, explained and validated in Saner, et al. (In Press), is based on assessing shared SA from objective measures of individual SA. We used the SAGAT methodology and compared participant responses to ground truth (reality) to evaluate the accuracy of their SA at a given moment in time. Despite the general validity of most existing methods for calculating SA scores, one limitation that they still possess is they often fail to account for errors in SA, in terms of both the amount of information that is thought to be relevant and in the accuracy of a persons knowledge of it. As such, assessments of SA derived from these methods may be inflated. In order to address both accuracy and similarity of SA, we developed a method for calculating shared SA that first derives true SA scores and then assesses the similarity between the scores of two individuals. Our preliminary analysis suggests that failure to compensate for error in SA might lead to overestimation of performance in a situation. In addition to developing this new measure of shared SA, Saner et al. (In Press) evaluated whether it was related to cognitive and social factors that often influence performance in team contexts. We investigated SA in two interdisciplinary, multi-team military training exercises. The first data set was obtained by measuring SA among 17 participants engaged in a joint rescue

ModelingandMeasuringSituationAwareness269
operation training exercise at the Joint Personnel Recovery Agency (JPRA). Participants were divided into four physically distributed team cells (Army, Navy, Special Forces, and Joint Operations), and had to coordinate rescue maneuvers in response to critical events. In the exercise, participants completed several distinct scenarios over several days, and were rotated into different cells for each scenario. The second data set was derived from a battalion command training exercise at Ft. Leavenworth. In this exercise, 24 participants were divided among four battalion level command positions and five core operational groups (Command Integration, Fires and Effect, Information Superiority, Maneuver and Support, and Build and Sustain), which were physically distributed. Participants remained in the same assigned positions throughout the duration of this exercise. In addition, there were clearly defined roles within each group, including officers in charge. The latter exercise was much more complex and there were more groups coordinating and more levels of rank, the situation was much more complex in this exercise. The initial expectation was that the more distance there is between an operator and the operational center of the organization, the less SA that operator will have. Results from the rescue operation data set revealed a significant relationship between shared SA and participants distance from the joint service cell, which was expected to act as the organizational hub of the C3 structure in that context. In contrast to what was expected, shared SA was significantly better the further away participants were from the hub., This finding, though counterintuitive, did provide evidence that an individuals role and position within an organization affect the level of shared SA that can be achieved with other individuals in the network. One possible explanation is that participants in the peripheral teams were doing more direct processing of information and did not need the help of the joint service cell to utilize information effectively, but this and other possibilities are still under investigation. The analysis of battalion command data is still in progress, and is focused more specifically on the social network factors. Cognitive load data was again collected, but detailed data on years of experience and communication were unavailable. However, the role of individuals within cells is being added as a predictor. The Command Integration cell was also an organizational hub in this exercise, so participants distance from this cell and physical distance from each other are factors in this analysis as well. The goals of this analysis are to replicate the relationships between the predictors and shared SA that are common to both samples, and also to extend our assessment of the social dynamic influences that were observed in the recovery operation context.

MainConclusionsfromMeasuresofIndividualandSharedSA

270GONZALEZETAL.
The success of the CMSA depends on finding appropriate and robust measures of individual and shared SA. The work described in this chapter extends the current state of the art on SA modeling and measurement. In addition, our shared SA measure builds upon work conducted with individual SA measures in an effort to develop new ways to account for error in SA that might otherwise misrepresent performance in a situation. We validated and extended the current understanding of individual measures of SA, developed procedures to measure the degree of shared SA between two team members and to improve the accuracy of shared SA scores, and created computational models that expand individual and team SA. In future work on the measurement of shared SA, we intend to address more of the dynamic factors involved in team work (e.g., information flow, communication flow, physical location, etc.), particularly the specific roles adopted by individuals and the distribution of task experts within the larger team. If a team is ultimately successful in completing their task, there must be a way to describe why particular knowledge, actions, or coordination patterns led to success. Without a measure of what particular beliefs the participants were operating on, and whether participants were operating on similar or different beliefs when they took similar actions, there is no sure way to reproduce success in later, similar situations. The effect of SA on performance is a question that is still unanswered in Shared SA research, and we plan to pursue this in future work.

ACKNOWLEDGMENTS

This work was supported by the Army Research Laboratory (DAAD19-01-20009) award to Cleotilde Gonzalez and Mica Endsley. Many others contributed to the research summarized in this paper. We would like to particularly thank Dr. Octavio Juarez for his contributions in building some of the tools and models summarized in this paper and originally reported in respective publications.

REFERENCES
Anderson,J.R.,&Lebiere,C.(1998).Theatomiccomponentsofthought.Hillsdale, NJ:LawrenceErlbaumAssociates. Borgatti,S.P.(1994).Aquorumofgraphtheoreticconcepts.Connections,17(2),47 49. Durso,F.T.,Bleckley,M.K.,&Dattel,A.R.(1996).DoesSAaddtothevalidityof cognitivetests?HumanFactors,48(4),721733. Durso,F.T.,Truitt,T.R.,Hackworth,C.A.,Crutchfield,J.M.,Nikolic,D.,Moertl,P. M.,etal.(1995).Expertiseandchess:Apilotstudycomparingsituationawareness

ModelingandMeasuringSituationAwareness271
methodologies.InD.J.Garland&M.R.Endsley(Eds.),Experimentalanalysisand measurementofsituationawareness(pp.295303).DaytonaBeach,FL:EmbryRiddle AeronauticalPress. Endsley,M.R.(1990).Amethodologyfortheobjectivemeasurementofsituation awareness.InSituationalAwarenessinAerospaceOperations(AGARDCP478)(pp.1/1 1/9).NeuillySurSiene,France:NATOAGARD. Endsley,M.R.(1995).Measurementofsituationawarenessindynamicsystems. HumanFactors,37(1),6584. Endsley,M.R.(2005).Designer'sSituationAwarenessToolkit.Fromhttp:// www.satechnologies.com/html/products.html. Endsley,M.R.,Bolte,B.,&Jones,D.G.(2003).EvaluatingdesignconceptsforSA.In M.R.Endsley,B.Bolte&D.G.Jones(Eds.),Designingforsituationawareness:An approachtohumancentereddesign(pp.218244).London:Taylor&Francis. Endsley,M.R.,&Garland,D.J.(2000).Pilotsituationawarenesstrainingingeneral aviation.InProceedingsofthe14thTriennialCongressoftheInternationalErgonomics Associationandthe44thAnnualMeetingoftheHumanFactorsandErgonomicsSociety (pp.2.357352.360).SantaMonica,CA:HumanFactorsandErgonomicsSociety. Endsley,M.R.,&Robertson,M.M.(2000).Trainingforsituationawarenessin individualsandteams.InM.R.Endsley&D.J.Garland(Eds.),Situationawareness analysisandmeasurement(pp.349366).Mahwah,NJ:LawrenceErlbaumAssociates. Endsley,M.R.,Sollenberger,R.,&Stein,E.(1999).Theuseofpredictivedisplaysfor aidingcontrollersituationawareness.InProceedingsoftheHumanFactorsand ErgonomicsSociety43rdAnnualMeeting(pp.5155).SantaMonica,CA:HumanFactors andErgonomicsSociety. Gonzalez,C.,Juarez,O.,Endsley,M.R.,&Jones,D.G.(2006).Cognitivemodelsof situationawareness:Automaticevaluationofsituationawarenessingraphicinterfaces. PaperpresentedattheFifteenthConferenceonBehaviorRepresentationinModeling andSimulation,Baltimore,MD. Gonzalez,C.,Juarez,O.,&Graham,J.(2004).Cognitiveandcomputationalmodels astoolstoimprovesituationawareness.InProceedingsofthe48thAnnualMeetingof theHumanFactorsandErgonomicsSociety.SantaMonica,CA:HumanFactorsand ErgonomicsSociety. Gonzalez,C.,&Wimisberg,J.(2007).Situationawarenessindynamicdecision making:Effectsofpracticeandworkingmemory.JournalofCognitiveEngineeringand DecisionMaking,1(1),5674. Graham,J.,Gonzalez,C.,&Doyle,M.(2003).Usingcommunicationpatternsinthe designofanadaptiveorganizationalstructureforcommandandcontrol.InProceedings ofthe47thAnnualMeetingoftheHumanFactorsandErgonomicsSociety.SantaMonica, CA:HumanFactorsandErgonomicsSociety. Graham,J.,Gonzalez,C.,&Schneider,M.(2007).Adynamicnetworkanalysisofan organizationwithexpertiseoutofcontext.InR.Hoffman(Ed.),Expertiseoutofcontext: ProceedingsoftheSixthInternationalConferenceonNaturalisticDecisionMaking(pp. 385402).NewYork:LawrenceErlbaumAssociates. Hogg,D.N.,Torralba,B.,&Volden,F.S.(1993).Asituationawarenessmethodology fortheevaluationofprocesscontrolsystems:Studiesoffeasibilityandtheimplicationof use.Storefjell,Norway:OECDHaldenReactorProject. Juarez,O.,&Gonzalez,C.(2003).MASA:Metaarchitectureforsituationawareness. PaperpresentedattheConferenceonBehaviorRepresentationinModelingand Simulation(BRIMS),Scottsdale,AZ.

272GONZALEZETAL.
Juarez,O.,&Gonzalez,C.(2004).Situationawarenessofcommanders:Acognitive model.PaperpresentedattheConferenceonBehaviorRepresentationinModelingand Simulation(BRIMS),Arlington,VA. Lebiere,C.,Gonzalez,C.,&Warwick,W.(2009).Convergenceandconstraints revealedinaqualitativemodelcomparison.JournalofCognitiveEngineeringand DecisionMaking. Matthews,M.D.,Pleban,R.J.,Endsley,M.R.,&Strater,L.D.(2001).Measuresof infantrysituationawarenessinavirtualMOUTenvironment.InProceedingsofthe1st HumanPerformance,SituationAwarenessandAutomationConference(pp.1519). Savannah,Georgia:SATechnologies,Inc. McCarley,J.S.,Wickens,C.D.,Goh,J.,&Horrey,W.J.(2002).Acomputational modelofattention/situationawareness.InProceedingsofthe46thAnnualMeetingof theHumanFactorsandErgonomicsSociety(pp.16691673).SantaMonica,CA:Human FactorsandErgonomicsSociety. Newell,A.(1990).Unifiedtheoriesofcognition.Cambridge,MA:HarvardUniversity Press. Riley,J.M.,&Kaber,D.B.(2001).Utilityofsituationawarenessandattentionfor describingtelepresenceexperiencesinavirtualtelepresencetask.InB.Das&W. Karwowski(Eds.),Proceedingsofthe2001InternationalConferenceonComputerAided ErgonomicsandSafety.Rome:Italy:InternationalErgonomicsAssociation. Saner,L.D.,Bolstad,C.A.,Gonzalez,C.,&Cuevas,H.M.(2009).Measuringand predictingsharedsituationawarenessinteams.Unpublishedmanuscriptunderreview. Wasserman,S.,&Faust,K.(1994).Socialnetworkanalysis:Methodsand applications.Cambridge:CambridgeUniversityPress.

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T A N D L A U R E L A L L E N D E R

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T AND LAUREL ALLENDER

This book, produced by the Partners of the Army Research Laboratory Advanced Decision Architectures Collaborative Technology Alliance, also serves as the Final Report for DAAD19-01-2-0009.

SECTIONIII.ACTINGONBATTLEFIELDINFORMATION

SECTIONIII ACTINGONBATTLEFIELDINFORMATION

256SECTIONIII

Chapter

15

15. ACOMPUTATIONALMODELOFNATURALISTICDECISIONMAKINGAND THESCIENCEOFSIMULATION

WALTERWARWICK,PH.D. AMYSANTAMARIA,PH.D. AlionScienceandTechnology,Boulder,CO

INTRODUCTION

With its roots in experimental psychology, computational cognitive modeling is most often seen as an exercise in theory testing. This view holds that theories of cognition are formalized as executable software programs; these programs are exercised as simulations of laboratory tasks to generate predictions that are then compared to actual human performance on those same tasks. Good fits validate good theories. Although this view has received its share of scrutiny, most of the concern has focused on the finer points of the method (e.g., the quality of the data used for validation) while the most basic assumption of the entire enterprisethat theories of cognition can be expressed directly as computational models remains unquestioned. Of course, some would argue that the relationship between cognitive theory and computer model is immediate. Indeed, Simon (1996) wrote, if computers are organized somewhat in the image of man, then the computer becomes the obvious device for exploring the consequences of alternative organizational assumptions for human behavior (pg. 21; emphasis added). Similarly, Newell (1990) proposed that the layered abstractions of computing systems provide a useful approach for describing and understanding knowledge systems. Even those who espouse a connectionist view of the

273

274WARWICKANDSANTAMARIA
human cognitive architecture (e.g., McClelland and Rumelhart, 1989) find comfort in computational models once the right functional units have been abstracted from the neurology. While cognitive modelers might debate the merits of symbolic versus distributed processing, they do not argue about the adequacy of productions systems or neural networks as computational models of those processes. The concern here is that the seemingly direct relationships between the computational models and the cognitive theories they purport to represent are dependent on the theories themselves and not on any general understanding of the role of computer simulation in cognitive science. What happens when we consider a theory of cognition that doesnt happen to map easily or directly onto a computational architecture? What general guidance is there to decide what needs to be included in a cognitive model? Is there a theoretically neutral account of what constitutes a proper simulation? These are not idle concerns. In fact, nowhere are these concerns more present than when we look at the role of computer simulation in representing the macrocognitive theories of cognition (e.g., Cacciabue and Hollnagel, 1995). These theories are, in large part, predicated on a rejection of the view of cognition as information processing and they are typically couched in descriptive terms far above the level of detail where a micro-level process, neural or otherwise, might be implemented. Moreover, research in macrocognition is typically conducted in the wild, beyond any laboratory control that might lend itself to computational representation. Simulating and comparing performance on a problem solving task like the Towers of Hanoi is far easier than trying to capture the vagaries of expertise as it is applied in a dynamic and perhaps ill-structured real world domain. In this chapter we report on our attempt to come to grips with these issues with the development of a computational model of the recognition-primed decision (RPD), a key function postulated by theories of macrocognition. We begin by describing the intended correspondence between the computational and theoretical model. We then briefly describe some tasks to which weve applied the model and the extent to which model predictions have fit human performance data. Though these applications provide a good degree of validation for the model, we conclude by discussing some of the broader implications we see in this kind of work for understanding cognitive modeling and simulation as a scientific endeavor.

THERPDANDTHECOMPUTATIONALMODEL

ModelingtheRecognitionPrimedDecision275
Inspired by Kleins model of the recognition-primed decision (Klein 1989, 1993, 1998), we have extended the available "decision types" in the Micro Saint Sharp task network modeling environment to include an "RPD" decision type. The RPD decision type is intended to support the representation of an experiencedriven decision-making process where courses of action are the emergent "byproducts" of recognition rather than the result of deliberative analysis or the application of rule-based knowledge. In particular, the RPD decision type implements computational analogues for three prominent features of Kleins theoretical model. First, just as the recognition-primed decision model emphasizes the importance of experience over the application of fixed, normative strategies, the RPD decision type depends on the accumulation of experience to shape decision-making performance during a simulation. Second, just as recognition-primed decision-making was presented as an alternative to analogical and case-based models of decision-making, recognition in the RPD decision type draws on the entirety of experience rather than focusing on any single past episode. Third, just as the course of action is an immediate "byproduct" of situation assessment in Kleins model, the RPD decision type is, at root, a mechanism for learning associations between situations, courses of action, and outcomes. With enough experience, the model simply reacts to each new situation with whatever course of action it has come to associate with that situation without engaging in any deliberative, optimizing or rule-based reasoning. Computationally, our approach extends Hintzmans multiple-trace memory model (1984, 1986a, 1986b). The basic idea is to represent a decision makers long-term memory as a set of episodes, each of which represents the situation that prompted a decision (encoded as a cue vector), the course of action taken (from a fixed set of discrete alternatives), and an outcome measure of that action (either successful or not). Recognition occurs when a new situation (i.e., cue vector) is presented. A "similarity value" is computed between the new situation and the corresponding portion of each of the remembered episodes. This value is used to determine the proportional contribution that each and every remembered episode makes to a composite recollection of courses of action taken in the past and their outcomes. The result is a distribution of recognition strengths across the available course of action given the new situation. At this point, the model depends on a fixed selection heuristic (e.g., choosing either the course of action with the greatest recognition strength or performing a weighted random draw across all recognized courses of action weighted with respect to recognition strength,), and the selected course of action for that situation is implemented, evaluated, and stored as a new episode in long-term memory for use in the next decision.

276WARWICKANDSANTAMARIA
There are several features of the approach worth noting here. Decision-making episodes are encoded using bit-strings and similarity values are calculated by taking a dot product between a cue vector representing the current situation and that portion of each remembered bit string that encodes the cue vectors of past situations. In order to ensure a uniform structure for these cue vectors, the cues that prompt recognition are re-represented internally using discrete enumerations of the ranges of values they can assume. This approach to representing cues and calculating similarity values implicitly supports fuzzy matching between "adjacent" cue values (the closer the value, the more similar the match), but it also has a more serious implication that determining similarity is a purely syntactic process within the RPD decision type; no matter what kinds of cues the internal enumerations encode, similarity is calculated bit-by-bit without any consideration of what those bits represent. The computational mechanisms that implement the similarity-based recall, the recognition of a crisp course of action from the distribution of recognition strengths, and the accumulation of experience are invariant features of the RPD decision type, but the content of the modeled decision is not. That is, any decision modeled using the RPD decision type in Micro Saint Sharp will use these same mechanisms, but the cues, courses of action, and outcome evaluations of those actions must be defined by the modeler. The following examples illustrate this process.

APPLYINGTHECOMPUTATIONALMODELOFTHERPD

During the course of the ADA, we have applied the computational model of the RPD to a variety of different tasks. The categorization and dynamic decision making tasks we describe here are especially relevant to issues of model validation and to the science of simulation generally.

CategorizationModels

Although there is a sense in which every decision we model can be thought of as a categorization task given the discrete nature of the course of action selection, we have built models of two classic categorization tasks from experimental psychology. The first we discuss is a model of the Brunswik Faces task (Warwick & Fleetwood, 2006). The Brunswik faces are a set of stimuli that

ModelingtheRecognitionPrimedDecision277
follow the 5-4 category structure, often used to study categorization (Smith & Minda, 2000, Gluck et al., 2001). Each face stimulus is defined by four binary features: eye height (high or low), eye separation (wide or narrow), nose length (long or short), and mouth height (high or low). Combining these four features yields 16 possible faces. These 16 faces can be grouped according to their features into two categories, A and B, defined by a complex rule that is not easily verbalized. The 5-4 category structure is intended to make categorization difficult, as only combinations of features are diagnostic. Participants are trained on nine of the faces, and once they meet a performance criterion, they are tested on the full set of 16 faces, including the seven faces that they have not seen before, to test their ability to generalize the categorization rule. We presented the nine training faces (each a set of four cue values), to the model in randomized order within training block. In one condition, the model was trained on a set number of training blocks. In the other condition, the model was trained until it met criterion performance (once through with perfect performance) and then tested on all 16 faces. Building the RPD model consisted of specifying the four binary cues and the possible two courses of action, Categorize as A and Categorize as B. The RPD model learned the mapping from cues to categories on the fly, without any deeper representation of the problem space, cues, or categories. When trained on a fixed number of exemplars, the model performed perfectly on the nine training faces and near-perfectly on the seven generalization faces (the trend fit is relatively good (R2 = 0.80), but the fit to individual data points is poor, reflected in the root mean square deviation of 0.248). This was unlike human performance because the model learned better than people do. By contrast, the model that stopped training after it reached criterion performance more closely matched human performance (R2 = 0.84; RMSD = 0.168). This model was a useful early demonstration of the viability of approach because it showed emergent behavior that reflected variability characteristic of human performance on the task, with a very direct representation of cues and courses of action.

278WARWICKANDSANTAMARIA

Figure15.1 Comparingmodelperformanceundertwotrainingconditions:fixed training(top);trainingtocriterion(bottom)

The second categorization task we modeled was the Weather Prediction task (described in Lagnado et al., 2006). The Weather Prediction task is a probabilistic category learning task that has been used to study implicit learning. As in the Brunswik faces task, participants predict a binary outcome (rainy or fine weather) based on four Tarot cards that are presented in different combinations. But, unlike the Brunswik faces task, the categorical outcome is probabilistically related to the 14 different stimuli (i.e., the combinations of cards, excluding the two cases where either all the cards are present or absent).

ModelingtheRecognitionPrimedDecision279
To construct the RPD model, we specified four cues to represent each of the four cards, being either present or absent, and two courses of action, predict-rainy and predict-fine. Using the distribution of patterns shown in Table 1:, we presented the model with 200 patterns in random order. After each pattern presentation, a random draw was performed to determine the weather (given the probability associated with that pattern) and the models prediction was reinforced accordingly (i.e., depending on whether the weather was, in fact, rainy or fine given that particular draw).

Table 1: Frequencies and probabilities of 14 patterns presented to the model of the Weather Prediction task.
Pattern 1 2 3 4 5 6 7 8 9 10 11 12 13 14 1 1 1 1 1 1 1 Cues (cardspresent) 2 3 2 4 2 3 4 3 4 2 2 2 2 3 3 4 4 3 3 Total 4 4 Frequency 19 9 26 9 12 6 19 19 6 12 9 26 9 19 200 Probabilityof fineweather 0.895 0.778 0.923 0.222 0.833 0.500 0.895 0.105 0.500 0.167 0.556 0.077 0.444 0.105 0.500

The model learned the probabilistic structure underlying the task, approaching optimal performance by the third 50-trial block. In this respect, model performance was both qualitatively and quantitatively similar to human performance. When examining performance for each pattern separately, we found that the model was sensitive to differences across the 14 patterns. The ushaped function in Figure 15.2 reveals that the model picked up on the statistics of the environment, performing at or near chance on weakly predictive patterns, better on moderately predictive patterns, and well on strongly predictive patterns. The model appeared to learn conjunctive information (patterns) rather than elemental information (individual cues).

280WARWICKANDSANTAMARIA

Figure15.2 Modelperformanceonthelast50trialsoftheWeatherPredictiontask, averagedacross30runs,separatedbypattern.ErrorbarsareSEM. ModelsofDynamicDecisionMaking

It is one this to predict categorization behavior given a fixed, albeit complicated, category structure or stationary probability distribution but quite another to capture behavior in dynamic environments, where the consequences of past decisions can shape future events. As a first step toward modeling tasks with a more naturalistic flavor, we developed two models of dynamic decision making tasks. First, we developed a model of the Prisoners Dilemma. The Prisoners Dilemma is classically presented as two people who are arrested for a crime. Each is offered a deal, in isolation, to testify against the other: if one prisoner stays silent, the other who testifies goes free and the prisoner gets the maximum sentence. If both prisoners testify, they both get a reduced sentence, and if both stay silent, they are both set free. Given these constraints, the rational choice

ModelingtheRecognitionPrimedDecision281
diverges from the optimal choice. That is, if each prisoner bases their decision on minimizing their own sentence no matter what their partner does, neither prisoner goes free. The optimal outcome is realized only if both prisoners remains silent, an outcome that depends on prisoners acting, irrationally, as partners in crime. We modeled the Prisoners Dilemma using the payoff structure shown in Table 2:. The two players were represented by two interacting RPD models each making its own decision either to cooperate (i.e., remain silent) or testify (i.e., defect). The game was iterated over a large number of trials, allowing each model to learn and adapt to the behavior of the other model.

Table 2: Payoff structure for the Prisoners Dilemma.

Playeraction BCooperate BDefect ACooperate APayoff=3 BPayoff=3 APayoff=0 BPayoff=4 ADefect APayoff=4 BPayoff=0 APayoff=1 BPayoff=1

Following the approach of Lebiere, Wallach, and West (2000), we viewed learning as matter of recognizing patterns in the opponents behavior (e.g., one prisoner recognizes that the other prisoner always defects after the first cooperates). Thus, the cues for each model were a fixed number past moves for it and its opponent, where the number of remembered past moves could be varied. The two possible courses of action were cooperate and defect. Success was defined in terms of the opponents decision; the decision to cooperate was rewarded only if the opponent did not defect, while the decision to defect was rewarded only if the opponent cooperated (its a cruel world). Figure 15.3 shows the cumulative outcome for a model of fixed memory (Player A) pitted against models of varying memory capacity (Player B). The cumulative outcome measure is calculated as follows: defecting when the opponent cooperates adds 1 point to a players score, cooperating when the opponent defects subtracts 1 point from a players score, and mutual cooperation or defection results in no change to the score. We captured two of the main qualitative effects reported by West, Lebiere and Bothell (2006). First, the average difference in score between opponents of equal or near-equal memory capacity is near zero, the result of averaging across many individual random walks, where a random walk is a series of successive, randomly determined steps. Second, having relatively more memory confers a systematic advantage.

282WARWICKANDSANTAMARIA

Figure15.3 InthePrisonersDilemmamodel,performanceispoorerwhenmemory fortheopponentsmovehistorygoesfrom3previousmovesto2to1.

The second task we modeled was Dynamic Stocks and Flows (DSF) task. The DSF task was developed by Gonzalez & Dutt (2007) to investigate how people perform in dynamically complex environments. It consists of a single stock whose rate of accumulation changes over time (the function that defines the rate of accumulation is an independent condition in DSF experiments). Inflows increase the stock level, and outflows decrease the stock level. Participants try to keep the stock within a certain range. The stock level is changed both by external changes (the rate of accumulation) and by user-controlled changes. Controlling the DSF is very difficult for most people: learning tends to be slow and sub-optimal even when participants perform a simplified version of the task that only varies inflow and not outflow. Unlike the models we described previously, where the relation between task and model was straightforward, we developed several very different representations of the DSF before we found one that could exhibit anything like human performance on the task. We began with what seemed to be the most natural way to represent the task by defining cues and courses of action in the absolute terms (i.e., as numeric representations of current inflow and stock). Despite

ModelingtheRecognitionPrimedDecision283
several variations, none of the models that used numeric representations produced human-like control of the stock. Ultimately, we used the degree of difference from desired stock as a single cue to the model that would decided among a small number of proportional adjustments to the current stock. Decisions were reinforced whenever the total stock was moved closer to the target stock (a more detailed discussion of the development process and the final model can be found in Lebiere, Gonzalez and Warwick (in press)). The model learned more slowly than humans and made more errors, but overall, the model exhibited plausibly human-like performance (see Figure 15.4). It also did a much better job controlling the stock in the decreasing condition than in the increasing condition. That result was particularly satisfying in that we modeled the decreasing inflow as a transfer condition; without making any representational or parameter changes, we applied the same model that had produced the first reasonable fit to the increasing inflow to the decreasing inflow and the right behavior simply emerged.

DISCUSSION

Looking across the results of the four models discussed previously (as well as other results not discussed here, e.g., Warwick and Hutchins, (2004); Warwick (2006)), we gain confidence in the basic computational mechanisms that underlie the models. Together the multiple-trace memory and similarity-based recall have produced reasonable fits, both qualitative and quantitative, to a variety of different tasks. Moreover, the effort needed to develop those models and, subsequently, to account for their behavior has been relatively modest. Practical, predictive and perspicuous are all welcome qualities in a decision making architecture. At the same time, we recognize the difficulties in staking claims to model validation. What we have presented here is only a small snapshot of all of the possible decision making domains to which our decision making architecture might be applied and it is hard to predict how the model might fare. Even in the examples we presented, there is room for improvement in the data fits. Finally, some might argue that the domains weve worked in are more conducive to preserving laboratory control than exploring human behavior in naturalistic settings.

284WARWICKANDSANTAMARIA

Figure15.4 Human(above)andmodel(below)performanceontheDynamicStocks andFlowstaskundertwodifferentinflowfunctions.

The foregoing concerns are all serious, but they are not peculiar to our work. Model validation is an essentially inductive endeavor and any attempt will be fraught with concerns about the scope, precision and generality of the results from which we infer model validity. Moreover, there is no new insight to be gained by observing that model validation is difficult. There is much deeper issue lurking here with implications for cognitive modeling as an empirical endeavor.

ModelingtheRecognitionPrimedDecision285
As we remarked at the outset, the received view of cognitive modeling holds that computer simulations are experiments in which software programs serve as models of theories. In this respect, cognitive modeling, and computer simulation generally, is seen to be continuous with experiments in the physical sciences. There is, however, an important difference between physical experiments and computer simulations, namely, the computer introduces a layer of abstraction between the phenomenon being modeled and the system doing the modeling. Of course, not every experiment in the physical sciences is conducted directly on the phenomenon of interest; some experiments are conducted using systems that are sufficiently like the phenomenon (e.g., a scale model of an airplane being tested in a wind tunnel) to justify generalization to the actual phenomenon. But establishing that the modeled system is sufficiently analogous to support generalization to the actual phenomenon is itself an empirical endeavor. That is to say, there is a theory of the experiment that accompanies such analogue simulation; there is a principled account of how the analogue system represents all and only the relevant features of the phenomenon and an explanation of why the dissimilar features of the analogue system can be ignored. What theory of the experiment is there for cognitive modeling? What is it that makes some representations suitable for simulating cognition and others not? This is not a question about the putative content of the representations (e.g., symbolic versus distributed representation) but about the representational strategies themselves. Our own experiences bring these issues into focus. For example, the two categorization models discussed previously demonstrate that identical internal representations can be used to predict behavior for two tasks that would otherwise seem to tease out important difference in cognition. The very same mode that can predict trained performance in deterministic categorization can also mirror the continuous learning that occurs in a probabilistic environment. Conversely, we find that one model of decision making in a dynamic environment crucially depends on the representing the recent history in memory while the other model can control a dynamic system using a purely reactive representation of the task. Though it is important to point out that the model architecture does impose some constraints on the modelernot everything goes when it comes to representing a decision making taskwe still lack a principled account to explain why a common representational strategy is appropriate in one case and not another. Absent such an account, we advocate a comparative approach to understanding representational strategieseither comparing how a given architecture has been applied to different task (as we have done here) or how different architectures have been applied to a common task. Working under the ADA we have collaborated with teams at Penn State University and Carnegie Mellon University on two different model comparison efforts. The work with Penn State

286WARWICKANDSANTAMARIA
turned on a simulation of the Three Block Challenge, a human-in-the-loop simulation where small, distributed command and control teams must decide how to allocate resources as they deal with insurgents in an urban stabilization operation (see Santamaria and Warwick, 2009, for details). The work with Carnegie Mellon turned on a simulation of the Dynamic Stocks and Flows, as discussed previously. While in both cases we identified interesting commonalities in the respective architectures and our own approach (e.g., the implementation on fuzzy matching in both ours and the R-CAST agent-based approach taken at Penn State; the use of instance-based models memory in ACT-R), the real benefit in the comparisons came in learning about the model comparison process itself. And most important among these insights was that a model comparison should not be a model competition in which winners and losers are decided solely on the basis of a quantitative measure of fit. In fact, the focus on quantitative fit often gets in the way of understanding how a task was modeled and whether the approach is reasonablea good fit does not excuse the dulling Occams razor and a bad fit should not be used to dismiss, out of hand, what could be a useful approach in other contexts. This comparative approach marks a complete reversal from the current methodology in cognitive modeling. Rather than view the computer as a virtual sand box in which we can advance our theories, we see the question of how any cognitive theory is represented computationally as an issue that must understood before experiments can be properly conducted. In the case of analogue modeling in physics, we are forced to confront and reconcile the manifest disconnects between the phenomenon and the simulated system. We can span this disconnect only with a theory of the experiment to justify the abstractions we employ. In cognitive modeling, the disconnect is overlooked because the relationship between theory and model is assumed to be so direct. But our work modeling the RPD reminds us how much work is needed to support this assumption; not only must we overcome the macrocognitive researchers general presumption that cognition cannot be understood as merely information processing but we must also explain away our own experiences in which our specific representational approaches have proven too flexible in one instance and not flexible enough in another. Cognitive modeling is and should be an empirical endeavor. The comparative approach we advocate is not just a practical solution for understanding how different architectures apply to a common task, but a theoretical prescription for identifying invariant representational structures among different architectures and understanding how they operate within simulation. It is this understanding that will undergird our theory of the experiment in cognitive modeling and will, finally, allow us realize the empirical promise of computer simulation.

ModelingtheRecognitionPrimedDecision287 REFERENCES
Cacciabue,P.C.andHollnagel,E.(1995).Simulationofcognition:Applications.InJ. M.Hoc,P.C.CacciabueandE.Hollnagel(Eds.),Expertiseandtechnology:Cognitionand humancomputercooperation(pp.5573).Hillsdale,NJ:LawrenceErlbaumAssociates. Fan,X.,Sun,B.,Sun,S.,McNeese,M.,andYen,J.(2006).RPDenabledagents teamingwithhumansformulticontextdecisionmaking.InAAMAS.Hakodate,Hokkaido, Japan. Gluck,K.A.,Staszewski,J.J.,Richman,H.Simon,H.A.,andDelahanty,P.(2001). Therighttoolforthejob:Informationprocessinganalysisincategorization.Proceedings forthe23rdAnnualMeetingoftheCognitiveScienceSociety,Edinburgh,Scotland. Gonzalez,C.,&Dutt,V.(2007).Learningtocontroladynamictask:Asystem dynamicscognitivemodeloftheslopeeffect.Paperpresentedatthe8thInternational ConferenceonCognitiveModeling,AnnArbor,MI. Hintzman,D.L.(1984).MINERVA2:Asimulationmodelofhumanmemory. BehaviorResearchMethods,Instruments&Computers,16,96101. Hintzman,D.L.(1986a).JudgmentsofFrequencyandRecognitionMemoryina MultipleTraceMemoryModel.Eugene,OR:InstituteofCognitiveandDecisionSciences. Hintzman,D.L.(1986b)."SchemaAbstraction"inaMultipleTraceMemoryModel. PsychologicalReview,93,411428. Klein,G.(1998).SourcesofPower:HowPeopleMakeDecisions.Cambridge,MA:The MITPress. Lagnado,D.A.,Newell,B.R.,Kahan,S.,&Shanks,D.R.(2006).Insightandstrategy inmultiplecuelearning.JournalofExperimentalPsychology:General,135,162183. Lebiere,C.,Gonzalez,C.,andWarwick,W.(InPress).Emergentcomplexityina dynamiccontroltask:Aqualitativemodelcomparison. Lebiere,C.,Gonzalez,C.,&Warwick,W.(2009).Acomparativeapproachto understandinggeneralintelligence:Predictingcognitiveperformanceinanopenended dynamictask.InProceedingsoftheSecondConferenceonArtificialGeneralIntelligence. AtlantisPress,AmestrdamParis,pp.103107. Lebiere,C.,Wallach,D.,&West,R.L.(2000).Amemorybasedaccountofthe prisonersdilemmaandother2x2games.InProceedingsofthe3rdInternational ConferenceonCognitiveModeling,185193. McClelland,J.L.,&Rumelhart,D.E.(1989).ExplorationsinParallelDistributed Processing:AHandbookofModels,Programs,andExercises.London:TheMITPress. Newell,A.(1990).UnifiedTheoriesofCognition.Cambridge,MA:HarvardUniversity Press. Newell,A.andSimon,H.A.,(1976).ComputerScienceasEmpiricalInquiry. CommunicationsoftheACM,19,113126. Santamaria,A.&Warwick,W.(2007).Anaturalisticapproachtoadversarial behavior:Modelingtheprisonersdilemma.InProceedingsofthe16thConferenceon BehavioralRepresentationsinModelingandSimulation. Santamaria,A.,Warwick,W.(2009).UsingaNaturalisticMechanismtoCapture CognitionandDynamicBehaviorsinTaskNetworkModels:AnOverview.Proceedingsfor theEighteenthConferenceonBehaviorRepresentationandSimulation.Sundance,UT. SISO.

288WARWICKANDSANTAMARIA
Santamaria,A.&Warwick,W.(2008).Modelingprobabilisticcategorylearningina tasknetworkmodel.InProceedingsofthe17thConferenceonBehavioral RepresentationsinModelingandSimulation. Smith,J.D.andMinda,J.P.(2000).Thirtycategorizationresultsinsearchofa model.JournalofExperimentPsychology:Learning,Memory,andCognition,26,327. Warwick,W.&Fleetwood,M.(2006),AbadHempelday:Thedecouplingof explanationandpredictionincomputationalcognitivemodeling.InProceedingsofthe Fall2006SimulationInteroperabilityWorkshop.Orlando,FL.SISO. Warwick,W.&Hutchins,S.(2004).Initialcomparisonsbetweena"naturalistic" modelofdecisionmakingandhumanperformancedata.InProceedingsofthe13th ConferenceonBehaviorRepresentationinModelingandSimulation. Warwick,W.,McIlwaine,S.,Hutton,R.J.B.,&McDermott,P.(2001).Developing computationalmodelsofrecognitionprimeddecisionmaking.InProceedingsofthe10th ConferenceonComputerGeneratedForces. Warwick,W.(2006).Givingupvindicationinfavorofapplication:Developing cognitivelyinspiredwidgetsforhumanperformancemodelingtools.Proceedingsofthe 7thInternationalConferenceonCognitiveModeling.

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T A N D L A U R E L A L L E N D E R

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T AND LAUREL ALLENDER

This book, produced by the Partners of the Army Research Laboratory Advanced Decision Architectures Collaborative Technology Alliance, also serves as the Final Report for DAAD19-01-2-0009.

SECTIONIII.ACTINGONBATTLEFIELDINFORMATION

SECTIONIII ACTINGONBATTLEFIELDINFORMATION

256SECTIONIII

Chapter

16

16. FIVELESSONSLEARNEDINHUMANROBOTINTERACTION

PATRICIAL.MCDERMOTT AlionScienceandTechnology,Boulder,CO DOUGLASGILLAN,PH.D. NorthCarolinaStateUniversity,Raleigh,NC JENNIFERRILEY,PH.D SATechnologies,Marietta,GA LAURELALLENDER,PH.D. ArmyResearchLaboratory,AberdeenProvingGround,MD

INTRODUCTION

The growing sophistication of robotics systems has enhanced the utility of unmanned systems for complex military operations (e.g., ordnance disposal or minefield remediation, Lemhofer, 1999; aerial reconnaissance and patrol, Chappell, 2007). The increasing capability and functionality of current and future robotics systems design requires comparable enhancements to robotics interfaces and displays developed for controlling unmanned systems. Many military operations involving robotic assets still require some level of teamwork,

289

290MCDERMOTTETAL.
so the interfaces must not only support single user control, but also promote team processes. Furthermore, displays and interactions mechanisms must be designed to facilitate quick and smooth transitions between multiple users collaborating to control multiple robots. This may occur under varying levels of autonomy and in a variety of tasks. Our goal for this research was to improve the human-robot interaction (HRI) through study of factors that impact aspects of Soldier performance in robotics operations. The research team identified key issues and challenges in monitoring and controlling robots with respect to situation awareness (SA), perception and performance, and workload. SA Technologies, Alion, New Mexico State University, and North Carolina State University have conducted numerous studies to understand a wide variety of factors that will affect Soldiers interactions with robots. These include the impact of bandwidth reduction, span of control, unmanned vehicle characteristics, level of automation, and interface designs (including control and feedback from the robot) as well as collaborative technologies and teamwork configurations. The research has included both theoretical and empirical investigations in a variety of contexts (e.g., search and rescue, homeland security operations, and military reconnaissance) and experimental mediums (e.g., real-world miniature environments, virtual environments, computer games, and field studies). Our HRI research has investigated multiple factors that can impact SA (and performance) including: mission or task factors, environmental factors, human and social factors, system design factors, and aspects of interface design. This paper summarizes five key lessons learned from this multiple-year research program within the Advanced Decision Architectures Collaborative Technology Alliance:

1. System and interface design must support the multiple facets of SA. 2. Operating a robot inherently requires multitasking. One solution to the human performance costs of multitasking is to automate certain tasks, but the effects of automation on performance may be limited by human and social factors. 3. Soldier-robot interfaces need to support effective teamwork and the communication of spatial information. 4. Soldiers need interface interventions and compensatory methods to ameliorate the effects of remote presence (e.g., reduction on natural ability to perceive, move, and manipulate objects in an environment) that can hinder navigation and limit Soldier ability to project his intent upon the environment. 5. Performance measures must be examined in tandem with SA measures to get better assess the effects of HRI factors on overall system performance.

HRILessonsLearned291

292MCDERMOTTETAL. SITUATIONAWARENESSDESIGNNEEDS Lesson1:Factorsassociatedwithhumancontrolofroboticsassetscan differentiallyimpactSA.Itiscriticaltodeterminetheinformationneeds ofSoldierswithinacontextandtosupportthemultiplefacetsofSA throughsystemandinterfacedesign

An important issue in the control of remotely operated vehicles is the limitation in development of SA due to impoverished sensory information, attention resource limits, task and environmental stressors, and system design faults (Endsley and Riley, 2004). In addition, where automated functionality is implemented to compensate for performance issues and to provide for lower operator-to-vehicle ratios, problems associated with passive control of robots can contribute to SA challenges. Several issues in acquiring and maintaining SA have been revealed. Environmental factors and perceptual demands associated with military operations with robots can be stressful and limit a Soldiers capability for perceptual awareness. For example, military operations in robot-assisted search and rescue or surveillance may involve complex environments with clutter, debris, darkness/limited visibility, and extremely unfamiliar layouts. This makes detection of targets and tracking of objects a difficult task (Riley, Murphy, & Endsley, 2006; Carlson & Murphy, 2004). Scenes can be hard to interpret due to soda-straw views and grainy images. Misleading viewpoints and perspectives also can lead to lack of object identification, misjudged distances and object size, and poor depth perception. Research shows that operators spend a great deal of time attempting to develop SA and disambiguate what they are viewing with robot cameras. Operators also struggle with comprehending where the robot is (spatial awareness) with respect to other landmarks or operational elements (e.g., operator location, troop locations, objectives, etc.). These localization issues related to understanding the direction of motion of the robot are not in isolation; they are in context of the mission requirements. Disorientation is a negative result of problems with vehicle localization. It is difficult for operators to track control actions for determining where the robot has been. This is critical in nonline of sight operators, common to Soldiers, where operators must maintain a mental model of where the robot has visited, what its encountered, and where it needs to go next. Similar effects can occur for latency and robot control lag.

HRILessonsLearned293
Degraded SA affects the perceived reliability of data gathered from robotic systems (e.g., with respect to target location, target tracking, etc.) and the ability to make operational projections (e.g., go/no-go areas, safety on route, etc.). These kinds of problems may also lead to less timely decision making and diminished tactical advantage expected from robotic systems. A poor understanding of how a robot is situated in the environment can affect performance with robots. This is an integration of location, configuration, and orientation of the robot. The important element is how the robot is situated in order to comprehend effects on mobility (related to mission and task awareness). Poor interface designs are an issue with current systems. Many interfaces provide too much low level data and too little integrated information that support at-a-glance awareness of things like robot situated-ness and status. They do not present the critical information requirements to the user to support meeting mission objectives. This causes delays in projection of robot state because of limited attention and working memory. In summary, the following are SA-related design needs for HRI:

1. Support for visually demanding tasks. Examples include object recognition support, strategies or interface support for depth perception and judging size and distance, and support for parallel processing and divided attention. 2. Support for understanding robot localization and situatedness. Examples include improved GPS for robots, mapping tools at the interface, interface designs that present data on orientation and robot configuration, and mission recording and playback for quick SA update). 3. Improved interface design for developing higher level SA. For example, conduct SA-oriented design of interfaces to facilitate integration of data at the interface and design for understanding of robot automation states and modes). 4. Systematic assessment of information needs and SA skills within a context. This can be done through goal-directed (cognitive) task analysis and behavioral and communications analysis.
MULTITASKINGANDTHEROLEOFAUTOMATION Lesson2:Operatingarobotinherentlyinvolvesmultitaskingandtask switching.Whendesigningautomationtoaidmultitasking,designers

294MCDERMOTTETAL. mustconsiderhowsocialandhumanfactorsinHRIcaninteracttonega tivelycounterthebenefitsofautomatedfunctions.

The task of an operator controlling a single robot actually involves many tasks, including navigating the robot (which itself may require the operator to watch one or more video screens, check paper or electronic maps, and coordinate with Soldiers in the field), monitoring displays to assess the robots status, and either performing the task that the robot is in place to do (e.g., search and rescue, ordinance disposal, etc.) or coordinating with team members who are expert in that task. Robot operators may also be required to control multiple robots or to coordinate with operators of other robotic assets, which may often involve integrating of information from different types of assets, such as Unmanned Ground Vehicles (UGVs) and Unmanned Aerial Vehicles (UAVs). When faced with multiple tasks, a single operator may try to engage in simultaneous performance of the tasks, but is more likely to switch among the tasks (Chadwick, Gillan, Simon, & Pazuchanics, 2004). Switching between tasks has substantial costs which can include loss of SA, loss of spatial awareness, and the need to retrieve task relevant information from memory (Chadwick, et al, 2004), and the costs can be a function of the specific tasks (Goodrich, Quigley, & Cosenzo, 2005). However, if certain tasks have natural break points or a period of down time, appropriate task switching can produce better performance than if the operator were to stay on one task for extended periods (Chadwick, et al, 2004). In addition, controlling two robots can be useful in conditions in which the second robot is positioned to provide visual information essential to completion of the first robots task, such as a UAV providing an aerial view for a UGV-oriented task (Chadwick, Pazuchanics, & Gillan, 2006). One high workload task that arises from the need to coordinate between UAVs and UGVs is integrating and interpreting visual information from the different sources (see Chadwick & Gillan, 2006). In a series of experiments, participants examined a ground-view image with a specified object as a target, then had to identify that target in an image of the same environment shown from an aerial perspective. In one experiment, the views were both photographs of real scenes (Figure 16.1), and the participants provided think aloud protocols as they attempted to integrate the two types of views. They described using strategies that included geon-based matching of multiple objects. When they could not match multiple objects in the two views, participants engaged in more opportunistic matching of features of the ground and aerial views including color, texture, markings, and shadows. Participants also used higher level

HRILessonsLearned295
cognitive strategies like imaging the aerial view based on the ground view, reasoning (e.g., the target is the guard shack which should be near the entrance), and hypothesis testing (e.g., if they matched a red object in the two scenes, and a tree was near the red object in the ground view, they would check for the tree in the aerial view).

Figure16.1 Reconcilingimageryfromgroundandaerialperspectives

In the second experiment, participants compared computer-generated scenes that were developed to examine view integration with greater experimental control over the scenes. The uniqueness of both shape and color cues in the scene had a huge impact on performance. For example, with a unique color and shape, participants could find the target shown in a ground view in less than two seconds on average in the aerial view, but if the objects had the same shape and color, the target took more than 10 times longer to identify. One important approach to dealing with the workload induced by the need to multitask for robot operators is the automation of certain tasks. Implementation of automation in robotics systems is attractive because of the expected benefits to overall system and operator performance. Automation is projected to facilitate operational configurations that include multiple users controlling multiple robotic systems, and allow operators to better multi-task in complex operations that involve multiple competing goals. While some research has provided results to support these hypotheses, our studies have revealed a need to examine critically the design of automation (i.e., what to automation and at what degree within a context).

296MCDERMOTTETAL.
An experiment assessed the effects of automation and supplemental robotics assets on team performance and SA in robotics tasks. Participants worked as a two-person team to complete a hazardous material surveillance and monitoring task with two personal manually controlled robots or with three robots (one personal robot per teammate and one semi-autonomous system that was shared, see Figure 16.2. Researchers hypothesized that teams with an additional semiautomated robot capable of navigating autonomously in the environment would both attempt and successfully complete more robotics tasks. Results indicated however, that the human-human coordination required to share control of the remote asset (in addition to driving another system and meeting mission objectives) limited the benefits of automation. Teams that did not have to share a robot attempted more tasks and completed more tasks correctly. In addition, teams that did share a robot reported higher perceptions of workload. The benefit expected by increasing the number of robots and adding automation may have been negatively countered by having to coordinate what tasks to allocate among robots during task performance (Riley et al., 2008).

Figure16.2 Miniroboticsenvironment

In summary, when implementing automation in robotic tasks, designers should keep in mind the following:

1. At natural break points in tasks or during down time, the system could prompt participants to switch tasks or robots. This would facilitate appropriate task switching strategies that produce better performance without sacrificing SA.

HRILessonsLearned297

2. Provide critical cues to help operators reconcile imagery from different perspectives such as aerial and ground or from two different ground perspectives. Shape and color were the most salient cues followed by texture, markings, and shadows. 3. Evaluate the coordination costs of two operators sharing one automated asset. The costs may outweigh the benefits, especially in an ad-hoc team.
TEAMFACTORSANDPROCESSES Lesson3:WhenroboticoperatorsaredistributedfromSoldierswho needtheinformationfromroboticassetstocompleteamission,the interfacesandcollaborativetechnologiesmustsupportthecommunica tionofhighlyspatialinformationinrelationtotheSoldierinthefield.

It is important that a robotics operator is provided with collaborative tools well suited to both his task and environment (Bolstad, 2005). For UV-provided information to be helpful, it must be shared in a manner that increases a units SA at all levels (Endsley, 1995), without hindering the mission. Often, the operator monitoring the video is separate from the Soldier in the field who needs that information to conduct a mission. In order to effectively relay the relevant information to the Soldier in the field, several issues arise. First, should the robotics controller be co-located with the unit in the field or is sufficient for them to relay information from a rear-ward command post? Second, what information needs to be conveyeddoes the Solider need the video feed itself or is it better to distill the sensor information to the key facts such as the location of enemy and whether a street is clear of enemy and civilians? Third, since the information is highly spatial in nature, what is the best communication mode spoken over the radio, presented visually on a map, or some combination of the two? In order to gain insight on these issues, game-based experiments were conducted with two-person teams. The teams completed rescue missions in which one team member navigated to a downed-helicopter pilot as quickly as possible without being detected by the enemy and another team member used an unmanned vehicle to scout out potential routes and enemy locations. One experiment investigated the impact of different configurations of human-UV teams and the utility of supporting communication technologies. We found that co-location of team members was beneficial to the speed and success of the team (McDermott, Luck, Allender & Fisher, 2005). A second experiment investigated which

298MCDERMOTTETAL.
aspects of being co-located aided teams in using UV-provided information. The principal finding was that mission times were faster with face-to-face communication, but only when operators did not have the ability to transmit static images such as the one in Figure 16.3. However, the transmission of images was only helpful when the sender provided clear information about where the picture was taken in relation to the Soldiers position (Luck, McDermott, Allender & Fisher, 2006). The experimental results also showed that eliminating one characteristic of colocation will not substantially affect the results, but eliminating multiple characteristics may (e.g., eliminating both transmission of images and face-toface communication). The subjective ratings backed this finding; post experiment reports show very small changes when one characteristic of colocation was removed and larger changes when multiple characteristics were removed. The fact that no conditions had a significant effect on detections by the enemy indicates that even without any of the advantages of co-location or the aiding technologies tested, teams can still perform the mission safely by

Figure16.3 Enemysoldiersdepictedinimagefromunmannedvehicle slowingdown.Lastly,themapanalysisindicatedthatamapdisplaying thelocationoftheSoldiersandUVinrealtimeisindeedhelpful,butonly ifdisplayedconstantly(Allender,McDermott,Luck,&Fisher,2006;Luck, McDermott,Allender,&Fisher,2006).

HRILessonsLearned299
A third experiment compared mode of communication between the two team members: radio only, visual only (by drawing on a shared map as in Figure 16.4), or both. Surprisingly, performance was best in the visual only condition, even surpassing performance in the visual plus radio condition. In both of those conditions, participants could draw or place icons on a real-time map that showed the location of the Soldier and the unmanned vehicle. The routes and enemy locations drawn on the map included an abundance of implicit information such as how to navigate relative to landmarks, which compass direction to move, exactly where to stop, what side to pass obstacles, and where the enemy is in relation to the route (McDermott, Fisher, Allender, 2008). Participants may have done better without the radio because it forced them to take extra care in the drawings because it was difficult to recover from a mistake or alert the participant to a new threat when participants could only communicate by drawing.

Figure16.4 Drawnroutethrougharurallandscape

In summary, the following solutions can be implemented to aid collaboration among humans with robotic tasks:

1. When robotics team members cannot be co-located, it is advantageous to incorporate aspects of co-location such as the ability to share imagery.

300MCDERMOTTETAL.

2. When sharing imagery, be sure to include enough reference information so that the recipient understands where the picture was taken in reference to his or her location. 3. Allow both team members to have full-time access to a map of the area without having to click between displays. 4. Allowing team members to communicate visually on a shared map can enhance the communication of highly spatial information.
PERCEPTION,NAVIGATION,ANDOPERATINGROBOTS Lesson4:Removingthehumanfromtheenvironmenttobenavigated disruptstheperceptionofvisualinformationabouttheenvironmentand themotion.TheHRIinterfacecanbedesignedtoreducethisdisruption andenhancetheoperatorsabilitytonavigatearobotthroughitsdistant environmentandtoperformthedesiredtasks.

Among the most natural and easy-to-do activities for mobile, adult humans is moving through space and extracting information from the environment. However, that ease and naturalness belies the complexity of the perceptual, cognitive, and motor processes that underlie visually-guided motion and information extraction. That complexity becomes clear when an operator at a distant location has to control or monitor a robot and identify targets or perform tasks using the robot. In this section, we review several experiments that examined techniques to ameliorate some of the problems faced by robot system operators. (See the original papers for detailed parametric information concerning the specific manipulations in those studies.) The Soda Straw Effect. One of the problems for the operator of a remote UGV is the narrow field of view (FOV) provided by a robots cameras, referred to as the soda straw effect (e.g. Riley, Murphy, and Endsley, 2006). A restricted FOV eliminates valuable visual information about the space surrounding a UGV, thereby making navigation and target identification difficult. Pazuchanics (2006) investigated the effect of supplying contextual information through a third-person, tethered, camera perspective that provided a view forward from behind the robots shoulder and included part of the robot as context. She asked whether this third person view could reduce the problems associated with a limited FOV. A virtual robot teleoperation task was created using a robotic vehicle simulation. Participants piloted the virtual robot through a series of obstacle courses under four different display conditions, based on the FOV of the main camera on the robot (narrow = a 30 degree FOV; wide = a 60

HRILessonsLearned301
degree FOV) and the presence or absence of a tethered camera: narrow FOV with no tethered camera, narrow FOV with tethered camera, wide FOV with no tethered camera, and wide FOV with tethered camera. Operator performance measures (mission time, number of objects destroyed due to accidental collision, and number of turn-arounds) and subjective ratings of piloting comfort were recoded. Both the wider FOV and the tethered camera aided navigation. Of the two methods, widening the FOV produced the greatest performance benefit. Thus, with a narrow FOV, capturing a third-person perspective via a tethered camera may facilitate certain aspects of navigation. Because the benefits associated of each condition were additive, the ideal video displays may incorporate both wide FOV and a third person view (see also Voshell & Woods, 2006). Bandwidth Trade-Offs. The distant relation between an operator and a robot also leads to problems when the communication channel between the robot and operator has limited bandwidth. If the operators task requires high-resolution full-motion color video, the operator may need to control the robot without all of the visual information. McDermott, Fisher, Hill (2009) studied participants who navigated a remote-control vehicle along specific routes while attempting to identify target objects. The operators received visual scene information from cameras mounted on the vehiclein full color to one group and in gray scale to a second group. The participants could change the spatial resolution and temporal resolution (i.e., frame rate) among five different levels, with resolution and frame rate negatively correlated. The resolution varied from 64 x 48 to 640 x 480 pixels (for the front camera) and the frame rate varied from 10 to 26 fps (frames per second). In general, color led to better navigation and target identification performance than did gray scale. Participants tended to select higher spatial resolution and lower frame rates with 58% of trials in the highest spatial resolution/lowest frame rate (HSR/LFR) setting and 31% in the moderately high spatial resolution/moderately low frame rate (MHSR/MLFR). The participants that selected the HSR/LFR did not differ from those who selected the MHSR/MLFR in the time to navigate the course, but were faster to identify the targets as friend or foe. In a second experiment (summarized in Chadwick & Pazuchanics, 2007), four groups searched for a target (the navigation was done automatically) with variations in the spatial resolution (high = 720 x 420 pixels, low = 48 x 32) and temporal resolution (high = 30 fps, low = 1 fps). The groups received high spatial/high temporal resolution (HS/HT), high spatial/low temporal resolution (HS/LT), low spatial/high temporal resolution (LS/HT), or low spatial/low temporal resolution (LS/LT). After viewing the video, participants identified the placement of the target and of six additional landmarks on a map of the environment through which the robot had navigated. For both target identification and landmark placement, high resolution of either type was

302MCDERMOTTETAL.
sufficient to produce good performance. Performance deteriorated only when both the spatial and temporal resolution was low. These studies, taken together, suggest that operators are likely to prefer high spatial resolution, but in conditions in which that cannot be provided, high temporal resolution may also be helpful. Guidance for the design of the operators interface for HRI systems. The studies reviewed above indicate that the perceptual problems caused by the humanrobot interface can also be reduced by careful design of the interface. In summary, the following solutions were found to aid in perception and navigation of robots in distant environments:

1. In order to compensate for the soda straw effect of remote viewing, the field of view (FOV) should be widened if possible. Providing a third-person perspective also benefits navigation and the benefits are additive. 2. Temporal resolution may be sacrificed if the resolution is high and the robot is traveling at relatively low speeds. A high spatial resolution aids in target detection and identification.
HRIMEASURES Lesson5:PerformancemeasuresandSAmeasuresshouldbeexamined togethertogetthewholestoryonoperationaleffectsinHRI.

There were several experiments in which the performance data and the SA data told slightly different stories. An example deals with the study of latency under different levels of automation. The study involved navigating a robot (Figure 16.5) through preset indoor courses with specific reconnaissance points. The levels of automation included teleoperation, guided teleoperation with automatic correction to avoid hitting objects, line-of-sight automation in which the user clicked on a position and the robot attempted the shortest route to that position, and full automation in which the robot used a map of the space to avoid obstacles and plan the best route. In terms of control performance, latency resulted in longer mission times. However, latency did not degrade SA. The navigation and SA marking mistakes did not increase between the latency and the no latency conditions, and in many cases actually decreased. This is most likely due to subjects slowing down due to increased control difficulty or switching to a move-and-wait control strategy, both of which allow more time for SA.

HRILessonsLearned303
Looking at the performance measures and the SA measures provides a richer understanding of the impact of latency. This holistic or cognitive-systems engineering approach to measurement provides insight beyond just the simple performance metrics. The application of these data depends crucially on this fuller examination of the various measures. If task completion time is critical (e.g., in a search and rescue mission in which delays may cost lives), then latency should be minimized, even if SA is not maintained at the highest possible levels. In contrast, in task environments in which the operator must provide detailed information about the situation and or predict upcoming features of the situation, latencies need not be minimized and operators should be allowed to move slowly enough to develop good SA.

Figure16.5 CourserouteandPioneer3DXRobot

To better explore important relationships among HRI response measures and examine potential tradeoffs, researchers should consider multiple measures of performance and SA for more comprehensive assessments. Measure should include several aspects of performance and SA. For example, performance can involve time to complete tasks, accuracy or number of errors, number of tasks or objectives met in a time period, decision making, monitoring

304MCDERMOTTETAL.
performance, ability to navigate the system, and interactions with the interface. SA can be measured with direct objective and subjective measures, as well as indirect objective measures, to evaluate both the outcome (knowledge acquired) and the process of acquiring awareness. Direct, objective measures, such as the Situation Awareness Global Assessment Technique (Endsley, 1995) directly assess the state of knowledge that a robot operator has acquired on both current and future operational elements by scoring operator responses to situation queries. Subjective measures survey operators regarding their perceptions of their SA and provide insight into the confidence a robotics operator has in his SA, which can impact behaviors with the system. Process measures, like team communications, scan patterns or eye tracking data, observations on planning and coordination, self-checking, or interface interactions, provide insight into how SA in acquired and support inferences on the degree of awareness an operator has achieved. HRI studies that include examination of interrelationships among these variables can enhance the body of literature by providing comprehensive results that describe operator behaviors and decision making.

IMPLICATIONSFORFUTUREARMYWORK

The lessons learned from this body of work have direct implications to the Army, and especially to the Armys Future Combat Systems. The lessons learned specify ways to help humans and robots work together more seamlessly in naturalistic environments where bandwidth is limited, the stakes are high, and robots are used to enter areas that humans can not safely enter. The implications are not only for display and interaction design, but for team configurations, collaborative technologies, and HRI metrics. One of the next logical steps in this research is turning the perceptual and cognitive requirements into system design requirements. This would push the human factors lessons learned into requirements for specific systems for system functionality, hardware, software, and associated interfaces. In addition, the authors have identified key areas of additional research. One fruitful area of future research involves examination of interrelationships in HRI. For example, what is the relation between spatial awareness and SA? In robotics tasks, one could hypothesize that increasing a Soldiers spatial awareness would have a high impact on overall situation awareness. What types of aids are most useful in enhancing spatial awareness and for what types are tasks are they most important? The relation between spatial awareness and mental map development is another related issue that would be useful to explore in future research. The research has implications for interface features to enhance spatial awareness and overall SA.

HRILessonsLearned305
A second area for future research involves HRI measures and metrics. More studies should be done on the interrelationships among human performance measures such as performance time and accuracy, SA, teamwork, and collaboration. A comprehensive assessment of these factors would support the development of HRI guidelines. It could provide guidance on how to design to promote the specific performance or behavior aspect that is of interest. It could also provide additional guidance on how the metrics should be examined in tandem to more fully understand the impact of design alternatives on performance including mission performance, situation awareness, and teamwork. Future research could help designers conduct tradeoffs to optimize as many aspects of performance as possible. The third area of future research involves automation and HRI. As robots technology advances, robots will become more automated and the ratio of operators to robots is likely to increase. A systematic study could examine automation, the number of robots being controlled, and interaction effects from the environment and the task. The study would shed light on what features should be automated, when, and to what level of automation. Other studies are needed to further understand trust in unmanned systems under automation and how systems should be designed to facilitate user acceptance and use.

REFERENCES
Allender,L.,McDermott,P.L.,Luck,J.,&Fisher,A.(2006).Teamcommunication withandwithoutaidsfortransmittingremoteinformation.Proceedingsofthe50th AnnualMeetingoftheHumanFactorsandErgonomicsSociety.HumanFactorsand ErgonomicsSociety,SantaMonica,CA. Bolstad,C.A.,&Endsley,M.R.(2005).Choosingteamcollaborationtools:Lessons learnedfromdisasterrecoveryefforts.ErgonomicsinDesigns,13(4),714. Carlson,J.&Murphy,R.R.(2004).HowUGVsPhysicallyFailintheField,IEEE TransactionsonRobotics. Chadwick,R.A.,&Gillan,D.J.(2006).Strategiesfortheinterpretiveintegrationof groundandaerialviewsinUGVoperations.ArmyScienceConference25,PosterSession. Availableatwww.sti.nasa.gove/Pubs/star0819.pdf. Chadwick,R.A.,Gillan,D.J.,Simon,D.,&Pazuchanics,S.(2004).Cognitiveanalysis methodsforcontrolofmultiplerobots:roboticson$5aday.InProceedingsofthe HumanFactorsandErgonomicsSociety48thAnnualMeeting.(pp.688692).Santa Monica,CA:HumanFactorsandErgonomicsSociety. Chadwick,R.A.&Pazuchanics,S.(2007).Spatialdisorientationinremoteground vehicleoperations.InProceedingsoftheHumanFactorsandErgonomicsSociety51st AnnualMeeting(pp.161165).SantaMonica,CA:HumanFactorsandErgonomics Society. Chadwick,R.A.,Pazuchanics,S.L.,&Gillan,D.J.(2006).Whattherobot'scamera tellstheoperator'sbrain.InN.Cooke,H.Pringle,H.Pedersen,andO.Conner(Eds.).

306MCDERMOTTETAL.
Advancesinhumanperformanceandcognitiveengineeringresearch:Humanfactorsof remotelypilotedvehicles(pp.373384).Amsterdam:Elsevier. Chappell,S.L.(2007).UAVcontrolwithpicturesandpointing.Proceedingsofthe AmericanInstituteofAeronauticsandAstronauticsConferenceandExhibit.Reston,VA: AmericanInstituteofAeronauticsandAstronautics. Endsley,M.R.(1995).Measurementofsituationawarenessindynamicsystems. HumanFactors,37,6584. Endsley,M.R.&Riley,J.M.(2004).Supportingsituationawarenessinhuman operatorUAV/UGVcollaboration.PresentationattheArmyAMRDECScienceand TechnologyWorkshop,MoffetField,AFB,CA. Goodrich,M.A.,Quigley,M.,&Cosenzo,K.(2005).Taskswitchingandmultirobot teams.InL.E.Parker,F.E.Schneider,&A.C.Schultz(Eds.),Multirobotsystems:From swarmstointelligentautomata,Vol.III.(pp.185195).Amsterdam:Springer. Lemhofer,A.J.(1999).Roboticsinbombdisposalanddemining.InProceedingsof the8thInternationalMeetingoftheAmericanNuclearSociety,Pittsburgh,PA,April 1999. Luck,J.,McDermott,P.L.,Allender,L.,&Fisher,A.(2006).Advantagesofcolocation foreffectivehumantohumancommunicationofinformationprovidedbyanunmanned vehicle.InHumanFactorsandErgonomicsSociety50thAnnualMeeting.SantaMonica, CA:HumanFactorsandErgonomicsSociety. McDermott,P.,Fisher,A,andHill,S.(2009).Thetradeoffofframerateand resolutioninarouteclearingtask:Implicationsforhumanrobotinteraction. ProceedingsoftheHumanFactorsandErgonomicsSociety52thAnnualMeeting,in press. McDermott,P.L.,Luck,J.,Allender,L.,&Fisher,A.(2005).Effectivehumanto humancommunicationofinformationprovidedbyanunmannedvehicle.InProceedings forthe2005HumanFactorsandErgonomicsSociety49thAnnualMeeting(pp.402406). SantaMonica,CA:HumanFactorsandErgonomicsSociety. Pazuchanics,S.L.(2006).Theeffectsofcameraperspectiveandfieldofviewon performanceinteleoperatednavigation.InProceedingsoftheHumanFactorsand ErgonomicsSociety50thAnnualMeeting(pp.15281532).SantaMonica,CA:Human FactorsandErgonomicsSociety. Riley,J.M.,Murphy,R.R.,&Endsley,M.R.(2006).Situationawarenessincontrolof unmannedgroundvehicles.InN.J.Cooke,H.Pringle,H.Pedersen,&O.Connor(Eds.), Advancesinhumanperformanceandcognitiveengineeringresearch,Vol7,Human factorsofremotelyoperatedvehicles,(1sted.)(pp.359371).SanDiego,CA:Elsevier. Riley,J.M.,Strater,L.D.,Sethumadhavan,A.,Davis,F.,Tharanathan,A.,&Kokini,C. (2008).PerformanceandSituationawarenesseffectsincollaborativerobotcontrolwith automation.ProceedingsoftheHumanFactorsandErgonomicsSociety52ndAnnual Meeting.SantaMonica,CA:HumanFactorsandErgonomicsSociety. Voshell,M.andWoods,D.(2005).Breakingthekeyholeinhumanrobot coordination:Methodandevaluation.ProceedingsoftheHumanFactorsand ErgonomicsSociety49thAnnualMeeting,pp.442446.SantaMonica,CA:HFES.

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T A N D L A U R E L A L L E N D E R

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T AND LAUREL ALLENDER

This book, produced by the Partners of the Army Research Laboratory Advanced Decision Architectures Collaborative Technology Alliance, also serves as the Final Report for DAAD19-01-2-0009.

SECTIONIII.ACTINGONBATTLEFIELDINFORMATION

SECTIONIII ACTINGONBATTLEFIELDINFORMATION

256SECTIONIII

Chapter

17

17. READINGINTENTANDOTHERCOGNITIVECHALLENGESININTELLIGENCE ANALYSIS

STONEYA.TRENT,PH.D. UnitedStatesMilitaryAcademy,WestPoint,NY MICHAELW.SMITH TheOhioStateUniversity,Columbus,OH DANIELJ.ZELIK TheOhioStateUniversity,Columbus,OH JUSTINB.GROSSMAN TheOhioStateUniversity,Columbus,OH DAVIDD.WOODS,PH.D. TheOhioStateUniversity,Columbus,OH

INTRODUCTION

307

308TRENTETAL.
Utilizing information for planning and decision making entails a process of analysis, which is subject to particular vulnerabilities given the demands confronting the analyst. In the context of military operations, intelligence analysis is a complex process involving many cognitive challenges. This project, through a series of studies, examined the cognitive challenges confronting intelligence analysts in their work. Over the course of the project, three important themes have emerged: First, capturing cognitive challenges of intelligence analysis requires methodological innovations. Requirements development methods which only look at normative textbook processes will fail to identify the challenges arising from complex situations, time pressures, resource limitations, and coping strategies. More naturalistic, ecologically valid methods are needed. However, the security constraints present in the intelligence domain require further innovations, such as more efficient utilization of limited access to the domain, and the use of non-classified analog scenarios. Multiple studies using complex but open source scenarios with professional analysts balanced these constraints in order to gain insight into cognitive challenges inherent in intelligence analysis. Second, assessing analytic performance is best accomplished with a structured yet adaptive model of the process. What constitutes expert analytic performance varies depending on demands, resources, the nature of the problem, and the needs of the decision makers. No simple algorithmic approach can adequately reflect the necessary trade-offs expert analysts make. Therefore, we studied how professional analysts decide what is sufficiently rigorous in context, and developed a metric for assessing the rigor of analytic processes. Third, reading the intent of others is both a core function of analysis, and a fundamental cognitive challenge in understanding complex social systems. Forecasting how a group will act in the future (including how the group might react to a Course Of Action) involves understanding the groups intentions. However, the research showed reading the intent of other groups accurately is complex and error prone. The series of studies look at the challenges of intelligence analysis with respect to the following:

The practice of analysis in real situations, with multiple teams, production demands, and environmental pressures (in other words, the Where, When, and with Whom of analysis) Reflections on the process of analysis, and interaction between analysis and supervisors/decision-makers (the How and for Whom)

CognitiveChallengesinIntelligenceAnalysis309

Analysis of other groups structures and intentions (the What and Why)
WHERE,WHEN,ANDWITHWHOM: CHALLENGESOFMILITARYINTELLIGENCEANALYSIS INSITU

This set of studies was conducted to explore the challenges confronting analysts, which can inform the design of support tools and training programs. The focus was on challenges in naturalistic contextswith time and production pressures, coordination problems, inaccurate data, etc., which are often lacking in more laboratory-based studies of cognitive work.

KnowledgeElicitation

A knowledge elicitation study was conducted to establish a foundation of knowledge of how military intelligence analysis is conducted in situ (Trent, Patterson, & Woods, 2007). It is the first part of a bootstrapping method for discovery and confirmation. Twenty-two analysts were interviewed (from DIA, NSA, and primarily the Army selected for participation based on recent experience in Iraq and/or Afghanistan). In order to capture multiple perspectives on analysis, seven supervisors or other systems/organizational development personnel (from the Army and the Pentagon), and 17 instructors or other training development personnel (from the Army) were also interviewed. In all, 46 experts participated. The researcher who conducted the interviews is an Army officer with some years of experience in military intelligence analysis, including hostile fire theaters. These methodological factors were important to the aim of understanding the process of intelligence analysis in real-world conditions. Analysis of the interviews identified four factors influencing intelligence analysis: environmental pressures (time, data overload, as well as weather, sand, and other threats to products and tools in field environments); learning (limited background education and opportunities to develop expertise); sustained attention (problems with motivation and focus upon analysis); and overestimations of analyst scope of expertise.

310TRENTETAL. ObservationalStudy#1

In order to confirm the challenges suggested in the knowledge elicitation study and reviews of the literature, and to explore examples of deficient analysis at a detailed level (of which the interviews, more amenable to broad reflections, elicited only a few), an observational study of military intelligence analysis was conducted (Trent et al., 2007). A four-day military intelligence training exercise that was conducted at the U.S. Army Intelligence Center in Ft. Huachuca provided the opportunity to observe eight analytical teams. The teams were concurrently working on a simulated complex problem representative of the type of work conducted in operational Army units. Each team (composed of 5 lieutenants and 2 experience intelligence sergeants) was assigned to support a simulated Tactical Operations Center. Observing teams increases the ecological validity of the research, and utilizes the communication between team members as a non-disruptive means of accessing the contents of the analysts cognition (Woods & Hollnagel, 2006). In the hypothetical scenario, U.S. forces were occupying a country after responding to its invasion by a neighboring regional power. Stability was being threatened by action cells of a larger insurgency movement conducting a wide range of activities. The implicit analytic tasks are: identify insurgent activity patterns; identify group structures; and identify operational methods, such as communication, logistics, and planning. Each team was given general orientation information, operations order and threat assessments, and other demographic details of varying importance, as well as a personality database of suspicious individuals. Teams were given simulated reports throughout the exercise, providing information about specific events. Instructors interacted with the teams to facilitate learning objectives and provided prompts when the teams appeared to be deviating too far from the objectives of the exercise. Teams provided briefings to the instructors at the end of each day. An observer (the same researcher with military intelligence experience who conducted the interviews in the knowledge elicitation study) rotated among the teams to collect data on team performance. The observers domain experience, knowledge of the scenario, and access to the instructors understanding of common problems, facilitated observations. Having only one observer supported consistent observational standards across the teams, but made it impossible to make concurrent observations for specific processes for all teams.

CognitiveChallengesinIntelligenceAnalysis311
Teams interpretations of the reports (presented in their regular briefs to their instructors) were compared to normative interpretations based on instructors knowledge of the scenario. Teams consideration of hypotheses or pieces of information was evaluated based on whether or not it was discussed. Instances of problems were observed for all of the following types of cognitive challenges (note that some instances fall into more than one category): mental set (inappropriate problem-solving strategies); fixation (failing to revise hypotheses in the face of disconfirming evidence); failing to use diagnostic data in hypothesis evaluation; time pressures; problems with trust between and within teams; and problems understanding and using analytic tools (i.e., collection systems).

312TRENTETAL. ObservationalStudy#2

The rich set of challenges observed in the first observational study and the difficulties of capturing them with only one observer, led to another observational study utilizing a very similar training exercise at Ft. Huachuca, but with improved observational methods (Grossman, Woods, & Patterson, 2007; Trent et al., 2007; Trent, Voshell, & Patterson, 2007; Trent, 2007). Participants were 40 military intelligence captains, divided into four teams. The exercise lasted five days (just daytime shifts, however). As with the earlier observational study, the teams were given orienting information and a personality database. Instructors provided periodic simulated intelligence reports, and intervened at predetermined times to guide teams, and to reassign leadership twice during the exercise. These events, and the daily briefings back to the instructors, served as consistently scheduled embedded probes, eliciting cognitive challenges and mindsets without disrupting the work process. The scenario involved an alternate history setting, with the task of supporting counter-insurgency operations by providing commanders with the organization and modus operandi of the insurgents and a list of high value targets that should be protected. Two main challenges designed into the exercise were:

1. A garden path problem, in which initial information suggested one particular group was responsible for the insurgent activity, but later reports provide information to discount that hypothesis in favor of another one attributing the activity to another group. 2. An emerging path problem, in which information on likely targets was made available only slowly over time.
These challenges make this a useful scenario for exploring analysis. It has been reformulated for use in abbreviated exercises, which have been conducted in a variety of contexts, including a one-hour version used to research macrocognitive processes with non-professional analysts, and a three-hour version used as a research exercise to study team coordination. Instead of a single observer monitoring multiple teams, multiple observers were used one with each team, and two in a command post. An ad-hoc wireless network was set up, and chat programs (supporting text, audio/video connection, and file transfer) were used to connect the team observers to the command post. Observers in the command post could monitor observation records real-time for

CognitiveChallengesinIntelligenceAnalysis313
quality control, cue observers to patterns emerging in other teams, respond to questions from team observers, and ask for clarification or audio/video feed from team observers. This allowed the primary investigators military intelligence background and experience conducting the prior observation study to be utilized more widely, while supporting multiple simultaneous observations. All the observers underwent an accelerated version of the exercise to become familiar with the scenario and its challenges. At the end of each day, team observers summarized their observations, and conducted a group hot wash to review observations and refine observation protocol. A process trace based on collected data produced these categories: team interpretation of report, team hypothesis for scenario, behavior related to instructor interventions, and other team activity. Further analysis, including comparisons across teams over time, resulted in three themes: workspace layout, cognitive work balance, and analytic strategy. Teams were allowed to set up the tables, dry erase boards, maps, etc. in their workspaces as they saw fit. Based on their chosen layouts, and the observed patterns of work activity, distinctions between the collaborative processes of the teams were made. The two teams that were judged by the instructors as successfully solving the scenario problems both utilized open workspaces, with collaborative activity occurring around centrally located shared artifacts (i.e., maps, diagrams). The unsuccessful teams used private workspaces, and work was done without much coordination. Related to these patterns is the balance between sensemaking work (done in private or small sub-groups) and coordination work. A commonly observed misbalance was for teams to spend too much time on sensemaking tasks, and too little on coordination, resulting in teams being unable to share observations or adequately prepare briefings. None of the teams used formal hypothesis testing methods much despite instructor suggestions; instead, they preferred to generate explanatory stories to account for evidence. The teams varied in their commitment to particular hypotheses, with some teams able to revise initial hypotheses in the garden path problem easily, and others fixated on the initial hypotheses.

314TRENTETAL. HOWANDFORWHOM: REFLECTINGONTHEANALYSISPROCESSAND ORGANIZATIONALCONTEXTS

Another study was completed with professional analysts from multiple agencies that focused on the concept of analytical rigor. The study was based on an energy/security scenario related to the importation of liquefied natural gas (LNG) in the Northeastern US in light of new terrorist risks circa 2006 (Zelik, Patterson, & Woods, 2007c). Nine professional analysts participated in a scenario walkthrough in which they critiqued the analysis processes of two junior analysts. In the study, participants assumed the role of an analyst supervisor responsible for deciding whether these analyses were ready to send forward to a decision maker. We describe this critical judgment task, and core aspect of the study, as the "Supervisor's Dilemma." The Supervisors Dilemma describes the generic situation wherein a supervisor must decide if the output product of an analyst is acceptably rigorous or if more resources must be invested in the analysis process before sending it forward. This judgment, while in principle an abstraction of a common occurrence, represents a critical decision point in the analysis cycle that, in practice, is often made tacitly. Importantly, analysts and decision makers face a parallel form of this judgment as well, as they must also judge, in effect, whether or not an analysis is sufficient for a given context. Thus, designing the study around the Supervisor's Dilemma provided a mechanism for studying how analysts and other stakeholders judge rigor, capturing the real world iterative dynamic of intelligence analysis as a sensemaking activity. Data from the study were analyzed by organizing participant feedback relative to four aspects of the study design: the participants' comparisons of briefing reports, comparisons of analysis processes, assessments of analytical rigor, and judgments with respect to the Supervisor's Dilemma. The comparison of briefing reports and comparison of process documents reflected participant responses regarding the cues they used to infer rigor and were relative to the various documents they interacted with throughout the study. Rigor assessment and Supervisor's Dilemma judgments captured how process insight influenced perceptions of rigor, given that participants made judgments both before and after gaining insight into the analysis processes of the junior analysts. Three general patterns on analytical rigor in intelligence analysis emerged from this analysis. First, the findings suggest a definition of analytical rigor that frames the concept as an emergent, multi-attribute measure of sufficiency, rather than as a measure of process adherence. Second, they suggest that similar cues are used by professional analysts to form assessments of rigor. The study found that participants made similar comparisons of the analysis briefings and that

CognitiveChallengesinIntelligenceAnalysis315
there was some agreement as to the order in which the process documents were seen as valuable in judging rigor. Third, it was found that providing insight into the analysis process influenced perceptions of rigor, as over half of the participants, after reviewing the analysis process documents, altered their assessments as to which analysis was judged as more rigorous. These findings are summarized in a multi-media TopicLandscape that organizes the results on rigor in information analysis (a podcast is also available; Zelik, 2009). These results also showed that analytical rigor is a powerful concept that can be used to better understand the difference between shallow and deep analysis, and the findings have spawned a number of ongoing research directions (Zelik, Patterson, & Woods, 2007a). First, based on the study, an attribute-based model of analytical rigor was developed, representing an empirically derived framework for assessing rigor and for evaluating the impact of software tools hypothesized to improve analytic performance. Second, the model has informed efforts to re-envision analyst briefing interactions and to design briefing tools around the criticality of communicating about the underlying rigor of the analysis process. Third, as this research suggests that supporting the judgment of analytical rigor is about revealing insight into the analysis process, it has led to explorations of the role of capturing and sharing analytic process as an approach for creating process insight and for revealing the processes behind analysis products (Zelik, Patterson, & Woods, 2007b). In sum, these findings highlight the critical inputs needed to build a processbased representation of analysis that reveals process in a way that not only allows analysts to better understand their ownand to better judge when analyses are sufficiently rigorous for a given contextbut that also allows them to meaningfully share their analysis processes with decision makers and other analysts. Innovating new and more effective forms of representing information analysisgrounded in an understanding of how professional analysts judge analytic rigoroffers direction for coping with the present-day deluge of available data, for avoiding the trap of shallow analysis, and for developing tools capable of communicating a meaningful understanding of analytical process. This research informs each of these efforts.

WHATANDWHY: CHALLENGESOFSOCIALSTRUCTURESANDINTENTIONS ResearchingChallengeCases

316TRENTETAL.
Instances of analysts over-simplifying phenomena were observed in the Army Intelligence training exercises. Collection systems were assumed to always function correctly; insurgents were assumed to communicate either electronically or face-to-face; and insurgent activities were assumed to be independent from outside social factors (Trent et al., 2007). Over-simplifications occur when a complex entity or attribute is simplified beyond the level of detail needed for adequate understanding (Feltovich, Hoffman, Woods, & Roesler, 2004). Seeing heterogeneous groups as homogenous, dynamic as static, and multi-faceted as flat are examples. Factionsas examples of social structuresare complex. They exist as groups with interests incongruent with larger, formal political structures; they are in dynamic states of tension, responding to changing landscapes of competition and social resources. Due to their complexity, they can serve as challenge cases for intelligence analysis and the exploration of over-simplifications in analysis. (Smith, Branlat, Stephens, & Woods, 2008; Smith, Patterson, Zelik, & Woods, 2007; Smith, Stephens, & Branlat, 2007) The dynamic and multi-faceted nature of social structures of interest to intelligence analysts (as exemplified by factions) evokes the difficulties of determining the intentions of groups. Determining the intentions of an adversary or other group is centrally important for intelligence analysis (as the basis for forecasting likely courses of action or reactions to interventions), but it is particularly challenging. Peoples ability to read the intentions of others in dayto-day interactions relies on the simplicity of the actions, familiarity with the person, and/or awareness of the situation, and their ability to interact with the others. The situations confronting intelligence analysts are different. A review of a set of challenge cases was conducted in order to explore the difficulties analysts face when attempting to read the intent of other groups. The corpus consists of:

Historical challenge cases where the record of (mis) reading intent has been documented: the assassination of Julius Caesar; and the Yom Kippur War of 1973 between Israel and an allied Egypt and Syria (both cases of analysts failing to understand changing threats). Contemporary socio-political challenge cases: the phenomenon of factions, as examined from ethnographic and political science perspectives; and the analysis for siting of Liquefied Nature Gas (LNG) terminals in the Northeast (discussed above in the Supervisors Dilemma study). Challenge scenarios situated in the near future, supporting exploration of new technologies and threats using variations in

CognitiveChallengesinIntelligenceAnalysis317

the future incident technique: the Angola 2011 case, in which energy investment and political unrest (on local and global levels) interact on the southwest coast of Africa; and the Kazakhstan 2012 case, involving analysis of a political coup in a former Soviet republic, and the confounding factor of cultural differences between central Asia and the West.
All of the cases were shown to a cross section of the intelligence community to assess the validity as analytic challenge cases. Subject matter experts were involved in developing and reviewing the LNG, Angola, and Kazakhstan cases. These cases were developed in a multi-disciplinary project where participants performed analysis of the situations and confronted their inherent challenges. Formal analytic tasks were performed under observation by both novice security analysts (using the Kazakhstan scenario) and experienced security analysts [the Kazakhstan scenario, and the LNG case (Zelik, Patterson, & Woods, 2007b)].

ChallengesinReadingIntent

Groups have more than one intention at a time. Actions can impact multiple goals simultaneously (supporting both, or supporting one while undermining another). The sensitivity of intentions to constraints, pressures, opportunities, etc. means that as these factors change, intentions change. This sensitivity is actuated by the group's perception of these factors. The members of groups do not all share identical perceptions of conditions, or identical knowledge, preferences, etc., meaning that different sub-sets of the group can differ in their understanding of the groups intent. These difference, and the constraints and opportunities inherent in the concrete situation, result in the actual actions executed being different than the intent or plan of the group (Jervis, 1976). Understanding the perspective of the other group is hampered by cultural differences and (especially in the case of asymmetric conflicts) organizational differences. Understanding the situation of the other group is challenged by the limits of observing at a distance.

318TRENTETAL.
Prominent findings from each case are as follows: Angola. The Angola scenario shows how the behavior multiple groups in the context of interacting pressures can produce noisy patterns, creating the risk of misinterpreting coincidental or random patterns as intentional. Caesar. Julius Caesars failure to recognize the threat that led to his assassination resulted from multiple factors, including failure to anticipate loss of support following his nomination to Dictator for life, and his expectations that opposition in the Senate would remain disjointed. Factions. The analyses of factions show how even small groups with powerful leaders and narrow agendas are affected by internal distinctions and dynamic pressures. Faction leaders must respond to member demands, competing factions, and external pressures. Liquefied Natural Gas (LNG). The challenges in analysis for LNG terminal siting also shows the risks of over-simplifying other groups and their intentions. Sides advocating and opposing a particular site are composed of different stakeholder groups, which may align themselves differently depending on various details, opportunities, and trade-offs. Yom Kippur. Israels expectations about Egypts and Syrias goals and capabilities formed the basis of its failure to adequately anticipate the surprise attack that started the Yom Kippur War of 1973. Israel did not understand Egypts vision of political victory independent of military victory. Kazakhstan. One of the main risks illustrated in the Kazakhstan scenario is inadequate consideration of parallel socio-cultural structures in a foreign society. Analysis of power in Kazakhstan that involves projection of Western social structures, and the assumption that only formal political structures (vs. traditional social structures) are relevant, would be erroneous. Particular patterns of over-simplification in analysis and attribution of intent have been identified in the social cognition and international relations literature, which correspond to the challenges seen in these cases. Intentional, Focused, and Orchestrated. People are prone to seeing intent in random or otherwise non-directed patterns. Likewise, the behavior of other groups is more likely to be seen as the result of coordinated, single-minded, and intentional action, rather than resulting from internal struggles; trade-offs, execution errors; etc. (Jervis, 1976; Stein, 1988). (Examples in Angola, Factions case) Static. Despite changes occurring in other groups, there is a tendency to see them as static. (Examples in Factions, LNG cases)

CognitiveChallengesinIntelligenceAnalysis319
Homogeneous. Groups are simplified as being more homogenous than they are. They are treated as a single unit; relevant internal distinctions are neglected. (Examples in Factions, LNG cases) Expectations. An observers expectations for another actors or organizations behavior (and the underlying beliefs and goals concerning that actor or organization) impact the perception of the others intent (Woods & Hollnagel, 2006, ch 8). (Examples in Yom Kippur, Caesar case) Projections. False assumptions about the universality of the observers experiences (i.e., false consensus) result in familiar patterns or relationships being projected onto the target, hiding ambiguous or unknown patterns. (Example in Kazakhstan case). Proportionality. This is the assumption that other groups will exert effort to a degree proportional to the importance of the goal of that effort (i.e., if the actions of the group appear costly, the observer will assume the goal is important). (Stein, 1988) (Example in Yom Kippur case) Over-estimation of Own Importance. The extent to which other groups are influenced by the observers group is exaggerated, and the role of other factors are neglected. (Jervis, 1976) (Example in Caesar case) Over-confidence. When simple explanations are available to account for the behavior of other groups, then the observer may falsely conclude that his or her understanding of the target organizations intentions, values, etc. is correct. (Stein, 1988) (Examples in Caesar, Kazakhstan cases) One approach to minimize the risk of these over-simplifications is the use of frameworks to support reflection on the process of reading intent. In addition to the rigor framework presented above, a framework based on the larger system of observation itself (including intelligence sources, sensors, networks, etc.) would support reflection. Furthermore, encompassing the broader context of multiple, adaptive groups interacting with one another would enable analysts to understand not only how the observer may be misunderstanding the intent of the target, but the target may be misunderstanding the intent of the observera key element in international conflict. This also includes issues of how intent is signaled to the other group (Jervis, 2002).

CONCLUSION

320TRENTETAL.
This series of studies looked at the Who, What, When, Where, Why, and How of military intelligence analysis, and in so doing has generated useful developments in three key areas: research methods, understanding the analysis process, and challenges in reading intent.

Method

The utilization of rich scenarios enables investigation of complex, realistic, facevalid intelligence analysis work while avoiding problems of access due to security or hostile environments. They also allow exploration of the impact of future technologies on analysis, and potential future security issues. Challenge cases (such as those incorporating factional groups and requiring the determination of intent) can support more focused exploration on difficulties in analysis. Scenarios can also be repurposed for use in new contexts. Collaborative work and embedded probes provide information on analysts cognitive processes without observer intervention or similar disruptions to the flow of work. The Supervisors Dilemma method provides a concrete and realistic scenario with which to trigger reflection on process and criteria for analysis and its products. The method of federated observation used in the 2nd Army Intelligence training exercise observation study developed a number of innovations, supporting consistency across multiple observers; enabling real-time quality control, and replanning and retasking; and more effective utilization of expert observer knowledge (Grossman et al., 2007; Trent et al., 2007). The methodological findings are integrated in a report on how to plan large scale training exercises as learning laboratories to assess cognitive challenges and envision requirements for new technology (Voshell & Woods, 2009).

Process

The findings from the observation studies show that the process of how analysis is conducted is not independent from the work environment, collaborative patterns, tools, or the nature of the analytic problems at hand. The evaluation of analytic work is affected by knowledge of the process by which it was created. The aspects of the process that illustrate analytic sufficiency do not conform to any simple prescriptive procedure. They are best understood as an emergent, multi-attribute construct.

CognitiveChallengesinIntelligenceAnalysis321

Intent

The process of reading intent has been identified as a particularly important yet challenging aspect of analytic work. Due to the complexities of the nature of groups of interest, the nature of intent itself, and the challenges in the observation process, intent reading in military intelligence analysis entails risks which normal day-to-day intent perception processes do not handle. As the role of stance evaluation of sources in the Rigor construct illustrates (Zelik, Patterson, & Woods, 2007a), understanding the process of how sources or groups signal intent, and how observers read intent, is necessary to mitigate the risks of intent misreading.

REFERENCES
Feltovich,P.J.,Hoffman,R.R.,Woods,D.,&Roesler,A.(2004).Keepingittoo simple:howthereductivetendencyaffectscognitiveengineering.IntelligentSystems, IEEE,19(3),9094. Grossman,J.B.,Woods,D.,&Patterson,E.(2007).SupportingTheCognitiveWork ofInformationAnalysisandSynthesis:AStudyOfTheMilitaryIntelligenceDomain.In ProceedingsoftheHumanFactorsandErgonomicsSociety51stAnnualMeeting(pp. 348352). Jervis,R.(1976).PerceptionandMisperceptioninInternationalPolitics.Princeton UniversityPress. Jervis,R.(2002).Signalingandperception:Drawinginferencesandprojecting images.PoliticalPsychology. Smith,M.,Branlat,M.,Stephens,R.,&Woods,D.(2008).CollaborationSupportvia AnalysisofFactions.InProceedingsofNATOResearchandTechnologyOrganization HumanFactorsandMedicineConference:AdaptabilityinCoalitionTeamwork(RTOMP HFM142).Copenhagen. Smith,M.,Patterson,E.,Zelik,D.,&Woods,D.(2007).FactionDisplay:Visualizing theSpectrumofPerspectives.InK.Moiser&U.Fisher(Eds.),ProceedingsoftheEighth InternationalNDMConference.PacificGrove,CA.Retrievedfromhttp://bss.sfsu.edu/ kmosier/ndm8.htm. Smith,M.,Stephens,R.,&Branlat,M.(2007).SpaceforFactionAnalysis.Posterand Presentation,Columbus,OH.Retrievedfromhttp://csel.eng.ohiostate.edu/smith/ FactionSpace/index.html. Stein,J.(1988).Buildingpoliticsintopsychology:themisperceptionofthreat. Politicalpsychology,9(2),245271. Trent,S.,Voshell,M.,Grossman,J.,Schoenwald,J.,Patterson,E.,Goldstein,S.,etal. (2007).FederatedObservationalResearch:Auniqueinvestigationstrategyforthestudy ofdistributedwork.InK.Moiser&U.Fisher(Eds.),ProceedingsoftheEighth

322TRENTETAL.
InternationalNDMConference.PacificGrove,CA.Retrievedfromhttp://bss.sfsu.edu/ kmosier/ndm8.htm. Trent,S.,Voshell,M.,&Patterson,E.(2007).TeamCognitioninIntelligence Analysis.InProceedingsoftheHumanFactorsandErgonomicsSociety51stAnnual Meeting(pp.308312). Trent,S.A.(2007).Teamcognitioninintelligenceanalysistraining.Dissertation.The OhioStateUniversity. Trent,S.A.,Patterson,E.S.,&Woods,D.D.(2007).ChallengesforCognitionin IntelligenceAnalysis.JournalofCognitiveEngineeringandDecisionMaking,1(1),7597. Voshell,M.andWoods,D.D.(2009).PlanningSupportforRunningLargeScale ExercisesasLearningLaboratories.CognitiveSystemsEngineeringLaboratory,Institute forErgonomics,TheOhioStateUniversity,ColumbusOH.March2009,Preparedfor ArmyResearchLaboratory. Woods,D.D.,&Hollnagel,E.(2006).JointCognitiveSystems:PatternsinCognitive SystemsEngineering.CRCPress. Zelik,D.J.(2009).Analyticalrigorininformationanalysis.Retrievedfromhttp:// csel.eng.ohiostate.edu/zelik/rigor/ Zelik,D.,Patterson,E.S.,&Woods,D.D.(2007a).UnderstandingRigorin InformationAnalysis.In8thInternationalConferenceonNaturalisticDecisionMaking, PacificGrove,CA. Zelik,D.,Patterson,E.S.,&Woods,D.D.(2007b).Supportingtheassessmentof rigor:Representinganalysistocreateprocessinsight.Paperpresentedatthe2ndAnnual WorkshoponMetaInformationPortrayal,Washington,DC. Zelik,D.J.,Patterson,E.S.,Woods,D.D.(2007c).Judgingsufficiency:How professionalintelligenceanalystsassessanalyticalrigor.ProceedingsoftheHuman FactorsandErgonomicsSociety51stAnnualMeeting(pp.318322).SantaMonica,CA: HumanFactorsandErgonomicsSociety.

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T A N D L A U R E L A L L E N D E R

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T AND LAUREL ALLENDER

This book, produced by the Partners of the Army Research Laboratory Advanced Decision Architectures Collaborative Technology Alliance, also serves as the Final Report for DAAD19-01-2-0009.

SECTIONIV.COMMUNICATINGINFORMATIONACCROSSTHETEAM

SECTIONIV COMMUNICATINGINFORMATIONACROSSTHETEAM

324SECTIONIV

Chapter

18

18. ADVANCINGOURUNDERSTANDINGOFBATTLEFIELDDECISIONMAKING

GaryKlein,Ph.D. ElizabethS.Veinott,Ph.D. WinstonR.Sieck,Ph.D. AppliedResearchAssociates,Inc.,KleinAssociatesDivision, Fairborn,OH

INTRODUCTION Theprocessofcommandandcontrol,whichiscentraltoU.S.Army operations,dependsheavilyoncognitivefunctionssuchasdecisionmak ing.Commandersandstaffmembersareengagedinplanningand replanning,ininterpretingtheactionsofadversaries,inreprogramming resources,inmakingpersonnelassignments,andinmanyrelatedactivi ties.Theyhavetomakehighstakesdecisionsunderuncertaintyandtime pressure.Inordertosupportthemthroughtraining,organizational design,anddecisionsupportsystems,itisessentialtounderstandhow theycanusetheirexperiencetomakeeffectivedecisionswhenfaced withcomplexconditions. In1989,theArmyResearchInstituteforBehavioralandSocialSci encessponsoredthefirstconferenceonNaturalisticDecisionMaking (NDM,Klein,Orasanu,Calderwood,&Zsambok,1993).TheNDMframe workwasdesignedtoinvestigatedecisionmakingunderdifficultcondi tions:vaguegoals,uncertainty,changingenvironmentalfeatures,high stakes,teamandorganizationalconstraintsonaction(Orasanu&Con nolly,1993).Itisimportanttounderstanddecisionmakingunderthese kindsofconditionsinordertodevelopeffectivetrainingprograms,infor mationtechnologyapplications,anddoctrine. OneofthebestknownoftheNDMaccountsistheRecognition PrimedDecision(RPD)model,whichexplainshowpeopleareableto makedecisionswithouthavingtocompareoptions(Klein,1998;Klein, 325

326KLEIN,VEINOTT,ANDSIECK Calderwood,&ClintonCirocco,1986).TheRPDmodelpositsthatpeople areabletorelyheavilyonthefirstoptiontheygeneratebecausetheir experiencehasbeencompiledaspatterns.Apatternmatchingprocess enablesthemtorapidlymatchasituationtoaprototypepatternthey havealreadycompiled,whichletsthemfocustheirattentiononrelevant cues,determineplausiblegoals,formexpectancies,andidentifypromis ingcoursesofaction.Therefore,inmostcasesthefirstoptiontheycon siderwillbeeffective.Theyonlyhavetogenerateadditionaloptions whentheirfirstoptionisevaluatedasunacceptable.Theydonoteven havetogenerateseveraloptionsinordertomakeacomparativeevalua tion.Theevaluationisconductedthroughaprocessofmentalsimulation (whichDeGroot,1946/1978,termed"progressivedeepening"). TheRPDmodelhasbeentested.Theinitialresearchwithfirefighters hasbeenextendedtootherdomains,andthedominantfindingisthat thestrategydescribedbythemodelisusedinthegreatmajorityofsitua tionsbetween80%and90%ofthetime.Rarelydodecisionmakers contrastalternativeoptionsusingacommonsetofevaluationcriteria. Thesefindingshavebeenreplicatedbydifferentresearchgroups.Klein (1998)hasprovidedacomprehensiveaccountofthetestsandreplica tionsoftheRPDmodel. BythetimetheArmyResearchLaboratoryAdvancedDecisionArchi tectures(ARLADA)programbeganin2001,theNDMframeworkhad becomewellestablishedforstudyingdecisionmakinginfieldsettings, andtheRPDmodelhadbecomewidelyacceptedasadescriptionofthe waypeopleactuallymakedecisions.TheARLADAprogramhasbeen extremelyvaluableformovingourunderstandingofdecisionmakingfor ward.ItdirectlyresultedinadvancesintheRPDmodel,andintheexten sionoftheNDMframeworktocoveradditionalcognitivefunctionsand processes.ThischapterdescribeshowtheARLADAprogramenabled KleinAssociatestomakeimportantprogress.OtherARLADAcontractors havealsomadeimportantprogress,particularlyinmodelingrecogni tionaldecisionmaking. THERECOGNITIONPRIMEDDECISION(RPD)MODEL OurextensionsoftheRPDmodelhavefocusedontwodirections:by gettingdeeperinsidethemodelandbyapplyingthemodeltorapidbat tlefielddecisionmakingduringtheplanningprocess.

BattlefieldDecisionMaking327 GettingInsidetheRPDModel OneofthechallengesoftheRPDaccountwastoformulatecomputa tionalmodels(Hutton,Warwick,Stanard,McDermott,&McIlwaine, 2001).Asweexpected,thecollaborationworkedintwodirections.It generatedanimpressivecomputationalmodelthathasbeenusedto guidefriendlyandadversaryelementsduringsimulations.Insteadof placingthemunderthecontrolofsimplerulesandrandomizers,theycan nowexhibitrudimentarydecisionmaking(Lebiere,Gonzalez,&Warwick, inpress).Thisdevelopmentrepresentsanimportantpotentialadvance inthestateoftheart(Warwick,Allender,&Yen,inpress)(see15.A ComputationalModelofNaturalisticDecisionMakingandtheScienceof Simulationonpage 273). ThecollaborationalsoforcedtheKleinAssociatesstafftothinkmore deeplyaboutcomponentsoftheRPDmodel,suchasthedevelopmentof prototypesandpatterns,theuseofpatternsinguidingattention,andthe inclusionofcoursesofactionwithinprototypesandpatterns.Perhaps themostvaluableanddifficultextensionwastothinkfurtheraboutthe natureofexpectancies,andthewaysthatexpectanciesaregenerated andassessed.Inadditiontogettingdeeperinsidethemodel,wehave beenapplyingthemodeltosupportflexibledecisionmakingduringthe planningprocess. ApplyingtheRPDModeltoRapidBattlefieldDecisionMakingandPlan ning TheFutureCombatSystem(FCS)programwasdesignedtosignifi cantlyincreasethepaceofoperationsforArmyforces.However,ifthe forcesremainedtetheredtohighlydeliberateplanninganddecision makingprocesses,theArmymighthavetroublerealizingtheadvantages ofFCS.Therefore,theArmybecameinterestedindecisionmakingand planningmethodsthatweremoreintuitiveeasiertouse,andfasterto apply. Asaresponsetothisneedforanincreaseddecisiontempo,Schmitt andKlein(1999)describeawaytousetheRPDmodelasthecenterpiece ofnewplanningdoctrine,callingittheRecognitionalPlanningModel (RPM).UsingtheRPM,acommanderisexpectedtobeconceptualizinga courseofactionfromtheverybeginning,uponlearningofthenewmis sion.Ratherthanwaitingforhisstafftogooffandgeneratethree coursesofaction,thecommandersimplyexplainshiscourseofaction. Thestaffthentakesitthenextstep,testingandoperationalizingit.Then

328KLEIN,VEINOTT,ANDSIECK thecommandercomesbackandtogetherwithhisstaff,wargamesthe courseofaction.Plannersandexecutorsparticipateinthewargame. Then,ifeverythinggoeswell,thestaffdevelopsanddisseminatesorders. Eventhedisseminationactivityisexpeditedbecausethestaffhasgotten advancenoticeofthelikelycourseofaction.SeeFigure18.1.

Figure18.1 RecognitionalPlanningModel TheRPMalsoincludesvariationsforcaseswherecommandersmay notimmediatelyknowwhattheywanttodo,orforcaseswherethestaff findsaflawinaplanoridentifiesasuperioroptionwhileengagedin operationalizingthecourseofaction. TheeffectivenessofRPMwasdemonstratedinasimulationexercise conductedattheBattleCommandBattleLaboratory(BCBL)atFt.Leav enworth.OurteamembeddedandassessedRPMaspartofatwoweek experimentthatassembledanObjectiveForceUnitofAction(brigade) staffalongwithanotionalUnitofEmploymentheadquarters,andsev eralbattalioncommanders.InanarticlepublishedinMilitaryReview,we havedescribedhoweasilyandeffectivelytheRPMwasused,replacing themoredeliberateMilitaryDecisionMakingProcess(MDMP)(Ross, Klein,Thunholm,Schmitt,&Baxter,2004). AlthoughintheendRPMwaseasilyused,severalparticipantsinthe experimenthadbeenskepticaloftheRPMattheoutset.Theyresisted theideaofusingRPMtoreplacetheMDMP,aprocedurethattheyhad allbecomefamiliarandcomfortablewith.Subsequenttotheexercise, theevaluationoftheRPMfoundthatparticipantshadlittledifficultyin learningandapplyingtheRPM.TheyestimatedthattheRPMtook30% lesstimethantheMDMPwouldhave.Astheexerciseprogressed,favor ablecommentsincreased.Severalparticipantsrecalledinstanceswhere theyhadtosufferwithinadequateplansinitiatedbyinexperiencedplan ningstaffmembers.RPMallowsthecommandertodrivetheprocess,

BattlefieldDecisionMaking329 increasingplanqualityaswellastempo.Butthisisnottheonlyreason thatRPMworkswell. AnimportantreasonthattheRPMworkssowellisthatSchmittand Kleindesignedittomirrorthewaycommandersactuallymakedecisions. Consequently,thisdoctrinedoesnotimposeunnaturalprocesses.Con sistentwiththisnotionofseamlessadoption,theBCBLroleplayers(pri marilyretiredArmyColonels),whenqueriedabouttheirreactionstothe newRPMprocedure,reportedthattherewasnothingnewtheywere alreadydoingit.Andthatwasthepoint,tohavearapiddecisionmaking andplanningdoctrinethatwasintuitive,easytolearn,andconsistent withcurrentpractices. TheRPMhasbecomethedoctrinalplanningmethodfortacticalmis sionsintheSwedishArmy,duetotheresearchofPeterThunholm(2003) whodemonstrateda20%increaseintempowithnoreductioninplan quality.ResearchintheUnitedKingdomhasshowntheArmyofficers prefertheRPMtoalternativetoolsdesignedtoincreasetempo. TheprocessofrevisingArmydoctrineisnottobetakenlightly. Amongotherfunctions,doctrineensuresthatsoldierswhohavenot extensivelytrainedtogethercanaccuratelyandreliablycommunicate witheachother,andcaneffectivelycoordinatetheiractions.Whilethe RPMisnowherenearreadytobecomepartofArmydoctrine,thenotion ofintuitivedecisionmakinghasactuallybecomedoctrine,andhasbeen includedinArmyFieldManual6.0CommandandControl.Theinclusion ofintuitivedecisionprocessesduringplanningandexecutionwasbased inlargepartontheresearchthatgeneratedandvalidatedtheRPD model. ReplanningisNotPlanningintheBattlefield Asimportantasplanningisduringexecution,soldiersaremuchmore heavilyengagedinreplanning,ofteninaverytimecompressedfashion. YettheArmyprovideslittleguidanceandnodoctrineforhowtoreplan. ArmyFieldManual60,describingdoctrineforCommandandControl,is approximately300pageslongandprovidesgreatdetailonhowtoplan, butthereisonlyasingleparagraphonreplanning,andthatofferslittle thatisinformativeoruseful. Weconductedoneofthefirstcognitiveinvestigationsintothe requirementsforreplanning.Klein,Wiggins,andLewis(2003)defined replanningasmodifyingthemissiongoalsandtheplaninprogressto accomplishthosegoals.Itisdifferentfromexecution,whichrefersto adjustmentsmadetoachievegoalswithoutsignificantlyalteringthose

330KLEIN,VEINOTT,ANDSIECK goals.Replanningisalsodifferentfromcontingencyplanning(identifying branchesandsequels)whichisdonepriortoinitiatingtheplan.Kleinet al.(2003)showedthatreplanningwasnotsimplyaformofrapidplan ning;ithasitsowndynamic,whichtheyrepresentedinadynamic replanningtemplate. Replanningisnotjustinitiatedwhenaplanrunsintotrouble;itcan alsobetriggeredwhenanopportunityisspotted.Replanningisnota meanstoachievethemissiongoals;itmayrequiregoalnegotiation.It doesnotalwaysrelyonfeedbackaboutprogress;feedbackmechanisms canbetooslowandsocommandersneedtoanticipateprogress.Replan ningdoesnotreplaceanoldplanwithanewone;thenewplanusually triestosalvageasmuchoftheoldplanaspossible.Replanningismore difficultthanplanningbecauseoftheaddeduncertaintyaboutresource expendituresduringplanexecution;logisticiansandcommandershave toestimatethestatusofresources.Replanningdoesnotendwiththe issuanceofafragmentaryorder(FRAGO)oroperationorderbecausethe FRAGOmaydisruptcommongroundandresultincoordinationbreak downs.Andreplanningdoesntincreaseflexibility,itreducesflexibility becauseitreducespredictabilityandmakescoordinationmorebrittle. ReplanningisnotalwaysinitiatedbyaFRAGOfromabove;sometimesit isselfinitiatedwhenacommanderdeterminesthattheinitialplanisno longerfeasibleordesirable.Andreplanningisnotsolelytheprovinceof theexecutionstaff;itoftenrequirestimepressuredcoordination betweenplannersandexecutors. Ourresearchonreplanninghassinceevolvedfurtheranddeveloped intotheconceptofManagementbyDiscovery.ManagementbyDiscov ery,asacontrasttoManagementbyObjectives,isessentialwhendeal ingwithilldefinedgoals,andrequiresthedecisionmakerstodefine theirgoalswhileintheactoftryingtoattainthosegoals(Klein,2007a, 2007b;Klein&Rothman,2008).WehavepresentedtheManagementby DiscoveryprocessinseveralseminarstotheSchoolforAdvancedMili taryStudyprogram(SAMS)atFt.Leavenworth,andtoaboutahalfdozen workshopsfornewlypromotedArmygeneralofficers.Allofthiswork originatedintheARLADAprojectonreplanning,andisanextensionof naturalisticdecisionmaking.

BattlefieldDecisionMaking331 MACROCOGNITIONFRAMEWORK:EXPANDINGBEYONDNATURALISTIC DECISIONMAKINGTOSTUDYMACROCOGNITIONINBATTLEFIELD SITUATIONS WhilewewereexpandingtheRPDmodelduringtheARLADApro gram,wewerealsoexpandingtheNDMframeworkitself.Wewerelearn ingthatdecisionmakingcannoteasilybeseparatedfromothercognitive functionssuchassensemaking.Wedefinesensemakingastheactivityof fittingobservedeventsanddataintoaframe,suchasanarrative,while usingaframetodefinedataelements.TheRPDmodelreallyaddresses bothsensemakinganddecisionmaking.Andbotharerelevanttoplan ningandreplanning.Kleinetal.(2003)proposedtheconceptofmac rocognitionasthenextstepintheevolutionofNDM.Theydescribeda smallsetoffunctionsandprocessesthatoccurrepeatedlyinnaturalset tings.Figure18.2presentsacurrentversionofthesemacrocognitive functionsandprocesses.

Figure18.2 Macrocognitiondiagram(figureprovidedbyRobertHoffman,allrights reserved) Schraagen,KleinandHoffman(2008)havedefinedmacrocognition asthecognitiveadaptationtocomplexity.Further,themacrocognitive functionsandprocessesarerelevantatseverallevels.Theyarerelevant

332KLEIN,VEINOTT,ANDSIECK forindividuals,whomakedecisions,makesenseofevents,andsoforth. Theyarerelevantattheteamlevelandattheorganizationallevel becauseteamsandorganizationsalsohavetomakedecisions,make senseofevents,planandreplan,detectproblems,andsoforth.Finally, theyarerelevantwhendesigningtoolstosupportamacrocognitivepro cess(Klein,DominguezandWiggins,inpress).Ateachlevel,fromindivid ualtoteamtoorganization,newchallengesemergeandnewstrategies comeintoplay. WiththeelaborationofNDMintoaMacrocognitiveframeworkcame arequirementtoprovidemethodsforstudyingmacrocognitionandits differentfacets.Workingminds:Apractitionersguidetocognitivetask analysis.(Crandall,Klein,&Hoffman,2006),waswrittenasanARLADA ResearchTask,inordertosupportcognitivesystemsengineeringefforts. Itisnowthebasictextinhowtoperformcognitivetaskanalysismeth ods.Inaddition,KleinandHoffman(2008)havedescribedmethodsfor usingCTAtoprobedifferentkindsofmentalmodels,oneofthemac rocognitiveprocesses. MacrocognitiveMeasures Eversincethemacrocognitionframeworkwaspresented,KleinAsso ciateshasbeeninterestedinidentifyingmeasuresforthedifferentfunc tionsandprocesses.TheARLADAprogramofferedanopportunityto workwithRobertHoffman(IHMC)andDavidWoods(OSU)toformulate asetofmeasures,inthecontextofevaluatingsomeofthetoolsdevel opedbyotherARLADAprojects(see6.HowGoodisthatNewSoftware Tool?TheMathematicalModelingofPerformanceMetricson page 107).Klein(inpress)hasdevelopedasetofmacrocognitivemea sures. Forexample,themacrocognitivefunctionofsensemakingincludes severalimportantaspectssuchasseekinginformation.Tomeasurethis aspect,researcherscanassessthetimespentsearchingforagivendata element,suchasadocument,andthequalityofthedataelements selectedforreview(aspreratedbysubjectmatterexperts).Thesewere themeasuresusedbyPattersonetal.(2001)intheirresearchonintelli genceanalystssearchingforunclassifiedinformationinanopendata base. Setsofmacrocognitivemeasureshavebeendevelopedfordecision making,planning/replanning,andcoordination,aswellasforsensemak ing.Thepurposeofthesemeasuresistoallowresearcherstostudythe effectsofdifferentinterventions.Forexample,ifatoolisproposedto

BattlefieldDecisionMaking333 improvesensemaking,somecombinationofmacrocognitivemeasures couldbehelpfulindeterminingwhichaspectsofsensemaking,ifany,are affectedbythetool. GuideScanningToolDevelopmentforIntelligenceAnalysts Themacrocognitiveframeworkisalsointendedtoguidethedevel opmentoftoolstosupportdifferentfunctionsandprocesses.The ARLADAprogramsupportedthedevelopmentofatooltohelpintelli genceanalystsperformmoreefficientandeffectivesearchesthrough largedocumentsets(O'Dea,HarrisThompson,Malek,Dominguez,& Crandall,2007).Thisactivitystemsdirectlyfromthesensemakingfunc tioninFigure18.2above.ItstemsfromtheData/Framemodelofsense making(Kleinetal.,2003)aboutthereciprocalnatureofsensemaking, usingdatatoidentify/developframes,andusingframestodefinedata. Fromthisperspective,informationsearchneedstobeinteractivethe intelligenceanalystsneedtolearnhowtoadapttheirsearchstrategies basedonwhattheyarediscovering.Theyareengagedinaprocessakin toperceptuallearning.Insteadoflearninghowtoseeorhearmore clearly,theintelligenceanalystsattheirworkstationsarelearninghow totunetheirsearchprocess.Tosomeextent,intelligenceanalystsare wrestlingwithdataoverload,butitmaybeamistaketotreattheirstrug gleasanoverloadproblemthatrequiresfiltering.Rather,weviewedthe issueasonethatrequiresmoreskillfulanddiscoverybasedsearchstrat egies. Inthiseffort,weconducted15cognitivetaskanalysisinterviewswith intelligenceanalystsacrossfourdifferentintelligenceorganizations: NationalAirandSpaceIntelligenceCenter,JointForceIntelligenceCom mand,JointWarfareAnalysisCenter,andNationalGroundIntelligence Center.Theinterviewsfocusedonthecognitiveworkassociatedwithset tingprofilesandqueries.Theinterviewswereconductedatanunclassi fiedlevel.Theinterviewsenabledustoidentifyanalyststrategiesfor settingprofilesandconductingsearches,alongwithsomeofthekey challenges.Weconfirmedthatintelligenceanalystsspendlargeamounts oftimedevelopingprofilesandconductingsearches,andthatbymaking evensmallimprovementsintheirefficiency,perhaps20minutessaved perday,wemightbeabletocreatelargesavings. Wedevelopedaconceptualdesignforatooltoassistintelligence analyststoconductmoreeffectiveandefficientsearches.Oneofseveral componentsprovidedameansforvisualizingtheeffectofenteringdif ferentsearchterms.SincethecompletionoftheARLADAprojectwe

334KLEIN,VEINOTT,ANDSIECK haveobtainedoutsidefundingthatallowedustotakethisprojecttothe nextlevel,developingaworkingprototypeofthetool. CoordinationandtheTeamLevelofMacrocognition Kleinetal.(2004a,2004b)studiedthemacrocognitivefunctionof coordination,andtherelatedprocessofmaintainingcommonground. Kleinetal.(2004a)describedtherequirementsforeffectivecoordination forteamsandorganizations,particularlytheroleofcommongroundand theresponsibilitytorepaircommongroundifitappearstobedegrading. Kleinetal.(2004b)describedtherequirementsofintelligentsystemsto support,maintain,andrepaircommongroundiftheyaretobeteam players.WithintheARLADAprogram,KleinAssociatesperformedthree setsofprojectstogainadeeperunderstandingofteamcoordinationof macrocognitivefunctions.Thefirstsetofprojectsfocusesontheques tion,howdoesorganizationalstructureaffectcoordinationandteam work?Thesecondsetfocusesoncollaborationinlargescalesociotechni calsystems.Thefinalsetofprojectsfocusesondifferencesinmacrocog nitionacrossculturesandasksthequestion,domultinationalteams sharedecisionmakingvalues? BattlefieldCoordinationandTeamwork.Weconductedadirect examinationofbattlefieldlevelcoordinationandteamworkduringaBat tleCommandBattleLaboratoryexerciseconductedin2002atFt.Leaven worth.Westationedeightobserversattheexerciseandalsoconducted postexerciseinterviewswithexerciseparticipantsusingaMacrocogni tionInterviewmethoddevelopedspecificallytoexploremacrocognitive functionsandprocesses.Otherinterviewsexaminedsensemaking,and usedmethodssuchasaCommanderQuickAssessment,aTaskDiagram, aWagonWheelexercisetotracecoordinationroutes,aDecisionrequire mentsexercise,andaTeamAuditinterview. TheBCBLexercisewasdesignedtoexploreaninnovativeorganiza tionalstructureemployingaUnitofActionstaffstructure.Therefore,it representedafruitfulopportunitytodeterminehowthenewstructure, andtheparticipantsunfamiliaritywithit,affectedmacrocognitivefunc tionsandprocesses.Ross,Crandall,andBattaglia(2003)foundthatthe newstructurewasnotwellunderstoodbytheexerciseparticipantsand ledtoanimbalanceinworkloadamongdifferentstafffunctions.They suggestedwaysofrevisingthestafffunctionsbycombiningoreliminat ingsomeroles.Theemphasisofthesesuggestionswasonimproving macrocognitivefunctionsattheorganizationallevelspecifically,to improvedecisionmaking,sensemaking,andplanning,whileimproving

BattlefieldDecisionMaking335 staffcommonground.Inadditiontoorganizationalstructureaffecting macrocognitiveprocesses,sodothetechnicalsystemsthatteamsinter actwithanduse. SociotechnicalInteraction.Morerecently,KleinAssociateshasstud iedtheeffectofcollaborativesystemsthemselvesonthesocialdynamics ofteamsastheyperformmacrocognitiveprocesses.Collaborativenet workedsystemshaveuniquechallengesrelativetosingleuser,stand alonesystemsinthattheyimpactateaminadynamic,contextdepen dentandsocialenvironment.Thisisespeciallytrueincommandandcon trolenvironments. Duetothehighlycontextualandcomplexnatureofcollaborativesys tems,itisdifficulttoanticipateallthewaysindividualsandgroupsmight usethesystems.Forexample,systemscanviolatepeoplesmentalmod elsofitsusefulness(Yates,Veinott,&Patalano,2003),theycanchange thegroupprocessesandfailtoachievecriticalmassbecausenotenough peopleusethem(Orlikowski,1992),ortheycanhaveorganizational impactsbyviolatingnormsandroles(Grudin,1995). Weconductedanonlineethnographytoevaluatealightweightdis tributedcollaborativetoolusedinarecentdisasterresponseeffort (Veinott,Cox,&Mueller,2009).Thiscollaborativetoolallowedshort (e.g.,140characters),crossplatformpostskeepinginformationonpoint; however,wefoundthatitwasdifficulttogainanoverallpictureofthe situationwiththistool.Duringtheactualdisaster,Googlemapsjoined theresponseeffortbyworkingaroundtheclocktoupdatetheirmaps withinformationpostedusingthislightweighttool.Thisresearchispart ofanongoingeffortthathasfocusedonevaluatingmethodsforimprov ingtheArmysabilitytoanticipatethesociotechnicaleffectsofasystem bycapturingthecomplexityofthecontext.Inadditiontobattlefielddeci sionmakingandcollaborationbeingaffectedbythetoolsthatpeople haveavailable,itisalsoaffectedbyculturalcontext.Whetheroneis workingwithlocalforces,oraspartofaJointForcesCommand,cultural understandingisakeycomponent. CulturalMacrocognition.Lookingattheteam/organizationallevel, KleinAssociateshasperformedasetofprojectsexploringthewaysthat macrocognitivefunctionsandprocessesvaryacrosscultures.Wecon ductedaseriesofstudytodeterminewhetherbeliefsaboutdecision makingvaryacrosscultures,howthesebeliefsvary,andhowthese beliefscanbemeasured. Inoneofthefirststudies,wetestedapreviouslyheldassumption thatcollaborationorteamworklooksthesame,nomatterwhereyou areintheworld.Giventhedegreeoffrictionanddisappointingperfor

336KLEIN,VEINOTT,ANDSIECK mancethathasplaguedmultinationalteamswethoughtthatcultures maydifferintheirviewsonteamwork.Thatis,Westernbasedmodelsof teamwork(e.g.,Salas,Sims,andBurke,2005BigFiveofteamwork) maynotaccuratelyrepresenttheprocessesemployedbymembersof nationswhoexhibitdistinctsocialandcognitivepatterns. AftersurveyingparticipantsfromChina,Taiwan,India,Chile,Japan, Brazil,Turkey,Korea,andtheU.S.regardinghowdecisionsaremadeina team,howconflictisaddressed,andtherolethattheleaderplays,we developedageneralmodelofthecollaborativedecisionmakingprocess. Thismodelrepresentscoreelementsofthecollaborativedecisionmak ingprocessthataresharedacrosscultures(i.e.,divergenceandconver gence,decisionpoint,commitment,execution,andopportunityfor change),butthatvaryinthemeansbywhichtheyareaccomplished.We foundthatdifferencesincollaborativedecisionmakingacrosscultural groups(particularlyWesternersandFarEasterners)arecomplexand nuanced,withvaluesforrespect,personalrelationshipsandtrust,con sensus,andpreservationoffaceplayingcrucialrolesinhowthedecision makingprocessunfolds,includingthedegreeoftransparencyinthepro cess(McHugh,Smith,&Sieck,2008).Theserepresentpointsoflikelydis connectormisunderstandinginmultinationalteams,andshouldbe targetedasareasofnegotiationamongmultinationalteammembers. Next,weconductedalaboratoryexperimenttoevaluatethecore hypothesisthathighlightingandreconcilingculturaldifferencesincollab orativedecisionmakinghasapositiveinfluenceonperformancewithin multinationalteams.Theliteraturehassuggestedthatthroughtheexpe rienceofinteractingwitheachother,culturallydiverseteamscan developasimplifiedsetofrules,norms,expectationsandroles(Ham brick,Davison,Snell&Snow,1998).Emergenceofahybridculturehas beenshowntofacilitateperformance(Casmir,1992). Surprisingly,ourresultsindicatedthatthegroupsthathadpartici patedinthereconciliationprocedureperformedsignificantlyworseon theideagenerationandrankorderingtasks,andtheyexperiencedlower overallteamefficacy.Webelievethatthereasonforthiseffectisthatthe reconciliationprocedureincreasedthesalienceofculturaldifferences (Rasmussen,Smith,&Sieck,2007).Thisinterpretationisconsistentwith afindingthatreducingthesalienceofculturaldifferencesbyusingcom municationtechnologiesimprovesdecisionsinshortterm,culturally diverseteams(Staples&Zhao,2006). Theseresultssuggestthatamoresubtleapproachtofosteringhybrid culturesinmultinationalteamsshouldbeconsidered.Specifically,we developedanewconceptinvolvingtheconstructionofculturalmetrics

BattlefieldDecisionMaking337 forprovidingabackgroundassessmentoftheteamsconsensusabout appropriatecollaborativedecisionprocesses,aswellasmeasuringany convergenceovertime(Sieck,W.R.,Smith,J.L.,McHugh,A.P.,2007). OurresearchhasnowdevelopedintoCulturalNetworkAnalysis (CNA)(SieckandRasmussen,2007).CNArepresentsaninterdisciplinary synthesisofmethodsdrawnfromthefieldsofcognitiveanthropology, culturalpsychology,naturalisticdecisionmaking,anddecisionanalysis. Theconsensusofindividualsmentalmodelsconstitutesaculturalmodel forthatgroup,typicallydepictedasanetworkrepresentationofthecul turallyprevalentinterrelatedconcepts,causalbeliefs,andvaluesthat influencekeydecisions.Thisapproachhasthepotentialtobeusefulin jointcommandandcontrolsituations. CONCLUSIONS TheARLADAprogramhasbeenofextremevalueinmovingthefield ofNaturalisticDecisionMakingforward.Duringtheeightyearsofpartici patinginARLADA,wehaveexpandedontheRecognitionPrimedDeci sion(RPD)model,clarifyingsomeofthefunctionswithinthemodel, helpingtodevelopacomputationalversionofthemodel,andextending themodeltoprovidearecommendationforaplanningdoctrinethat wouldgaintempowithoutsacrificingquality. Inaddition,wehaveexpandedthebordersofNaturalisticDecision Makingtothepointwhereweidentifiedalargerscaleframeworkof Macrocognition.InmovingNDMtothenextlevel,wehaveinitiateda rangeofactivitiestopreparemacrocognitivemeasuresforthevarious functions,toprepareasearchtooltosupportsensemakinginintelli genceanalysts,andtoworkouttheimplicationsofmacrocognitionat theteamandorganizationalleveltheimpactoftechnologyonteam macrocognition,andtheexistenceofculturaldifferencesinmacrocogni tion. ComparedtothestateoftheartforNDMin2001,atthebeginning oftheARLADAprogram,weseeconsiderableprogressandmaturityin thefield.TheARLADAprogramhasofferedanopportunitytogroundour workinnaturaltasksandchallengesrelevanttotheArmy,toworkwith creativeandstimulatingcolleagues,andtoexploredifferentpathsin movingthemodelsandmethodsforward. REFERENCES
Casmir,F.L.(1992).Thirdculturebuilding:Aparadigmshiftforinternationaland interculturalcommunication.CommunicationYearbook,16,407428.

338KLEIN,VEINOTT,ANDSIECK
Crandall,B.,Klein,G.,&Hoffman,R.R.(2006).Workingminds:Apractitioner'sguide toCognitiveTaskAnalysis.Cambridge,MA:TheMITPress. DeGroot,A.D.(1946/1978).Thoughtandchoiceinchess.NewYork,NY:Mouton. Grudin,J.(1995).Interactivesystems:Bridgingthegapbetweendevelopersand users.InR.M.Beacker,J.Grudin,W.Buxton&S.Greenberg(Eds.),Readingsinhuman computerinteraction:Towardstheyear2000(2nded.,pp.293304).SanFrancisco,CA: MorganKaufmann. Hambrick,D.,Davison,S.,Scott,S.,Snow,C.(1998).Whengroupsconsistofmultiple nationalities:Towardsanewunderstandingoftheimplications,OrganizationalStudies, 19,181205. HeadquartersDepartmentofArmy(2002).FM60,MissionCommand:Command andControlofArmyForces.Washington,D.C. Hoffman,R.R.,&Militello,L.G.(2009).Perspectivesoncognitivetaskanalysis: Historicaloriginsandmoderncommunitiesofpractice.NewYork,NY:Taylor&Francis. Hutton,R.,Warwick,W.,Stanard,T.,McDermott,P.,&McIlwaine,S.(2001). ComputationalModelofRecognitionPrimedDecisions(RPD):ImprovingRealismin ComputerGeneratedForces(CGF).PaperpresentedattheHumanFactorsand ErgonomicsSociety,Minneapolis,St.Paul,MN. Klein,G.(inpress).Macrocognitivemeasures.InE.S.PattersonandJ.Miller(Eds.) Macrocognitionmetricsandscenarios:Designandevaluationforrealworldteams. AshgatePublishing. Klein,G.(1998).Sourcesofpower:Howpeoplemakedecisions.Cambridge,MA:MIT Press. Klein,G.(2007a).Flexecutionasaparadigmforreplanning,Part1.IEEEIntelligent Systems,22(5),7983. Klein,G.(2007b).Flexecution,Part2:Understandingandsupportingflexible execution.IEEEIntelligentSystems,22(6),108112. Klein,G.,Feltovich,P.J.,Bradshaw,J.M.,&Woods,D.D.(2004a).Commonground andcoordinationinjointactivity.InW.B.Rouse&K.R.Boff(Eds.),Organizational simulation.NewYork:JohnWiley&Sons. Klein,G.,&Hoffman,R.(2008).Macrocognition,mentalmodels,andcognitivetask analysismethodology.InJ.M.Schraagen,M.L.G,T.Ormerod&R.Lipshitz(Eds.), Naturalisticdecisionmakingandmacrocognition(pp.5780).Hampshire,U.K.:Ashgate. Klein,G.,Lewis,W.R.,&Klinger,D.W.(2003).Replanningthearmyunitofaction commandpost(AnnualProjectReportsubmittedunderContractDAAD1901S0009 preparedfortheArmyResearchLaboratory).Fairborn,OH:KleinAssociatesInc. Klein,G.,Ross,K.G.,Moon,B.M.,Klein,D.E.,Hoffman,R.R.,&Hollnagel,E.(2003). Macrocognition.IEEEIntelligentSystems,18(3),8185. Klein,G.,&Rothman,J.(2008).Newdirections:Stayingoncoursewhenyour destinationkeepschanging.TheConferenceBoardReview,November/December,2427. Klein,G.,Wiggins,S.L.,&Dominguez,C.O.(inpress).Teamsensemaking. TheoreticalIssuesinErgonomicScience. Klein,G.,Wiggins,S.L.,&Lewis,W.R.(2003).ReplanningintheArmybrigade commandpost.InProceedingsofthe2003CTASymposium[CDROM]. Klein,G.,Woods,D.D.,Bradshaw,J.M.,Hoffman,R.R.,&Feltovich,P.J.(2004b). Tenchallengesformakingautomationa"teamplayer"injointhumanagentactivity.IEEE WintertreeSystems.,19(6),9195.

BattlefieldDecisionMaking339
Klein,G.A.,Calderwood,R.,&ClintonCirocco,A.(1986).Rapiddecisionmakingon thefireground.ProceedingsoftheHumanFactorsandErgonomicsSociety30thAnnual Meeting,1,576580. Klein,G.A.,Orasanu,J.,Calderwood,R.,&Zsambok,C.E.(Eds.).(1993).Decision makinginaction:Modelsandmethods.Norwood,NJ:Ablex. Lebiere,C.,Gonzalez,C.,&Warwick,W.(inpress).ConvergenceandConstraints RevealedinQualitativeModelComparison.JournalofCognitiveEngineeringandDecision Making. McHugh,A.P.,Smith,J.,&Sieck,W.R.(2008).Culturalvariationsinmentalmodels ofcollaborativedecisionmaking.InJ.M.C.Schraagen,L.Mitello,T.Omerod,&R.Lipshitz (Eds).MacrocognitionandNaturalisticDecisionMaking.Aldershot,UK:Ashgate PublishingLtd. O'Dea,A.,HarrisThompson,D.,Malek,D.A.,Dominguez,C.O.,&Crandall,B.W. (2007).Indicatorsforassessingcollaborationtechnologies.PaperpresentedattheEighth InternationalConferenceonNaturalisticDecisionMaking,Asilomar,CA. Orasanu,J.,&Connolly,T.(1993).Thereinventionofdecisionmaking.InG.A.Klein, J.Orasanu,R.Calderwood&C.E.Zsambok(Eds.),Decisionmakinginaction:Modelsand methods(pp.320).Norwood,NJ:Ablex. Orlikowski,W.(1992).Learningfromnotes:Organizationalissuesingroupware implementation,Proceedingsofthe1992ACMConferenceonComputersupported cooperativework(pp.362369).Toronto,Ontario,Canada. Patterson,E.S.,Roth,E.M.,&Woods,D.D.(2001).PredictingVulnerabilitiesin ComputerSupportedInferentialAnalysisUnderDataOverload.Cognition,Technology andWork,3,224237. Rasmussen,L.,Smith,J.L.,&,Sieck,W.(2007).Effectsofaculturalreconciliation procedureonmultinationalcollaborativedecisionmaking.Proceedingsofthe2007 GroupNegotiationandDecisionMakingMeeting. Ross,K.G.,Crandall,B.,&Battaglia,D.A.(2003).Complexcognitionofbattle commandintheobjectiveforceunitofaction(UA).InProceedingsofthe2003CTA Symposium. Ross,K.G.,Klein,G.,Thunholm,P.,Schmitt,J.F.,&Baxter,H.C.(2004).The recognitionprimeddecisionmodel.MilitaryReview,LXXIV(4),610. Rudolph,J.W.(2003).Intothebigmuddyandoutagain.UnpublishedDoctoral Thesis,BostonCollege,Boston,MA. Salas,E.,Sims,D.E.,&Burke,C.S.(2005).Istherea"BigFive"inteamwork?Small GroupResearch,36(5),555599. Schraagen,J.M.,Klein,G.,&Hoffman,R.(2008).Themacrocognitiveframeworkof naturalisticdecisionmaking.InJ.M.Schraagen,L.Militello,T.Ormerod&R.Lipshitz (Eds.),Naturalisticdecisionmakingandmacrocognition(pp.326).Hampshire,England: Ashgate. Schmitt,J.&Klein,G.(1999).HowWePlan.MarineCorpsGazette,October,1826, Sieck,W.R.,&Rasmussen,L.(2007).CulturalNetworkAnalysis.Proceedingsofthe FirstAnnualConferenceoftheUS/UKInternationalTechnologyAlliance,Baltimore,MD. Sieck,W.R.,Smith,J.,&McHugh,A.P.(2007).Crossnationalcomparisonofteam competencyvalues.Proceedingsofthe51stHumanFactorsandErgonomicsSociety, October2007. Staples,D.S.,&Zhao,L.(2006).Theeffectsofculturaldiversityinvirtualteams versusfacetofaceteams.GroupDecisionandNegotiation,15,389406.

340KLEIN,VEINOTT,ANDSIECK
Thunholm,P.(2003).Militarydecisionmakingandplanning:Towardsanew prescriptivemodel.UnpublishedDoctoraldissertation,StockholmUniversity, DepartmentofPsychology,Stockholm. Veinott,E.,Cox,D.,&Mueller,S.(2009).SocialMediaSupportingDisasterResponse: Evaluationoflightweightcollaborativetool.PaperpresentedattheTenthInternational ConferenceonNaturalisticDecisionMaking.,London,England. Warwick,W.,Allender,L.,&Yen,J.(inpress).DevelopingandUnderstanding ComputationalModelsofMacrocognition.JournalofCognitiveEngineeringandDecision Making. Yates,J.F.andVeinott,E.S.,andPatalano,A.L.(2003).Harddecisionsandbad decisions:Ondecisionqualityanddecisionaiding.In(S.L.SchneiderandJ.Shanteau, Eds.)EmergingPerspectivesonJudgmentandDecisionResearch.CambridgeUniversity Press.

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T A N D L A U R E L A L L E N D E R

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T AND LAUREL ALLENDER

This book, produced by the Partners of the Army Research Laboratory Advanced Decision Architectures Collaborative Technology Alliance, also serves as the Final Report for DAAD19-01-2-0009.

SECTIONIV.COMMUNICATINGINFORMATIONACCROSSTHETEAM

SECTIONIV COMMUNICATINGINFORMATIONACROSSTHETEAM

324SECTIONIV

Chapter

19

19. ATASKBASEDEVALUATIONMETHODFOREMBEDDEDMACHINE TRANSLATIONININSTANTMESSAGINGSYSTEMS

WilliamOgden,Ph.D. NewMexicoStateUniversity,LasCruces,NM

INTRODUCTION Crosslanguageinstantmessaging(CLIM)systemsareintendedtobe usedbygroupsofpeoplewhodonotspeakthesamelanguagebutneed toworktogethertoaccomplishcommontasks.Suchsystemscouldhelp coordinateteamsandfacilitatecommunicationsbyautomaticallytrans latingtextmessagessenttoteammembers.Membersoftheteamwould beabletoreadandentermessagesintheirownlanguagesandstillcom municatewithallmembersofthegroup.Thiscouldbeparticularlyuseful formanymultinationalcivilianandmilitarymissions,whereparticipants maynotspeakEnglishfluentlyoratall. Inthischapter,wedescribeaseriesofuserstudiesdesignedtoinves tigatetheusefulnessofthesesystemsforthesepurposes.Thesestudies werefocusedonthreedistinctgoals.First,wewantedtodevelopanew taskbasedmethodforevaluatingtheembeddedmachinetranslation (MT)technologyinthesesystems.PreviousMTevaluationshavenot beenfocusedonthistypeofinteractiveapplication.Second,wewanted tousethismethodtoevaluatetheusefulnessofMTtechnologyfor instantmessagingandsuggestspecificimprovementsforthisapplica tion.Finally,wewantedtoconductformativeevaluationsoftheuser interfaceforcrosslanguageinstantmessagingtoimprovetheuserexpe rienceofthesesystems.Consequently,weexaminedthehabitabilityof thesystemforaccomplishingrealisticcollaborativetasksandtocommu nicaterelevantinformationwheninteractingwithcollaboratorswho speakdifferentlanguages.Insum,wewereinterestedintechnicaldesign

341

342OGDEN issuesofboththeuserinterfaceandtheMTsoftwareattheuserinterac tionlevel. Inthisefforttodate,wehavedevelopednewparadigmsforevaluat ingmachinetranslationtechnologyforpurposesofcollaborativecom munication.Wehavebeenconductingformativeusertestingofa translingualinstantmessagingsysteminterface(TrIM),developedbyThe MITRECorporationtohelpmilitarycoalitionpartnerstocommunicate usingtheirownlanguages(Milleret.al.,2001).Inthispaper,wewill reviewissuesofconductingtaskbasedevaluationsofembeddedMTand discusssomeoftheresultsandimplicationsforMTtechnologyembed dedinCLIM.Nextweusetheseimplicationstodesignandtestan improvementtotheTrIMuserinterface.Thenwedescribeastudytotest thesensitivityofthemethodwehavedevelopedbycomparingtwo KoreanEnglishtranslationsystems. BACKGROUND Instantmessaginghasbecomepopularwithmilitaryusersbecause ofitsimmediacyandlowbandwidthconsumption.Becausethe applicationpermitsrealtimecommunicationswhencomparedwiththe delaysthatemailcanencounter,instantmessagingsystemshave becomeanimportanttoolforconductingjointcommunications, coordinationandinformationsharingformilitarypersonnel. TrIM,alsoknownastheCoalitionChatLine(CCL),providesrealtime, multilingualtexttranslationinaninstantmessagingenvironmentin whichtwoormoreparticipantsareabletointeractbyreadingandtyping usingtheirownpreferredlanguages.Thesystemtranslateseachusers messagesintothelanguageoftheotherparticipantsanddisplaysboth thesourcelanguageandtargetlanguageversionsofeachmessage. TrIMstranslationservicesareprovidedbytheCyberTranssystem,which providesacommoninterfaceforvariouscommercialandresearchtext translationsystems.TrIMcurrentlysupportslanguagesincludingEnglish, Spanish,Portuguese,German,Italian,French,Korean,Thai,Japanese, Chinese(simplified),Russian,Polish,Ukrainian,Arabic,andHebrew.The CCLhasalreadybeentestedinmanymultinationalmilitaryexercises.Ina recentoperationaldeploymentofCCLtoIraq,potentialmedicalapplica tionsbecameapparent.Accordingtothereport,theCCLenabledpre cise,rapidreportingandconsultationonillnessandinjury,aswellas locationandavailabilityofmedicalresources,instantaneouslyinthirteen languages(CampbellandHillenbrand,2005).Othersuccessfuldemon strationshavebeenreported.

EmbeddedMachineTranslationinInstantMessaging343 TheTRiM(TranslingualInstantMessaging)language applicationtoolisanexampleofamarvelousapplicationthat crossesthelanguagebarrier.AstheFireSupportCoordinator (FSC)forJWID03,responsibleforallgroundartillery,navalgun andcloseairsupport,IwasrequiredtoworkwiththeSpanish Army.TRiMeffectivelyenabledmetowritemymessageona whiteboardandsenditstraighttoSpain.Theyreceiveitin SpanishandcanthenrespondtomeandIreceiveitin English."(Joyce,2003) Itwouldseemthatcrosslanguageinstantmessagingisatechnology thatisrapidlybeingadoptedbytheArmyandotherbranchesofthemili taryforcoalitionoperations.Thesemilitaryexercisesareagoodwayof exploringhowwelltheTrIMintegrateswithothercollaborativetechnolo gies.Itdemonstratestheappealofthetoolbutdoesnotnecessarily allowthetypeofdetailedanalysisnecessarytodiscoverwaystoimprove thetechnologytomakeitmoreuseful.Milleret.al.(2001)suggestthat translingualsharingofinformationcannothappenuntilwetodevelop toolsforcapturing,analyzing,andenablingtranslingualinformation sharing.Further,theysuggestthatthetranslationcapabilitymustbe morereflectiveofthelanguagestyleandterminologythatisactually requiredforcollaborative,coalitionwork. Weareworkingtoanswerthecriticalandfundamentalissues regardingtranslationcapabilityinwhatappearstobeanimportantnew applicationforMT. DEVELOPINGATASKBASEDEVALUATIONMETHOD Wehavebeeniterativelydevelopingamethodthatwouldbesensi tivetotheevaluationofembeddedmachinetranslationinaninstant messagingapplication.Wewantedtostudytheperformanceofmachine translationtechnologyinthecontextofrealtasks.Incontrast,automatic evaluationmethods,suchasthepopularBLEUmethod(Papinenietal., 2002),areusedtocomparesystemsagainsteachotherandarespecifi callydesignedtoremovethepotentialbiasataskcontextmayimposeon theevaluation.Webelieveitisimportanttotesttheperformanceofa systeminwhichthetranslationsaretobeunderstoodbypeoplewho actuallyneedtheinformationtoaccomplishtheirgoals.Consequently, wehaveconductedaseriesofstudiesinwhichpairsofparticipantsusing differentlanguagesareaskedtoworktogetherusingaCLIMsystemto accomplishatask.Thetasksgiventotheparticipantshaveevolvedfrom

344OGDEN studytostudybutallrequirethatparticipantsshareinformationthat eachhasinordertocompletethetask.Tasksweredesignedtoinvoke realisticconversationsbetweenparticipants. Todate,wehaveconductedseveninformationsharingexercisesto measurehowmuchinformationhumanparticipantscantransferthrough theTrIMsystem.Completedetailsofallofthesestudiesincludingall stimulusmaterials,results,andannotatedmessagelogs,willbemade availableonlinebycontactingtheauthor. Thegoaloftheevaluationwastoassessusabilityofthesystemas determinedby: Theabilityofpartnerstoshareandunderstandinformation definedbytheinformationsharingtask. Thequalityoftranslationasmeasuredbytheparticipants ratings. Linguisticanalysisoftranslationerrorsanddialogrecovery moves. Taskmeasuressuchastimetocomplete,accuracy,andnumber ofmessagestocomplete. GeneralMethod Participantsweretestedinpairs.OnenativeEnglishspeakerwas pairedwithanonEnglishnative.WehaveusednativespeakersofJapa nese,Korean,ChineseandSpanish.AgroupofEnglishEnglishpairswere usuallyincludedforbaselinecomparisons.NonEnglishparticipantswere recruitedfromapoolofinternationalstudentsandtheirspouseswith therestrictionthattheywerewellpracticedinkeyboardtextentryin theirnativelanguage.Becausetheseparticipantsalsotypicallyhavea highfluencyinEnglish,averyusefulfeatureoftheTrIMsystem,which showstheEnglishsideoftheconversation,hadtobehiddenfromthem. Eachparticipantwasgivenataskthatrequiressharingofinformation betweenthepartners.Theyweregivenpartialinformationandneeded tocommunicatewiththeirpartnertocompletethetask.Throughoutthe dialog,eachuserwasexpectedtoobtainmissinginformationfromthe partner.Eachpartnerhaddifferentbitsmissingsotheconversationpart nershadtotaketurnsandnegotiatewhohadtheroleofinformation providerorreceiver. Inourinitialstudies,participantsweregiventablesofinformation withemptycellsmarkedbyquestionmarksandtheirtaskwastoask theirpartnerforinformationthatcouldfillinthemissingcells.Aproblem withthisapproachwasthatparticipantssometimesadoptedveryidio

EmbeddedMachineTranslationinInstantMessaging345 syncraticlanguagestylesheavilyinfluencedbythetaskstructure.For example,aparticipantmayenterintotheirchatwindowawholerowof theirtablecompletewiththequestionmarks,ratherthanconstructinga naturallanguagequestion.Invokingrealisticlanguagehaslongbeenan issueintheevaluationofnaturallanguageinterfaces(e.g.Ogden&Ber nick,1997). Clearly,taskcharacteristicsarepowerfuldeterminatesofcommuni cationstrategies.Consequently,oneofourinitialmajorfindingsisthat thereareaclassoftasksforwhichCLIMmaynotbeagoodsolution. Tasksthatinvokestructuredmessageswillprobablybebestservedby structuredinterfaceswithcustom,humantranslated,versionsrather thanwiththeopenendedcapabilitiesofunstructuredcrosslanguage chatapplicationsprovidedbygeneralMTsoftware.Indeedasubsequent studyusingamilitaryoperationplanningtaskshowedthatparticipants adoptedasimilar,structuredmessagecommunicationstyle,againsug gestingtherearetypesoftasksnotsuitedforCLIM. LogisticsTask Inournextiteration,wesolvedtheproblemthattaskstructurehad onconversationstylesbypresentingalogisticstaskdevelopedincollabo rationwithourArmyResearchLaboratorypartnersinvolvedwithmulti nationalteamplanningatFtHuachuca,Arizona.Participantsweregiven mapswithinformationaboutsuppliesandlocations.Eachparticipant hadhalfoftheinformationwhiletheirpartnerhadthesamemapbut withtheotherhalfoftheinformation.Theyweregivenlistsofsupplies neededandaskedtouseTrIMtonegotiatethebestsupplysources. ThesetaskshaveresultedingeneratingverygoodIMsessionswhich madepossiblemanyoftheresultshighlightedinthenextsection. However,analysisoftheIMdialogsisextremelychallengingandtime consumingpartiallyduetotheopenendednatureofthetask.Inpost hocanalysisofthedialogs,itissometimesdifficulttounderstandthe currentconversationalgoalsoftheparticipants.Webelieveopenended dialogisindeedmorerealisticanddesirable,butwewantedtobetter understandparticipantsproblemsandperformancebyintroducing morestructuretothetaskenvironment.Tothisend,wehavekeptthe fundamentalsofthelogisticmaptaskbuthaveintroducedadiscretetrial method.

346OGDEN

Figure19.1 Discretelogistictaskwebforms InFigure19.1,onecanseehoweachtrialispresentedtothepartici pants.Underthecontrolofawebapplication,eachparticipantsees eitherthetaskscreenorthesolutionscreen.Eachtaskdisplaysonelogis tictaskwithasingleitemandquantity.Bothparticipantsfillouttheform whentheyhaveagreeduponthebestsolution.Thismethodprovides directknowledgeoftheconversationalgoalsoftheparticipants,aswell asimportanttaskmeasuressuchassolutiontimeandaccuracy.Perhaps mostimportantly,sincebothparticipantshavetoexplicitlyindicatetheir understandingofthesolution,wehaveadirectmeasureofshared understandingindicatedbythenumberofanswersmatchingtheirpart ner.ThismethodwasusedintheKoreancomparisonstudydescribed below. GeneralResults MachineTranslationsoftwaretypicallysuffersfromasetofknown problems.Forexample,syntacticandlexicalambiguityinthesourceand targetlanguagescanleadtopoortranslationsduetowordselection errors.Sofar,oneofthemostpervasiveoffindingsinourinvestigations istheobservationthatthesepoorlytranslatedmessagesslowpartici pantsdown,butdonotpreventthemfromcommunicatingandsharing taskrelevantinformation.Participantsengageindialogstocorrectmis understandings(repairdialogs)andotherstrategiesthatlargelydepend onknowledgeofthetaskdomain.

EmbeddedMachineTranslationinInstantMessaging347 Table19.1Typicalfindingfortheinformationsharingtask LanguagePair ChineseEnglish JapaneseEnglish KoreanEnglish SpanishEnglish EnglishEnglish TaskTime(hrs) 1.4 1.35 1.2 1 .8 MapErrors 6 6 8 2 1 MessageCount 162 199 191 160 96

Theresultshavethusfarshownthattheteamscontainingnon Englishmemberstakeconsiderablylongertocompletesomewhatlessof thetaskthantheEnglishonlyteams.Thus,thetranslationsoftware,as expected,isnotperfect.However,mostofeachtaskwascompletedby thenonEnglishonlypairs.Inonestudy,thetaskcompletionratefor English,Chinese,JapaneseandKorean,was1.0,.98,.88,.84,respec tively.Table19.1showstheresultsofatypicalstudy.Translationslows butdoesnotpreventthetaskfrombeingcompleted.Theparticipants relyingonMTtocommunicatetooklonger,hadafewmoretaskerrors, andusedmanymoremessagestocompleteatask.Manyoftheseaddi tionalmessagesaremetacomments.Metacommentsaremessages abouttheconversationtoclarify,repairorfacilitatetheconversation, andmadeupalargerportionofthetranslationparticipantsmessages comparedtoEnglishonlyparticipantsmessages(22%vs.10%).Although thetranslationswerenotperfect,humanjudgedtranslationquality showedthattranslationsareadequatemorethan80%ofthetime.In pairscontainingnonEnglishsides,thetranslationqualitydependsonthe languagepairandthesyntacticalcomplexity.Thecomplexityofthesyn taxofthesentencesalsoaffectsthequalityoftranslations.Grammati callysimplesentenceshavebettertranslationsthancomplexsentences. Also,useofpunctuationinsentenceswillsometimesimprovethetrans lation. Figure19.2showsasampleconversationbetweenanEnglishpartici pantandaKoreanparticipantusingtheTrIMsysteminaninformation sharingtask,andillustratessomeoftheproblemsencounteredbythe participantsaswellasillustratinghowthetranslationsoftwarecouldbe improvedforthistask.Eachlineinthewindowrepresentsonemessage. Thesourcelanguageisontheleft,andtheresultingmachinetranslation isontheright.(Thisorderisshownhereforexplanation,butinthe actualstudytheparticipantwouldseeonlytheresponsesintheirown language).

348OGDEN Onthefirstlineofthesampleconversation,theEnglishparticipantis askingforclarificationbyaskingtheKoreanparticipanttorepeatwhat wasenteredearlier.TheKoreanOnthefirstlineofthesampleconversa tion,theEnglishparticipantisaskingforclarificationbyaskingthe Koreanparticipanttorepeatwhatwasenteredearlier.TheKoreantypes amessagethatmeans,Thatisallbutthewordallgetstranslatedtoa possiblesynonym,universal,whichunfortunatelyismeaninglessinthis contOnthefirstlineofthesampleconversation,theEnglishparticipantis askingforclarificationbyaskingtheKoreanparticipanttorepeatwhat wasenteredearlier.TheKoreantypesamessagethatmeans,Thatisall butthewordallgetstranslatedtoapossiblesynonym,universal, whichunfortunatelyismeaninglessinthiscontext.TheEnglishpartici pantagainasksforaclarificationofthelastmessage,butthistimethe word,universalisnottranslatedbacktotheoriginalKoreanword,all buttoanotherpotentialsynonyminKoreanwhichmeans,general.This isaresultoftheunderlyingMTtechnologynotbeingsensitivetothepre cedingcontextorsupportingreversiblelanguagetranslation.Typically, MTsystemsarenotreversible,thatis,thetranslationdirectionsbetween twolanguagesproducedifferentresults.TheEnglishtoKoreansystemis mostlikelyaseparatesystemfromtheKoreantoEnglishsystem. NowtheKoreanparticipantisconfusedandtypesaKoreanreplythat roughlymeansthathedidntsayuniversalbutinthiscase,averypoor translationisproduced.However,knowingthatthetranslationsystemis probablytheissueandthathisoriginalanswerwasnotunderstood,he triestorepairtheconversationbyreplyingagaintotheoriginalquestion inthelastlineofthesampleinFigure19.2.Thisgetstheconversation backontrackand,perhapsamazingly,thepartnerseventuallyworkout themeaningandaccomplishtheirgoal.

EmbeddedMachineTranslationinInstantMessaging349

Figure19.2 AsampleEnglishKoreanconversation ThisexamplehighlightsoneimportantlessonfordevelopersofMT softwareforinteractiveapplications.Providingreversibilitysothata translationusercouldsignalwhichwordsarepoorlytranslatedinthe originalsourcewouldbeagreatbenefit.Thiswouldenabletheauthorof thetexttomakespecificeditstoimprovethetranslation.Thiscapability hasnottraditionallybeenimportantinMTsystemswhichhavebeen builtfordocumenttranslationstasksbecause,usually,theauthorofthe textisnotavailable.ButintheinteractiveworldofCLIM,theauthoris immediatelyavailableandisengagedincommunicatingintendedmean ing,muchthesameaspeopleengagedinnormal,samelanguagecon versationwherenegotiationsofmeaningarecommon. ThisobservationalsohighlightswhyevenimperfectMTissosuccess fulinCLIMapplicationsandmaybeanevenbetterapplicationforMT

350OGDEN thandocumenttranslation.Becausethereishumanintelligenceonboth sidesofthetranslatedmessage,itisnotascriticalforthemachineto alwaysrepresentthetruemeaning.Peopleareusedtocommunicating thoughsometimesnoisychannelsandhavelearnedtonegotiatewhen thingsdontmakesense.Theideathattherecanbemultipleattemptsat agoodtranslationisapowerfulone. IMPROVINGTHEINTERACTIONWITHMETABUTTONS Thefindingthatmuchoftheinteractioninourearlierstudieswas theresultofrepairdialogsledtoacloserexaminationofthisresulttotry tounderstandandtopossiblyfixtheproblemsbyenhancingtheuser interface.First,weclassifiedallthemetacommentsintooneofseveral semanticcategories.Theresultofthisanalysisisshownin Table19.2. Table19.2Analysisofthemetacomments Occurrences (total/unique) Chinese 54/8 16/6 92/13 27/24 40/32 11/9 8/5 Korean 107/26 29/9 85/11 22/20 94/71 8/7 4/3 PercentPoorly Translated Chinese 15 19 19 11 32 9 37 Korean 18 4 9 23 32 0 50

Semantic Category Yes No OK Ready What? Understand? Wait

Thetableshowsthesevencategoriesthataccountedforthemost frequentlyoccurringmetamessages.Foreachsemanticcategory,the tableshowsthetotalanduniquenumberofoccurrencesobserved.Thus, forexample,weobservedChineseparticipantssendingareadymes sage27timesandKoreanparticipants22times.Ofthe27Chineseoccur rences,24wereuniquemessagesand20messageswereuniquein Korean.Thus,therewerealargenumberofdifferentwaysthesame

EmbeddedMachineTranslationinInstantMessaging351 messagewasexpressed.Thelasttwocolumnsofthetableshowthepor tionofthosemessagesthatwerejudgedtohavebeenpoorlytranslated. Thus,againfortheReadymessages,11percentoftheChineseand23 percentoftheKoreanmessageswerepoorlytranslated. Giventhelargenumbersofmetacommentsaccountedforbythe sevenmetacategorieslistedinthetableandthefairlylargeportionof poorlytranslatedversionsofthese,wehypothesizedthatifthesemes sagescouldbesentinastandard,andconsistentlywelltranslatedman ner,communicationusingthesystemwouldbeimproved.Thuswe constructedanewversionoftheTrIMinterfaceaddingmetabuttons whichcouldbeusedtosendthesesevenstandardmessagesandcon ductedastudycomparingthetaskperformancewithandwithoutthese buttons.

Figure19.3 ModifiedTrIMinterfaceshowingstandardmetabuttonsfor English(top)andKorean(bottom). ThemodifiedTrIMinterfaceisshowninFigure19.3.Buttonswere addedwithlabelsforoneofthesevenmajorcategoriesofmetames sagesasdiscoveredinourempiricalanalysis.Userscouldclickabutton ratherthantypeorsendamessageandthecorrectlytranslatedmessage

352OGDEN wouldbesenttotheirIMpartner.TheKoreanexampleshowninFigure 19.3showsthatalistofsemanticallyrelatedmessagespopsupwhenthe mousepointerhoversaboveoneofthebuttons(therightmostinthe example).Thesebuttontipsweretohelpusersunderstandthetypeof meaningforwhichthebuttoncouldbeused.Forexample,thepopup listfortheEnglishReadybuttoncontainedthephases;Imready, Areyouready?Letsstart,andGo. MethodandResults Twelvepairsofparticipantswereaskedtoworktogethertofilllogis ticrequestsusingtheproceduredescribedearlier.Eightofthegroups pairedanEnglishlanguageuserwithaKoreanlanguageuserandfourof thegroupspairedanEnglishuserwithaChineselanguageuser.Within eachofthesegroups,halfusedtheTrIMsystemwiththeadditional metabuttonsandhalfusedthenormalTrIMsystem. Table19.3Resultsofthemetabuttonstudy(numbersareaveragefor eachgroup). Without MetaButtons Metamessages Taskmessages Tasktime 31 65 69min With MetaButtons 47 54 52min

Weexpectedthatinthenobuttonconditionstherewouldbemore repairmovesandmetacomments,duetothelargermarginoferrorand misunderstanding.Therewasnoguaranteethatanywordsorphrases wouldbeunderstood.Inthebuttoncondition,wherethebuttonsare programmedtocorrectlytranslateacertainphrase(i.e.,ready,under stand,yes,no),weexpectedtofindfewerrepairmovessincethere wouldbefewermisunderstandings.However,wefoundjusttheoppo site.AscanbeseeninTable19.3,participantsinthebuttoncondition hadanaverageof16moremetamessagespergroupthanthepartici pantsinthenobuttoncondition,t(10)=2.36,p<.05. Whenthebuttonwasavailable,theparticipantstendedtodevotea greaterpercentageoftheirmessagestometacommentsandalesser

EmbeddedMachineTranslationinInstantMessaging353 percentagetotaskrelatedmessagesthanwhenthebuttonwasnotavail able.Sowhenthebuttonswereavailabletheywereusedfrequently. Thequestioniswhethertheavailabilityanduseofthebuttonshelp orhinderworkonthetask?Theaveragetaskrelatedmessagecountwas lowerforthegroupsusingthemetabuttonwhichmayindicatemore efficienttaskperformancebutthisdifferenceisnotsignificant.However, theaveragetimetocompletetheexperimentforthebuttongroupis only52minutes,whereasittookthenobuttongroupabout69minutes, andthismightbeamorereliabledifference,t(10)=1.66,p=.06(onetail test).Soitwouldseemthattheadditionofthemetabuttonshadaposi tiveeffectonparticipantsmethodofsolvingthecommunicationprob lemsandworkingtogetheronthetask.Italsoseemsthatitmayhave helpedparticipantscommunicateimportantinformationfasterandper hapsbetter. COMPARINGTWOTRANSLATIONSYSTEMS Method Totestthesensitivityofourevaluationmethodwewereabletouse TrIMwithtwodifferentKoreanEnglishtranslationsystemsprovidedto usbyMITRE.Inourearlierstudies,taskresultswiththeKoreantransla tionsweretypicallyworsethanwithanyoftheotherlanguagepairs.A newandbettertranslationsystemwasprovidedandwetestedboth withourdiscretelogistictrialmethod.SixKoreanEnglishpairsofpartici pantswereassignedtoeachtranslationsystem.SixEnglishEnglishpairs werealsoincludedforcomparison. Table19.4Taskperformancecomparisons Average SolutionTime Seconds EnglishControl 109 Korean1 155 Korean2 179 Correct solutions N=18 16.5 16.2 14.5 16.0 17.3 15.0 Matching solutions

354OGDEN Results TaskPerformance.Table19.4showstheaveragetaskmeasuresfor eachgroupofthesixpairsofparticipantsinthestudy.Eventhoughthere wereonlysixpairsineachgroup,ananalysisofvarianceshowsthattask solutiontimesweresignificantlydifferentbetweenthegroups,F(2,15)= 5.38,p=.017.Thisisconsistentwithallofourpreviousstudies.As expected,thegroupswhomustcommunicateusingautomatic translationsoftwaretakelongertocompletetheirtasksthanthepairs usingEnglishonly.Despitethis,theothermeasureoftaskperformance, theaccuracymeasure,isnotsignificantlydifferentbetweenthegroups, F(2,15)=.99,p=.39.Thisisalsoconsistentwithourearlierstudies.Using embeddedMTslowstaskperformancebutdoesntpreventtasksfrom beingcompletedsuccessfully. WhencomparingthetwoKoreantranslationsystems,thepoorest performancesseemtohavecomefromthepairsusingKoreanSystem2. Ourmethodgivesusyetanothergoodmeasureofhowwellthetransla tionssystemsarebeingusedtocommunicate.Sinceeachmemberofthe pairmustindependentlyenterthetasksolution,acountofthenumber oftimestheirsolutionsagreeisagoodmeasureoftheirsharedunder standing.Givenourprimaryinterestintestingthesensitivityofourmea suretocomparetranslationsystems,wecomparedtheaveragenumber ofmatchingsolutionsforjustthosepairsusingthetwotranslationsys temsandfoundthatTranslationSystem2wassignificantlyworse,t(10)= 2.36,p<.05.TheparticipantsusingTranslationSystem2hadanaverage of2.3fewermatchingsolutionsthandidtheparticipantsusingTransla tionSystem1.Thisfindingvalidatesthatthetaskmeasuresusedinour studiesaresensitivetotranslationsystemdifferences. HumanJudgmentsofTranslationQuality.ThefindingthatTransla tionSystem2isapoorersystemthanTranslationSystem1matchedsub jectiveexperiencegainedthroughcasualinteractionwiththetwo systems.Wewantedtoseehowamoretraditionalmeasureoftransla tionqualitycomparedwiththetaskbasedmeasures.Twopeoplefluent inKoreanandEnglishindependentlyassignedoneofthreecategoriesto eachtranslatedmessagegeneratedbybothsystemsfortheinstantmes sagesenteredbyourtaskparticipants.Agoodratingwasgivenfor translationsthatcapturedmostoralloftheintendedmeaningandpro ducedafluenttargetlanguagemessage.Anadequateratingwasgiven fortranslationsthatcapturedmuchofthemeaningbuthadnonnative fluency.Apoorratingwasgiventoincomprehensibletranslations.

EmbeddedMachineTranslationinInstantMessaging355 Table19.5Humanjudgmentsoftranslationquality KoreanSystem1 KoreanSystem2 Good 47% 49% Adequate 40% 40% Poor 13% 11%

Eachofthetworatersindependentlyjudgedall3,469messages enteredbythe12pairsofparticipantsandthenmettogethertobuilda consensus.Theydecidedtochange568judgmentswhichreflectan agreementrateof.84.Table19.5showsthattheresultingqualityjudg mentsforeachKoreantranslationsystemwerealmostequal.Thus,sur prisingly,KoreanSystem2,whichresultedinpoorertaskperformance, hadthesameproportionsofqualityratingsasKoreanSystem1.Itshould benotedthatatotalof349moremessageswereenteredbythegroups usingKoreanSystem2,andthushadlargernumbersofbothgood, adequateandpoorjudgments.Needingmoremessagestocomplete thetaskisanotherindicatorofpoorerperformanceoftheKoreanSystem 2andsuggeststhatparticipantswereabletolearnandadapttheir instantmessagingtocircumventthepoorertranslationsystem.Sothe costofaworsetranslationcapabilityisnotshowninthehumanratings, butitisshowninthetaskmeasures. IMPLICATIONSFORTHEFUTUREARMY OneofthemoreimportantfindingsofthisworksuggeststhatMT softwaredoesnthavetobeperfecttobeveryusefulforArmyinstant messagingapplications.Thisislikelyaresultofhavingintelligentlan guageusersonbothsidesoftheMTsoftware.However,whilewehave observedmanysuccessfulrepairdialogs,thereisnothingintheunderly ingtechnologyofthesystemoroftheuserinterfacewhichsupportsthis repairactivity.Althoughourinitialevaluationsofinterfaceenhance ments,suchasmetabuttons,showpromise,moreeffectivecollabora tiontoolsareneededtominimizepotentialcommunicationbreakdowns amongdistributedmultilingualteammembers. Weseemanyexamplesofuserstryingtoclarifymisunderstood wordsorpartsofthemessagebyhighlightingorechoingbackjustthose parts.Unfortunatelythisstrategyusuallyleadstomoremisunderstand ingsbecausetheechoedpartsaretranslatedbacktotheoriginatordif ferently.ThisisaproblemwiththeasymmetricaldesignofmostMT systems.Thesystemgoingfromonelanguageisdifferentthantheone goingbacktooriginal.Ifamechanismcouldbeimplementedthatwould

356OGDEN allowfeedbacktobegiventotheoriginatorofthemessageaboutparts thatwereincorrectlytranslatedwepredictagreatbenefit. Another,moreavailablefeatureofsomeMTsystemsistheabilityto providealternatetranslations.CurrentlyTrIMshowsonlyonepossible translation.Wepredictabenefitifuserscouldrequestalternatetransla tionsdirectlyfromtheMTsoftware,ratherthansolelyfromtheirpart ner.Wecurrentlyareconductingastudywithanew,custombuiltweb basedCLIMsystemthatusesfreelyavailableonlinetranslationsitesto generatemorethanonetranslationforeachusermessage.Theinterface allowstheusertoselectalternativetranslationswhentheydontunder standthecurrenttranslations.Wearecomparingtaskperformanceusing thiscapabilitytotheperformancewhenusershaveonlyonetranslation source. Wecanexpectthegreatestsuccessintasksituationswherethereisa greatdealofsharedunderstandingbetweenthepartnersaboutthecon versationalpossibilitiesprescribedbythetasksituation.Inthefuture,we wanttotestthishypothesisbycomparingtasksituationsinwhichthe contentoftheconversationiscloselyconstrained,totasksinwhichthe contentcanbemoreopenended.Inourcurrentlogisticstask,partici pantsknowallpossiblesupplyitems.Wecanmodifyourtasktoincludea conditioninwhichtheyarelesscertainaboutwhatwillbeincludedin thelogisticsrequests.Wewillanalyzedifferencesinuserrepairstrategies whenlessinformationcanbeobtainedforthetasksituationandmore mustbeobtainedfromtheirpartner. Fromtheoutset,wehavebeenestablishingmetricswhichcouldbe usedtoevaluateMTtechnologiescontainedinembeddedtasks.Wenow haveawelltestedmethodtodothenecessarylinguisticanalysisand taskmeasurestotestfeaturesofMTsoftwareandCLIMapplicationsthat supportcollaborativedialogs. ACKNOWLEDGMENTS TheauthorwouldliketothankJohnWarner(ARL/HRED)formallyat Ft.Huachuca,Arizonaforhisinstrumentalinvolvementinallaspectsof theevaluationandinparticulardevelopingarealisticlogisticstask.The authorwouldalsoliketothankthefollowinglistofcollegesatNewMex icoStateUniversitywhohadsignificantrolesintheresearch:SieunAn, EmilyChaffin,YukiIshikawa,WanyingJin,JongHwanKim,YosipKim, RobertoMontalvo,JeffSorgeandRonZacharski.Finally,wewouldliketo thankRodHollandandGalenWilliamsonofTheMITRECorporationfor providingaccessandsupportfortheTrIMsystem.

EmbeddedMachineTranslationinInstantMessaging357

REFERENCES
KishorePapineni,SalimRoukos,ToddWard,andWeiJingZhu.(2002).Bleu:a MethodforAutomaticEvaluationofMachineTranslation.InProceedingsofthe40th AnnualMeetingoftheAssociationforComputationalLinguistics. JamesR.CampbellandChrisHillenbrand.(2005).CCLforOperationalMedical Support,MilitaryMedicalTechnologyOnlineArchives,Jan26,2005inVolume:9Issue:1 Retrievedfromhttp://www.militarymedicaltechnology.com/article.cfm?DocID=784 JohnR.Joyce.(2003).CoalitionInteroperabilityTestedatDahlgrenDuringJWID 2003.CHIPSTheDepartmentoftheNavyInformationTechnologyMagazine.Fall2003, Retrievedfromhttp://www.chips.navy.mil/archives/03_fall/PDF/JWID_2003.pdf KeithJ.Miller,FlorenceReeder,LynetteHirschmanr,DavidD.Palmer.(2001). MultilingualProcessingforOperationalUsers.MITRETechnicalReport,Retrievedfrom http://www.mitre.org/work/tech_papers/tech_papers_01/miller_multilingual/ index.html Ogden,W.C.&BernickP.(1997)."Usingnaturallanguageinterfaces."InHelender, M.,LandauerT.,&PrabhuP.(Eds.),HandbookofHumanComputerInteraction,2nd Edition,Amsterdam:NorthHolland.

358OGDEN

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T A N D L A U R E L A L L E N D E R

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T AND LAUREL ALLENDER

This book, produced by the Partners of the Army Research Laboratory Advanced Decision Architectures Collaborative Technology Alliance, also serves as the Final Report for DAAD19-01-2-0009.

SECTIONIV.COMMUNICATINGINFORMATIONACCROSSTHETEAM

SECTIONIV COMMUNICATINGINFORMATIONACROSSTHETEAM

324SECTIONIV

Chapter

20

20. TOOLSTOSUPPORTADHOCTEAMS

LauraStrater SATechnologies,Marietta,GA SandroScielzo,Ph.D. SATechnologies,Marietta,GA MichelleLenoxTinsley SATechnologies,Marietta,GA CherylA.Bolstad,Ph.D. SATechnologies,Marietta,GA HaydeeM.Cuevas,Ph.D. SATechnologies,Marietta,GA MicaEndsley,Ph.D. SATechnologies,Marietta,GA

INTRODUCTION FutureForceoperationsrelyonrapidlyformingadhocteamsthat aremobile,versatile,anddistributedintime,space,andpurpose,and canbringtobearmultipleperspectivesonaproblem.Membersarecon venedforaspecificpurposeandchosenprimarilyfortheirexpertise, drawingfrommultipleArmyspecialtiesandechelonsaswellasJoint Forcesormultinationalentities.Consequently,teammembersoften havedifferentbackgroundsandexperiences,withfewsharedexperi encesattheonsetoftheirmission.Theuniquenatureoftheseteams, therefore,posessignificantchallengestobothmilitarycommandersand teammemberswhomustrelyononeanotherforcriticalinformationto 359

360STRATERETAL. quicklydevelopsufficientTeamandSharedsituationawareness(SA)for decisionmakingandachievingoperationalgoals. InvestigatingAdHocTeams ResearchersfromSATechnologiesandARLLeavenworthteamedto investigatethefactorsthatinfluenceadhocteamperformanceduring theU.S.ArmysUrbanResolve2015humanintheloopdistributed experimentatFortLeavenworth.Findingsindicatedasignificantneedfor bettertoolstosupporttheuniquerequirementsofadhocteams.Nota bly,themostsuccessfulteamsandteammemberswerethosewhowere inpositionswhereinformationsharingwasaprimarycomponentoftheir position,suchastheIntelligenceandOperationsgroupsandtheBattle Captainroles.Personnelintheserolesservednotmerelyasinformation conduits,butratherprovedtobevitalresourcesthathelpedfocusthose inotherspecialtiesandrolesonthecriticalaspectsofthesituation.This findingemphasizedtheneedfortoolsthatsupportinformationsharing amongpersonnelwithdiversebackgrounds,experienceandobjectives. Inordertodeveloptoolstosupportthesituationawarenessneedsof adhocteams,weneededtoselectaspecifictypeofadhocteamforcon tinuedinvestigation.PersonnelattheBattleCommandKnowledgeSys tems(BCKS)groupatFt.Leavenworthinvitedustoworkwiththemto meettheneedsofaparticulargroupofadhocteamsthatwereamajor focusoftheirefforts,militarytransitionteams(MiTT)deployedtoIraq andAfghanistan.MiTTandsimilargroups,suchasbordertransition teamsandpolicetransitionteams,areacriticalcomponentoftheU.S. Armysstrategyinwinningconflictsagainstnonstateentitiessuchasal QaedaandtheTaliban.MiTT,positionednearhostnationforcesoperat ingbasesasadvisoryteams,arechargedwiththemissionofsupporting localforceseffortsandabilitytodevelopasustainingforcecapableof securingtheirowninternaldefense,apillarofirregularwarfare(Depart mentofDefense,2007). Infact,irregularwarfareaspracticedtodaydependslessonmajor combatoperations,andmoreoninfluencingthepopulaceandempow eringthehostnationtosecuretheirowndefenseagainstinsurgency efforts.Forexample,amajorshiftinthefocusofOperationIraqiFree domresultedwiththedecisiontotransitionfromprovidingsecurity directlytopreparingIraqiSecurityForcestotakeoverthejobofsecuring theirowncountry,whichsubsequentlyrequiredthecreationofthevery adhoctransitionteamsstudiedhere(e.g.,Nagl,2008).Anunderstanding ofthehumanterrainandmaintainingcontinuityweredeterminedtobe

ToolstoSupportAdHocTeams361 criticaltothemissionsuccessoftheseadvisoryteams(FoxandStowell 2008;Petraeus2008). ChallengesFacingAdHocMiTT UnlikeSpecialForcesteamsorstandingunitsofsoldiers,adhoc teamsareaffectedbyanumberoffactorsthatmaynegativelyimpact theirfunctioning.Sweller,VanMerrinboer,andPaas(1998)usedan intrinsicandextrinsicloadfactorcategorizationtodeterminethecogni tivedemandsimposedbyspecifictrainingcontent.Abstractingthis notiontoadhocteamoperations,adistinctionbetweenintrinsicand extrinsicfactorsaffordsapragmaticwaytocategorizefactorslikelyto impactteamfunctioning.Specifically,intrinsicandextrinsiccharacteris ticsmayimposeteamdemandsthatmustbesuccessfullymitigatedto achieveoperationalsuccess. Intrinsiccharacteristics(i.e.,characteristicsinherenttotheteam) suchasteamsize,teammembersexperience,missiondifficulty,and trainingquality,cansignificantlyimpactthecomplexityofteamopera tions.Adhocteamstendtobedistributedandmaylackcolocatedtrain ing,whichcansignificantlyimpedethedevelopmentofteamtrust.In turn,alackofcohesionmayarise,inpartduetothelackofcommon backgroundandknowledge.Mergingpersonnelfromdiversespecialties, echelons,servicebranches,orcountriesintoanefficientteamcanpose significantchallengesforthecommander.Theverydiversitythatenables teamsynergiescanalsobeadetrimenttoteamoperations. Adhocteamsarealsoinfluencedbyextrinsicfactorsthatreflectout sideinfluences(e.g.,logistics,humanterrain,technologicaldevices,cul turalconstraints,etc.).Adhocteamsoftenhaveilldefinedorabstract goals,suchasMiTTwhomusteffectivelysupportthehostnationforces infindingalocalsolutionforalocalproblem.Inaddition,adhocteams maybedeployedinanunfamiliarenvironment,oftenwithunclearand lessefficienthierarchiesandlinesofcommunications.Therecanbelan guageandothercommunicationsbarriersbetweenpersonnelofdiffer entbackgrounds.EvenUSmilitarypersonnelfromdifferentbranchesof servicefacechallengesincommunicationsduetoterminologydiffer ences. Inadditiontotheseintrinsicandextrinsicteamcharacteristics,ad hocteamsfacesignificantchallengestotheirteamsituationawareness (SA).Insimpleterms,situationawarenessinvolvesknowingwhatisgoing onaroundyou,understandingwhatitmeansintermsofyourgoalsand objectives,andanticipatingrelevantnearfutureevents.Awidely

362STRATERETAL. accepteddefinitionofSAistheperceptionoftheelementsintheenvi ronmentwithinavolumeoftimeandspace,thecomprehensionoftheir meaningandtheprojectionoftheirstatusinthenearfuture(Endsley 1988,p.97). Adhocteamsfaceparticularchallengesindevelopingandmaintain ingSAdueinlargeparttothetemporaltimelinesuponwhichtheyoper ate.Theseteamsoftendonotformasawhole,operate,andthen disbandasawhole.Rather,aneedmaybeidentifiedandpersonnelare taskedtotheteamtoaddresstheneed.Overtime,additionalskilland experiencegapsarediscovered;sonewteammembersareaddedto coverthesegaps.Similarly,teammembersoftengainadhocteamduties inadditiontotheirpermanentassignment,requiringmultitaskingand taskswitchingbetweenmultipleassignments.Thiscanleadtodisjointed teamoperations,requiringanupdatingprocesstodeterminethemost currentinformationwhenswitchingbacktotheadhocteamtasks. Often,permanentdutiescantakepriorityoveradhocteamduties,and maynecessitateleavingtheadhocteam.Withhigherteamturnoverand therequirementsformultitaskingandtaskswitching,thedynamic knowledgeofagivensituationmaybemoredifficulttoacquireormay bemissingifaknowledgeableteammemberleavestheteamorisnot availableatacriticaltime.Asaresult,theteamsSAsuffers,potentially negativelyimpactingmissioncriticalinformationsharing(e.g.,Endsley& Jones,2001;Nofi,2002).Managinginformationflowandsharinginad hocteamsisasignificantchallengeforcommandersoftheseteams,and thelackofefficientinformationsharingcannegativelyaffectteamSA andperformance(e.g.,Endsley,Bolt,&Jones,2003). Overcomingtheintrinsicandextrinsicfactorschallengingadhoc teamsoperationsrequirestoolstosupportteammembersSA,thereby increasingteameffectiveness.Thegoalofthecurrentresearcheffortis toaddressthesechallengesfortheuniqueneedsofMiTT. COHORTTOOL ImprovingMiTToperationaleffectivenessrequirestoolsthatmiti gatetheeffectsofintrinsicandextrinsicfactorsonsharedSA.Accord ingly,wedevelopedaprototypetool,theCollectiveObservationsof HumanOrganizationalandRelationalTerrain(COHORT),asatheoreti callybasedandresearchdrivensolution.Theconceptualdesignofthe COHORTtoolispresentednext.

ToolstoSupportAdHocTeams363 COHORTInformationRequirements Thefirststepintheprocessofdevelopingatoolthatwouldsupport theSArequirementofMiTTmemberswastoisolatethoseSArequire mentsthatweremostcriticalforteamperformanceformultiplekey teamroles,thatis,thecriticalsharedSArequirements.Astandarddefini tionofSharedSAis"thedegreetowhichteammemberspossessthe sameSAonsharedSArequirements"(Endsley&Jones,1997,p.47; 2001,p.48).Eachteammemberhascriticalinformationrequirements, theirSAneeds,whichsupporttheirabilitytoachievetheirownindivid ualgoalsasateammember.Someoftheinformationrequirementsare uniquetothemembersroleontheteam,whileotherSArequirements aresharedamongmultipleteamroles.ThesearetheteamsSharedSA requirements.Forexample,theLogisticsOfficermightneeddetails abouteachitemintheinventory,whiletheOperationsOfficermayonly needtoknowthataplannedtrainingexercisewillnotdepletethese inventorystoresbelowanacceptablelevel. Aslightlydifferentconstruct,TeamSA,isalsorelevantforteamoper ationsandisdefinedasthedegreetowhicheveryteammemberpos sessestheSArequiredforhisorherresponsibilities(Endsleyetal., 2003,p.196).Thus,TeamSArepresentsthesumofeachteammembers knowledgeofneededsituationawarenesselements.InFigure20.1the shadedareainthegraphontheleftshowsateamsSharedSA,whilethe chartontheleftshowstheTeamSA.Asapracticalpoint,then,notall informationfromeachteammemberneedstobeshared,butonlythe informationthatwhenappropriatelymanagedbytheteamisneeded bymultipleteammemberstoachieveoperationalsuccessandgoal accomplishment.

364STRATERETAL.

Figure20.1 ThedifferencebetweenSharedSAandTeamSA(EndsleyandJones, 2001).AdaptedwithpermissionfromNewtrendsincooperativeactivi ties:Understandingsystemdynamicsincomplexenvironments,byM. McNeese,E.Salas&M.Endsley,2001,SantaMonica,CA:HFES.Copy right2001byHFES. Fromatheoreticalstandpoint,achievingSharedSAcanbeviewedas theproductofsharingmissionrelevantbitsofindividualteammembers SAelements.EndsleyandJones(1997)conceptualizedaframeworkfor TeamSAthatisdependentuponfourprimaryfactors(seeFigure20.2): (a)TeamSArequirements;theinformationteammembersneedacross threelevelsofSA(i.e.,perception,comprehension,andprojection),(b) TeamSAdevices,suchasdisplaysandcommunicationstoolsthatare availabletosupportinformationsharingandcollaboration,(c)TeamSA processes,whichinclude,forexample,standingoperatingproceduresor, simply,schemasorcommonmethodsforsharingandanalyzinginforma tion,and(d)TeamSAmechanisms,whichrefertointernalcognitive mechanisms,suchasmentalmodels,whichcanfacilitatethedevelop mentofsharedSA.Similarmentalmodelscandevelopovertimethrough commontrainingandsharedexperiencesandthusitmaybeparticularly problematicforadhocteamstodevelopthesecommoncognitivemech anisms.Asaresult,itisvitalthatdisplaysprovidesupportforthedevel opmentofthehigherlevelsofSA,comprehensionandprojection,sothat teammemberscanmaintainacommonunderstandingonthesecritical informationrequirements. Typically,SArequirementsanalyseshavebeenconductedusinga formofcognitivetaskanalysisknownasagoaldirectedtaskanalysis (GDTA)(Endsley,1993;Endsleyetal.,2003).TheGDTAfocusesonthe

ToolstoSupportAdHocTeams365 basicgoalsforeachteamrole(whichmaychangedynamically),the majordecisionsthatshouldbemadetoaccomplishthesegoals,andthe SArequirementsforeachdecision.TheGDTAmethodologyattemptsto determinewhatoperatorswouldideally liketoknowtomeeteachgoal, evenifthatinformationisnotavailablewithcurrenttechnology.GDTA methodologyhasbeenusedextensivelytodetermineSArequirementsin awidevarietyofoperationsincludingcommercialaviation(Endsley,Far ley,Jones,Midkiff,&Hansman,1998),bothenrouteandTRACONair trafficcontrol(Endsley&Rodgers,1994),infantryplatoonleaders (Strater,Jones,&Endsley,2001),Navyairwarfarecoordinators(Strater etal.,2004),operatorsofunmannedair,surfaceandunderwatervehicles (Strater,Connors&Davis,2008),andarmybrigadeoperations(Bolstad, Riley,Jones&Endsley,2002).

Figure20.2 FrameworkofTeamSA(fromEndsleyandJones,2001).Adaptedwith permissionfromNewtrendsincooperativeactivities:Understandingsys temdynamicsincomplexenvironments,byM.McNeese,E.Salas&M. Endsley,2001,SantaMonica,CA:HFES.Copyright2001byHFES. WiththesupportoftheBCKSgroupatFt.Leavenworth,SATechnolo giesconductedaseriesofGDTAinterviewswithMiTTmembersreturning toFt.RileyfromdeploymentsinIraqandAfghanistan.Interviewees servedinoneoffourrolesselectedfortheinitialevaluation,Com mander,OperationsOfficer,IntelligenceOfficerandIntelligencenon commissionedofficer(NCO).Duetosimilaritiesinthepositions,theroles ofIntelligenceOfficerandIntelligenceNCOwerecombined;thusgoal

366STRATERETAL. hierarchiesweregeneratedfortheMiTTCommander,Intelligence Officer,andOperationsOfficer. ApplyingtheSAorienteddesignprocessasdescribedbyEndsleyet al.(2003),thesegoalhierarchiesdrovethedesignoftheCOHORTtool,as partofalargerwikilikesetoftoolstoprovideLocalInformationfor OperationalNeeds(LION),referredtohereastheTeamSALION(seeFig ure20.3).Awikiisacollectionoflinkedwebpagesthatemployssimple editingtoolstoallowanyuserwithaccesstomodifythecontent. AlthoughtheTeamSALIONwillbesimilartoawikiinthatitisaliving collectionofknowledgeaboutthelocalenvironmentandinhabitants,it differsfromatraditionalwikiinthataccessanduserpermissionsmay varysignificantlyacrosspersonnel.Themajorityofthisdiscussionwill centerontheCOHORTtool;weintroducetheTeamSALIONtoprovide anoverviewofthetoolsuiteinwhichtheCOHORTwillreside,andacon ceptualframeworkoftheinteractionsandlinkagesbetweentools.

Figure20.3 TeamSALION(LocalInformationforOperationalNeeds) TheCOHORTprototypewasdesignedtopromoteMiTTmembers SharedSAwithregardtopeopleteammembers,localcounterparts, andthelocalcivilianpopulationwithinthecontextoftheorganiza tionalandoperationalenvironment.Toillustrate,theMiTTCommander mustdevelopanaccuratementalmodeloftheIraqiArmy(IA)battalion (BN)tocommunicateeffectivelywithhis/herIAcounterpartandothers withintheIraqicommandstructuretoachievethemissionobjectiveof autonomousIAoperations.Table20.1providesacomparisonofsomeof thegoalsfortheMiTTCommander,OperationsOfficerandIntelligence

ToolstoSupportAdHocTeams367 Officer,showingsignificantoverlapbetweenthegoalsofeach.Table20.2 presentsthesesamegoalsfromtheMiTTCommanderGDTA,andillus trateshowtheinformationrequirementsrelatedtothesegoalscanbe incorporatedintotheCOHORTprototype. Overall,theGDTAsenabletheconceptualizationoftheCOHORT interfacedesign,mainlyintermsofrequirementsandfunctionality, whicharesupplementedwiththeusabilityrequirementsdiscussedinthe nextsection. COHORTUsabilityRequirements AlowfidelityCOHORTprototypehasbeendeveloped,andamore fullyinteractiveversionisnearingcompletioninpreparationforuser testing.Thissectiondetailstheusabilityrequirementsestablishedforthe COHORTprototypeusingsamplescreenshotsdevelopedforthelow fidelityandinteractiveprototypes. Table20.1GDTAgoalsindicatingsimilaritybetweengoalsfor comparisonforCommander,OperationsandIntelligenceofficer
CDRgoals
EmpowerIAtomaximize operationsuccess Ensureoperationreadiness Supporttrainingprograms developmentand enforcement Ensureproper communicationsare established FosterrelationshipwithIraqi counterpart EnsureBNcontinuity Promotelocalstability Adoptculturallysensitive leadershipstyle

Opsofficergoals
Improvecounterpart competenceandhostnation Independence Ensureoperationwithin establishedMiTTparameters FostercounterpartS3training andprofessionaldevelopment Ensureeffective communicationflow FosterpositiveIraqiUS relations Promotecontinuity Ensureinsurgentsdonot impedeIAdevelopment Operateeffectivelywith culturaldifferences

Intelofficergoals
Mitigateknowledgegapsand provideintelstructurefor MiTTandIraqi Sustainintelacquisition, usageandsecurity Developandevaluate trainingprograms Sustainintelacquisitions, processing,distributionand security FosterIraqirelationships Ensurecontinuity Collectanddisseminateintel topromoteIraqistability Accountforcultural differences

368STRATERETAL. Table20.2CMDGDTAgoalsandCOHORTimplementation
GDTAgoals EmpowerIAtomaximize operationsuccess Ensureoperationreadiness Supporttrainingprograms developmentandenforcement FosterrelationshipwithIraqi counterpart Ensurepropercommunications areestablished EnsureBNcontinuity Adoptculturallysensitive leadershipstyle COHORTdesignimplementation Rosters;Visualizationofchainofcommand;Profile pagesonunitmembers Radargraphsofunitandindividualproficiency; Timelineoftrainingimplementation Radargraphsofunitandindividualproficiencyover time;Abilitytosubjectivelyrankunitmembers Profileinformationofcounterpart;Accessto previousMiTTexperiencewithcounterpart throughlogsandnotes Networkvisualizationsofrelationships;Address bookfunctionality Logfunctionality;AccesstopriorMiTTteam membersaccomplishmentsandnotes Inclusionoftribalinformationonprofilepages; SoldiersrankdisplayedinbothEnglishandIraqi equivalent;HyperlinkstoWIKImaterials

UserCharacteristics.TheuserinthiscaseisaMiTTmember;thatis, aU.S.Armyofficer,withtheaptitudesandskillsconsistentwithsuchper sonnel.Thereisonephysiologicallimitationthataffectsmanymale users:dyschromatopsia.Dyschromatopsia,knownascolorblindness,can altertheperceptionofcolorinformationandhuecontrast,potentially negativelyimpactinginteractionwithaninterfacethatemployscolor codingtoimpartmeaning.Thus,redundanciesforcolorcodingsuchas shapecodingwerenecessary. OperationalConditions.TheCOHORTtoolisenvisionedforusein twoenvironments.First,theCOHORTtoolcanbeaccessedfromnet workedcomputerterminalsonanymilitarybase,allowinguserstointer actwithafamiliarpointandclickinterfacewithinarelativelycontrolled environment.However,thisoptionlimitstheavailabilityoftheCOHORT toolwhenneededthemost,suchaswhileonpatrol,atcheckpoints,in convoys,oratotherlocationsthatdonothavecomputerterminals. Therefore,asecondoptionallowsforaportabledevicethatprovides mobileaccess. Accessibility.TheusabilityoftheCOHORTtoolwilldependinparton itsaccessibility,sinceitcapitalizesontimelyandaccuratedatabaseman agementtopresentuptodateinformationtosupportSA.Forexample, theprocessofdataentryinanoperationalenvironmentmaysuffer underlowvisibilityconditions(e.g.,nighttime).Furthermore,asetof procedures,permissionsandcapabilitieswouldhavetobeestablishedto

ToolstoSupportAdHocTeams369 allowtrustedpersonnel(e.g.,MiTTCommander)toacquireandupdate datawhetherinthefieldoronacomputerterminal. DataRequirements.Gatheringintelligenceintheareaofoperations (AO)isbeneficialsincemuchofthisinformationisobtainedwhileinter actingwiththelocalpopulationoronpatrol.Thus,aportable,security enableddevicewithcapabilitiessimilartotheiPhone3Gisplannedto supportrealtimeinformationgathering(seeFigure20.4).Forexample,a snapshotcouldbetakenofapersonofinterest(e.g.,anIraqiArmysol diercommunicatingwithasuspectedinsurgent)withatimeandlocation stamp(viaGPS).Also,datacouldbeupdatedaboutpreferences,charac teristicsandhistoryofkeylocalsastheinformationisgathered.This informationcouldbeimmediatelyupdatedthroughanencryptednet work,orupdatedandanalyzeduponreturntobase.Toillustrate,anindi vidualisselectedintherosterontheleft,andpeoplewithestablished linkagestothatindividualareshowninthenetworkontheright.Thedif ferentcloudcolorsinthenetworkcanbeusermodifiedtorepresentdif ferentrelationshipsorcharacteristics(e.g.relationshiptypefamilial, work,neighbors,friends;persontypeIraqiArmy,IraqiPolice,localcivil ians,localgovernmentpersonnel,USmilitary,etc.).

Figure20.4 ExampleofaprototypeCOHORTscreendevelopedfortheiPhone3Gfor demonstrationpurposes KnowledgeRequirements.TheCOHORTtoolcapitalizesonexisting militarydomainknowledge.ByleveragingCOHORTusersexistingknowl edge,wecanpresentmeaningfulinformationtheusercanprocess quicklyandunambiguously.Forexample,asillustratedinFigure20.5,the useoffamiliarmilitaryunitsymbolscansupportrapidunderstandingof thelocalcommandhierarchyandpersonnel.Inthisexample,selecting

370STRATERETAL. anindividualintherosterhighlightsthatindividualspositioninthecom mandhierarchy,andselectingapositioninthehierarchyhighlightsthe individualintherosterwhoholdsthatposition. DatabaseRequirements.Ultimately,theCOHORTtoolwilluseexist ingmilitarydatabasestoupdateinformation.However,withtheproto type,asocialnetworkdatabaseformatisbeingimplementedforthe initialdevelopment.

Figure20.5 ExampleofaCOHORTscreenusingmilitarysymbolstodepictanIraqi battalionsstructure COHORTSocialNetworkVisualizations COHORTvisualizationswillrelyontheabilitytoimplementafree ware,Javabasedtoolsuitewiththirdpartysocialnetworkvisualization software.Aninvestigationoftheavailablesocialnetworkvisualization softwarehasidentifiedseveraldesirablecharacteristicsandfunctional capabilitiesthattheselectedproductshouldprovide.Theseincludenet workanalysis,multicolorednodes,nodeclouds,theabilitytouse imagesanddifferentshapestorepresentnodes,animation,dynamic visualization,multiplelinetypes,multiplelinecolors,andedgelabeling (seeFigure20.6).Specifically,relationshipstrengthcanberepresented bythethicknessofthelineslinkingpersonnel,whilecloudsandoutlines canbeusedtorepresentotheruserdefinedvariables,suchasservice branch,levelofsupportfortheU.S.orcoalitionmission,tribalaffilia tions,etc.PrototypedevelopmentoftheCOHORTsocialnetworkvisual

ToolstoSupportAdHocTeams371 izationemploysVizster,whichcurrentlyprovidesthebestfitforthe desiredcapabilities.

Figure20.6 COHORTtoolscreenillustratingnetworkvisualizationcapabilities COHORTMetricsVisualizations Currently,socialnetworkmeasurescanquantifymanyaspectsofa network,suchasnodecentrality,closeness,andbetweenness,allhelping todeterminethecriticalcharacteristicsofthenetwork(e.g.,Bender deMoll,&McFarland,2006;Carley,&Reminga,2004;Wasserman& Faust,1994).ThesemeasuresenabletheCOHORTsocialnetworkvisual izationtool;theyinteracttocreateaccuratenetworks,provideinputinto algorithms,andpromotesearchfunctionality.Oneofthemostimportant implicationsofnetworkvisualizationistheabilitytoconvey,ataglance, considerablequantitativeinformation.However,tocapitalizeonthe potentialcapabilitiesoftheCOHORT,wearedevelopingadditionalmet ricstoallowrelevantvisualizations(seeFigure20.7foracustommetric indevelopmenttodepictunitreadiness). Thenatureofrelationshipscanalsobedescribedthroughqualitative metrics,asidentifiedintheMiTTGDTAs.Onesuchmeasurewouldbea relationshipindexthatprovidesaqualitativeindicationof,(a)thelikeli hoodthatthepersonposesathreat,(b)thepersonsoveralltraining, experienceorcapabilities(seeFigure20.8),and(c)theleveloftrustin thatperson.Suchinformationwillbedrawnfrombothobjectivesources (e.g.,soldierstrainingrecord)andsubjectivesources(e.g.,collective observations,reviewsandratingsfromauthorizedpersonnel).

372STRATERETAL.

Figure20.7 WindowinCOHORTshowingacustommetricforunitreadiness,withan aggregateofindividualreadinessevaluationsforindividualsorunits

Figure20.8 WindowinCOHORTillustratinganIraqisoldierscapabilities GuidelinestoSupportAdHocTeams Thusfar,theCOHORTprototypeispresentedinrelationtoitsinter facedesignrequirements,fromoperationaltofunctional,andusability requirements.However,thissectionfocusesonanumberofgeneral guidelinesthatwereadoptedwhendevelopingtheCOHORTprototype, whicharespecifictoaugmentingSharedSAforMiTTteams.Theseguide linesarepresentedinamannerthatwouldbeapplicabletomanyadhoc

ToolstoSupportAdHocTeams373 teams,thus,theseguidelinesmaybeofvaluewhendevelopingtoolsfor supportingadhocteamsSharedSA. Adhocteamsneedincreasedsupportforcomprehensionandprojec tion.Withlesscommontrainingandbackground,adhocteamsneed increasedsupportforhigherlevelSA.Informationdisplaysshouldpro moteacommonunderstandingofthestatusorimpactofvariousevents, alongwithprojectionsoflikelyfuturestatesbasedonevents. Adhocteamscanbenefitfromavisualorgraphicalrepresentationto betterunderstandcomplexrelationships.Thisrepresentationshould maximizetheamountofinformationconveyedinapreattentivefashion (e.g.,colorcoding,nodesize,etc.)aswellasprovideflexibilityandadapt abilityintermsoftheinformationitdisplaysviausercontrol(e.g.,filters, queries,etc.). Visuallyrepresentingmissioncontinuityandprogressviachartsor figurescanhelpsupporttheaccurateprojectionofgivenstatesandsitua tionsrelatingtomissiongoals.Thisrepresentationshouldinclude progresstowardstrategicgoalstheteamsoverarchingobjectives,as wellasmoredetailedassessmentsoftacticalgoalsthelowerleveltask ingandoperationaldetailsnecessarytomeetthestrategicgoals. Visuallyrepresentingteamknowledgeviachartsorfigurescanhelp supporttheaccurateprojectionofgivenstatesandsituationsrelatingto unitsorindividuals.Thesevisualrepresentationscanbetailoredviaspe cificalgorithmsthatautomaticallyadaptthesefiguresandchartsasthe informationcontainedinthedatabaseevolves. Adhocteamsneedtohaveinformationavailableonculturalnorms astheyrelatetomissiongoals.Ideally,thisshouldbeachievedby embeddingcontextcriticalculturalinformationfromthehumanterrain (e.g.,tribe,appropriatediscussiontopics,etc.)directlyintonodeprofiles, andbyprovidingaccesstomoreindepthculturaltopicsviaadditional TeamSALIONtools. ADVANCINGTHESTATEOFTHEART TechnologyTransition Thefunctionalprototypepresentedinthischapterwasdeveloped basedoncurrentandfuturetechnologyneeds.Specifically,theinterface andnavigationarefullyfunctionalandtheinformationcontainedinthe variousvisualizations(e.g.,humanterrainnetwork,unitandindividual performance,etc.)isbasedonwhatsocialnetworkvisualizationtools anddatabasescandobasedonourtechnologyassessment(seeprevious

374STRATERETAL. sectionregardingvisualizationsandvisualizationmetrics).However,cur rentCOHORTassumptionsarebasedonawebbaseddeliveryofthetool andpositiveinteractionsbetweenvariousprograms(e.g.,Java,database synchronization,visualnetworksoftware,etc.). Futuretechnologychallengesaremainlycenteredonintegratingthe variousinteractingpartsoftheCOHORTtoolaswellasaugmentingits availabilityandportabilitytoalargerspectrumoftheoperationalenvi ronment.ThisincludesusingtheCOHORTtoolonhandhelddeviceson patrols,atcheckpoints,etc.,thatwillrequireasecurityenabledportable militarydevice.Anotherchallengeliesindevelopingalgorithmscapable ofquantifyingpresentandpossiblefuturethreatlevelsbasedonthe quantificationofvariousintranodeparametersinvisualnetworkstofur thersupportadhocteamsSharedSA.Afinalchallengewillbetointe gratethetooltoleveragedataalreadyavailableincurrentsystems,and toinsurecontinuousupdatingoftheinformationacrossmultiplesystems thatwillrequirethesamedatasets. FutureResearch Futureplannedresearchincludes(a)completingtheiterativeand usercentereddevelopmentoftheCOHORTtoolbeyonditsprototype state,(b)assessingtheutilityoftheCOHORTtoolinsupportingadhoc teamsSharedSA,and(c)continuingtocollaboratewiththemilitaryto obtainvaluableinputfromsubjectmatterexpertsandpotentialusers andestablishingthevalidityoftheCOHORTtoolintheoperationalenvi ronment. CONCLUSIONS Missioneffectivenessofadhocteamsperformingintodayscomplex militaryoperationsiscriticallydependentuponaccesstotechnological toolsthatpromoteTeamandSharedSA.Inthispaper,wedescribeda collaborativeeffortbetweenSATechnologiesandARLLeavenworththat ledtotheconceptualdesignofourCOHORTtool,proposedasatheoret icallybasedandresearchdrivensolutiontoaddressthisimportantchal lenge.TheCOHORTtoolsdesigncapitalizesonexistingtechnologiesto displaysalientaspectsofhumanterrainwhileadheringtousercentered principles.TheCOHORTtoolprovidesawayofnotjustsharingrawdata aboutrelevantobservationsanddetailsaboutthehumanterrain,butit alsosupportsvisualizationsthatpromotethehigherlevelsofSA.Byinte gratingdataacrossmultipleobservationsinmeaningfulways,the

ToolstoSupportAdHocTeams375 COHORTallowsallmembersoftheteamaccessnotonlytotheirindivid ualobservations,butalsototheunderstandingofthehumanterrainthat comesfromtheaggregationofmultipleobservations. Particularbenefitsofthistoolarethatitpromotesarapidunder standingofthesituation,thussupportingSAeveninthefaceofthetem poralperturbationsthatcandisruptadhocteamoperations.Recent informationprovidedtousindicatesthatMiTTpersonnelhaveatwo weekreliefinplaceoverlap,wheremembersoftheoutgoingteampro videdetailsoftheoperationalenvironmenttomembersoftheincoming team.Someoutgoingteamsprovidepermanentdocumentationoftheir experiencesintheenvironment,suchaswrittennotesoreventdescrip tions,whileothersprovidealltheirtransitioninformationverbally.In eithercase,ittakesasignificantrampupperiodforeachincomingMiTT togainanaccuratefeelforthelayofthelandparticularlyintermsof thehumanterrain.WithnetworkedCOHORTandSAWikitools,person nelcouldbeginstudyingtheirassignedareaintheirpredeployment training,thusarrivingincountrywithamuchbetterabilitytounderstand andpredicthumandynamics.ThesetoolswillallowMiTTpersonnelto improvetheirindividual,teamandSharedSA,andthuswillimprovetheir abilitytorespondtodevelopingsituationsintheirareaofoperation. FutureworkwillfocusonfurtherdevelopingtheCOHORTtoolsfea turesaswellasexploringthefeasibilityandutilityofothercomponents ofaTeamSAWikiforsupportingcriticalcomponentsofTeamSAamong adhocteammembers. ACKNOWLEDGMENTS WewouldliketoextendourdeepestappreciationtoMr.RonPruyt andMr.MikeDoyleoftheBattleCommandKnowledgeSystemsGroup fortheirassistancewiththisproject,andforarranginginterviewsforus. WewouldalsoliketothankeachoftheTransitionTeammemberswho tookthetimetomeetwithusontheirreturntoFt.Riley,atatimewhen alltheywantedtodowastobewithfamilyandfriends.Wehaveendeav oredtoaccuratelyrepresenttheinformationtheywerekindenoughto share. Inaddition,wegratefullyacknowledgethemanycontributionsof Ms.DianeUngvarskyoftheU.S.ArmyResearchLaboratory,FortLeaven worth,forheressentialandcontinuedsupportthroughoutthisproject. Shehasbeenanintegralmemberofourteamsincewebeganinvestigat ingadhocteamsatFtLeavenworth,andhasguidedusthroughthechal lengesoflargescalemilitaryexperimentationefforts.Withouther

376STRATERETAL. experience,expertise,goodideas,goodhumor,anddailydistributionof chocolate,muchofourdatacollectioneffortswouldnothavebeensuc cessful. REFERENCES

BenderdeMoll,S.,&McFarland,D.A.(2006).Theartandscienceofdynamic networkvisualization.JournalofSocialStructure,7(2),246. Bolstad,C.A.,Riley,J.M.,Jones,D.G.&Endsley,M.R.(2002).Usinggoaldirected taskanalysiswithArmybrigadeofficerteams.ProceedingsoftheHumanFactorsand ErgonomicsSociety46thAnnualMeeting,472476.SantaMonica,CA:HumanFactors andErgonomicsSociety. Carley,K.,&Reminga,J.(2004).ORA:OrganizationRiskAnalyzer.CarnegieMellon University,SchoolofComputerScience,InstituteforSoftwareResearchInternational, TechnicalReportCMUISRI04106. DepartmentofDefense(2007,September11).Irregularwarfare(IW)joint operatingconcept(JOC).RetrievedMarch26,2009,fromhttp://www.dtic.mil/ futurejointwarfare/concepts/iw_joc1_0.pdf Endsley,M.R.(1988).Designandevaluationforsituationawarenessenhancement. ProceedingsoftheHumanFactorsSociety32ndAnnualMeeting,Vol.1:97101.Santa Monica,CA:HumanFactorsSociety. Endsley,M.R.,Bolt,B.,&Jones,D.G.(2003).DesigningforSituationAwareness. BocaRaton,FL:CRCPress. Endsley,M.R.,Farley,T.C.,Jones,W.M.,Midkiff,A.H.,&Hansman,R.J.(1998). Situationawarenessinformationrequirementsforcommercialairlinepilots(ICAT981). Cambridge,MA:MassachusettsInstituteofTechnologyInternationalCenterforAir Transportation. Endsley,M.R.,&Jones,W.M.(1997).Situationawareness,informationdominance, andinformationwarfare(No.AL/CFTR19970156).WrightPattersonAFB,OH:United StatesAirForceArmstrongLaboratory. Endsley,M.R.,&Jones,W.M.(2001).Amodelofinterandintrateamsituation awareness:Implicationsfordesign,trainingandmeasurement.InM.McNeese,E.Salas &M.Endsley(Eds.),Newtrendsincooperativeactivities:Understandingsystem dynamicsincomplexenvironments.SantaMonica,CA:HumanFactorsandErgonomics Society. Endsley,M.R.,&Rodgers,M.D.(1994).Situationawarenessinformation requirementsforenrouteairtrafficcontrol(No.DOT/FAA/AM94/27).Washington,D.C.: FederalAviationAdministrationOfficeofAviationMedicine. Fox,J.andD.Stowell(2008).AmericanAdvisors:AWayAhead.AirLandSea Bulletin,2,2630 Mayer,R.E.(2001).Multimedialearning.NewYork:CambridgeUniversityPress. Nagl,J.A.(2008).Institutionalizingadaptation:Itstimeforanarmyadvisor command.MilitaryReview,88(5),2126. Petraeus,D.H.(2008).MultinationalforceIraqcommanderscounterinsurgency guidance.MilitaryReview,SeptemberOctober2008,24. Strater,L.D.,Connors,E.S.,&Davis,F.(2008).CollaborativeControlof HeterogeneousUnmannedVehicles.PaperpresentedattheTexasHumanFactorsand ErgonomicsConference.

ToolstoSupportAdHocTeams377
Strater,L.D.,Endsley,M.R.,&Plott,C.(2004).Developmentofanautomated measurementbatterytoassessinnovativeconceptsandtechnologiesforshipboard applications:PhaseIFinalReport.Dahlgren,VA:NavalSurfaceWarfareCenter,Dahlgren Division. Strater,L.D.,Jones,D.G.,&Endsley,M.R.(2001).Analysisofinfantrysituation awarenesstrainingrequirements(No.SATech0115).Marietta,GA:SATechnologies. Sweller,J.,VanMerrinboer,J.,&Paas,F.(1998).Cognitivearchitectureand instructionaldesign.EducationalPsychologyReview,10,251296. Wasserman,S.,&Faust,K.(1994).SocialNetworksAnalysis:Methodsand Applications.Cambridge:CambridgeUniversityPress.

378STRATERETAL.

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T A N D L A U R E L A L L E N D E R

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T AND LAUREL ALLENDER

This book, produced by the Partners of the Army Research Laboratory Advanced Decision Architectures Collaborative Technology Alliance, also serves as the Final Report for DAAD19-01-2-0009.

SECTIONIV.COMMUNICATINGINFORMATIONACCROSSTHETEAM

SECTIONIV COMMUNICATINGINFORMATIONACROSSTHETEAM

324SECTIONIV

Chapter

21

21. DIAGRAMMATICREASONINGINARMYSITUATIONUNDERSTANDING ANDPLANNING:ARCHITECTUREFORDECISIONSUPPORTAND COGNITIVEMODELING

B.Chandrasekaran,Ph.D. BonnyBanerjee,Ph.D. UnmeshKurup,Ph.D. JohnR.Josephson,Ph.D. TheOhioStateUniversity,Columbus,OH RobertWinkler ArmyResearchLaboratory,Adelphi,MD

INTRODUCTION Diagramsareubiquitousinarmydecisionmaking.Commandersrep resent,communicate,andmonitortheirsituationunderstandingand plansbymeansofdiagramsdrawnoverterrainorothermaps.Diagrams aresocentralinmilitaryapplicationsthatfieldmanualshavestandard izedtheelementsofsuchrepresentations.Theyarenotonlyusefulin spatialreasoningtasks,butalsointasksthatarenotdirectlyrelatedto space,asevidencedbybarandpiechartsandVenndiagrams.Thereason fortheirusefulnessisthattheyenableanoptimalcombinationofvisual perceptionandsymbolicreasoning:visualperceptionpicksupinforma tionthatisperceptuallyavailableinthediagram,avoidingchainsofinfer ence,whilesymbolicreasoningenablesinferencesthatcanonlybemade withabstractanddistalknowledge. Wehavetwodifferentsetsofmotivationsforresearchindiagram maticrepresentations,andbothareespeciallyrelevanttoarmydecision support.Thefirstistheneedtounderstandwhatmakesforgooddia gramsforatask,i.e.,whatgraphicalpropertiesofadiagrammakeiteas ierforahumanusertoobtaintheneededinformationfastandwithout 379

380CHANDRASEKARANETAL. error.Thisknowledgecanbeusedtodesigneffectivediagramsanddia grammaticinterfacesforinteractivesystems.Exploringtheinteraction betweentaskdemands,graphicalproperties,humancognitiveandper ceptualarchitecture,andtheusersbackgroundknowledgethatgointo usingadiagramisbestdonebybuildingcognitivemodels.Designingand buildingacognitivearchitecturethatrisestothechallengeofmodeling diagrammaticreasoninghasbeenamajorconcernofourresearch. Thesecondmotivationarisesfromtheopportunitytoenhancedeci sionmakingbyautomatingsomeofthereasoningtasks.Forexample, considerreplanningarouteinrealtimeinanurbanengagementin responsetorapidchangesinthesituation,especiallyifthechangesare occurringoutsideoftheimmediatesurroundingsofasoldier.Forasol dierintheheatoftheaction,thistaskwillbeverydemandingatbest andimpossibleatworst.Anautomateddiagrammaticreasoningsystem thatcanperformtheneededreroutingfastanddisplaytheresultsina formthatthesoldiercancomprehendcanbeveryhelpful.Asanother example,aplannermightdesiretocritiqueaplannedavenueof approachwithrespecttoriskofpotentialambush.Thecomplexityofthe terrainmaymakethetasktimeconsuming,butanautomateddiagram maticreasoningsystemthatcanperformthistask,asademonstration systemwehavebuiltdoes,canaddrealvalue.Boththeresearchgoals automationandcognitivemodelinghaveacommonelement:ageneral cognitivearchitecturethatcanperformdiagrammaticreasoningby appropriatelycombiningspatialinformationandabstractknowledge. HistoricalBackground Researchindiagrammaticreasoninghasburgeonedinrecentyears, sowhatfollowsisintendedtogiveonlyaflavorofpreviousresearch. Moderninterestindiagrammaticreasoningasageneralconcept,not particulardiagramsforparticularproblemsaroseinthenineteenth centuryinthefoundationsofmathematics,especiallyinrelatingEuclid totheemergingnotionsoflogic.Euclidsmethodsweredeemednotrig orousenough,andaconsensusformedthatonlysymbolicreasoningof therightsortcouldprovidetherigorneeded.Inrecenttimes,aschoolof logicemerged(BarwiseandEtchemendy,1998)thattakesissuewiththis consensus,andhasattemptedtoestablishthebasisforrigorousdia grammaticproofs. Inpsychologyinthe196070s,thenatureofmentalimageswasa subjectofcontroversy(seePylyshyn(2002)forasummary),withdis putesaboutwhatitmeanttohaveanimageinsidethehead,howsuch

DiagrammaticReasoningArchitecture381 imagesdifferedfromexternalimages,andhowtheymightberepre sentedinthemind/brain.Inanotherstream,Pinker(1990)andahostof psychologistsconsideredtheprocessesinvolvedincomprehending graphs.Humanuseofdiagrams,mostlyintheformofmentalimages,in syllogisticreasoninghasbeenstudied(JohnsonLaird,1983).Phenomena suchasdistortionsingeographicrecall(StevensandCoupe,1978)were identifiedandinvestigatedexperimentally,resultingintheoriesabout theorganizationofhumanspatialmemories.Noneofthisworkwas explicitlycomputational,thoughtheyallassumedthatoperationson representationswereinvolved. WiththeemergenceofAI,whosecomputationalframeworkpro videdadualfunctionalityofbuildingAIsystemsaswellascomputational modelsofhumantaskperformance,computationalcognitivemodeling becameanactiveareaofresearch.However,AIanditssister,Cognitive Science,focusedalmostexclusivelyonsymbolicrepresentationsasthe basisforcognition,withafewexceptions,e.g.,Gelernter(1959)ongeo metrictheoremproving.Bytheearly1990s,however,interestindia grammaticreasoninginAIhadgrownsufficientlyforasubstantialedited collectionofarticles(Glasgow,NarayananandChandrasekaran,1995). Ourworkondiagrammaticreasoningisdistinguishedbyourdesire tounderstandthestrengthsandlimitationsofhumandiagrammaticrea soning,buildanarchitecturethatinvokesthediagrammaticandsymbolic modesofreasoningasappropriate,andusethearchitecturetobuild advanceddecisionsupportsystems(Chandrasekaran,etal.,2002)andto buildcognitivemodelsofspatialreasoning(KurupandChandrasekaran, 2007). DIAGRAMSANDREASONINGWITHTHEM Perhapsthebestwaytoexplaintheideasbehinddiagrammaticrea soningistodescribeanactualapplicationoftheideatoarealproblem. Thisexamplehasbeenimplementedusingthetechnologythatwebuilt, soitisanintroductiontoourtechnologyaswell. EntityReidentificationbyInformationFusion TheentityreidentificationproblemisacoretaskintheAllSource AnalysisSystem(ASAS).TheASASsystemreceivesanewreportabouta sightingofentity,sayT3,oftypeT(tanksforarmy,planesforAF).Thesys temhastodecideifthenewsightingisthesameasanyoftheentitiesin itsdatabaseofearliersightings,oranentirelynewentity.Reasoninghas

382CHANDRASEKARANETAL. tofuseinformationfromdifferentsourcesdatabaseofsightings,capa bilitiesofvehicles,sensorreports,terrainandmapinformationto makethedecision. Figure21.1(a)showsadiagramofthreeregionssuperimposedona terrainmap,thethreeregionscorrespondingtoimpassableareasforthe typeofvehiclesofinterest.Thefigureswithbackgroundterrainimages arefordisplay.Theactualdiagramthatisusedinreasoningisshownin Figure21.1(f),andisrepresentedusingtheDRSsystemthatwewill describeshortly.Figure21.1(f)alsoshowsthreepointobjectsT1,T2and T3,whereT1andT2arelocationsoftwovehiclessightedearlierandof thesametypeasT3,andT3isthelocationofthelatestsighting(thetime ofreasoning).

Figure21.1 Variousstagesinentityreasoningaboutwhetheranewlysightedentity iseitheroftheearliersightedentities.Seetextfordetails.

DiagrammaticReasoningArchitecture383 Thereasoningmachineatthecoreisafusionengine(FE),which incorporatesaformofabductiveinference(JosephsonandJosephson, 1996)thatisappropriateforthetask.FEworkswithadiagrammaticrea soner(DR)thatappliesperceptionsandactionstothediagramintheDRS representation,inordertoachievethesubgoalssetbyFE.Perceptions obtaininformationfromthediagramandactionsmodifythediagramin variousways.Thefollowingstepscanbereadastheactualactionsofthe variouscomponentsinourimplementation,andasstepsthatahuman problemsolver,givenasimilardiagramandtaskmightperform. ThestrategyofFEistoseeifthereexistpathsthatT1andT2could havetakentogettoT3withinthetimeallowed,ifsuchpathsareconsis tentwithotherevidence.Basedontheresultsofthesesteps,determine whetherT1,T2,oranewpreviouslyunknownvehicleisthebesthypothe sistoexplainT3.FEasksDRtoidentifypathsthatT1mighthavetakento reachthelocationofT3.DR(byusingitsactionrepertoire(Banerjeeand Chandrasekaran,2004;BanerjeeandChandrasekaran,2006a))responds withtworoutes(seeFigure21.1(b))thatavoidtheimpassableregions, andgivestheirlengths.DRSisupdatedwiththesetwocurveobjects,cor respondingtothetworoutes.Rte1istheshorteroneatthebottom,and Rte2istheother.FErulesoutRte2asbeingtoolong,basedonthevehi clesmaximumspeedandthelengthoftheroute.FEsimilarlyasksDRto identifypossibleroutesforT2,DRrespondswithFigure21.1(c).Inthis case,Rte3istheonethatloopsaroundtheleftimpassableregion,and getstoT3bypassingbetweenthetwoleftregions.Rte4goesbetween thetoptworegions.FErulesoutRte3astoolong.FEnextasksasensor databaseifthereareanysensorfieldsintheareaofinterest,andifthey reportedanysightings.Thereareindeedtwosensorfieldsintheareaof interestbuttherehavebeennoreportsfromthemofvehiclecrossings. FEpassesthelocationstoDR.TheDRSisupdatedwithtworegionscorre spondingtothesensorfields(twosmallcirclesinFigure21.1(d)). FEasksDR(i)ifRte1andRte4intersectwiththesensorfields,and(ii) ifeitherofthemdoes,whetherthepathcouldbemodifiedtoavoidthe correspondingsensorfield.ForRte1(forT1),DRreportsthatitindeed intersectsoneofthefields,anditcouldbemodifiedtoavoiditasshown Figure21.1(d).DRalsoconcludesthatRte4cannotbemodifiedtoavoid thecorrespondingsensorfield(Figure21.1(e)). BecauseT2couldnothavegottentoT3withouttriggeringthesen sors,butT1couldhave,FEgivesT1ahigherplausibilityasthecandidate answer.FEalsoconsidersthehypothesisthatthenewsightingcorre spondstoanewentity,areasoningsequencewedontdescribehere.

384CHANDRASEKARANETAL. Thebesthypothesisattheendisthatthenewsightingisthatofvehicle T1,andfurtherthatitmostlikelytookRte1. ThepointofthedescriptionaboveisnotthedetailsoftheFEstrat egythereaderisencouragedtosuppressanyquibbleshemighthave buttheillustrationofhowthediagramisusedinacomplexreasoning task.FEsstrategyoritsvariationscanbeimplementedinmanyframe worksthatsupportknowledgerepresentationandgoaldirectedreason ing,butweneedadditionalcapabilitiesforthediagrammatic component.Wenextturntothesecapabilities. WhatisaDiagram? Diagramsarespatialrepresentations,createdbyanagentforaspe cificreasoningtask.Therequirementsofthetaskdeterminewhich aspectsofthespatialinformationareessential,andthosethatarenot areabstractedaway.Theobjectsinadiagramrepresentabstractorcon creteobjectsinthedomainofinterest,andtheirrelevantspatiality (Chandrasekaran,etal.,2004).Intheentityreidentificationexample,the relevantdiagrammaticelementswerenogoregions,locationpointsof vehiclesightings,andcurvesforvariousroutes.Thedistinctiveproperties ofdiagrammaticrepresentationsinspatialdomainsmaybeunderstood bycontrastingamapadiagrammaticrepresentationofaregionwith anaerialphotographofthesameregion.Themapandthephotograph agreespatiallyonseveralthings.Therearehowever,severalimportant differences. Abstractinginto(spatial)objects.Incontrasttothephotograph, themapidentifiescertainlocations,linesandregionsasobjects, i.e.,asthingsthatmaybenamedandtalkedabout,suchascities, highways,lakes,andsoon. Abstractingawayspatialinformationnotrelevanttothetask, suchasthereflectionofsunlightoffthewaterofalakeorthe varyingwidthofaroad. Iconizationorattachingsymbolstovisualobjects.Theidentityor typeoftheobjectsisoftenprovidedinthemapbysymbolsor icons,e.g.,aChurchiconoritsnameatthelocationofachurch. Ahashedcurvemightbearailroad.

DiagrammaticReasoningArchitecture385 WhatisDiagrammaticReasoning? Westartbyclarifyingwhatisnotdiagrammaticreasoning,inour senseoftheterm.Itdoesnotrefertoimageprocessing,patternrecogni tion,orcomputergraphics,thoughtheyareconnections. Problemsolvingisaprocessinwhichaproblemagoalisbroken downintosubgoalsrecursivelyuntilsubgoalsarereachedthatcanbe readilysolved.Howtodecomposeagoalintosubgoalsrequiresknowl edgeaboutthesubjectmatterdomain.Itiscommonplaceinmanyprob lemdomainsforpeopletousediagrams,eitherasexternal representationsorinimagination,toassistinproblemsolving.Thisis donenotonlyindomainsthathavespatialaspects,suchasspatialplan ning,butalsoindomainspartsofwhichhavespatialanalogs:Venndia gramsinsettheoryproblems,piechartstoshowbudgetcomponents andtheirshares,bargraphstorepresentsalesovertime,andscatter plotsthatbringoutcorrelationsbetweenvariablesinadataset. Intheentityreidentificationexample,aperceptioninthediagram maticpartsolvedthesubtaskofdecidingifarouteintersectswithasen sorfield.Adiagrammaticactioncanalsomodifythediagram,as happenedintheexamplewhenDRshowedthataroutecouldbemodi fiedtoavoidasensorregion.Thesymbolicpartappliednumericalinfor mationaboutmaximumspeeds,routelengths,andelapsedtimeto decidewhichrouteswerenotpossible.Itisthisopportunisticprocess wherebyperception/actionondiagramsandinferencefromsymbolically representedabstractknowledgeworktogether,bysolvingsubgoalsthat eachisbestsuitedforthatgivesdiagrammaticreasoningitspower. Perceptionisespeciallygoodatrecognizingemergentobjectsand relations.Forexample,giventhatpointAistotheleftofpointBandthat BistotheleftofC,thatAistotheleftofCisautomaticallytrueinadia gramthatonemightdrawtorepresentthesituation,andperceptioncan pickupthisinformation.Theavailabilityofthisinformationinadiagram forpickupbyperceptionhasbeencalledafreeride(Shimojima,1996). DRS,THEDIAGRAMREPRESENTATIONSYSTEM ThePhysicalversustheIntendedDiagram Aphysicaldiagramisanimage,anarrayofmarksonasurface.How ever,asarepresentation,itisaspatialconfigurationofdiagrammaticele ments,composedoutofpoints,curves,andregions.(Weassumeforthe currentdiscussionthatthediagramsdonotuseintensityvariationto

386CHANDRASEKARANETAL. representinformation,suchastemperatureisinsomeweathermaps.) Therearedifferencesbetweentheelementsinthephysicaldiagramand theelementsintheintendedone.Inadiagramofatriangledrawnto assistinprovingatheoreminplanegeometry,thetrianglessideswillnot beidealizedlinesofzerothicknesstheintendedrepresentationbut willbenarrowregions.Similarly,theverticesintheintendeddiagram havenodimensionatall,butinthediagramwillbesmallregions.The regionsoccupiedbythemarkscorrespondingtoalphabeticsymbolsthat nametheobjectshavenospatialrepresentationalsignificance,once theyarerecognizedassymbols.Sotheintendeddiagram,asaspatial representation,isacollectionofthreelinesmeetingatthreevertex pointsandeverythingelseinthephysicaldiagramistheretohelpwith memoryandperception,butisnotinfactspatiallyrepresentational. DRSisintendedtocapturethissenseoftheintendeddiagram,andis adomainindependentsystemforrepresentingdiagramsingeneral.A diagramisaconfigurationofdiagrammaticobjects,eachofwhichisone ofthreetypes:point,curve,andregion.Pointobjectsonlyhavelocation (i.e.,nospatialextent),lineobjectsonlyhaveaxialspecification(i.e.,do nothaveathickness),andregionobjectshavelocationandspatial extent.Intheentityreidentificationexample,thelocationsofthevehi clesandthesightingarepointobjects,theroutesthataregeneratedare curveobjectsandtheimpassableareasareregionobjects.Associated witheachobjectisthespecificationofthespatialityoftheobject,and additionalfeaturessuchassymboliclabels,colors,andiconicmarks(such ashashmarksonatrianglesideinthetriangleexampleabove).These labels,beingsymbols,havenospatialconnotation.Thespatialspecifica tionofobjectscanbeextensional,suchasaspecificationofa2Darray andtheintensityvaluesoftheelementsofthearray.Thespecification canbeintensional,e.g.,algebraicexpressionsorequationsthatdescribe point,lineorregionobjects.Asadatatype,howthespatialspecifica tionsarerepresentedislefttotheimplementation.Wehaveimple mentedboththeextensionalandintensionalversions(Banerjee,2007). Theusercaninputadiagramusingagraphicaltablet,assignobjecttypes totheobjects,andaddsymbolicinformation.Formilitaryapplications, ourinputsystemconnectstotheTacticalObjectDatabase,sothatthe objectscanbetaggedwithmilitarilyrelevantsymbolicinformation.The diagramscanbedisplayedasalayerinESRIsArcMapsystemdisplaying terrainimagesoftheregionofinterest.

DiagrammaticReasoningArchitecture387 PerceptionsandActions Asmentionedearlier,diagrammaticreasoninginvolvesapplying,at variouspointsduringproblemsolving,eitherperceptiontoobtaininfor mationfromthediagram,oractiontomodifythediagramtoreflect actualorproposedchangesinthesituation.Theperceptionsinvolved herearetobedistinguishedfromtheactivityofexternalperceptionof animagearray.DRScorrespondstoanimageonwhichafigureground separationhasalreadybeenperformed,sonoimageprocessingopera tionsasinearlyvisionareinvolved.Theperceptionsdonotalsoinvolve recognition,suchasseeingacollectionoflinesinacartoonasCharlie Brown.Theperceptionscangiveinformationaboutcertainproperties (lengths,angles,areas)oftheobjects,relations(inside,leftof,longer than,etc.)andnotingthepresenceofemergingobjectsthatresultfrom theconfigurationofobjects(suchastheintersectionpointortheseg mentsthatresultwhentwocurvesintersect).Examplesofperceptions andactionsweregivenintheentityreidentificationapplication. Abstractions.Aparticulartypeperception/actioninvolvesabstrac tion,suchasgestaltperceptionofemergentobjects.Examplesaregiven inFigure21.2.Dependinguponthetask,wewishtotreattheobjectsin thefigureasasetofobjects,asaregioncontainingobjects,orjustasa regionoranannularregion.

Figure21.2 Abstractingaclusterofobjectsintoaregion(left)andabstractingan annularregion Abstractingaclusterofobjectsintoaregion(left)andabstractingan annularregionThesetsofperceptionsandactionsareopenended(Ull man,1984).However,overtheyears,wehavebuiltarepertoirethatis usefulacrossawidevarietyofapplications(BanerjeeandChandraseka ran,2006b),andwhichcanbeusedtoasbuildingblocksforadditional perceptionsandactions.Wehavealsodevelopedanapproachwherethe systembuilderspecifiesthepropertiesofdesiredperceptionsand actionsandanautomaticprogrammingsystemcomposestheappropri atealgorithmsforthem(BanerjeeandChandrasekaran,2007;Banerjee, 2007).

388CHANDRASEKARANETAL.

BISOAR,THECOGNITIVEARCHITECTURE CognitiveArchitectures Acognitivearchitectureisadescriptionoftheconstituentelements andtheirinteractionsofthatpartofageneralcognitiveagentthatis involvedinthinkingorreasoning.Itisusuallydistinguishedfrommodules thatareresponsibleforexternalperceptionandmotorsystems.Soar (Laird,NewellandRosenbloom,1987)andACTR(AndersonandLebiere, 1998)aretwowellknowngeneralcognitivearchitecturesinthesymbolic representationframework.TheyenvisionanagentashavingaLongTerm Memory(LTM),aWorkingMemory(WM)andacontrolsystemthatorga nizesinteractionbetweenLTMandWMaswellasactivitiesinWMin ordertoachievecognitivegoals.Anagentthatinteractswiththeexter nalworldinordertoachievegoalsintheworldwillhaveadditionalper ceptionandmotormodules.SoarandACTRmodelthinkingas operationsonpredicatesymbolicrepresentations,i.e.,descriptionsof propertiesofandrelationsbetweenobjectsinthedomainofdiscourse, e.g.,On(A,B)vInside(B,C).Goals,worldstates,problemsolvingstates, andelementsofLTM(inAI,thetermknowledgebaseisusedtoreferto LTM)andWMareallsymbolicinthissenseoftheterm.WechoseSoaras thebasearchitectureforourwork,butmuchofwhatwedidcanbe directlyappliedtoActRaswell(seeMatessa,2007). AsimplifieddescriptionofSoarforourcurrentpurposesisasfollows. SoarsLTMconsistsofknowledgeintheformofproductionrules.Sup posetheagenthasaproblemsolvinggoalandarepresentationofthe worldstateinWM.ThecontentsofWMarematchedagainsttheleft handside(LHS)oftherulesinLTM,andthebestmatchingruleischosen. Therighthandside(RHS)oftheruleeitherdirectlysolvesthegoal,oritis usedtosetupasetofsubgoals,thesolutionofwhichwillsolvetheorig inalgoal.Ifanyofthesubgoalscannotbesolved,thisprocessofretriev ingknowledgeandsettingupsubgoalscontinuesuntiltheycanbesolved andthegoalsupthegoalstackaresolved,ornomorerelevantknowl edgeisavailableandSoarstops.Whenaproblemissuccessfullysolved,a ruleiscachedorchunkedinSoarterminologyinLTM,sothatthenext timesuchacombinationofgoal/cognitivestateoccurs,thesolutioncan bedirectlyretrieved,avoidingthesearch.Soarcanbeusedasaplatform tobuildAIsystems,aswellasforcognitivemodeling.Inthelattercase, LTMholdstheagentsknowledgeandproblemsolvingstrategies.WM sizecanberestrictedtomodelthelimitedhumansorttermmemory.

DiagrammaticReasoningArchitecture389 ArchitecturalSupportforOperationsonInternalImages.Evenwith externalrepresentations,imaginationmentaloperationsinwhich internalimageelementsaremoved,rotated,etc.isoftenrequired.For example,intheentityreidentificationtask,forahumanagent,deciding whetheraspecificroutecouldbemodifiedtoavoidasensorfield requiresimagination,notjustexternalperception.Soar(andActR)lack architecturalsupportfortheseimagisticoperations.BiSoarremedies thislack. BiSoar BiSoar(Kurup,2008)isanextensionofSoarthatretainsSoarsbasic architecturalelements,itsviewofhowproblemsolvingworks,andthe chunkingmechanismforlearning,buttreatsthecognitivestateasbi modal.Oneofthemodesisthetraditionalsymbolicstaterepresentation, butanewone,adiagrammaticmode,isadded.KnowledgeinLTM,rep resentationsinWM,andgoalscanallbebimodal. ConsiderasimpleexampleofaworldstateasdepictedinFigure 21.3(a)inwhichthreeblocksA,BandCarestackedonatable.Statesin biSoararebimodal,withaDRSrepresentationof(a)providingtherepre sentationforthediagrammaticmode,asintherightsideof(b)and(c). Specifically,theDRSwouldconsistofthreeregionsforthethreeblocks, andonefortheTable.ThepropositionsthatthatAisontopofB,etc., wontbepartoftheDRSrepresentationtheyaresymbolicdescriptions producedasneededbyapplyingappropriateperceptions.Thesymbolic componentsofthecognitivestatein(b)and(c)aresomewhatdifferent, pointingtothefactthatthesameworldstatecanbedescribedindiffer entwaysusingdifferentvocabularies.TheDRSrepresentationiscom pleteinthenarrowsensethatbothsymbolicdescriptionsin(b)and(c) canbeproducedfromit,butthesymbolicdescriptionsarenotcomplete inthesamesense.

390CHANDRASEKARANETAL.

Figure21.3 (a)TwobiSoarrepresentationsofworldstate;(b)and(c)havesameDRS component,butdifferentsymbolicdescriptions PerceptionandActionAlgorithms.Inthecurrentimplementationof biSoar,theperceptionsandactionsarenotdesignedtomimictheprop ertiesofthecorrespondinghumanoperationsindetail.Thereisgrowing literatureonvarioushumanjudgmentoperations,suchasestimatingan angleorthelengthofaline,comparingthelengthsoftwolinesatdiffer entorientations,etc.Afullcomputationalaccountofsuchperceptual activitiesthatexplainthetiminganderrorratedataisyettobedevel oped.Ourgoalistohelpmodelthesequenceofreasoning,perception andactionstepsthatareinvolvedinproblemsolving,andexplorethe interactionbetweenknowledge,strategies,attentionandmemory. Dependingonthegoalsofthemodeler,aperceptioncanbemod eledasasinglestep,orasavisualproblemsolvingactivitythatdeploys moreelementaryperceptionsandactions,usingavisualproblemsolving strategy. FromExternalRepresentationtoWorkingMemory.Whenwelook brieflyatadiagramwithmanyobjects,someofwhichhavecomplex shapes,whatweareabletorecallashortwhilelatermaybejustasubset oftheobjects,andtheobjectsrecalledhavesmoothershapesthanthey hadinthediagram.Thissimplificationisduetothelimitedcapacityof attention.Sincethegeneralsimplificationphenomenonistaskindepen dent,thisattentionmechanismispartofthedesignofbiSoar.

DiagrammaticReasoningArchitecture391 BimodalLTM.Bimodalchunkingtransferstherelevantcontentsof WMtoLTM,includingdiagrammaticelements.Forretrieval,wecur rentlyonlyusethesymboliccomponentstotriggerproductionrules, sincethereareunresolvedtheoreticalissuesinmatchingshapesfor recall. BISOARMODELSOFDISTORTIONINGEOGRAPHICRECALL AccordingtoStevensandCoupe(1978),whensubjects,withsome familiarityofthegeographyofwesternUS,wereaskedabouttherelation betweenSanDiegoandReno,mostansweredthatSanDiegowastothe westofReno,eventhoughofcourseRenoiswestofSanDiego.What roleisplayedbythepropertiesofthecognitivearchitectureandwhat rolebyagentspecificproblemsolvingstrategiesinthisphenomenon? Thiskindoflimitationofhumanmemoryisscientificallyinteresting,but alsopracticallyimportant,sincetheycanresultinunexpectedperfor mancedegradation. Webuiltthreedifferentmodelsofproblemsolvingforthistask,each providingadifferentexplanation(KurupandChandrasekaran,2007). Model1isofanagentthathasasinglesimplifiedmetricalrepresentation ofCaliforniaandNevadainLTM(andWM)likeinFigure21.4(a).Inthis particularexample,SanDiegototheWestofReno,butanagentthat paidparticularattentiontothesecitiesmayhaveametricalrepresenta tionwiththecitiesintheircorrectrelationshiptoeachother.Thispro videsanarchitecturalexplanation.Model2involvesaspecificproblem solvingstrategy.TheagenthassymbolicinformationinLTMthatSan DiegoisSouthofSanFranciscoandthatRenoisEastofSanFrancisco.It constructsadiagram(Figure21.4(b))inWMusingthisinformationand extractsthe(wrong)answerfromthediagram.Model3correspondstoa hierarchicalspatialrepresentation.Theagenthassymbolicinformation inLTMthatSanDiegoisinCalifornia,RenoinNevadaandthatCalifornia istotheWestofNevada.Thisinformationisusedtoconstructadiagram (Figure21.4(c))andthe(wrong)answerextractedfromit. Withthesethreemodelsathand,themodelerisinapositionto definespecificadditionalexperimentsthatcoulddistinguishbetween theseexplanations.Suchfinegrainedcognitivemodelsopenupnew avenuesofresearchonspatialmemoryphenomena.

392CHANDRASEKARANETAL.

Figure21.4 DiagramsinLTMsofthreeagents CONCLUDINGREMARKS Ourdualgoalstoautomatediagrammaticreasoningtoreduce decisionmakerscognitiveloadandtohelpbuildcognitivemodelsof diagrammaticproblemsolvingaresupportedbythesameunderlying architecture.Thescopeforautomationisbroad,giventheubiquityof spatialrepresentationsinArmytasks.Inadditiontotheentityreidentifi cationtask,wehaveimplementeddemonstrationsystemsforrerouting inanurbanmission(BanerjeeandChandrasekaran,2007;Banerjee, 2007),forcritiquingaproposedavenueofapproachwithrespecttopos sibilitiesofambush,andformaneuverrecognition(Chandrasekaran,et al.,2002).Anotherpossibleapplicationisinpersistentvideosurveil lance,whereamapofabuildinganditsenvirons,e.g.,canbeusefulin reasoningaboutmovementsofvehiclesandpersons.Insuchautoma tion,ourarchitecturescorrespondencewiththehumanarchitecture makesitpossiblefortheuserandthemachinetocommunicateusing mutuallyunderstandablediagrams.Similarly,inadditiontoacognitive

DiagrammaticReasoningArchitecture393 modelofgeographicalrecall,wehavebuiltmodelsofrecallofroutes, andofgraphcomprehension. Therearemanyavenuesforfutureresearch,allwithpotentialbene fitstoArmysneeds.Inadditiontoautomationandcognitivemodeling, tutorialsystemsthatusedynamicdiagramstoexplainconceptsand mechanisms,andteststudentsunderstandingcanincreasetheeffec tivenessoftraining.Ourdistinctionbetweenphysicalandintended, abstractdiagramsraisesissuesforfutureresearch:howtogofromphysi caldiagramstointendeddiagrams,andconversely,howtoconvert abstractdiagramsintodisplaysthatsatisfytherequirementsforpercep tibilityandcomprehensibilitybyhumanusers? Asacognitivemodelingtool,biSoarcandocertainthingswell,but notothers.Thecurrentimplementationsofperceptionsandactionsare functionaltheydothejobbutthealgorithmsarenotpsychologically realistic.Forreliablepredictionsofhumanperformance,weneedmuch moreinformationabouthowthehumanperceptionworks. Weendwiththeideathatourconceptofknowledgeneedstobe enlargedsothatvariouskindsofperceptualrepresentations,including diagrams,areproperlythoughtofasknowledge.Forspatialknowledge, viewingknowledgeasbimodal,asintheknowledgerepresentationwe haveadoptedforBiSoar,seemstobeapromisingidea. REFERENCES
Anderson,J.R.andLebiere,C.(1998).TheAtomicComponentsofThought, LawrenceErlbaumAssociates. Banerjee,B.andChandrasekaran,B.(2006b).Onautomatingperceptionsand actionsinreasoningwithmilitarydiagrams.Proc.25thArmyScienceConference,Orlando, FL. Banerjee,B.(2007).SpatialProblemSolvingforDiagrammaticReasoning.Ph.D. dissertation,Dept.ofComputerScience&Engineering,TheOhioStateUniversity, Columbus,OH. Banerjee,B.andChandrasekaran,B.(2007).Aconstraintsatisfactionframeworkfor visualproblemsolving.TrendsinConstraintProgramming,F.Benhamou,N.Jussienand B.OSullivan,Editors,HermesScience,Chapter26. Barwise,J.andEtchemendy,J.(1998).Acomputationalarchitecturefor heterogeneousreasoning.InI.Gilboa,editor,Proc.7thConferenceonTheoretical AspectsofRationalityandKnowledge,pp.127.MorganKaufmann. Chandrasekaran,B.,Josephson,J.R.,Banerjee,B.,Kurup,U.andWinkler,R.(2002). Diagrammaticreasoninginsupportofsituationunderstandingandplanning.Proc.23rd ArmyScienceConference,Orlando,FL. Chandrasekaran,B.,Kurup,U.,Banerjee,B.,Josephson,J.R.andWinkler,R.(2004). Anarchitectureforproblemsolvingwithdiagrams.InDiagrammaticRepresentationand

394CHANDRASEKARANETAL.
Inference,A.Blackwell,K.MarriottandA.Shimojima,Editors,LectureNotesinArtificial Intelligence2980,Berlin:SpringerVerlag,pp.151165. Gelernter,H.(1959).Realizationofageometrytheoremprovingmachine.Proc.of theInternationalConferenceonInformationProcessing,Paris,1959.Reprintedin Computers&Thought,MITPressCambridge,MA,1995,pp.134-152. Glasgow,J.,Narayanan,N.H.andChandrasekaran.B.(1995).Diagrammatic Reasoning:CognitiveandComputationalPerspectives.AAAIPress/MITPress. JohnsonLaird,P.N.(1983).MentalModels:TowardsaCognitiveScienceof Language,Inference,andConsciousness.Cambridge,MA,CambridgeUniversityPress Josephson,J.R.andJosephson,S.G.(1996).AbductiveInference:Computation, Philosophy,Technology.CambridgeUniversityPress. Kurup,U.andChandrasekaran,B.(2007).ModelingMemoriesofLargeScaleSpace UsingaBimodalCognitiveArchitecture.Proceedingsofthe8thInternationalConference onCognitiveModeling,R.L.Lewis,T.A.Polk,andJ.E.Laird,Editors,Universityof Michigan,AnnArbor,MI. Kurup,U.(2008).DesignandUseofaBimodalCognitiveArchitecturefor DiagrammaticReasoningandCognitiveModeling.Ph.D.dissertation,Dept.ofComputer Science&Engineering,TheOhioStateUniversity,Columbus,OH. Laird,J.,Newell,A.andRosenbloom,P.(1987).SOAR:AnArchitectureforGeneral Intelligence.ArtificialIntelligence,33(1),pp.164. Matessa,M.,Archer,R.,andMui,R.(2007)DynamicSpatialReasoningCapabilityin aGraphicalInterfaceEvaluationTool.Proceedingsofthe8thInternationalConferenceon CognitiveModeling,R.L.Lewis,T.A.Polk,andJ.E.Laird,Editors,UniversityofMichigan, AnnArbor,MI. Pinker,S.(1990).Atheoryofgraphcomprehension.InR.Freedle,editor,Artificial IntelligenceandtheFutureofTesting,pp.73126.LawrenceErlbaum,Hillsdale,NJ. Pylyshyn,Z.W.(2002).Mentalimagery:Insearchofatheory,BehavioralandBrain Sciences,25:157182. Shimojima,A.(1996).Reasoningwithdiagramsandgeometricalconstraints.InJ. SeligmanandD.Westerstahl,Editors,Logic,LanguageandComputation,Volume1. Stanford:CSLIPublications,pp.527540. Stevens,A.andP.Coupe(1978).DistortionsinJudgedSpatialRelations.Cognitive Psychology10:422437. Ullman,S.(1984).Visualroutines.Cognition,18:97159.

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T A N D L A U R E L A L L E N D E R

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T AND LAUREL ALLENDER

This book, produced by the Partners of the Army Research Laboratory Advanced Decision Architectures Collaborative Technology Alliance, also serves as the Final Report for DAAD19-01-2-0009.

SECTIONIV.COMMUNICATINGINFORMATIONACCROSSTHETEAM

SECTIONIV COMMUNICATINGINFORMATIONACROSSTHETEAM

324SECTIONIV

Chapter

22

22. MODELS,METHODS,ANDTOOLSFORTHEADAPTABILITYOFTEAMS

PeterW.Foltz,Ph.D. PearsonandNewMexicoStateUniversity,Boulder,CO AdrienneY.Lee,Ph.D. NewMexicoStateUniversity,LasCruces,NM GaryBond,Ph.D. WinstonSalemStateUniversity,WinstonSalem,NC MelanieMartin,Ph.D. CaliforniaStateUniversity,Stanislaus,Stanislaus,CA

INTRODUCTION SSTR(Stability,Security,TransitionandReconstruction)missions oftenrequirethatdiverse,distributedexpertsfrommultinationalforces, nongovernmentalorganizations,andothergovernmentagencieswork together.CommandersandteamstrainedasWarfightersoftenmust acquiredifferentskillstoperformeffectivelyforSTTRoperations.While someknowledgecantransferfromwarfightingskills,otherskillsmaybe needed,suchastheabilitytoperformeffectivedecisionmakinginnovel culturalsituationsanddevelopingmultifunctionalteamsfromcoalition forces.Thisraisesanumberofissuesabouthowtoeffectivelyassessand measureperformanceinsuchoperationsandhowtoolscanbedevel opedthatprovideappropriatesupportduringSSTRoperations. ThedomainofSSTRoperationsprovidesarichareaforexamining cognitive,social,andculturalaspectsofcollaborationwhilealsoprovid inginformationthatshouldbegeneralizabletomilitaryteamperfor manceoverawidenumberofdifferentdomains.Forexample, commandersandteammembersmustunderstandhowculturaland 395

396FOLTZETAL. socialdifferencesinfluencebehaviorssuchasinformationsharingand managinguncertaintyorrisk.However,collaborativetechnologythat merelysupportsinformationexchangewillbenecessary,butnotsuffi cienttoinsurecollaborationamonghighlydiverseteammemberssolving complexproblems.Theoverallobjectiveofthepresentworkisto developandvalidatemodels,methods,andautomatedtoolsformea surementofperformanceinteamsperformingStability,Security,Transi tion,andReconstruction(SSTR)operations. Theworkdescribedinthischapterfocusesonmethodstoassessand enhancecollaborationinteamsandtheeffectsofcultureandsocial issues(e.g.,trust)onteamperformance.Learningandtransferofthe teamsskillswasexaminedbyhavingteamsperforminanSSTRtaskenvi ronmentwheretheywereeithercollocatedwithotherteammembersor separatedandonlyabletocommunicatethroughcollaborativesystems. Finally,automatedmethodsweredevelopedtopredictteamperfor mancethroughanalysisofthecommunicationamongtheteammem bers.Wefirstdescribesomeoftheissuesassociatedwiththeareasof learningandtransfer,trustinteams,andautomatedcommunication analysis.Thenweprovideanoverviewofexperimentsperformedto examinetheseissues. TeamLearningandTransferIssues Inmostrealworldsituations,studentslearninonecontext(school) andthenareexpectedtoapplytheirknowledgetoadifferentcontext (job).Inthemilitary,teamsaretrainedtogetherorseparately,andthen areexpectedtoperform(test)eitherincolocatedorindistributedareas oftheworld. Althoughtransferoflearninghasbeenstudiedextensively(for reviewsseeLee,1998;Singley&Anderson,1989),mostresearchhas focusedonindividuallearningandtransfer,ratherthanatteamlevel. However,moststudentsareexpectedtolearningroupsorteams,and oftenareexpectedtoperforminteamsintheworkplaceatalatertime. Grouplearninghasbeenstudiedatasociallevel,butteamtrainingand cognitionhasonlyrecentlybecomeafocusforresearchers(Cooke,Salas, CannonBowers,&Stout,2000). Studiesperformedinmilitarysettingstendtoshownodifferencein achievementbetweendistributedlearnersandcollocatedlearners.In threestudiesthatcompareddistributedandcollocatedlearnersusing Armyparticipants,Hahn(1990)foundnodifferencesbetweendistrib utedandcollocatedlearnersusingAirCombatCommand(ACC).Phelps

AdaptabilityofTeams397 etal.(1991)alsofoundnodifferencesusingtheComputerMediated Communication(CMC)system,andKeeneandCary(1992)showeda benefitfordistributedlearnersovercollocatedlearnersusinginteractive videoteleconferencingtraining.Additionally,BrambleandMartin(1995) foundthatcommunitycollegescouldprovideadequatetrainingforthe militarywhenusingtheU.S.Armystwowayaudio/videoTeletraining Network.Thus,distributedlearningisthoughttobecosteffective,andit canbeanefficientwaytoprovidetraining. Ontheotherhand,distributedlearningmayhaveadifferentmean inginthecontextofcollaborativetasks.Fewstudieshaveexaminedcol laborativedistributedtasksandevenfewerhaveexaminedteam distributedtaskperformance(Barry&Runyan,1995;Frost&Fukami, 1997;Kerka,1996).Computerinternet/webcontextshavegenerally beendesignedfortheindividual(Calvani,Sorzio,&Varisco,1997);how ever,groupwareandCMChasallowedforthepossibilityofthedistrib utedteamwork.ResultsfromstudiescomparingCMCtocolocated learninghavefoundmixedresults.Forexample,studieshavesuggested thatCMCcanbeahindrancetothecreationofmeaning(Mantovani, 1996)andcanleadtolengthydecisionmaking(Hedlund,Ilgen,&Hollen beck,1998)incomparisontocollocatedtasks.Thesestudiesarelimited becausetheytendtofocusonorganizationalandsocialfactorsrather thanteamcognitionandperformance. TrustIssuesinDistributedTeams Collaborativeteamsareusedextensivelyintheworkplace,ingovern ment,andinorganizationstoperformtasks(Keen,1990).Increasingly, studentsinschoolsarelearningingroupsorteamsandareexpectedto performinteamsintheworkplaceatalatertime(Lee,Bond,Scar brough,Gillan,&Cooke,2007).ThroughoutAmericanhistory,military organizationshaveemployedteamworkathierarchicallevels,from infantrysquadsinthefieldtoJointChiefsofStaffteamsintheboard room(Blascovich&Hartel,2008).Withtheincreaseduseofteamsin organizationsworldwide,scholarsinmanagementandpsychologyhave notedthattheconceptoftrustisafundamentalandcriticalelementin increasingteameffectiveness(Sabel,1993),andthattrustisafacilitator ofcohesionandcollaborationbetweenteammembers(Mayer,Davis,& Schoorman,1995). Trustisadifficultconstructtoconceptualize(e.g.,Costa,2003; Mayer,etal.,1995;McAllister,1995).Mostdefinitionsincludeawilling nesstobecomevulnerable(Mayeretal.,1995;Zand,1972).Definitions

398FOLTZETAL. oftrustsometimesincludepositiveoroptimisticexpectations(Elangovan &Shapiro,1998),butGambetta(1988)assertsthatminimally,trustisan expectationthatotherswillactinawaythatisnotdetrimentaltothe trustortrusteerelationship.Trustatteamlevelinvolvesmultipletrustors andtrustees. Antecedentstotrustmustbeinplaceinordertocreatetrustingrela tionshipsindyadsorinteams.Antecedentsarefactorsofperceivedchar acteristicsoftrustworthinessofothers,andapropensitytotrust.Mayer etal.(1995)advancedaparsimonioustrustmodel,theIntegrativeModel ofOrganizationalTrust,inwhichpeopleholdageneralizedpropensity,or anenduringpredisposition,totrust,andpeopleperceivecharacteristics oftrustinothers.Perceivedtrustworthinessisderivedfromobservations andinterpretationsofteammembersbehaviorovertimeandcanvary overtasks,situations,andpeople(Zand,1972). Mayerandcolleaguesproposedthatpeoplelooktoothersabilities, benevolence,andintegrityinordertoformabasisforatrustingrelation ship.Abilityisasetofskillsandcharacteristicsexhibitedwithinadomain (Mayer&Davis,1999).Benevolencerepresentsanaffectivebond betweenpeoplethatinvolvesprovidingmutualsupport(Aubert& Kelsey,2003)andamutualdesiretodowell(Mayeretal.,1995).Integ rityiscomposedofhonestyandmorality,definedasthetrustorspercep tionthatatrusteesactionsadheretoanacceptablesetofprinciples (Mayer&Davis,1999).Thosethreefactorsareproposedtobeindepen dent,inthateachofthemcanvaryalongacontinuum.Onecouldper ceivethatanothermemberoftheteamdemonstrateshighabilityina task,yetisnothelpful(benevolent)tootherteammembers,anddoes notactwithintegrity(isnotethical,forexample).Inadditiontothethree factors,McAllister(1995)alsosuggeststhatevaluationsofcompetence arealsomadebyatrustor.Mayerandcolleagues(1995)incorporated competencewithintheirabilitydimension;however,apersonmightevi dencegreatnaturalabilityinatask,suchasanabilitytomakerepeated freethrowsinbasketball,andyetnotdemonstratecompetence,whichis aseparateconstructwhichincludespossessionofspecificknowledge, skills,orqualifications. ModelingPerformanceinDistributedTeams Teamsprovidearichsourceofinformationabouttheirperformance throughtheirverbalcommunication.Thecontentoftheinformation communicatedprovidesdetailedindicatorsoftheknowledgeteam membershave,whatinformationtheytellothers,andtheircurrentsitu

AdaptabilityofTeams399 ation.Additionally,communicationdatacanprovideinformationabout teamcognitivestates,knowledge,errors,informationsharing,coordina tion,leadership,stress,workload,intent,andsituationalstatus.Anum berofstudieshaveshownthatanalysisofcommunicationcanprovide valuableindicatorsofteamperformance(Achille,SchulzeandSchmidt Nielsen,1995;Fischer,McDonnell&Orsanu,2007;Kiekeletal.,2002).If thecommunicationcanbeanalyzedbycomputer,thenmodelsofperfor mancecouldbeautomaticallyderived. Usingamixtureofcomputationallanguageanalysisandmachine learningtechniques(seeFoltzetal.,2006;Foltzetal.,2008),bothverbal andwrittencommunicationdatacanbeconvertedintoacomputational representationwhichincludesarangeofmeasuresofthecontent(what teammembersaretalkingabout),quality(howwellteammembers seemtoknowwhattheyaretalkingabout),andfluencyandflow(how wellteammembersaretalkingaboutitandtowhom).Underthis approachamodeloftherelationshipbetweenfeaturesofthecommuni cationandthemetricsofperformanceisbuilt,whichcanthenbeusedto analyzeanynewcommunicationstreamandgenerateperformancepre dictions.Thesystemisinitiallytrainedondatafromratingsmadeby SMEs,orbasedonobjectiveperformancemeasures,andthenautomati callylearnstojudgethecommunicationtoproducetheperformance measures.Becausethesystemisfirsttrainedonhumanratedorobjec tivedata,itcanprovidepredictionsoverawiderangeofperformance metrics,aswellasprovidejudgmentswhichraisealarmsifperformance fallsbelowthresholdedlevels. Thisapproachhasbeensuccessfullyusedtoautomaticallygenerate performancemetricsforteams.Theapproachusesamachinelearning algorithmtoautomaticallyassociatecommunicationeventswithindica torsofperformancewithhighdegreesofaccuracyandgeneralization. Themethodhasbeentestedoncommandandcontrolcommunication datasets,includingNavalshipCommandandControlteams,UAVopera tionteams,F16swithAWACcontrollers,andteamstraininginvirtual andliveconvoyoperationsattheNationalTrainingCenter.Acrossthe cases,thetechnologyisabletoprovideaccuratepredictionsofoverall teamperformance.Typicalcorrelationsbetweenthecomputerpredic tionsandsubjectmatterexpert(SME)judgmentsorobjectivemeasures rangedfromr=0.5to0.8andwerecloselyequivalenttotheinterrater judgmentsmadebySMEs(seeFoltz,2005;Foltzetal.,2008).

400FOLTZETAL. SSTRTEAMLABORATORYEXPERIMENTS Twoexperimentswereconductedinahighlyconfigurablelaboratory atNewMexicoStateUniversityusingstudentteamsparticipatingina simulatedSSTRcollaborativedecisionmakingenvironmentusing AptimasDynamicDistributedDecisionmaking(DDD)software.The experimentsassessedlearningandtransferinphysicalcontextcondi tions,trustamongteammembers,andmodeledcommunicationasit relatedtoteamperformance. OVERVIEW:EXPERIMENT1 SixtythreepersonteamsperformedeightmissionsinusingaDDD simulationofSSTRtasksinBosnia.Eachoftheteammembershaddiffer entrolesandreceiveddifferentinformation.TheS3(armyoperations officer/unitstaff)controlledpatrolresourcesthatgatheredinformation orcarriedoutsecurityoperations.TheG3(armyoperationsofficer/gen eralstaff)couldsendinathirdpartytointerveneinproblemsituations, andcouldgatherinformationfrominfantryandpoliceactivity.TheG2 (armyintelligenceofficer/generalstaff)couldsendinintelligence resourcestogatherinformation.IconsappearedonamapofBosniato indicateproblemareas.Aparticipantcouldclickoniconsinordertoview informationaboutthecurrentsituation.Teammembersdetermined whichresourcestosendingivenanevent.Eachteammembercould monitorperformancescorescontinuouslyineachmission. Twomanipulationswereperformedontheteams.First,approxi matelyhalftheteamsusedpushtotalkcommunication,whiletheother halfhadanopencommunicationchannel.Second,atransfertaskwas usedwheresomeoftheteamswerecolocatedandsomeoftheteams weredistributedandwerenotabletoseeeachother. Aftertraining,teamsconductedeightmissions.Therewerefourran domlyassignedcontextbasedconditions:CC(traincolocated,testcolo cated),CD(traincolocated,testdistributed),DC(traindistributed,test colocated),andDD(traindistributed,testdistributed).Trainingwascon ductedinthefirstcontextovermissions16,andtestmissions7and8 wereconductedinthesecondcontext. Trainingwaspresentedintwosessions.Thefirstsessiongaveateam levelintroductiontopeacekeeping,anintroductiontoBosnianhistory, andashortsocial/personalexchangeofinformation.Thesecondsession presentedateamleveloverviewofrolespecificresourcesandvideos examplingteamcommunications.Rolespecificresourcesweretaught

AdaptabilityofTeams401 specificallytoS3(DismountedPatrol,Presence,andTHTresources);to G2(CIMIC,OSINT,andC12imagingaircraftresources);andtoG3(Pres ence,BILAT,andIPTFresources). Procedure Participantsarrivedatdifferentareasofabuildinganddidnotsee eachotherunlesstheywereincolocatedconditions.Colocatedworksta tionswerearrangedsothatteammembersfacedeachotherinatriangu larconfiguration.Participantsindistributedroomsdidnotseeeach other,butcouldverballycommunicateviaheadsets.Participantstrained forapproximately1hours.Teamsthenperformedsixtrainingmissions. Afterthesixthmission,experimenterseithermovedtheworkstationsto distributedroomsortoacolocatedroom,ortheworkstationsremained distributedorcolocatedforthetwotestmissions. TeamPerformanceResults TeamperformancescoreswereautomaticallymeasuredbytheDDD systembycomparingthesolutionsgeneratedbytheteammembersto SMEratedoptimalsolutions.Teamsscoreswerehigherthegreaterthe degreethattheiractionsmatchedthoseoftheoptimalsolution.Results oftheteamperformancescoresacrosstheeightmissionsareshownin Figure22.1andFigure22.2.Theresultssuggestthatteamsusingopen channelstocommunicateperformedbetterthanteamswhousedpush totalk,andthatteamsinthesamecontextattrainingandtestper formedbetterthanteamsthattransferredtodifferentcontexts. Overall,distancepushtotalkteamsperformedbetterthancolo catedteamsintraining;butattest,thecolocatedteamsperformedbet terthandistanceteams.Allteamsimprovedperformanceforthelasttest mission.Ontheotherhand,nodifferenceswerefoundbetweendistance andcolocatedopenchannelteamsduringtraining.Inaddition,attrans fer,teamswhochangedcontext(colocatedtodistanceordistanceto colocated)performedworsethanthoseteamswhostayedinthesame context.Althoughallteamsagainimprovedbetweenthetwotestmis sions,theteamsthatchangedcontextincreasedtheirperformanceata greaterrate(i.e.,theybouncedback).Additionaldetailsonthelearning andtransferbytheteamsmaybefoundinLee,Bond,Scarbrough,Gillan &Cooke(2007).

402FOLTZETAL.

Figure22.1 Teamperformanceinpushtotalkvsopenchannelconditions
80

70 CC CD DC DD

60

50

40 M1 M2 M3 M4 M5 M6 M7 M8 Missions

Figure22.2 Openchannelteamperformanceresults OVERVIEW:EXPERIMENT2(TEAMTRUST) Experiment2implementedacontextbasedtrustperformance manipulation.TheCC,CD,DC,andDDconditionswereusedforExperi ment2,followingthedesigninExperiment1.Perceivedtrustworthiness

AdaptabilityofTeams403 ofteammemberswasassessedinExperiment2toexamineanyrelation shipsbetweenteamtrustandperformance. ColocatedTeams Perceivedtrustworthiness(refertotheTrustIssuessectioninthe introduction)waspredictedtobestrongestforcolocatedteamsbecause presumably,thedevelopmentoftrustinthetaskrequiresfacetoface interaction.Distributedteamswerepredictedtoshowlowerperceived trustworthinesssincetheywouldbeunabletotouchingeographic proximity,communicatingonlyvirtually. DimensionsofPerceivedTrustworthiness Sincethiswasacomplextaskwheresituationsweretechnicallydiffi cultandambiguous,abilityandcompetencewaspredictedtobemost importanttoperceivedtrustworthinessinteams,followingMayerand Davis(1999).Thisresearchwasconductedovertwostudies.Truststudy1 investigatedfactorsinperceivedtrustworthinessandeffectsofthosefac torsonperformanceinfourcontextconditions.InTrustStudy2,amodel oftrustvariables(propensitytotrust,perceivedtrustworthiness,and trustbehaviors)predictedteamperformance. Truststudy1incorporateda48itempostmissionperceivedtrust worthinessquestionnaireadministeredafterfourofeightmissionsinan 8missionpeacekeepingsimulation.Ability,competence,integrity,and benevolencefactorsinperceivedtrustworthinesswereassessedto determinewhichwereimportantincontributingtoteamperformance, sothatashortenedmetriccouldbedesignedforStudy2which addressedthemostimportantdimensionsofperceivedtrustworthiness inacomplexteamtask. TrustStudy1Methods Threepersonteamsparticipatedaspeacekeepersinasimulationas inExperiment1.Aftermissions1,5,7,and8,teamsanswereda48item perceivedtrustworthinessquestionnaire.Fiftyoneteamsparticipated. Thestudywasconductedononedayforapproximatelysixhours,with breaks.ThesameprocedureusedinExperiment1wasusedinthisstudy, exceptthattrustquestionnaireswereadministeredafterfourofthemis sions.MaterialsandStimuliwerethesameasExperiment1.

404FOLTZETAL. PerceivedTeamTrustworthinessResultsandDiscussion Aperceivedtrustworthinessscalewasbuiltusingfourcommonfac torsoftrustworthiness(ability,competence,benevolence,andintegrity). Abilityquestionsinvolvedskill,ability,andcapability.Competenceques tionsassessedqualification,knowledge,andcapacity.Benevolenceques tionsassessedkindness,goodwill,andgenerosity.Integrityquestions assessedhonesty,morality,andethicalbehaviors.Questionswerecom prisedofmy(self)andtheir(other)ability,competence,integrity, andbenevolence.Onequestiondesignedtomeasuremyability,for example,askedtheteammember,Onasevenpointscale,howwould yourteammatesrateyourskillindoingyourjob?Theteammember wasalsoaskedthesamequestionabouttheothertwoteammembers (theirability).Analysesoftheperceivedtrustworthinessquestionnaire consistedofaprincipalcomponentsfactoranalysiswithvarimaxrota tion.Overall,theMyAbility/CompetenceFactorexplained53.2%of varianceinperceivedtrustworthiness,andtogetherwithBenevolence (9.9%),TheirAbility/Competence(6.6%),andIntegrity(5.4%), explained75.1%ofvarianceinperceivedtrustworthiness. PerceivedTeamTrustworthinessinContext:TrainingandTest.A seriesof2(MissionType:train,test)X4(Context:CC,CD,DC,DD)facto rialanalyseswereconductedoneachoftheperceivedtrustworthiness factorstodiscoverwhetheranyorallofthefactorschangeovermis sions.Omnibusanalysesofvariance(ANOVA)resultsshowedthatmost factorscoresincreasedfromtrainingtotest,exceptBenevolence(e.g., trustincreased). Onefactorinperceivedtrustworthinesswhichtranscendedvirtual ness(i.e.,whenteamsareonlyworkinginadistributedfashion)(referto Kirkman&Mathieu,2005)isthatatteamlevel,perceivedtrustworthi nessismainlyaboutmyability/competence.Resultspartiallysupport thesuggestionthatvirtualteamsneedtohavesomecolocatedtouch inordertobuildtrustwhenteamsareworkingonacomplextask. TrustStudy2Methods Ability,competence,integrity,andbenevolencewereinvestigatedin Study1todeterminewhichfactorsweremostimportantinperceived trustworthinessinacomplexteamtask.ResultsindicatedthattheMy Ability/Competencefactorwasmostimportant.Therefore,aftereach mission,teamsinExperiment2ratedhowwelltheyfeltthateachofthe otherteammemberstrustedthemtodotheirownjob(MyAbility/

AdaptabilityofTeams405 Competence),andhowwelltheytrustedeachoftheothermembersto dotheirjobs(TheirAbility/Competence).Thesevariablescompriseda shortenedmetricmeasuringperceivedtrustworthiness.Experiment2 assessesperceivedtrustworthiness,propensitytotrust,andtrustbehav iorsaspredictorsofperformancebycomparingtrustmeasurestohuman ratingsoftrustbehaviors. TwentyoneteamsparticipatedusingthesameprocedureasinStudy 1.However,participantscompletedtheTrustscale(Rotter,1967)before training.Aftermissions1,5,7,and8,teamscompletedthenewabbrevi atedperceivedtrustworthinessmetric. TrustBehaviorCoding.Transcriptionsweremadeofall21teams.To codeutterances,twocodersworkedthroughoneteamtranscript togethertodefinecooperativeandmonitoringlanguage.Cooperative utterancesweredefinedasanswerstoanotherteammembersstate mentswhereagreementorconsensuswasreached.Forexample,state mentssuchas,Iagreewithyou,G2,or,S3,thatsoundslikeaplan, werecodedasonecooperativeutterance.Monitoringutteranceswere questionstootherteammembers,wheresomesuspicion,checking,or monitoringbehaviorwaspresentinthequestion.Cooperativeormoni toringutterancesconsistedofthepercentageofeachteammembers totalutterancesinamission(individualtrustbehaviors).Intercoderreli abilityoncooperativeutteranceswasr=.96.Intercoderreliabilityon monitoringutteranceswasr=.93. TrustStudy2ResultsandDiscussion ReliabilityofPropensitytoTrustScale.Reliabilityofthe25itemRot terscaleshowed=.41.Nodifferencesinpropensitytotrustwerefound betweenteammembersinanyofthecontextconditions(CC,CD,DC, DD),F(3,232)=0.92,ns. ModelPredictingPerformance.Toassesspropensitytotrust,per ceivedtrustworthiness,andtrustbehaviorsaspredictorsofperfor mance,amultipleregressionmodelexaminedelementsoftrustas predictorsofperformance.Predictorvariableswereteampropensityto trust,teamtrust,monitoringbehavior,andcooperatingbehavior.Pro pensitytotrustwassummed.Thesummedvalueofperceivedtrustwor thinessitemscompletedaftermissions1,5,7,and8constitutedteam perceivedtrustworthinessscore.Reliabilityofperceivedtrustworthiness scalewas.96.Themodelaccountedfor12.9%ofthevarianceinperfor mance,F(4,79)=4.06,p=.005.Teamperceivedtrustworthinesspre dictedperformance(B=0.71,t=3.32,p=.001);however,propensityto

406FOLTZETAL. trust,monitoringandcooperativebehaviorsdidnotpredictperfor mance. PerceivedTrustworthinessinContext.Perceivedtrustworthinesspre dictedteamperformancescores:CC(Adj.R2=.12,p=.003);CD(Adj.R2= .25,p<.0001);DC(Adj.R2=.20,p=.0003);andDD(Adj.R2=.08,p= .02).Resultssuggestthattheperceivedtrustworthinessmetricisuseful inmeasuringtherelationshipbetweenthemostimportantperceived trustworthinessfactorsandperformanceinacomplextask. TrustBehaviorsAnalyses.Independentttestsrevealeddifferences betweencooperativeandmonitoringbehaviorswithincontextcondi tions,CC:t(23)=7.32,p<.0001;CD:t(27)=6.52,p<.0001;DC:t(15)= 8.38,p<.0001;andDD:t(15)=4.25,p<.0003).Meandifferencescores betweenteamcooperativeandmonitoringbehaviorswerecomputed andweretermedtrustbehaviors(meancooperativemeanmonitoring =overalltrustbehaviors).AnANOVArevealeddifferencesintrustbehav iors,F(3,80)=4.22,p=.008.Comparisonsrevealedthattrustbehaviors inCC(M=0.13,SE=0.02)weregreaterthaninCD(M=0.08,SE=0.01), andweregreaterthanteamsinDD(M=0.07,SE=0.02).CDtrustbehav iorsweregreaterthanbehaviorsshowninDC(M=0.14,SE=0.02). MODELINGTEAMPERFORMANCEWITHAUTOMATED COMMUNICATIONANALYSIS Thegoaloftheperformancemodelingwastoapplycomputational languagemodelingtechniquestoanalyzetheteamcommunicationin ordertopredicttheteamsperformance.Thecompleteteamtranscripts fromExperiment1wereanalyzedusingLatentSemanticAnalysisanda varietyofothercomputationallanguagetechniques.Detailsofthemod elingapproachcanbefoundinFoltzetal.(2005)andFoltzetal.(2008). Theperformanceforeachteamforeachmissionisthenpredictedbased onamodeloftheperformanceoftheotherteamsontheirmissions. Overall,themodelwasabletoprovideaccuratepredictionsoftheteam performancescore(r=0.62,p<.0001).Figure22.3showsascatterplotof theobservedversuspredictedteamscores.Overalltheresultsindicate that,giventhecommunicationfromtheteam,themodelcanaccurately determinehowwelltheteamisperforming.Thisapproachpermitsfast andaccuratemodelingofteamsandcanbeusedforrapidalarmsforper formanceproblemsinteamsaswellasallowingdetectingavarietyof cognitiveerrorsthatmayleadtofailuresandtheabilitytoprovidefeed backtoteamsandcommanders.

AdaptabilityofTeams407

Figure22.3 Predictedvs.observedteamperformance GENERALDISCUSSION Thisresearchstudiedalargenumberoftemporaryteamswithspe cificrolesandfunctionsworkingdynamically,interdependently,and adaptively(e.g.,Salasetal.,1992)toachievethegoalofsuccessful peacekeepinginacomplex,multiplemissionteamsimulation.The researchfocusedonlearning,transfer,trust,andcommunicationinorder tomodelwhichfactorshaveimportanteffectsontheperformanceofthe teamsinSSTRoperations.Overalltheresultsaddressissuesof1)transfer effectswhenchangingcontextswhilelearningandperformingteam tasks2)culturalandsocialaspectsofteamperformance,includingatti tudes,stereotypes,andtrustandtheirroleinteamperformance,and3) automaticanalysisofcommunicationappliedtomodelingtheperfor manceofteams.Belowwediscussadditionalissuesrelatedthesefac tors. TeamTrustinContext MayerandDavis(1999)suggestthatteamtaskcomplexitymight makesomeperceivedtrustworthinessfactorsmoreimportantthanoth ers.Thepeacekeepingsimulationinthisresearchwasdifficultand ambiguous,andfindingsindicatethatatrusteesabilityandcompetence

408FOLTZETAL. werethemostimportantfactorsinperceivedtrustworthiness.Integrity andbenevolencedidnotplaysignificantrolesinpredictingperformance. Themostimportantantecedentstotrustonteamsarebeliefsaboutoth ersperceptionsofonesownabilityandcompetence.TheMyAbility/ Competenceperceptionstrengthensoverthecourseofworkperformed byateam.Ifonefeelsableandcompetentatdoinghisorherjob,one willbelievethatothersontheteamperceivethatabilityandcompe tence,andreciprocaltrustwilldevelop. Whenteamsareworkingonacomplextaskincolocatedanddistrib utedgeographicareas,ifteammembersareonlyassessingperceptions oftheirownabilityandcompetence,contextdoesnotmatter,nordoes touch,asshowninExperiment1wheretherewerenodifferences betweencontextbasedteams.WhenMyandTheirabilityandcom petenceteamlevelratingswerecombinedinExperiment2toassessper formance,thecombinedmetricpredictedteamperformanceacrossall contextconditions.Perceivedtrustworthinessfactorsrequiredtouch onlywhenbenevolenceandintegrityweretakenintoaccount. TrustNeedsTouch? CDandDCteamsindirectlytestedtheHandy(1995)trustneeds touchassertion,investigatingwhetherteamsthathadanextended opportunitytoworkfacetofaceeitherbeforeoraftermovingintosepa ratecontextsmightshowgreatertrustversusteamsthatwereseparated. Trustbehaviorsweregreatestwhenteamsworkedfacetoface.Ifteams weregivenachancetotouchandthenmovetoseparatecontexts, trustbehaviorswerelowerthanmeetingfacetoface.However,findings suggestthatatleastsometouchbeforemovingintoseparatecontextsis betterthanifteamsworkseparatelyandtouchafterwards. KirkmanandMathieu(2005)challengetraditionaldefinitionsofvir tualteams,andsuggestthatlevelsofvirtualityaremoreimportantthan ateamsgeographicdispersion.Colocatedteamsinthisstudyoperatedin proximity,butvirtuallytheysharedthesametechnologicalequipmentas distributedteams.Wefoundthatabilityandcompetencefactorstran scendedcontextualbarriers.However,analternativeexplanationmight bethatpeopleworkedonacomplextasktotheexclusionofphysical context,andthatperceivedtrustworthinessfactorsimportanttothevir tualtaskweresimilaracrossasinglelevelofvirtualness.SmithandVela (2001)suggestthatonesdefaultpolicyistoencodephysicalcontextin memory,becauseoneisawareofonessurroundingsduringlearning. However,conditionsduringlearningmightcauseparticipantstofocuson

AdaptabilityofTeams409 learningthetasktotheexclusionofthephysicalenvironment,andso physicalcontextchangewillshowlessofaneffect(Smith&Vela,2001,p. 206).Perhapsvirtualenvironmentsworktosuppress(Matzel,Schacht man,&Miller,1985;Mazur,1986)contextifthetaskiscomplex.Future researchshouldvarytheamountofvirtualnessinphysicalcontextsand varytaskcomplexityduringlearningtodeterminewhetherreciprocal perceptionsofabilityandcompetenceinperceivedtrustworthinessare factorswhichtranscendphysicalcontext,orwhethertheydependto someextentuponlevelsofvirtuality. CONCLUSIONS Theexperimentsdescribedinthischapterareuniquebecausethey studytheknowledgeacquisitionofmanyteamsinvaryingcommunica tionanddistributedcontextsovermultiplemissions.Whileonlyapor tionoftheresultsarediscussedhere,thefindingshighlightimportant aspectsforunderstandingandmodelingteaminteractionsindistributed complexdecisionmakingtasks. Intheexperiments,wedirectlyexaminedtheeffectsofchangesin contextonteamsincludingpushtotalkversusopenchannelandcolo catedversusdistributed.Theresultsresembledwhatonewouldexpect achangeofphysicalcontextandsoftwarecontextresultedinmorediffi cultiesthanstayinginthesamecontext.Amoregeneralconclusionis thatindividuals(andteams)needtoacquireknowledgeontheaspectsof thetechnologytheyareusingbeforefocusingonthetargetknowledge availableinthesituations.Oneneedstoexaminetheaspectsofthehard ware(andsoftware)tolocatepotentialdifficultiesbeforefocusingon whetherthegoalsoflearningcanbeachieved. Theresultsfurtherindicatedtheimportanceofcommunicationtrain ing.Teamswhomaybecomposedofpeoplewhodonotknoweach other(fromdifferentpartsofthecountry)mayneedtobecomecomfort ableincommunicatingwitheachotherbeforeateamtaskcanbeper formed.Unfortunatelypeopleareoftenexpectedtolearnhowto communicatethroughlearnbydoingorbyexposure. Thesestudiesindicatethatmanydifferenttypesofcontextsmight existfortheindividualorteam.Thereisthephysicalcontext(same room/differentroom),thehardwarecontext(largeimposing/unfamiliar orfamiliar)andthesoftwarecontext(SASOscountrymap&stories, UAV/UGVmaps&buildings).Ifthecontext(simulationorhardware)is particularlycompelling,thenthephysicalcontext(room,building,chairs,

410FOLTZETAL. etc.)maynotplayaslargearole.However,identifyingwhatthecritical contextisfortheteamorindividualmaybethehardertask. Teamperformancecanresembleindividualperformance.Forinter ference,adecrementwouldbeexpectedattransfertoadifferentcon text,butafteralittlemoretraining,theperformancewilljumpbackto theoriginalperformancebeforetransfer.Formostteams,thisresultwas found.Thus,inmostcases,ateamwhoperformedSSTRtasksinone countrycouldbebroughttogetherinanothercountrywithonlyatempo rarylossofperformance.Wedonotyetknowhowmuchofadecrement wouldoccurorhowlongitwouldtaketobouncebackbutatleastwedo knowthattheknowledgegainedisnotlostpermanentlyanddoestrans fertothenewsituation. Finally,theresultsfromthestudiesperformedshowthatonecan assesstherelationshipofperformanceinmissionstoawiderangeof metricsofteamperformance,culture,communication,andtrust.The goalbeyondthepresentworkistosynthesizealltheresultsintoaframe workforteamperformancemetricsinSSTRoperations.Bydetermining theimportantinteractionsamongthemeasuresofteamcognition,cul ture,communication,andtrust,onecandevelopadditionalgeneraliza tionsthatcanbeusedforassessmentofteamperformance.Ongoing workisfocusingonadaptingthemostpromisingmeasuresintotoolsfor performingembeddedassessmentbefore,duringandafterdeployment ofWarfightersandotherSSTRteammembers.Thepresentsetofexperi mentshelpsdeterminethemeasuresandthetechnologiesthatcanbe usedtomeasureperformancethatcanbestbeadaptedtoproviding effectiveassessmenttools.Ultimately,thesemethodsandtechnologies canbeintegratedintocollaborativesystemstoprovideenhancedsup portforcommandersandteammemberssotheyarebetterabletoman ageandcomprehendtheinformationflowingfromteammemberswhile performingcomplexoperations. ACKNOLWEDGMENTS TheauthorsaregratefulforhelpwiththeexperimentsfromPam Scarbrough,CarloGonzalez,DeborahRussell,DanMalloy,AaronPerez andJeremyTostandformodelingassistancefromMarkRosenstein.Cor respondencemaybeaddressedtopfoltz@nmsu.edu.

AdaptabilityofTeams411 REFERENCES
Achille,L.B.,Schulze,K.G.,&SchmidtNielsen,A.(1995).Ananalysisof communicationandtheuseofmilitarytermsinNavyteamtraining,MilitaryPsychology, 7,95107. Aubert,B.A.,&Kelsey,B.L.(2003).Furtherunderstandingoftrustandperformance invirtualteams.SmallGroupResearch,34,575618. Barry,M.,&Runyan,G.B.(1995).AreviewofdistancelearningintheU.S.military. TheAmericanJournalofDistanceEducation,9,3756. Blascovich,J.J.,&Hartel,C.R.(2008).Humanbehaviorinmilitarycontexts. Washington,DC:NationalAcademiesPress. Bramble,W.,&Martin,B.(1995).TheFloridateletrainingproject:Militarytraining viatwowaycompressedvideo.TheAmericanJournalofDistanceEducation,9,626. Calvani,A.,Sorzio,P.,&Varisco,B.(1997).Interuniversitycooperativelearning:An exploratorystudy.JournalofComputerAssistedLearning,13,271280. Cooke,N.J.,Salas,E.,CannonBowers,J.A.,&Stout,R.(2000).Measuringteam knowledge.HumanFactors,42,151173. Costa,A.C.(2003).Workteamtrustandeffectiveness.PersonnelReview,32,605622. Elangovan,A.R.,&Shapiro,D.(1998).Betrayaloftrustsinorganizations,Academy ofManagementReview,23,547566. Fischer,U.,McDonnell,L.&Orsanu,J.(2007).Linguisticcorrelatesofteam performance:Towardatoolformonitoringteamfunctioningduringspacemissions. Aviation,SpaceandEnvironmentalMedicine,78,5,pp.,B86B95. Foltz,P.W.(2005).ToolsforEnhancingTeamPerformancethroughAutomated ModelingoftheContentofTeamDiscourse.InProceedingsofHCIInternational.. Foltz,P.W.,LaVoie,N.,Oberbreckling,R.Rosenstein,M,Psotka,J.&Chatham,R. (2008)DARCAAT:DARPACompetenceAssessmentandAlarmsforTeams.Interservice/ IndustryTraining,SimulationandEducationConference(I/ITSEC2008),Orlando,FL., December14,2008. Foltz,P.W.,Martin,M.J.,Abdelali,A.,Rosenstein,M.&Oberbreckling,R.(2006). AutomatedTeamDiscourseModeling:TestofPerformanceandGeneralization.Proc.of the28thAnnualConferenceoftheCognitiveScienceSociety,Vancouver,BC. Frost,P.J.,&Fukami,C.V.(1997).Teachingeffectivenessintheorganizational sciences:Recognizingandenhancingthescholarshipofteaching.Academyof ManagementJournal,40,12711281. Gambetta,D.(1988).Trust:Makingandbreakingcooperativerelations.Blackwell: NewYork. Hahn,H.(1990).Distributedtrainingforthereservecomponent:Remotedelivery usingasynchronouscomputerconferencing(ReportNo.2Q263743A794).Boise,ID:Army ResearchInstitute. Handy,C.(1995).Trustandthevirtualorganization.HarvardBusinessReview,73, 4050. Hedlund,J.,Ilgen,D.R.,&Hollenbeck,J.R.(1998).Decisionaccuracyincomputer mediatedversusfacetofacedecisionmakingteams.OrganizationalBehaviorand HumanDecisionProcesses,76,3047. Keene,D.,&Cary,J.(1992).EffectivenessofdistanceeducationapproachtoU.S. Armysearchcomponenttrainings.InM.G.Moore(Ed.),Distanceeducationforcorporate andmilitarytraining(ACSDEResearchMonographNo.3,pp.97103).UniversityPark, PA:PSU;AmericanCenterfortheStudyofDistanceEducation.

412FOLTZETAL.
Kerka,S.(1996).Distancelearning,theInternet,andtheWorldWideWeb(ERIC DigestNo.168).Columbus,OH:ERICClearinghouseonAdult,Career,andVocational Education,CenteronEducationandTrainingforEmployment. Kiekel,P.A.,Cooke,N.J.,Foltz,P.W.,Gorman,J&Martin,M.J.(2002).Some promisingresultsofcommunicationbasedautomaticmeasuresofteamcognition. ProceedingsoftheHumanFactorsandErgonomicsSociety46thAnnualMeeting. Kirkman,B.L.,&Mathieu,J.E.(2005).Thedimensionsandantecedentsofteam virtuality.JournalofManagement,31,700718. Lee,A.Y.(1998).Transferasameasureofintellectualfunctioning.InS.Soraci&W.J. McIlvane(Eds.),Perspectivesonfundamentalprocessesinintellectualfunctioning:A surveyofresearchapproaches,Vol.1(pp.351366).Stamford,CT:Ablex. Lee,A.,Bond,G.,Scarbrough,P.,Gillan,D.&Cooke,N.(2007).Teamtrainingand transferindifferingcontexts.CognitiveTechnology,12,1729. Leifer,R.,&Mills,P.K.(1996).Aninformationprocessingapproachfordesigning controlstrategiesandreducingcontrollossinemergingorganizations,Journalof Management,22,112137. Mantovani,G.(1996).Newcommunicationenvironmentsfromeverydaytovirtual. Bristol,PA:TaylorandFrancis. Matzel,L.D.,Schachtman,T.R.,&Miller,R.R.(1985).Recoveryofan overshadowedassociationachievedbyextinctionoftheovershadowingstimulus. LearningandMotivation,16,398412. Mazur,J.E.(1986).Learningandbehavior.EnglewoodCliffs,NJ:PrenticeHall. Mayer,R.C.,&Davis,J.H.(1999).Theeffectoftheperformanceappraisalsystemon trustformanagement:Afieldquasiexperiment.J.ofAppliedPsychology,84,123136. Mayer,R.C.,Davis,J.H.,&Schoorman,F.D.(1995).Anintegrativemodelof organizationaltrust.AcademyofManagementReview,20,709734. McAllister,D.J.(1995).Affectandcognitionbasedtrustasfoundationsfor interpersonalcooperationinorganizations.AcademyofManagementJournal,38,2459. Phelps,R.,Wells,R.,Ashworth,R.,&Hahn,H.(1991).Effectivenessandcostsof distanceeducationusingcomputermediatedcommunication.TheAmericanJournalof DistanceEducation,5,719. Rotter,J.B.(1967).Anewscaleforthemeasurementofinterpersonaltrust.Journal ofPersonality,35,651665. Sabel,C.F.(1993).Constitutionalorderinginhistoricalcontext.InF.W.Scharpf(Ed.), Gamesinhierarchiesandnetworks(pp.65123).Boulder,CO:Westview. Salas,E.,Dickinson,T.L.,Converse,S.A.,&Tannenbaum,S.I.(1992).Towardan understandingofteamperformanceandtraining.InR.W.Swezey&E.Salas(Eds.), Teams:Theirtrainingandperformance(pp.329).Norwood,NJ:Ablex. Singley,M.K.,&Anderson,J.R.(1989).Thetransferofcognitiveskill.Cambridge, MA:HarvardUniversityPress. Slavin,R.(1996).Researchoncooperativelearningandachievement:Whatwe know,whatweneedtoknow.ContemporaryEducationalPsychology,21,4369. Smith,S.M.,&Vela,E.(2001).Environmentalcontextdependentmemory:Areview andmetaanalysis.PsychonomicBulletinandReview,8,203220. Wickens,D.D.(1987).Thedualmeaningsofcontext:Implicationsforresearch, theory,andapplications.InD.S.Gorfein&R.R.Hoffman(Eds.),Memoryandlearning: TheEbbinghauscentennialconference(pp.135152).Hillsadale,NJ:LEA. Zand,D.E.(1972).Trustandmanagerialproblemsolving.AdministrativeScience Quarterly,17,229239.

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T A N D L A U R E L A L L E N D E R

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T AND LAUREL ALLENDER

This book, produced by the Partners of the Army Research Laboratory Advanced Decision Architectures Collaborative Technology Alliance, also serves as the Final Report for DAAD19-01-2-0009.

SECTIONIV.COMMUNICATINGINFORMATIONACCROSSTHETEAM

SECTIONIV COMMUNICATINGINFORMATIONACROSSTHETEAM

324SECTIONIV

Chapter

23

23. ASYNCHRONOUSCOMMUNICATIONOFARMYOPERATIONSORDERS

PhilipJ.Smith,Ph.D. TheOhioStateUniversity,Columbus,OH JenniferBower UnitedStatesMilitaryAcademy,WestPoint,NY AmySpencer TheOhioStateUniversity,Columbus,OH

INTRODUCTION Thefocusofthisresearchisonthesupportofasynchronouscommu nicationofplansandplanmodificationsamongmilitaryunits.Theneed tobettersupportasynchronouscommunicationinthemilitaryisevident, especiallygiventheincreasinglydistributedbattlefieldexpectedinthe U.S.Armyofthefuture.Insuchadistributedbattlefield,thecommand ersandplannersofoperationswillnotbecolocatedand,becauseoftime constraints,willattimesrequireasynchronouscommunication.This increasedrelianceondistributedresponsibilityalsoplacesmoreempha sisonindividualresponsibilitytomakelocaldecisions,thusmaking sharedsituationawarenessandsharedunderstandingofintenteven moreimportant. Inthefollowing,twoempiricalstudiesarediscussedthatlookat issuesassociatedwiththecommunicationofArmyoperationsordersand taskings.Inthefirstexperiment,46ROTCcadetseachreceivedacom panyoperationsorder,andbasedonthatwererequiredtowriteapla toonoperationsorderfollowedbytworecallquestions.Halfofthe cadetsreceivedthecompanyoperationsorderinaconventionaltextfor matthatincludedastaticmapoverlay.Theotherhalfreceivedthesame operationsorder,butwith4subsectionspresentedwithattachedmulti 413

414SMITH,BOWER,ANDSPENCER mediapresentationsthatincludedsynchronizedanimationandvoice narration.Thegroupthatviewedthemultimediapresentationaccurately recalledmissioncriticalinformation26%moreoften. Inthesecondstudy,aformativeevaluationwascompletedmaking useofamultimediacommunicationstooldesignedtosupportandstudy asynchronouscommunication.Whileparticipatinginajointforcesexer cise,tencaptains,majors,andcolonels(4soldiersfromtheUnited States,2fromCanada,2fromFrance,1fromIsraeland1fromGermany) hadtheoptionofusingthismultimediacommunicationtoolwhenever theyfeltitwouldhelpthemtocommunicateinformationtocommand ersinotherunits.Theycreatedatotalof15messages.Twoofthemes sagesconsistedofonewaycommunications.Theremaining13were asynchronousdialogs.Inthesemessages,theofficers: Madeextensiveuseofpointing,drawingandembeddedwritten notes. Usedtheseasynchronousdialogstodetectandrepairmiscon ceptionsthatarosefromlivefacetofacebriefings(6/13dialogs). Usedtheseasynchronousdialogstoshareexpertisewhiledevel opingaplan(13/13dialogs). Detailsareprovidedbelow. STUDY1:THEIMPACTOFPRESENTATIONMEDIUMONRECALLAND COMPREHENSION Study1wasaformalexperimentcomparingtheimpactofpresenta tionmediumontherecallandcomprehensionofOperationsOrders (OPORDs).UnderlyingStudy1isthehypothesisthatuseofthecurrent textformatofOPORDsmakesithardertoasynchronouslycommunicate contentthathasimportantdynamicvisualdimensionstosubordinate leadersinadistributedbattlefield.Thismakesitmoredifficulttoestab lishcommonground(ClarkandBrennan,1991)andtothusensurean understandingofthesituationandthecommandersintentandorders (ShattuckandWoods,2000).Basedonthishypothesis,theprimary researchquestionaddressedbythisstudywastherefore: Cantheinclusionofasynchronousmultimediasupplementstoa writtenOPORDimpartmoreaccurateandcompleteunderstanding/ recallofkeypointsinthatOPORD,ascomparedtothecurrentmedium usedforcommunicatingOPORDs(usingtextandstaticgraphics)?

AsynchronousCommunicationandCoordination415 Study1Background Inordertocontrasttheimpactofamultimediapresentationof OPORDcontentwiththatofatextbasedpresentation,aspecificmulti mediapresentationtooldevelopedatOhioStateUniversitywasused, theCollaborativeSLideANnotationToolorCSLANT(Bower,2004;Chap manandSmith,2003).Thiscommunicationtoolallowstheusertocreate aslideshowwhereeachslidecanhaveavoicerecordingthatissynchro nizedwithpointingandwithdynamicannotationsmadeonavisualdis play(slide)whilecreatingthevoicerecording.Italsosupports asynchronousdialogs,astherecipientofamessagecancreateandsend responsestomessagesthathavebeenreceived. FeaturesoftheMultimediaCommunicationTool.Althoughthisstudy madeuseofaspecificasynchronouscommunicationtool(CSLANT),what isimportantarethespecificfunctionssupportedbythistool.These includethefollowing: Usesslidestochunkorsegmentthevisualdata. Supportssynchronizedrecordingofvoice,pointing,anddynamic annotations. Allowsthedesignofgraphicannotationspriortorecording, whichcanbedisplayedandthenhiddenagainwithasingleclick duringavoicerecording. Allowsuseofaslideshowwithintheslideshowtopresenttext bulletsthatreinforcethespokenmessage. Supportsasynchronoustwowaycommunication. Thesefeaturesbuilduponpreviousresearchanddevelopment projectsandcommercialapplicationsincludingFreestyleamultimedia communicationssystemwhichincorporatedimagecapture,scanners,fax interfacecards,email,andhighresolutiongraphics(developedbyWang Laboratories,1988);MicrosoftResearchAnnotationSystem(MRAS), whichsupportedstreamingvideoonthewebwithpersonalizedannota tionsbythosewhoviewedit;andfeaturesincorporatedincommercial slideshowandscreencapturesoftwaresuchasCamtasia,Captivate, CaptureProfessional,FullShot,HyperSnapDX,PowerPointandSnagIt. Noneofthesepreviousefforts,however,haveincludedthefullsetof featuresofferedbyCSLANT.Furthermore,thesepreviouslydeveloped toolshavefocusedprimarilyonsupportingonewaycommunication (presentations)ratherthantwowaydialogs. Inadditiontoconsideringwhatfunctionsshouldbeprovidedbysuch acommunicationtool,itisimportanttorecognizethatprovidingkeyfea turesisonlyhalfoftheequation.Thesefeaturesmustthenbeapplied

416SMITH,BOWER,ANDSPENCER effectivelybythemessagecreator.FaradayandSutcliffe(1997)havepro videdguidelinesfordesignersofmultimediapresentations.Afewof theseguidelinesandconsiderationsthatwereappliedinthepresenta tionsinthisstudyareasfollows: Animatedobjectscuedbyspeechtrackarebetterrecalledthan thesameobjectswithoutspeechcueing. Arrowsymbolsshiftfixationtoitemsthearrowspointto. Peoplesattentionisdrawntomotion. Objectmotioncanbeusedtocontrolattentionandviewing order. Gradualrevealingoflabels,symbols,andobjectscanbeusedto controlviewingorder. Anobjectanditslabelshouldappeartogethertoimproveidenti fication. Allowreadingtimeaftercueingalabel. Suchguidelinesaresupportedbycognitivetheory(Mayer,2005)aswell asbyempiricalstudiesfocusingonmultimedialearning. Anexamplefromoneofthemultimediapresentationsusedinthis studyisshowninFigure23.1.Inthisparticularpresentation,thetextbox (whichwasyellowaspresentedtoparticipants)isessentiallyatitleor advanceorganizerindicatingthattheassociatedmessageisaboutwhere toassumedefensivepositions.Itappearedonthedisplayasthecom manderintroducedthisgraphicinhisvoiceover.Thecommanderalso drewarrowspointingtotheareaswherethesedefensivepositionswere locatedashewastalkingaboutthoseareas.

AsynchronousCommunicationandCoordination417

Figure23.1 Screenshotfromthemultimediapresentation Study1Methods Inthisstudy,463rdand4thyearcollegemilitarysciencestudents wereaskedtoreadandinterpretacompanylevelOPORD(fromVisconti, 2002)andwriteaplatoonOPORD.ThetextofthisOPORDincludeda numberofreferencestothelocationoftheenemyforcessuchas[The enemyBrigade]isintheprocessofmovingfromtheircurrentstronghold 200milestooursouth.Theywerethenaskedtoalsoansweraquestion toassessrecall/understating.23cadetsreceivedthetextOPORDpre sentedonacomputerdisplay,alongwithacorrespondingmilitarymap showingfriendlyandenemyforcepositions(textonlycondition).The other23cadetsreceivedthesametextOPORD,withlinkstomultimedia presentationscreatedusingCSLANT.Additionalmethodologicaldetails areprovidedinSmith,etal.(2007). Study1ResultsandDiscussion RecallAssessment:IdentificationofEnemyApproach.TheCSLANT groupcorrectlydeterminedtheenemydirectionofapproach22/23times (95.6%),whilethetextgroupwaslesssuccessful,with16/23(69.6%)

418SMITH,BOWER,ANDSPENCER answeringcorrectly.Thisdifferenceisstatisticallysignificantatp<.05.An answerwasconsideredcorrectifthecadetdrewthearrowindicatingthe enemysapproachfromasoutherlydirection. Intermsoftheprimaryresearchquestion,thisisthemostinforma tivefindingofStudy1.Itprovidesevidencesuggestingthepotentialfor multimediapresentationstomoreeffectivelyemphasizeandcommuni catethoseportionsofanOPORDthatarehighlyvisual(maporiented)in nature.Note,however,thatbecauseoftimeconstraintsontheavailabil ityoftheparticipants,itwasnotpossibletoconductamorecomplete evaluationoftheirunderstandingandrecallofallofthecontentinthe OPORDimportanttoaplatooncommander.Suchamoredetailedstudy, usingdifferentvariationsontextalonevs.textplusmultimediapresenta tionswouldbevaluableinordertogainfurtherinsightsintohowtouse differentmediatomoreeffectivelycommunicatedifferentkindsofcon tentwithinanOPORD. Inaddition,itshouldberecognizedthatinanactualbattlesetting withfullytrainedsoldiers,itislikelythattheywouldquicklybecome awareofamajorconcernsuchasthedirectionofapproachofthe enemy.Thus,thesefindingsshouldbetakenasacautionregardingthe potentialformoresubtlebutstillimportantpointstobemisunderstood, missedorforgottenwheninformationispresentedinanOPORDinaless thaneffectiveformatormedium.Thesefindingsalsosuggestthatthe useofmultimediacommunicationtoolscanbeveryhelpfulforasynchro nouscommunicationofmessagesthatfocusoninformationthathasa strongspatial/visualcomponent. EvaluationofOPORDCompleteness.Thesectionsoftheplatoon OPORDswrittenbythe46cadetswerealsoexaminedtoascertain whetherthecadetsmadereferencetokeypointsthatwereexpectedby theCommandertobeincludedinaplatoonOPORD.Regardingthe approachoftheenemyfromthesouth,inthemultimediagroup10/23 (43.5%)referredtotheenemyapproachfromthesouth,while12/23 (52.2%)inthetextonlygroupmadethisreference.Thisdifferenceisnot statisticallysignificant.Thus,eventhoughalmostallofthecadetsinthe multimediaconditiondemonstratedthattheyknewtheenemywas approachingfromthesouth,fewerthanhalfofthemchosetoinclude thiscriticalinformationintheplatoonOPORDsthattheywrote. Similarly,alargepercentageofthecadetsinbothtreatmentgroups failedtoproviderequiredsquadlevelinformationintheirplatoon OPORDsregardingtaskstosquadleaders.Inthemultimediagroup10/23 (43.4%)wrotetaskstosquadleadersintheirOPORDS;inthetextonly

AsynchronousCommunicationandCoordination419 group14/23(60.8%)wrotetaskstosquadleaders.Thesedifferences werenotstatisticallysignificant. Thus,intermsofthegenerationoftheplatoonOPORDS(asopposed torecallofthecontentsofanOPORD),therewasnoevidencetoindicate thattheuseofmultimediapresentationsincreasedtheinclusionof requiredcontent.SincetheOPORDwasstillavailablewhilethecadets werewritingtheirplatoonOPORDS,thefactthatmanyofthemleftout importantinformationislikelyduetoinsufficienttrainingandexperience withwritingOPORDsatthispointintheircareers,ratherthananindica tionofwhetherornottheycouldrememberthatcriticalinformation. STUDY2:USEFULNESSANDUSABILITYASSESSMENTOFANASYNCHRO NOUSMULTIMEDIACOMMUNICATIONTOOLTOSUPPORTBATTLE PLANNINGANDEXECUTION Study1illustrated,thatforvisuallyorientedinformation,multimedia presentationscanenhanceunderstandingandrecall.Study2buildsupon thesefindings,providingacasestudylookingattheuseofsuchmultime diapresentationstosupportasynchronouscommunication. Study2Overview Thegoalofthissecondstudywastohelpassessthepotentialuseful nessandusabilityofasynchronousmultimediacommunicationtoolsin thecontextoftheplanningandexecutionofabattle,andtoassessthe perceivedusefulnessofparticularfeaturesforinclusioninsuchatool.To completethisevaluation,CSLANTwasusedtostudycommunication withinadistributedmultinationalforceconductingasimulatedjoint forcesoperationintheBalkans.Thegeneralcharacteristicsofthistool weredescribedinthesectiononStudy1.Tenexperiencedofficersfrom5 countries(Canada,France,Germany,IsraelandtheUnitedStates)pro videdinputforthisevaluationastheyusedCSLANTtoexchangeinforma tionduringthebattle. Study2ResearchQuestions Thefocusofthisstudywasthecompletionofaformativeevaluation ofthisasynchronousmultimediacommunicationstool,CSLANT,inthe contextofacomputerstagedbattleconductedbyamultinationalcoali tionofCanadian,French,German,Israeli,andU.S.Armyforces.More specifically,threequestionswereaddressed:

420SMITH,BOWER,ANDSPENCER Wouldthisgeneraltypeofmultimediacommunicationtoolbeof valueforArmyoperations? Forwhatuseswouldsuchatoolbeofvaluetoanarmy? Withinthedesignofthisparticularmultimediacommunication tool,whichfunctionsareperceivedtobemostimportant? Theresearchwasconductedinthecontextofplanningandbattle executionusingCERDECsWebC2Portal,atoolthatmakesuseofa semanticrepresentationofanOperationsOrder,inordertosupport accesstothisOrderthroughthevarietyofdifferentplanningtoolsused bythearmiesofdifferentcountries.Inthiscontext,amorespecificform ofthefirstquestionwasalsoaddressed: Wouldthisgeneraltypeofmultimediacommunicationtoolbeof valueinsupportingexplanationandasynchronouscommunicationdur ingtheplanningandexecutionofjointforcesusingCERDECs WebC2Portalinthepreparationforandtheexecutionofbattles? Study2Methods CSLANT(describedearlier)wastheasynchronouscommunication toolusedinthisstudyaswell(seeFigure23.2). Study2Setting.ThisformativeevaluationofCSLANTwasembed dedinalargerstudyofothertoolsassociatedwithCERDECsSINCE(Sim ulationandC2InformationSystemsConnectivityExperiments)Program. CSLANTwasstudiedinthiscontextasatooltosupportexplanationand communicationasamultinationalforcedeveloped,executedandrevised plansforabattleinaregionwithgeographylikethatoftheBalkans. ThefriendlyforcesconsistedofamultinationalDivision.Withinthis Division,the1stBrigadehadthreeBattalions.The1stand3rdBatalions wereGermanforces,whilethe2ndBattalionwasamultinationalforce madeupofCanadian,Israeli,andU.S.companies.Eachoftheseforces hadaccesstotheplanningtoolsnormallyusedbyitsowncountrys forces,aswellasaccesstothecollaborativeplanningenvironmentpro videdbyCERDECsWebC2Portal.ThePortalnotonlyprovidedaninter facefordevelopingacourseofaction,italsowaslinkedtotheplanning systemsforeachoftheparticipatingcountries.Thislinkage,whichmade useofasemanticrepresentationoftheoveralloperationsorder,madeit possibletodevelopplanswithintheWebC2Portalplanningenvironment andtoexporttheseplansintothetoolsuniquetoeachoftheparticipat ingnationalforces.

AsynchronousCommunicationandCoordination421

Figure23.2 CreatingaCSLANTmessagetosendtoanotherBattalion Study2TheBattleContext:TheOperationsOrder.Thegeneralset tingforthisexercisewasanOPORDthatwasbriefedtoallofthepartici pantsatthebeginningoftheexercise.ThisOPORDrepresentedabattle planinfictitiouscountriesforahypotheticalmilitaryencounter.Asthe exerciseproceeded,Commandersforthedifferentunitswereallowedto adapttheirplans,communicatingasnecessarytocoordinatetheiractivi ties. Study2Participants.Tenexperiencedsoldierswhowerepartofthe largerCERDECstudyparticipatedintheformativeevaluationofCSLANT. Thisincludedsoldierswiththefollowingranks:Major(5),Captain(1),Lt. Colonel(2),andColonel(2)withameanof21.7yearsofmilitaryexperi ence(range=1230years).FoursoldierswerefromtheUnitedStates,2 fromCanada,2fromFrance,1fromIsrael,and1fromGermany. Study2Procedure.AdescriptionandwalkthroughofCSLANTand itsfunctionswaspresentedinabriefinggiventoalloftheparticipantsas agroup.Theywerethenaskedtowatchforopportunitieswherethey thoughtaCSLANTmessagewouldprovideahelpfulformofcommunica tioniftheywerereallyconductingsuchabattleaspartofthisdistributed multinationalforce. Asthebattleplanningandexecutionprogressed,whenanofficer indicatedthathewasatapointwherehefeltaCSLANTmessagewould

422SMITH,BOWER,ANDSPENCER behelpfulifthisexercisewasactuallyoccurring,withthevariousunits distributedacrossthebattlefield: Thatofficerwastrainedtoproficiencyonhowtousethefunc tionsinCSLANT.Theofficerwasgivenabriefreviewofthefunc tionsthatwerepresentedinthegeneralbriefingandaskedto demonstratethathecouldusethembeforecreatinghisown message. TheofficerwasthenallowedtocreatetheactualCSLANTmes sagethathethoughtwoulddemonstrateausefulapplicationof thetool. Ifthemessagecreatedwasintendedforoneoftheotherofficers participatinginthemockbattle,itwassenttothatofficer,who wasallowedtorespondtoit(bycreatingaresponseusing CSLANT)ifhethoughtthatwasappropriate. Asamessagewascreated,alloftheactionscompletedandcom mentsmadebytheofficerwererecorded. Followingthecreationofthemessage,theofficerwasaskedto completeaquestionnaireaskingthreeLikertScalequestions concernedwithusefulnessandusability,andwasthenasked fouropenendedquestionsabout:1)Whetherhethoughtthis typeofmultimediacommunicationtoolwouldbeusefulforthe Army;2)Thekindsofsituationsormessageswherehethougha toollikeCSLANTwouldbeusefulfortheArmy;3)Whichaspects ofthetoolhethoughwereparticularlyimportantforArmyappli cations;and4)Whatadditionalfunctionsorimprovementshe wouldsuggestforthistypeofcommunicationtool.(Ifapartici pantcreatedmorethanonemessage,hewasgiventheopportu nitytoaddtoorrevisehisresponses.) Study2Results Performancewasassessedintermsofthesubjectiveratingsand comments(seeSmith,etal.,2007foradditionaldetails)thebehavioral protocols(whattheparticipantsdidwhilecreatingaCSLANTmessage), andthecontentsofthemessagesthemselves. LikertScaleResponses.Below,theresponsestothethreeLikertscale questionsaresummarized. 1. Thistypeofcommunicationtoolwouldbeusefulforcertainmili taryapplicationsMean:6.2(1=stronglydisagree;7=strongly agree)

AsynchronousCommunicationandCoordination423 2. ItwaseasytousethiscommunicationtoolMean:6.4 (1=stronglydisagree;7=stronglyagree) 3. ThemultimediamessagesIcreatedwouldhelpmetocommuni catemoreeffectively.Mean:6.4(1=stronglydisagree;7=strongly agree) ResponsestoOpenEndedQuestions.Below,welistarepresentative setofresponsestotwoofthefourquestionsthatwereaskedafterthe participantscompletedtheLikertscalequestions. Whataspectsofthistoolareespeciallyimportant? Thisallowsthecommandertohearitfromthebrigadecom mandersmouthandtheresnoroomformisinterpretation. TheproblemwithdigitalcommunicationintheArmynowis thatitispassivetextandpictures.Voicegivesyouasenseof urgency,offear,ofpanic.CSLANTenablesthecommanderto communicatewithhistroopsmoreeffectively,indicatingwithhis voicewhathesfeelingandwhatisimportant,givingtheman intuitivefeelforimportant. CSLANTputsahumantouchtothemessage,indicatingwhere thecommanderisstressingthings.Youknowwhatsimportant tothecommander.Youcantgetthatfromtextandstaticgraph ics. Youcouldusethistodetectandclearupmisconceptions. Inaninternationalgroup,ifyoudontspeakfluentEnglish,you wouldbeabletoseewhatpeoplearetalkingabout,reducing misunderstandings. Swiftnessisthekeytothistool.Icanmakeamessagein5min utes,quickanddirty. Ithasmorerichnessthananemailmessage.Itletsyouprovide arichcontextinanasynchronousmode.Therecipientgetsmuch morerichnessthanevenemailwithanattachedslideshow.You alsogetamuchrichercommunicationthanovertheradio.The radioisabadmeansofexplainingwhatyouwant. Itispersistentsoitcanbeviewedagainlatertomakesureyou understandwhatwas.Evenwitharichsynchronouscommunica tion,themessageisgone.Idontknowwhathesaid5minutes later. Theabilitytopointatwhatyouretalkingaboutiscritical.It letsyoudrawtheirattentiontotherightplaceonthemap.

424SMITH,BOWER,ANDSPENCER Theyrelikelytounderstandandrememberthemessagewhen itspresentedthisway. Thetoolhastobesimple.Dontaddtoomanyfeaturesand makeittoodifficulttouse.

Whatadditionalfunctionsorimprovementsareneededinthecurrent designofCSLANT? ThehandlingofthesymbologyshouldbethesameastheCom mandandControltool. Ifitsbeingusedduringtheexecutionofaplan,itneedstobe tiedtoactualdatasourcestomakesureyourenotlaunchingan attackonelectrons[unitsshownonthemapdisplaythatarenot inthesamelocationintherealworld]. SupposethecommanderstartsworkinginCapes[AU.S.Army planningtool]andthencreatesaCSLANTmessageabouthiscur rentplan.Heshouldbeabletomovetheunitswhilecreatinghis messageinCSLANT.Thisdatashouldthenbeexportedbackto theplanningtool.Thecommunicationtoolandtheplanningtool needtosharethesameview,thesamerepresentations. TheabilitytoeasilyexportcontentfromCSLANTintoreports,or toembedlinkstoCSLANTslideshowsinreports. Reducetheredundancyinthepopupwindowsforrecording.

AnalysisoftheBehavioralProtocols.The10participantscreateda totalof15CSLANTcommunicationsoverthecourseoftheexperiment. (Giventhetimedemandsfortestingothersoftwaretoolsinthestudy, thisshouldnotbetakenasanindicationofhowoftenCSLANTwouldbe usedinatruebattlefieldenvironment.) Ifwedefineamessageastheinformationcreatedbyasingle officertoconveycertaininformation,inprinciplesuchasinglemessage couldbeaCSLANTpresentationencompassingoneormoreslides.An asynchronousdialogwouldthenconsistofasequenceofCSLANTmes sagessentbackandforthamongtwoormoreofficers.Twoofthecom municationsconsistedofonewaycommunications.Theremaining13 wereasynchronousdialogs.Onaverage,themessageattachedtoeach slidelastedameanof51.6secs.(range:11137secs.). Theresultsalsoshowthat,inallbutthreecases,themessagepre paredbyoneindividualwasasingleslidemessage.Inthecaseofadia log,thismeansthatonecommandersentasingleslidemessageto

AsynchronousCommunicationandCoordination425 anothercommander,whosentasingleslidemessageback.Inthree cases,theofficersusedsequencesof24slidestocreatetheirmessages. Allofthemessagesfocusedondiscussingthecommandersintent and/ortaskingforasubordinateorparallelunit.Inthemessage sequences(asynchronouscollaborations),therewerethreeparticularly significanttypesofinteractions: Detectingandcorrectingmisconceptionsthatarosefromprevi ouslivefacetofacebriefings(6/13dialogs).Afullexampleis illustratedinExampleDialog1below. Addingimportantmissingdetailstoaplan(12/13dialogs)(See ExampleDialog2belowforafullexample.) Sharingknowledgerelevanttoanoperationthatoneofficerhas becauseofhispreviousexperienceortraining(13/13dialogs). Thesefindingsareparticularlyimportant,astheyindicateconsider ablevalueintheuseofsuchmultimediamessagesasawaytoshare knowledgeandtoensurethateveryoneisonthesamepage. UseofSpecificCSLANTFunctions.Inthefollowing,wesummarizethe specificCSLANTfunctionsusedincreatingthesemessages. Priortomakingarecording,70%oftheparticipantschangedthepen colortoincreasecolorcontrast.Inaddition,overhalfoftheparticipants markedupthebackgroundslide(map)usingtextboxes,arrows,free handdrawingswiththepen,etc. Whilemakingtherecording,thepausefunctionwasusedfrequently (by60%oftheparticipants)inordertodecidewhattosaynext.Allofthe participantspointedatobjectsandlocationsonthemapwhiletalking; 70%alsosometimesdrewonthemapwhilerecording. Theresultsindicatedthatallofthefunctionsmadeavailablein CSLANTwereused.Ofparticularimportance,however,wasthefactthat everyonemadeuseoftheabilitytopointtovariousobjectsandlocations ontheslide(mapdisplay)ortodrawonthemapdisplayastheywere speaking.Thishighlightstheperceivedvalueofbeingabletousepoint inganddrawingtofocusattentiononkeyfeaturesonthemapdisplay whiletalkingaboutthem,andtosometimesleaveapersistentreminder (drawing)onthemapdisplay. SampleDialogs.Allofthemessagesfocusedondiscussingthecom mandersintentand/ortaskingforasubordinateorparallelunit. SampleDialog1.DetectingandCorrectingaMisconception.Thefol lowingdialogoccurredafterthebrigadecommandergavealivebriefing onthemissionforthebattalionsunderhiscommand.Inthisdialogthe commanderofthe2ndBattalionfirstindicatedthetaskingforhisunits basedonhisunderstandingofthelivebriefing.Thisorderwasthen

426SMITH,BOWER,ANDSPENCER reviewedbythebrigadecommander,whodetectedamisunderstanding andcorrecteditviaamessagebacktothebattalioncommander. 2ndBattalionCommander(CSLANTmessage):OK.Atthistimeeach oftheelementsfromthe2ndBattalion,US,wehaveAlpha,Bravoand CharlieCompanies,setinourassemblyarea.Weregoingtobemoving alongRouteBlue,whichbasicallytakesusupasfarasthePristinaairport area,whichisinthisvicinity.Eachoftheelementswillbegivenapieceof theroutetodosecuring,andbasicallydoingaguardandscreenmission. Alphacompanywillbasicallyhavethisareatothisareawheretheyll guardandscreen.Bravowillbasicallyhavethemidpoint,andCharliewill takeitallthewayuptothePristinaarea.Theobjectiveofthegameis basicallytoestablishadefensiveperimetersothatanylargerelements movingalongtheroutehavetheopportunitytogetthroughsecurely. Alpha,BravoandCharliewillbeoneoftheelementsthatwillbeprovid ingaguardandscreenmissionalongtheentireroute.(Whilesaying this,theBattalioncommanderwaspointingtotherelevantareasonthe map. Brigadecommander(inrespondingtobattalioncommanders CSLANTmessage):[BattalionCommandersname],Ibasicallyconcur withyourmovementasfarastheairportisconcerned.Imgladtosee thatyouareconcernedwithsecurityalongthatroute.Iwantyoutokeep inmindthatyourenottiedtoRouteBlueandthatyouhavesomerecon assetsattachedtoyoufromtheprovisionalRSTAsquadron.Iamcon cernedthatyoudidntmentionhowyouplannedtoutilizethem.They shouldkickoutwellbeforeyourstarttimesotherouteshouldbegener allysecured.Bearinmindthatoneoftheiradditionaltasksistolookfor alternativeroutes.Inparticular,Imconcernedabouttheareaaroundthe airportinPristinabecausethatsecuritymissionbelongstothe1stGer manandIdontwantyoubumpingintothem.Idontwantthemtoslow yourmovementuptoMitrovica.IneedyoutogetuptoMitrovicato establishpositionsandbeginsecurityandstabilityoperations.Yourroute ofmarchwillcontinuewellnorthoftheairport.Notsurewhyyoucon cludedattheairport.OneoftheconsiderationsisthatIdliketoseeyou workwiththeRSTApeople. TheRouteBlueactuallydoesafewrivercrossingsupinthePristina area.Maybetheycouldworktofindroutesthatminimizerivercrossings andcutyourexposureonpossiblechokepointsthatcouldbecomea problem.Iwanttomakesureyouhavearelativelyunimpededroute northintotheMitrovicaarea.ReworkthatandintegratetheRSTAele mentsandshowmeyourrevisedplanassoonaspossible.[TheBrigade commanderwaspointinganddrawingashesaidthesethings]

AsynchronousCommunicationandCoordination427 SampleDialog2.CollaborativelyDevelopingaPlan.Thisdialog betweenanIsraeliCommanderandaCanadianCommanderillustrates themworkingtogetherasynchronouslytoworkoutthedetailsofaplan. CSLANTMessagea:IsraelicommanderWeregoingtogivesome explanationregardingthemissionweveplannedforthereconbattalion combinedofIsraeli,CanadianandFrenchforces.Wehaveafewkeyareas here.Thisisonekeyareawehavetocontrol;thisisthe2ndkeyareawe havetocontrol.Bydoingthatwewillallowtheforcestogothroughthis route[saidwhileannotatingtherouteontheCSLANTmap],whichis reallythemainmissionofoneofthebrigadebattalions. Inordertoachievethattaskwehavedecidedtogaincontrolofthis keyareaatnightwithwheeled,relativelysilentvehiclessuchashumvees andsquadsthathaveelectroopticalmeansthatcanoverseethiswhole valley.Then,afterthishasbeenaccomplished,wecanhaveaconvoyof APCstogothroughthiswholeroutewhichisamajorpartofourmis sion.(TheIsraelicommanderwasspeaking,pointinganddrawingonthe mapwhilesayingthesethings.) CSLANTMessageb:CanadianCommanderThiswholeterrain,this wholevalleyisreallyasensitivepoint.Itsakillingzone.Andthatswhy itsessentialthatwegaincontrolofthis[pointingtothemap]andthis [pointingtomap]bysiteandfireabilitiessothatweknowbythetime wehavethebrigadecommandermoveforwardwiththebattalionhe wantstomovehere,thatitsnolongerakillingzone,nolongeraprob lem. Werestartinginourassemblyareasdownhere.Ourfirstphaseisto securetheheightsinordertopassthroughanotherforcetoclearthis fieldanddeployouthere,andthatstheendofthefirstphase.The2nd phase,theaimistosecuretheotherheights.Inordertogettherewe havetocrossthisvalley,1016kilometers,andgetaforceupthehillto securetheheightsandthencontinuetosecureRouteRed.WhilePhase2 ishappeningthegroupthathassecuredtheseheightswilldeployeast andreconnoitertheseroutesasalternativeroutesfortheforce,andwill establishsecurityintheseheightsforsubsequentreconnaissanceand movementupthisvalley.(TheCanadianCommanderpointedtothe appropriateareasonthemapashewastalkingaboutthem.) CSLANTMessagec:IsraelicommanderOnecommentregarding thismission.Oncewesecurethisandwehaveanotherforcehere,in ordertoliveuptoourmissionweregoingtohavetohavepatrolsof APCsoccasionallyrunningupanddownthisroutetomakesurethe enemyisntpreparinganythingforthebattalionthatwillcomefurtherin

428SMITH,BOWER,ANDSPENCER ourmission.(ThisCommanderpointedtothelocationsofinterestalong theroutewhilediscussingthem.) ExampleDialogsImportantPoints.Thecontentsofthesemessages (andtheothersproducedinthisstudy)highlightseveralpoints: Theofficersmadeextensiveuseofpointinganddrawingwhile creatingtheirvoiceoversinordertohelpcommunicatetheir messagesmoreeffectively. Suchasynchronousdialogsmakeitpossibletodetectandrepair misconceptions(seeExampleDialog1aboveforanillustration). Suchasynchronousdialogsprovideanopportunitytoshare expertiseindevelopingaplan(seeExampleDialog2abovefor anillustration). Study2Discussion Thiscasestudyservestoveryconcretelyillustratethepotential valueofsuchanasynchronousmultimediacommunicationtooltosup portdistributedworkintheArmy.Theofficerswhoparticipatedinthis studyindicatedavarietyofspecificusesforsuchatool,including: Communicationofthecommandersintentandspecifictaskings aspartofanoperationsorder. CommunicatinginformationaspartofaFRAGO. Preparingafteractionreviews(mentionedinquestionnaire responseseventhoughthestudyitselfdidnotinvolvecreating afteractionreviews). Providinginstructionstostaffduringthepreparationofanoper ationsorderorinthecompletionofsomeothertaskrequiring staffsupport. Collaborationtoshareknowledgeandexpertiseduringtheplan ningofandexecutionofabattle. Collaborativeworkandtrainingingeneral(outsideofbattle preparationandexecution). Itwasfurtherpointedoutinthefeedbackfromtheparticipating officersinthisstudythatthisricherformofcommunicationwouldbe especiallybeneficialinamultinationalforceenvironment,asthevisual reinforcementofverbalinstructionswouldhelptoreducemisunder standingsduetolanguagebarriers. Itshouldbenotedthatthisformofasynchronouscommunication shouldnotbeconsideredareplacementforothertraditionalmedia,but ratherasanadditionaltooltoaddtothesoldiersrepertoireofcommu nicationtools.Therewillbetimeswhensynchronouscommunications

AsynchronousCommunicationandCoordination429 arevital,whetherintheformoflivebriefings,radiomessages,orbrief ingswhileviewingshareddisplaysoncomputers.Therewillbeother timeswhenasynchronouscommunicationisnecessary,butwheretext withstaticgraphicsisthepreferredmedium.However,inthedistributed battlefieldofthefuture,thereisclearlyanimportantroleforthetypesof richmultimediamessagescreatedaspartofthisstudy. IntermsofthespecifictypesoffeaturesfoundinCSLANT,theresults illustratethevalueofprovidingvoicerecordingsthataresynchronized withpointinganddynamicannotations.Theuseofthesecapabilitieswas pervasiveinthemessagescreatedduringthisstudy. Finally,previousstudieshaveshownhowmultimediapresentations createdbyCSLANTandothersuchtoolscanincreasetheunderstanding andrecallofinstructions.Thisstudyfurtherillustratesthevalueofsup portingrichasynchronousdialogs,allowingsoldierstosharetheirknowl edgeandexpertiseduringtheplanningandexecutionofabattle,to collaborativelydevelopplans,andtohelpthemtodetectandrepairmis understandingsandmisconceptions. Study2FutureWork Fromadevelopmentperspective,oneofthemajorrecommenda tionsarisingfromtheseresultsistheneedtoexplorehowtoembedsuch acommunicationtoolintomilitaryplanningenvironments,sothatcom municationisaseamlesspartoftheplanningprocess.Equallyimportant, thereisaneedtobetterunderstandexactlyhowandwhentointegrate theuseofsuchmultimediacommunicationtoolsintomilitaryplanning andexecutionprocesses.Newproceduresandprocessesneedtobe developedandtestedtodeterminehowtobesttakeadvantageofthese newcapabilities.Similarly,thereisaneedtodevelopandevaluatetrain ingtechniquestoensurethatthesoldiersofthefutureareabletomake effectiveuseofthistechnology. ACKNOWLEDGMENTS TheworkwassupportedunderagrantfromtheArmyResearchLab oratorythroughfundingundertheAdvancedDecisionsArchitectures Consortium.Portionsofthisworkwerealsomadepossiblethroughthe supportofCERDECstaffatFt.Monmouth,whowereresponsibleforthe overalldesignandconductoftheSINCEexperimentatFt.Dixthatwas usedasthecontextforStudy2.SpecialappreciationisduetoIsraelMayk andalloftheCERDECstaffwhorantheoverallexperiment,andtothe

430SMITH,BOWER,ANDSPENCER officersfromCanada,France,Germany,IsraelandtheUnitedStateswho participatedintheexperiment. REFERENCES

Chapman,R.andSmith,P.J.(2003).Theimpactofcommunicationsmodeon asynchronouscollaborationintheNAS.Proceedingsofthe2003AviationPsychology Symposium,Dayton,OH.Clark,H.H.(1996).Arrangingtodothingswithothers.Computer HumanInteraction,165167. Clark,H.H.,andBrennan,S.(1991).Groundingincommunication.InL.B.Resnick,J. M.Levine,andS.D.Teasley(Eds.),Perspectivesonsociallysharedcognition(pp. 127149).Washington,DC:AmericanPsychologicalAssociation. Faraday,P.M.andSutcliffe,A.G.(1997).Designingeffectivemultimedia presentations.ProceedingsofCHI97,272279. Mayer,R.(2005).Cognitivetheoryofmultimedialearning.InR.Mayer(ed.),The CambridgeHandbookofMultimediaLearning(pp.3148).NewYork:Cambridge UniversityPress. Shattuck,L.G.andWoods,D.D.(2000).Communicationofintentinmilitary commandandcontrolsystems.InC.McCann&R.Pigeau(eds.),TheHumaninCommand (pp.279291).NewYork:KluwerAcademic/PlenumPublishers. Smith,P.J.,Bower,J.andSpencer,A.(2007).AsynchronousCommunicationofArmy OperationsOrders.OSUCSELTechnicalReport200714.ColumbusOH. Visconti,Sheila(Ed.).(2002).LeadershipandProblemSolving.Boston:McGrawHill Companies.

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T A N D L A U R E L A L L E N D E R

A D VA N C E D D E C I S I O N A R C H I T E C T U R E S F O R T H E WA R F I G H T E R :
F O U N D AT I O N S A N D T E C H N O L O G Y

E D I T E D B Y PAT R I C I A M C D E R M O T T AND LAUREL ALLENDER

This book, produced by the Partners of the Army Research Laboratory Advanced Decision Architectures Collaborative Technology Alliance, also serves as the Final Report for DAAD19-01-2-0009.

SECTIONIV.COMMUNICATINGINFORMATIONACCROSSTHETEAM

SECTIONIV COMMUNICATINGINFORMATIONACROSSTHETEAM

324SECTIONIV

Chapter

24

24. UNDERSTANDINGTHESIGNIFICANCEOFADARESEARCH

WalterWarwick,Ph.D. AlionScienceandTechnology,Boulder,CO

INTRODUCTION OverthenineyearcourseoftheADA,theArmyhasundertakena varietyoftechnological,organizational,andinstitutionalchanges.Atthe sametime,theArmyhasbeenengagedintwolongtermunconventional conflicts.TheArmystransitiontoadigitalbattlefieldconductedunder thepressuresofasymmetricwarfarehasledtosignificanttransforma tionsintactics,equipment,andleadership.Thistransitionwascharacter izedbyacceleratedtechnologicaltransformationswhichincreasedthe numberofsensorsonthebattlefield,expandeduseofunmannedand autonomousassets,andleveragedimprovementsinweaponslethality, responsiveness,andaccuracy,whiledramaticallyincreasingdistance betweenechelons.Thecumulativeeffectoftheseconditionsresultedin moredecentralizationofcommandwithkeydecisionmakingtasks pushedtolowerandlowerechelons.Combatinclosequartersincreased thespeedofthekillchainandledtoacommensurateshorteningof theorientobservedecideactloop.Theoperationalenvironmentwas furthercomplicatedbyincreasedinterdependencewithotherservices, relianceonMultiNationalForces,andthedemandtooperateacrossdif ferentdimensions,includingpolitical,informational,economic,and social.Atthetacticallevelofcombat,dailyoperationsdecisivelyshifted fromdeliberatetomoreadaptiveplanning. Onthesurface,ADAresearchcouldbeviewedasareactiontothese newchallenges.Indeed,muchofthesuccessoftheADAcanbeattrib utedtothecommitmentADAresearchersmadetomaintainingopera tionalrelevance.ButthebasicresearchoftheADAshouldnotbeviewed solelythroughthelensoftodayscircumstances.ThescienceoftheADA

431

432WARWICK ismuchmorethanareactiontocurrenttechnologytrendsandArmy contingenciesbut,rather,restsinthedevelopmentofadeeperunder standingoftheinvariantchallengesofwarfare. UNDERSTANDINGTHESOLDIER TheSoldieristheoneconstantamidtheArmystechnologydriven transformationandthecomplexnatureofasymmetricconflict.Soldiers havealwaysandwillalwaysneedtomakesenseoftheirenvironments andactontheirunderstanding.Theywilldosogivenuncertaininforma tionaboutcontinuouslyevolvingthreats.Theywillcontinuallyconfront newenemieswhoarethemselvesflexible,unpredictable,andadaptive. Andwhilethenatureoftheconflictmightbeeverchanging,Soldierswill alwayshavetoovercomeanenemyonthegroundthroughcollaboration amongteamsoffighterstorealizeacommandersintent. Neithertechnologynorshiftingoperationalcontextswillchangethe essentialnatureofthechallengestheSoldierfaces.Theeverincreasing complexityoftacticalcombatmakeitallthemoreimportanttounder standtheSoldierscapabilitiesandlimitationsfromahumancentered perspective.Forexample,moreisnotalwaysbetterwhenitcomesto information,soitisimperativetounderstandhowtheSoldierwill developandmaintainsituationawarenessgiventhefloodofinformation thatwillbeavailableonthedigitalbattlefield.Additionally,whileeffec tivecollaborationdependsontrust,itisfarfromclearhowtoengender theSoldierstrustinasystemofsystemsthatcomprisesbothhumanand autonomousagentspotentiallyactingacrosslargedistances.Finally, eveniftheadvanceofnetworkcentrictechnologiesisdescribedby Mooreslaw,advancesinhumaninformationprocessingarenot.Hence, theresultingincreasewemightexpectinoperationaltempofrom improvementsinaSoldiersaccesstoinformationwillstillbeconstrained bythismanifestasymmetryofthehumancomputerinteractions. FortheADA,thedefiningquestionofArmytransformationisnot whatistechnologicallypossiblebut,rather,whatishumanlypossible giventhetechnology. ADARESEARCH Fromthisperspective,theADAhasviewedtransformationthrougha lensinwhichthecapabilitiesoftheSoldierasahumandecisionmaker needtobeunderstood.Changesinthetechnology,theoperationalenvi ronment,andthehumanterrainallservetofocusperennialquestions

Conclusion433 aboutsituationawareness,managinguncertainty,planning,collaborat ing,anddecisionmaking.Indeed,muchoftheworkdescribedinthisVol umehassoughttoexplainhowdecisionmakinganditssupporting processescanbemeasured,modeled,andpredictedinthecontextof newtechnologiesand,further,howwecananticipatenewtechnology trendstoensurethathumancentereddesignbeginsearlyinthedevel opmentprocess.ThisisthesenseinwhichtheADAhasservedto advancefoundationalresearchaboutdecisionmaking. ButtheADAhasalsoadvancedthescienceofdecisionmakinginnew directionsbynurturingworkthatisnecessarilyinterdisciplinaryand, moreover,deeplyintegrative.Anadvanceddecisionarchitectureis,at essence,aframeworkforsupportinghumancognitionbywayoftechnol ogy.Thestudyofthesearchitecturesthereforedependsonmorethan justanunderstandingofthecomponenttechnology(whichoftenhasa lifeofitsown)ortheexplorationofhumancognitioninisolationofthe technology.Thescienceofadvanceddecisionarchitecturesdependson understandingcognitionintechnologicalandoperationalcontexts. Astheprecedingchaptershavedemonstrated,ADAresearchhas progressedbybringingtogetherleadingresearchersincognitivescience, humanfactorsengineering,andcomputersciencetoworkintandem withArmystakeholders.Theseinterdisciplinaryteamswerekeyto understandingdecisionmakingincontext.Thecompositionofthese teamsalsoreflectedtheADAscommitmenttohumancentereddesign and,bythesametoken,madeitpossibletoanticipatenewtechnological trendsbyputtingsystemdevelopersinclosecontactwithbasicresearch ers.Finally,theinterdisciplinaryapproachwasnotonlynecessaryto advancethestudyofdecisionmakingintechnologicalcontexts,butalso toensurethattheresearchhadrelevancetoArmystakeholders.Someof themostimportantresultsofADAresearcharethosethatexplorethe uniquerequirementsofArmydecisionmakersandidentifyanddevelop thetechnologiesthatarebestsuitedtothoserequirements.Underthe ADA,Armyrelevancehasthusbeentransformedfromaposthoc requirementintoaconsiderationthatcanitselfdriveresearch. WhilethisinterdisciplinaryapproachhasbeenahallmarkofADA research,itwasnotpursuedasanendinitselfbut,moresignificantly,as ameanstoadeeplyintegrativescienceofadvanceddecisionarchitec tures.Inparticular,theADAhassupportedintegrationatthreelevels. First,totheextentthattheADAhascompriseddifferentengineeringdis ciplines,therehasbeensystemslevelintegration.Thislevelofintegra tionisevidentinmanyoftheprecedingchaptersindiscussionsofmodel interoperability,interfacedevelopmentandagentbasedsolutions.Sec

434WARWICK ond,therehasbeenconceptualintegration.Thisisevidentinthediscov eryofcommonformalizations(e.g.,thecomputationalmodelsof cognitiveprocesseslikerecognitionprimeddecisionmaking,planning, andsituationawareness),metrics,analyticalandexperimentalstrate gies,andtheapplicationofideasfromthedisparatefieldsofresearch thathavebeenbroughttogetherundertheADA.Third,andmostimpor tant,therehasbeenintegrationinthesenseofbringingtheinsightsand researchresultsoftheADAtotheSoldier.Thisisthelevelatwhichthe ADAmissionissatisfiedanditisevidentinthetremendoussuccessthe ADAhasachievedintransitionaccomplishingover$60Mintransition throughthefirsteightyearstocustomers. CONCLUSION Thesignificanceofthistransitionrecordcannotbeoverstated. Indeed,theimpactofeventhemostsophisticatedtechnologiesorpro foundinsightsintohumancognitionwillbelostiftheycannotbetransi tionedtotheSoldierinamannerthatsupportsanawarenessand understandingofthetacticalsituationandfacilitatesmoretimelyand moreaccuratedecisionmaking.TheADACTAhasmetthischallenge.The ADAtransitionrecordalsostandsasavindicationoftheCTAbusiness model.TheADAtransitionrecordispowerfulevidencethatcollaborative basicresearchproductscanbedevelopedwithcrosscuttingimplications andclearArmyrelevance. Finally,inadditiontoreflectingprogrammaticsuccess,theADAtran sitionrecordunderscoresthescientificsuccessoftheCTA.Heresuccess isjudgednotonlybywhathasbeenaccomplished,butbywhatisto come.WhilethetransitionofexistingADAresearchproductsissignifi cantandwillprovideafoundationforfutureresearchacrossavarietyof disciplines,itisthedevelopmentoftheverynotionofadvanceddecision architecturesthatwillshaperesearchtocome.Sucharchitecturesfocus onthehumanandtheyarepredicatedontheunderstandingthatnew trendsandtechnologiesnevereliminateoldproblems,theymerely recasttheminnewanewlight.Moreover,thedevelopmentofadvanced decisionarchitecturesdependsontrulyinterdisciplinaryandintegrative research.TheADAhasbeenasuccessnotonlyforwhatithastoldus aboutdecisionmakingbutforwhatithastaughtaboutconductingthe scienceofdecisionmaking.