MULTI-MODAL INTERACTIONS IN COCKPITS

HONEYWELL AEROSPACE DESIGN COMPETITION

Submitted By Vishnu M S Anoop Balakrishanan PGDPD 2013 National Institute of Design, Gandhinagar

Methodology
Background Research

Activity Mapping

Problem Identification

Ideation

Concept

Background Research
Background Study was conducted to get familiarized with the area of study. Literature review was conducted to study the procedures in flight operations, words used etc. Some References
http://en.wikipedia.org/wiki/Instrument_landing_system http://instrument.landing-system.com/ http://en.wikipedia.org/wiki/Flight_management_system http://en.wikipedia.org/wiki/Flight_plan http://multimodalblog.wordpress.com/category/multimodal-theories-and-methods/ http://multimodalityglossary.wordpress.com/multimodality/ http://www.quora.com/Airplanes/What-do-all-the-controls-in-an-airplane-cockpit-do

Datas and Facts

That they frequently make mistakes - About altitude. About control settings. About airspeed. When landing. However, since the overwhelming majority of the time the mistakes are caught by themselves, by the co-pilot or dont result in a mishap, they wont tell an already nervous flying public. -We forget about the fasten seatbelt sign all the time. When you look up at the sign (and disregard it typically) and it has been illuminated for the last 45 minutes in smooth air, we simply forgot. Lots of guys will leave it on all the time. However, sometimes we do have reports of choppy air ahead and will leave it on until we either experience it or take a wild guess that the air ahead will be smooth.
Source : What are some things that airline pilots wont tell you? http://www.quora.com/Airlines/What-are-some-things-that-airline-pilots-wont-tell-you

Most of the flight accidents are caused by pilots according to statistics

Datas and Facts

There have been many criticisms of the current flight management/autoflight system in modern aircraft (Weiner, 1998). In 1996, the FAA determined that a major problem with the modern flight deck was the disconnect between the pilot and the autoflight systems in the aircraft (Abbott, et. al, 1996). This disconnect led to instances where pilots would remark, Why is the aircraft doing that? or Whats it doing now?
Source : Towards an Improved Pilot-Vehicle Interface for Highly Automated Aircraft: Evaluation of the Haptic Flight Control System Paul Schutte, Kenneth Goodrich

Activity mapping
Interactions inside the cock pit was studied and classified in to four types 1. 2. 3. 4. Controls / input realted to the displays in cockpit Control / input that controls the flight instruments Output / informations ( visual or souds) Output / Warnings (visual or sound)

23

23

28

26

24

27

30

30

5 1 3 4

18

19

15 13 14

29

20 30 30

2
22 21

16

17

31

10 7 6 11 12

32

33

34

35
35

1 2 3 4 5 6 7 8 9 10 11 12

Airspeed Tape Altitude Indicator Altitude Tape Rate of Climb Purple text -Autopilot speed and altimeter setting Yellow text Green text Green text Green text Dotted Purple Line Green text Green text

Left Engine Thrust Right Engine Thrust 15 Engine Warnings 16 Left Engine Exhaust Temp 17 Left Engine Exhaust Temp
14
18 19 20 21 22
23

13

24
26

27 28 29
30 31 32 33 34

APP - Toggles Landing Displays HP/IN - Toggles Units Flight Course Settings Altimetre Settings Resets the Display for Heavy Manuering
23

35 35

Glidescope alert active warning Switch Displays Warning Lights Master warning lights [ Speed Brake Control] Clock display Clock controls Nose Wheel steer toggle Foot air Wind Sheild air Dipslay Brightness FMC [ Flight Managment Computer]

Aeroline Control And Display

39

43 44 40

38

37

42 45
46

41

46

36

36

37 38 39
40

41 42
43 44

45
46

Landing Gear Gear status Indicators Flaps Indicators Auto Break Malfunction warning Break Settings Anti Skid malfunction warning Flaps extending Indications Airspeed reference Settings Thrust Reference settings Fuel flow Indication Lower DU display Control

53

51

52

54

58

56

57
55

59

51 52
53 54 55 56

57
58 59

Fire warning Main warning / with grid Tiller Wheel / Steer flight Throttle Fuel cut off Trim Wheel trim indicator Flap indicator Trim set/ auto vs manual

62

61

60

62

60

60

61

63

64 65
68

77

66

65
67

69

69 76

80

80

76 70 70

82

71 73

81

81

83

78
72

75 75

79

74

60

61 62
63

64 65
66 67 68 69 69 70 70 71 72

Fire extingusher APU Engine 1 and 2 Over heat in engine Fire warning wheel well + 4 other areas Bottle Test Switch Fire bottle empty alert Freq Display COMM radio Transfer Freq from active to next Tune Frequency Select Comm device[ VHF/ HF etc] Cargo Fire Detection Panel NAV radio Control Radio Volum Control Mask/ Boom Switch

73 74 75 75 76 76 77

78 79 80 80 81 81
82

83

HUD display Control Landing Trim Controls ADF radion Controls IRS controls Internal LAN Stats/ Flaps indicator PSEU warning / land gear pos detection error Observer Intercom - 3rd person in deck Caution - Engine thrust device error black box control Warning test light

84

84 85
84

88 88 87

91
94

91

89 89 90 90

86

92

92

93

84 84

85 86
87 88 88 89 89 90 90 91

92 92 93
94

Alternate Hydralic system and warning electrical info and faults Exit door open or close information Cabin temperature controls Anti ice [ engine ] warning and controls Standy by battery warning and controls Hydraulic pump warnign and control dual bleed warning air control panel Fuel value information and warnings Check lists Circuit Breakers

Problem Identification
1. 2. 3. 4. 5. Warnings in the current system either have audio or visual modes of interaction which may be missed by the pilots in abundance of information. Visibility of warning lights like [Standby battery warnings] etc. which is in the overhead panel is less. Lesser embodiment due to few mode of interactions and autopilot features. Pilots can tend to forget things as they are in multitasking. Change in flight paths ATC delays. It involved dialing in the new flight path to FMC ac curately and quickly.

SYNAPSE
Synapse brings multimodality in to the air plane cockpits. With out affecting the present interface synapse helps pilots to carry out tasks easly and effectively on the run. It auguments the present interface by providing precise and taske realted information right on the palm. It uses visual haptic and sound feedback to alert the pilot at times. It keeps the pilots connects to the interaface.

What it has ?
Sound Input Device to Send commands to FMC Rings in the fingers are mapped to the interface display on the basis of color. Menu in the display has a color and folding the finger with corresponding color activates the option show in the display. Haptic interactions to alert the pilots on emergency warnings with the wrist color Turing red and error displayed on the screen.

Display Screen That shows Relevant Information of activities in the cockpit

Ring in the wrist shows the status of device. E.g : Green light Depicts Check list mode.It can turn red for warning which are not so important like Seat belt sign off alert.

Features
Autopilot mode alerts the pilot s about what actions are taken by autopilots. Check list mode gets the check list raeady at the palm. You have to shut off all the finger ring ligths to clear the check lists. ILS landing mode - Ring color changes acordign to the apporach beacon. Haptic warning system makes sure that pilot doesnt miss any critial warnings. Speech to command system helps fast reprogramming of FMC

Haptic Feedback for immediate warnings like [ Overheat in Engine ].

Augmenting interface with relevant information which has greater visibility right in your hands.

Speech to Command to program FMC. Easy input method to FMC

It works in different modes

Autopilot mode : Alerts Pilots about the decisions taken by autopilot.

ILS Landing Mode : Color Changes and haptic feedback

Checklist Mode : Do the necessary actions for check listing.

Scenario
Emergency landing at near by airport due to fire warning in the APU during bad weather conditions using ILS approach. 1. Fire warning : Haptic feedback displaying fire in APU

Pilot acknowledges the fire warning and turns on the APU fire Extingusher. He decided to land the flight to near by airport.

2. Contacts the ATC and reprogrames FMC to near buy airport for landing.

Pilot reprogrammes the FMC with speech

3. Checklisting for landing

Pilot does all the actions in the order shown in the Check list display So as to be sure that he has done everything for landing All ring lights are off. Now its safe to land !

4 . ILS landing approach taken

Color of light changes with hapticfeedback as flight crosses each beakon it is the feeback along with the beep sound from cockpit.

Thank you