2014

Proposal for the

measurement of Real Time
Pilot Mental Health Index
(RPMHI)
ELYSIAN AERO TECH

I. Abstract
RPMHI is proposed as a means of dynamically recording in flight pilot biometrics with an aim to detect real
time variations in stress levels and perform long term analysis on pilot mental health. The proposed system
has two main system components, measurement and data infrastructure. In this work we lay out our
proposed methods for both the measurement and recording of in-flight pilot biometrics. This work is
aimed at commercial airline companies with the goal of reducing the risk of pilot mental breakdown.
II. Introduction
The demand for air travel has significantly increased in the past decade. Many people choose flight as a
mode of transportation because it is convenient and time saving, however, the recent disappearance of
Malaysian Airlines flight MH370 suggests that the safety systems present in the aviation industry can be
improved. One of the biggest problems with the rapidly increasing demand for air travel is the upcoming
shortage of airline pilots, this shortage means that pilots are flying more now than ever before which is
leading to huge increases in stress in the cockpit. Stress in the cockpit can be disastrous for the passengers
of the plane as the pilot is required to make complex decisions in seconds while flying the plane. In some
cases, increased stress can lead to mental breakdowns in the cockpit, this was seen in March 2012 when
JetBlue Pilot Clayton F. Osbon left the cockpit and began screaming at passengers, he had to be retained
and refused re-entry to the cockpit by the cabin crew [7].
It has been recorded that human error can be held in account for between 58% and 97% of all aviation
accidents [8]. As human error and pilot wellbeing are inversely correlated, the proposed solution to the
problem of mounting stress and the decline of pilot mental health is to carefully monitor the mental health
of the pilot using an inflight biometrics measurement system.
The remainder of this paper is organised as follows: III. overview of system where a basic high level
abstraction of the intended solution is given. IV. baseline creation where the means of establishing a pilots
in flight baseline is examined. V. system data in which we detail the movement and treatment of data
within the proposed solution. VI. cockpit sensor array, this section constitutes the hardware overview of
the system. Second to last is VII. Summary and Conclusion where the details of proposal are rounded up.
Finally there is VIII. References.
III. Overview of System
The proposed solution to the problem of monitoring the mental wellbeing of the pilot is an advanced
sensor array built into the plane cockpit that measures the biometrics of the pilot. The raw data from the
sensors is managed and analysed using a python/SQL server; this data is compared to the Baseline data
information which is gathered about the pilot in order to create a unique pilot profile. If the inflight
biometrics are sufficiently dissimilar to the Baseline data the server will provide advice to the pilot and
airline staff relating to the mental and physical wellbeing of the pilot. This solution can act as an early
warning system for pilots whose mounting stress may lead to a mental breakdown while in flight.
P a g e 1 | 18


Operation Flow Chart


Figure1:Overviewofproposedsolutionsystem
IV. Baseline Creation
This section of the report will describe a strategy for creating a unique set of Baseline parameters for each
individual pilot profile. The baseline consists of a set of values which describe the regular biometrics and
lifestyle of the pilot. As per the sensor design, the biometrics that will be measured are:
Heart Rate
Perspiration
Respiration
Hand Tremor
Motor irregularity (sudden twitching/fidgeting)
These Biometric parameters will also be measured during every in-flight monitoring session using the
sensors installed in the cockpit of the plane. The initial testing of the pilot biometrics establishes the pilots
regular biometric information. Before the initial biometrics tests, the Pilot will take a Profile Questionnaire
which aims to extract information about the pilots lifestyle. The information that will be sought from this
Questionnaire will include:
Pilot Sleep Patterns
Pilot Caffeine intake
In Flight
Monitor
System
Data
Aggregation
Post Flight
Data
Transfer
Baseline
Creation
P a g e 2 | 18


Pilot Alcohol intake
Pilot Stress levels on a day to day basis
Level of Exercise completed on a day to day basis by the pilot
Pilot tobacco intake (Does the pilot smoke?)
Pilot medication intake
Whether the pilot suffers from hyperhidrosis or hypohidrosis
Pilot Weight
Pilot Height
These lifestyle parameters will have an impact on the pilots Biometric information and it can be used to
verify the Biometric data. For instance, a pilot who regularly drinks coffee will have a higher than average
heart rate and a pilot who suffers from hypohidrosis will have a lower than average perspiration level. The
Biometrics measurement process will be repeated on every flight cycle completed by the pilot until there is
sufficient data to finalise the acceptable baseline range.


Figure 2: Baseline Creation Work Flow
The data below shows how the pilot biometrics would be displayed. This system allows the data managers
to determine the unique baseline parameters for the pilot; in this case the data was taken over the course
of 74 flights. The data shown below was generated using a random normal distribution using ranges from
academic studies aiming to measure Heart Rate [9], Perspiration [10], and Hand Tremor [11].
Profile
Questionnaire
Biometrics
Measurement
Data
Aquisition
Baseline
Creation
P a g e 3 | 18


Sample data for Biometric Testing

Figure 3.1: Heart Rate vs. Number of Flights

Figure 3.2: Perspiration (Sweat Quantity) vs. Number of Flights

Figure 3.3: Tremor Frequency (Hz) vs. Number of Flights

74
75
76
77
78
79
80
81
82
83
84
0 10 20 30 40 50 60 70 80
Heart Rate (bpm) vs Number of Flights
0
0.01
0.02
0.03
0.04
0 10 20 30 40 50 60 70 80
Perspiration (Sweat Quantity) vs Number of
Flights
0
2
4
6
8
0 10 20 30 40 50 60 70 80
Tremor Frequency (Hz) vs Number of Flights
P a g e 4 | 18


Baseline Profile Example

Figure 4: A typical pre-flight read out of a pilots recorded status
The SQL database will be capable of providing the pilot or airline company with information about their
pilot. This information includes a breakdown of the pilots general mental health and whether the pilot is
deemed to be fit to fly the aircraft at present.
V. System Data
The key components of the proposed data storage and analysis reside on a small control server running a
Linux server OS such as CentOS, CirrOS, Red Hat or Ubuntu server. The reason for utilizing Linux operating
systems is the ease in which these systems may be customised and scaled for application. The server may
run on a simple a SoC (system on a chip) and networked drives in RAID configuration.
On the server SQL [1], OpenCV [2] and Python 2.7.4 [3] libraries will be installed. SQL will serve as the
database component for storing and ordering recorded data. SQL is currently one of the most popular
packages for general database creation, use and management .Python serves as the base primary
programming language to perform analysis, establish secure transparent connections when transferring
data and managing data. Python has the added benefits of being written in C. With the underlying C
libraries recompilation is possible to optimise the Python interpretation and minimise the data and
compute overheads. OpenCV is an open source package for computer vision applications. The computer
vision program written in C using OpenCV libraries is designed to help analyse pilot eye movement, posture
and general movement. The use of the aforementioned packages is not required these are merely
suggested suitable packages.
P a g e 5 | 18



Figure 5: A high level view of the proposed data operation workflow
Pre-Flight
Import: take-off data relating to both the pilot and co-pilot is imported from the central operating pilot
database, see below. The data is imported as a compressed encrypted SQL database file and contains the
major recorded data for both the pilot and co-pilot and all related metadata. This data will later be used to
create the baseline for the upcoming flight.
Pre-Protocol: Data protection and security is of vital importance and as such pilot data must be properly
encrypted to avoid fraud and malicious activity. Once the database is imported the data must be
decrypted [4]. Once data is decrypted the database must be prepped for writing and modification this is a
can be done through basic SQL shell scripting. As a slight side note, uniform formatting of data necessary as
data may be recorded over several years. Any changes to data format through system updates may be
taken into account at this stage.
Baseline: With the data imported and decrypted we are now in a position to run the algorithm to
determine the pilot and co-pilots flight baseline. Both previous pilot data and the data created from the
pre-flight survey must be considered in the creation of the pre-flight baseline to which everything will be
compared.
Pre-Flight
Post-Flight
Mid-Flight
P a g e 6 | 18


Mid-Flight
Data Recording: Sensory data is recorded in real time and maintained in raw format files. A sampling
frequency of 2 Hz is used therefore 12 data points are created per second which likely result in
approximately 50 Bytes per second. This data will then be processed by the central analyser. Each data
point will likely be restricted to a single scalar value which will represent an offset from the established
flight baseline. For example, for heart rate a stream of data points: 0, 1, 2, 1, 3, 2, 0, -1 would represent the
beats per minute from the baseline resting heart rate of the pilot or co-pilot.
The openCV program will run constantly to analysis visual data. Due to the nature of visual data the
majority of the data analysed through the openCV program will be temporary. Ultimately only large
variations in visual data will be recorded and received by the analyser. Again, data may be represented as
the integers showing deviation from the baseline by some pre-defined scale.
Analysis: The central analysis Python program will likely run a straightforward algorithm defined for the
RPMHI but machine learning techniques may also be implemented to help create pseudo-personalised
algorithms for each pilot or co-pilot being monitored. Stress data will likely only be analysed every 10
minutes or so to minimise wasteful computation and lessen the effect of non-significant anomalous data.
Considering that flight times can vary from 30 minute up, an analysis frequency of 6 analyses per hour
should be acceptable. It is important to re-state that the proposed system aims at picking up chronic and
long term mental health issues in pilots, issues relating to short term stress induced from intense or
dangerous flying situations fall outside the scope of this proposal.
Analysis is ultimately intended to result in the calculation of the RPMHI of the pilot and co-pilot. Extreme
analysis results may then be reported back to flight staff or ground crew through satellite based
communications. Flight protocol will determine the action taken in response to any real-time data. This
proposal does not suggest any direct intervening with any flight systems by the RPMHI system nor is any
real-time feedback to either the pilot or co-pilot suggested as this may worsen a stressful situation.
Analysis will result in one final data set, which will be the averages and variances of each metric since the
last analysis was carried out. These data points will pay the primary role in determining the pilots RPMHI
result.
Cleaning: At this stage superfluous data is discarded and entries to the SQL database are made. The
cleaning stage is important to minimise unnecessary data and increase the ease of packaging, compression
and encryption during post-flight operations. 10 minute analysis data is maintained along with some final
metadata representing the frequency of large metric variation which is intended to aid in data
interpretation.
Post-Flight
Export: The baseline calculated during pre-flight operation will be packaged alongside all recorded data.
Packaging of the SQL database will be directly followed by compression and encryption operations. Finally,
the transmission of the encrypted data to the central pilot information data is performed. At this point all
P a g e 7 | 18


sensory technology is inactive and the sensory system may undergo some recalibration during the post-
protocol.
Post-Protocol: In the post-protocol a confirmation of receipt from the central pilot information database is
requested. Once this is received all local data is wiped and the system is once again readied to run pre-
flight operations to import and prep incoming pilot data.
Central Pilot Information Database
The central pilot information database is in principle a single distributed pilot data backing store.
Compressed and encrypted pilot data is received by the database and then stored for later use. In order to
decrease unnecessary complication the received data is decrypted and uncompressed and then combined
with any previous pilot data. Once a request is made to the database for data, all combined pilot data is
compressed and encrypted to be sent by SFTP [6] or similar secure transfer protocols to the intended on
plane system.
The database infrastructure is proposed as a frontal request and incoming worker backed by numerous
worker nodes. The cyber infrastructure used is not of any great importance, but large amounts of storage
will be required. Dedicated is most likely more suitable than cloud or grid based resources in this instance
due to the homogeny of operation and the singular large resource requirement.
VI. Cockpit Sensor Array
The cockpit sensor array consists of 5 sensors to detect various biometrics for the pilot. These sensors are:
Pressure sensor in the back of the seat to detect whether the pilot is in the seat and to measure
breathing
Accelerometer on the joystick to measure hand tremor
A Ventilation Chamber combined with a Thermocouple and Humidity sensor adapter mounted on
the joystick to measure perspiration
An ECG mounted to the headband of the pilots headset to monitor heart rate
A strain gauge built into the seatbelt to measure respiration

In addition to this there will be a camera mounted to the roof of the cockpit which can monitor the pilots
motor functions and a second camera also mounted to the roof of the cockpit that aims to detect pupil
dilation, which can occur if the pilot is under stress.
P a g e 8 | 18


Pressure sensor in the rear of the seat

Figure6:Proposedcockpitpressuresensorconfiguration[12]
A weight sensor in the rear of the pilot seat can be used to detect whether the pilot is currently sitting in
the chair. If the pilot is not present then the system will not record any data. Data recording will begin as
soon as the plane takes off with a registered pilot in the seat. The sensor itself consists of a series of strain
gauge resistors which generate an electrical signal in response to the weight on the seat from the
occupant. In addition to this, this sensor can also be used in conjunction with the seatbelt mounted
breathing sensor in order to more accurately measure the respiration biometric.
P a g e 9 | 18



Figure7:Graphshowingtheoutputsignalfromtheseatmountedsensor
The graph above shows a sample output from the weight sensor. At approximately 40 minutes the pilot sits
down and the flight begins. Approximately 200 minutes later, the pilot leaves the seat and the flight ends.
Note that when the pilot is sitting in the seat there is an extra signal from the pilots breathing.

0
1
2
3
4
5
6
7
8
0 50 100 150 200 250 300 350
V
o
l
t
a
g
e

f
r
o
m

S
e
n
s
o
r

(
V
o
l
t
s
)
Time (minutes)
Sensor Voltage Vs Time (minutes)
P a g e 10 | 18


Seatbelt Mounted Breathing sensor

Figure8:Proposedseatbeltmodification
The Seatbelt mounted breathing sensor uses the rise and fall of the pilots chest to calculate a breathing
rate. The breathing rate of the pilot is a key indicator to the pilots wellbeing. If the pilot is breathing
irregularly he may be experiencing health problems and these can be addressed on the landing of the
plane.
The seatbelt breathing sensor consists of a strain gauge that runs along the length of the seatbelt. When
the seatbelt is plugged in, the circuit is complete and the gauge is activated. When the pilot breathes the
rise and fall of his/her chest will cause the strain gauge to output a voltage. This signal is saved as raw data
and processed by the on board control system every 10 minutes with a sample speed of 2Hz.

P a g e 11 | 18


Figure9:SchematicfortheSeatbeltSensor
Joystick Sensor Systems

Figure10:Modificationtopilotsjoystick
There are two different sensors present on the Joystick of the plane; the Perspiration sensor and the
Accelerometer to detect hand tremor. The Joystick is the main hub for sensory data in the cockpit as hand
tremor and perspiration are two of the most prevalent symptoms of Stress.

P a g e 12 | 18


The Accelerometer
The Accelerometer will detect hand tremor. In order to rule out turbulence and the movement of the
plane two accelerometers will be used on the joystick and the difference between the two will provide the
control system with a value for pilot hand tremor. The accelerometer system has been used to measure
hand tremor with success in alternative studies [13]. The data from the Accelerometer will be sent to the
Control System which will aggregate the data at a sample speed of 2 Hz (2 samples per second). If the pilot
shows signs of serious hand tremor the Airline and Pilot will be given advice in order to ensure the pilot
stays in good mental health.

Figure11:Sampleaccelerometerreadings
Perspiration Sensor
In the report studied by the group, a Perspiration sensor was constructed using a Relative Humidity sensor
adapter (Specifically the THT-B121 resistive type transmitter), a Flow Meter and a calibrated type K
thermocouple to measure temperature. The perspiration sensor recorded the number of grams of
perspiration from the test subject. This could be implemented to the joystick of the aircraft and could
measure the amount of sweat on the hands of the pilot. The figure below shows the block diagram for the
P a g e 13 | 18


Perspiration measurement system. This sensor system in particular is crucial in determining the mental
wellbeing of the pilot as sweaty palms are a sign of agitation, anxiety, stress and paranoia.

Figure12:BlockDiagramforPerspirationSensor
P a g e 14 | 18


ECG Headset

Figure13:BasicECGconfiguration
Built into the headset is an ECG Sensor which is pressed against the pilots temple. The outboard
equipment for the ECG is stored within the control panel of the cockpit to allow for a light and comfortable
headset design. The ECG will record the heart rate of the pilot. The .edf files from the ECG will be sent to
the control system of the plane and if the heart rate is more rapid than the baseline profile then it can be
shown that the pilot is feeling stressed or under pressure and measures can be taken to ensure the pilots
health returns to normal.
Camera
A camera will be present in the cockpit in order to detect pilot movement. One of the key symptoms for
anxiety and stress is fidgeting and rapid twitching movements. The system will use edge detection
algorithms to determine any irregular pilot movement during a flight.
P a g e 15 | 18



Figure14:Samplecockpitviewusingedgedetection
Fingerprint Scanner

Figure15:Joystickwithaddedfingerprintscanner
A fingerprint scanner will be mounted to the top of the joystick will allow the pilot to login to the system,
thus activating it. The system will not record any data until a pilot with a profile has logged in. In order to
login the pilot simply presses a finger onto the scanner.
VII. Summary and Conclusion
This work outlines a proposal for a data based solution to the rising problem of monitoring commercial
pilot mental health. Described are three main features of a system to aid in the measurement and real-
time analysis of physical factors giving insight into the general mental health and wellness of active
commercial pilots and co-pilots. As mentioned these features are the baseline profile, cockpit sensor array
and the data processing.
P a g e 16 | 18


Research into this area shows the need for an elegant and efficient solution to monitoring pilot mental
health. At present pilot mental health checks are at the discretion of the airline and the FAA and current
active protocols are insufficient to ensure the long-term mental wellness of active pilots. As pilots are in a
position of great responsibility it is our hope that this work will, to some extent, shed light on the ease and
practicality with which a system may be implemented to help monitor pilot mental health.
The cost of the various system components will likely not exceed $5000. However, the central pilot
information database may require addition of extra cyber infrastructure on top of existing facilities. This
could be improve through the use of opportunistic and/or cloud resources. Added cyber infrastructure will
drive pricing up yet, in the context of the multi-billion airline industry the cost of retrofitting current
aircraft with our proposed system is effectively trivial. Large development cost is a one-time consideration
and will likely focus on the computer vision and data analysis side as the sensory and data technologies are
typical.
In conclusion it is our belief that the work outlined in this paper provides a viable solution to the
monitoring of pilot mental health, and with further development the implementation of such a system is
highly viable. Future work could include the use of similar sensors and techniques in a controlled
environment to provide proof of concept. With further developments this solution will not only ensure the
long term physical and mental health of commercial pilots but also provide greater peace of mind to
customers and companies of the aviation industry.
VIII. References
[1] SQL homepage http://www.sql.org/ August 2014
[2] OpenCV hompage http://www.opencv.org/ August 2014
[3] Python homepage https://www.python.org/ August 2014
[4] Olumofin, Femi and Goldberg, Ian. Privacy-preserving Queries over Relational Databases. Proceedings
of the 10th International Conference on Privacy Enhancing Technologies 75-92 2010

[5] Database compression techniques http://technet.microsoft.com/en-
us/library/dd894051(v=SQL.100).aspx August 2014
[6] SFTP overview http://compnetworking.about.com/od/ftpfiletransfer/g/sftp-definition.htm August 2014
[7] Rusche, D. (2012). The JetBlue pilot's breakdown and the high stress of 'safety sensitive
positions'. Available: http://www.theguardian.com/world/2012/mar/30/jetblue-pilot-breakdown-mental-
health. Last accessed 7th Aug 2014.
[8] .[J].199910(4):234.
[9] Wood, R. (2012). Resting Heart Rate Table. Available: http://www.topendsports.com/testing/heart-
rate-resting-chart.htm. Last accessed 8th Aug 2014.
P a g e 17 | 18


[10] Yingjui Tsai, Chiachung Chen. (2010). Development and testing of a perspiration measuring
system. Medical Engineering & Physics. 32 (356-362), 359.
[11] William Dayton, Patricia Collins, Michael Smith, and Ted Selker (2009).Measuring Hand Tremor with a
Mobile Device. Carnegie Mellon, Silicon Valley: Mobility Research Centre.

[12] Rainey, Robert R. (Elkhart, IN, US) . (2005). Vehicle seat weight sensor.Available:
http://www.freepatentsonline.com/6969809.html. Last accessed 8th Aug 2014.
[13] William Dayton, Patricia Collins, Michael Smith, and Ted Selker (2009).Measuring Hand Tremor with a
Mobile Device. Carnegie Mellon, Silicon Valley: Mobility Research Centre.
P a g e 18 | 18