Usability Evaluation of the Flight Simulators

Human-Computer Interaction

Yanbin Shi1,2(&) and Dantong Ouyang2

1
Aviation University of Air Force, Changchun 130022, China
shiyanbin_80@163.com
2
Key Laboratory of Symbolic Computation and Knowledge Engineering
of Ministry of Education, Jilin University, Changchun 130012, China

Abstract. Attention is concentrated on the usability of the flight simulators

Human-Computer Interaction system, especially on the cockpit display systems.
Based on the User-Centered Design tool, used the cloud model to evaluate the
usability of the cockpit display system from learnability, efciency, memora-
bility, errors and satisfaction. Cloud model based on grey system theory and the
normal grey whitenization weight function used in conversion between quali-
tative and quantitative uncertainty model.

Keywords: Usability
Flight simulator
User-Centered Design

Human-computer interaction
Multimodal

1 Introduction

A flight simulator is a device that articially re-creates aircraft flight and the envi-
ronment in which it flies, for pilot training, design, or other purposes [1]. It is used for a
variety of reasons, including flight training (mainly of pilots), the design and devel-
opment of the aircraft itself, and research into aircraft characteristics and control
handling qualities.
User-Centered Design (UCD) is the process of designing a tool, such as flight
simulators or applications user interface, from the perspective of how it will be
understood and used by a pilot. Rather than requiring pilots to adapt their attitudes and
behaviors in order to learn and use simulator, simulator can be designed to support its
intended pilots existing beliefs, attitudes, and behaviors as they relate to the tasks that
the simulator is being designed to support. Recent years, many organizations and
researchers have studied the coordination design theory and methods for aircraft
cockpit human-computer. Tang [2] and Zhang [3] considered the pilots cognition
characteristic and perception motion characteristic. Ai [4] researched the relationship of
the main equipments arrangement in the cockpit and the ergonomics. Wang [5] taking
the ecological interface design (EID) theory as the basis, summarized the guidelines of
designing the interface of the ghter cockpit.

Springer International Publishing Switzerland 2016

D.-S. Huang and K.-H. Jo (Eds.): ICIC 2016, Part II, LNCS 9772, pp. 693702, 2016.
DOI: 10.1007/978-3-319-42294-7_62
694 Y. Shi and D. Ouyang

2 Usability

According to the international standard, ISO 9241-11 [6], the denition of usability is
the extent to which a product can be used by specied users to achieve specied goals
with effectiveness, efciency and satisfaction in a specied context of use, which is
affected by the users, their goals and the usage situation.
Usability is dened by 5 quality components [7]:
Learnability: How easy is it for users to accomplish basic tasks the rst time they
encounter the design?
Efciency: Once users have learned the design, how quickly can they perform
tasks?
Memorability: When users return to the design after a period of not using it, how
easily can they reestablish prociency?
Errors: How many errors do users make, how severe are these errors, and how
easily can they recover from the errors?
Satisfaction: How pleasant is it to use the design?
There are many methods for studying usability, but the most basic and useful is
user testing, which has 3 components [8]:
Get hold of some representative users.
Ask the users to perform representative tasks with the design.
Observe what the users do, where they succeed, and where they have difculties
with the user interface.
In order to evaluate the usability of the flight simulators, many advanced and
primary pilots respectively be selected to complete designated training missions.
Recording what the pilots do and where they have difculties with the cockpit.

3 Flight Simulators Human-Computer Interaction

The cockpit display systems is the major human-computer interaction means, they
provide a visual presentation of the information and data from the aircraft sensors and
systems to the pilot to enable the pilot to operate the aircraft safely and carry out the
mission [9]. They are thus vital to the operation of any aircraft as they provide the pilot,
whether civil or military, with primary flight information, navigation information,
engine data, airframe data and warning information. The military pilot has also a wide
array of additional information to view, such as: infrared imaging sensors, radar, tac-
tical mission data, weapon aiming, threat warnings.
The pilot is able to rapidly absorb and process substantial amounts of visual
information but it is clear that the information must be displayed in a way which can be
readily assimilated, and unnecessary information must be eliminated to ease the pilots
task in high work load situations. A number of developments have taken place to
improve the pilotdisplay interaction and this is a continuing activity as new tech-
nology and components become available. Examples of these developments are Head
up displays (HUD), Helmet mounted displays (HMD), Multi-function colour displays,
 Usability Evaluation of the Flight Simulators Human-Computer Interaction 695

Digitally generated color moving map displays and Synthetic pictorial imagery, Dis-
plays management using intelligent knowledge based system (IKBS) technology,
Improved understanding of human factors and involvement of human factors spe-
cialists from the initial cockpit design stage.
Equally important and complementary to the cockpit display systems in the
human-computer interaction are the means provided for the pilot to control the oper-
ation of the avionic systems and to enter data [10]. Again, this is a eld where continual
development is taking place [11, 12]. Multi-function keyboards and multi-function
touch panel displays are now widely used. Speech recognition technology has now
reached sufcient maturity for direct voice input control to be installed in the new
generation of military aircraft. Audio warning systems are also now well established in
both military and civil aircraft. The integration and management of all the display
surfaces by audio/tactile inputs enables a very signicant reduction in the pilots
workload to be achieved in the new generation of single seat ghter/strike aircraft.
Other methods of data entry which are being evaluated include the use of eye trackers,
sounds and body languages.
In this research, as the major HCI system, the power gauges, turn & slip, compass,
altimeter, ASI, AI and VSI are considered.

4 Evaluation Index System

A usable system is created by getting hold of enough of the right usability information
and using the information well. Usually people thinking that measured values are
quantitative and that criteria application yields quantitative results, that it is important
that the target value of a usability evaluation criterion can be either quantitative or
qualitative. Traditional, there are generally three types of usability evaluation methods
[1316]: Testing, Inspection, and Inquiry. In Usability Testing approach, representative
users work on typical tasks using the system (or the prototype) and the evaluators use
the results to see how the user interface supports the users to do their tasks. In Usability
Inspection approach, usability specialists and sometimes software developers, users
and other professionals examine usability-related aspects of a user interface. In
Usability Inquiry approach usability evaluators obtain information about users likes,
dislikes, needs, and understanding of the system by talking to them, observing them
using the system in real work (not for the purpose of usability testing), or letting them
answer questions verbally or in written form.
In order to evaluate the usability of the flight simulator, these three kinds of
methods all be used in studying. The factors which influence the efciency of the flight
simulators human-computer interaction mainly include: learnability, efciency,
memorability, errors and satisfaction. Here, based on the grey system theory, using
cloud model [1726] and the normal grey whitenization weight function to evaluate the
usability of the flight simulator.
696 Y. Shi and D. Ouyang

4.1 Index Quantization and Weight Value

The evaluation model of the flight simulators human-computer interaction is multi-
indexes issue, a prominent characteristic is that these indexes without unied a mea-
suring standard, their fetching value ranges of initial data and tolerance units have
nothing in common with each other, it is unable to compare each other. So, assess the
question as to many indexes of a known index attribute value, before conrming the
index weight and carrying on comprehensive evaluation, must have no dimension
treatment (standardization or standardize to the attribute value of the index), its essence
is to utilize certain mathematics to vary and turn the attribute value with different
dimension, property into by comprehensive treatment Quantization value, Generally
vary every attribute value to the range of fetching value in unison. In assessing models,
most index have clear number value, and it can utilize method of normalization their in
unison for have dimension, distribute in [0, 1] number of the space to value. But some
indexes can only be described in natural language by qualitative method, such as
satisfaction, this kind index can utilize the cloud model to determine the natural to
quantitative good conversion.

4.2 Normalization of the Index

The normalization method should carry on according to the request of index type,
practical problem characteristic and policymaker, assess index can be divided into four
kinds, they are benet type, cost type, xed type and interval type.
If labeled the subscripts set of all of the indexes fi i = 1; 2;


; m as
U f1; 2;


; mg, xij is the original value, yij is the normalization value, the ways of
the dimensionless are:

About the benet type index valve: yij = xij max xij ;


About the cost type index valve: yij = min xij xij ;
About the xed type index valve:
8  
<  1  xij  xi
yij      ; 1
 
: 1  xij  xi  
max xij  xi xij 6 xi
1jn

About the interval type index valve:

8  

< 1  xLi  xij =gi xij \xLi

yij  1 
xij 2 xLi ; xUi 2
:
1  xij  xU
i =gi xij [ xUi

Where, i = 1; 2;


; m (m is the number of the indexes), j = 1; 2;


; n (n is the

number of the objects for evaluating); xi is the optimal value of the xed type index

valve; closed interval xLi ; xU i is the optimal interval, and gi = maxxLi  min xij ;
u
max xij  xi .
 Usability Evaluation of the Flight Simulators Human-Computer Interaction 697

4.3 Conversion Method of the Natural Language from Qualitative

to Quantitative
The normal Cloud is the most basic tool to express the language value, and can be
generated by the Clouds digital character (Ex; En; He), and its mathematic expec-
tation curve (MEC) is:

MECA x = expx  Ex2 /2En2  3

The generating algorithm of the normal Cloud is that:

(1) xi = G(Ex; Eneih , generating a normal random number xi , whose expected
value is Ex and standard deviation is En;
0 0
(2) Eni = G(En; He), generating a normal random number Eni , whose expectation
value is En and standard deviation is He ;
2
(3) Calculate li = exp[  xi2E
Ex
0 , (xi ; li ) is a cloud drop.
ni
(4) Repeat the step 1 to step 4, until the Cloud drops were generated enough.
Figure 1 gives out well achievements languages which were described by one-
dimension Cloud.

Fig. 1. Straight attitude cloud models and digital characteristics

Cloud is combined by many Cloud drops, and a Cloud drop may be footy, but the
whole shape of the Cloud reflects the important characters of the quantity concept [6].
Cloud is used to compare to an uncertainty mapping between quantity and quality
because this distribution looks like the Cloud in the sky, and it has a clear shape when
far looking, but there is no borders in nearly. The digital character of Cloud can be
expressed by Ex(Expected value), En(entropy) and He(Hyper entropy).
Expected value Ex is the center value of concept in the theory eld, and its the
representative value of the qualitative concept.
Entropy En is the measuring of the fuzziness of qualitative concept, reflects the
numerical range which can be accepted by this concept in the theory eld, and
698 Y. Shi and D. Ouyang

embodies the uncertain margin of the qualitative concept. The bigger the entropy is, the
bigger numerical range can be accepted by the concept, and the fuzzier are the
concepts.
Hype Entropy He reflects the dispersion of the Cloud drops. The bigger the Hyper
Entropy is, the bigger of its dispersion and the randomness of degree of membership,
and this is the Clouds thickness.
So, the fuzziness (the uncertainty of qualitative concept) and randomness (ran-
domness of degree of membership) are completely combined together by three char-
acteristic numbers of the Cloud models, which make up the mapping between quality
and quantity as the foundation of knowledge expression.
According to the habit of natural languages of people, the performance of
description things of determining the natural generally uses ve graduation laws, i.e.
better, good, general, difference and worse. Using the cloud model to
express these ve formulations, the desired value Ex can be expressed by 1, 0.75, 0.5,
0.25 and 0; but at the same time, these values are approximate, so they can be adjusted
by the entropy, En = 0.04, by this way, the cloud drops can cover the [0, 1] interval;
Hype Entropy He = 0.005, as showed by the Fig. 2.

Fig. 2. Natural description language changed to quantitative cloud

4.4 Quantization of the Weight Value

The right value also can adopt cloud model, conrm right value adopt nine graduation
law generally, the [0, 1] block turns nine graduations, i.e. 0.1, 0.2, 0.3, 0.4, 0.5, 0.6,
0.7, 0.8, 0.9, The important intensity shown is increased progressively sequentially, the
importance can be adjusted through the determination of entropy value. Figure 3 shows
the cloud of right value.

4.5 Arithmetic of Clouds

The arithmetic of the cloud is the algorithm changing the base cloud to the empty cloud.
The so-called empty cloud means the digital characteristics of some already designated
clouds will carry on a certain operation, cloud getting the new digital characteristic and
 Usability Evaluation of the Flight Simulators Human-Computer Interaction 699

Fig. 3. Weight value cloud chart expresses by nine graduation law

constructing. These designated clouds used for calculating the empty cloud are called
the base cloud. Because in exceeding the evaluation research of the efciency of the
flight simulators Human-Computer Interaction system, only use the weighted sum
algorithm, so only introduce addition and multiplication of the cloud here.
If there is a given cloud in xed domain C1 Ex1 ; En1 ; He1 , C2 Ex2 ; En2 ; He2 , the
calculated result of arithmetic operation for C1 and C2 is CEx ; En ; He .
Its operation of multiplication can be expressed as:

Ex = Ex1 Ex2 ; 4
s
 2  2
En1 En2
En = jEx1 Ex2 j + ; 5
Ex1 Ex2
s
 2  2
He1 He2
He = jEx1 Ex2 j + ; 6
Ex1 Ex2

Its operation of addition can be expressed as:

q q
Ex = Ex1 + Ex2 ; En R En12 + E2 ; H =
n2 e He1 2 + H2
e2

After get the positive-going, antidromic generate algorithm and the weighted sums
algorithm, can use the cloud model theory to carry on the evaluation of efciency for
the Human-Computer Interaction system.

5 The Evaluation Example

In order to get the usability information, research chooses 10 pilots with different flying
experience, aiming for a reasonable balance regarding age and rank, i.e. at least 40 %
must be primary and at least 40 % must be senior rank pilots. About age, approxi-
mately one-third must be under 30 years old, between 30 and 40 and above 40 %.
700 Y. Shi and D. Ouyang

Regarding operation skills, the representative user group will divide subjects into
two groups, each balanced with respect to age, i.e.:
(1) Subjects who have little experience in specied flight simulators human-computer
interaction system but have strong interest in flying.
(2) Subjects who are familiar with specied flight simulators human-computer
interaction system.
So, the 10 pilots including 4 primary ranks and 4 senior ranks, the other 2 are
middle rank pilots.
Through the meetings with discussion, observation and interaction with system
three kinds methods to get the usability information. During the pilots completing
interaction with cockpit display system, the Digital Airborne Video Recorder recorded
the pilots sight scan path in different training process or different mission.
Choose four kinds of flight simulators human-computer interaction, about
availability in Fig. 1, the index and afliated subordinate index are calculated. The
experts give the ve indexes learnability, efciency, memorability, errors and satis-
faction, respectively a different weight assignment 0.2, 0.2, 0.4, 0.2, and
change them into digital characteristic of the cloud, likewise, the experts provides and
appraises languages to the system among them, the digital characteristic changed into
the cloud as shown in Table 1.

Table 1. The weight and evaluation cloud of the availability

Index Weight Weight cloud Evaluation cloud
Learnability 0.2 (0.2, 0.04, 0.005) (0.9, 0.04, 0.005)
Efciency 0.2 (0.2, 0.02, 0.005) (0.4, 0.02, 0.005)
Memorability 0.2 (0.2, 0.08, 0.005) (0.3, 0.02, 0.005)
Errors 0.2 (0.2, 0.02, 0.005) (0.6, 0.02, 0.005)
Satisfaction 0.2 (0.2, 0.02, 0.005) (0.2, 0.01, 0.005)

Fig. 4. Cloud chart for the availability index

 Usability Evaluation of the Flight Simulators Human-Computer Interaction 701

According to number value described above and weighted sum operation method of
the cloud, it can be obtained through calculating, the system is shown in Fig. 1, the
digital characteristic of cloud of the learnability index is: (0.42, 0.0458, 0.0058), its
cloud chart is shown as Fig. 4.

6 Conclusion

This paper analyzed the index system of Human-Computer Interaction system for
usability evaluation, and representative evaluation model has been set up. The nor-
malization method of the index and the conversion method of the natural language
from qualitative to quantitative are described. Taking the flight simulators
human-computer interaction in four different periods to evaluate, which accords with
the evaluation result. According to the model, sensitivity analysis can be carried on, by
thus promote the usability of flight simulators human-computer interaction system.

Acknowledgement. This work is supported by the Education Department of Jilin Province

13th Five-Year scientic research planning project (2016-514).

References
1. Flight Simulator (2016). www.baidubaike.com
2. Tang, Z., Zhang, A., Bi, W.: Aircraft Cockpit Human-Computer Coordination Design
Theory and Methods. Northwestern Polytechnical University Press, xi'an (2015)
3. Zhang, W., Ma, Z., Yu, J.: Human-Computer Integrative Design of the Civil Aircraft
Cockpit. Northwestern Polytechnical University Press, xi'an (2015)
4. Ai, L.-Y.: Ergonomics-apply-research on cockpits equipment arrangement. Aircr. Des. 32,
7880 (2012)
5. Wang, H., Bian, T., Xue, C.: Layout design of display interface for a new generation ghter.
Electro-Mech. Eng. 27, 5761 (2011)
6. ISO 9241-11:1998: Ergonomic requirements for ofce work with visual display terminals
(VDTs) Part 11: Guidance on usability (2016). http://www.iso.org
7. Niels, O.B., Laila, D.: Multimodal Usability, p. 68. Springer, London (2009)
8. Collinson, R.P.G.: Introduction to Avionics Systems, 3rd edn, pp. 1920. Springer,
Heidelberg (2011)
9. Kong, Y., Qu, K.: Cockpit Engineering. Aviation industry press (2015)
10. Guo, Y., Gu, X., Shi, L.: Virtual Reality and Interaction Design. Wuhan University Press
(2015)
11. Tan, H., Tan, Z., Jin, C.: Automative Human Machine Inerface Design. Publishing House of
Electronics Industry (2015)
12. Zhao, C.: Usability Evaluation and Development for Mobile Phone Products Based on
Simulator. Dalian Maritime University (2004)
13. Hua, Q.: Research on Usability Oriented Software Interaction Design of Smart Phone.
Northwestern University (2011)
14. Whitten, A., Tygar, J.D.: Why Jonny cant encrypt: a usability evaluation of PGP 5.0. In:
Proceedings of the Ninth USENIX Security Symposium, p. 14 (1999)
702 Y. Shi and D. Ouyang

15. Usability Evaluation (2016). http://www.usabilityhome.com/

16. Wang, J., Xiao, W., Sheng, W.: An improved effectiveness evaluation method based on grey
cloud model. Microcomput. Inf. 4, 277278 (2009)
17. Zhang, Y., Wu, L., Wei, G.: Improved ant colony algorithm based on membership cloud
models. Comput. Eng. Appl. 27, 1114 (2009)
18. Yang, S.: An improved system cloud grey neural network model. In: Proceedings-2nd
International Conference on Genetic and Evolutionary Computing, pp. 158160 (2008)
19. Wang, G.: Rough set based uncertain knowledge expressing and processing. In: Kuznetsov,
S.O., lzak, D., Hepting, D.H., Mirkin, B.G. (eds.) RSFDGrC 2011. LNCS, vol. 6743,
pp. 1118. Springer, Heidelberg (2011)
20. Zhao, T., Hu, J.: Performance evaluation of parallel le system based on lustre and grey
theory. In: Proceedings 9th International Conference on Grid and Cloud Computing,
pp. 118123 (2010)
21. Chen, Z., Yu, B., Hu, W., Luo, X., Qin, L.: Grey assessment and prediction of the urban heat
island effect in city. J. Xian Jiaotong Univ. 38, 985988 (2004)
22. Chen, Y.: The comprehensive ranking evaluation of flood disaster based on grey-cloud
whitening-weight function. In: Proceedings of 2011 International Conference on Electronic
and Mechanical Engineering and Information Technology, vol. 4, pp. 19321934 (2011)
23. Zhang, G., Zhang, M., Zhong, H., Huang, X.: Research of transportation scientic and
technological projects evaluation based on improved triangular whitenization weight
function. 2010 International Conference on Logistics Systems and Intelligent Management,
vol. 3, pp. 15001503 (2010)
24. Liu, J., Xiao, M., Li, B.: An evaluation model based on grey trigonometry whitenization
weight function for logistic center. In: Proceedings of the 2nd International Conference on
Transportation Engineering, pp. 345, 20552060 (2009)
25. Zeng, B., Liu, S.-F., Cui, J.: Prediction model for interval grey number with known
whitenization weight function. Control Decis. 25, 18151820 (2010)
26. Yang, X.-W., Wu, S.-L., Xu, C.-R., Jie, C., Shun-Peng, Z.: The comparison of two types of
whitenization weight function. In: 2009 International Conference on Apperceiving
Computing and Intelligence Analysis, pp. 6972 (2009)