GENERIC FAULTS - THE FIRST WORD

Dennis G Cannon

ABSTRACT

The s u b j e c t o f g e n e r i c f a u l t s and g e n e r i c computer systems and t h e e r r o r s t h a t soon

e r r o r s i s n o t something new. People have been plagued users q u i c k l y l e d t o pragmatic f a c t i o n s
concerned about t h e occurrence o f e r r o r s o f a l l which d e a l t p r i m a r i l y w i t h c r e a t i n g systems t h a t
types i n f l i g h t c o n t r o l systems f o r many years. c o u l d t o l e r a t e f a u l t s r a t h e r t h a n t h o s e which
However, g e n e r i c e r r o r s i s a t o p i c which i s were e r r o r f r e e ( o r p e r f e c t ) systems. Although
extremely controversial, and t h e c o n t r o v e r s y s i g n i f i c a n t progress has been made i n t h e area
stems l a r g e l y f r o m t h e f a c t t h e r e has been a o f c r e a t i n g l i m i t e d f a u l t t o l e r a n t systems, we
v e r y poor language standard used t o d i s c u s s t h e are s t i l l plagued w i t h t h e occurrence o r
phenomena o f " g e n e r i c " e r r o r s . The a b i l i t y t o p o t e n t i a l occurrence o f t h o s e t y p e e r r o r s which
improve t h e q u a l i t y o f b o t h hardware and can render even redundant systems i n o p e r a t i v e ,
s o f t w a r e a v i o n i c s equipment f o r f u t u r e f l i g h t v i z . t h e so-called "generic" e r r o r .
c r i t i c a l c o n t r o l system a p p l i c a t i o n s i s l a r g e l y
e f f e c t e d by how p r e c i s e l y one i s a b l e t o The l a c k o f a standard language and poor
identify those elements that presently d e f i n i t i o n o f terms r e s u l t e d i n s u b s t a n t i a l
c o n t r i b u t e t o degraded q u a l i t y . The f a c t t h a t miscommunication about e a r l y computing systems
c e r t a i n f a u l t s do e x i s t , g i v i n g r i s e t o e r r o r s and about f a u l t t o l e r a n c e concepts. This i s a
and f a i l u r e s t h a t o c c u r almost s i m u l t a n e o u s l y i n p a r t i c u l a r l y s i g n i f i c a n t i s s u e when one r e a l i z e s
each separate channel o f a redundant system t h a t t h e concepts r e l a t i n g t o f a u l t t o l e r a n t
c o n f i g u r a t i o n , can be a s e r i o u s problem w i t h systems, redundant systems, issues of
t h e i r occurrence independent o f how p r e c i s e l y o r architecture, error detection, and error
vagL. .y t h e y a r e d e s c r i b e d . T h i s paper i s based prevention are sometimes quite difficult
upon t h e a s s e r t i o n t h a t " g e n e r i c t 1 f a u l t s do n o t t e c h n i c a l concepts t o understand; u l t i m a t e l y ,
exist. system d e s i g n e r s d i v i d e d i n t o two camps. One
camp i s devoted t o c r e a t i n g s i m i l a r systems
BACKGROUND u s i n g s i m i l a r a r c h i t e c t u r e i n hardware and
s o f t w a r e , and another camp e q u a l l y sure t h a t t h e
The s u b j e c t o f g e n e r i c e r r o r s i n i t s most o n l y way t o achieve h i g h l y r e l i a b l e systems i s
broaa f o r m i s n o t something new. People have t o use d i s s i m i l a r hardware and d i s s i m i l a r
been concerned about t h e occurrence o f e r r o r s o f software.
a l l t y p e s i n f l i g h t c o n t r o l systems f o r many
years. More s p e c i f i c a l l y i n t h e computer r e l a t e d Another d i v i s i o n among t h e e n g i n e e r i n g teams
f i e l d s and p a r t i c u l a r l y i n t h e e a r l y development i s i n r e l a t i o n t o t h e k i n d o f s o f t w a r e language
stages o f d i g i t a l computers, t h e r e were o f t e n ( p r o g r a m i n g language) t h a t must be used. One
e r r o r s which o c c u r r e d t h a t seemed t o be somewhat camp f a v o r s h i g h o r d e r languages w h i l e another
" u n e x p l a i n a b l e by t h e u s e r s . " These e r r o r s were camp f a v o r s t h e lower l e v e l (assembly) language
l o o s e l y r e f e r r e d t o as " g e n e r i c " e r r o r s by t h e approach. U n f o r t u n a t e l y . t h e s e d i s t i n c t i o n s were
users of those computer systems. The sometimes t h e r e s u l t o f business pressures,
seriousness o f these s o - c a l l e d " g e n e r i c " e r r o r s p r e j u d i c e s o r o u t r i g h t stubborness and were n o t
was immediately recognized by developers o f the result of making clear technical
computing systems and t h e y s e t about w i t h distinctions between what one specific
v a r i o u s p l a n s t o i d e n t i f y , i s o l a t e and c o n t r o l a r c h i t e c t u r a l c o n s i d e r a t i o n o r one s o f t w a r e
those t y p e s o f e r r o r s . The o v e r a l l goal o f t h e implementation technique had t o o f f e r o v e r
developer was t o simp1y promote systems which another.
behaved as expected; t h a t i s t o say, e r r o r f r e e
and f a u l t t o l e r a n t . Based on t h e r e c o r d o f o u r p a s t performance,
o u r a b i l i t y t o d e s i g n e r r o r f r e e systems and
Today t h e s u b j e c t o f g e n e r i c e r r o r s i s a e f f e c t i v e f a u l t t o l e r a n c e systems has n o t been
t o p i c which i s e x t r e m e l y c o n t r o v e r s i a l , and t h e totally successful. We have experienced
c o n t r o v e r s y stems l a r g e l y f r o m t h e f a c t t h e r e shortcomings by people a t a l l l e v e l s : system
has been a v e r y poor use o f a standard language d e f i n e r , system designer, system implementer.
t o d i s c u s s t h e phenomena o f " g e n e r i c " e r r o r s . system testers, system user, and system
The l a c k o f s t a n d a r d i z a t i o n was aggravated by m a i n t a i n e r . However, we a l s o have h i s t o r i c a l l y
t h e f a c t t h a t q u i t e o f t e n t h e r e was an a t t e m p t n o t been pressed t o r e l y 100 p e r c e n t on t h e
t o g l o s s o v e r these e r r o r s t h a t had been p r o d u c t s o f o u r design, because i n t h e event o f
committed as a d e f e n s i v e a c t i o n by those who had a system m a l f u n c t i o n , f l i g h t crews were t h e r e t o
made t h e e r r o r s and who were a l s o r e s p o n s i b l e be r e l i e d on t o r e c o n f i g u r e t h e system o r even
f o r f i n d i n g t h e e r r o r and u l t i m a t e l y c o r r e c t i n g t o d i s c o n n e c t t h e system t o t a l l y and t o f l y t h e
t h e system. The r a p i d growth o f d i g i t a l v e h i c l e manually.
Released to AIAA to publish in all forms.
 The f u t u r e a p p l i c a t i o n s o f f 1y-by-wi r e and occurred; t h e f a c t o f nonoccurrence; l a c k o f
f l y - b y - l i g h t f l i g h t c o n t r o l systems w i l l t a k e s a t i s f a c t o r y performance o r e f f e c t ; a f a i l i n g o r
away some o f t h a t cushion t h a t we've had i n t h e a lapse; d e f i c i e n c y and a l a c k ; t h e f a c t o f
past. I n o r d e r t o achieve t h e performance b e i n g c u m u l a t i v e l y inadequate o r n o t matching
b e n e f i t s t h e p a y o f f t o t h e users o f t h e s e hopes o r e x p e c t a t i o n s ; further it i s the
airplanes b o t h commercial and m i l i t a r y , t h e i n a b i l i t y t o perform a v i t a l function; a
a v i o n i c system d e s i g n e r s have much g r e a t e r collapsing, f r a c t u r i n g o r g i v i n g way under
i n c e n t i v e t o make s u r e t h a t t h e systems t h e y s t r e s s .I1
design will operate with extremely high
r e l i a b i l i t y , a v a i l a b i l i t y . and m a i n t a i n a b i l i t y With regard t o generic versus specific
w h i l e b e i n g as s a f e as c u r r e n t systems o r s a f e r . e r r o r s , f a u l t s and f a i l u r e s , what i s r e a l l y t r u e
i s t h a t people make q u i t e s p e c i f i c e r r o r s which
DEFINITION OF TERMS lead to certain specific faults in the
definition, design, construction, and
It i s necessary t o d e f i n e s e v e r a l terms q u i t e maintenance o f hardware and s o f t w a r e elements;
s p e c i f i c a l l y so t h a t we can be c l e a r when we t h i s i s i l l u s t r a t e d i n F i g u r e 1. These f a u l t s ,
d i s c u s s t h e process o f c o n t r o l l i n g f a u l t s i n which a r e a g a i n q u i t e s p e c i f i c . g i v e r i s e t o
future fly-by-wire systems. L e t us f i r s t f a i l u r e s o f t h e intended f u n c t i o n o f t h e
dispense w i t h t h e t e r m " g e n e r i ~ . ' ~ I a s s e r t t h a t hardware and s o f t w a r e system. Again each o f t h e
g e n e r i c f a u l t s do n o t e x i s t , and t h a t what has errors, faults, and failures were quite
been c a l l e d llgenericll are r e a l l y "specific" s p e c i f i c ; t h e y were n o t g e n e r i c . Such d e f e c t s
faults. can occur i n each s i m i l a r o r d i s s i m i l a r channel
o f a redundant system even i f t h e channels a r e
I f one c o n s u l t s t h e d i c t i o n a r y (Ref. 1 ) you "independent", s i n c e i n a c t u a l f a c t , something
will f i n d t h a t t h e t e r m " g e n e r i c " i s an may have made them dependent. They may o n l y
a d j e c t i v e which means: " r e l a t i n g o r applied t o appear t o be independent.
o r d e s c r i p t i v e o f a l l members o f genus, species,
c l a s s o r group, common t o a c h a r a c t e r i s t i c o f a
whole group or class; not specific or
i n d i v i d u a l , g e n e r a l ; a v a i l a b l e f o r comnon use, PEOPLE WHICH PRODUCE
SYSTEM
n o t p r o t e c t e d by trademark o r r e g i s t r a t i o n .
non-proprietary.. ." HARDWARE SERVICE
SOFTWARE
To be more c l e a r about " g e n e r i c " f a u l t s n o t
b e i n g g e n e r i c , b u t b e i n g s p e c i f i c , i t ' s good t o
examine and c o n t r a s t t h e meaning o f s p e c i f i c
with generic. Specific is defined as
" c o n s t i t u t i n g o r fa1l i n g i n t o the category Figure 1 The Reiationship o f Errors,
specified; i t i s being p e c u l i a r t o t h e t h i n g o r Faults, and Failures
r e l a t i o n i n q u e s t i o n ; r e s t r i c t e d by n a t u r e t o a
particular individual, situation, relation o r
affect; it is peculiar; further it is F a u l t s may a l s o be c l a s s i f i e d as l a t e n t
c h a r a c t e r i z e d by p r e c i s e f o r m u l a t i o n o r a c c u r a t e f a u l t s o r exposed f a u l t s i . e . f a i l u r e s . The
r e s t r i c t i o n ; f r e e f r o m such a m b i g u i t y as r e s u l t s d i s t i n c t i o n here i s t h a t l a t e n t f a u l t s have y e t
f r o m c a r e l e s s l a c k o f p r e c i s i o n o r f r o m omission t o be d e t e c t e d o r t o be exposed through any k i n d
o f pertinent matter." of a failure. Be v e r y c l e a r , however, t h a t t h e
human e r r o r has a l r e a d y been committed and t h a t
So i t appears obvious t h a t when t h e t e r m has r e s u l t e d i n t h e f a u l t . I n o t h e r words t h e
g e n e r i c f a u l t i s used i t i s t r u l y a misnomer. q u a l i t y o f t h e p r o d u c t i s s t i l l i n q u e s t i o n as
It i s n o t a g e n e r i c f a u l t t h a t i s common t o o r l o n g as t h e r e a r e s i g n i f i c a n t l y many l a t e n t
c h a r a c t e r i s t i c o f t h e whole group o r c l a s s t h a t f a u l t s p r e s e n t i n t h e hardware o r s o f t w a r e
causes problems; i t i s a v e r y s p e c i f i c f a u l t . element. The o p p o s i t e o f a l a t e n t f a u l t i s a
f a u l t t h a t g e t s exposed through an a c t u a l
It i s a l s o i m p o r t a n t t o draw d i s t i n c t i o n s physical f a i l u r e . The e f f e c t o f each s p e c i f i c
between f a u l t s , e r r o r s and f a i l u r e s . Fault i s a f a u l t must be q u a n t i f i e d as e i t h e r b e i n g
noun d e f i n e d as, "a d e f e c t i n q u a l i t y o r significant or insignificant. Certain f a u l t s
constitution; a defect in character or a r e never observed because t h e i r e f f e c t i s n o t
d i s p o s i t i o n ; a f a i l i n g ; a weakness; It i s a detectable/measurable.
f l a w o r blemish.
Another t e r m we need t o define quite
An error i s "an a c t o; a c o n d i t i o n o f o f t e n specifically i s specification. The a b i l i t y t o
i g n o r a n t , imprudent d e v i a t i o n f r o m a code o f comnunicate s p e c i f i c a t i o n s w i l l be discussed
behavior. F u r t h e r , i t ' s an a c t i n v o l v i n g an l a t e r i n t h i s paper. A s p e c i f i c a t i o n i s "the
u n i n t e n t i o n a l d e v i a t i o n f r o m t r u t h o r accuracy, a c t o r process o f i d e n t i f y i n g o r making s p e c i f i c
a m i s t a k e i n p e r c e p t i o n , reasoning r e c o l l e c t i o n through t h e s u p p l y i n g o f p a r t i c u l a r i z i n g d e t a i l ;
o r expression. An a c t t h a t through ignorance, a decreasing o f a g e n e r a l i t y o r vagueness as o f
deficiency o r accident departs from o r f a i l s t o a concept by determining or supplying
achieve what should be done." c h a r a c t e r i s t i c s t h a t d e l i m i t a more p r e c i s e
applicability; i t i s a detailed, precise,
A f a i l u r e i s a noun meaning, "omission o f e x p l i c i t p r e s e n t a t i o n o f something, a p l a n o r
performance o f an a c t i o n o r t a s k ; n e g l e c t o f an proposal f o r something as a w r i t t e n statement
assigned expected o r a p p r o p r i a t e a c t i o n ; t h e c o n t a i n i n g a minute d e s c r i p t i o n o r enumeration
f a c t o f a c e r t a i n a c t i o n o r process n o t h a v i n g of particulars.
 We w i l l a l s o need t h e d e f i n i t i o n o f t h e t e r m I t i s v e r y c l e a r t h a t where we a r e a i m i n g t o
management. Management is "direction, end up w i t h o u r systems i s t o p r o v i d e commercial
a d m i n i s t r a t i o n ; i t i s an overseer; management i s a v i o n i c s and v e h i c l e s t h a t use those a v i o n i c s
t h e process o f j u d i c i o u s use o f means t o systems t h a t a r e s a f e and economical f o r t h e
accomplish an end; i t i s a s k i l l f u l t r e a t m e n t ; m i l i t a r y and a i r l i n e s t o operate, systems t h a t
j u d i c i o u s means t o be d i r e c t e d o r governed by are highly r e l i a b l e and t h a t a r e e a s i l y
sound judgment; wise means." So management has maintained.
t h e t a s k t o e s t a b l i s h and m a i n t a i n t h e p h y s i c a l
o r g a n i z a t i o n , t h e procedures and p r a c t i c e s by I n o r d e r t o achieve o u r goal w i t h f l y - b y - w i r e
which t h e systems a r e developed. systems, t h e r e a r e some obvious d i r e c t i o n s we
must t a k e f r o m where we a r e now. It involves
We w i l l a l s o need t h e t e r m comnunication o r having everyone who participates in the
t o comnunicate. Comnunicate means, " t o i m p a r t ; development process t a k e a c r i t i c a l l o o k a t
t o convey; t o make known; i t means t h a t which i s e x a c t l y how t h e y a r e d o i n g business today and t o
comnunicated; t h e a c t o r p r a c t i c e o f conveying, ask t h e q u e s t i o n , " w i l l t h i s process l e a d t o a
i m p a r t i n g o r making known." The a b s o l u t e h i g h q u a l i t y r e l i a b l e system?" The key elements
importance of effective and positive o f such i n t r o s p e c t i o n a r e shown i n F i g u r e 2.
communication w i l l be expanded subsequently.
F i g u r e 2 shows t h e p r o g r e s s i o n o f a f a u l t
FUTURE DIRECTION through development and testing to its
m a n i f e s t a t i o n as a r e a l f a i l u r e d u r i n g s e r v i c e
I n o r d e r t o understand t h e f u t u r e d i r e c t i o n , use. There a r e two v e r y i m p o r t a n t aspects shown
we f i r s t have t o be v e r y c l e a r about e x a c t l y i n t h i s figure. F i r s t , t h a t attempts t o c o r r e c t
where we a r e now. As engineers and managers, we a f a u l t , i . e . t o remove f a u l t s , may n o t be
do n o t have a s p o t l e s s t r a c k r e c o r d r e g a r d i n g successful. I n f a c t , i f one f a u l t e f f e c t has
o u r a b i l i t y t o produce hardware and s o f t w a r e p a r t i a l l y masked a n o t h e r f a u l t e f f e c t , i t may be
a v i o n i c systems t h a t a r e e r r o r f r e e . We have p o s s i b l e t h a t t h e t o t a l system performance a f t e r
r e l i e d e x t e n s i v e l y on crew i n t e r v e n t i o n t o c o r r e c t i n g t h e f a u l t i s much worse t h a n b e f o r e
complete a m i s s i o n when c o n f r o n t e d w i t h what the correction. The second p o i n t i s t h a t t h e
m i g h t o t h e r w i s e have been t o t a l f a i l u r e o f t h e e f f e c t o r r e s u l t o f any one f a i l u r e ranges from
system. We have a l s o r e l i e d p r i m a r i l y on s i m p l y i m p e r c e p t i b l y low t o p o t e n t i a l l y c a t a s t r o p h i c
adding people t o s o l v e d i f f i c u l t development proportions. The o b j e c t i v e s and methods t h a t
problems when a b e t t e r d i r e c t i o n m i g h t have been are subsequently proposed are aimed at
to consider the effectiveness of the m i n i m i z i n g t h e e f f e c t o f a l l f a i l u r e s t o be l e s s
o r g a n i z a t i o n and development techniques t h a t t h a n annoying.
were b e i n g u t i l i z e d .
AREAS OF MOST SIGNIFICANT IMPACT
Managers o f t e n g e t v e r y d e d i c a t e d t o i n t e r n a l
m i l e s t o n e s t h a t may have been e s t a b l i s h e d The h e a r t o f t h e m a t t e r f o r t h e development
w i t h o u t a c l e a r understanding o f t h e t e c h n i c a l o f h i g h q u a l i t y a v i o n i c systems i s i n t h e area
issues: t h e near t e r m problems a r e t h e focus. o f p r e c i s e communication. The f i r s t source o f
e r r o r s which l e a d t o f a u l t s which l e a d t o

I CATASTROPHIC

0 UNACCEPTABLE
(MORE COSTLY)

DISRUPTIM/COSTLY

IMPERCEPTIBLE

Figure 2 Assessment o f E f f e c t or Result

subsequent e r r o r s , f a u l t s and f a i l u r e s which t h e t e c h n i c a l community was c a r r i e d o v e r i n t o
c o u l d a f f e c t a l l channels o f a redundant system, o u r i n a b i l i t y t o a c c u r a t e l y comnunicate t h e
i s i n t h e area o f t h e communication o f t h e progress o f o u r success o f t h e development
functional requirements. The functional process t o management f o r t h e i r review. Lacking
requirements i n t h e p a s t have a p p a r e n t l y n o t accurate descriptions of progress, upper
been w e l l understood a t t h e o u t s e t o f a program; management q u i t e o f t e n was a t a l o s s t o
t h i s i n c l u d e s t h e s p e c i f i c a t i o n o f t h e most understand exactly how to participate
g e n e r a l i n t e n d e d crew i n t e r f a c e f u n c t i o n a1 1 t h e e f f e c t i v e l y i n g e t t i n g t h e resources r e q u i r e d t o
way down t o t h e b e s t way t o achieve a s p e c i f i c s o l v e p a r t i c u l a r o r apparent problems i n t h e
f u n c t i o n o f t h e f l i g h t c o n t r o l system. Poor development o f a v i o n i c s equipment.
understanding o f t h e f u n c t i o n a l requirements
f u r t h e r complicated o u r a b i l i t y t o s p e c i f y This communication barrier between
requirements t o s u p p l i e r s , t o upper management, engineeringhanagement and management/upper
t o t h e m i l i t a r y and a i r l i n e customer f o r t h e i r management a f f e c t e d t h e a b i l i t y t o communicate
evaluation. In short, our inability to t o t h e customer exact1 y what capabi 1i t y t h e i r
a c c u r a t e 1y communicate the requirements systems would and would n o t p r o v i d e . Customers
produced a s i t u a t i o n which f a c i l i t a t e d f a u l t s were o f t e n faced w i t h making d e c i s i o n s about t h e
t h a t would a f f e c t a l l channels o f redundant a p p r o p r i a t e f u n c t i o n o f a new c a p a b i l i t y w i t h o u t
system. being able t o evaluate t h e f u l l y integrated
system i n a s i m u l a t i o n , b u t r a t h e r t o see o n l y
Communication o f t h e s p e c i f i c a t i o n f o r t h e p a r t o f t h e system, t h a t was implemented i n
f l i g h t c o n t r o l system i t s e l f was complicated by phases, and t o hear about t h e upcoming o r new
t h e wide v a r i e t y o f methods o f p r e s e n t i n g t h e capability.
d a t a and t h e language used. A highly linear
f o r m o f t e x t used t o d e s c r i b e v e r y c o m p l i c a t e d Probably t h e most complicated area was t o be
f u n c t i o n and l o g i c was o f t e n mixed w i t h (and n o t a b l e t o communicate t o t h e c e r t i f y i n g agencies
consistent with) h i g h l y n o n l i n e a r forms o f e x a c t l y what t h e development process was, and i f
communication o f v e r y d e t a i l e d c o n t r o l law, mode a t any g i v e n t i m e t h e r e was a need t o
l o g i c , and mode t r a n s i t i o n diagrams. As a demonstrate consistency between the
r e s u l t i t was q u i t e easy t o m i s i n t e r p r e t t h e requirements, t h e s p e c i f i c a t i o n , t h e design, t h e
s p e c i f i c a t i o n , and i n f a c t , t h e s p e c i f i c a t i o n implementation, and the test method and
was n o t a unique one-to-one mapping o f t h e procedures. I t was q u i t e d i f f i c u l t t o show t h a t
requirement i n t o a r e q u e s t f o r a s p e c i f i c t h e r e was a one-to-one correspondence between
intended function. t h e requirements and t h e r e a l system. I n other
words, i t was d i f f i c u l t t o communicate t h a t t h e
To proceed f r o m t h e s p e c i f i c a t i o n t o t h e r e q u i r e d system was a c t u a l l y b e i n g designed and
communication o f t h e design. t h e d e s i g n e r s were tested. Furthermore, p a s t methods t o achieve
o f t e n plagued w i t h a problem t h a t their certification were based largely on the
understanding of the requirement was experience o f t h e d e s i g n e r s and t h e i r i n t u i t i v e
c o n s i d e r a b l y o u t o f phase w i t h new o r changing f e e l i n g s about t h e systems. F u t u r e system
requirements. Q u i t e o f t e n the designers o f t h e certification w i l l have t o r e l y more on
avionic systems would be i n t e r a c t i n g w i t h a n a l y t i c a l techniques t o demonstrate t h e i r h i g h
engineers f r o m t h e i r own companies who were r e l i a b i l i t y w i t h l e s s r e l i a n c e on t h e p a s t
trying to reinterpret or restate the experience o f system engineers, since the
requirements t h a t were i n i t i a t e d by t h e m a j o r i n t u i t i v e approach would r e q u i r e even a more
a i r f r a m e manufacturers. Consequently, a t any complete understanding o f t h e e n t i r e f l i g h t
one p o i n t i n time, t h e s p e c i f i c design o f t h e c o n t r o l system t h a n i n t h e p a s t .
a v i o n i c system lagged t h e most c u r r e n t system
requirement, and t h i s phase l a g has caused THE PAYOFF -- INCREASED SAFETY AND REDUCED COSTS
considerable problems i n c o m u n i c a t i ng t h e
c u r r e n t requirements t o t h e engineers who were The new s o f t w a r e and hardware development
r e s p o n s i b l e t o r e v i e w requirement changes and t o techniques t h a t a r e subsequently discussed a r e
p a r t i c i p a t e i n d e s i g n walk-throughs of the expected t o have a dramatic impact on r e d u c i n g
a v i o n i c systems. t h e costs associated w i t h e r r o r c o r r e c t i o n since
t h e goal i s t o e l i m i n a t e o r reduce t h e number o f
T h i s phase l a g a l s o c o n t r i b u t e d t o poor human e r r o r s t h a t a r e produced, i.e. t o minimize
comnunication of test methods and test the number of original faults that are
procedures t h a t were i n t e n d e d t o achieve "100 i n t r o d u c e d i n t o t h e system designs. There i s
p e r c e n t " t e s t coverage. Since t h e requirements a l s o a p o t e n t i a l f o r reduced c o s t s r e s u l t i n g
were somewhat dynamic, t h e s p e c i f i c a t i o n s were from revised organizational boundaries and
f a i r l y complicated and m i s l e a d i n g . It was p a r t i t i o n i n g o f work e f f o r t s . There w i l l be
obviously quite difficult to design test l e s s d u p l i c a t i o n o f processes. There w i l l be
procedures and t e s t methods which could, i n i n c r e a s e d automation which w i 11 c r e a t e code from
f a c t , m a i n t a i n anywhere n e a r 100 p e r c e n t t e s t a s i n g l e s p e c i f i c a t i o n , and t h e r e w i l l be t h e
coverage. Furthermore i t was o f t e n d i f f i c u l t t o automatic generation of test methods and
c r e a t e new t e s t procedures t h a t e f f e c t i v e l y procedures t h a t map one-to-one back t o t h e
tested the added requirements and added requirements and t h e s p e c i f i c a t i o n documentr.
functions during this dynamic process of The most i m p o r t a n t c o s t aspect i s t o p r o v i d e l o
updating requirements, specifications, and the airline customers extended vehic l c
designs. performance t h a t w i l l l e t them r e a l i z e i n c r e a \ c d
p r o f i t s w h i l e r e d u c i n g i n i t i a l system c o 5 t \ ,
Obviously t h e i n a b i l i t y t o a c c u r a t e l y s p e c i f y operating costs, maintenance costs whI 1 0
requirements, d e s i g n and t h e t e s t methods w i t h i n
 maintaining o r improving current levels of fly-by-wire and/or f ly-by-light. And the
a v a i l a b i l i t y and s a f e t y . problem i s t h a t a s u s t a i n e d l o s s o f f u n c t i o n o r
a m a l f u n c t i o n can be a c a t a s t r o p h i c event. No
P
The p o t e n t i a l f o r f a u l t s t o e x i s t t h a t can a i r p l a n e manufacturer can accept f a u l t s o r
a f f e c t t h e f i n a l system performance ( p o t e n t i a l l y combination o f f a u l t s t h a t expose t h e a i r p l a n e
i n each channel o f a redundant system) o c c u r i n t o t h i s situation during the f l e e t lifetime.
t h e requirements area, t h e s p e c i f i c a t i o n , t h e Therefore, t h e goal must be t o develop systems
design implementation, t e s t and maintenance t h a t have s u f f i c i e n t l y h i g h r e l i a b i l i t y t h a t
aspect o f t h e a v i o n i c s system. Managers o f a l l operators never experience a failure,
aspects of the development process must p a r t i c u l a r l y a catastrophic f a i l u r e , during the
understand t h e f a c t t h a t i n o r d e r t o achieve l i f e t i m e o f the f l e e t . We a r e i n v o l v e d i n t h e
h i g h l y r e l i a b l e systems t h e y must be c o m n i t t e d f i r s t a t t e m p t i n t h e U.S. t o d e s i g n f u l l - t i m e
t o making t h e key resources a v a i l a b l e t o t h e commercial f l i g h t c r i t i c a l c o n t r o l systems. In
development process a t an e a r l i e r s t a g e t h a n has t h e p a s t we have developed systems which had
been done i n t h e p a s t . A t t h e same time, t h e r e flight critical phases w i t h very limited
must be techniques i n p l a c e t o measure and t o exposure times. F o r example d u r i n g t a k e o f f and
m o n i t o r t h e success t h a t i s b e i n g r e a l i z e d i n autoland, t h e t a r g e t exposure t i m e s were 20 t o
t h e development process. What t h i s means i s 30 seconds p e r f l i g h t .
t h a t i t w i l l n o t be p o s s i b l e t o s i m p l y add
manpower i n a r a t h e r w a s t e f u l c r a s h e f f o r t t o As shown i n F i g u r e 3, t h e p o t e n t i a l f o r
"save t h e design," because t h e poor q u a l i t y f a u l t s occurs i n b o t h hardware and s o f t w a r e
would have a l r e a d y been designed i n t o t h e system elements o f t h e system. They can o c c u r as t h e
and i t w i l l be t o o l a t e t o salvage t h e design. r e s u l t o f requirements e r r o r s , specification
By measuring and m o n i t o r i n g , managers can add errors, systems design e r r o r s , implementaton
people t o c o r r e c t a problem. The key i s t h a t e r r o r s , and e r r o r s t h a t r e s u l t f r o m o p e r a t i o n o r
t h e y w i l l now know where t o add resources t o maintenance and support o f t h a t equipment. And,
c o r r e c t t h e r e a l problem. o f course, what we a r e r e a l l y t a l k i n g about a r e
t h o s e f a u l t s which have been c a l l e d " g e n e r i c "
APPROACH TO FAULT TOLERANT DESIGN f a u l t s , b u t which a r e r e a l l y s p e c i f i c f a u l t s
t h a t c o u l d a f f e c t a l l channels o f a redundant
What we a r e t a l k i n g about here i s t h e d e s i g n system s i m u l t a n e o u s l y o r n e a r l y s i m u l t a n e o u s l y .
o f f u l l t i m e f l i g h t c r i t i c a l systems u t i l i z i n g

D E F I N I T I O N OF MU - D E S I G N WEAKNESS, IMPERFECTION,
M I S T A K E , THE RESULT OF ERROR

FAULTS

t
RANDOM FAULTS
t
(HARDWARE F A I L U R E S )
(COULD EFFECT ALL
CHANNELS SIMULTANEOUSLY)

COMMON MOD; FAULTS COMMON MODE FAULTS

DUE TO HUMAN ERROR DUE TO NATURAL CAUSES

I 4 t
I
A t
SYSTEM REQUIREMENT D E S I G N ERRORS OPERATIONAL/SUPPORT
ERRORS
/'Z
HARDWARE SOFTWARE /'"ROR\
HARDWARE SOFTWARE

S P E C I F I C A T I O N ERRORS IMPLEMENTATION ERRORS

HARDWARE SOFTWARE
/-=' \

Figure 3 Sources o f Faults

 The new d i r e c t i o n t h a t we must t a k e I s shown Hardware and software s p e c i f i c a t i o n e r r o r s
i n F i g u r e 4 as an expanded approach designed t o
minimize f a u l t s e a r l y i n t h e design phase and/or
t o eliminate/minimize/tolerate t h e i r impact.
w i l l be c o n t r o l l e d o r minimized through t h e use
of structured methods and program design
languages which p r o v i d e a p r e c i s e language w i t h
-
The goal i s t o minimize o r e l i m i n a t e systems which t o communicate t h e s p e c i f i c a t i o n . Use of
requirement e r r o r s u s i n g extensive s i m u l a t i o n s s t r u c t u r e d methods and program d e f i n i t e language
to validate requirements early in the w i l l a l s o a l l o w completeness o f design and l o g i c
development process. T h i s w i l l a l s o i n c l u d e t h e states.
use o f l a b o r a t o r y t e s t i n g , f l i g h t t e s t s and
prototypes to assure that the system The s t r u c t u r e d methods w i l l a l s o be used t o
requirements are complete prior to any minimize hardware and software design e r r o r s .
commitment t o f i n a l system design. It i s a l s o Automated design t o o l s and program design
planned t o use t e s t teams which a r e independent languages wi 11 a l s o c o n t r i b u t e t o reduced e r r o r s
from those who have developed t h e requirements d u r i n g t h e design phase. Another technique t h a t
and s p e c i f i e d t h e system. w i l l be used t o minimize t h e design e r r o r s i s t o
i s o l a t e t h e c r i t i c a l f u n c t i o n s from t h e l e s s

Figure 4 Expanded Approaches t o Minimize F a u l t s

and/or The i r Impact

FAULTS

RANDOM FAULTS COMMON MODE COMMON MODE FAULTS

REDUNDANCY/RECONFIGURATION
(MANMADE) COMPREHENSIVE SYSTEM DR6O

L S I N L S I FOR R E L I A B I L I T Y
.f EXPANDED SYSTEM LEVEL TESTING:
R I G 6 AIRPLANE

SYSTEM REQUIREMENT DESIGN OPERA?IONAL/SUPPORT

ERRORS ERRORS ERRORS
EXTENSIVE SIMULATION
TO VALIDATE REQT'S
HARDWARE
f 2
SOFTWARE HARDWARE
f
SOFTWARE
LAB 6 FLIGHT TEST *STRUCTURED METHODS EXPANDED B I T E .ISOLATED
CRITICAL
- DEMONSTRATOR .ISOLATED CRITICAL FUNCTIONS FUNCTIONS

0 AUTOMATED PRDGRM DESIGN .NO AIRLINE

DESIGN LANGUAGE MODIFICATIONS
INDEPENDENT TEST TOOLS TO CRITICAL
TEAMS CODE
SPECiF ;ATION IMPLEMENTATION
-
ERF ,RS f ERRORST

. (HARDWAREISOFTWARE)
*STRUCTURED METHODS
HARDWARE SOFTWARE
DISSIMILARITY

OPROGRM DESIGN LANGUAGE 0 REDUNDANCY/RECONFIGURATION

0 EXPANDED SYSTEM LEVEL

VALIDATION TESTING

.AUTOHATED VERIFICATION
TESTING

ADA

*PROVEN TRANSPORTABLE CODE

c r i t i c a l f u n c t i o n s and t o c o n c e n t r a t e on t h e Management should be encouraged not t o
d e s i g n o f those c r i t i c a l f u n c t i o n s as a separate e s t i m a t e those f a c t o r s t h a t can a c t u a l l y be
entity. c a l c u l a t e d o r computed, e.g., p r e d i c t o r s can
sometimes be computed f r o m a c t u a l p h y s i c a l d a t a
Hardware and s o f t w a r e implementation e r r o r s r a t h e r t h a n e s t i m a t e d f r o m t h e h i p pocket.
w i l l be minimized by t h e use o f d i s s i m i l a r Hanagement must understand how t o use t h e
redundancy or by using system component structured methods documents to obtain
reconfiguration when appropriate. Expanded p r e d i c t o r s o f how b i g t h e system w i l l be, how
system l e v e l v a l i d a t i o n t e s t i n g w i l l a l s o be complex t h e system w i l l be and how l o n g i t w i l l
used t o c o n t r o l implementation e r r o r s . This t a k e t o design, e.g., a count o f t h e p r i m i t i v e s
w i l l i n c l u d e automated v e r i f i c a t i o n t e s t i n g , t h e f r o m t h e d a t a f l o w diagrams and a count o f t h e
use o f Ada as a common s o f t w a r e design language "big items" that are involved in being
t h a t leads t o a v i s i b l e and h i g h l y t e s t a b l e t r a n s p o r t e d f r o m one f u n c t i o n t o a n o t h e r i n t h e
implementation, and t o t h e use o f h i g h l y proven d a t a f l o w diagram may be v a l u a b l e i n d i c a t o r s .
p r i m i t i v e code elements. Management must understand t h a t t h e e s t i m a t i n g
process i s a dynamic process and so n o t o n l y i s
An expanded b u i l t - i n test f u n c t i o n and an e s t i m a t e i m p o r t a n t and v a l u a b l e , b u t t h e t i m e
capability w i l l contribute t o the control of a t which t h e e s t i m a t e was made i s o f c r i t i c a l
e r r o r s d u r i n g o p e r a t i o n o f equipment and d u r i n g importance. F o r example, e s t i m a t e s made d u r i n g
i t s support l i f e . The f a c t t h a t t h e c r i t i c a l t h e i n i t i a l d e s i g n phase a r e t o t a l l y d i f f e r e n t
functions are isolated from noncritical than those made halfway through the
f u n c t i o n s may a l l o w customer m o d i f i c a t i o n s t h a t design-complete phase.
do n o t a f f e c t t h e c r i t i c a l hardware o r s o f t w a r e
elements provided that initial and Managers should be aware o f and understand
r e c e r t i f i c a t i o n c o s t s a r e reasonable. s o f t w a r e schedule development l i m i t s based on
estimates of "time-to-del i v e r y versus
Random f a u l t effects w i l l be minimized m a n - m ~ n t h s ' ~as discussed i n Reference 2. There
t h r o u g h use o f redundancy and r e c o n f i g u r a t i o n , e x i s t s some boundary t h a t i s based on t h e i r
and by u s i n g l a r g e s c a l e i n t e g r a t e d ( L S I ) company's way o f d o i n g business ( e m p i r i c a l l y t h e
c i r c u i t s and VLSI technology, f o r improved nominal t i m e - t o - d e l i v e r y i s equal t o 2.5 t i m e s
reliability. The approach t o minimize f a u l t s t h e cube r o o t o f t h e work e f f o r t i n man-months
due t o common modes i s t o develop comprehensive and t h e s t a t i s t i c a l l i m i t , i s about 75 p e r c e n t
systems design requirements and o b j e c t i v e s of the nominal time) that defines a
(ORLO) and t o expand t h e system l e v e l t e s t i n g , " s t a t i s t i c a l l y i m p o s s i b l e r e g i o n " where i t i s
i.e. i r o n b i r d s . "copper" b i r d s e t c . and a l s o i m p o s s i b l e t o reduce d e l i v e r y t i m e s i m p l y by
expanded a i r p l a n e f l i g h t t e s t i n g . p o u r i n g more manpower o n t o t h e problem.

Each o f t h e f o r e g o i n g elements add t o Managers must a l s o e x p l o r e how t o i d e n t i f y

i n c r e a s e d u p f r o n t c o s t s which must be o f f s e t and r e c o g n i z e t h e n e g a t i v e producers on t h e i r
against the extremely high costs o f c o r r e c t i n g team. They should understand t h a t t h e c o s t s o f
e r r o r s l a t e i n t h e program and p a r t i c u l a r l y when removing d e f e c t s f r o m t h e system i s f a r g r e a t e r
t h e equipment i s i n s e r v i c e . than the cost o f avoiding defects i n the f i r s t
place. Further, i n o r d e r t o end up w i t h
CONCLUSIONS h i g h - q u a l i t y t e s t e d code one must c r e a t e h i g h
q u a l i t y code i n t h e b e g i n n i n g and n o t count on
The s o l u t i o n t o achieve h i g h l y r e l i a b l e , " t e s t i n g o u t t h e defects.'
safe, f u l l - t i m e c r i t i c a l c o n t r o l system designs
that include f ly-by wire or f ly-by-light Hanagement should do a l l t h e y can t o
subsystems, i s t h r o u g h e a r l y and thorough d e s i g n discourage t h e i d e a t h a t " s o f t w a r e bugs and
d e f i n i t i o n and e v a l u a t i o n , t o u t i l i z e improved system bugs a r e a f a c t o f l i f e " o r t h a t " i t ' s
development processes which w i 11 reduce t h e t o t a l l y human t o make mistakes.'
l i k e l i h o o d o f a f a u l t being introduced i n t o t h e
design, and t o i n c r e a s e t h e l i k e l i h o o d o f a Engineers. scientists and software
f a u l t b e i n g i d e n t i f i e d and/or t o l e r a t e d by u s i n g s p e c i a l i s t s charged w i t h d o i n g t h e s p e c i f i c s
redundant system a r c h i t e c t u r a l condepts and should e l i m i n a t e t h e t e r m g e n e r i c from t h e i r
reconfiguration. vocabulary. I f t h e best a d j e c t i v e from t h e i r
perspective i s "generic," t h e n perhaps t h e y
Management needs t o have i n p l a c e a way t o should n o t be t h e one communicating. A f t e r a l l ,
measure t o t a l system q u a l i t y e a r l y on i n t h e t h e r e i s one o r more e x p e r t s o u t t h e r e who view
design phase because i t e l s c l e a r t h a t i t i s each r e p o r t e d g e n e r i c e r r o r as a v e r y s p e c i f i c
i m p o s s i b l e t o c o n t r o l what cannot be measured. and r e s o l v a b l e e r r o r .
Management must understand t h e c r i t e r i a f o r
b e i n g complete a t each o f t h e d e s i g n phases, as REFERENCES
a v i t a l p a r t o f the metric t h a t i s required t o
assess whether we a r e a c h i e v i n g o u r systems 1) Philip B. Gove, Websters Third New
design goals. Hanagement a l s o has t o understand International Dictionary o f the English
t h a t l a r g e samples o f meaningless d a t a a r e s t i l l Lanquaqe, 1976. Merriam-Webster P u b l i s h i n g
meaningless. What t h i s means i s t h a t t h e d a t a Company, S p r i n g f i e l d , Massachusetts.
used t o examine t h e system progress has t o come
f r o m s e v e r a l sources i n o r d e r t o keep c e r t a i n 2) Barry W. Boehm, Software Enqineering
people from t e l l i n g a s t o r y t h a t l o o k s o v e r l y Economics, 1981. Prentice Hal 1, Inc.
optimistic. Englewood C l i f f s , New Jersey.