FVSysID ShortCourse 6 Validation

Validation of Flight Vehicle System Identification
Maneuver
Input
Optimized
Input
Actual
Response
FlightVehicle
Measurements
DataCollection
&Compatibility
Methods
APrioriValues,
lower/upper
bounds
Estimation
Algorithm/
Optimization
Models
Mathematical
Model/
Simulation
Model
Structure
Identification
Criteria
Response
Error
Parameter
Adjustments
ModelResponse
Identification Phase
Complementary
Flight Data
Dr. Ravindra Jategaonkar
Model
Validation
Validation Phase
AIAA Short Course: Flight Vehicle System Identification in Time Domain, Aug.2006
Validation/1
This page is left intentionally blank.
Validation/2
Data Base Validation (1)

System identification:
Given the system inputs and responses, what is the model?
Model Validation:
How do you know that you got the right answer?
Definition:
Validation refers to the process of confirming that the conceptual
model is applicable or useful by demonstrating an adequate
correspondence between the computational results of the model and
the actual data (if it exists) or other theoretical data.
Broad classification:
1) Statistical properties of the estimates,
2) Residual analysis, and
3) Model predictive quality
Validation/3
Data Base Validation (2)

Several criteria
1) Standard derivations
2) Correlation among the estimates
3) Goodness of fit
4) Plausibility of estimates (WT data base)
5) Statistical analysis of residuals (bias, variance, covariance, and PSD)
6) Model deficiencies in terms of residual control inputs (inverse simulation)
7) Model predictive capability
"ACID TEST
Simulation and comparison with flight data not used in identification
- Criteria to be used in conjunction with each other

- Basic philosophy remains same for simple as well as for global models
Validation/4
Statistical Accuracy of Parameter Estimates

Measure of accuracy
Clues into the effectiveness, or lack thereof, of model parameters
Fischer information matrix provides a good approximation to the
parameter error covariance matrix P:
N y (t )T 1 y (t )
k
k
P
R

k =1
Standard deviations (Cramer-Rao bounds):

i =
pii
Correlation coefficients:
i j =
pij
pi p j
Validation/5
Statistical Accuracy: Practical difficulties

Standard deviations:
- Estimated error bounds are too optimistic
-> use fudge factor of 5-10
(ad hoc approach to bridge the gap between theory and practice).
-> Scatter in the estimates larger than standard deviations

(non trivial issue; scatter can not be avoided)
- Variances of the estimated parameters and the covariances

(off diagonal elements of the error matrix) must be small.
- Correlation coefficients:
-> Linear dependency (> 0.9; >0.95; >0.98..)
Validation/6
Residual Analysis: Goodness of fit

Cost function value:
- ML estimation: determinant of R (covariance matrix of residuals)
- Most direct way to evaluate model quality
- Cost function value should be small:
Value depends on the number of system outputs, noise level, and units
of the variables (radians or degrees; g or m/s2; m/s or kts)
- Absolute value alone of limited use

- No consistent workable criteria has been put forward
- Old adage: Goodness of Fit is no criteria.
It is a necessary but not a sufficient condition.
Peculiarity of cost function:
Very low value does not guarantee equally good matching of all variables;
In worst case a perfect match of one variable yields minimum.
Validation/7
Residual Analysis: Overall Fit and

Decomposition of Fit Error
Theils inequality coefficient
TIC =
2
z( t ) y( t )
1 N
k
N k=1 k
2
z ( t )
1 N
+
N k=1 k
2
y( t )
1 N
N k=1 k
Ratio of the root mean square fit error and the root mean square
values of the measured and estimated signals summed together
TIC is normalized: between 0 and 1
0 --> perfect fit
1 --> worst case ==> two time series are very significantly different
Thumb rule: Acceptable TIC value depends on the application, in
general 0. 25 to 0.3 indicates a good agreement
Validation/8
Residual Analysis: Decomposition of Fit Error

Bias proportion:
U iM
( z i yi ) 2
(1 / N )
measure of systematic
[ zi (t k ) yi (t k )]2
error in the identified model
( zi y ) 2
i
represents the models ability
[ zi (t k ) yi (t k )]2
to duplicate the variability in

the true system
k =1
Variance proportion:
U iS
(1 / N )
k =1
2 (1 i ) z i yi
Covariance proportion: U iC =
(1 / N )
[ zi (t k ) yi (t k )]2
k =1
measure of non-systematic
error
U iM + U iC + U iS = 1
zi and y i denote the mean vales;
and the standard deviations and correlation coefficients.
Bias and variance proportions should be very small (typically less than 0.1),
in an ideal case, zero;
Covariance proportion should be close to one.
Validation/9
Inverse Simulation
Simulation:
given inputs u and system model f, find system output y
Inverse
simulation
Process of calculating desired controls, for the given

system model and response.
Input
Aircraft
u
u
Feedback
controller
Mathematical
model
Measured Response
z
+
_
y
Computed Response
Closely similar to the control problem (namely, given system model f and
output y, find control input u).
Subtle difference:
Classical control problem does not require measured system responses.
Inverse simulation explicitly needs measured responses, and leads to
controls based on residuals.
Validation/10
Model Plausibility (1)

Example: Control surface effectiveness - ATTAS VFW-614
Elevator
-1.0
-.16
/rad
/rad
-1.2
-.20
Cm
Aileron
Rudder
-.14
/rad
Cn
-.16
Cl
-.24
-1.4
-.18
-1.6
-.28
0
10
20
Angle of attack, deg
0
10
20
Angle of attack, deg
-10
0
10
Angle of sideslip, deg
Wind tunnel / analytical prediction

Estimates from flight data
Validation/11

Damping Derivatives - ATTAS VFW-614
Short period motion
Rolling motion
Dutch roll motion
-4.0
-.80
-.10
/rad
/rad
/rad
Cm.
Cn.
-.90
-6.0
Clp
-.40
-1.0
-8.0
Cmq
-10.0
Cnr
0
0.2
0.4
0.6
-1.1
Mach number
0.2
0.4
Mach number
0.6
-.70
0
0.2
0.4
0.6
Mach number
Wind tunnel / analytical prediction

Estimates from flight data
Validation/12
Eigenvalues and eigenvectors completely characterize

the system behavior
Eigenvalue = ( -1.61196 , + - 0 )
Linearized system matrix
0.3
0.2
WK
THE
UK
VK
PSI
PHI
Eigenvalue = ( -0.965076 , + - 1.75372 )
0.1
Dutch roll
UK
WK
THE
VK
Q
PSI
PHI
Im [rad/s]
Eigenvalue = ( -0.142527 , + - 1.54814 )
Short period
1
UK
-0.2
WK
-0.4
THE
Re [1/s]
-0.6
-0.8
PSI
VK
-1
PHI
-1.2
-1.4
-1.6
Validation/13
Model Predictive Capability (1)

Proof-of-Match (POM):
An important part of flight simulator certification and acceptance.
Compare the flight measured system responses with those predicted
by the model for the same (identical) control inputs.
In this POM process, the identified (aerodynamic) model is kept fixed.
Other important issues:
1) Proper choice of the data set to compare against,
2) Initial conditions on the state variables (trim), and
3) Criteria to check adequate correspondence between
model response and measured aircraft outputs.
Validation/14

Proof-of-Match (POM):
Data set: Complementary flight data, i.e., flight maneuvers not used
in the estimation the of aerodynamic (Acid test)
FAA has defined a set of roughly a little more than 100
different cases, covering different modes of aircraft motion,
and configurations
Initial conditions: Ideally, the simulation is to be started from the
same initial conditions as in the flight
Criterion: FAA tolerances (to avoid subjective evaluation)
Non trivial task: Measurements corrupted by noise
Presence of turbulence
Modeling errors
Allow small biases on the selected initial conditions and on the
measured control deflections
Validation/15

Stand-alone and integrated models
Measured
Aircraft
Motion
Variables
Reversible
Flight Control
Measured
Dynamics
Control
Surface
Deflection
Measured
Control
Surface
Deflect.
Rigid Body
Dynamics
Aircraft
Motion
Variables
Pilot Input
Forces
Flight controls stand-alone
Pilot
Input
Forces
Reversible
Flight Control
Dynamics
Rigid-body stand-alone
Control Surface
Deflection
Rigid Body
Dynamics
Aircraft
Motion
Variables
Integrated model
Validation/16

Validation test 2c11: Short period dynamics
10
deg
Time domain verification:

Elevator
deflection
Majority of VT in time domain

e.g., 1.5/s for rates,
0.1 g for accelerations
0
-5
15
deg/s
Pitch rate
Few are in terms of damping

ratio and frequency; e.g.,
0.02 damping ration,
10% period
-10
0.5
g
Recent effort:
Frequency domain criteria
Vertical
acceleration
-2.5
0
2.5
time
7.5
sec
10
Validation/17

Frequency domain criteria
Bode plots of measured to
model estimated response
e.g., qm/q (error function)
Validation test 2c11: Short period dynamics

10
dB
Magnitude 0
Ideally, for a perfect match:

0 dB magnitude and
0 deg phase angle
over the frequency range
Boundaries are based on the
LOES (Low Order Equivalent
System) mismatch criteria:
unnoticeable dynamics
Brings our more clearly the
range of model applicability
Important for high authority
FCS where aeroservoelastic
effect may be dominant
-5
-10
40
deg
Phase
-40
-80
0.3
10
Frequency
rad/s 20
More restrictive tolerance band in the range of 1 to 5 rad/s

pilot cross-over frequency, allows more error outside of it
(LOES phase-lead leniency), and are asymmetric at low
frequencies (allow more phase lead error than phase lag
error) to have better fidelity with respect to lag error which
might cause pilot-induced oscillations.
Validation/18
From Identified model:
P / AILERON and P / AILERON

dB
Linearized system matrices A, B, C, D
5
0
-10
Frequency response matrix
Magnitude
G( j ) = C ( jI A )1 B + D
-20
-30
-180
deg
From measured flight responses:

Approximation of frequency responses
through FFT techniques
-240
Phase
-300
-360
Noisy measured data: smoothing
Frequency
10
rad/s
Validation/19
References
Jategaonkar, R. V.,
Flight Vehicle System Identification: A Time Domain Methodology,
Volume 216, AIAA Progress in Astronautics and Aeronautics Series
Published by AIAA Reston, VA, Aug. 2006, ISBN: 1-56347-836-6
http://www.aiaa.org/content.cfm?pageid=360&id=1447
Hamel, P. G. and Jategaonkar, R. V., Evolution of Flight Vehicle System Identification, Journal of
Aircraft, Vol. 33, No. 1, Jan.-Feb. 1996, pp. 9-28.
Hodgkinson, J. and Mitchell, D., Handling Qualities, in Flight Control Systems, Pratt, R. W. (Ed.),
AIAA Progress in Astronautics and Aeronautics Series, Vol. 184, 2000, Chapter 4.
Jategaonkar, R.V., Determination of Aerodynamic Characteristics from ATTAS Flight Data Gathering for
Ground-Based Simulator, DLR-FB 91-15, 1991.
Murray-Smith, D. J., Methods for the External Validation of Continuous System Simulation Models:
A Review, Journal of Mathematical and Computer Modelling of Dynamical Systems, Vol. 4, No. 1,
1998, pp. 5-31.
N.N., Airplane Simulator Qualification, FAA Advisory Circular, AC 120-40C,Interim Version, Jan. 1995.
N.N., Joint Aviation Requirements - Aeroplane Flight Simulators, JAR-STD 1A, Westward Digital Ltd.,
Cheltenham, England, April 1997.
Tischler, M. B., System Identification Methods for Aircraft Flight Control Development and Validation,
NASA TM 110369, Oct. 1995.
see also the References from the Section Examples
Validation/20

FVSysID ShortCourse 6 Validation

Загружено:

Сведения о документе

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

FVSysID ShortCourse 6 Validation

Загружено:

Авторское право:

Доступные форматы

Validation of Flight Vehicle System Identification

Dr. Ravindra Jategaonkar

This page is left intentionally blank.

Dr. Ravindra Jategaonkar

Data Base Validation (1)

Dr. Ravindra Jategaonkar

Data Base Validation (2)

- Criteria to be used in conjunction with each other

Statistical Accuracy of Parameter Estimates

Standard deviations (Cramer-Rao bounds):

Dr. Ravindra Jategaonkar

Statistical Accuracy: Practical difficulties

-> Scatter in the estimates larger than standard deviations

- Variances of the estimated parameters and the covariances

Dr. Ravindra Jategaonkar

Residual Analysis: Goodness of fit

- Absolute value alone of limited use

Residual Analysis: Overall Fit and

Residual Analysis: Decomposition of Fit Error

error in the identified model

represents the models ability

to duplicate the variability in

zi and y i denote the mean vales;

and the standard deviations and correlation coefficients.

given inputs u and system model f, find system output y

Process of calculating desired controls, for the given

Model Plausibility (1)

Wind tunnel / analytical prediction

Model Plausibility (2)

Dutch roll motion

Wind tunnel / analytical prediction

Model Plausibility (3)

Eigenvalues and eigenvectors completely characterize

Linearized system matrix

Eigenvalue = ( -0.965076 , + - 1.75372 )

Eigenvalue = ( -0.142527 , + - 1.54814 )

Dr. Ravindra Jategaonkar

Model Predictive Capability (1)

Dr. Ravindra Jategaonkar

Model Predictive Capability (2)

Model Predictive Capability (3)

Flight controls stand-alone

Model Predictive Capability (4)

Time domain verification:

Majority of VT in time domain

Few are in terms of damping

Dr. Ravindra Jategaonkar

Model Predictive Capability (5)

Validation test 2c11: Short period dynamics

Ideally, for a perfect match:

More restrictive tolerance band in the range of 1 to 5 rad/s

Model Predictive Capability (6)

From Identified model:

P / AILERON and P / AILERON

Linearized system matrices A, B, C, D

Frequency response matrix

From measured flight responses:

Noisy measured data: smoothing

Dr. Ravindra Jategaonkar

Вам также может понравиться