Nature of Model Validation

 “A model cannot be expected to have

absolute validity.”(Forrester, 1968)
Model Evaluation & Validation  A model is developed for a purpose.
 A model should be valid for its purpose but may
be irrelevant or wrong for some other purposes.
 “No positive proof is conceptually possible for
any theory or model.”(Forrester, 1968)
 No absolute proof but only a degree of hope and
Wei-Tsong Wang confidence on the validity of a model (validity is a
“relative”concept).
IIM, NCKU  No universal standards for comparing the validity
of models constructed for different purposes.
1 3

Overview Nature of Model Validation (Cont.)

 Nature of model validation
 What is validation
 Two aspects of model validation
 Historical review of the discussions on System
Dynamics model validation
 Role of statistical tests in model validation
Some statistical tests for model validation

2 4
 “Model validation is inherently a social,
judgmental, qualitative process: models
cannot be proved valid but can be
judged so.”(Barlas & Carpenter, 1990)
 A model is a imperfect representation of
the reality and is valid if it serves as a
useful tool for decision making. (Barlas
& Carpenter, 1990)
 Nature of Model Validation (Cont.) What is Validation (Cont.)
 Some statistical tests can be applied to  An operating definition of validation:
SD model validation, but some of them ”an on-going mix of activities embedded
are inappropriate (Barlas, 1989; throughout the iterative model-building
process.”(Lane, 1998)
Forrester & Senge, 1980)
 Features of model validation (Barlas, 1989;
Barlas, 1996; Forrester & Senge, 1980; Lane,
1998)
 Focus on “
structure”
 Focus on “
behavior”

5 7

The system dynamics modeling

What is Validation process
Empirical and
 Determine “ whether the conceptual simulation Perceptions of Inferred Time
System Structure
model is ...... an accurate representation of the Series
System
system under study.”(Kleijnen, 1995)
Conceptualization
 Validation is a process of establishing Comparison and Comparison and
confidence in the soundness and usefulness of a Reconcilation Reconciliation.
Model
model. (Forrester 1973; Forrester & Senge 1980)
Formulation
 Evaluate the “ usefulness”of the model in a Representation of Deduction Of
Model Structure Model Behavior
specific social context, or for a particular
purpose. (Barlas & Carpenter, 1990; Forrester,
1968; Lane, 1998)
Adapted from G. P. Richardson’
s presentation on Model Validation
6 Source: Saeed, 1992 8
 Processes focusing on system Two Kinds of Validating
structure Processes
Mental Models, Mental Models, Literature,
Experience, Experience, Experience
Empirical Empirical
Literature Literature
Evidence Evidence Empirical and
Perceptions of Perceptions of
System Structure Inferred Time
System Structure Series
System
System
Conceptualization
Comparison and Conceptualization
Comparison and Comparison and
Reconcilation Structure Behavior
Reconcilation Validating Validating Reconciliation.
Model
Processes Model Processes
Formulation
Representation of Formulation
Representation of Deduction Of
Model Structure
Model Structure Model Behavior
Diagramming and
Diagramming and Computing
Description Tools
Description Tools Aids
9 11

Processes focusing on system Forrester and Senge’s

behavior Propositions on Confidence
Literature,

Empirical
Experience  Forrester and Senge’
s (1980) tests for
Evidence Empirical and
Inferred Time
model validation
Series
System
 Tests of model structure
 Tests of model behavior
Conceptualization
Comparison and
 Tests of policy implication
Reconciliation.
Model
Formulation
Deduction Of
Model Behavior
Computing
Aids

10 12
 Tests of Model Structure
 Structure-verification test
 Compare the structure of a model with
structure of the real system
 Parameter-verification test
 Whether or not the parameters in a model
correspond conceptually and numerically to
the real system
Source: Forrester & Senge, 1980
Tests of Model Structure (Cont.)
 Extreme-conditions test
 A model should allow extreme conditions of
levels in the system to be represented
 Boundary-adequacy (structure) test
 Whether or not the structure satisfies a
model’
s purpose?
 Proper level of aggregation?
 Includes all relevant structure?
Source: Forrester & Senge, 1980
Tests of Model Structure (Cont.)
 Dimensional-consistency test
 No unit errors
 Other tests
13 Source: Forrester & Senge, 1980 15
Tests of Model Behavior
 Behavior-reproduction test
 “How well model-generated behavior matches
observed behavior of the real system?”
 A number of tests are available for this matter, such
as symptom-generation test and multi-mode test
 Behavior-prediction test
 Pattern prediction test: generating qualitatively
correct patterns of the future?
 Event prediction test: generating qualitatively correct
patterns in response to a particular change in
circumstances?
14 Source: Forrester & Senge, 1980 16
 Tests of Model Behavior (Cont.) Tests of Model Behavior (Cont.)
 Behavior-anomaly test  Behavior-sensitivity test
 Do anomalous features of model behavior that  Do plausible shifts in model parameters
conflict with the behavior of the real system exist?
cause a model to fail to generate observed
 Family-member test patterns of behavior or behave implausibly?
A model usually represents a family of social
Other tests


systems 

 With proper changes on parameters and tables,

the model should be able to represent a particular
member in the family

Source: Forrester & Senge, 1980 17 Source: Forrester & Senge, 1980 19

Tests of Model Behavior (Cont.) Tests of Policy Implication

 Surprise-behavior test  System-improvement test
Is the unexpected behavior generated by the
Does the model help us find policies which

model the same as that in the real system? 

 Extreme-policy test are beneficial?

 Does the model behave like the real system when
the same extreme policy is applied?
 Boundary-adequacy (behavior) test
Does the model include all the structures that are

necessary to address the issues to which the model
is designed for?
 Changed-behavior-prediction test
 Does the model accurately predict the
behavioral change of the system
corresponding to a change in the original
policy?

Source: Forrester & Senge, 1980 18 Source: Forrester & Senge, 1980 20
 Barlas’Framework on Model
Tests of Policy Implication (Cont.) Validation (Cont.)
 Boundary-adequacy (policy) test  Structure validity tests
 Does the addition of a new structure  Direct structure tests
dramatically change the policy  Directly compare the model structure with the structure
recommendations? of the real system
Structure-oriented behavior tests
Policy-sensitivity test



 To what degree policy recommendations

are influenced by uncertainty in parameter  Behavior validity tests
Assess how accurately a model can reproduce the
values? 

behavioral patterns exist in the real system

Source: Forrester & Senge, 1980 21 23

Barlas’Framework on Model Lane’s Practical Criteria for

Validation Confidence on SD Models

 Barlas (1996) proposes that confidence  Lane (1998) discusses about building
confidence on the generic structures of
on a model builds on its structure
simulation models
validity and behavior validity
 3 Generic structures
 However, he also proposes that  Canonical Situation Model (fully formulated
behavior validity is meaningful only if simulation model)
we have sufficient confidence in the  Abstracted Micro-structure (stock and flow
structure)
structure
 Counter-intuitive System Archetype (causal-loop
structure)
22 Source: Lane, 1998 24
 Criteria for Confidence on SD Summary of Model Validation
Models (Cont.) Tests
Focusing on Focusing on
 Features of the criteria for confidence STRUCTURE BEHAVIOR
 Each criterion include concerns on both
Testing SUITABILITY for
“structure”and “ behavior” PURPOSE
 Each criterion considers about

“representativeness”and “ usefulness” Testing CONSISTENCY

with REALITY
 Each criterion can be applied to each of the

3 generic structures Contributing to UTILITY

& EFFECTIVENESS
Summarized by G. P. Richardson
Source: Lane, 1998 25 Sources: Forrester, 1973; Forrester & Senge, 1980; Richardson & Pugh, 1981 27

Criteria for Confidence on SD Role of Statistical Tests in Model

Models (Cont.)
 Lane’
s (1998) 3 criteria for confidence
 Perceived representativeness of models (PRoM)
 Whether a model’ s structure, data, and behavior
represent the real system?
Analytical quality of policy insights (AQ)
 
 Whether a model produce policy insights and
recommendations?
 Process effectiveness of the intervention (PEI)
 Whether a model satisfies the target clients?
Source: Lane, 1998
Validation
 General problems on using statistical tests for SD
model validation (Barlas, 1996):
 Technical problem #1: Most statistical tests assume
that data:
 Serially independent (not autocorrelated)
Not cross-correlated

Normally distributed
However, data in an SD model is autocorrelated and cross-
correlated by nature
 Technical problem #2: In SD models no single
“output variable”one can focus on in validity testing
(multiple-hypothesis problem may occur when using
statistical hypothesis testing)
26 28
 Role of Statistical Tests in Model
Validation (Cont.)
 Philosophical problem:
 The cost of rejecting the true null hypothesis or
accepting the false null hypothesis is difficult to
determine.
What would be our tolerable level of significance?
 Alternative:
Report the obtained validity statistics, but no
Role of Statistical Tests in Model
Validation (Cont.)
 For model behavior:
“Kalman filter”might be useful (Peterson,
1979, quoted in Forrester & Senge, 1980).
 It aims to compare model behavior to data
instead of comparing model structure to data
like the traditional statistical tests do.
 Might be more appropriate for SD since it

binary significance testing (no “

reject”or “ fail extracts measurement errors when testing
to reject”decision). causal hypotheses in the SD models.

29 31

Role of Statistical Tests in Model Some Statistical tests for Model

Validation (Cont.) Validation
Barlas (1989) proposes a series of tests (in a
From the perspective of model structure


sequential order) for evaluating model
and behavior (Forrester & Senge, 1980) behavior
 For model structure:  Step 1: Trend comparison and removal
Traditional statistical tests are not sufficient  Step 2: Comparing the periods
for testing the causal hypotheses in a  Step 3: Comparing the means
system dynamics model, but might be  Step 4: Comparing the variations
useful for discovering flaws.  Step 5: Testing for phase lag
 Step 6: Discrepancy coefficient U as a summary
measure

30 32
 Some Statistical tests for Model
Validation (Cont.) Discussion
 Kleijnen (1995) proposes that some
simple tests are available for comparing
the output of a SD model with historical
What do you think about
output of the real system, such as:
Model validation now?
 Eyeball time paths
 Schruben-Tuning test

 Traditional statistical tests

33 35

What can we do about model

validation in general?
Validation, as an integrated social process, is
 system dynamics type of simulation models.”
present at every step
 Conceptualizing:
 Do we have the right people?
 The right dynamic problem definition?
 The right level of aggregation?
 Mapping: Developing promising dynamic hypotheses
 Formulating: Clarity, logic, and extremes
 Simulating: Right behavior for right reasons
 Deciding: Implementable conclusions
 Implementing: Requires conviction!
34
Sources: G. P. Richardson’
s presentation on “
Model Validation”
Reference
Barlas, Y. (1989). “Multiple tests for validation of

European Journal of Operational Research, 42(1),
59 –87
 Barlas, Y. (1996). “Formal aspects of model validity
and validation in system dynamics.”System
Dynamics Review, 12(3), 183 - 210
 Barlas, Y., & Carpenter, S. (1990). “ Philosophical
roots of model validation: two paradigms.” , System
Dynamics Review, 6(2), 148 - 166
 Forrest, J. W. (1968). “Industrial Dynamics –A
Response to Ansoff and Slevin.”Management Science,
14(9), 601 –618
36
 Reference (Cont.)
 Forrester, J. W. (1973). “ Confidence in Models of
Social Behavior –With Emphasis on System
Dynamics Models.”M.I.T. System Dynamics Group
 Forrester, J. W., & Senge, P. M. (1980). “ Tests for
building confidence in system dynamics models.”in A.
A. Legasto, JR, J. W. Forrester, & J. M. Lyneis (Ed.),
System Dynamics: TIMS Studies in the Management
Science, 14, New York: North-Holland, 209 –228
 Kleijnen, J. P. C. (1995). “Verification and validation
of simulation models.”European Journal of
Operational Research, 82(1), 145 –162

37

Reference (Cont.)
 Lane, D. C. (1998). “Can we have confidence in
generic structures?”The Journal of the Operational
Research Society, 49(9), 936 - 947
 Peterson, D. W. (1979). “ Statistical tools for system
dynamics” , in J. Randers (Ed.), Elements of the
System Dynamics Method, Cambridge, MA: MIT Press
 Richardson, G. P., & Pugh, A. L. (1981). Introduction
to System Dynamics Modeling with DYNAMO.
Cambridge, MA: Productivity Press. Reprinted by
Pegasus Communications
 Saeed, K. (1992). "Slicing a complex problem for
systems dynamics modeling." System Dynamics
Review, 8(3), 251-262
38