Model Development Unit

Strategy Directorate
1
Developing & Maintaining
Dynamic Micro-Simulation
Models at DWP
Sally Edwards
Simon Gault

ESRC 2
nd
April 2009
2
Model Development Unit
Strategy Directorate
Developing Models at DWP
1. Overview
2. Objectives
3. Development Standards
4. Simplifying Complexity
5. Maintenance Protocol
6. What worked well and Lessons Learned
3
Model Development Unit
Strategy Directorate
Introductions
Simon Gault
Statistician, with a strong background in Economics
Extensive experience in developing models
Manages the Forecasting Simulation models team

Sally Edwards
Systems / Business Analyst
Predominantly worked in the private sector on large scale IT systems
Manage the Genesis Simulation Model

Members of a the Model Development Unit (MDU) which has
responsibility for Micro-Simulation Models at DWP
MDU consists of 20+ people
4
Model Development Unit
Strategy Directorate
Micro-Simulation Models

Dynamic Micro-Simulation Models, based on our standard
architecture, Genesis:
Pensim2 private & state pension income
Inform Integrated Forecasting model for working age
benefit claimants
Individual Benefit Forecasting for RP, IB, DLA/AA
Employment Model, currently being validated

Static Micro-Simulation Models:
Policy Simulation Model (PSM)
Employment and Hours Model (EHM)

5
Model Development Unit
Strategy Directorate
Key factors taken into Consideration during
Development
Key factors considered when we designed our first dynamic micro-simulation
model, Pensim2 - also valid for our other models:

Simplify the complexity of the model as much as possible, using a modular
approach and allowing a simple prototype model to be developed quickly

Several developers work on each model at the same time

Large staff turnover

Models are intended for long term use (e.g. Pensim2 anticipated to have
15-20 year life) so most of the people using/maintaining the models are
not involved in their development

Clear documentation, processes, training and user guides are essential

Excel and SAS are the preferred tools, as both are well understood and
extensively used among the Analysts at DWP
6
Model Development Unit
Strategy Directorate
Developing Models at DWP
1. Overview
2. Objectives for Pensim2 (first DMS model)
3. Development Standards
4. Simplifying Complexity
5. Maintenance Protocol
6. What worked well and Lessons Learned
7
Model Development Unit
Strategy Directorate
Original Objectives for Pensim2
Pensim2 Feasibility Study document (2002) Objectives
1. Flexibility 8. Efficiency (in use of memory)
2. Robustness 9. Timeliness
3. Transparency 10. Done with own expertise
4. User-friendliness 11. Done with own resources
5. Speed 12. Ease of handover
6. Reliable Output 13. Ease of maintenance
7. Availability 14. Independence of base data
(i.e. desktop model)
8
Model Development Unit
Strategy Directorate
Decision to Split the Model
Impossible to satisfy all the objectives in a single model solution so
decision was made to split the model into two separate parts,
satisfying most of the objectives:












9
Model Development Unit
Strategy Directorate
Decision to Split the Model
Impossible to satisfy all the objectives in a single model solution so
decision was made to split the model into two separate parts,
satisfying most of the objectives:

Excel based front end Pensim2
User friendly, transparent code in a standard format
Easy to understand and flexible, enabling analysts to change
parameters & structure of model without understanding the
underlying code.
Easy to maintain and handover
Allowed independence of base data




10
Model Development Unit
Strategy Directorate
Decision to Split the Model
Impossible to satisfy all the objectives in a single model solution so
decision was made to split the model into two separate parts,
satisfying most of the objectives: :

Excel based front end Pensim2
User friendly, transparent code in a standard format
Easy to understand and flexible, enabling analysts to change
parameters & structure of model without understanding the
underlying code.
Easy to maintain and handover
Allowed independence of base data

SAS code generator - Genesis
Developed with our own expertise and own resources
Robust generator, producing reliable outputs
11
Model Development Unit
Strategy Directorate
Basic structure of Genesis
Dynamic Micro-Simulation Models at DWP
GENESIS Model Engine
generates SAS model
and runs the simulation
Simulation
Model
parameter
sheets (Excel)
SAS Base
Data
SAS post-
simulation
Data
- Standard Excel sheets
used to define the model
- SAS chosen for the model engine and the output data,
as SAS is the standard data analysis tool used
SAS Static
code
12
Model Development Unit
Strategy Directorate
Developing Models at DWP
1. Overview
2. Objectives
3. Development Standards
4. Simplifying Complexity
5. Maintenance Protocol
6. What worked well and Lessons Learned
13
Model Development Unit
Strategy Directorate
Development Standards
Two separate development teams, with different
development standards adopted

Genesis Model Engine
Developed by IT staff
Used standard IT project development procedures

Pensim2 Analytical team
Economists and Statisticians
Less structured approach to development
14
Model Development Unit
Strategy Directorate
Genesis Model Engine Development Standards
Highly structured approach to the development

Project Management Protocol based on Prince2 methodology, although not
all aspects were needed

Documentation 2 types
Development documentation produced at each stage of project, that
was signed off and used as input to the next stage
Key Project documentation (*) - maintained and kept up-to-date with
each new release

Development Processes simple process diagrams to provide clear
guidelines, particularly useful for new staff

Detailed Requirements explaining the full functionality of the model
engine. Document translates Economist language into IT language (*)
15
Model Development Unit
Strategy Directorate
Genesis Model Engine Development Standards
Design Diagrams and documentation used in development stage only

Change Control process changes estimated and impacted before being
accepted/rejected/on hold (*)

Problem Log process problems recorded, prioritised, fixed/removed
from the scope/user error (*)

Validation Process - detailed Test Plan based on Requirements document,
used to ensure that each requirement was fully satisfied

Test Pack ensured existing functionality continued to work when changes
were made to the model engine (*)

User Guide contains details of functionality written in simple format (*)

Training materials, inc presentations and training text for self-study (*)
16
Model Development Unit
Strategy Directorate
Pensim2 Model Development
Pensim2 Analytical development followed a less structured
approach

Generally this was appropriate, as detailed requirements were not
possible to establish up-front

Effort was predominantly spent on regression analysis

Analysis results reviewed to determine the key components to
include and the most appropriate sources for the assumptions

Using the standard Genesis templates, the Pensim2 model was
more easy to develop, rather than if a traditional DMS model had
been built

17
Model Development Unit
Strategy Directorate
Developing Models at DWP
1. Overview
2. Objectives
3. Development Standards
4. Simplifying Complexity
5. Maintenance Protocol
6. What worked well and Lessons Learned
18
Model Development Unit
Strategy Directorate
Key Aim of Genesis Structure - Simplify Complexity
No Hard Coding Rule which allows complete flexibility for:

Structure of the simulation, i.e. order of events processed

Data structures, i.e. data tables, data relationships and variables

Parameters, e.g. indexes, rates, dates

Regression Equations

Selection filters

Static SAS code, if required (e.g. addition of new cases to a model)

Ability to add / remove modules easily

Each model is defined in simple Excel spreadsheets, using standard templates,
that users can easily understand and modify
19
Model Development Unit
Strategy Directorate
Key Aim of Genesis Structure - Simplify Complexity
Data Access routines
Enable easy access to data so user does not have to write complex code

Default data access will process the current object in the current simulation
year

Facility to process a variable belonging to a related object, e.g. partner PAR;
Relationships are not hard-coded and new relationships can easily be
defined

Data manipulation routines provide commonly required data facilities,
e.g. maximum value during past 10 years = MA10

E.g. MA10_PAR_pa_salary returns the maximum value in the past 10 years of
the salary variable for the individuals partner, taken from the pa table

Keep Equations simple
Equations, filters & process flows are defined in a simple, standard format,
that is unambiguous. Hence the models are easy to maintain
20
Model Development Unit
Strategy Directorate
Developing Models at DWP
1. Overview
2. Objectives
3. Development Standards
4. Simplifying Complexity
5. Maintenance Protocol
6. What worked well and Lessons Learned
21
Model Development Unit
Strategy Directorate
Change Management
Change Management Process is followed for all models

Every change goes through the formal Change Control procedure, i.e.
requirements documented, impact assessed, agreed by users & signed off

Change Register listing all changes, each with a separate reference number

For each change to the model, a separate Change Control form is produced

The model code contains the Change ref number, with a comment

Some changes are repeated regularly, e.g. assumptions, so clear
documentation is helpful for the repeat change

Questions arise after a change has been implemented - these can be easily
answered by referencing the Change Control documentation
22
Model Development Unit
Strategy Directorate
Problem Management
Problem Management Process is followed for all models

Every problem/error discovered in a model goes through the formal
Problem Log procedure, i.e. requirements documented, impact is assessed,
agreed by users & signed off

Problem Log Register, listing all problems identified, including those that
turn out to be User Errors or misunderstandings; PL ref number assigned

For each problem recorded, a separate Problem Log form is produced

Users are notified of significant problems that affect the model results

Many of the problems are minor and are picked up by the development
team these are recorded formally, even if they do not impact the results

23
Model Development Unit
Strategy Directorate
User Model
Our models fall into 2 categories:
Maintained & owned by the Model Development Unit
Developed by Model Development Unit; handed over to User
team, who then own and maintain the model

Models owned by the Model Development Unit have a User Group
and/or a Steering Group consisting of representatives from each
team using the model

The purpose of these groups is to agree the content, priority and
timing of changes made to the model


24
Model Development Unit
Strategy Directorate
User Group Purpose
Specifically for Pensim2, our largest model:

Change Requests and Problem Logs are reviewed by the User
Group, where appropriate

A key user sponsor is assigned to each change. The sponsor
ensures that the change is accurately specified & will sign-off
testing

The User Group are responsible for signing off new versions of
the model before they are released

The users also have a separate User Forum where they present
analysis carried out on the model outputs and explain how the
results have been used
25
Model Development Unit
Strategy Directorate
Release Management
Again, specifically for Pensim2:

New Releases are implemented approximately every 6 to 9 months

Each Release includes one or two key components, e.g. Private Pension
Reform, and a bundle of smaller changes and problem fixes

The timing and content of each Release are agreed with the User Group

When a new Release is built:
Each change/fix is added to the model one at a time
Hence each change is reviewed and signed off separately
Enables us to fully understand and agree the effect of each change.
Time consuming but gives confidence in the results

Clear audit trail

All previous versions of the model remain available
26
Model Development Unit
Strategy Directorate
Developing Models at DWP
1. Overview
2. Objectives
3. Development standards
4. Simplifying Complexity
5. Maintenance protocol
6. What worked well and Lessons Learned
27
Model Development Unit
Strategy Directorate
Original Objectives Satisfied
Pensim2 Feasibility Study document (2002) Objectives
1. Flexibility 8. Efficiency (in use of memory)
2. Robustness 9. Timeliness
3. Transparency 10. Done with own expertise
4. User-friendliness 11. Done with own resources
5. Speed 12. Ease of handover
6. Reliable Output 13. Ease of maintenance
7. Availability 14. Independence of base data
(i.e. desktop model)
28
Model Development Unit
Strategy Directorate
Objectives Partially Satisfied
Pensim2 Feasibility Study document (2002) Objectives
1. Flexibility 8. Efficiency (in use of memory)
2. Robustness 9. Timeliness
3. Transparency 10. Done with own expertise
4. User-friendliness 11. Done with own resources
5. Speed 12. Ease of handover
6. Reliable Output 13. Ease of maintenance
7. Availability 14. Independence of base data
(i.e. desktop model)
29
Model Development Unit
Strategy Directorate
Objectives that proved more tricky
Pensim2 Feasibility Study document (2002) Objectives
1. Flexibility 8. Efficiency (in use of memory)
2. Robustness 9. Timeliness
3. Transparency 10. Done with own expertise
4. User-friendliness 11. Done with own resources
5. Speed 12. Ease of handover
6. Reliable Output 13. Ease of maintenance
7. Availability 14. Independence of base data
(i.e. desktop model)
30
Model Development Unit
Strategy Directorate
What went well
Generic framework has proved successful and easy to use

Genesis architecture has been remarkably robust

Modules have been reused/copied from one model to another

Quicker development of new dynamic micro-simulation models,
compared with a traditional build

Automated documentation facility and error checking produced
using VBA, listing where variables are used and providing an
overview of each module, to ensure consistency

Lessons learnt from Pensim2 development were used when
developing subsequent models
31
Model Development Unit
Strategy Directorate
Lessons Learned
Key lessons learnt from Pensim2 development:

Tried to make the model too complicated particularly the analytical side

We should have developed a set of simple modules for a Phase 1 delivery,
then added complexity to the more important modules for Phase 2

The structure of some of our modules was more complicated than the data
would support, e.g. Labour Market process

Insufficient documentation produced by the Analytical team during the
development (e.g. Base Data Creation)

Genesis Engine code is difficult to understand and hence modify, although
it is robust and does not require much modification

Some Genesis Engine functionality was over-ambitious and subsequently
dropped (e.g. Alignment of polychotomous variables)
32
Model Development Unit
Strategy Directorate
Any questions?