ETL Detailed Design Document

ETL Detailed Design Template V1.0

Document Details
Project Name
Current Version
List of Contributors
Customer Contact Information

Prepared by/Date Reviewed by/Date Approved by/Date

Revision History
Date of Revised Description of Reason for Affected Approved
Version
Revision By Change Change Sections By

Affected Groups

DRAFT Confidential Page 2 of 15

 ETL Detailed Design Template V1.0

Table of Contents

1 INTRODUCTION 5

1.1 DOCUMENT PURPOSE AND SCOPE.................................................................................

1.2 PROJECT BACKGROUND................................................................................................

1.3 PROJECT SCOPE..........................................................................................................

1.4 DATA SOURCES AND CONTACT DETAILS.........................................................................

1.5 REFERENCES................................................................................................................

2 TOOLS 6

3 LOAD STRATEGY 7

3.1 OVERVIEW....................................................................................................................

3.1.1 Initial Load........................................................................................................

3.1.2 Incremental Load..............................................................................................

4 DETAILED ETL DESIGN 8

4.1 <MODULE NAME/SUBJECT AREA>.................................................................................

4.1.1 Source Definition..............................................................................................

4.1.2 Target Definition................................................................................................

4.1.3 Transformation and Load..................................................................................

4.2 PROCESSING LOGIC......................................................................................................

4.2.1 Process Flow....................................................................................................

4.2.2 Error Handling...................................................................................................

4.2.3 Reconciliation Logic..........................................................................................

4.2.4 Dependencies...................................................................................................

5 RESTART AND RECOVERY 10

6 GENERAL PURPOSE ROUTINES 11

6.1 <ROUTINE>................................................................................................................

6.1.1 Input Parameters............................................................................................

6.1.2 Output Parameters..........................................................................................

DRAFT Confidential Page 3 of 15

 ETL Detailed Design Template V1.0

6.1.3 Frequency.......................................................................................................

6.1.4 Error Handling.................................................................................................

6.1.5 Notification......................................................................................................

6.1.6 Dependencies.................................................................................................

7 SCHEDULER 12

7.1 SCHEDULING INTERFACE.............................................................................................

8 DATA RETENTION AND ARCHIVING 13

9 TRACEABILITY MATRIX 14

10 ACRONYMS AND GLOSSARY 15

DRAFT Confidential Page 4 of 15

 ETL Detailed Design Template V1.0

1 Introduction

Brief introduction about the project

1.1 Document Purpose and Scope

Brief the objective of the project

1.2 Project Background

1.3 Project Scope

Brief the scope of the project

1.4 Data Sources and Contact Details

Data Source Name Contact Details

1.5 References

Document name Author Version / Date Location of document

DRAFT Confidential Page 5 of 15

 ETL Detailed Design Template V1.0

2 Tools

List the tools used, if any

DRAFT Confidential Page 6 of 15

 ETL Detailed Design Template V1.0

3 Load Strategy

3.1 Overview

Detail the load strategy.

3.1.1 Initial Load

Describe the first time load.

3.1.2 Incremental Load

Describe the incremental loading process.

3.1.2.1 Full Refresh

Describe the incremental loading which fully refreshes the data in the tables after the first time load

3.1.2.2 Delta

Describe delta incremental loading

DRAFT Confidential Page 7 of 15

 ETL Detailed Design Template V1.0

4 Detailed ETL Design

This section should describe in detail the extraction, cleansing, transformation and load strategy for all the modules
and the programs within them.

4.1 <Module Name/Subject Area>

This section should describe in detail the extraction, transformation, cleansing and load strategy for the subject area
/module under consideration with the data flow diagram.

4.1.1 Source Definition

Table/File Column/Field Column/Field Data Type Length

Name Name Description

Numeric/Alpha-
Numeric

4.1.2 Target Definition

Table/File Column/Field Column/Field Data Type Length

Name Name Description

Numeric/Alpha-
Numeric

4.1.3 Transformation and Load

This section should describe in detail the transformation and load process - to load the target tables/files(Staging
Area or Data warehouse database) using all the source tables/files .

Target Target Target Target Master Source Source Data Transformation

Table/ File Column/ Column/ Column/ Field Table/ File Table / File Column/ Field
Name Field Field Data Lookup Name Name
Name Default Validation information
value checks
(null/length,
data type)

DRAFT Confidential Page 8 of 15

 ETL Detailed Design Template V1.0

4.2 Processing Logic

4.2.1 Process Flow

Process flow will contain the algorithm or a similar step by step representation of the processing done by the routine.

4.2.2 Error Handling

This section should detail the error handling for the program under consideration.

4.2.3 Reconciliation Logic

This section explains the logic used to keep track of the number of records fed, number of records processed, number of valid
records, number of records corrected and number of records rejected and how they are reconciled.

4.2.4 Dependencies

This section should detail the dependencies of the program under consideration with other programs or General
purpose routines or any other external system.

DRAFT Confidential Page 9 of 15

 ETL Detailed Design Template V1.0

5 Restart and Recovery

Detail the restart and recovery steps

DRAFT Confidential Page 10 of 15

 ETL Detailed Design Template V1.0

6 General Purpose Routines

Detail each routine that will be used in general (ex: date conversion, some specific validations across programs etc)
or for data extraction or for data loading.

6.1 <Routine>

Detail the scope of the routine and its funtionality.

6.1.1 Input Parameters

Detail the input parameters used in the routine.

6.1.2 Output Parameters

Detail the outputs of the routine.

6.1.3 Frequency

Detail the frequency and time when this routine will be executed if independent.

6.1.4 Error Handling

This section should detail the error handling.

6.1.5 Notification

This section should brief the notification procedure.

6.1.6 Dependencies

This section should brief the dependencies on programs, sub-programs or any other external system.

DRAFT Confidential Page 11 of 15

 ETL Detailed Design Template V1.0

7 Scheduler

7.1 Scheduling Interface

Details of schedullng tool to be used and the schedules for the programs.

DRAFT Confidential Page 12 of 15

 ETL Detailed Design Template V1.0

8 Data Retention and Archiving

This section gives details of data retention and archiving.

DRAFT Confidential Page 13 of 15

 ETL Detailed Design Template V1.0

9 Traceability Matrix

This Section lists the traceability information to trace the requirements from Requirements Analysis document to
Design document. The locations of design specifications should be mapped to the corresponding sections of the
requirements in the Requirements Analysis document.

Document reference Id & Description: (Doc Id from which this document is derived)

Serial Reference document Design document

No. Requirement/Feature (Section ID/Name) (Section ID/Name)

DRAFT Confidential Page 14 of 15

 ETL Detailed Design Template V1.0

10 Acronyms and Glossary

This section describes the acronyms of various technical terms used in the document.

DRAFT Confidential Page 15 of 15