Luncheon Webinar Series

June 3rd, 2010

Deep Dive MetaData Workbench
Sponsored By:
1

Deep Dive MetaData Workbench

Questions and suggestions regarding presentation
topics? - send to editor@dsxchange.com
Downloading the presentation
http://www.dsxchange.net/MetaDataWorkbench.html
Replay will be available within one day with email with details

Pricing and configuration - send to editor@dsxchange.net

Bonus Offer Free premium membership for your DataStage
Management! Submit your managements email address and we will offer
him access on your behalf.

Email Info@dsxchange.net subject line Managers special.

Join us all at Linkedin http://tinyurl.com/DSXmembers

Tips and Tricks for Managing,

Administering Metadata Successfully
TSB-3403

Marc Haber
Functional Architect, Infosphere Metadata Tools

Disclaimer
Copyright IBM Corporation 2010. All rights reserved.
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP
Schedule Contract with IBM Corp.
THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL
PURPOSES ONLY. WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY
OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED AS IS WITHOUT
WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON
IBMS CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM
WITHOUT NOTICE. IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE
OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING
CONTAINED IN THIS PRESENTATION IS INTENDED TO, NOR SHALL HAVE THE EFFECT OF, CREATING
ANY WARRANTIES OR REPRESENTATIONS FROM IBM (OR ITS SUPPLIERS OR LICENSORS), OR
ALTERING THE TERMS AND CONDITIONS OF ANY AGREEMENT OR LICENSE GOVERNING THE USE OF
IBM PRODUCTS AND/OR SOFTWARE.
IBM, the IBM logo, ibm.com, Infosphere, and are trademarks or registered trademarks of International Business
Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are
marked on their first occurrence in this information with a trademark symbol ( or ), these symbols indicate U.S.
registered or common law trademarks owned by IBM at the time this information was published. Such trademarks
may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available
on the Web at Copyright and trademark information at www.ibm.com/legal/copytrade.shtml

Agenda

Introduction
InfoSphere Information Server
InfoSphere Foundation Tools
Metadata Primer

Getting Started
Goals
Architecture
Administration Tasks

Product Demonstration
Import, Manage and Deliver

Summary and Conclusion

Introduction

InfoSphere Vision
An Industry Unique Information Platform

Simplify delivery of Trusted Information

Accelerate Client Value
Promote Collaboration
Mitigate Risk
Modular, yet Integrated
Scalable Project to Enterprise

InfoSphere Information Server

IBM InfoSphere Information Server
Unified Deployment

Discover, model,
define, and govern
information structure
and content

Standardize, merge,
and correct information

Combine and
restructure information
for new uses

Unified Metadata Management

Synchronize, virtualize
and move information
for in-line delivery

InfoSphere Foundation Tools

Business Glossary

Information Analyzer

FastTrack

Manage
Business Terms

Assess Data
Quality

Capture
Design Specifications

Metadata

Data Architect

Discovery
Understand Data
Relationships

Design
Enterprise Models

Metadata
Workbench
Monitor
Data Flows

IBM Industry Models

Leverage Industry Best Practices

InfoSphere Foundation Tools Portfolio

Enterprise Projects
Test Data Generation

Discover and understand

the data across
heterogeneous systems

Application Retirement &

Consolidation
Data Archival

Design trusted
information structures for
business optimization

Data De-identification
Data Quality

Govern that information

over time

Data Integration
Master Data Management
Data Warehousing

InfoSphere Foundation Tools

Manage Business
Terms

Discover Data
Relationships

Design Enterprise
Models

Capture Design
Specifications

Assess, Monitor,
Manage Data Quality

Monitor Data Flows

Business Glossary

New Discovery

Data Architect

FastTrack

Information Analyzer

Metadata Workbench

Discover
10

Design

Govern

Infosphere Metadata Workbench

Governance

Asset catalog and metadata

reporting for Data Governance
initiatives and requirements

Compliance

Analysis Reporting for compliance

measures in ensuring data quality
and trust of Data Sources

Standards

Data Flow reports are requirements

of Sarbanes Oxley, Basel II and
other regulatory standards

Change
Understanding and reacting to the impact of
change of Data Sources and structures

Metadata Primer
Literally, data about data
helps to describe a companys information from
business, technical, and operational perspectives

Practically, information that is important and critical,

information that is difficult to grasp or fully understand,
information that is continually emerging and processed

Metadata Primer standard definition

Business Metadata
Audience: Business users
Purpose: Business rules, definitions, terminology, glossaries, algorithms
and lineage using business language

Technical Metadata
Audience: Specific Tool Users BI, Data Integration, Profiling, Modeling
Purpose: Defines source and target systems, table and field/attribute
structures, derivations and dependencies

Operational Metadata
Audience: Operations, Management
Purpose: Information about application runs: frequency, record counts,
component by component analysis, other statistics

Metadata Primer user definition

Meaning
Understand the true meaning of a concept, what business process or
entity does it represent, what business rules govern it, what
specifications define it, what concepts are related

Size and Construct

Understand the length, type and structure of a concept

Metrics
Understand the cardinality, range, valid values, frequency of a concept

Usage
Trace the data flow through systems and applications, understand
what processes and logic is involved in moving, transforming or
otherwise aggregating data

Metadata Business Drivers

Governance and Compliance Regulations are increasing
How do organizations comply and meet documentation requirements?

How can organizations ensure accountability and responsibility?

Business Competition continues to grow

How do organizations individualize their customer experience?
How can organizations get access to information to make correct decisions?

Costs and system complexities are expanding

How can organizations drive optimization with integration?
How do organizations manage complex software environments?

Metadata Primer Design Metadata

Job Design Analysis:
Analysis is defined as the projected flow of information, across different
DataStage Jobs where the target and source Stages share a common source.
Such information is necessary to determine the Impact of Change or Data Flow
Analysis Reports delivered by the Infosphere Metadata Workbench.
Linkage of Jobs via their common Stage Types and properties.
Requires Automated Linkage service to be invoked
Does not require user to load or use Physical Schemas or Files

Metadata Primer Operational Metadata

Job Operational Analysis:
Analysis is defined as the actual flow of information, from a Source data item
through a set of actions defined within a DataStage Job and written to a Target
data item, based upon the Operational Job Run logs of the Job. Form a
complete ETL Data Flow diagram, analyzing the sources of information, Job Run
statistics and Transformation logic.
Linkage of Jobs via their Job Run Operational Logs
Requires import of Operational Metadata
Requires Automated Linkage service to be invoked

InfoSphere Metadata Workbench

Exploration and Analysis of Information Assets

Features

Explore, analyze and manage assets

Data Lineage and Impact Analysis

Extended visibility to enterprise integration

flows outside of Information Server

Full searching and querying across

information Assets

Benefits

Mitigate risk for change management

Support compliance and governance initiatives

Comprehensive understanding of data lineage for

trusted information

IT Developers
Administrators

Project Managers
& DBAs

Data Lineage

View end-to-end lineage

including design metadata,
operational metadata, user-defined
metadata

View context-specific details

including stewards, term, description, Job
image, Job operational metadata details,
etc.

Business Lineage
Business oriented view of
Data Lineage Analysis report

Business Lineage is configured

within the Metadata Workbench, explicitly
including only key Data Assets

Catalog and Display

Data Catalog
browse data structures, including
Database, Data File, BI Report and
Job assets

Asset Details
display asset information, including
relationships and usage details

Asset Display Information

Asset Information
display base information, including
description, container and relationships

Asset Usage
understand ETL Jobs or Mapping
consumption, Business Glossary defined
meaning, Data Steward, Mapping
Specification requirement from FastTrack or
Analysis Profiling data from Information
Analyzer

Search and Query

Homepage
quickly search, display or query
Information Assets

Query Results
formatted as a spreadsheet, for
easier understanding and readability

Query Result Information

Results
Formatted as a spreadsheet, for easier understanding and readability
Grouped according to Type
Ability to save as Spreadsheet or Text File

Query Construction

Create specific ad-hoc Reports

Select Information Asset properties and Relationships or their propertiers
Add specified conditioning filters
Publish Queries for all users

Getting Started

Design Specification
Design Document
Abstract definition and
specification which govern the
flow of information from Source
System for Reporting, OLAP
and Mining deliverables.
Governance and Auditing
requirements dictate the need
for Data Lineage reporting
analysis.

Identify and Plan the Tasks

1.
2.
3.
4.

5. DataStage Jobs
6. Data Scripts

System Application
Data File
Database Warehouse & Mart
BI Reports

7. Data Flow Analysis

4

Goals
Data Lineage
Ability to view Data Flow, validate Systems of Record, validate
Business Logic

Data Reporting
Ensure compliance and data re-use, understand data consumption

Data Terminology
Ensure standardized language, descriptions and methodology

Data Consistency
Ensure proper Data Formatting, Data Type and Value Range

Metadata Preperation

Import metadata about Database Tables and Files that are used
in Job Design and Production

Import metadata about BI Reports used to publish information

Define and import Extended Data Sources and external Data

Mappings for a complete end-to-end lineage flow

Publish shared metadata as necessary

Generate and import operational metadata from job runs

Invoke Metadata Workbench administrative services

Did you know? Design metadata for DataStage and QualityStage jobs is automatically stored
in the metadata repository as well as metadata from all other suite tools.

Data Lineage and Impact Analysis

Data Reporting and Querying

Metadata Workbench Architecture

AUTHOR AND LINK TO IT ASSETS

BUSINESS
GLOSSARY

FAST TRACK

METADATA
WORKBENCH

INFORMATION
ANALYZER

MANAGE CONTENT

INFOSPHERE
DATA
ARCHITECT

Metadata
Workbench

METADATA SERVER
Data Structure
Lineage
Operational

Business
ETL Design

Technical
IMPORT/EXPORT MANAGER OR
DATASTAGE CONNECTORS

ETL Operational

BI Structure

Querying

IT ASSETS

BI REPORTS, PHYSICAL SCHEMAS, DS/QS JOBS

Understanding

Infosphere Import Export Manager

Features
Import capabilities for 3rd party BI tools (Cognos,
Business Objects, MicroStrategy), data modeling
tools (ERwin, RDA) and databases (ODBC
connections to all major RDBMS)

Metadata Bridges interchange metadata with each

specific application a consist of a model, a decoder,
and an encoder which require no coding.

Support a variety of import formats including XMI,

XML, UML, CWM and CSV metadata exchange
formats

Benefits

Visibility of data modeling to ETL to report layer

minimizes
risks
of
overlooking
critical
dependencies

Leverage
common
metadata
exchange
environment
for
application
development
consistency

IT Developers
IT Administrators

Infosphere Import Extended Data Source

IT Developers

Data Source
import and maintain application,
procedure or file definitions from
spreadsheets

Data Flow
import and maintain source to target
mappings, their business logic and
function from spreadsheets

IT Administrators

Infosphere Import Extended Data Mapping

IT Developers

Data Flow Mapping

document and express the
transformation or business logic
between source and target

IT Administrators

Custom Attributes
extend the properties of a mapping to
record specific and proprietary
information, including runtime data,
specification or organizational data

Create or Import
create Extended Data Flow Mapping
documents within the Metadata
Workbench or import from a file

Infosphere Data Lineage Administration

Metadata
Administrators

Ability to include or exclude Projects

Intelligent metadata linking
Ability to schedule Analysis Services
Ability to map Database Aliases
Enhanced and extended support for Stages

Allows administrators to minimize time

maintaining and managing
metadata assets as well as reduce
the numbers of errors introduced
from manual reconciliation
processes.

DataStage and QualityStage Development

As a developer creates the Job canvass, they are building a flow of data
from the Source to the Target of the Job. That flow, connected with other
Job flows, will translate into Data Lineage.
The Metadata Workbench Linkage Services will infer a relationship between
both DataStage Jobs, based upon a common Data Set.

DataStage and QualityStage Job Design

Ensuring a proper Job Design, while maintaining standards for naming and data
connectivity will ensure greater linkages between the Job Design and the
imported Data Source.

Database Connectors
Job Parameters and Environment Variables
Load Column information from Shared Table
Supported DataStage Stage Types
DataStage Common Connector Stages
Build SQL vs. User Defined SQL

Infosphere Data Lineage Support

The
following
DataStage
and
QualityStage stages are supported by
the IBM Metadata Workbench analysis
service in determining cross Job
relationships based upon the values of
the Stage properties.
Other types of DataStage Stages may
be manually associated to Database
Tables or Data File Elements.

(S) = Server Canvas

(P) = Parallel Canvas
(M) = Mainframe Canvas

DB2 Native

DB2 UDB API (S, P)

DB2/UDB Enterprise (P)
DB2 UDB Load (S, P)

Server Name
Schema Name
Table Name

RDBMS Native

Dynamic RDBMS (S, P)

Server Name
Schema Name
Table Name

MSOLE Native

MS OLEDB (S)

Server Name
Schema Name
Table Name

MSSQL Native

MS SQL Server Load (S)

SQL Server Enterprise (P)

Server Name
Schema Name
Table Name

Oracle Native

Oracle 7 Load (S)

Oracle Enterprise (P)
Oracle OCI (S)
Oracle OCI Load (S)

Server Name
Schema Name
Table Name

Sybase Native

Sybase BCP Load (S)

Sybase Enterprise (P)
Sybase IQ 12 Load (S)
Sybase OC (S)

Server Name
Schema Name
Table Name

ODBC

ODBC (S)
ODBC Connector (P)
ODBC Enterprise (P)

Server Name
Schema Name
Table Name

TeraData

Teradata API (S, P)

Teradata Connector (P)
Teradata Enterprise (P)
Teradata Export (M)
Teradata Load (S, M)
Teradata Multiload (S, P)
Teradata Relational (M)

Server Name
Table Name

Complex Flat File

Complex Flat File (S, P, M)

File Name

Other Flat File

Delimited Flat File (M)

Fixed-width Flat File (M)
Multi-format Flat File (M)

File Name

Hash File

Hashed File (S)

File Name

Sequential File

Sequential File (S, P)

File Name or Pattern

Product Demonstration

Summary and Conclusion

Summary
Step 1: Understanding the objectives
Step 2: Defining the Tasks
Step 3: IBM Infosphere Delivering Lineage and Understanding

Thank You!
Your Feedback is Important to Us

44

Dont Miss these Foundation Tools Sessions!!

Wed - May 19

Future Directions in Integrated

Data Quality
USL-3873 02:00 PM - 04:00 PM
Introduction and Overview InfoSphere Foundation Tools
Featuring Business Partner:
Accantec Information Solutions
TSB-3392 03:00 PM - 03:50 PM
The Evolution of a Complex Data
Warehouse with InfoSphere
Foundation Tools
Customer: Consip S.p.a
TSB-3333 05:15 PM - 06:05 PM
Building Business-led
Informational Solutions with
Industry Models, InfoSphere
Warehouse, Business Glossary
and Cognos
TSB-3593 05:15 PM - 06:05 PM

Fri - May 21

Thu May 20

Data Discovery & Mapping to Accelerate

Information Centric Projects
TSB-3405 07:45 AM - 08:45 AM
Using Information Analyzer for Data Quality Health
Monitoring
TSB-3410 07:45 AM - 08:45 AM
Reduce costs, speed collaboration, and access
critical data w/ low impact using Foundation Tools
HOL-3845 10:30 AM - 01:30 PM
Get the Most Out of Your Data Modeling & Metadata
Customer: Danske Bank
TSB-3496 11:45 AM - 12:35 PM
A Metadata Based Approach to Data Governance
Customer: Deutsche Bank
BLD-3615 02:00 PM - 02:50 PM
InfoSphere Foundation Tools Deep Dive & Roadmap
TSB-3393 02:00 PM - 02:50 PM
Governing Your Information Supply Chain
TSB-3379 02:00 PM - 02:50 PM
Do You Really Trust Your Information? See How
You Can - Live Demos Included
TSB-2902 02:00 PM - 02:50 PM

Tips & Tricks for Managing &

Administering Successful Metadata
TSB-3403 07:45 AM-08:45 AM
Succeed In Getting All Stakeholders
Involved Using Business Glossary
TSB-3414 07:45 AM-08:45 AM
Delivering Smart Analytics, ROI &
Business Benefits through the
InfoSphere Portfolio
Customer: 3UK
BLD-3493 9:00 AM 9:50 AM
Accelerate Master Data Design and
Definition using InfoSphere
Discovery
TSB-3545 9:00 AM 9:50 AM
Industry Models for Basel II
Compliance and Risk Management
Customer: CitiGroup
BLD-3022 12:30 PM 01:30 PM

** Visit Our Live Demos Every Day @ The Demo Room! **

Understand and Map Your Distributed Data
Integrated Metadata for Enterprise Collaboration and Trust

Assess Information Quality and Health

Proven Models that Accelerate Your Information Agenda

Customer Sessions, Presentations, Usability Sessions, Live demos, Hands-On Labs

Contacts us for more information about

IBM InfoSphere Metadata Workbench
Marc Haber march@il.ibm.com
Functional Architect, Metadata Tools
Infosphere Metadata Workbench product specialist
Farnaz Erfan erfan@us.ibm.com
Metadata Product Marketing Manager