How to Prepare Your Data for the Cloud

Leveraging Data Migration Best Practices to Facilitate the Move to Oracle Cloud

Kate Miller
Premier International
kate_miller@premierintl.com
Presenter Introduction
Kate Miller, Principal
➢ 17 years with Premier International
➢ Focused specifically on Data Migration
Founded: 1985
➢ Professional services & best practices developed specifically for Data Migration Headquartered:
➢ Deliver Oracle Cloud ERP / PLM / HCM Data Migration Solutions Downtown Chicago
No offshoring
Specializing:
➢ Our Mission: Remove Data Migration from the critical success path, delivering Data Assessment,
the highest quality results on time - every time Data Migration,
Data Archiving
Strategies
Supporting
migration for :
ERP
HCM
PLM

Any legacy/target
Agenda

➢ Introduction to Data Migration

➢ Understanding the Oracle Cloud Load Templates

➢ Preparing Data Prior to Load

➢ Populating Oracle Cloud Load Templates

Data Migration is Critical

“Businesses spend billions of dollars migrating data between information-intensive applications. Yet up
to 75 percent of new systems fail to meet expectations, often because flaws in the migration process
result in data that is not adequately validated for the intended task.”

Oracle – Successful Data Migration

http://www.oracle.com/technetwork/middleware/oedq/successful-data-migration-wp-1555708.pdf

➢ ERP systems are more critical to business then ever before, but most legacy system data is in no
fit state to take advantage of the latest advancements

➢ Business Intelligence derived from the latest ERP applications continues to accelerate as
companies look to leverage their data assets

➢ Moving from a traditional ERP / PLM / HCM to Oracle Cloud can vastly increase Data Migration
complexity – highlighting the importance of proper planning & preparation
 What Makes Data Migration Problematic?
➢ Data Quality
➢ The data quality is never better that you think. Often, it is much worse – especially when you have multiple
sources. Oracle Cloud is less forgiving of these types of issues.
➢ Immense Scope
➢ Every data value, in every column/field, from every row/record, in every table/file, in every data source must be
understood, transformed, and moved to the correct location in the new system (or purposefully left behind) before
the old systems can be retired.

➢ Cloud Database Access ➢ Complex transformations

➢ Unlike traditional On Prem solutions, Oracle Cloud ➢ The old system data structure and content often
doesn’t allow direct SQL access to update / correct / doesn’t “fit” the new system.
back out data if something is incorrect. Additionally, ➢ Disparate Legacy Data Sources
Cloud ‘pods’ cannot be refreshed / restored on the fly
➢ Combining multiple data sources into a single target
– refreshes must be scheduled in advance, making
system requires wrangling multiple, unrelated
every data load critical.
structures into a common format
➢ Unknown & Misunderstood data
➢ Endless Requirements Changes
➢ Documentation is often out-of-date or missing entirely,
resulting in inaccurate specifications ➢ The intersection of immense scope and complex
requirements based on inaccurate data knowledge
sets the stage for a nightmare
Using Oracle Load Utilities

FBDI – File Based Data Import

➢ Populate spreadsheets, following a defined layout, and use Oracle load utility to stage and
process data set

ADFdi - ADF Desktop Integration

➢ Populate spreadsheets, following a defined layout, and upload data directly from Excel

➢ FBDI and ADFdi both enforce data integrity validation / cross-checks to prevent invalid
data scenarios

➢ Ensuring FBDI / ADFdi content is accurate, consistent and clean is critical to ensuring
successful data loads
The Actual Data Migration Programs
➢ Data Migration programs are not the Oracle Cloud loaders
➢ Data Migration programs include routines that:
➢ Extract from the legacy databases
➢ Apply data cleansing/harmonization rules
➢ Transform the data into the new format
➢ Ensure integrity across related data areas
➢ Identify missing configurations prior to the actual load into Oracle Cloud
➢ Identify which records will fail prior to the execution of the loader program
➢ Provide opportunity for the legacy data owners \SMEs to review it before it is loaded into
the Cloud
➢ Automation and Repeatability are crucial
➢ Any manual intervention needs to be kept to a minimum
➢ Excel and vlookups are not highly automated and repeatable solutions
FBDI: Understanding the
Load Templates
FBDI: Understanding the Load Templates
➢ Each FBDI template includes instructions on population and the load process

➢ Files are loaded as CSVs through the Cloud’s Load Interface File process

➢ Most FBDI templates contain multiple tabs for related data sets – the snippet below shows
part of the ITEM load template
FBDI: Understanding the Load Templates
➢ Each FBDI template includes instructions on population and the load process
➢ Files are loaded as CSVs through the Cloud’s Load Interface File process
FBDI: Understanding the Load Templates
FBDI: Understanding the Load Templates

➢ Most FBDI contain basic sample data to show layout / content options – but this often doesn’t
include more complex business scenarios

➢ While some columns have detailed comments outlining how to populate that field, not all are
so forthcoming or intuitive
FBDI: Understanding the Load Templates

➢ The templates outline what to do once populated – the trick is getting legacy data
to match the format, content and structure expected by Oracle Cloud

➢ How do you get from what you have today, to what you need tomorrow?
Preparing Data Prior to Load
Overall Data Migration Methodology
START 1 2 3 4

Define Extract Legacy Perform Define Legacy

Conversion Tables/Files to Data Profiling to Cloud
Objects / Data Consolidated and Analysis Conversion
Requirements Data Repository Requirements

8 7 6 5

Accurate Data with

Analyze Pre- Pre-validate Transform/ Validate Conversion better quality
validation Data Before Cleanse Data Requirements
and Resolve Attempting Within Data Using Data Profiling
Errors Load Repository and Analysis Early availability of
data for testing
FINISH

9 10 11

Analyze Reduced risk for

Load Data to Perform
Exceptions and data conversions
Cloud and Reconciliations
Analyze Resolve Errors
Validations
Phases of a Legacy Data Assessment

Make Decisions Based on Facts, Not Assumptions

What’s the First Step of the Assessment?
➢ Explore / Identify – Start to confirm / update assumptions about the data
➢ Which tables are used
➢ Which fields are used
➢ Values in each field
➢ Data patterns across each field

➢ Start to uncover data quality issues

➢ Missing, unexpected, invalid values
➢ Ways users ‘hacked’ the solution to get around system limitations

➢ Start the conversation with the business about data

➢ Ask questions about complexities / data entry procedures
➢ Understand current data governance standards
➢ Uncover additional data sources
➢ Excel spreadsheets
➢ Rogue databases
➢ Additional data sources that might not be known

➢ Profiling is also invaluable through out the project:

➢ Assisting with the data mapping specifications
➢ Addressing data issues
➢ Validating migrated data
➢ Testing conversion processes
You know the Facts - Now what?

It’s time to dig deeper!

➢ Talk to business users in regards to current data issues in the system
➢ Discuss the profiles, you’ll be surprised that additional data sources will be uncovered during these interviews

➢ Take the profiling results and dive deeper into the data to unravel data
➢ Unexpected values
➢ Invalid values

➢ Start to identify data issues

➢ Duplicate candidates
➢ Integrity issues
➢ Harmonization issues

➢ Compare datasets between disparate systems.

➢ Verify assumptions about how the systems interact.
➢ Are the datasets truly in-sync?
➢ What is different about the same piece of data in multiple systems?

➢ Leverage data profiles and this additional analysis to drive the detailed data mapping specifications
Harmonizing of Disparate Systems

Disparate legacy systems and data sources ranging from 2 to 200 may need to be harmonized, merged, and
transformed before hitting Oracle Cloud.

During phased rollouts or rollouts into an ongoing application, Oracle Cloud also becomes a system that
the legacy data needs to be reconciled with and incorporated into via the harmonization process.

➢ When systems are related / integrated:

➢ There are always data integrity issues between the systems
➢ There are always exceptions to the assumption that the data is supposed to be same

➢ When systems are not integrated, but contain equivalent data:

➢ Additional thought needs to be put into harmonizing the data sources
➢ Decisions need to be made concerning the consolidation of overlapping datasets

➢ Each of the disparate systems will have their own data issues that will compound once they start to
merge
Identifying Duplicates & Maintaining Referential Integrity

➢ Criteria for identifying duplicates is significantly more than just looking for exact matches
➢ Noise words, Phonetic Encoding, Combinations of different fields, etc

➢ Often requires some level of manual review to approve the consolidations

➢ Referential integrity across the consolidated data needs to be maintained or downstream

conversions will fail.

Warning: If the duplicate data is not resolved before or during the migration, the duplicate data
will either silently load and cause problems or error out during the load and cause problems.
Data Quality Strategy

Data Quality Issues Can Sink Any Implementation

➢ Establish a Data Quality Strategy Early – Before any conversions are built
➢ Process for resolving future data issues as they are encountered
➢ Catalog data issues, data gaps, and the procedures for resolving them
➢ There are three primary methods for managing data cleansing:
➢ Manual
➢ Automated
➢ Hybrid
When Should the Data Work Start?

➢ Leaders/Sponsors don't understand the data complexities so data work is not started early

➢ There are no corporate policies to follow, because the process happens so infrequently
➢ Near the beginning of the project or just through the first test cycles, there comes the
realization that the disciplines and skills required are considerably more involved than was
anticipated
➢ Data is always more work than anticipated and
assumptions will be proven wrong
➢ When the client is responsible for preparing the
data templates, anticipate issues

START EARLY!
Populating Data Load
Templates
Complete Data Migration Path
The Actual Data Migration Programs
➢ Data Migration programs are not the Oracle Cloud loaders
➢ Data Migration programs include routines that:
➢ Extract from the legacy databases
➢ Apply data cleansing/harmonization rules
➢ Transform the data into the new format
➢ Ensure integrity across related data areas
➢ Identify missing configurations prior to the actual load into Oracle Cloud
➢ Identify which records will fail prior to the execution of the loader program
➢ Provide opportunity for the legacy data owners \SMEs to review it before it is loaded into
the Cloud
➢ Automation and Repeatability are crucial
– Any manual intervention needs to be kept to a minimum
– Excel and vlookups are not highly automated and repeatable solutions
Where Should Data Preparation Occur?

➢ Legacy system: A terrible idea

➢ Target system: Not possible with Oracle Cloud and generally not a good idea. The
environment is either not ready in time or too unstable when a large chunk of the
data migration work needs to take place.

➢ Centralized Repository: A centralized data repository which can connect to all

disparate systems and produce output in the necessary load format.

Excel is not a centralized repository.

Benefits of a Centralized Data Repository

➢ All disparate legacy data in a single place

➢ Facilitates analysis of legacy system data without impacting any of the active systems
➢ Facilitates mini-test cycles into a mocked-up version of the target system
➢ Facilitates a way for data cleansing activities to occur outside the legacy and target systems
➢ Facilitates an easy way to utilize additional transformation and enhancement rules
➢ Facilitates testing and retesting on a stationary data set
➢ Facilitates the ability to reconcile data post conversion
➢ Removes impact on legacy production environments
➢ Allows pre-conversion test runs
➢ Allows testing on stationary sets of data
➢ Simplifies post conversion reconciliation
Pre-Load Validation

➢ A large portion of the validation should occur well before the data load is attempted

➢ A loaded record doesn’t necessarily mean that the record is right

➢ Data Validation should confirm test both the technical accuracy and the accuracy of
the business requirement accuracy

➢ Waiting for the Oracle Cloud load process to point out errors and
exceptions with data delays the migration effort – and may not
catch everything
In Closing…

➢ The complexity of today’s systems have caused the need for Data Migration resources to be
highly specialized and equipped to deal with the challenges data brings to implementations.
It is always more effort than anticipated!

➢ While FBDI and ADFdi streamline the process of loading data to the Cloud, the real effort is in
getting cleansed, formatted and business appropriate data into the templates

➢ If left to their own devices, most organizations will struggle to produce data load files which
load seamlessly – even with the information provided by the FBDI templates
 How to Prepare Your Data for the Cloud

Leveraging Data Migration Best Practices to Facilitate the Move to Oracle Cloud

Kate Miller
Premier International
kate_miller@premierintl.com