Oracle GoldenGate

:::::::::Overview:::::::::::

Sreenivaas Bolisetty For more info : bolisettyvaas@gmail.com

Agenda

Oracle Golden Gate Overview Deep-dive Architecture Demo

Overview of Oracle Golden Gate

Oracle GoldenGate provides real-time data integration to support high availability capabilities across heterogeneous systems to achieve business continuity. GoldenGate provides the ability to capture and deliver updates of critical data as the changes occur and provide continuous synchronized data across heterogeneous environments. Use Cases: Online banking, Claims processing and Online shopping can use GoldenGate to operate without any disruption.

Overview of Golden Gate Contd.

Provides Low-impact capture Routing Transformation and Delivery of transactional data across Heterogeneous environments in Real time

Zero Downtime Migration and Upgrades

New DB/ OS/HW/App

Active-Active High Fully Active Availability Distributed Database Key Features

Log Based, RealTime Change Data Capture

Query Offloading

Performanc e

Non-intrusive, low-impact, sub-second latency

GoldenGate

Reporting Database

ETL ODS ETL EDW

Flexible / Extensible

Open, modular architecture - Supports heterogeneous sources and targets Maintains transactional integrity - Resilient against interruptions and failures

Heterogeneous Source Systems

Real-time BI

EDW

Reliable

Data Distribution

Global Data Centers

SOA/EDA

Oracle GoldenGate
Capture: Committed transactions are captured (and can be filtered) real time, from the

DB Redo logs.

Trail: stages and queues data for routing. Pump: distributes data for routing to target(s). Route: data is compressed, encrypted for routing to target(s). Delivery: applies data with transaction integrity, transforming the data as required.

Capture

Trail

Pump LAN/WAN Internet TCP/IP

Trail

Delivery

Source Oracle & Non-Oracle Database(s)

Target Oracle & Non-Oracle Database(s)

Oracle GoldenGate
Capture: Committed transactions are captured (and can be filtered) real time, from the

DB Redo logs.

Trail: stages and queues data for routing. Pump: distributes data for routing to target(s). Route: data is compressed, encrypted for routing to target(s). Delivery: applies data with transaction integrity, transforming the data as required.

Capture

Trail

Pump LAN/WAN Internet TCP/IP

Trail Delivery (SQL)

ource racle & Non-Oracle atabase(s)

Bi-directional

Target Database(s)

GoldenGate Deployment Topologies

Unidirectional Query Offloading Zero-Downtime Migration Bi-Directional Hot Standby or Active-Active for HA Peer-to-Peer Load Balancing, Multi-Master

Broadcast /Fan-out Data Distribution

Integration/Consolidation/Fan-in Data Warehouse

Data Distribution via Messaging

BPM

DELIVERY CAPTURE Architecture Components (Processes)

Log-based VAM-based Local/Remote Queuing Filtering Parallel Coordination Local/Remote

Native database API ODBC Batched Operations Filtering Transaction/Operations Grouping Parallel Coordination Local/Remote

PUMP (routes from one side to other)

MANAGER (Start/Stop/Report on thresholds, trail purging etc.)

Filtering Local/Remote TCP/IP Compression Encryption

Queue (Trail) Management Process Management (Dynamic) Monitoring and Administration (Lag reports) Reporting (Events, Errors, Thresholds) Receive/Route requests from UI

On-disk Components

Trail Files Checkpoint Files (Capture, Pump, Delivery) Data Definition Files (SOURCEDEFS/TARGETDEFS) Configuration Files Discard Files Report/Log Files Executables

Oracle GoldenGate Components

Communications and Startup

GGSCI
Interactive Interface

Manager

Manager

GGSCI
Interactive Interface

Network
Source Trail
Target Trail

Source

Capture

Pump

Collector

Delivery

Target

Target Database

TCP/IP Process Start Shared Memory (for Monitoring)

GoldenGate Capture
GoldenGate Parameter File
Messaging Subsystem Checkpoint Manager Parser Memory Manager

GoldenGate Event Log

Transaction Logs
Redo API (Log reader interface)

GoldenGate Checkpoint File

Transaction Log Reader

Formatting

Filtering

Transformation and Mapping

Serialization and Routing

Metadata Manager

Metadata Manager

SQL Interface

Transformation Engine

GoldenGate Trail Files

Database Access To gather Metadata (to interpret log data) & for SQL interface for data lookup, advanced filtering etc.

Command and Control

Source Database

Capture

GoldenGate Command Interpreter

GoldenGate Delivery
GoldenGate Parameter File
Messaging Subsystem Checkpoint Manager Parser Memory Manager

GoldenGate Event Log

GoldenGate Checkpoint File

GoldenGate Trail Reader

Read API

Formatting

Filtering

Transformation and Mapping

SQL Generation

GoldenGate Trail Files

Metadata Manager SQL Interface Transformation Engine

Command and Control

Database Access For Metadata access & SQL interface

GoldenGate Command Interpreter

Delivery

Target Database

GoldenGate Instantiation
Current source Commit Sequence Number (CSN) is 356

Start delivery at or after given backup CSN (245 in this case)

Once Delivery is current the source and target are in sync

Consistent as of CSN 356 In Sync!

Current CSN is 356

CSNs 222 through 356

Commit Ordered Source Trail

Deliver transactions after CSN 245

Source Database

Capture

Pump

Commit Ordered Target Trail

Delivery

Target Database

Take Backup as of CSN 245

Apply Backup as of CSN 245

Initial Initial data Load load1 Create the Data extract process
ADD EXTRACT load1, SOURCEISTABLE Create Parameter file for Load1: Edit params load1

Source:

Target: Create initial data load task load2

ADD REPLICAT load2, SPECIALRUN Create Parameter file for Load2: Edit params load2

Start Initial Load:

START EXTRACT load1 INFO EXTRACT load1

Online Change Synchronization

1.

Create a GoldenGate Checkpoint table

EDIT PARAMS ./GLOBALS

GGSCHEMA GGS_OWNER CHECKPOINTTABLE GGS_OWNER.CHKPTAB

DBLOGIN USERID ggs_owner, PASSWORD ggs_owner ADD CHECKPOINTTABLE GGS_OWNER.CHKPTAB

1.

Create an Extract groups on Source

ADD EXTRACT ext1, TRANLOG, BEGIN NOW ADD EXTRACT dpump, EXTTRAILSOURCE /ora/app/stage/GoldenGate-Zip/ggs/dirdat/lt

1.

Create a parameter file for the online Extract group

EDIT PARAMS ext1

1.

Create a Trail on Source & Target

On the source/target system, the trail files will be created having a prefix rt/lt which will be used by the Replicat/Extract processes also running on the target /source system(s) ADD EXTTRAIL /ora/app/stage/GoldenGate-Zip/ggs/dirdat/lt, EXTRACT EXT1 ADD RMTTRAIL /ora/app/stage/GoldenGate-Zip/ggs/dirdat/rt, EXTRACT dpump

1.

Create a Replicat group on the Target

ADD REPLICAT rep1, exttrail /ora/app/stage/GoldenGate-Zip/ggs/dirdat/rt, checkpointtable chkptab

Significance of Data Pump

When Data Pump is not used, the Extract process writes to a remote trail that is located on the target system using TCP/IP When Data Pump is configured, Extract process writes to a local trail for Data Pump to read and write over the network to the remote trail located on the target system The advantages of Data Pump can be seen as it protects against a network failure as in the absence of a storage device on the local system, the Extract process writes data into memory before the same is sent over the network Any failures in the network could then cause the Extract process to abort (abend) Additionally, if there are any complex data transformations or filtering, Data Pump can be used to perform these