Bringing Oracle Big Data SQL To OBIEE 11g and ODI12c

Bringing Oracle Big Data SQL
to OBIEE 11g and ODI12c

Mark Rittman, CTO, Rittman Mead
BIWA Summit 2015, January 2015, San Francisco
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or
+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : info@rittmanmead.com
W : www.rittmanmead.com
About the Speaker

Mark Rittman, Co-Founder of Rittman Mead
Oracle ACE Director, specialising in Oracle BI&DW
14 Years Experience with Oracle Technology
Regular columnist for Oracle Magazine
Author of two Oracle Press Oracle BI books
Oracle Business Intelligence Developers Guide
Oracle Exalytics Revealed
Writer for Rittman Mead Blog :
http://www.rittmanmead.com/blog
Email : mark.rittman@rittmanmead.com
Twitter : @markrittman
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

About Rittman Mead

Oracle BI and DW Gold partner
Winner of five UKOUG Partner of the Year awards in 2013 - including BI
World leading specialist partner for technical excellence,
solutions delivery and innovation in Oracle BI
Approximately 80 consultants worldwide
All expert in Oracle BI and DW
Offices in US (Atlanta), Europe, Australia and India
Skills in broad range of supporting Oracle tools:
OBIEE, OBIA
ODIEE
Essbase, Oracle OLAP
GoldenGate
Endeca
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or
Introducing Hadoop
A new approach to data processing and data storage
Rather than a small number of large, powerful servers, it spreads processing over
large numbers of small, cheap, redundant servers
Spreads the data youre processing over
lots of distributed nodes
Job Tracker
Has scheduling/workload process that sends
parts of a job to each of the nodes
- a bit like Oracle Parallel Execution
And does the processing where the data sits
Task Tracker
Task Tracker
Task Tracker
Task Tracker
- a bit like Exadata storage servers
Shared-nothing architecture
Low-cost and highly horizontal scalable
Data Node
Task Tracker
Data Node
Task Tracker
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Apache Hive : SQL Access + Table Metadata Over HDFS

Apache Hive provides a SQL layer over Hadoop, once we understand the structure (schema)
of the data were working with
Exposes HDFS and other Hadoop data as tables and columns
Map
Provides a simple SQL dialect for queries called HiveQL
Task
Map
Task
SQL queries are turned into MapReduce jobs under-the-covers
JDBC and ODBC drivers provide
Map
Task
SELECT a, sum(b)
access to BI and ETL tools
FROM myTable
Hive metastore (data dictionary)
WHERE a<100
leveraged by many other Hadoop tools
Reduce
Reduce
Task
Apache Pig
GROUP BY a
Task
Cloudera Impala
etc
Result
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Hive SerDes & Storage Handlers

Plug-in technologies that extend Hive to handle new data formats and semi-structured sources
Typically distributed as JAR files, hosted on sites such as GitHub
Can be used to parse log files, access data in NoSQL databases, Amazon S3 etc
CREATE EXTERNAL TABLE apachelog (
host STRING,
identity STRING,
user STRING,
time STRING,
request STRING,
status STRING,
size STRING,
referer STRING,
agent STRING)
ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
"input.regex" = "([^ ]*) ([^ ]*) ([^ ]*) (-|\\[[^\\]]*\\])
([^ \"]*|\"[^\"]*\") (-|[0-9]*) (-|[0-9]*)(?: ([^ \"]*|\"[^\"]*\")
([^ \"]*|\"[^\"]*\"))?",
"output.format.string" = "%1$s %2$s %3$s %4$s %5$s %6$s %7$s %8$s %9$s"
)
STORED AS TEXTFILE
LOCATION '/user/root/logs';
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

CREATE TABLE tweet_data(

interactionId string,
username string,
content string,
author_followers int)
ROW FORMAT SERDE
'com.mongodb.hadoop.hive.BSONSerDe'
STORED BY
'com.mongodb.hadoop.hive.MongoStorageHandler'
WITH SERDEPROPERTIES (
'mongo.columns.mapping'='{"interactionId":"interactionId",
"username":"interaction.interaction.author.username",
"content":\"interaction.interaction.content",
"author_followers_count":"interaction.twitter.user.followers_count"}'
)
TBLPROPERTIES (
'mongo.uri'='mongodb://cdh51-node1:27017/datasiftmongodb.rm_tweets'
)
Oracles Big Data Products

Oracle Big Data Appliance
Optimized hardware for Hadoop processing
Cloudera Distribution incl. Hadoop
Oracle Big Data Connectors, ODI etc
Oracle Big Data Connectors
Oracle NoSQL Database
Oracle Data Integrator
Oracle R Distribution
OBIEE, BI Publisher and
Endeca Info Discovery
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Oracle Big Data Connectors

Oracle-licensed utilities to connect Hadoop to Oracle RBDMS
Bulk-extract data from Hadoop to Oracle, or expose HDFS / Hive data as external tables
Run R analysis and processing on Hadoop
Leverage Hadoop compute resources to offload ETL and other work from Oracle RBDMS
Enable Oracle SQL to access and load Hadoop data
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Oracle Business Analytics and Big Data Sources

OBIEE 11g can also make use of big data sources
OBIEE 11.1.1.7+ supports Hive/Hadoop as a data source, via specific Hive ODBC drivers
and Apache Hive Physical Layer database type
But practically, it comes with limitations
Current 11.1.1.7 version of OBIEE only ships with HiveServer1 ODBC drivers
HiveQL is a limited subset of ISO/Oracle SQL
and Hive access is really slow
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Oracle Big Data SQL

Part of Oracle Big Data 4.0 (BDA-only)
Also requires Oracle Database 12c, Oracle Exadata Database Machine
Extends Oracle Data Dictionary to cover Hive
Extends Oracle SQL and SmartScan to Hadoop
SQL Queries
Extends Oracle Security Model over Hadoop
Fine-grained access control
Data redaction, data masking
Exadata Database
Server
Uses fast c-based readers where possible
(vs. Hive MapReduce generation)
SmartScan
Map Hadoop parallelism to Oracle PQ
Exadata
Hadoop
Big Data SQL engine works on top of YARN
Storage Servers
Cluster
Like Spark, Tez, MR2
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

SmartScan
Oracle Big
Data SQL
View Hive Table Metadata in the Oracle Data Dictionary

Oracle Database 12c 12.1.0.2.0 with Big Data SQL option can view Hive table metadata
Linked by Exadata configuration steps to one or more BDA clusters
DBA_HIVE_TABLES and USER_HIVE_TABLES exposes Hive metadata
Oracle SQL*Developer 4.0.3, with Cloudera Hive drivers, can connect to Hive metastore
SQL>
SQL>
SQL>
2
col database_name for a30

col table_name for a30
select database_name, table_name
from dba_hive_tables;
DATABASE_NAME
-----------------------------default
default
default
default
default
default
default
default
TABLE_NAME
-----------------------------access_per_post
access_per_post_categories
access_per_post_full
apachelog
categories
countries
cust
hive_raw_apache_access_log
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Big Data SQL Server Dataflow

Direct-path reads
C-based readers when possible
Use native Hadoop classes otherwise
3%
External$Table$Services$
3 Apply SmartScan to Oracle bytes
10110010%
Smart$Scan$
2 Translate bytes to Oracle
2%
RecordReader%
Data$Node$
Disks%
10110010%
SerDe%
Apply filters
Project columns
Parse JSON/XML
Score models
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Big$Data$SQL$Server$
10110010%
1 Read data from HDFS Data Node
1%
Hive Access through Oracle External Tables + Hive Driver

Big Data SQL accesses Hive tables through external table mechanism
ORACLE_HIVE external table type imports Hive metastore metadata
ORACLE_HDFS requires metadata to be specified
Access parameters cluster and tablename specify Hive table source and BDA cluster
CREATE TABLE access_per_post_categories(
hostname varchar2(100),
request_date varchar2(100),
post_id varchar2(10),
title varchar2(200),
author varchar2(100),
category varchar2(100),
ip_integer number)
organization external
(type oracle_hive
default directory default_dir
access parameters(com.oracle.bigdata.tablename=default.access_per_post_categories));
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Use Rich Oracle SQL Dialect over Hadoop (Hive) Data

Ranking Functions
rank, dense_rank, cume_dist,
percent_rank, ntile
Window Aggregate Functions
Avg, sum, min, max, count, variance,
first_value, last_value
LAG/LEAD Functions
Reporting Aggregate Functions
Sum, Avg, ratio_to_report
Statistical Aggregates
Correlation, linear regression family,
covariance
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Linear Regression
Fitting of ordinary-least-squares
regression line to set of number pairs
Descriptive Statistics
Correlations
Pearsons correlation coefficients
Crosstabs
Chi squared, phi coefficinet
Hypothesis Testing
Student t-test, Bionomal test
Distribution
Anderson-Darling test - etc.
Leverages Hive Metastore for Hadoop Java Access Classes

As with other next-gen SQL access layers, uses common Hive metastore table metadata
Provides route to underlying Hadoop data for Oracle Big Data SQL c-based SmartScan
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Extending SmartScan, and Oracle SQL, Across All Data

Brings query-offloading features of Exadata
to Oracle Big Data Appliance
Query across both Oracle and Hadoop sources
Intelligent query optimisation applies SmartScan
close to ALL data
Use same SQL dialect across both sources
Apply same security rules, policies,
user access rights across both sources
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Example : Using Big Data SQL to Add Dimensions to Hive Data

We want to add country and post details to a Hive table containing page accesses
Post and Country details are stored in Oracle RBDMS reference tables
Hive Weblog Activity table
Oracle Dimension lookup tables
Combined output
in report form
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Create ORACLE_HIVE External Table over Hive Table

Use the ORACLE_HIVE access driver type to create Oracle external table over Hive table
ACCESS_PER_POST_EXTTAB now appears in Oracle data dictionary
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Import Oracle Tables, Create RPD joining Tables Together

No need to use Hive ODBC drivers - Oracle OCI connection instead
No issue around HiveServer1 vs HiveServer2; also Big Data SQL handles authentication
with Hadoop cluster in background, Kerberos etc
Transparent to OBIEE - all appear as Oracle tables
Join across schemas if required
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Create Physical Data Model from Imported Table Metadata

Join ORACLE_HIVE external table containing log data, to reference tables from Oracle DB
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Create Business Model and Presentation Layers

Map incoming physical tables into a star schema
Add aggregation method for fact measures
Add logical keys for logical dimension tables
Remove columns from fact table that arent measures
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Create Initial Analyses Against Combined Dataset

Create analyses using
full SQL features
Access to Oracle RDBMS
Advanced Analytics functions
through EVALUATE,
EVALUATE_AGGR etc
Big Data SQL SmartScan feature
provides fast, ad-hoc access
to Hive data, avoiding MapReduce
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Oracle / Hive Query Federation at the RDBMS Level

Oracle Big Data SQL feature (not BI Server) takes care of query federation
SQL required for fact table (web log activity) access sent to Big Data SQL agent on BDA
Only columns (projection) and rows (filtering) required to answer query sent back to Exadata
Storage Indexes used on both Exadata Storage Servers and BDA nodes to skip block reads
for irrelevant data
HDFS caching used to speed-up
access to commonly-used
HDFS data
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Access to Full Set of Oracle Join Types

No longer restricted to HiveQL equi-joins - Big Data SQL supports all Oracle join operators
Use to join Hive data (using View over external table) to a IP range country lookup table
using BETWEEN join operator
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Add In Time Dimension Table

Enables time-series reporting; pre-req for forecasting (linear regression-type queries)
Map to Date field in view over ORACLE_HIVE table
Convert incoming Hive STRING field to Oracle DATE for better time-series manipulation
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Now Enables Time-Series Reporting Incl. Country Lookups
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Oracle Data Integrator 12c

Oracles data integration tool for loading, transforming and integrating enterprise data
Successor to Oracle Warehouse Builder, part of wider Oracle DI platform
Connectivity to most RBDMS, file and application sources
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Part of the Wider Oracle Data Integration Platform

Oracle Data Integrator for large-scale data integration across heterogenous sources and
targets
Oracle GoldenGate for heterogeneous data replication and changed data capture
Oracle Enterprise Data Quality for data profiling and cleansing
Oracle Data Services Integrator
for SOA message-based
data federation
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Oracle Data Integrator on Hadoop

ODI provides an excellent framework for running Hadoop ETL jobs
ELT approach pushes transformations down to Hadoop - leveraging power of cluster
Hive, HBase, Sqoop and OLH/ODCH KMs provide native Hadoop loading / transformation
Whilst still preserving RDBMS push-down
Extensible to cover Pig, Spark etc
Process orchestration
Data quality / error handling
Metadata and model-driven
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Combining Oracle and Hadoop (Hive) Data in Mappings

Example scenario : log data in Hadoop needs to be enriched with customer data in Oracle
Hadoop (Hive) contains log activity and customer etc IDs
Reference / customer data held in Oracle RBDMS
How do we create a mapping that joins both datasets?
movieapp_log_odistage.custid = CUSTOMER.CUSTID
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Options for Importing Oracle / RDBMS Data into Hadoop

Could export RBDMS data to file, and load using
IKM File to Hive
Oracle Big Data Connectors only export to Oracle,
not import to Hadoop
One option is to use Apache Sqoop, and new
IKM SQL to Hive-HBase-File knowledge module
Hadoop-native, automatically runs in parallel
Uses native JDBC drivers, or OraOop (for
example)
Bi-directional in-and-out of Hadoop to RDBMS
Join performed in Hive, using HiveQL
With HiveQL limitations (only equi-joins)
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Sqoop extract
movieapp_log_odistage.custid =
customer.custid
New Option - Using Oracle Big Data SQL

Oracle Big Data SQL provides ability for Exadata to reference Hive tables
Use feature to create join in Oracle, bringing across Hive data through ORACLE_HIVE table
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Oracle Big Data SQL and Data Integration

Gives us the ability to easily bring in Hadoop (Hive) data into Oracle-based mappings
Allows us to create Hive-based mappings that use Oracle SQL for transforms, joins
Faster access to Hive data for real-time ETL scenarios
Through Hive, bring NoSQL and semi-structured data access to Oracle ETL projects
For our scenario - join weblog + customer data in Oracle RDBMS, no need to stage in Hive
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Using Big Data SQL in an ODI12c Mapping

By default, Hive table has to be exposed as an ORACLE_HIVE external table in Oracle first
Then register that Oracle external table in ODI repository + model
3
External table creation in Oracle
2
Register in ODI Model
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Logical Mapping using just Oracle tables
Custom KM : LKM Hive to Oracle (Big Data SQL)

ODI12c Big Data SQL example on BigDataLite VM uses a custom KM for Big Data SQL
LKM Hive to Oracle (Big Data SQL) - KM code downloadable from java.net
Allows Hive+Oracle joins by auto-creating ORACLE_HIVE extttab
definition to enable Big Data SQL Hive table access
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

ODI12c Mapping Creates Temp Exttab, Joins to Oracle

Big Data SQL Hive External Table created as temp object
1
Register in ODI Model
Main integration SQL routines uses regular Oracle SQL join
4
Hive table AP uses LKM Hive to Oracle (Big Data SQL)
2
IKM Oracle Insert
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Summary
Oracle Big Data SQL extends Exadata capabilities over Hadoop and Hive
Makes Hive Metastore visible through Oracle Data Dictionary
Register Hive tables as ORACLE_HIVE external tables and include in Oracle SQL queries
Used with OBIEE, allows RPDs to be created across both Oracle + Hive data, with
query federation handled by Oracle RBDMS rather than BI Server
Enables use of Oracle Advanced Analytics functions over Hadoop data
Useful for ODI as way of using full set of join operators on Hive data, and simplifying
the addition of Hive data to Oracle mappings
For developers working with Exadata + BDA, useful addition to the data access toolkit
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Thank You for Attending!

Thank you for attending this presentation, and more information can be found at http://
www.rittmanmead.com
Contact us at info@rittmanmead.com or mark.rittman@rittmanmead.com
Look out for our book, Oracle Business Intelligence Developers Guide out now!
Follow-us on Twitter (@rittmanmead) or Facebook (facebook.com/rittmanmead)
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Bringing Oracle Big Data SQL

to OBIEE 11g and ODI12c
Mark Rittman, CTO, Rittman Mead
BIWA Summit 2015, January 2015, San Francisco
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Bringing Oracle Big Data SQL To OBIEE 11g and ODI12c

Загружено:

Сведения о документе

Исходное описание:

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Bringing Oracle Big Data SQL To OBIEE 11g and ODI12c

Загружено:

Авторское право:

Доступные форматы

Bringing Oracle Big Data SQL

to OBIEE 11g and ODI12c

About the Speaker

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

About Rittman Mead

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Apache Hive : SQL Access + Table Metadata Over HDFS

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Hive SerDes & Storage Handlers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

CREATE TABLE tweet_data(

Oracles Big Data Products

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Oracle Big Data Connectors

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Oracle Business Analytics and Big Data Sources

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Oracle Big Data SQL

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

View Hive Table Metadata in the Oracle Data Dictionary

col database_name for a30

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Big Data SQL Server Dataflow

3 Apply SmartScan to Oracle bytes

2 Translate bytes to Oracle

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

1 Read data from HDFS Data Node

Hive Access through Oracle External Tables + Hive Driver

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Use Rich Oracle SQL Dialect over Hadoop (Hive) Data

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Leverages Hive Metastore for Hadoop Java Access Classes

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Extending SmartScan, and Oracle SQL, Across All Data

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Example : Using Big Data SQL to Add Dimensions to Hive Data

Hive Weblog Activity table

Oracle Dimension lookup tables

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Create ORACLE_HIVE External Table over Hive Table

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Import Oracle Tables, Create RPD joining Tables Together

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Create Physical Data Model from Imported Table Metadata

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Create Business Model and Presentation Layers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Create Initial Analyses Against Combined Dataset

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Oracle / Hive Query Federation at the RDBMS Level

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Access to Full Set of Oracle Join Types

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Add In Time Dimension Table

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Now Enables Time-Series Reporting Incl. Country Lookups

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Oracle Data Integrator 12c

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Part of the Wider Oracle Data Integration Platform

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or

Oracle Data Integrator on Hadoop