BW Basic Architecture

SAP Product Line BI

Klaus Majenz

Overview
complete DW & BI product, comprising ...
ETL tools (extractors, transformation, monitoring, scheduling, ...) OLAP engine data mining engine repository analytical front-end (web- or Excel-based, agents, GIS, ...) prepacked models, built by SAP application departments

client-server architecture
SAP web application servers database server: 7 commercial RDBMS platforms supported (Oracle, MS, 4IBM, SAP)

part of SAP Netweaver

SAP's open integration and application platform more details: http://www.sap.com/solutions/netweaver/

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 2

Overview

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 3

Scenario (1)

Year 1998 1998 1998 1998 1998 1998 1998 1998 1998 1998 1998

Month 011998 011998 011998 011998 011998 011998 011998 011998 011998 011998 011998

Day City Region Country Sales Pers. 19980101 BERLIN DE-NORTH DE JIM 19980101 BERLIN DE-NORTH DE MARISA 19980102 BERLIN DE-NORTH DE JACK 19980103 BERLIN DE-NORTH DE MANDY 19980120 MILAN IT-NORTH IT MONICA 19980121 MILAN IT-NORTH IT MONICA 19980122 MILAN IT-NORTH IT BILL 19980123 MILAN IT-NORTH IT JOE 19980124 MILAN IT-NORTH IT JIM 19980124 FRANKFURT DE-SOUTH DE KIM 19980125 FRANKFURT DE-SOUTH DE BILL

Division SOFTWARE SOFTWARE SOFTWARE SOFTWARE SOFTWARE SOFTWARE HARDWARE HARDWARE HARDWARE HARDWARE HARDWARE

Distr. Ch. INTERNET INTERNET AGENT RETAIL AGENT INTERNET AGENT RETAIL AGENT INTERNET INTERNET

Sales Org. AMERICAS AMERICAS EUROPE EUROPE EUROPE EUROPE AMERICAS AMERICAS AMERICAS EUROPE AMERICAS

Product Product Grp. Quantity in PC Profit in $ PAINT OFFICE 730 6763 PAINT OFFICE 390 2614 PAINT OFFICE 780 3866 WIN-OS OS 970 -3734 WIN-OS OS 190 1355 WIN-OS OS 810 7565 PC-3 PC 250 -861 PC-3 PC 40 -44 PC-3 PC 160 -503 PC-3 PC 50 18 PC-3 PC 990 6468

Characteristics

Key Figures

Infoobjects

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 4

Scenario (2)
Dimension Time
Day Month Year

Dimension Product

Product Product Group

Dimension Region
City Region

Key Figures

Quantity (in pieces) Profit (in US$)

Country

Dimension Sales Org

Sales Person Division

Distribution Channel Sales Organization

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 5

An adequate BW Infocube IUSALES

Dimension IUSALEST

Dimension IUSALES3

0CALDAY 0CALMONTH

IUPROD IUPRODGRP

0CALYEAR

Dimension IUSALES1

Key Figures

IUCITY IUREGION IUCOUNTRY IUSALPER

IUQUAN IUPROFIT

Dimension IUSALES2

IUDIV
IUDCHAN IUSALORG

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 6

Data Flow in BW
Aggregate
Initial Fill, Roll-Up

Infocube: E fact table

Compression

Cube Query

Infocube: F fact table

Infocube Upload (from ODS) Infocube Upload (from PSA) ODS Query

BW Query

Operational Data Store (ODS)

ODS Upload ODS Activate

Persistent Staging Area (PSA)

Extraction

V.P. Query

Source System (e.g. R/3, other DB, File, ...)

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 7

V.P. Query

Data Flow in BW what we will look at

Aggregate Infocube
Initial Fill, Roll-Up

Infocube: E fact table

Compression

Cube Query

Infocube: F fact table

Infocube Upload (from PSA)

BW Query

ODS

Operational Data Store (ODS)

ODS Upload ODS Activate

PSA

Persistent Staging Area (PSA)

Extraction

Source System (e.g. R/3, other DB, File, ...)

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 8

PSA
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 9

PSA table
request package (within request) partition no. record no. (within package)

huge number of individual INSERTs no UPDATE SELECT * FROM WHERE "REQUEST" = mass deleteion: DELETE WHERE "PARTNO" = / DROP PARTITION

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 10

ODS
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 11

ODS object = 3 tables

active data : /BIC/AOIUSALES00 modified data ("activation queue") : /BIC/AOIUSALES40 delta data ("change log") : /BIC/B0008215000 (PSA)

1.

ODS upload: INSERT INTO "/BIC/AOIUSALES40" ODS data activation: UPSERT "/BIC/AOIUSALES00" delta records: INSERT INTO "/BIC/B0008215000" (mass) DELETE FROM "/BIC/AOIUSALES40" infocube delta upload from ODS: SELECT * FROM "/BIC/B0008215000"

2.

3.

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 12

ODS tables: /BIC/AOIUSALES00, /BIC/AOIUSALES40

active data

same as in PSA table

modified data
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 13

ODS Object (BW 3.0)

Active data
Doc.No I Value 4711 I 10 4711 I 30

Change log
Req.ID I Pack.ID I Rec.No ODSRx I P 1 I Rec.1I4711I 10 ODSRy I P 1 I Rec.1I4711I-10 ODSRy I P 1 I Rec.2I4711I+30

Upload to Activation queue

Data

from different requests are uploaded in parallel to the activation queue activation the data is sorted by the key fields of active data plus key fields of Activation queue. guaranties the correct sequence of the records and allows inserts instead of table locks .

Activation Activation queue

REQU1 I P 1 I Rec.1I4711I 10 REQU2 I P 1 I Rec.1I4711I 30

Activation
During This

Req1 Req2 Req3 Staging Engine

Before-

and After Image

Request After

ID in activation queue and change log differ from each other. update, data in the activation queue is deleted.

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 14

Infocube
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 15

InfoCube: Star Schema

(1) Fact Table (2) Dimension (3) time-independent-SID time-dependent-SID master SID Char S (4) SID Attr

F, E

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 16

Infocube IUSALES
Facttable

Dimension 1

X (City)

S (Population)

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 17

Infocube Indexing (1) Oracle

line item dimension
Facttable

Dimension 1

S (Population) X (City)

Bitmap Index B-Tree (unique) B-Tree (non-unique)

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 18

Infocube Indexing (2) MS SQL Server

line item dimension
Facttable

Dimension 1

S (Population) X (City)
B-tree Index (nonunique, nonclustered) B-Tree (unique, clustered)

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 19

Infocube Indexing (3) Oracle

F Facttable

E Facttable

partitioning column (for E facttable)

"P-index"

single column indexes support queries P-index: compress additional bitmap index on part. column
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 20

Bitmap Index B-Tree (unique) B-Tree (non-unique)

Infocube Indexing (4) MS SQL Server

F Facttable

Does not exist on MS-SQL

E Facttable

"P-index"
B-tree Index (nonunique, nonclustered)
B-Tree (unique, nonclustered)

single column indexes support queries P-index: compress

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 21

Infocube Operations (1)

INSERT: only F facttable
array INSERT if array INSERT fails: UPSERT logic

DELETE request (mass deletion): only F facttable

DELETE FROM "/BIC/FIUSALES" WHERE KEY_IUSALESP = alternatively: DROP PARTITION

DELETE specified data

DELETE FROM WHERE

UPSERT: only E facttable

infocube compression (separate slide)

SELECT
separate slide

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 22

Infocube Compression

(ex.: request 3)

UPDATE INSERT

before after

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 23

Infocube Compression (2)

Oracle (via stored procedure; on DB server)
loop over rows for request REQ in F facttable
attempt if

UPDATE of E facttable

UPDATE fails then INSERT rowid into temporary table INS

do mass INSERT INTO E facttable using INS DROP PARTITION corresponding to REQ in F facttable

MS SQL-Server (via ABAP; via application server)

loop over rows for request REQ in F facttable
attempt if

UPDATE of E facttable

UPDATE fails then attempt INSERT

DELETE FROM F facttable WHERE requestid = REQ

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 24

Aggregate Fill
INSERT INTO [/BIC/E100010] SELECT [D1].[SID_IUCITY] AS [KEY_1000101], [D2].[SID_IUSALPER] AS [KEY_1000102], 0 AS [KEY_100010P], SUM ([F].[/BIC/IUPROFIT]), SUM ([F].[/BIC/IUQUAN]), COUNT(*) AS [FACTCOUNT] FROM [/BIC/FIUSALES] [F], [/BIC/DIUSALES1] [D1], [/BIC/DIUSALES2] [D2], [/BIC/DIUSALESP] [DP] WHERE [F].[KEY_IUSALES1] = [D1].[DIMID] AND [F].[KEY_IUSALES2] = [D2].[DIMID] AND [F].[KEY_IUSALESP] = [DP].[DIMID] AND [DP].[SID_0CHNGID] = 0 AND ( [F].[KEY_IUSALESP] = 0 OR [F].[KEY_IUSALESP] = 2 ) AND [DP].[SID_0REQUID] BETWEEN 0 AND 40 GROUP BY [D1].[SID_IUCITY], [D2].[SID_IUSALPER]

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 25

Aggregate Roll-Up
INSERT INTO [/BIC/F100011] SELECT [D1].[SID_IUCITY] AS [KEY_1000111], [D3].[SID_IUPROD] AS [KEY_1000112], 7 AS [KEY_100011P], SUM ([F].[/BIC/IUPROFIT]), SUM ([F].[/BIC/IUQUAN]), COUNT(*) AS [FACTCOUNT] FROM [/BIC/FIUSALES] [F], [/BIC/DIUSALES1] [D1], [/BIC/DIUSALES3] [D3], [/BIC/DIUSALESP] [DP] WHERE [F].[KEY_IUSALES1] = [D1].[DIMID] AND [F].[KEY_IUSALES3] = [D3].[DIMID] AND [F].[KEY_IUSALESP] = [DP].[DIMID] AND [DP].[SID_0CHNGID] = 0 AND [F].[KEY_IUSALESP] = 5 AND [DP].[SID_0REQUID] = 498 GROUP BY [D1].[SID_IUCITY], [D3].[SID_IUPROD]

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 26

Infocube Query Example: Infocube IUSALES

city region country

(1) Fact Table (2) Dimensions (3) Characteristics (simplified)

day

sales person division

month
distribution channel year sales organization

product
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 27

product group

Query Example & Processing (under Oracle)

region country = 'US'

(1) Fact Table (2) Dimensions (3) Characteristics (simplified)

month

year = [98-99]

product group
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 28

Step 1: Restrictions Master Data Dimensions

region country = 'US'

(1) Fact Table (2) Dimensions (3) Characteristics (simplified)

month

year = [98-99]

product group
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 29

Typical Query Processing

Step 2: Restrictions Dimensions Fact Table

(1) Fact Table (2) Dimensions (3) Characteristics (simplified)
bitmap index bitmap index

product group
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 30

Typical Query Processing

Step 3: Assemble Result

region country = 'US'

(1) Fact Table (2) Dimensions (3) Characteristics (simplified)

month

small subset of facttable

year = [98-99]

product group
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 31

Typical Query Processing

Query Example (1) simple

SELECT "DT"."SID_0CALMONTH" AS "S____081" ,"DT"."SID_0CALYEAR" AS "S____083" ,"D1"."SID_IUCOUNTRY" AS "S____520" ,"D3"."SID_IUPRODGRP" AS "S____524" , COUNT( * ) AS "1ROWCOUNT" , SUM ( "F"."/BIC/IUPROFIT" ) AS "IUPROFIT" , SUM ( "F"."/BIC/IUQUAN" ) AS "IUQUAN" FROM "/BIC/FIUSALES" "F" , "/BIC/DIUSALEST" "DT" , "/BIC/DIUSALES1" "D1" , "/BIC/DIUSALES3" "D3" , "/BIC/DIUSALESP" "DP" WHERE "F"."KEY_IUSALEST" = "DT"."DIMID" AND "F"."KEY_IUSALES1" = "D1"."DIMID" AND "F"."KEY_IUSALES3" = "D3"."DIMID" AND "F"."KEY_IUSALESP" = "DP"."DIMID" AND ( "DT"."SID_0CALMONTH" = 200007 AND "DT"."SID_0CALYEAR" = 2000 AND "DP"."SID_0REQUID" <= 745 ) GROUP BY "DT"."SID_0CALMONTH", "DT"."SID_0CALYEAR", "D1"."SID_IUCOUNTRY", "D3"."SID_IUPRODGRP"

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 32

Query Example (2) navigational attribute

SELECT "DT"."SID_0CALMONTH" AS "S____081" ,"DT"."SID_0CALYEAR" AS "S____083" ,"D1"."SID_IUCOUNTRY" AS "S____520" ,"X1"."S__IUCOLOR" AS "S____530" , COUNT( * ) AS "1ROWCOUNT" , SUM ( "F"."/BIC/IUPROFIT" ) AS "IUPROFIT" , SUM ( "F"."/BIC/IUQUAN" ) AS "IUQUAN" FROM "/BIC/FIUSALES" "F" , "/BIC/DIUSALEST" "DT" , "/BIC/DIUSALES1" "D1" , "/BIC/DIUSALES3" "D3" , "/BIC/XIUPROD" "X1" , "/BIC/DIUSALESP" "DP" WHERE "F"."KEY_IUSALEST" = "DT"."DIMID" AND "F"."KEY_IUSALES1" = "D1"."DIMID" AND "F"."KEY_IUSALES3" = "D3"."DIMID" AND "D3"."SID_IUPROD" = "X1"."SID" AND "F"."KEY_IUSALESP" = "DP"."DIMID" AND ( "DT"."SID_0CALMONTH" = 200007 AND "DT"."SID_0CALYEAR" = 2000 "DP"."SID_0REQUID" <= 745 AND "X1"."OBJVERS" = 'A' ) GROUP BY "DT"."SID_0CALMONTH", "DT"."SID_0CALYEAR", "D1"."SID_IUCOUNTRY", "X1"."S__IUCOLOR"

AND

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 33

Query Example (3) external hierarchy

SELECT "DT"."SID_0CALYEAR" AS "S____083" ,"DT"."SID_0CALMONTH" AS "S____081" ,"D1"."SID_IUCOUNTRY" AS "S____520" ,"H1"."PRED" AS "S____524" , COUNT( * ) AS "1ROWCOUNT" , SUM ( "F"."/BIC/IUPROFIT" ) AS "IUPROFIT" , SUM ( "F"."/BIC/IUQUAN" ) AS "IUQUAN" FROM "/BIC/FIUSALES" "F" , "/BIC/DIUSALES3" "D3" , "/BIC/DIUSALEST" "DT" , "/BIC/DIUSALES1" "D1" , "/BIC/DIUSALESP" "DP" , "/BI0/0300148611" "H1" /* This is a (UNION) view! */ WHERE "F"."KEY_IUSALES3" = "D3"."DIMID" AND "F"."KEY_IUSALEST" = "DT"."DIMID" AND "F"."KEY_IUSALES1" = "D1"."DIMID" AND "F"."KEY_IUSALESP" = "DP"."DIMID" AND "D3"."SID_IUPRODGRP" = "H1"."SUCC" AND ( "DT"."SID_0CALYEAR" = 2000 AND "DP"."SID_0REQUID" <= 745 AND "H1"."SUCC" <> 2000008999 ) GROUP BY "H1"."PRED", "DT"."SID_0CALYEAR", "DT"."SID_0CALMONTH", "D1"."SID_IUCOUNTRY"

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 34

Examples of Conceptual Modeling in SAP BW

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 35

Examples
Reveal why pure RDBMS technology ... sometimes requires an additional conceptual layer on top, is not sufficient is some cases, has no chance in some situations because it has to be more general than necessary. Examples

example 1: infoproviders in SAP BW

uniform

view on differing physical layouts

example 2: non-cumulative key figures in SAP BW

semantic

relationship between table columns

example 3: aggregates in SAP BW

could

be implemented by using materialized views (or equivalent) but they have proved to be inferior
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 36

Example 1: Infoprovider (1)

An infoprovider in SAP BW ... comprises a reporting scenario, is the entity on which a query is defined, combines (aggregated or non-aggregated) operational data with master data (e.g. product, customer, ... data), or constitutes a master data entity

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 37

Example 1: Infoprovider (2) Examples

Example A: a cube is an infoprovider fact table holds operational data on certain granularity dimensions hold master data

Example B: customer master data can be an infoprovider same UI as for other infoproviders

selections, projections, summaries using attributes (e.g. address, customer category, region, ...)

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 38

Example 1: Infoprovider (3) -- Overview (SAP BW 3.x) Infoprovider

Infocube

ODS-Object

Characteristic (master data)

Multiprovider

Infoset

Virtual Infoprovider API e.g. remote access

UNION

JOIN

multi-dim. analytical sales cube

flat operational POS data

flat master data product data real time

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 39

Example 2: Non-Cumulative Key Figures (1)

also: "semi-additive measures" example: account balance

conceptually:
account A B A B A B A B A B A B day 29-Sep 29-Sep 30-Sep 30-Sep 1-Oct 1-Oct 2-Oct 2-Oct 3-Oct 3-Oct 4-Oct 4-Oct balance 100 500 110 500 110 400 110 400 110 400 50 400 - 60 - 100 delta 10

physically:
account A B A A B day 29-Sep 30-Sep 3-Oct 4-Oct 4-Oct ref point no no no yes yes delta 10 - 100 - 60 50 400

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 40

Example 2: Non-Cumulative Key Figures (2)

non-cumulative key figures / semi-additive measures
balance can be reconstructed for any moment in the past that information has not to be physically stored advantages
significantly better more

reduced data volumes

performance

flexibility

however: algorithms are required for

reconstruction
insertion

read

load

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 41

Example 3: Aggregates in SAP BW

SAP BW constraints: only SUM, MIN, MAX aggregations are materialized uploaded data (in an infocube) can still be identified delta roll-ups are simple

Materialized or Indexed Views / Automatic Summary Tables could be used in theory however: maintenance is considerably slower

... due to expensive tracking and logging mechanisms that are necessary if the general case has to be covered

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 42

Summary

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 43

Summary
brief introduction to SAP BW three examples:
1. an additional conceptual layer on top of the relational one

2. a semantical pattern that is frequently used in business 3. an object that might suffer from the generic approach

Do the examples reveal shortcomings of RDBMS or are they application domain specific ?

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 44