Understanding Oracle RAC

(12.1.0.2) Internals:
The Cache Fusion Edition

Markus Michalewicz
Director of Product Management
Oracle Real Application Clusters (RAC)
October 1st, 2014
@OracleRACpm
http://www.linkedin.com/in/markusmichalewicz
http://www.slideshare.net/MarkusMichalewicz

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 3

The Secret to the Scalability of Any System –
A straight vertical line

Simplification 1 Simplification 2

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 4

Three Alternative Architectures – The Same Idea Applies

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 5

Three Alternative Architectures – The Same Idea Applies

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 6

Program Agenda

1 Cache Fusion Overview

2 (Dynamic Re-) Mastering
3 Handling Contentions
4 RAC Meets Multitenant
5 RAC Meets In-Memory

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Oracle Confidential – Internal/Restricted/Highly Restricted 7
Program Agenda

1 Cache Fusion Overview

2 (Dynamic Re-) Mastering
3 Handling Contentions
4 RAC Meets Multitenant
5 RAC Meets In-Memory

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Oracle Confidential – Internal/Restricted/Highly Restricted 8
Oracle RAC Combines it All & Adds Services
• Requester

• Listener

• Service
Block, unless
otherwise stated.
• Holder
Instance
• Master

• Session PQ/PX:

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 9

Listeners and Services Direct Work to an Instance
• In an Oracle environment, listeners
establish connections with an instance.
• In an Oracle RAC environment two types
of listeners are used (for this purpose):
1. SCAN and SCAN Listener
2. Local Listeners per node
• Services are used to load balance work.
• Local listeners will establish connections
on instances where the service is offered.
• Once a session is established within an
instance, the sessions stays until closed.

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 10

Getting Access to Data

• Local Cache Hit • Remote Cache Hit • Data is on disk

• Access time: nanoseconds • Access time: microseconds • Flash cache access time: microseconds
• Disk controller cache: access time: micros.
Whether local or remote, cache hits are always • Spinning disk access time: milliseconds
faster than reading from spinning disks

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 11

Maximum Three Way Communication
write
Case 1 (all local)
• All entities in one instance
• Data holding instance
• Session “holding” instance
• Mastering Instance
write write
Message Message
Block
Case 2 (local / remote) Case 3 (all distributed)
• Some entities in one instance • All entities are dispersed
•Data holding instance • Three way communication
• Session “holding” instance to obtain a given block of data
• Message
operations have been subject to
• Two way communication improvements since Oracle RAC 10g times.
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 12
Dynamic Retrieval of Data
• In order to fulfill a given request,
+ + Oracle RAC can decide to:
• Assemble data spread across instances and disks
• Perform a (full) disk read for (parts of) the data
• Use assembly and (full) disk reads dynamically
1 2

• The decision which access path to use is

3 dependent on various factors (e.g. IO capacity,
network utilization)
• Respective parameters are monitored
• The access path can change
accordingly and dynamically.

Once data has been shipped to an instance, it resides in

the cache for further (local) access. Updates will be
communicated as required (via messages).

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 13

Program Agenda

1 Cache Fusion Overview

2 (Dynamic Re-) Mastering
3 Handling Contentions
4 RAC Meets Multitenant
5 RAC Meets In-Memory

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Oracle Confidential – Internal/Restricted/Highly Restricted 14
Mastering and Dynamic Re-Mastering
write write

Oracle RAC Oracle RAC Oracle RAC

Oracle GI | HUB Oracle GI | HUB Oracle GI | HUB Oracle RAC Oracle RAC Oracle RAC
germany argentina brazil Oracle RAC Oracle RAC Oracle RAC
Oracle GI | HUB Oracle GI | HUB Oracle GI | HUB Oracle GI | HUB Oracle GI | HUB Oracle GI | HUB
germany argentina brazil germany argentina brazil

• In Oracle RAC, every object is mastered (as
part of Global Cache Services / GCS) by an
instance.
• The Global Resource Directory (GRD) is
used to “keep track” of mastering.
• Illustration above is symbolic for this matter.
• If a session requests write access • Based on user access patterns, a Dynamic
Re-Mastering (DRM) can be triggered,
to a block that is not mastered by changing the master for an object from one
the instance hosting the session, instance to another.
the master instance needs to be • Relocating the master to the instance
where data is requested most reduces the
contacted (via message). messaging.

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 15

Dynamic Re-Mastering (DRM) – Recommendations
write • DRM is used internally and externally.
• Externally means access patterns can
influence the mastering of user data.
• Internally
DRM is used for the management
of UNDO data across instances for example.
• DRM activity is also performed upon
instances leaving or joining the cluster.

• Do not turn DRM off (_gc_policy_time=0)!

• Optimize _gc_policy_minimum to run / trigger DRM
Oracle RAC Oracle RAC Oracle RAC
at a convenient moment under normal operations.
Oracle GI | HUB Oracle GI | HUB Oracle GI | HUB
germany argentina brazil
• Consider using smaller SGA sizes.
• See MOS note 1619155.1 – “Best Practices and
Recommendations for RAC databases using SGA
larger than 300GB” as applicable
• “dbms_cacheutil” can be used to manually set and
release affinity under well understood circumstances.

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 16

New in 12.1.0.2: Cache Fusion Accelerator
write • The Cache Fusion Accelerator (CFA) is an
• OS kernel (Linux & Solaris only) module
• which can respond directly to
certain lock requests via RDSv3.
• The lock state is saved in memory shared
by the database and the kernel.
• CFA saves user/kernel context switches, frees up
CPU cycles in LMS, and “speeds up” messages.
Oracle RAC Oracle RAC Oracle RAC
Oracle GI Oracle GI Oracle GI • CFA will be activated on Engineered Systems over
germany argentina brazil
time, including the Oracle Database Appliance.
Lock grant
• CFA contributes to a better, linear scaling.
• CFA is one of a long list of improvements.

LMS process Accelerator

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 17

New in 12.1.0.2: Cache Fusion Accelerator
write • The Cache Fusion Accelerator (CFA) is an
• OS kernel (Linux & Solaris only) module
• which can respond directly to
certain lock requests via RDSv3.
• The lock state is saved in memory shared
by the database and the kernel.
• CFA saves user/kernel context switches, frees up
CPU cycles in LMS, and “speeds up” messages.
Oracle RAC Oracle RAC Oracle RAC
Oracle GI Oracle GI Oracle GI • CFA will be activated on Engineered Systems over
germany argentina brazil
time, including the Oracle Database Appliance.
• CFA contributes to a better, linear scaling.
• CFA is one of a long list of improvements.

LMS process Accelerator

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 18

Program Agenda

1 Cache Fusion Overview

2 (Dynamic Re-) Mastering
3 Handling Contentions
4 RAC Meets Multitenant
5 RAC Meets In-Memory

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Oracle Confidential – Internal/Restricted/Highly Restricted 19
How To Handle Contention – Basics
write write write write
write
write
write
write write write
write
write write write

• Contention can occur in any multi-
user system (even in SI databases)
• User data as well as “metadata” (e.g.
an index) can be subject to contention
• You can still guide users to only one
instance in an Oracle RAC environment.
• As soon as you scale out, contention
can occur between instances (not only
within an instance).
• From a contention perspective, a 2-node
setup is basically the same as a 3 or more
nodes setup.
• The difference is in the required message-
based communication to obtain a block
considering the mastering.

Note: for scalability, only write/write contention needs to be considered.

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 20
How To Handle Contention – Considerations
write write write
write write write
write write write
write write write

Sequence

REDO

• Frequent transactional changes to the
same data blocks in all instances may
result in “write hot spots”.
• Block with pending changes may
be “pinged” by other instances.
• Pending redo must be written to log • Contention can affect related data as
before the block can be transferred much as it can affect the actual “user data”.
• Right growing indexes and
• Latency for a deferred block transfer index contention are common.
becomes dependent on delay for log IO
• In 99% of OLTP performance issues,
write hot spots occur on indexes.

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 21

How To Handle Contention – General Solutions
write write write
write write write
write write write
write write write

Sequence

REDO

• Avoid frequent transactional changes • Place redo logs on fast storage • Use either for better cache locality:
to the same data blocks in all instances if performance critical; e.g. SSDs
using partitioning and services. •Implemented in 11.2.2.4 of Exadata and Oracle • Global hash partitioned indexes
Database Appliance by default (Smart Logs and
• Logically partition data so that sub- SSDs, respectively) • Locally partitioned indexes
data is handled in one instance only. •Separate disks for logs from other IO busy disks
Guide users via services accordingly. • Drop unused indexes
Get the best out of Oracle Partitioning - A practical guide and reference
https://www.slideshare.net/secret/nHJV1CjWWOlGOl

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 22

How To Handle Contention – Use Connection Pools
write write
write write
Connection Pool Connectionwrite
Pool Connection Pool
write busy idle

• UCP supports “Connection Affinity”:
• Transaction-Based Affinity
• Web Session Affinity
http://docs.oracle.com/database/121/JJUCP/rac.htm#JJUCP8197
• Connection Pools limit the number • Fast Application Notification (FAN)
of connections to the database – example:
– enabled connection pools receive
• Oracle Universal Connection Pool (UCP)
Load Balancing information based on
• Use “Database Resident Connection the Workload Repository and on a
Pooling” when no. of active sessions per-service basis.
much smaller then no. of open session.

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 23

Program Agenda

1 Cache Fusion Overview

2 (Dynamic Re-) Mastering
3 Handling Contentions
4 RAC Meets Multitenant
5 RAC Meets In-Memory

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Oracle Confidential – Internal/Restricted/Highly Restricted 24
Oracle Multitenant and Oracle RAC – a Symbiosis

cons_1 cons_2

Oracle RAC Oracle RAC

Oracle GI | HUB Oracle GI | HUB
germany argentina

REDO

•Pluggable Databases (PDBs) represent themselves as services in an
Oracle RAC Multitenant Database, ensuring extremely fast failover.
• PDBs can be used to conveniently align users to instances with
all the benefits, providing higher consolidation benefit and agility.
• Oracle RAC Multitenant Databases help to consolidate.
• Future improvements will ensure greater efficiency.
• Multithreaded Redo Log writer already implemented.

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 25

Oracle Multitenant and Oracle RAC – a Symbiosis

cons_1 cons_2

Oracle RAC Oracle RAC

Oracle GI | HUB Oracle GI | HUB
germany argentina

REDO

•Pluggable Databases (PDBs) represent themselves as services in an • Oracle RAC Multitenant Databases help to consolidate.
Oracle
MoreRAC Multitenant
information, Database,
see: ensuring extremely
Oracle Multitenant meetsfast failover.
Oracle RAC -
• PDBshttp://www.slideshare.net/MarkusMichalewicz/oracle-
can be used to conveniently align users to instances with • Future improvements will ensure greater efficiency.
multitenant-meets-oracle-rac-ioug-2014-version
all the benefits, providing higher consolidation benefit and agility.
• Multithreaded Redo Log writer already implemented.
• Per-PDB/service optimized GCS operations coming in future.

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 26

Program Agenda

1 Cache Fusion Overview

2 (Dynamic Re-) Mastering
3 Handling Contentions
4 RAC Meets Multitenant
5 RAC Meets In-Memory

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Oracle Confidential – Internal/Restricted/Highly Restricted 27
Oracle In-Memory and Oracle RAC

Breakthrough: Generate
Dual Format Reports
Database Instantly

In-Memory
Column Store
Column
Store Full HA &
Replaces Integration
Analytic with
Indexes Industry
Standards

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 28

Oracle In-Memory and Oracle RAC – A Dream-Team

Breakthrough: Addresses
Dual Format
Database

In-Memory
Column Store
Column
Store
Replaces Addresses
Analytic
Indexes

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 29

“Full support for RAC scale-out
means Oracle Database In-Memory
can be used on our largest Data
Warehouse, enabling more near
real-time analytics.”
– Sudhi Vijayakumar, Senior Oracle DBA
Yahoo Inc.

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 30

Oracle Database In-Memory: Unique Fault Tolerance

• Similar to storage mirroring

• Duplicate in-memory
columns on another node
• Enabled per table/partition
• Application transparent

• Downtime eliminated by
using duplicate after failure
Only Available on Engineered Systems

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 31

“Downtime is extremely costly for our
business. Oracle’s In-Memory
architecture takes the right approach
to balancing real-time speed with
continuous availability.”
– Jens-Christian Pokolm
Analyst IT-DB Architecture & Engineering
Postbank Systems AG

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 32

Optimized Used of Memory in Every Layer
• Compressed
In-Memory
Format • Lock mgmt.
In-Memory
Format

simplified
Normal Normal
Buffer Cache Buffer Cache ? ? ?
Oracle RAC Oracle RAC Oracle RAC
Oracle GI | HUB Oracle GI | HUB Oracle GI | HUB
germany argentina brazil

Coming soon…

• In-Memory compresses data, • PCI Flash works mostly locally.

optimizing the use of memory.
• Memory allocated for In-Memory
is subject to simplified lock mgmt.
• Shared solutions require
• In-Memory Speed and Capacity of
RAC external synchronization.
Low Cost Disk
• Coming soon on ODA: a fully
integrated shared flash cache.

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 33

Oracle Learning Streams: http://education.oracle.com/streams/
• Database and Middleware streams available
• Videos
– Feature different groups and presenters
– Cover a broad range of topics and products

• FREE OF CHARGE Oracle RAC PM / Dev contributions:

– Oracle Flex Cluster: Optimized Resource Management for the Cloud - Ian Cookson
– Oracle Grid Infrastructure 12c Bundled Agents – Shankar Iyer
– ACFS Product Overview and Use Cases - Ara Shakian
– The Oracle Real Application Clusters (RAC) Family of Solutions - A User Guide – Markus Michalewicz
– Next Generation Oracle Automatic Storage Management - Jim Williams
– Implementing DBaaS with Oracle RAC 12c and Quality of Service Management - Mark Scardina
– Oracle RAC Practical Performance Management and Tuning – Markus Michalewicz

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 36