Avoiding Common RAC Problems

Session #301
Dan Norris DanNorris@DanNorris.com

DanNorris.com

This session
You Me Oracle RAC
2

DanNorris.com

DanNorris.com

DanNorris.com

DanNorris.com

DanNorris.com

DanNorris.com

Terminology
Database, Instance Clusterware SAN, Local Storage, Shared Storage Multi-Instance, ASM, Oracle Services
8

DanNorris.com

RAC Review, briey

ONE Database
RAC1

Shared Storage

RAC2

RAC3

DanNorris.com

How can I manage all these users/processes/ connections?

DanNorris.com

10

Services*
Use them! More than one is usually a good idea Dont use the default service for users Track statistics per service: V$SERVICE_STATS V$SERVICE_WAIT_CLASS
* see Jeremy Schneiders paper at ardentperf.com/publications
11

DanNorris.com

Avoid SPOF for Network Interfaces

12

DanNorris.com

Network Conguration
Lack of redundant NICs (bonding, IPMP,
etc.)

Dont overbuy for the interconnect Crossover cables VIPs *must* be used!
13

DanNorris.com

Add redundant NICs

RAC1

Shared Storage

RAC2

RAC3

14

DanNorris.com

Storage Issues - Cong

Check certication (ASM, CFS, etc.) Use correct mount options OCFS2: datavolume NFS has very specic mount options Use direct I/O whenever possible
15

(rw,bg,hard,nointr,tcp,vers=3,ti
meo=300,rsize=32768,wsize=32768, actimeo=0)

DanNorris.com

SAN = Storage Area Network

16

DanNorris.com

Storage Issues Multipathing

Multipathing: use proper software & device
names

Ensure asm_diskstring is set properly

or risk bypassing multipathing

/dev/emcpowerXX is a pseudo device

used with PowerPath, but /dev/sdXX will still work...until a path fails
DanNorris.com

17

Add Storage Multipathing

RAC1

Shared Storage

RAC2

RAC3
DanNorris.com

18

Most failed RAC implementations arent due to technical problems.

19

--Dan Norris

DanNorris.com

 Training required DBAs Storage admins - possibly Developers - maybe not a full course, but
some internal training

People Issues Knowledge

Set expectations properly and realistically

20

Managers

DanNorris.com

People Issues Teamwork

RAC has non-RDBMS requirements: shared
storage, clustering plan, same goal

Everyone has to be on board: same project

21

DanNorris.com

Testing - Failure Tests

Once in a lifetime opportunity: break stuff! single NIC, two NICs, storage access,
pub/priv network, aborts, crashes

Plan each test, predict outcome, verify Dont stop until you know what will happen If building N+1, test for that, too
22

DanNorris.com

Our 3-node cluster runs at 80% busy on each node. Highly available?
23

DanNorris.com

Testing - Stress Tests

These are hard to perform well (realistic) Often expensive, but... Probably most important testing to do Consider Database Replay Employ Resource Manager if necessary
24

DanNorris.com

Testing - Connectivity
Connectivity testing is part of other tests PRECONNECT, BASIC; SESSION,
SELECT

Observe load balancing (client & server) Test services, workload management Ensure remote_listener,
25

local_listener set properly; use VIPs

DanNorris.com

Listener Conguration
What does local_listener do? What does remote_listener do? Which one is more important or will cause
bigger problems if you mess it up?

26

DanNorris.com

inst1.local_listener=(ADDRESS=(PROTOCOL=TC P)(HOST=n1.dannorris.com)(PORT=1521)) inst1.local_listener=(ADDRESS=(PROTOCOL=TC P)(HOST=n1-vip)(PORT=1521)) inst1.local_listener=(ADDRESS=(PROTOCOL=TC P)(HOST=n1-vip.dannorris.com)(PORT=1521)) inst1.local_listener=(ADDRESS=(PROTOCOL=TC P)(HOST=52.32.15.60)(PORT=1521)) # n1-vip inst1.local_listener=(ADDRESS=(PROTOCOL=TC P)(HOST=localhost)(PORT=1521)) inst1.local_listener=LISTENER_N1
27

DanNorris.com

App Testing - Txn Mgt

Conn Pools + OO Dev can = RAC prob Perform all related work in single session Know how to recognize GCS thrashing Plan your transactions; NO auto-commit
28

DanNorris.com

Performance Tuning
RAC is often not to blame, except when it
is to blame :)

Understand RAC-specic wait events When is the interconnect to blame? When is the interconnect just not working
well (HW issue)?

Who/what is monitoring the NICs?

29

DanNorris.com

Stick with the tuning methods you already know.

DanNorris.com

30

Performance Tuning Sequences*

DFS Lock Handle SQ enqueue High cache transfer rates for SEQ$ blocks Easy to x, needs business sign-off though
CACHE 1000 NOORDER
* see Joel Goodman preso archived at oracleracsig.org
31

DanNorris.com

The Top 5
I. Make friends with sysadmin, storage admin II. Get smart III. Set expectations IV.Do technical stuff right V. Follow a systematic approach for identifying problems
32

DanNorris.com

Oracle RAC SIG

See www.oracleracsig.org for details
Webcasts: Average 2x per month, live & recorded Conference Events (this week!):
Birds of a Feather, Tues 12:15pm, W222B Experts Panel, Tues 4:30pm, W222A Customer Panel (Vitrualization), Weds 4:30pm, W222B

Forums (via OTN): Lots of participation from RAC SIG as well as Oracle gurus Elections: Nominations open now on the website Join the RAC SIG at www.oracleracsig.org!

DanNorris.com

34

DanNorris.com

(Please nudge the person sleeping next to you.) DanNorris@DanNorris.com

Thank You!

DanNorris.com

Legal
The information contained herein should be deemed reliable but not guaranteed. The author has made every attempt to provide current and accurate information. If you have any comments or suggestions, please contact the author at: dannorris@dannorris.com You may request redistribution permission from dannorris@dannorris.com.

36

DanNorris.com