Detail Methodology Agenda 1. Introduction 2. Oracle SQL Tuning Skill Set 3. Basic Skill 4. Recommended Methodology Case Study 1: Case Study 2:
5. Additional SQL Tuning Idea
6. Q&A Introduction The purpose of this sharing session is to provide Huawei a generic & systematic method that will help Huaweis developers to find and fix a SQL performance problem in Exadata environment. The recommended methodology will attempt to guide developer along a process that will diagnose most common Oracle SQL performance problems. Oracle SQL Tuning Skillset 1. Basic Skill How to pick problematic SQL How to get the execution plan 2. Advanced skill - How to pick problematic SQL Identify the problematic SQL by: AWR report top elapsed time, top CPU time, top buffer get etc GV$SESSION &G V$SQL Enable 10046 trace at session level Report by user - Automatic Workload Repository (AWR) in Oracle Database 11g Under SQL Statistics section, they are many useful information SQL ordered by Elapsed Time SQL ordered by CPU Time SQL ordered by User I/O Wait Time SQL ordered by Gets SQL ordered by Reads SQL ordered by Physical Reads (UnOptimized) SQL ordered by Executions SQL ordered by Parse Calls SQL ordered by Sharable Memory SQL ordered by Version Count SQL ordered by Cluster Wait Time Complete List of SQL Text - Automatic Workload Repository (AWR) in Oracle Database 11g By example, under SQL ordered by Elapsed Time Easy to identify SQL that run very long time - Automatic Workload Repository (AWR) in Oracle Database 11g In general, Tune SQL that Elapsed Time per execution is high Tune SQL that Buffer Get per execution is high Tune SQL that CPU Time per execution is high Ask DBA for the AWR report - How to pick problematic SQL If the SQL is still running, query GV$SESSION and GV$SQL Example : select g.inst_id, g.sid, g.serial#, g.sql_id, g.event, g.machine, g.sql_exec_start, l.sql_text, l.PLAN_HASH_VALUE, g.blocking_session, g.SERVICE_NAME, g.status, g.LOGON_TIME from gv$session g , (select distinct sql_id, CHILD_NUMBER, PLAN_HASH_VALUE, SQL_text from gv$sql) l where g.username is not null and status = 'ACTIVE' and g.sql_id = l.SQL_ID and g.SQL_CHILD_NUMBER = l.CHILD_NUMBER order by g.sql_exec_start ; - How to pick problematic SQL Identify problematic SQL that is running for a long time refer to SQL_EXEC_START Oracle SQL Tuning Skillset 1. Basic Skill How to pick problematic SQL How to get the execution plan 2. Advanced skill Why we need execution plan? For SQL tuning, we need to review and understand the execution plan. Understand the execution plan is more important than understand the program logic in SQL tuning - How to get SQL execution plan If you get the problematic SQL from AWR, you can get the execution plan by running select * from table(Dbms_Xplan.display_awr(<SQL_ID>', null, null, 'ALL')) - How to get SQL execution plan If you get the problematic SQL from AWR, you can get the execution plan by running SQL> select * from table(Dbms_Xplan.display_awr('0rbu5r3j5f7fg', null, null, 'ALL')) 2 / PLAN_TABLE_OUTPUT -------------------------------------------------------------------------------- SQL_ID 0rbu5r3j5f7fg -------------------- SELECT DISTINCT T.COMPANY_ID, T.CONTRACT_NUMBER, T.PRODUCT_ID FROM BL_RCR_RATE_SEPCIAL_TMP1 T WHERE T.PERIOD_ID = :B2 AND T.SOURCE_TYPE_ID = :B1 AND T.COMPANY_ID IN (SELECT G.COMPANY_ID FROM APD_RCR_SCH_BATCH_COMPANY G WHERE G.BATCH_ID = :B3 ) AND T.PROGRAM_SOURCE IN ('REV_SPECIAL_PRE1', 'REV_SPECIAL_PRE2') Plan hash value: 2906480639 -------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes |TempS -------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | | | | 1 | HASH UNIQUE | | 112K| 6819K| 886 | 2 | NESTED LOOPS | | 112K| 6819K| | 3 | INDEX RANGE SCAN| APD_RCR_SCH_BATCH_COMPANY_U1 | 31 | 310 | | 4 | INDEX RANGE SCAN| BL_RCR_RATE_SEPCIAL_TMP1_N1 | 3633 | 184K| -------------------------------------------------------------------------------- Query Block Name / Object Alias (identified by operation id): ------------------------------------------------------------- 1 - SEL$5DA710D3 3 - SEL$5DA710D3 / G@SEL$2 4 - SEL$5DA710D3 / T@SEL$1 - How to get SQL execution plan You can get SQL_ID - unique for each SQL statement and Plan hash value unique for each execution plan for this SQL for further tuning use. SQL> select * from table(Dbms_Xplan.display_awr('0rbu5r3j5f7fg', null, null, 'ALL')) 2 / PLAN_TABLE_OUTPUT -------------------------------------------------------------------------------- SQL_ID 0rbu5r3j5f7fg -------------------- SELECT DISTINCT T.COMPANY_ID, T.CONTRACT_NUMBER, T.PRODUCT_ID FROM BL_RCR_RATE_SEPCIAL_TMP1 T WHERE T.PERIOD_ID = :B2 AND T.SOURCE_TYPE_ID = :B1 AND T.COMPANY_ID IN (SELECT G.COMPANY_ID FROM APD_RCR_SCH_BATCH_COMPANY G WHERE G.BATCH_ID = :B3 ) AND T.PROGRAM_SOURCE IN ('REV_SPECIAL_PRE1', 'REV_SPECIAL_PRE2') Plan hash value: 2906480639 -------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes |TempS -------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | | | | 1 | HASH UNIQUE | | 112K| 6819K| 886 | 2 | NESTED LOOPS | | 112K| 6819K| | 3 | INDEX RANGE SCAN| APD_RCR_SCH_BATCH_COMPANY_U1 | 31 | 310 | | 4 | INDEX RANGE SCAN| BL_RCR_RATE_SEPCIAL_TMP1_N1 | 3633 | 184K| -------------------------------------------------------------------------------- Query Block Name / Object Alias (identified by operation id): ------------------------------------------------------------- 1 - SEL$5DA710D3 3 - SEL$5DA710D3 / G@SEL$2 4 - SEL$5DA710D3 / T@SEL$1 - How to get SQL execution plan If you get the problematic SQL from running session, you can view the execution plan by running DBMS_XPLAN in particular instance Set long 20000000 Set pagesize 0 Set linesize 200 Select * from table(dbms_xplan.display_cursor(<SQL_ID>,null , ALLSTATS LAST)) - How to get SQL execution plan If you get the problematic SQL from runnnig session, you can view the execution plan by using SQL Monitor Report Set long 20000000 Set pagesize 0 Set linesize 200 Select dbms_sqltune.report_sql_monitor(sql_id => SQL_ID) from dual <- Click me for sample output <- Click me for sample output - How to get SQL execution plan If you only know the package, enable trace 10046 before execute the package 1. Example: 2. SQLPLUS>alter session set events 10046 trace name context forever, level 12; 3. SQLPLUS>exec PKG_BL_RCR_REV_FACT.SP_BL_RCR_REV 4. After done, go the user_dump_dest and get the trace file. 5. OS> cd /u01/app/oracle/diag/rdbms/dwdb/DWDB1/trace 6. OS> tkprof tracefile_name outputfile_name explain=hwdw/password waits=y <- Click me for sample output Oracle SQL Tuning Skillset 1. Basic Skill How to pick problematic SQL How to get the execution plan 2. Advanced skill Recommended Methodology - Recommended Methodology Most common SQL problems that are easy to identify and easy to fix by below steps. 1. Are the underlying tables and indexes analyzed? A. If yes, go to next step B. If no, gather stats and re-test 2. Does the SQL already have optimized joining method? A. If yes, go to next step B. If no, check if inappropriate hint, if yes, remove / add it and re-test 3. Is the SQL using smart scan? A. If yes, go to next step B. If no, invisible indexes or add FULL() hint or remove INDEX() or similar hints 4. Does SQL has appropriate join predicates? A. If yes, go to next step B. If no, adding appropriate HINT 5. For low volume SQLs, are there any Full Table/Partition Scans? A. If yes, check if appropriate to add index B. If no, go to next step - Recommended Methodology Most common SQL problems that are easy to identify and easy to fix by below steps. 6. Do you compare 9i, 10g and 11g execution plan? A. Compare each plans and bind the best to SPM - Recommended Methodology Most common SQL problems that are easy to identify and easy to fix by below steps. 1. Are the underlying tables and indexes analyzed? A. If yes, go to next step B. If no, gather stats and re-test 2. Does the SQL already have optimized joining method? A. If yes, go to next step B. If no, check if inappropriate hint, if yes, remove / add it and re-test 3. Is the SQL using smart scan? A. If yes, go to next step B. If no, invisible indexes or add FULL() hint or remove INDEX() or similar hints 4. Does SQL has appropriate join predicates? A. If yes, go to next step B. If no, adding appropriate HINT 5. For low volume SQLs, are there any Full Table/Partition Scans? A. If yes, check if appropriate to add index B. If no, go to next step - Recommended Methodology 1. Are the underlying tables and indexes analyzed? A. If yes, go to next step B. If no, gather stats and re-test Why accurate statistic is so important? The cost based optimizer has the challenging job of evaluating any SQL statement and generating the "best" execution plan for the statement. Object metadata The DBA controls the quality of the metadata via the dbms_stats package. This data includes the number of rows in a table, the distribution of values within a column and other critical information about the state of the tables and indexes. Disk I/O speed The cost of disk I/O is the single most important factor in SQL optimization. Disk I/O is measured in thousandths of a second, an eternity for a database, and something that needs to be avoided whenever possible. Why accurate statistic is so important? Case study S1: SQL> SELECT SUM(MTA.BASE_TRANSACTION_VALUE) ABSORPTION_AMOUNT FROM ODS_GL_CODE_COMBINATIONS GCC, ODS_MTL_TRANSACTION_ACCOUNTS MTA WHERE MTA.REFERENCE_ACCOUNT = GCC.CODE_COMBINATION_ID(+) AND GCC.SEGMENT3 LIKE '128%' AND MTA.ACCOUNTING_LINE_TYPE <> 15 AND MTA.COST_ELEMENT_ID <> 1 AND MTA.TRANSACTION_ID = 2455022551; Why accurate statistic is so important? Case study S1: SQL> SELECT SUM(MTA.BASE_TRANSACTION_VALUE) ABSORPTION_AMOUNT FROM ODS_GL_CODE_COMBINATIONS GCC, ODS_MTL_TRANSACTION_ACCOUNTS MTA WHERE MTA.REFERENCE_ACCOUNT = GCC.CODE_COMBINATION_ID(+) AND GCC.SEGMENT3 LIKE '128%' AND MTA.ACCOUNTING_LINE_TYPE <> 15 AND MTA.COST_ELEMENT_ID <> 1 AND MTA.TRANSACTION_ID = 2455022551; If No / Inaccurate table statistic: Delete statistic in tables exec dbms_stats.delete_table_stats('ODSPUB', 'ODS_GL_CODE_COMBINATIONS') exec dbms_stats.delete_table_stats('ODSCST', 'ODS_MTL_TRANSACTION_ACCOUNTS') Why accurate statistic is so important? Case study S1: Execution plan with no table statistic --------------------------------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart| Pstop | --------------------------------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | 117 | 484 (1)| 00:00:07 | | | | 1 | SORT AGGREGATE | | 1 | 117 | | | | | |* 2 | HASH JOIN | | 22690 | 2592K| 484 (1)| 00:00:07 | | | |* 3 | TABLE ACCESS BY GLOBAL INDEX ROWID| ODS_MTL_TRANSACTION_ACCOUNTS | 22690 | 1440K| 2 (0)| 00:00:01 | ROWID | ROWID | |* 4 | INDEX RANGE SCAN | ODS_MTL_TRANSACTION_ACCOUNT_N2 | 262K| | 1 (0)| 00:00:01 | | | | 5 | TABLE ACCESS BY INDEX ROWID | ODS_GL_CODE_COMBINATIONS | 210K| 10M| 481 (0)| 00:00:07 | | | |* 6 | INDEX RANGE SCAN | ODS_GL_CODE_COMBINATIONS_N3 | 210K| | 448 (0)| 00:00:07 | | | --------------------------------------------------------------------------------------------------------------------------------------- 1 rows selected. Elapsed: 00:00:08.00 Why slow ? Because incorrect index was chosen Why accurate statistic is so important? Case study S1: SQL> SELECT SUM(MTA.BASE_TRANSACTION_VALUE) ABSORPTION_AMOUNT FROM ODS_GL_CODE_COMBINATIONS GCC, ODS_MTL_TRANSACTION_ACCOUNTS MTA WHERE MTA.REFERENCE_ACCOUNT = GCC.CODE_COMBINATION_ID(+) AND GCC.SEGMENT3 LIKE '128%' AND MTA.ACCOUNTING_LINE_TYPE <> 15 AND MTA.COST_ELEMENT_ID <> 1 AND MTA.TRANSACTION_ID = 2455022551; With accurate table statistic: Gather statistic for tables using auto sample size, for all columns size auto exec dbms_stats.gather_table_stats(ownname=>'ODSCST',tabname=>'ODS_MTL_TRANSACTION_ACCO UNTS',estimate_percent=>dbms_stats.auto_sample_size, degree=>128, cascade=>True, method_opt=>'FOR ALL COLUMNS SIZE AUTO'); exec dbms_stats.gather_table_stats(ownname=>'ODSPUB',tabname=>'ODS_GL_CODE_COMBINATIONS' ,estimate_percent=>dbms_stats.auto_sample_size, degree=>32, cascade=>True, method_opt=>'FOR ALL COLUMNS SIZE AUTO'); Why accurate statistic is so important? Case study S1: Execution plan with accurate table statistic ---------------------------------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart| Pstop | ---------------------------------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | 37 | 9 (0)| 00:00:01 | | | | 1 | SORT AGGREGATE | | 1 | 37 | | | | | | 2 | NESTED LOOPS | | | | | | | | | 3 | NESTED LOOPS | | 2 | 74 | 9 (0)| 00:00:01 | | | |* 4 | TABLE ACCESS BY GLOBAL INDEX ROWID| ODS_MTL_TRANSACTION_ACCOUNTS | 2 | 46 | 5 (0)| 00:00:01 | ROWID | ROWID | |* 5 | INDEX RANGE SCAN | ODS_MTL_TRANSACTION_ACCOUNT_N2 | 4 | | 4 (0)| 00:00:01 | | | |* 6 | INDEX UNIQUE SCAN | ODS_GL_CODE_COMBINATIONS_U1 | 1 | | 1 (0)| 00:00:01 | | | |* 7 | TABLE ACCESS BY INDEX ROWID | ODS_GL_CODE_COMBINATIONS | 1 | 14 | 2 (0)| 00:00:01 | | | ---------------------------------------------------------------------------------------------------------------------------------------- 1 rows selected. Elapsed: 00:00:01.00 Correct index was chosen with accurate statistic Why accurate statistic is so important? Case study S2: SQL> SELECT * FROM table (DBMS_XPLAN.DISPLAY_CURSOR('&SQL_ID', NULL, 'ALLSTATS LAST')); Enter value for sql_id: 0wbgug6xbr7gg old 1: SELECT * FROM table (DBMS_XPLAN.DISPLAY_CURSOR('&SQL_ID', NULL, 'ALLSTATS LAST')) new 1: SELECT * FROM table (DBMS_XPLAN.DISPLAY_CURSOR('0wbgug6xbr7gg', NULL, 'ALLSTATS LAST')) PLAN_TABLE_OUTPUT ---------------------------------------------------------------------------------------- SQL_ID 0wbgug6xbr7gg, child number 0 ------------------------------------- SELECT 830 RECORD_TYPE_ID, TMP.PERIOD_ID, TMP.COMPANY_ID, TMP.COMPANY_CODE, TMP.FIN_REGION_ID, TMP.REGION_CODE, TMP.ACCOUNT_ID, TMP.ACCOUNT_CODE, TMP.PHYSICAL_PRODUCT_ID, TMP.PHYSICAL_PRODUCT_CODE, TMP.SERVICE_PRODUCT_ID, TMP.SERVICE_PRODUCT_CODE, PCD.CUSTOMER_ID CUSTOMER_ID, TMP.CUSTOMER_CODE, TMP.PROJECT_ID, TMP.PROJECT_CODE, TMP.COST_CENTER_ID, TMP.CURRENCY_ID, TMP.FIN_CONTRACT_ID, TMP.CONTRACT_NUMBER, TMP.SUB_PROJECT_ID, TMP.SUB_PROJECT_CODE, TMP.EXPENSE_TYPE, TMP.SOURCE_TYPE, TMP.ATTRIBUTE1, TMP.ATTRIBUTE2, TMP.ATTRIBUTE3, TMP.ATTRIBUTE4, TMP.ATTRIBUTE5, TMP.ATTRIBUTE6, TMP.ATTRIBUTE7, TMP.SER_ASSIGN_STEP, TMP.PHY_ASSIGN_STEP, TMP.REPORT_TYPE_FLAG, TMP.INDUSTRY_CLASS_ID, NVL(TMP.AMOUNT, 0) AMOUNT, NVL(TMP.FUNC_AMOUNT, 0) FUNC_AMOUNT, NVL(TMP.RMB_AMOUNT, 0) RMB_AMOUNT FROM BL_GTS_CST_CLASSIFY_TMP TMP, BL_PROJECT_CUST_DIM PCD WHERE EXPENSE_TYPE IN ('C610', 'C630') AND TMP.PERIOD_ID = :B1 AND NVL(TMP.RMB_AMOUNT,0) <> 0 AND TMP.SUB_PROJECT_ID = PCD.PROJECT_ID(+) Why accurate statistic is so important? Case study S2: ----------------------------------------------------------------------- | Id | Operation | Name | E-Rows | ----------------------------------------------------------------------- | 0 | SELECT STATEMENT | | | | 1 | NESTED LOOPS OUTER | | 1 | |* 2 | TABLE ACCESS STORAGE FULL| BL_GTS_CST_CLASSIFY_TMP | 1 | |* 3 | INDEX RANGE SCAN | BL_PROJECT_CUST_DIM_N1 | 1 | ----------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - storage(("TMP"."PERIOD_ID"=:B1 AND INTERNAL_FUNCTION("EXPENSE_TYPE") AND NVL("TMP"."RMB_AMOUNT",0)<>0)) filter(("TMP"."PERIOD_ID"=:B1 AND INTERNAL_FUNCTION("EXPENSE_TYPE") AND NVL("TMP"."RMB_AMOUNT",0)<>0)) 3 - access("TMP"."SUB_PROJECT_ID"="PCD"."PROJECT_ID") Optimizer uses nested loop to join those tables because estimated row is small for both tables Why accurate statistic is so important? Case study S2: SELECT COUNT(1) from BL_GTS_CST_CLASSIFY_TMP WHERE PERIOD_ID = 201103 ----------------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads ----------------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 1 |00:00:00.07 | 4294 | 4286 | 1 | SORT AGGREGATE | | 1 | 1 | 1 |00:00:00.07 | 4294 | 4286 |* 2 | TABLE ACCESS STORAGE FULL| BL_GTS_CST_CLASSIFY_TMP | 1 | 1 | 293K|00:00:00.07 | 4294 | 4286 ----------------------------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - storage("TMP"."PERIOD_ID"=201103) filter("TMP"."PERIOD_ID"=201103) Based on table statistic, E-rows (estimate row: 1) VS A- rows (actual rows: 293K) is huge different Why? Why accurate statistic is so important? Table Actual number of record during execution BL_GTS_CST_CLASSIFY_TMP 30,395 BL_PROJECT_CUST_DIM 196,557 Rows (1st) Rows (avg) Rows (max) Row Source Operation ---------- ---------- ---------- --------------------------------------------------- 30395 30395 30395 HASH JOIN OUTER (cr=3690 pr=3399 pw=0 time=1306343 us cost=647 size=6076431 card=30231) 30395 30395 30395 TABLE ACCESS STORAGE FULL BL_GTS_CST_CLASSIFY_TMP (cr=2965 pr=2959 pw=0 time=44754 us cost=451 size=5743890 card=30231) 196557 196557 196557 INDEX STORAGE FAST FULL SCAN BL_PROJECT_CUST_DIM_N1 (cr=725 pr=440 pw=0 time=56939 us cost=65 size=2162127 card=196557) Based on 10046 trace file, number of row processed is as above Why accurate statistic is so important? Case study S2: ----------------------------------------------------------------------- | Id | Operation | Name | E-Rows | ----------------------------------------------------------------------- | 0 | SELECT STATEMENT | | | | 1 | NESTED LOOPS OUTER | | 1 | |* 2 | TABLE ACCESS STORAGE FULL| BL_GTS_CST_CLASSIFY_TMP | 1 | |* 3 | INDEX RANGE SCAN | BL_PROJECT_CUST_DIM_N1 | 1 | ----------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - storage(("TMP"."PERIOD_ID"=:B1 AND INTERNAL_FUNCTION("EXPENSE_TYPE") AND NVL("TMP"."RMB_AMOUNT",0)<>0)) filter(("TMP"."PERIOD_ID"=:B1 AND INTERNAL_FUNCTION("EXPENSE_TYPE") AND NVL("TMP"."RMB_AMOUNT",0)<>0)) 3 - access("TMP"."SUB_PROJECT_ID"="PCD"."PROJECT_ID") Optimizer uses nested loop to join those tables but actual rows count for both tables are huge Why is wrong with using Nested Loop for joining? Indexed Nested Loops is used primarily in low volume joins; it is efficient over small volumes and versatile enough to be used in a variety of situations. Although it is fully scalable, Indexed Nested Loops is inefficient over large data volumes. Why accurate statistic is so important? What is wrong??? Incorrect table statistic & histogram Why accurate statistic is so important? Case study S2: SQL> select TABLE_NAME, COLUMN_NAME, LAST_ANALYZED, NUM_DISTINCT, LOW_VALUE, HIGH_VALUE, HISTOGRAM from dba_tab_columns where TABLE_NAME = 'BL_GTS_CST_CLASSIFY_TMP' TABLE_NAME COLUMN_NAME LAST_ANALYZED NUM_DISTINCT LOW_VALUE HIGH_VALUE HISTOG ------------------------------ ------------------------------ --------------- ------------ ---------- ---------- ------ BL_GTS_CST_CLASSIFY_TMP PERIOD_ID 11-JUN-11 1 C3150C05 C3150C05 FREQUE SQL> select utl_raw.cast_to_number(high_value) from dba_tab_columns where table_name = 'BL_GTS_CST_CLASSIFY_TMP UTL_RAW.CAST_TO_NUMBER(HIGH_VALUE) --------------------------------- 201104 SQL> select utl_raw.cast_to_number(low_value) from dba_tab_columns where table_name = 'BL_GTS_CST_CLASSIFY_TMP UTL_RAW.CAST_TO_NUMBER(LOW_VALUE) --------------------------------- 201104 Checked the table histogram, the highest / lowest value in period_id column is 201104. Should have no 201103 data exist in the table. Why accurate statistic is so important? Case study S2: ----------------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads ----------------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 1 |00:00:00.07 | 4294 | 4286 | 1 | SORT AGGREGATE | | 1 | 1 | 1 |00:00:00.07 | 4294 | 4286 |* 2 | TABLE ACCESS STORAGE FULL| BL_GTS_CST_CLASSIFY_TMP | 1 | 1 | 293K|00:00:00.07 | 4294 | 4286 ----------------------------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - storage("TMP"."PERIOD_ID"=201103) filter("TMP"."PERIOD_ID"=201103) If select 201103 data, Optimizer estimates maxium 1 row need to process because statistic indicates no 201103 data Optimizer, which uses nested loop to join those tables, is correct according the table statistic & histogram Only table statistic & histogram is incorrect Why accurate statistic is so important? What is wrong??? How it happens??? Step 1 : Gather table statistic on today 1:00pm Only have 201104 data in table B_GTS_CST_CLASSIFY_TMP Step 2 : Execute ONE PASS at night 11:00pm Delete all data in BL_GTS_CST_CLASSIFY_TMP and then insert 201103 data Step 3 : Execute the same SQL Bad performance due to incorrect statistic bad joining method Why accurate statistic is so important? Case study S2: Lets see what happen after gather statistic Gather statistic for tables using auto sample size, for all columns size auto exec dbms_stats.gather_table_stats(ownname=>BLREP',tabname=>' BL_GTS_CST_CLASSIFY_TMP',estimate_percent=>dbms_stats.auto_sample_size, degree=>32, cascade=>True, method_opt=>'FOR ALL COLUMNS SIZE AUTO'); exec Check the DBA_TABLES & DBA_TAB_COLUMNS statistic information If data_type = NUMBER, SELECT UTL_RAW.CAST_TO_NUMBER(low_value), UTL_RAW.CAST_TO_NUMBER(HIGH_VALUE) FROM DBA_TAB_COLUMNS WHERE TABLE_NAME = 'ODS_PROJ_CON_REG_RELATION' AND COLUMN_NAME = 'PERIOD_ID' If data_type = VARCHAR2, SELECT UTL_RAW.CAST_TO_VARCHAR2(low_value), UTL_RAW.CAST_TO_VARCHAR2 (HIGH_VALUE) FROM DBA_TAB_COLUMNS WHERE TABLE_NAME = 'ODS_PROJ_CON_REG_RELATION' AND COLUMN_NAME = 'PERIOD_ID' Why accurate statistic is so important? Case study S2: ----------------------------------------------------------------------- | Id | Operation | Name | E-Rows | ----------------------------------------------------------------------- | 0 | SELECT STATEMENT | | | | 1 | HASH JOIN OUTER | | 30K | |* 2 | TABLE ACCESS STORAGE FULL| BL_GTS_CST_CLASSIFY_TMP | 30K | |* 3 | INDEX RANGE SCAN | BL_PROJECT_CUST_DIM_N1 | 196K | ----------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - storage(("TMP"."PERIOD_ID"=:B1 AND INTERNAL_FUNCTION("EXPENSE_TYPE") AND NVL("TMP"."RMB_AMOUNT",0)<>0)) filter(("TMP"."PERIOD_ID"=:B1 AND INTERNAL_FUNCTION("EXPENSE_TYPE") AND NVL("TMP"."RMB_AMOUNT",0)<>0)) 3 - access("TMP"."SUB_PROJECT_ID"="PCD"."PROJECT_ID") Estimate rows become 30K Optimizer uses HASH JOIN, execution plan changed Nested Loops VS Hash Join When to use Hash joins At least one side of the join is returning many rows Low cardinality indexes are not available on the join keys. The join predicates use only equals (=) conditions. Nested Loops VS Hash Join Nested Loops join is acceptable in a high volume SQL are: When the driving (inner) table will return 0 or 1 row. If you have a join query where one of the tables is supplied with the whole of a primary or unique key, Oracle can retrieve the row (if there is one) and then perform a full table scan on the second table. This is more efficient than either a sort-merge or a hash join. When the outer (second) table is very small (ie. fewer than 100 rows) and can fit into a single block. Since a single block is the smallest amount of data Oracle can read, a Table that fits into a single block can be accessed very fast with a Full Table Scan. Bad SQL execution plan caused by incorrect statistic Elapsed time with correct statistic : 6min (201104) Elapsed time with incorrect statistic : 129min (201103) Elapsed time after gather statistic: 6min (201103) - Recommended Methodology 1. Are the underlying tables and indexes analyzed? A. If yes, go to next step B. If no, gather stats and re-test Incorrect statistic cause Optimizer : Choose wrong INDEX Choose wrong JOIN method - Recommended Methodology For this case, Step 1 : Gather table statistic on today 1:00pm Only have 201104 data in table B_GTS_CST_CLASSIFY_TMP Step 2 : Execute ONE PASS at night 11:00pm Delete all data in BL_GTS_CST_CLASSIFY_TMP and then insert 201103 data Step 3 : Execute the same SQL Bad performance due to incorrect statistic > bad joining method - Recommended Methodology Solution 1: Gather statistic on dynamic working table after bulk insert / delete In the package logic / MOIA job: 1. Delete table BL_GTS_CST_CLASSIFY_TMP 2. Insert record to BL_GTS_CST_CLASSIFY_TMP 3. Add Gather table statistic -BL_GTS_CST_CLASSIFY_TMP 4. Run other jobs - Recommended Methodology Solution 2 : ----------------------------------------------------------------------- | Id | Operation | Name | E-Rows | ----------------------------------------------------------------------- | 0 | SELECT STATEMENT | | | | 1 | HASH JOIN OUTER | | 30K | |* 2 | TABLE ACCESS STORAGE FULL| BL_GTS_CST_CLASSIFY_TMP | 30K | |* 3 | INDEX RANGE SCAN | BL_PROJECT_CUST_DIM_N1 | 196K | ----------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - storage(("TMP"."PERIOD_ID"=:B1 AND INTERNAL_FUNCTION("EXPENSE_TYPE") AND NVL("TMP"."RMB_AMOUNT",0)<>0)) filter(("TMP"."PERIOD_ID"=:B1 AND INTERNAL_FUNCTION("EXPENSE_TYPE") AND NVL("TMP"."RMB_AMOUNT",0)<>0)) 3 - access("TMP"."SUB_PROJECT_ID"="PCD"."PROJECT_ID") If I know this should be the correct execution plan - Recommended Methodology Solution 2 : Use SPM (SQL PLAN MANAGEMENT) for a statement, subsequent executions of that statement will use the SQL plan baseline. Example 1. Create a SPM for a correct execution plan 2. Next month when select data 201105, even high / low value statistic only have 201104, as long as the SQL is the SAME, the SQL will use the correct SQL execution plan - Recommended Methodology Solution 2 : 1. Create a SPM for a correct execution plan ----------------------------------------------------------------------- | Id | Operation | Name | E-Rows | ----------------------------------------------------------------------- | 0 | SELECT STATEMENT | | | | 1 | HASH JOIN OUTER | | 30K | |* 2 | TABLE ACCESS STORAGE FULL| BL_GTS_CST_CLASSIFY_TMP | 30K | |* 3 | INDEX RANGE SCAN | BL_PROJECT_CUST_DIM_N1 | 196K | ----------------------------------------------------------------------- SQL_ID=0wbgug6xbr7gg PLAN_HASH_VALUE=656489123 SQLPLUS> VARIABLE CNT NUMBER ; SQLPLUS> execute :CNT :=DBMS_SPM.LOAD_PLANS_FROM_CURSOR_CACHE(SQL_ID => 0wbgug6xbr7gg, PLAN_HASH_VALUE => 656489123) ; Check the added SPM SQLPLUS> SELECT * FROM DBA_SQL_PLAN_BASELINES ORDER BY LAST_MODIFIED - Recommended Methodology Solution 2 : 1. Create a SPM for a correct execution plan Check the added SPM: SQLPLUS> SELECT SQL_HANDLE, PLAN_NAME, LAST_MODIFIED, ENABLED, ACCEPTED, SQL_TEX FROM DBA_SQL_PLAN_BASELINES ORDER BY LAST_MODIFIED Check the SPM execution plan: SQLPLUS> SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_SQL_PLAN_BASELINE(SQL_HANDLE) - Recommended Methodology Solution 2 : 2. Next month when select data 201105, even high / low value statistic only have 201104, as long as the SQL is the SAME, the SQL will use the correct SQL execution plan PLAN_TABLE_OUTPUT ---------------------------------------------------------------------------------------- SQL_ID 0wbgug6xbr7gg, child number 0 ------------------------------------- SELECT 830 RECORD_TYPE_ID, TMP.PERIOD_ID, TMP.COMPANY_ID, TMP.COMPANY_CODE, TMP.FIN_REGION_ID, TMP.REGION_CODE, TMP.ACCOUNT_ID, TMP.ACCOUNT_CODE, TMP.PHYSICAL_PRODUCT_ID, TMP.PHYSICAL_PRODUCT_CODE, TMP.SERVICE_PRODUCT_ID, TMP.SERVICE_PRODUCT_CODE, PCD.CUSTOMER_ID CUSTOMER_ID, TMP.CUSTOMER_CODE, TMP.PROJECT_ID, TMP.PROJECT_CODE, TMP.COST_CENTER_ID, TMP.CURRENCY_ID, TMP.FIN_CONTRACT_ID, TMP.CONTRACT_NUMBER, TMP.SUB_PROJECT_ID, TMP.SUB_PROJECT_CODE, TMP.EXPENSE_TYPE, TMP.SOURCE_TYPE, TMP.ATTRIBUTE1, TMP.ATTRIBUTE2, TMP.ATTRIBUTE3, TMP.ATTRIBUTE4, TMP.ATTRIBUTE5, TMP.ATTRIBUTE6, TMP.ATTRIBUTE7, TMP.SER_ASSIGN_STEP, TMP.PHY_ASSIGN_STEP, TMP.REPORT_TYPE_FLAG, TMP.INDUSTRY_CLASS_ID, NVL(TMP.AMOUNT, 0) AMOUNT, NVL(TMP.FUNC_AMOUNT, 0) FUNC_AMOUNT, NVL(TMP.RMB_AMOUNT, 0) RMB_AMOUNT FROM BL_GTS_CST_CLASSIFY_TMP TMP, BL_PROJECT_CUST_DIM PCD WHERE EXPENSE_TYPE IN ('C610', 'C630') AND TMP.PERIOD_ID = :B1 AND NVL(TMP.RMB_AMOUNT,0) <> 0 AND TMP.SUB_PROJECT_ID = PCD.PROJECT_ID(+) - Recommended Methodology Solution 2 : 2. Next month when select data 201105, even high / low value statistic only have 201104, as long as the SQL is the SAME, the SQL will use the correct SQL execution plan ----------------------------------------------------------------------- | Id | Operation | Name | E-Rows | ----------------------------------------------------------------------- | 0 | SELECT STATEMENT | | | | 1 | HASH JOIN OUTER | | 30K | |* 2 | TABLE ACCESS STORAGE FULL| BL_GTS_CST_CLASSIFY_TMP | 30K | |* 3 | INDEX RANGE SCAN | BL_PROJECT_CUST_DIM_N1 | 196K | ----------------------------------------------------------------------- Note ----- - SQL plan baseline SYS_SQL_PLAN_fcc170b0a62d0f4d used for this statement\ The same execution plan will use. If SPM is used, a note will indicate it when you view the execution plan - Recommended Methodology 1. Are the underlying tables and indexes analyzed? A. If yes, go to next step B. If no, gather stats and re-test YES, go to next step - Recommended Methodology Most common SQL problems that are easy to identify and easy to fix by below steps. 1. Are the underlying tables and indexes analyzed? A. If yes, go to next step B. If no, gather stats and re-test 2. Does the SQL already have optimized joining method? A. If yes, go to next step B. If no, check if inappropriate hint, if yes, remove / add it and re-test 3. Is the SQL using smart scan? A. If yes, go to next step B. If no, invisible indexes or add FULL() hint or remove INDEX() or similar hints 4. Does SQL has appropriate join predicates? A. If yes, go to next step B. If no, adding appropriate HINT 5. For low volume SQLs, are there any Full Table/Partition Scans? A. If yes, check if appropriate to add index B. If no, go to next step - Recommended Methodology 2. Does the SQL already have optimized joining method? A. If yes, go to next step B. If no, check if inappropriate hint, if yes, remove / add it and re-test Case study J1 Problematic SQL: SELECT COT.CONTRACT_NUMBER ,COT.ITEM_CODE ,COT.QUANTITY , Q_L.QUOTATION_ITEM_CODE ,Q_L.QUOTATION_QUANTITY FROM (SELECT /*+INDEX(DT ODS_CP_ORDER_OM_DAILY_TEMP_N1)*/ DT.CONTRACT_NUMBER, DT.ITEM_CODE, SUM(DT.QUANTITY) QUANTITY FROM ODS_CP_ORDER_OM_DAILY_TEMP DT GROUP BY DT.CONTRACT_NUMBER, DT.ITEM_CODE) COT, (SELECT /*+USE_NL(QH,QL) LEADING(QH) INDEX(QH ODS_CP_QUOTATION_HEADERS_U1) INDEX(QL ODS_CP_QUOTATION_LINES_N1)*/ QH.CONTRACT_NUMBER, QL.QUOTATION_ITEM_CODE, SUM(QL.QUOTATION_QUANTITY) QUOTATION_QUANTITY FROM ODS_CP_QUOTATION_HEADERS QH, ODS_CP_QUOTATION_LINES QL, (SELECT DT.CONTRACT_NUMBER, DT.ITEM_CODE, SUM(DT.QUANTITY) QUANTITY FROM ODS_CP_ORDER_OM_DAILY_TEMP DT GROUP BY DT.CONTRACT_NUMBER, DT.ITEM_CODE) ODT WHERE QH.QUOTATION_HEADER_ID = QL.QUOTATION_HEADER_ID AND ODT.CONTRACT_NUMBER = QH.CONTRACT_NUMBER AND ODT.ITEM_CODE = QL.QUOTATION_ITEM_CODE GROUP BY QH.CONTRACT_NUMBER, QL.QUOTATION_ITEM_CODE) Q_L WHERE COT.CONTRACT_NUMBER = Q_L.CONTRACT_NUMBER(+) AND COT.ITEM_CODE = Q_L.QUOTATION_ITEM_CODE(+) Case study J1 Problematic SQL: SELECT COT.CONTRACT_NUMBER ,COT.ITEM_CODE ,COT.QUANTITY , Q_L.QUOTATION_ITEM_CODE ,Q_L.QUOTATION_QUANTITY FROM (SELECT /*+INDEX(DT ODS_CP_ORDER_OM_DAILY_TEMP_N1)*/ DT.CONTRACT_NUMBER, DT.ITEM_CODE, SUM(DT.QUANTITY) QUANTITY FROM ODS_CP_ORDER_OM_DAILY_TEMP DT GROUP BY DT.CONTRACT_NUMBER, DT.ITEM_CODE) COT, (SELECT /*+USE_NL(QH,QL) LEADING(QH) INDEX(QH ODS_CP_QUOTATION_HEADERS_U1) INDEX(QL ODS_CP_QUOTATION_LINES_N1)*/ QH.CONTRACT_NUMBER, QL.QUOTATION_ITEM_CODE, SUM(QL.QUOTATION_QUANTITY) QUOTATION_QUANTITY FROM ODS_CP_QUOTATION_HEADERS QH, ODS_CP_QUOTATION_LINES QL, (SELECT DT.CONTRACT_NUMBER, DT.ITEM_CODE, SUM(DT.QUANTITY) QUANTITY FROM ODS_CP_ORDER_OM_DAILY_TEMP DT GROUP BY DT.CONTRACT_NUMBER, DT.ITEM_CODE) ODT WHERE QH.QUOTATION_HEADER_ID = QL.QUOTATION_HEADER_ID AND ODT.CONTRACT_NUMBER = QH.CONTRACT_NUMBER AND ODT.ITEM_CODE = QL.QUOTATION_ITEM_CODE GROUP BY QH.CONTRACT_NUMBER, QL.QUOTATION_ITEM_CODE) Q_L WHERE COT.CONTRACT_NUMBER = Q_L.CONTRACT_NUMBER(+) AND COT.ITEM_CODE = Q_L.QUOTATION_ITEM_CODE(+) If Nested Loop is used in this case ------------------------------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads | ------------------------------------------------------------------------------------------------------------------------------------- ...... | 8 | TABLE ACCESS BY INDEX ROWID | ODS_CP_QUOTATION_HEADERS | 1 | 10M| 10M|00:00:05.51 | 316K| 0 | | 9 | INDEX FULL SCAN | ODS_CP_QUOTATION_HEADERS_U1 | 1 | 10M| 10M|00:00:01.33 | 11771 | 0 | ..... | 16 | TABLE ACCESS STORAGE FULL | ODS_CP_QUOTATION_LINES | 1 | 180M| 180M|00:00:22.25 | 615K| 615K| If nested loop is used for ODS_CP_QUOTATION_HEADERS (rows 10M) and ODS_CP_QUOTATION_LINES (rows 180M) .. 14604 rows selected. Elapsed: more than 2hr Case study J1 If removed all HINTs and let Optimizer chooses: SELECT COT.CONTRACT_NUMBER ,COT.ITEM_CODE ,COT.QUANTITY , Q_L.QUOTATION_ITEM_CODE ,Q_L.QUOTATION_QUANTITY FROM (SELECT /*+INDEX(DT ODS_CP_ORDER_OM_DAILY_TEMP_N1)*/ DT.CONTRACT_NUMBER, DT.ITEM_CODE, SUM(DT.QUANTITY) QUANTITY FROM ODS_CP_ORDER_OM_DAILY_TEMP DT GROUP BY DT.CONTRACT_NUMBER, DT.ITEM_CODE) COT, (SELECT /*+USE_NL(QH,QL) LEADING(QH) INDEX(QH ODS_CP_QUOTATION_HEADERS_U1) INDEX(QL ODS_CP_QUOTATION_LINES_N1)*/ QH.CONTRACT_NUMBER, QL.QUOTATION_ITEM_CODE, SUM(QL.QUOTATION_QUANTITY) QUOTATION_QUANTITY FROM ODS_CP_QUOTATION_HEADERS QH, ODS_CP_QUOTATION_LINES QL, (SELECT DT.CONTRACT_NUMBER, DT.ITEM_CODE, SUM(DT.QUANTITY) QUANTITY FROM ODS_CP_ORDER_OM_DAILY_TEMP DT GROUP BY DT.CONTRACT_NUMBER, DT.ITEM_CODE) ODT WHERE QH.QUOTATION_HEADER_ID = QL.QUOTATION_HEADER_ID AND ODT.CONTRACT_NUMBER = QH.CONTRACT_NUMBER AND ODT.ITEM_CODE = QL.QUOTATION_ITEM_CODE GROUP BY QH.CONTRACT_NUMBER, QL.QUOTATION_ITEM_CODE) Q_L WHERE COT.CONTRACT_NUMBER = Q_L.CONTRACT_NUMBER(+) AND COT.ITEM_CODE = Q_L.QUOTATION_ITEM_CODE(+) Case study J1 : Checking ------------------------------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads | ------------------------------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 14604 |00:20:21.00 | 957K| 2383K| |* 1 | HASH JOIN OUTER | | 1 | 99751 | 14604 |00:20:21.00 | 957K| 2383K| | 2 | VIEW | | 1 | 14604 | 14604 |00:00:00.14 | 12708 | 0 | | 3 | HASH GROUP BY | | 1 | 14604 | 14604 |00:00:00.14 | 12708 | 0 | | 4 | TABLE ACCESS STORAGE FULL | ODS_CP_ORDER_OM_DAILY_TEMP | 1 | 390K| 390K|00:00:00.11 | 12708 | 0 | | 5 | VIEW | | 1 | 2665K| 12200 |00:20:20.85 | 945K| 2383K| | 6 | HASH GROUP BY | | 1 | 2665K| 12200 |00:20:20.85 | 945K| 2383K| |* 7 | HASH JOIN | | 1 | 46M| 541K|00:09:55.83 | 945K| 2383K| | 8 | TABLE ACCESS BY INDEX ROWID | ODS_CP_QUOTATION_HEADERS | 1 | 10M| 10M|00:00:05.51 | 316K| 0 | | 9 | INDEX FULL SCAN | ODS_CP_QUOTATION_HEADERS_U1 | 1 | 10M| 10M|00:00:01.33 | 11771 | 0 | | 10 | VIEW | VW_GBC_5 | 1 | 46M| 559M|00:20:02.72 | 628K| 2368K| | 11 | HASH GROUP BY | | 1 | 46M| 559M|00:18:31.21 | 628K| 2368K| |* 12 | HASH JOIN | | 1 | 46M| 627M|00:02:08.95 | 628K| 615K| | 13 | VIEW | | 1 | 14604 | 14604 |00:00:00.13 | 12708 | 0 | | 14 | HASH GROUP BY | | 1 | 14604 | 14604 |00:00:00.13 | 12708 | 0 | | 15 | TABLE ACCESS STORAGE FULL| ODS_CP_ORDER_OM_DAILY_TEMP | 1 | 390K| 390K|00:00:00.10 | 12708 | 0 | | 16 | TABLE ACCESS STORAGE FULL | ODS_CP_QUOTATION_LINES | 1 | 180M| 180M|00:00:22.25 | 615K| 615K| ------------------------------------------------------------------------------------------------------------------------------------- Check if E-Rows close to A-Rows If yes, go to next step If no, gather statistics and re-test Are the underlying tables and indexes analyzed? YES Case study J1 : Checking ------------------------------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads | ------------------------------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 14604 |00:20:21.00 | 957K| 2383K| |* 1 | HASH JOIN OUTER | | 1 | 99751 | 14604 |00:20:21.00 | 957K| 2383K| | 2 | VIEW | | 1 | 14604 | 14604 |00:00:00.14 | 12708 | 0 | | 3 | HASH GROUP BY | | 1 | 14604 | 14604 |00:00:00.14 | 12708 | 0 | | 4 | TABLE ACCESS STORAGE FULL | ODS_CP_ORDER_OM_DAILY_TEMP | 1 | 390K| 390K|00:00:00.11 | 12708 | 0 | | 5 | VIEW | | 1 | 2665K| 12200 |00:20:20.85 | 945K| 2383K| | 6 | HASH GROUP BY | | 1 | 2665K| 12200 |00:20:20.85 | 945K| 2383K| |* 7 | HASH JOIN | | 1 | 46M| 541K|00:09:55.83 | 945K| 2383K| | 8 | TABLE ACCESS BY INDEX ROWID | ODS_CP_QUOTATION_HEADERS | 1 | 10M| 10M|00:00:05.51 | 316K| 0 | | 9 | INDEX FULL SCAN | ODS_CP_QUOTATION_HEADERS_U1 | 1 | 10M| 10M|00:00:01.33 | 11771 | 0 | | 10 | VIEW | VW_GBC_5 | 1 | 46M| 559M|00:20:02.72 | 628K| 2368K| | 11 | HASH GROUP BY | | 1 | 46M| 559M|00:18:31.21 | 628K| 2368K| |* 12 | HASH JOIN | | 1 | 46M| 627M|00:02:08.95 | 628K| 615K| | 13 | VIEW | | 1 | 14604 | 14604 |00:00:00.13 | 12708 | 0 | | 14 | HASH GROUP BY | | 1 | 14604 | 14604 |00:00:00.13 | 12708 | 0 | | 15 | TABLE ACCESS STORAGE FULL| ODS_CP_ORDER_OM_DAILY_TEMP | 1 | 390K| 390K|00:00:00.10 | 12708 | 0 | | 16 | TABLE ACCESS STORAGE FULL | ODS_CP_QUOTATION_LINES | 1 | 180M| 180M|00:00:22.25 | 615K| 615K| ------------------------------------------------------------------------------------------------------------------------------------- Does the SQL already have optimized joining method? Case study J1 : Checking ------------------------------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads | ------------------------------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 14604 |00:20:21.00 | 957K| 2383K| |* 1 | HASH JOIN OUTER | | 1 | 99751 | 14604 |00:20:21.00 | 957K| 2383K| | 2 | VIEW | | 1 | 14604 | 14604 |00:00:00.14 | 12708 | 0 | | 3 | HASH GROUP BY | | 1 | 14604 | 14604 |00:00:00.14 | 12708 | 0 | | 4 | TABLE ACCESS STORAGE FULL | ODS_CP_ORDER_OM_DAILY_TEMP | 1 | 390K| 390K|00:00:00.11 | 12708 | 0 | | 5 | VIEW | | 1 | 2665K| 12200 |00:20:20.85 | 945K| 2383K| | 6 | HASH GROUP BY | | 1 | 2665K| 12200 |00:20:20.85 | 945K| 2383K| |* 7 | HASH JOIN | | 1 | 46M| 541K|00:09:55.83 | 945K| 2383K| | 8 | TABLE ACCESS BY INDEX ROWID | ODS_CP_QUOTATION_HEADERS | 1 | 10M| 10M|00:00:05.51 | 316K| 0 | | 9 | INDEX FULL SCAN | ODS_CP_QUOTATION_HEADERS_U1 | 1 | 10M| 10M|00:00:01.33 | 11771 | 0 | | 10 | VIEW | VW_GBC_5 | 1 | 46M| 559M|00:20:02.72 | 628K| 2368K| | 11 | HASH GROUP BY | | 1 | 46M| 559M|00:18:31.21 | 628K| 2368K| |* 12 | HASH JOIN | | 1 | 46M| 627M|00:02:08.95 | 628K| 615K| | 13 | VIEW | | 1 | 14604 | 14604 |00:00:00.13 | 12708 | 0 | | 14 | HASH GROUP BY | | 1 | 14604 | 14604 |00:00:00.13 | 12708 | 0 | | 15 | TABLE ACCESS STORAGE FULL| ODS_CP_ORDER_OM_DAILY_TEMP | 1 | 390K| 390K|00:00:00.10 | 12708 | 0 | | 16 | TABLE ACCESS STORAGE FULL | ODS_CP_QUOTATION_LINES | 1 | 180M| 180M|00:00:22.25 | 615K| 615K| ------------------------------------------------------------------------------------------------------------------------------------- Because the A-Rows (390K, 10M, 180M) are big numbers, hash join is the most appropriate join Does the SQL already have optimized joining method? YES Case study J1 If using the correct joining method 14604 rows selected. Elapsed: 00:20:21.85 Nested Loops VS Hash Join Is Hash Join better than Nested Loop? Example: SELECT SD.SALESREP_ID,SD.SALESREP_CODE,SD.SALESREP_DESCRIPTION, OE.CUST_PO_NUMBER FROM ODS_OE_ORDER_HEADERS_ALL OE, BL_FIN_SALESREP_DIM SD WHERE OE.SALESREP_ID = SD.SALESREP_ID AND SD.SOURCE_CODE_ID = 1 AND OE.FLOW_STATUS_CODE <> 'CANCELLED' AND NVL(OE.PURGE_DELETE_FLAG,'N') = 'N' AND OE.SALESREP_ID IS NOT NULL AND OE.CUST_PO_NUMBER = '0006821002170H' AND ROWNUM = 1 Case J2 Nested Loops Elapsed: 00:00:00.00 --------------------------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads | --------------------------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 1 |00:00:00.01 | 7 | 4 | |* 1 | COUNT STOPKEY | | 1 | | 1 |00:00:00.01 | 7 | 4 | | 2 | NESTED LOOPS | | 1 | | 1 |00:00:00.01 | 7 | 4 | | 3 | NESTED LOOPS | | 1 | 1 | 1 |00:00:00.01 | 6 | 4 | |* 4 | TABLE ACCESS BY INDEX ROWID| ODS_OE_ORDER_HEADERS_ALL | 1 | 2 | 1 |00:00:00.01 | 4 | 4 | |* 5 | INDEX RANGE SCAN | ODS_OE_ORDER_HEADERS_ALL_N4 | 1 | 2 | 1 |00:00:00.01 | 3 | 3 | |* 6 | INDEX UNIQUE SCAN | BL_FIN_SALESREP_DIM_U1 | 1 | 1 | 1 |00:00:00.01 | 2 | 0 | | 7 | TABLE ACCESS BY INDEX ROWID | BL_FIN_SALESREP_DIM | 1 | 1 | 1 |00:00:00.01 | 1 | 0 | --------------------------------------------------------------------------------------------------------------------------------- Are the underlying tables and indexes analyzed? YES Does the SQL already have optimized joining method? YES, those result set are small, 1row for ODS_OE_ORDER_HEADERS_ALL & 1 row for BL_FIN_SALESREP_DIM Nested Loops VS Hash Join What if using Hash join? Example : SELECT /*+ USE_HASH(OE SD) */ SD.SALESREP_ID,SD.SALESREP_CODE,SD.SALESREP_DESCRIPTION, OE.CUST_PO_NUMBER FROM ODS_OE_ORDER_HEADERS_ALL OE, BL_FIN_SALESREP_DIM SD WHERE OE.SALESREP_ID = SD.SALESREP_ID AND SD.SOURCE_CODE_ID = 1 AND OE.FLOW_STATUS_CODE <> 'CANCELLED' AND NVL(OE.PURGE_DELETE_FLAG,'N') = 'N' AND OE.SALESREP_ID IS NOT NULL AND OE.CUST_PO_NUMBER = '0006821002170H' AND ROWNUM = 1 Case J2 Hash Join Elapsed: 00:00:00.00 ---------------------------------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | OMem | ---------------------------------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 1 |00:00:00.01 | 68 | | |* 1 | COUNT STOPKEY | | 1 | | 1 |00:00:00.01 | 68 | | |* 2 | HASH JOIN | | 1 | 2 | 1 |00:00:00.01 | 68 | 841K| |* 3 | TABLE ACCESS BY INDEX ROWID | ODS_OE_ORDER_HEADERS_ALL | 1 | 5 | 5 |00:00:00.01 | 8 | | |* 4 | INDEX RANGE SCAN | ODS_OE_ORDER_HEADERS_ALL_N4 | 1 | 5 | 5 |00:00:00.01 | 3 | | |* 5 | TABLE ACCESS STORAGE FULL FIRST ROWS| BL_FIN_SALESREP_DIM | 1 | 1591 | 3314 |00:00:00.01 | 60 | | ---------------------------------------------------------------------------------------------------------------------------------------- Estimate rows and actual rows almost the same Same Elapsed time Nested Loops VS Hash Join NESTED LOOPS HASH JOIN Elapsed Time: 00:00:00.01 00:00:00.01 Buffers Get: 7 68 NESTED LOOPS has lower Buffers Get in small result set joining During SQL tuning, objective : Reduce logical reads / buffer gets Most reliable metric Reduce CPU time Optimizer generates execution plans based on the 2 metrics Given a specific plan, CPU time and Buffer Gets wont change. But other metrics such as Elapsed time, Physical reads could change - Recommended Methodology 2. Does the SQL already have optimized joining method? A. If yes, go to next step B. If no, check if inappropriate hint, if yes, remove / add it and re-test YES, go to next step - Recommended Methodology Most common SQL problems that are easy to identify and easy to fix by below steps. 1. Are the underlying tables and indexes analyzed? A. If yes, go to next step B. If no, gather stats and re-test 2. Does the SQL already have optimized joining method? A. If yes, go to next step B. If no, check if inappropriate hint, if yes, remove / add it and re-test 3. Is the SQL using smart scan? A. If yes, go to next step B. If no, invisible indexes or add FULL() hint or remove INDEX() or similar hints 4. Does SQL has appropriate join predicates? A. If yes, go to next step B. If no, adding appropriate HINT 5. For low volume SQLs, are there any Full Table/Partition Scans? A. If yes, check if appropriate to add index B. If no, go to next step - Recommended Methodology 3. Is the SQL using smart scan? A. If yes, go to next step B. If no, invisible indexes or add FULL() hint or remove INDEX() or similar hints Exadata Overview - Hardware Architecture Database Grid Storage Server InfiniBand Network Redundant 40Gb/s switches Unified server & storage network 14 High-performance low-cost storage servers 8 Dual-processor x64 database servers OR 2 Eight-processor x64 database servers 100 TB High Performance disk, or 336 TB High Capacity disk 5.3 TB PCI Flash Data mirrored across storage servers Exadata Features Exadata Smart Scans 10X or greater reduction in data sent to database servers Exadata Storage Indexes Eliminate unnecessary I/Os Hybrid Columnar Compression Efficient compression increases effective storage capacity and increases user data scan bandwidths by a factor of up to 10X Exadata Smart Flash Cache Breaks random I/O bottleneck by increasing IOPs by up to 20X Doubles user data scan bandwidths I/O Resource Manager (IORM) Enables storage grid by prioritizing I/Os to ensure predictable performance Smart IO what? Smart IO is not Block IO Block IO - data is shipped to the location where it can be processed - RDBMS Smart IO Some of the processing is shipped to where data resides Exadata Storage Server Results from the storage layer may be further processed in the RDBMS Smart IO why? ( for Performance) Reduced network IO Data get filtered due to smart IO operations offloaded to the storage layer Reduces the processing burden on the host Horizontal parallelism Concurrent processing of the smart IO requests by many exadata storage servers Concurrent processing of smart IO requests, from a single database process, by many threads within a single exadata storage server Vertical (pipeline) parallelism Exadata storage servers processing more results while database is consuming results already returned Smart IO How? Smart IO implementation is distributed across both RDBMS and Exadata storage server(s) RDBMS implements smart IO applications and may choose to use smart IO as opposed to block IO RDBMS drives smart IO Exadata storage server serves smart IO Smart Scan Pre-requisite There must be a full scan on an object FTS (TABLE ACCESS STORAGE FULL) INDEX_FFS (INDEX STORAGE FAST FULL SCAN) BITMAP INDEX SCAN (BITMAP INDEX STORAGE FAST FULL SCAN) The scan must use Oracles Direct Path Read mechanism Mechanism changed in 11g favoring Exadata If a table smaller than _small_table_threshold, the table will still be cached in SGA though PARALLEL is used with PARALLEL_DEGREE_POLICY=MANUAL _small_table_threshold = 400M default The object must be stored on Oracles Exadata Storage Smart Scan showed in execution plan doesnt mean its really using Smart Scan Explain plan table scan no exadata ----------------------------------------------------------------------------------- | Id | Operation | Name | ----------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | | * 1 | HASH JOIN | | | * 2 | HASH JOIN | | | * 3 | TABLE ACCESS FULL | SALES | | * 4 | TABLE ACCESS FULL | SALES | | * 5 | TABLE ACCESS FULL | SALES | ------------------------------------------------------------------------------------ Predicate Information (identified by operation id): ------------------------------------------------------------------------------------ 1 - access("T"."CUST_ID"="T2"."CUST_ID" AND "T1"."PROD_ID"="T2"."PROD_ID" AND "T1"."CUST_ID"="T2"."CUST_ID") 2 - access("T"."PROD_ID"="T1"."PROD_ID") 3 - filter("T1"."PROD_ID"<200 AND "T1"."AMOUNT_SOLD"*"T1"."QUANTITY_SOLD">10000 AND "T1"."PROD_ID"<>45) 4 - filter("T"."PROD_ID"<200 AND "T"."PROD_ID"<>45) 5 - filter("T2"."PROD_ID"<200 AND "T2"."PROD_ID"<>45) Explain plan table scan - exadata ----------------------------------------------------------------------------------- | Id | Operation | Name | ----------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | | * 1 | HASH JOIN | | | * 2 | HASH JOIN | | | * 3 | TABLE ACCESS STORAGE FULL | SALES | | * 4 | TABLE ACCESS STORAGE FULL | SALES | | * 5 | TABLE ACCESS STORAGE FULL | SALES | ------------------------------------------------------------------------------------ Predicate Information (identified by operation id): ------------------------------------------------------------------------------------ 1 - access("T"."CUST_ID"="T2"."CUST_ID" AND "T1"."PROD_ID"="T2"."PROD_ID" AND "T1"."CUST_ID"="T2"."CUST_ID") 2 - access("T"."PROD_ID"="T1"."PROD_ID") 3 - storage("T1"."PROD_ID"<200 AND "T1"."AMOUNT_SOLD"*"T1"."QUANTITY_SOLD">10000 AND "T1"."PROD_ID"<>45) filter("T1"."PROD_ID"<200 AND "T1"."AMOUNT_SOLD"*"T1"."QUANTITY_SOLD">10000 AND "T1"."PROD_ID"<>45) 4 - storage("T"."PROD_ID"<200 AND "T"."PROD_ID"<>45) filter("T"."PROD_ID"<200 AND "T"."PROD_ID"<>45) 5 - storage("T2"."PROD_ID"<200 AND "T2"."PROD_ID"<>45) filter("T2"."PROD_ID"<200 AND "T2"."PROD_ID"<>45) Case study J1 ------------------------------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads | ------------------------------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 14604 |00:20:21.00 | 957K| 2383K| |* 1 | HASH JOIN OUTER | | 1 | 99751 | 14604 |00:20:21.00 | 957K| 2383K| | 2 | VIEW | | 1 | 14604 | 14604 |00:00:00.14 | 12708 | 0 | | 3 | HASH GROUP BY | | 1 | 14604 | 14604 |00:00:00.14 | 12708 | 0 | | 4 | TABLE ACCESS STORAGE FULL | ODS_CP_ORDER_OM_DAILY_TEMP | 1 | 390K| 390K|00:00:00.11 | 12708 | 0 | | 5 | VIEW | | 1 | 2665K| 12200 |00:20:20.85 | 945K| 2383K| | 6 | HASH GROUP BY | | 1 | 2665K| 12200 |00:20:20.85 | 945K| 2383K| |* 7 | HASH JOIN | | 1 | 46M| 541K|00:09:55.83 | 945K| 2383K| | 8 | TABLE ACCESS BY INDEX ROWID | ODS_CP_QUOTATION_HEADERS | 1 | 10M| 10M|00:00:05.51 | 316K| 0 | | 9 | INDEX FULL SCAN | ODS_CP_QUOTATION_HEADERS_U1 | 1 | 10M| 10M|00:00:01.33 | 11771 | 0 | | 10 | VIEW | VW_GBC_5 | 1 | 46M| 559M|00:20:02.72 | 628K| 2368K| | 11 | HASH GROUP BY | | 1 | 46M| 559M|00:18:31.21 | 628K| 2368K| |* 12 | HASH JOIN | | 1 | 46M| 627M|00:02:08.95 | 628K| 615K| | 13 | VIEW | | 1 | 14604 | 14604 |00:00:00.13 | 12708 | 0 | | 14 | HASH GROUP BY | | 1 | 14604 | 14604 |00:00:00.13 | 12708 | 0 | | 15 | TABLE ACCESS STORAGE FULL| ODS_CP_ORDER_OM_DAILY_TEMP | 1 | 390K| 390K|00:00:00.10 | 12708 | 0 | | 16 | TABLE ACCESS STORAGE FULL | ODS_CP_QUOTATION_LINES | 1 | 180M| 180M|00:00:22.25 | 615K| 615K| ------------------------------------------------------------------------------------------------------------------------------------- The INDEX FULL SCAN prevents a Smart Scan on the table Smart Scan Pre-requisite There must be a full scan on an object FTS (TABLE ACCESS STORAGE FULL) INDEX_FFS (INDEX STORAGE FAST FULL SCAN) BITMAP INDEX SCAN (BITMAP INDEX STORAGE FAST FULL SCAN) The scan must use Oracles Direct Path Read mechanism Mechanism changed in 11g favoring Exadata If a table smaller than _small_table_threshold, the table will still be cached in SGA _small_table_threshold = 400M default Case study J1 Try performing a full table scan instead and compare the performance. If you have an INDEX() hint, remove it. If you have an RULE hint, remove it. Add a FULL hint / INVISIBLE index to force a full table scan. Case study J1 ------------------------------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads | ------------------------------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 14604 |00:20:21.00 | 957K| 2383K| |* 1 | HASH JOIN OUTER | | 1 | 99751 | 14604 |00:20:21.00 | 957K| 2383K| | 2 | VIEW | | 1 | 14604 | 14604 |00:00:00.14 | 12708 | 0 | | 3 | HASH GROUP BY | | 1 | 14604 | 14604 |00:00:00.14 | 12708 | 0 | | 4 | TABLE ACCESS STORAGE FULL | ODS_CP_ORDER_OM_DAILY_TEMP | 1 | 390K| 390K|00:00:00.11 | 12708 | 0 | | 5 | VIEW | | 1 | 2665K| 12200 |00:20:20.85 | 945K| 2383K| | 6 | HASH GROUP BY | | 1 | 2665K| 12200 |00:20:20.85 | 945K| 2383K| |* 7 | HASH JOIN | | 1 | 46M| 541K|00:09:55.83 | 945K| 2383K| | 8 | TABLE ACCESS BY INDEX ROWID | ODS_CP_QUOTATION_HEADERS | 1 | 10M| 10M|00:00:05.51 | 316K| 0 | | 9 | INDEX FULL SCAN | ODS_CP_QUOTATION_HEADERS_U1 | 1 | 10M| 10M|00:00:01.33 | 11771 | 0 | | 10 | VIEW | VW_GBC_5 | 1 | 46M| 559M|00:20:02.72 | 628K| 2368K| | 11 | HASH GROUP BY | | 1 | 46M| 559M|00:18:31.21 | 628K| 2368K| |* 12 | HASH JOIN | | 1 | 46M| 627M|00:02:08.95 | 628K| 615K| | 13 | VIEW | | 1 | 14604 | 14604 |00:00:00.13 | 12708 | 0 | | 14 | HASH GROUP BY | | 1 | 14604 | 14604 |00:00:00.13 | 12708 | 0 | | 15 | TABLE ACCESS STORAGE FULL| ODS_CP_ORDER_OM_DAILY_TEMP | 1 | 390K| 390K|00:00:00.10 | 12708 | 0 | | 16 | TABLE ACCESS STORAGE FULL | ODS_CP_QUOTATION_LINES | 1 | 180M| 180M|00:00:22.25 | 615K| 615K| ------------------------------------------------------------------------------------------------------------------------------------- The INDEX FULL SCAN prevents a Smart Scan on the table ALTER INDEX ODS_CP_QUOTATION_HEADERS_U1 INVISIABLE Candidate Indexes to Invisible for Smart Scan Candidate Indexes to Invisible: INDEX RANGE SCAN - Oracle is reading 0 or more contiguous rows from the index. INDEX FULL SCAN - Oracle is reading all rows from the index, and may be accessing these rows in the underlying table. INDEX SKIP SCAN - Oracle is reading 0 or more rows from different parts of the index, and may be accessing these rows in the underlying table. Generally Dont Invisible: UNIQUE INDEX UNIQUE SCAN - Oracle is reading 0 or 1 rows from the index. INDEX FAST FULL SCAN - Oracle is reading all rows from the index, and is not accessing these rows in the underlying table. ie. The index contains all columns required to resolve the query without having to lookup the table. Before and After removing the index() hint ------------------------------------------------------------------------------------------------------------------------------------ | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads | ------------------------------------------------------------------------------------------------------------------------------------ | 0 | SELECT STATEMENT | | 1 | | 14604 |00:03:32.09 | 740K| 615K| |* 1 | HASH JOIN OUTER | | 1 | 14604 | 14604 |00:03:32.09 | 740K| 615K| | 2 | VIEW | | 1 | 14604 | 14604 |00:00:00.14 | 12708 | 0 | | 3 | HASH GROUP BY | | 1 | 14604 | 14604 |00:00:00.14 | 12708 | 0 | | 4 | TABLE ACCESS STORAGE FULL | ODS_CP_ORDER_OM_DAILY_TEMP | 1 | 390K| 390K|00:00:00.11 | 12708 | 0 | | 5 | VIEW | | 1 | 25723 | 12200 |00:03:31.94 | 728K| 615K| | 6 | HASH GROUP BY | | 1 | 25723 | 12200 |00:03:31.94 | 728K| 615K| |* 7 | HASH JOIN | | 1 | 25723 | 548K|00:03:32.07 | 728K| 615K| | 8 | TABLE ACCESS STORAGE FULL | ODS_CP_QUOTATION_HEADERS | 1 | 10M| 10M|00:00:01.44 | 99833 | 0 | |* 9 | HASH JOIN | | 1 | 46M| 627M|00:02:03.78 | 628K| 615K| | 10 | VIEW | VW_GBF_14 | 1 | 14604 | 14604 |00:00:00.14 | 12708 | 0 | | 11 | HASH GROUP BY | | 1 | 14604 | 14604 |00:00:00.14 | 12708 | 0 | | 12 | VIEW | | 1 | 14604 | 14604 |00:00:00.14 | 12708 | 0 | | 13 | HASH GROUP BY | | 1 | 14604 | 14604 |00:00:00.13 | 12708 | 0 | | 14 | TABLE ACCESS STORAGE FULL| ODS_CP_ORDER_OM_DAILY_TEMP | 1 | 390K| 390K|00:00:00.10 | 12708 | 0 | | 15 | TABLE ACCESS STORAGE FULL | ODS_CP_QUOTATION_LINES | 1 | 180M| 180M|00:00:20.74 | 615K| 615K| ------------------------------------------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads | ------------------------------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 14604 |00:20:21.00 | 957K| 2383K| |* 1 | HASH JOIN OUTER | | 1 | 99751 | 14604 |00:20:21.00 | 957K| 2383K| | 2 | VIEW | | 1 | 14604 | 14604 |00:00:00.14 | 12708 | 0 | | 3 | HASH GROUP BY | | 1 | 14604 | 14604 |00:00:00.14 | 12708 | 0 | | 4 | TABLE ACCESS STORAGE FULL | ODS_CP_ORDER_OM_DAILY_TEMP | 1 | 390K| 390K|00:00:00.11 | 12708 | 0 | | 5 | VIEW | | 1 | 2665K| 12200 |00:20:20.85 | 945K| 2383K| | 6 | HASH GROUP BY | | 1 | 2665K| 12200 |00:20:20.85 | 945K| 2383K| |* 7 | HASH JOIN | | 1 | 46M| 541K|00:09:55.83 | 945K| 2383K| | 8 | TABLE ACCESS BY INDEX ROWID | ODS_CP_QUOTATION_HEADERS | 1 | 10M| 10M|00:00:05.51 | 316K| 0 | | 9 | INDEX FULL SCAN | ODS_CP_QUOTATION_HEADERS_U1 | 1 | 10M| 10M|00:00:01.33 | 11771 | 0 | | 10 | VIEW | VW_GBC_5 | 1 | 46M| 559M|00:20:02.72 | 628K| 2368K| | 11 | HASH GROUP BY | | 1 | 46M| 559M|00:18:31.21 | 628K| 2368K| |* 12 | HASH JOIN | | 1 | 46M| 627M|00:02:08.95 | 628K| 615K| | 13 | VIEW | | 1 | 14604 | 14604 |00:00:00.13 | 12708 | 0 | | 14 | HASH GROUP BY | | 1 | 14604 | 14604 |00:00:00.13 | 12708 | 0 | | 15 | TABLE ACCESS STORAGE FULL| ODS_CP_ORDER_OM_DAILY_TEMP | 1 | 390K| 390K|00:00:00.10 | 12708 | 0 | | 16 | TABLE ACCESS STORAGE FULL | ODS_CP_QUOTATION_LINES | 1 | 180M| 180M|00:00:22.25 | 615K| 615K| ------------------------------------------------------------------------------------------------------------------------------------- Exadata - SmartScan Before with good joining method: 14604 rows selected. Elapsed: 00:20:21.85 After using smart scan: 14604 rows selected. Elapsed: 00:03:32.90 Verify Smart Scan 10046 event trace Elapsed times include waiting on following events: Event waited on Times Max. Wait Total Waited ---------------------------------------- Waited ---------- ------------ SQL*Net message to client 976 0.00 0.00 SQL*Net message from client 976 256.36 304.17 asynch descriptor resize 13 0.00 0.00 gc cr multi block request 51 0.00 0.00 cell multiblock physical read 51 0.00 0.17 gc cr grant 2-way 18 0.00 0.00 cell single block physical read 18 0.00 0.00 reliable message 1 0.00 0.00 enq: KO - fast object checkpoint 2 0.00 0.00 cell smart table scan 728 0.00 0.19 Verify Smart Scan EXPLAIN PLAN / DBMS_XPLAN package Doesnt tell if Smart Scan really happen or not 10046 Trace cell smart table scan cell smart index scan V$SESSTAT / V$MYSTAT cell scans V$SQL Offload Eligible Bytes IO_CELL_OFFLOAD_ELIGIBLE_BYTES IO_INTERCONNECT_BYTES DBMS_SQLTUNE.REPORT_SQL_MONITOR MONITOR hint - Recommended Methodology 3. Is the SQL using smart scan? A. If yes, go to next step B. If no, invisible indexes or add FULL() hint or remove INDEX() or similar hints YES, go to next step Many Oracle developers - usually those working on OLTP systems - are told early in their careers that Full Table Scans are bad. Many will then hold on to this prejudice and never learn the truth. - Recommended Methodology Most common SQL problems that are easy to identify and easy to fix by below steps. 1. Are the underlying tables and indexes analyzed? A. If yes, go to next step B. If no, gather stats and re-test 2. Does the SQL already have optimized joining method? A. If yes, go to next step B. If no, check if inappropriate hint, if yes, remove / add it and re-test 3. Is the SQL using smart scan? A. If yes, go to next step B. If no, invisible indexes or add FULL() hint or remove INDEX() or similar hints 4. Does SQL has appropriate join predicates? A. If yes, go to next step B. If no, adding appropriate HINT 5. For low volume SQLs, are there any Full Table/Partition Scans? A. If yes, check if appropriate to add index B. If no, go to next step - Recommended Methodology 4. Does SQL has appropriate join predicates? A. If yes, go to next step B. If no, adding appropriate HINT Case Study H1 SQL Text ------------------------------ select /*+ FULL(ODS_OM_MTL_TRANSACTIONS) FULL() SQLID=4rf90vv0dbf8r */ t.transaction_id, t.transaction_date, t.organization_id, p.segment1 company_code, p.segment3 account_code, p.segment4 product_code, p.segment6 region_code, p.segment7 to_ic, r.segment1 item_code, s.subinventory_code, s.trx_source_line_id, tt.description, q.transaction_type_name transaction_type, s.currency_code, t.primary_quantity, t.base_transaction_value from ods_mtl_transaction_accounts t, ods_mtl_material_transactions s, ods_mtl_system_items_b r, ods_mtl_transaction_types q, ods_gl_code_combinations p, ods_oe_order_lines_all ol, ods_oe_order_headers_all oh, ods_oe_transaction_types_tl tt where tt.description like '%EMS%' and t.transaction_id = s.transaction_id and t.organization_id = s.organization_id and s.transaction_type_id = q.transaction_type_id and t.inventory_item_id = r.inventory_item_id and t.organization_id = r.organization_id and t.reference_account = p.code_combination_id and s.trx_source_line_id = ol.line_id and ol.header_id = oh.header_id and oh.order_type_id = tt.transaction_type_id -- and p.segment3 not in ('1260100', '1260200') and s.transaction_type_id in (33, 15) and t.transaction_date >= to_date('2011-01-01', 'YYYY-MM-DD') and t.transaction_date < to_date('2011-02-01', 'YYYY-MM-DD') and t.organization_id = 17221 This SQL is using Smartscan but can I speed it up? Case Study H1 Execution plan
SQL Plan Monitoring Details (Plan Hash Value=1317483436)
========================================================================================================================================================================================================================== | Id | Operation | Name | Rows | Cost | Time | Start | Execs | Rows | Read | Read | Mem | Activity | Activity Detail | | | | | (Estim) | | Active(s) | Active | | (Actual) | Reqs | Bytes | | (%) | (# samples) | ========================================================================================================================================================================================================================== | 0 | SELECT STATEMENT | | | | | | 1 | | | | | | | | 1 | NESTED LOOPS | | | | | | 1 | | | | | | | | 2 | NESTED LOOPS | | 15 | 181K | | | 1 | | | | | | | | 3 | HASH JOIN | | 15 | 181K | 17 | +6 | 1 | 0 | | | 4M | | | | 4 | TABLE ACCESS BY INDEX ROWID | ODS_MTL_SYSTEM_ITEMS_B | 5940 | 3931 | 22 | +1 | 1 | 107K | 19758 | 309MB | | 9.09 | Cpu (3) | | | | | | | | | | | | | | | cell single block physical read (17) | | 5 | INDEX RANGE SCAN | ODS_MTL_SYSTEM_ITEMS_B_N3 | 5940 | 36 | 17 | +6 | 1 | 107K | 615 | 10MB | | 0.45 | cell single block physical read (1) | | 6 | NESTED LOOPS | | | | | | 1 | | | | | | | | 7 | NESTED LOOPS | | 7510 | 177K | | | 1 | | | | | | | | 8 | HASH JOIN | | 7467 | 148K | 1 | +22 | 1 | 0 | | | 1M | | | | 9 | INDEX STORAGE FAST FULL SCAN | ODS_OE_TRANSACTION_TYPES_TL_N1 | 117 | 4 | 1 | +22 | 1 | 15 | | | | | | | 10 | HASH JOIN | | 56419 | 148K | 199 | +22 | 1 | 0 | | | 157M | 0.45 | Cpu (1) | | -> 11 | NESTED LOOPS | | | | 199 | +22 | 1 | 2M | | | | | | | 12 | NESTED LOOPS | | 56419 | 128K | 199 | +22 | 1 | 2M | | | | 0.45 | Cpu (1) | | -> 13 | NESTED LOOPS | | 56419 | 15041 | 199 | +22 | 1 | 2M | | | | | | | 14 | INLIST ITERATOR | | | | 1 | +22 | 1 | 1 | | | | | | | 15 | TABLE ACCESS BY INDEX ROWID | ODS_MTL_TRANSACTION_TYPES | 2 | 2 | 25 | +22 | 2 | 1 | | | | | | | -> 16 | INDEX UNIQUE SCAN | IDX_ODS_MTL_TRANS_TYPES_U1 | 2 | 1 | 199 | +22 | 2 | 2 | | | | | | | -> 17 | TABLE ACCESS BY INDEX ROWID | ODS_OM_MTL_TRANSACTIONS | 28210 | 14660 | 200 | +22 | 2 | | | | | | | | | | | | | | | cell single block physical read (116) | | 18 | INDEX RANGE SCAN | ODS_OM_MTL_TRANSACTIONS_N13 | 53721 | 379 | 199 | +22 | 2 | 2M | 4148 | 65MB | | 1.36 | Cpu (2) | | | | | | | | | | | | | | | cell single block physical read (1) | | 19 | INDEX UNIQUE SCAN | ODS_OE_ORDER_LINES_ALL_U1 | 1 | 1 | 199 | +22 | 2M | 2M | 2M | 218K | 3GB | | 61.82 | Cpu (20) | 85602 | 1GB | | 5.91 | Cpu (8) | | | | | | | | | | | | | | | cell list of blocks physical read (5) | | 20 | TABLE ACCESS BY INDEX ROWID | ODS_OE_ORDER_LINES_ALL | 1 | 2 | 199 | +22 | 3M | 2M | 414K | 6GB | | 20.45 | Cpu (12) | | | | | | | | | | | | | | | cell list of blocks physical read (31) | | | | | | | | | | | | | | | cell single block physical read (2) | | 21 | VIEW | index$_join$_007 | 3M | 17539 | | | | | | | | | | | 22 | HASH JOIN | | | | | | | | | | | | | | 23 | INDEX STORAGE FAST FULL SCAN | ODS_OE_ORDER_HEADERS_ALL_N3 | 3M | 4123 | | | | | | | | | | | 24 | INDEX STORAGE FAST FULL SCAN | ODS_OE_ORDER_HEADERS_ALL_N9 | 3M | 9185 | | | | | | | | | | | 25 | INDEX RANGE SCAN | ODS_MTL_TRANSACTION_ACCOUNT_N2 | 4 | 3 | | | | | | | | | | | 26 | TABLE ACCESS BY GLOBAL INDEX ROWID | ODS_MTL_TRANSACTION_ACCOUNTS | 1 | 4 | | | | | | | | | | | 27 | INDEX UNIQUE SCAN | ODS_GL_CODE_COMBINATIONS_U1 | 1 | 1 | | | | | | | | | | | 28 | TABLE ACCESS BY INDEX ROWID | ODS_GL_CODE_COMBINATIONS | 1 | 2 | | | | | | | | | | ========================================================================================================================================================================================================================== Table ods_oe_order_headers_all was executed 3M times. Why inappropriate execution plan in Case Study H1? Oracle's Cost Based Optimizer works by analyzing several of the possible execution paths for a SQL and choosing the one that it considers best. For instance, a two table join could drive off table A and lookup table B for each row returned, or it could drive off table B. By adding in the possibilities of join methods and index selection, the number of possible execution paths increases. Why inappropriate execution plan in Case Study H1? 2 tables : table A & table B Select * from A, B where A.a=B.a 1. Driving table A -> table B Or 2. Driving table B -> table A Why inappropriate execution plan in Case Study H1? 3 tables : table A & table B & table C Select * from A, B, C where A.a=B.a and B.b=C.b 1. Driving (table A -> table B result set) -> table C 2. Driving (table B -> table A result set) -> table C 3. Driving (table B -> table C result set) -> table A 4. Driving (table C -> table B result set) -> table A 5. Driving table A -> (table B -> table C result set) 6. Driving table C -> (table B -> table A result set) Why inappropriate execution plan in Case Study H1? A three table join has three times as many alternatives, a four table join has four times the alternatives of a three table join. In general, the number of possible execution paths for a join statement is proportional to n! (ie. n x n-1 x n-2 x ... x 2 x 1), where n is the number of tables in the join. No. of tables No. of possible execution plan Why inappropriate execution plan in Case Study H1? The problem of choosing the absolute best execution path becomes near impossible as n increases. Mathematicians call this an np-hard - or non- polynomial - problem. Why inappropriate execution plan in Case Study H1? If you have a table join (ie. a FROM clause) with five or more tables, and you have not included a hint for join order (eg. ORDERED or LEADING ), then Oracle may be joining the tables in the wrong order. How to fix it If the tables are being joined in the wrong order, you can supply a hint to suggest a better order. If Oracle is just starting with the wrong table, try a LEADING hint to suggest the best table to start with. SQLs with equi-joins will often get the rest of the joins right if only they know where to start. For ultimate control, update the FROM clause to list the tables in the exact order that they should be joined, and specify the ORDERED hint. Solution for Case Study H1 SQL Text ------------------------------ select /*+ FULL(S) LEADING(ol) full(ol) full(oh) LEADING(t) FULL(t) FULL(p) SQLID=4rf90vv0dbf8r */ t.transaction_id, t.transaction_date, t.organization_id, p.segment1 company_code, p.segment3 account_code, p.segment4 product_code, p.segment6 region_code, p.segment7 to_ic, r.segment1 item_code, s.subinventory_code, s.trx_source_line_id, tt.description, q.transaction_type_name transaction_type, s.currency_code, t.primary_quantity, t.base_transaction_value from ods_mtl_transaction_accounts t, ods_mtl_material_transactions s, ods_mtl_system_items_b r, ods_mtl_transaction_types q, ods_gl_code_combinations p, ods_oe_order_lines_all ol, ods_oe_order_headers_all oh, ods_oe_transaction_types_tl tt where tt.description like '%EMS%' and t.transaction_id = s.transaction_id and t.organization_id = s.organization_id and s.transaction_type_id = q.transaction_type_id and t.inventory_item_id = r.inventory_item_id and t.organization_id = r.organization_id and t.reference_account = p.code_combination_id and s.trx_source_line_id = ol.line_id and ol.header_id = oh.header_id and oh.order_type_id = tt.transaction_type_id -- and p.segment3 not in ('1260100', '1260200') and s.transaction_type_id in (33, 15) and t.transaction_date >= to_date('2011-01-01', 'YYYY-MM-DD') and t.transaction_date < to_date('2011-02- 01', 'YYYY-MM-DD') and t.organization_id = 17221 Added HINT execution plan With Smart Scan 470068 rows selected. Elapsed: ~30mins With Smart Scan + appropriate join 470068 rows selected. Elapsed: 498 sec - Recommended Methodology 4. Does SQL has appropriate join predicates? A. If yes, go to next step B. If no, adding appropriate HINT YES, go to next step - Recommended Methodology Most common SQL problems that are easy to identify and easy to fix by below steps. 1. Are the underlying tables and indexes analyzed? A. If yes, go to next step B. If no, gather stats and re-test 2. Does the SQL already have optimized joining method? A. If yes, go to next step B. If no, check if inappropriate hint, if yes, remove / add it and re-test 3. Is the SQL using smart scan? A. If yes, go to next step B. If no, invisible indexes or add FULL() hint or remove INDEX() or similar hints 4. Does SQL has appropriate join predicates? A. If yes, go to next step B. If no, adding appropriate HINT 5. For low volume SQLs, are there any Full Table/Partition Scans? A. If yes, check if appropriate to add index B. If no, go to next step - Recommended Methodology 5. For low volume SQLs, are there any Full Table/Partition Scans? A. If yes, check if appropriate to add / use index B. If no, go to next step High Level Guideline for using INDEX If you think Oracle should be using an index to resolve your query and it is not doing so, then make sure the index exists. the index status is USABLE Discuss with the DBA the prospect of adding a new index. Providing the index is efficient High Level Guideline for using INDEX Index is good when Used on a medium-large table (> 500 rows) as the outer table of a Nested Loop join. Used on a medium-large table (> 500 rows) in a Nested Sub-Query. Still remember the step by step tuning Methodology? - Recommended Methodology Most common SQL problems that are easy to identify and easy to fix by below steps. 1. Are the underlying tables and indexes analyzed? A. If yes, go to next step B. If no, gather stats and re-test 2. Does the SQL already have optimized joining method? A. If yes, go to next step B. If no, check if inappropriate hint, if yes, remove / add it and re-test 3. Is the SQL using smart scan? A. If yes, go to next step B. If no, invisible indexes or add FULL() hint or remove INDEX() or similar hints 4. Does SQL has appropriate join predicates? A. If yes, go to next step B. If no, adding appropriate HINT 5. For low volume SQLs, are there any Full Table/Partition Scans? A. If yes, check if appropriate to add index B. If no, go to next step - Recommended Methodology Most common SQL problems that are easy to identify and easy to fix by below steps. 6. Do you compare 9i, 10g and 11g execution plan? A. Compare each plans and bind the best to SPM Case Study C1 SQL is running good in 9i & 10g but not after upgrade to 11g Compare execution plan To get the same 9i execution plan in 11g, before run the SQL, SQL>alter session set "optimizer_features_enable"= '9.2.0.8'; To get the same 10g execution plan in 11g, before run the SQL, SQL>alter session set "optimizer_features_enable"= 10.2.0.5'; Compare execution plan If the execution plan in 9i is better than 11g, how can we ensure the SQL using 9i plan to have better performance? Added SQL>alter session set "optimizer_features_enable"= '9.2.0.8'; in program code ??? Smart developer learns and do thing in a smart way Smart Way Use SPM (SQL PLAN MANAGEMENT) for a statement, subsequent executions of that statement will use the SQL plan baseline. - Recommended Methodology Most common SQL problems that are easy to identify and easy to fix by below steps. 1. Are the underlying tables and indexes analyzed? A. If yes, go to next step B. If no, gather stats and re-test 2. Does the SQL already have optimized joining method? A. If yes, go to next step B. If no, check if inappropriate hint, if yes, remove / add it and re-test 3. Is the SQL using smart scan? A. If yes, go to next step B. If no, invisible indexes or add FULL() hint or remove INDEX() or similar hints 4. Does SQL has appropriate join predicates? A. If yes, go to next step B. If no, adding appropriate HINT 5. For low volume SQLs, are there any Full Table/Partition Scans? A. If yes, check if appropriate to add index B. If no, go to next step - Recommended Methodology Most common SQL problems that are easy to identify and easy to fix by below steps. 6. Do you compare 9i, 10g and 11g execution plan? A. Compare each plans and bind the best to SPM Additional SQL tuning idea Additional SQL tuning idea Select * from BL_PROJECT_CUST_DIM where PRIMARY_KEY_COLUMN = xxxxxxxx ----------------------------------------------------------------------- | Id | Operation | Name | E-Rows | ----------------------------------------------------------------------- | 0 | SELECT STATEMENT | | | | 1 | TABLE ACCESS STORAGE FULL| BL_PROJECT_CUST_DIM | 1 | |* 2 | UNIQUE INDEX SCAN | BL_PROJECT_CUST_DIM_U1 | 1 | ----------------------------------------------------------------------- Elapsed time : 0.0004sec A simple SQL using a primary key and only have 1 record Statistic are correct Using correct index What if this SQL in a CURSOR LOOP??? Additional SQL tuning idea Process time per record (sec) No .of record in loop Elapsed time (sec) 0.0004 10 0.004 0.0004 100 0.04 0.0004 1,000 0.4 0.0004 10,000 4 0.0004 100,000 40 0.0004 1,000,000 400 0.0004 10,000,000 4000 10046 Trace file in a cursor statement call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Execute 4 0.00 0.00 0 0 0 0 Execute 4 0.00 0.00 0 0 0 0 Execute 4 0.00 0.00 0 0 0 0 Execute 4 0.00 0.00 0 0 0 0 Execute 80 0.00 0.00 0 0 0 0 Execute 101407 6.77 6.77 0 0 0 0 Execute 101392 9.04 9.13 0 0 0 0 Execute 101407 13.60 13.78 0 0 0 0 Execute 101407 4.36 4.36 0 0 0 0 Execute 101407 9.22 9.06 0 0 0 0 Execute 101407 6.83 6.81 0 0 0 0 Execute 101407 11.23 11.13 0 0 0 0 Execute 101407 102.52 103.07 13 1234 721786 101407 Execute 22204 2.42 2.48 0 0 0 0 Execute 22089 6.11 6.28 6 5 177139 0 Execute 161265 6.69 6.72 0 0 0 0 Execute 161265 6.33 6.44 0 0 0 0 Execute 1 0.06 0.06 0 0 0 0 Execute 2 0.00 0.00 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Execute 139176 8.49 8.56 0 0 0 0 Execute 1 0.00 0.00 0 0 0 0 Execute 139176 302.03 306.74 4750 6 2091868 0 Execute 1 0.00 0.00 0 0 0 0 Execute 139176 272.18 280.70 6351 557250 151754 139176 Execute 1 0.00 0.00 0 0 0 0 Execute 1 0.00 0.00 0 62 0 0 Execute 1 0.00 0.00 0 1 14 1 Execute 1 0.00 0.00 0 0 0 0 Execute 1596000 772.70 793.85 49170 596532 3867306 341993 Long term tuning suggestions Rewrite cursor statement Too many loop cause by cursor statement Rewrite cursor loop logic to batch / single statement Effort to rewrite: High for existing, low for new POC on SP_BL_CC_INV_MTL_BK1 Elapsed time using cursor loop : 80min Elapsed time after rewrite : 20min Record processed : ~1,300,000 END