Ing. Eduardo Castro, PhD ecastro@grupoasesor.

net

http://ComunidadWindows.org

Session Objectives:
Get familiar with database performance tuning tools, especially what SQL Server 2008 offers Understand methodology of database performance bottleneck analysis Learn features/improvements of SQL Server 2008 that help improve performance

Target the most challenging and innovative SQL Server applications

10+ TB DW, 3k/tran/s OLTP, Large 500GB+ Cubes, Competitive migrations, Complex deployments, Server Consolidation (1000+)

Invest in large scale, referenceable SQL Server projects across the world
Provide SQLCAT technical & project experience Conduct architecture and design reviews covering performance, operation, scalability and availability Offer use of HW lab in Redmond with direct access to SQL Server development team

Performance Tuning Tools I/O Bottleneck CPU Bottleneck Memory Bottleneck

System Performance Monitor (Perfmon) SQL Profiler Trace Dynamic Management Views/Functions DBCC commands SQLDiag (Shipped with SQL 2005/2008) SQLIO, SQLIOSim KernRate Debugging tools (Windbg, DebugDiag,) More

Data Collector & Management Data Warehouse (MDW) Extended Events Activity Monitor New DMVs/DMFs

MDW Host

MDW Target Server

Snapshot Data

Client Workstation

SSIS Pkgs
Collection Set Metadata Job Metadata

dcexec.exe
Cache Files

Management Studio

MDW Reports

SQL Agent

msdb

Time line

Navigation control

Resource usage

Highly scalable with low overhead Future of SQL tracing

Built-in system_health session

select event_session_id, name, startup_state from sys.server_event_sessions

Session definition example

create event session session_error on server ADD EVENT sqlserver.error_reported (action (sqlserver.sql_text, sqlserver.tsql_stack, sqlserver.client_app_name) where error = 50001) ADD target package0.ring_buffer with (max_dispatch_latency = 1 seconds) go alter event session session_error on server state = start; go

Memory allocation tracking

sys.dm_os_memory_brokers sys.dm_os_memory_nodes

Reqource Governor
sys.dm_resource_governor_configuration sys.dm_resource_governor_resource_pools sys.dm_resource_governor_workload_groups

Query performance (hash value for query and plan)

sys.dm_exec_query_stats Sys.dm_exec_requests sys.dm_exec_procedure_stats sys.dm_exec_trigger_stats

More

Bottleneck A point or component of a process where throughput is the slowest. Understanding where time is spent
Elapsed Time = Running + Waiting

Key is the waiting - what type of wait(s)?

Monitor and identify I/O bottleneck

Resource Monitor & Performance Monitor MDW DMV & Extended Events

What is driving high I/O load?

Inefficient query plan Too much data to read/write Slow I/O subsystem

File level I/O throughput and disk response time

Runtime Performance monitoring

System Reports

What counters to look at?

Physical Disk Object:

Avg. Disk Queue Length Avg. Disk Sec/Read Avg. Disk Sec/Write %Disk Time Disk Transfers/Sec Avg. Disk Bytes/Read Avg. Disk Bytes/Write Disk Read Bytes/Sec Disk Write Bytes/Sec

SQL Server Buffer Manager Object: (memory pressure could cause I/O pressure)
Buffer Cache hit ratio Page Life Expectancy Checkpoint pages/sec Lazywrites/sec

Page I/O waits in Buffer pool

WriteLog and LogBuffer waits

High I/O Latch Wait Time

sys.dm_os_wait_stats sys.dm_exec_requests

pending/stalled I/O

sys.dm_io_pending_io_requests sys.dm_io_virtual_file_stats

ERRORLOG warnings of I/O delays

15 second delay in IO completion

This is way too slow

sqlos package async_io_requested async_io_completed wait_info wait_info_external

sqlserver package physical_page_read physical_page_write file_read file_write long_io_detected

2009-04-13 18:30:10.170 spid5s SQL Server has encountered 1 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [T:\tempdb\templog.ldf] in database [tempdb] (2). The OS file handle is 0x0000000000000670. The offset of the latest long I/O is: 0x000007eecac000 king longer than 15 seconds to complete on file [e:\myslowdb.mdf] in database [myslowdb] (1). The OS file handle is 0x00000438. The offset of the latest long IO is: 0x000000513ce000

Look if there is memory pressure Further narrow down to what queries are generating the I/O workload. Tune these queries. Explore data compression to reduce I/O load Upgrade storage hardware

Compress tables/indexes to reduce database size Compression ratio depends on schema & data distribution (avg 50-60%) Reduce query runtime I/O (increase cpu) Identify compression candidates
sp_estimate_data_compression_savings sys.dm_db_index_operational_stats

Sustained %Processor Time > 80%

Performance Monitor/Task Manager MDW Server Activity Report DMV & Extended Events

What is driving high CPU?

Recompilations Cursors Inefficient query plans More

Resource Monitor/Task Manager Performance Monitor Counters System Data Collector Reports Windows Performance Toolkit (XPerf)

Monitor it Live Watch Live or Log counters you like Formatted Perfmon counter based report Dig deeper using ETW

Processor object:
%Privileged Time %Processor Time %User Time
New in SQL Server 2008 for Resource Governor feature

Process object (SQL Server):

%Privileged Time %Processor Time %User Time

SQL Server:Workload Group Stats

CPU usage % Max request cpu time

SQL Server:Resource Pool Stats

CPU usage % CPU usage target %

SQL Server:SQL Statistics

Batch requests/sec SQL Compilations/sec SQL Re-Compilations/sec

SQL Server:Cursor Manager by Type SQL Server:Cursor Manager Total

Concurrent processing of multiple cursors could be cpu intensive

Total CPU waits

CPU cycles that did the real work

Wait in runnable queue

Thread voluntarily yields

Sys.dm_exec_sessions

sys.dm_exec_requests

sys.dm_exec_query_stats sys.dm_exec_procedure_stats sys.dm_exec_trigger_stats

Sys.sysprocesses

cpu_time

cpu_time

total_worker_time
cpu

total_scheduled_tim e

total_elapsed_time

total_elapsed_time

total_elapsed_time

query_hash

Sql_handle/query_h ash

Backward compatibility view

sqlos package scheduler_monitor_non_yielding_ring_buffer_recorded spinlock_backoff Wait_info wait_info_external

sqlserver package sql_statement_completed sp_statement_completed cursor_manager_cursor_end checkpoint_end

Extended Event can track individual statement level events like Profiler does with minimal impact ADD EVENT sqlserver.sql_statement_completed (action (sqlserver.sql_text) WHERE duration > 0), ADD EVENT sqlserver.sp_statement_completed (action (sqlserver.sql_text) WHERE duration > 0) select top 10 CONVERT(xml, event_data).value('(/event/data/value)[4]','int') as 'cpu'

High cpu doesnt necessarily mean its a problem. Drill down to specific workload and queries that drive cpu usage. Tune them. Use Resource Governor to protect high priority workloads Upgrade hardware

Resource Governor
allocate cpu/memory to prioritized workloads based on connection string parameters Mixed workloads: OLTP, Reporting, Maintenance

Soft Numa
Node configuration (soft numa node and cpu affinity) Network Configuration (soft numa node and tcp port)

Monitor and identify memory pressure

Performance Monitor/Task Manager MDW Server Activity Report DMV & Extended Events

What is eating up my memory?

External or internal memory pressure Inefficient query plan (hashing, sorting,)

AWE
32-bit uses AWE to cache data/index pages 64-bit uses AWE to implement Locked pages in memory

Environment 32-bit on 32-bit OS 32-bit on 64-bit OS (Wow) 64-bit on 64-bit OS

Virtual Address Space 2 GB 4 GB 8 TB

Max physical memory 64 GB 64 GB 2 TB

Perfmon is golden tool to identify memory pressure Memory object:

Available K/M bytes Commit Limit Paging File %Usage (Peak) Page reads/sec

Process object:
Private bytes Virtual bytes Working set

SQL Server Memory Manager object:

Connection Memory Lock Memory Optimizer Memory Memory Grants Pending
Somebody is waiting on memory

SQL Server Buffer Manager object:

Buffer cache hit ratio Checkpoint pages/sec Free list stalls/sec Free pages Lazy writes/sec Page life expectancy Stolen pages Target pages Total pages

How long will the page live?

SQL Server Plan Cache object SQL Server Resource Pool Stats object

Query execution waits on memory for sorting, hashing,

DBCC MEMORYSTATUS sys.dm_os_sys_memory

system_memory_state_desc a good indicator of available physical memory

sys.dm_os_process_memory
process_physical_memory_low = 1 indicates low physical memory. memory_utilization_percentage value below 100% indicates external memory pressure.

sys.dm_os_memory_clerks

sys.dm_os_ring_buffers
RING_BUFFER_SCHEDULER_MONITOR RING_BUFFER_RESOURCE_MONITOR RING_BUFFER_OOM RING_BUFFER_MEMORY_BROKER RING_BUFFER_BUFFER_POOL

sqlos package page_heap_memory_allocated page_heap_memory_freed memory_node_oom_ring_buffer_recorded

sqlserver package sort_memory_grant_adjustment plan_cache_cache_hit plan_cache_cache_attempt cursor_manager_cursor_memory_usage

2009-04-14 09:27:15.43 spid62 Failed allocate pages: FAIL_PAGE_ALLOCATION 1

Check memory configuration sp_configure External memory pressure Identify what other processes are competing Internal memory pressure Identify which SQL component is main consumer Workload/query tuning (sorting, hash join) Resource Governor Hardware upgrade

Focus on methodology no guess Correlate data points from multiple tools Performance tuning is iterative process bottleneck could move Resource contention impacts each other
Lack of memory -> high cpu & I/O Slow I/O -> tempdb contention

Ing. Eduardo Castro, PhD ecastro@grupoasesor.net

Troubleshooting Performance Problems in SQL Server 2008 http://msdn.microsoft.com/enus/library/dd672789.aspx