1

<Insert Picture Here>

MySQL Connector/Java: Best Practices & Tips for Java-based Product Developers
Todd Farmer <todd.farmer@oracle.com> Director, MySQL Engineering

The following/preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracles products remains at the sole discretion of Oracle.

DRIVING MySQL INNOVATION

MySQL Enterprise Monitor 2.2 MySQL Cluster 7.1 MySQL Cluster Manager 1.0 MySQL Workbench 5.2 MySQL Database 5.5 MySQL Enterprise Backup 3.5 MySQL Enterprise Monitor 2.3 MySQL Cluster Manager 1.1

MySQL Enterprise Backup 3.7 Oracle VM Template for MySQL Enterprise Edition MySQL Enterprise Oracle Certifications MySQL Windows Installer MySQL Enterprise Security MySQL Enterprise Scalability

MySQL Cluster 7.2 MySQL Cluster Manager 1.4 MySQL Utilities 1.0.6 MySQL Migration Wizard MySQL Enterprise Backup 3.8 MySQL Enterprise Audit MySQL Database 5.6

MySQL Cluster 7.3 DMR

All GA! All GA! Available Now!

All GA!

MySQL Database 5.6 DMR* MySQL Cluster 7.2 DMR MySQL Labs! (early and often)

A BETTER MySQL
2012-13

2010

2011
*Development Milestone Release

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

MySQL 5.6: Best Release Ever!

IMPROVED PERFORMANCE AND SCALABILITY
Scales to 48 CPU Threads Up to 230% performance gain over MySQL 5.5

IMPROVED INNODB
Better transactional throughput and availability

IMPROVED OPTIMIZER
Better query exec times and diagnostics for query tuning and debugging

IMPROVED REPLICATION
Higher performance, availability and data integrity

IMPROVED PERFORMANCE SCHEMA

Better Instrumentation, User/Application level statistics and monitoring

New! NoSQL ACCESS TO INNODB

Fast, Key Value access with full ACID compliance, better developer agility
Copyright 2012, Oracle and/or its affiliates. All rights reserved. 5

3,000+ ISV & OEM Customers

Network Management & Security

Hardware/Appliances/Devices

SaaS

Telecommunications

Software

Rely on MySQL
Copyright 2012, Oracle and/or its affiliates. All rights reserved. 6

Agenda
Fault Tolerant Deployments & Applications Extending Connector/Java Optimizing Performance Monitoring and Management Strategies Q&A
<Insert Picture Here>

Eliminating .JAR File Confusion

Install the JDBC driver correctly Once per-WEB-INF/lib (assuming J(2)EE) Once per Application Server Not both!

Integrate Logging
Use a logging framework Integrate Connector/J into it (category MySQL):
logger=Jdk14Logger logger=Slf4JLogger

Connection Thread Safety

Limited by MySQL client/server architecture
No multiplexing Connections are sessions

Thread Safety in Connector/Java

Recent improvements help reduce client-side deadlocks Avoid sharing Connection objects

10

Avoiding CommunicationExceptions
Configure wait_timeout server variable appropriately for maximum connection idle period
Use interactive_timeout if using interactiveClient C/J option)

Make sure socketTimeout is not set (or set sensibly high) Minimize duration connections left idle
Check connection pool connection testing policies Use lightweight validation query starting with /* ping /* Choose validation strategy wisely

Assume connection may be unusable

Explicitly validate or protect with retry logic

11

Dealing with Failed Connections

Dont clean up failed/dead connections with Connection.close() Use Connection.abort()
Added in JDBC-4.0 Shoots the connection dead, no locks, no niceties

Lobby your pool vendor to do the same

12

Understanding Automatic Reconnect

autoReconnect does not transparently negotiate network disconnects
Will always throw SQLException to the application Attempts reconnect so that Connection object can be used later

Potential inconsistent state

User variables Temporary tables Session-scoped server variables Transactional state

13

High Availability
HA does not mean never any Exceptions! HA means quick to recover and resume processing Application needs to be prepared to handle failover situations
Implement retry logic, if appropriate See earlier discussion of CommunicationExceptions for details

14

MySQL HA Deployment Options

How are your MySQL servers deployed?
Single server Semi-automated server failover Replication slave promotion O/S HA options Oracle VM or DRBD Windows Cluster Dual master/replication ring/active cluster

MySQL Server HA whitepapers:

http://mysql.com/products/enterprise/high_availability.html

15

Semi-Automated Failover
Only route to secondary server(s) if primary is offline
Use jdbc:mysql://master,secondary,[]/ URL format

Set failOverReadOnly=false if write operations are OK Will failover entail creating new Connection objects?
If so, leaving autoReconnect=false is fine

Failing back
secondsBeforeRetryMaster and queriesBeforeRetryMaster throttle how often Driver will try master again

16

Dual-Master, Ring, and Cluster

Use jdbc:mysql:loadbalance://host1,host2 JDBC URL failOverReadOnly is useless - all hosts are read/write Connection object retains physical connections, max one for each defined host (internal connection pool)
Make sure to check connection state after rebalancing opportunity

Host management:
loadBalanceBlacklistTimeout keeps hosts from being retried loadBalanceValidateConnectionOnSwapServer checks connections before choosing loadBalanceEnableJMX and loadBalanceConnectionGroup allow live manipulation of hosts
17

Agenda
Fault Tolerant Deployments & Applications Extending Connector/Java Optimizing Performance Monitoring and Management Strategies Q&A
<Insert Picture Here>

18

Extension Points
Lifecycle Interceptors Statement Interceptors Exception Interceptors Loadbalancing Strategies

19

Lifecycle Interceptors
com.mysql.jdbc.ConnectionLifecycleInterceptor Intercept JDBC method calls, return false to suppress driver standard behavior
commit() rollback() setAutoCommit() close() setCatalog()

20

Statement Interceptors
com.mysql.jdbc.StatementInterceptorV2 Override pre or post-execution method hooks
Return null if no changes Return implementation of ResultSetInternalMethods to override actual response from server.

Useful to change behavior of without application-level changes Can be chained for different use cases

21

Exception Interceptors
mysql.jdbc.ExceptionInterceptor Can change/observe exceptions In interceptException(SQLException, Connection)
Return a different exception Add information to exception, such as output from SHOW WARNINGS Do something related to given Exception, then return same exception E.G. SQLState 08007 Transaction resolution unknown, panic() or at least invalidate cache state

22

Load Balancing Strategies

com.mysql.jdbc.BalanceStrategy Key method is pickConnection()
Returns ConnectionImpl

Connector/J has several examples:

RandomBalanceStrategy BestResponseTimeBalanceStrategy

23

Agenda
Fault Tolerant Deployments & Applications Extending Connector/Java Optimizing Performance Monitoring and Management Strategies Q&A
<Insert Picture Here>

24

Optimize Disk Usage

Not everything from a MySQL image is needed
Tests

Keep core tools:

Mysqld Mysqld_safe Mysqladmin (although shutdown() is available in the JDBC driver) Mysqldump Mysqlbinlog Mysql_upgrade (plus mysqlcheck important later)

Investigate Innodb options

File-per-table Compressed tables
25

Optimizing Memory Usage

Connection instance is around 88K
40K of that buffers for I/O with the server 10K of that is information about the server

Pooling active connections is a win

As long as your pool is scalable The connections arent idle for very long

Caching server information is a win

cacheServerConfiguration=true Saves 4K/connection useDynamicCharsetInfo=false Saves 5K/Connection (if your code calls RSMD.getColumnDisplaySize())

26

Streaming Result Sets

MySQL returns all row data to the client Default mode of most clients (including JDBC) is to buffer entire result set in memory
OLTP/Webapp optimization

Timeseries data (graphs), ETL, bulk unload are possible use cases Caveat: Only one active query at a time perconnection! Server-side locks for the duration of the statement (or transaction)

27

Streaming Result Sets (cont)

Statement.setFetchSize(n > 0) does not work with MySQL Statement.setFetchSize(Integer.MIN_VALUE) causes row-at-a-time com.mysql.jdbc.Statement.enable/disableStreamingR esults() also works

28

Optimize JDBC Driver Configuration

Made easy with standard bundles Found in com/mysql/jdbc/configs directory
maxPerformance.properties solarisMaxPerformance.properties fullDebug.properties coldFusion.properties

Use as a reference starting point Include with useConfigs=maxPerformance

29

Caching Prepared Statements

Use cachePrepStmts=true Caches parsing for client-side prepared statements Caches references to server-side prepared statements Caches results of suitability check for server-side prepared statements Stored in a connection-specific HashMap of configurable size (prepStmtCacheSize, default 25), uses LRU purge SQL statements larger than prepStmtCacheSqlLimit (default 256 characters) will not be cached
30

Caching Callable Statements

Use cacheCallableStmts=true Caches parameter information by schema/SQL Stored in a connection-specific HashMap of configurable size (callableStmtCacheSize, default 100), uses LRU to purge Client pulls CallableStatement metadata from server
INFORMATION_SCHEMA SHOW CREATE mysql.proc table

31

Caching Server Configuration

Use cacheServerConfiguration=true Caches server-side state information
COLLATION VARIABLES

Cache is static (per JVM, not per Connection object)

Lives on even if MySQL Server is restarted! Requires application restart if MySQL Server configurations are changed

32

Caching Session State

Use useLocalSessionState=true Assumes use of standard JDBC operations:
setTransactionIsolation() setAutocommit() setCatalog()

Avoids roundtrip to confirm state with server

33

Caching Auto-Commit State

Use elideSetAutoCommits=true Prevents sending auto-commit state command to server if new state matches last server-reported state
Earlier Server versions have bug which returns state out of query cache, so not guaranteed to be accurate Avoids roundtrip to set state on server

34

Caching ResultSetMetaData
Use cacheResultSetMetaData=true Caches ResultSet metadata per SQL statement, per connection Default cache size is 50, uses LRU ResultSets must be static
no dynamic SQL in stored procedures

35

Disable Query Timeouts

Use enableQueryTimeouts=false Used in Statement.setQueryTimeout()
Starts second thread to establish second physical connection and issue KILL statement against slow connection

Uses RAM, even when timeout threshold is not reached Disable if not using Statement.setQueryTimeout() in high-load environment

36

Disable Connection Attributes

Use connectionAttributes=none Disables sending connection attributes to MySQL Server 5.6 (or higher) Performance only affects new connection creation only likely not meaningful for pooled connections
Very roughly 10% performance penalty (tested on ancient hardware) Measured average connection time at 7.7ms vs. 6.8ms.

37

Disable Query Timeouts

Use enableQueryTimeouts=false Used in Statement.setQueryTimeout()
Starts second thread to establish second physical connection and issue KILL statement against slow connection

Uses RAM, even when timeout threshold is not reached Disable if not using Statement.setQueryTimeout() in high-load environment

38

Agenda
Fault Tolerant Deployments & Applications Extending Connector/Java Optimizing Performance Monitoring and Management Strategies Q&A
<Insert Picture Here>

39

Optimizing for Managability Bad Usage

Pre-production
useUsageAdvisor=true Uses ProfilerEventHandler (default dumps to configured logger) You can implement your own, implement com.mysql.jdbc.profiler.ProfilerEventHandler ProfilerEvent.TYPE_WARN (for Usage Advisor events)

40

Managability - Slow Queries

Pre-or-Post Production
logSlowQueries=true (optionally with autoSlowLog=true) ProfilerEvent.TYPE_PREPARE/EXECUTE/FETCH

Note, all profiler events have:

Event time Event duration Thread stack of event occurrence

41

Monitoring Performance

42

What to Collect
CPU (User, Sys, Idle) Memory (real, virtual, swap, per-important-process?) Throughput (http, sql) Service times (95 %-ile if possible) Slow SQL, app units of work (requests) I/O (disk, network) JVM (GC) MySQL (Connections, SQL, Sorts, Temp Tables, Buffer Pool)

43

How to Collect It
MySQL Enterprise Monitor(MEM) Already in-place monitoring tools iostat, sar, recorded to database tables Servlet filters (response times, counts)

44

MEM Query Analysis

Find hotspots quickly
Where is most time spent in doing database work? Which operations are most frequently executed? Which operations return the most rows? What operations are executing when MySQL is busiest?

Use Connector/Java plugin to get application stack trace for problematic/expensive/frequent statements Get EXPLAIN output for sample queries

45

MEM Query Analysis

46

MEM Query Analysis

47

Agenda
Fault Tolerant Deployments & Applications Extending Connector/Java Optimizing Performance Monitoring and Management Strategies Q&A
<Insert Picture Here>

48