Академический Документы
Профессиональный Документы
Культура Документы
Dave Beulke
Joe Burns
Dan Luksetich
Craig S. Mullins
Anne Stevens
Julian Stuhler
Brought to you by
DBAzine.com
DBAzine.com
ii
DBAzine.com
Table of Contents
A DB2 for z/OS Performance Roadmap..........................................................................................1
Tuning the System.......................................................................................................................1
No Magic Bullet.......................................................................................................................2
Tuning the Databases ..................................................................................................................3
Creating Indexes in the Dark.......................................................................................................5
Tuning the Applications ..............................................................................................................6
The Bottom Line .........................................................................................................................7
Can Your Left Outer Joins Perform Better? ....................................................................................9
Introduction .................................................................................................................................9
Left Outer Joins the Good and the Bad ....................................................................................9
A Simple Example of a Left Outer Join ....................................................................................10
Leveraging the ON Clause for Performance Benefits ...........................................................10
The ON Clause Predicate Does Not Affect the Result Set....................................................11
Benchmarking the Performance Improvement..........................................................................12
Design Considerations and Where to Use This Technique.......................................................15
Conclusion.................................................................................................................................15
Notes:.........................................................................................................................................16
Utilizing DB2 UDB for OS/390 and z/OS UNION Everywhere for Large Table Design ............17
Introduction ...............................................................................................................................17
Increasing Table Size and Concurrency with Union in View ...................................................17
Partitioning Tablespaces............................................................................................................18
The UNION Solution ................................................................................................................19
Predicate Distribution................................................................................................................20
Query Block Pruning.................................................................................................................21
Predicate Transitive Closure Applied to Joins and UNION in View........................................23
Limitations to Union Everywhere .............................................................................................24
Join Impact on UNION in View................................................................................................26
The Role of UNION in View in V8 ..........................................................................................29
Conclusion.................................................................................................................................29
REORG Redux: A Review of the DB2 REORG Utility................................................................31
Introduction ...............................................................................................................................31
What Is REORG For?................................................................................................................31
How Does REORG Work?........................................................................................................34
When Should You Run REORG? .............................................................................................35
Conclusion.................................................................................................................................37
DB2 Buffer Pool Essentials ...........................................................................................................39
Memory Basics..........................................................................................................................39
DBAzine.com
iii
iv
DBAzine.com
DBAzine.com
vi
DBAzine.com
Forward
One of the biggest problems faced by companies today is keeping applications running at
peak efficiency. It can be a never-ending task to diligently keep collecting, examining, and
reacting to performance-oriented statistics. This is certainly the case for DBAs charged with
maintaining optimal performance of DB2 databases and applications.
DB2 performance is a three-headed beast encompassing the program logic and SQL in your
applications, the database objects being accessed, and the database system itself all of which
must be tamed to assure optimal performance. Many experts agree that inefficient SQL is
probably the number one cause of poor DB2 performance. But that doesnt mean you can
safely neglect the state of your database objects or your DB2 subsystems.
Monitoring and managing these three aspects of performance can be a full-time job. Further
complicating this job is the constantly changing state of DB2. Just when you think you have
mastered one aspect of DB2, along comes that new release or PTF that changes everything.
Fortunately, you have this DB2 for z/OS Performance Primer in your hands to keep you upto-speed with DB2 performance issues. If you read the articles contained in this primer you
will be better armed to combat the three-headed DB2 performance beast.
Start off with the article I wrote titled A DB2 for z/OS Performance Roadmap. This article
offers up a high-level overview of DB2 performance issues and will start you on your way to
becoming a DB2 performance expert. Then move on to learn more about solving DB2
application performance issues by reading Joe Burns Can Your Left Outer Joins Perform
Better? Joe offers up a wealth of information about squeezing optimal performance out of
outer joins. We follow this with Dan Luksetichs Utilizing UNION Everywhere for Large
Table Design. In this informative article Dan teaches us how to bridge the gap between
application and database performance by targeting a newer SQL feature to simplify the design
of large DB2 tables.
Then we take full aim at your database objects with REORG Redux: A Review of the DB2
REORG Utility by Julian Stuhler. This article takes you through the paces of when, why, and
how to reorganize your DB2 databases to achieve optimal database performance. And we
dont neglect the system aspect of DB2 performance either. Anne Stevens discusses DB2
Buffer Pool Essentials for tuning your subsystem performance.
DBAzine.com
vii
Finally, Dave Beulke wraps it all up with his insightful article titled The Key to Performance:
Business Process Analysis. In it he discusses how to achieve business objectives using
business process analysis and better not bigger databases.
So, lets start our journey into the world of DB2 performance management
Craig S. Mullins
Director, Technology Planning
BMC Software
viii
DBAzine.com
For example, when DB2 data is accessed using CICS, multiple threads can be active
simultaneously, giving multiple users concurrent access to a DB2 subsystem of a single CICS
region. A mechanism named the CICS Attach Facility connects CICS with DB2. Using the
CICS Attach Facility, you can connect each CICS region to only one DB2 subsystem at a
time. You can connect each DB2 subsystem, however, to multiple CICS regions
simultaneously. DB2 provides services to CICS via MVS TCBs. All of these TCBs reside in
the CICS address space and perform cross-memory instructions to execute the SQL code in
the DB2 database services address space (DSNDBM1).
Furthermore, the resource control table (RCT) must be configured for each DB2 program that
will run under CICS. The RCT applies only to CICS transactions that access DB2 data; it
defines the manner in which DB2 resources will be used by CICS transactions. In particular,
the RCT defines a plan for each transaction that can access DB2.
Additionally, it defines parameters detailing the number and type of threads available for
application plans and the DB2 command processor.
DB2 DBAs also must ensure that appropriate DB2 system parameters are set using DB2
commands and DSNZPARMs. One of the most important areas for tuning here is memory
usage. DB2 uses memory for buffer pools, the EDM pool, RID pool, and sort pools to cache
data and structures in memory. The better memory is allocated to these structures, the better
DB2 will perform.
When allocating DB2 buffer pools, keep these rules in mind:
Dont allocate everything to a single buffer pool (e.g., BP0); use a multiple buffer pool
strategy.
Explicitly specify a buffer pool for every tablespace and index.
Isolate the DB2 Catalog in BP0; put user and application DB2 objects into other buffer
pools.
Consider separating indexes from tablespaces with each in their own dedicated buffer
pools.
Consider isolating heavily hit data into its own buffer pool to better control performance.
Consider isolating sorts into a single buffer pool and tuning for mostly sequential access
(e.g., BP7).
Consider separating DB2 objects into separate buffer pools that have been configured for
sequential verses random access.
No Magic Bullet
Forget about trying to follow a cookie-cutter approach to buffer pool management. Every
shop must create and optimize a buffer pool strategy for its own data and application mix.
DBAzine.com
DB2 offers the following buffer pool tuning knobs that can be used to configure virtual
buffer pools to the type of processing they support:
DWQT This value is the deferred write threshold; it is expressed as a percentage of
the virtual buffer pool that might be occupied by unavailable pages. When this threshold
is reached, DB2 will start to schedule write I/Os to externalize data. The default is 50
percent, which is likely to be too high for most shops.
VDWQT This value is the vertical deferred write threshold; it is basically the same as
DWQT, but for individual data sets. The default is 10 percent, which once again is quite
likely to be too high for many shops.
VPSEQT This value is the sequential steal threshold; it is expressed as a percentage
of the virtual buffer pool that can be occupied by sequentially accessed pages. Tune
buffer pools for sequential access such as scans and sorting by modifying VPSEQT to a
larger value. The default is 80 percent.
VPPSEQT This value is the sequential steal threshold for parallel operations; the
default value is 50 percent.
VPXPSEQT This value is assisting parallel sequential threshold; it is basically the
VPPSEQT for operations from another DB2 subsystem in the data sharing group.
These parameters can be changed using the ALTER BUFFERPOOL command. Additionally,
hiperpools can be created to back up DB2 virtual buffer pools with additional memory. DB2
provides several tuning knobs for hiperpools, too, including HPSIZE to adjust the size of
hiperpools and HPSEQT to adjust the hiperpool sequential steal threshold.
The EDM pool is used for caching internal structures used by DB2 programs. This includes
DBDs, SKCTs, CTs, SKPTs, and PTs. It also includes the authorization cache for plans and
packages, as well as the cache for dynamic SQL mini-plans. As a general rule, you should
shoot for an 80 percent hit rate with the EDM pool; this means that in only one out every five
times should a structure need to be loaded from disk into the EDM pool.
Finally, remember that buffer and EDM pool tuning are in-depth subjects that cannot be
adequately covered in a high-level article such as this. Additionally, there is much more to
proper DB2 system performance tuning than allied agent and memory tuning. Other system
elements requiring attention include: locking, logging, and Parallel Sysplex configuration and
management for DB2 data-sharing shops.
database is not optimally organized or stored, the data it contains will be difficult or slow to
access. The performance of every application that requires this data will be negatively
impacted.
The first component of database optimization is assuring an optimal database design.
Database design is the process of transforming a logical data model into a physical database
design and then implementing the physical model as an actual database. Proper database
design requires up-front data modeling and normalization in other words, a logical data
model is required before you can even begin to design a physical database. Assuming a
logical model exists, it should be used as the basis for creating the physical database. The
physical database is created by transforming the logical data model into a physical
implementation based on an understanding of the DBMS to be used for deployment. But a
one-to-one mapping of logical entity to physical table is unlikely to result in an optimal
physical database design.
Successfully developing a physical database design requires a good working knowledge of
DB2s features. More precisely, the DBA must possess in-depth knowledge of physical
database objects supported by DB2 as well as the physical structures and files required to
support those objects. This must include knowledge of the manner in which DB2 supports
indexing, referential integrity, constraints, data types, and other features that augment the
functionality of database objects. Armed with the correct information, the DBA can create an
effective and efficient database from a logical data model.
The DBA creating the physical DB2 database implementation should keep these rules of
thumb in mind when building the databases:
Keep the physical database as normalized as possible. However, performance should win
out over aesthetics. In other words, dont let data modelers dictate physical design. If
you need to reorder columns in the table to optimize logging, do it.
In general, put each table in its own tablespace. Exceptions can be made for very small
tables such as code and reference tables.
In general, favor partitioned and segmented tablespaces over simple tablespaces. And
dont specify a DSSIZE greater than 4GB unless you really need a large tablespace
(doing so will waste space).
Dont create base table views.
Use NULL sparingly.
Use appropriate DB2 data types (e.g., use DATE for dates instead of CHAR or numeric
data types).
Consider using DB2 compression instead of using VARCHAR columns. With
compression, there is less overhead and no programmatic handling is required.
Favor using DB2 declarative RI instead of programmatic RI. It will usually perform
better and is easier to administer. However, its generally a bad idea to use RI for lookup
and reference tables.
DBAzine.com
Avoid the DDL defaults they are usually wrong. Explicitly code every single DDL
option when creating DB2 objects.
Calculate the appropriate amount of free space for both PCTFREE and FREEPAGE
based on the frequency of modification. Do not just default every tablespace to 10
percent free space. DB2 DBAs should always keep in mind that a proper indexing
strategy can be the primary factor to ensure optimal performance of DB2 applications.
First, take care of unique and primary key constraints by creating unique indexes on those
columns. Then consider creating indexes for each foreign key to optimize RI.
The next step is to examine the most heavily used queries. Look at the predicates and
build indexes to support these queries. Consider overloading indexes by adding columns
to encourage index only access.
Next, examine queries that invoke sorts. Look for ORDER BY, GROUP BY, UNION,
and SELECT DISTINCT. Consider building indexes to support these clauses so DB2 can
avoid initiating an expensive sort. Look at your indexes and make sure youve chosen the
first column wisely. In general, the first column of the index should have a high
cardinality.
In general, consider avoiding indexes on variable columns because DB2 will expand the
VARCHAR column to its full length in the index. After coming up with a first stab
indexing strategy consider the insert, update, and delete implications of those indexes. If
the columns of those indexes must be modified in any way, DB2 will incur additional
overhead keeping the indexes up-to-date.
Over time, as data is modified and updated, DB2 will have to move the data around within the
database. Such activity causes the data to become fragmented and inefficiently ordered. The
longer the database remains online and the more changes made to the data, the more
inefficient database access can become. To overcome disorganized and fragmented databases,
the DBA can run a reorganization utility to refresh the data and make the database efficient
once again. But the key to successful reorganization is to reorganize only when the database
requires it; instead, some companies over-reorganize by scheduling regular database
reorganization jobs to be run whether the database is fragmented, or not. This wastes valuable
CPU cycles.
But reorganization is only one of many database performance tasks performed by the DBA.
DBAzine.com
Others include data set placement, partitioning for parallel access, managing free space, and
assuring optimal compression.
Let SQL do the work instead of the application program. For example, code an SQL join
instead of two cursors and a programmatic join.
Simpler is generally better, but complex SQL can be very efficient.
Retrieve only the columns required, never more.
Retrieve the absolute minimum number of rows by specifying every WHERE clause that
is appropriate.
When joining tables, always provide join predicates. In other words, avoid Cartesian
products.
Favor using Stage 1 and Indexable predicates.
Avoid sorting if possible by creating indexes for ORDER BY, GROUP BY, and
DISTINCT operations.
Avoid black boxes - that is, avoid I/O routines that are called by programs instead of
using embedded SQL.
Avoid deadlocks by updating tables in the same sequence in every program.
Issue data modification statements (INSERT, UPDATE, DELETE) as close as possible to
the COMMIT statement as possible.
Be sure to build a COMMIT strategy into every batch program that changes data. Failing
to COMMIT can cause locking problems.
Every DBMS provides a method of inspecting the actual access paths that will be used to
satisfy SQL requests. The DBA must thoroughly understand the different types of access
paths and know which ones are best in a given situation. For DB2, the DBA must be able to
interpret the output of the access path explanation produced by EXPLAIN. This information
is encoded in the PLAN_TABLE and must be interpreted. To make matters more difficult, the
PLAN_TABLE doesnt contain 100 percent of the information required to determine if the
SQL will perform efficiently. The DBA (and, indeed, programmers too) must be able to read
the PLAN_TABLE in conjunction with the SQL code, host language code, and information
from the DB2 catalog to judge the efficiency and effectiveness of each SQL statement.
DBAzine.com
Host language code refers to the application programs written in C, COBOL, Java, Visual
Basic or the programming language du jour. SQL statements are usually embedded into host
language code and it is quite possible to have finely tuned SQL inside of inefficient host
language code. And, of course, that would cause a performance problem.
DBAzine.com
DBAzine.com
Introduction
Have you ever heard the saying, The best performing SQL statement is the one that never
executes? This means that if you can avoid doing an unnecessary SQL call, then you have
saved all of the resources that would have been used by that call. But there should be a similar
guideline that says, The best performing Left Outer Join is the one that never does a join
except when it should.
Left Outer Joins have been part of DB2 for a while now, but starting in V6, the join
processing became much more sophisticated. Changes were made in how the ON predicates
are evaluated, and we can take advantage of those changes to improve the performance of our
Left Outer Joins. This article covers a technique that we are using in our shop to significantly
reduce the CPU costs of our most expensive Left Outer Joins.
DB2 doesnt find a child row. This results in a lot of fruitless access to the child table looking
for rows that often are not going to be there. In a high-volume, high-performance application,
this can be a real problem. Spending resources looking for child rows that usually are not
there is far from an ideal design.
But are there actual alternatives to the issue? After all, how can DB2 determine whether a
child row does or doesnt exist unless it actually goes to the child table and looks? Believe it
or not, this can be done using careful database design and well-coded Left Outer Joins.
This works, and it will return the desired rows and columns. However, keep in mind that most
of the time there is no matching COMPLAINT row. Despite this, the SQL is still constantly
probing the COMPLAINT table looking for one. If our customer service area is a highvolume application, then this SQL is spending a lot of valuable CPU time looking for
COMPLAINT rows that often are not there.
Fortunately, due to the way that Left Outer Joins are processed (V6 and above), there is an
alternative available to us. The ON clause can accept predicates that allow DB2 to eliminate
the unsuccessful accesses to the child table. The key concept to remember is that the ON
clause tells DB2 how to join the tables together, but it does not provide filtering. This sounds
a little cryptic, but if we understand it, we can leverage it to our benefit. The best way to see
this is by going back to our example.
DBAzine.com
to include the indicator in the ON clause. It is essential that this indicator is added to the
ON clause and not in the WHERE clause.
SELECT CU.cols.., CP.cols..
FROM CUSTOMER
CU LEFT OUTER JOIN
COMPLAINTS CP ON CU.CUST_ID = CP.CUST_ID
AND CU.COMPLAINT_IND = Y
WHERE CU.CUST_NAME LIKE %ABC%
This small change will not affect the actual result set in any way; however, it will dramatically
affect the way the Left Outer Join performs. Remember, the key concept was that the ON
clause tells DB2 how to join the tables, but does not provide filtering. Because it does not
filter the data in any way, the CU.COMPLAINT = Y predicate in the ON clause has no
effect on the final result set. In the example, DB2 will return a row whether or not the
COMPLAINT_IND = Y. The predicate in the ON clause is simply ignored when it
comes to determining the final result. This can seem very confusing, because as SQL coders,
it looks very much like a filtering predicate; consequently, we think that it is going to limit the
result set in some way. But it is not, and we must remember this important detail.
If we, on the other hand, had actually added a true filtering predicate in the WHERE clause,
then we would have a serious problem with the result set.
SELECT CU.cols.., CP.cols..
FROM CUSTOMER
CU LEFT OUTER JOIN
COMPLAINTS CP ON CU.CUST_ID = CP.CUST_ID
WHERE CU.CUST_NAME = ABC Company
DBAzine.com
11
In this example, we have added a true filtering predicate in the WHERE clause. This is not
at all what we wanted, since this will only return rows for customers if they have complaints.
We really do not need a customer service application that only returns data for those
customers who complain all of the time. Instead, we prefer that it return a row whether or not
a customer had a complaint.
12
DBAzine.com
without the additional ON predicate. As you can see the Left Outer Join with the additional
ON predicate is clearly the better performer.
DBAzine.com
13
Using the SQL with the ON predicate results in a very low number of Getpages against the
child tables index. This is because DB2 only accesses the child table if the indicator is set on
the parent. In fact, the Getpage activity literally drops off the chart as the number of child
rows decreases. On the other hand, the Getpages are consistently very high in the SQL
without the ON predicate. This is because DB2 has to check the child table every time to
determine if a child row exists. Note that the Getpage activity is still declining somewhat, but
this is only due to the reduction in the size and number of levels in the child index.
From the benchmarks, we can see that there are obvious performance savings from this
approach. However, another less elegant but somewhat simpler solution may jump to mind. If
we are putting an indicator on the CUSTOMER table that tells us whether a complaint exists
or not, then why bother doing a Left Outer Join at all? Why not just read the customer row
and, if the complaint indicator is a Y, go read the COMPLAINT table. Much like the Left
Outer Join, it would seem that we are now avoiding those fruitless lookups to the child table.
After all, we can check the complaint indicator just as easily as DB2 did.
And indeed, this is an option that should be considered and benchmarked since there is the
possibility that it is actually faster than letting DB2 do the check as part of a Left Outer Join.
Joins of any type are not free, and CPU cycles must be used to perform them.
In fact, in the extremely simple example covered in this article, it very well may execute
faster as two separate SQL calls. However, at our shop, we have found that our real-world
situations are not nearly as simple as the example we covered. As a result, more often than
not, the Left Outer Join outperforms the two separate SQL calls. The main reason is that there
will be overhead associated with each separate SQL call since both would need to be coded as
14
DBAzine.com
cursors. And, of course, each cursor would require a separate open, fetch, and close statement.
So, as the number of times the second SQL call needs to be executed goes up, it eats into any
potential performance savings. The Left Outer Join with the additional ON predicate does
not suffer from this degradation since it is all done in a single SQL call (one cursor).
Conclusion
We have seen that there are definite performance advantages to using this technique. We can
save CPU and reduce Getpages by adding some simple predicates in the ON clause. An
added benefit is that it often simplifies code since Left Outer Joins allow us to combine access
to tables in a single SQL. So, to take full advantage of the powerful functionality offered by
the Left Outer Join, consider using this interesting technique.
DBAzine.com
15
Notes:
1. The benchmarks used involved a Left Outer Join that was using a Nested Loop Join
method as the access path. If DB2 had selected a Merge Scan Join access path instead,
there would be no performance benefits to this technique. This is because the Merge Scan
never probes the child index like the Nested Loop join; instead, it sorts and scans the child
table.
2. In the key concepts, it was shown that the ON predicate does not affect the result set.
This is true because the ON predicate is referencing the table on the left (the parent
table). If the ON predicate were specified for the table on the right (the child table), then
it could have affected the result set. This is because DB2 will apply the child ON
predicate to the child table during the join, which may affect the final result set.
3. All benchmarks were performed using DB2 for z/OS version 7.
16
DBAzine.com
Introduction
Around the time of the introduction of DB2 for OS/390 and z/OS version 7, many people
began talking about the concept of utilizing the new UNION everywhere feature to
accommodate some of our large table designs. This new feature of DB2 allowed UNIONs to
be placed inside of nested table expressions, subqueries, and views. If we could put a UNION
clause into a view, then that view could represent a large table to our applications. This view
could exceed the current table size limits, as well as some of our current large table
concurrency issues.
The idea of storing the contents of a single DB2 table in many physical tables is nothing new.
There are situations in which the amount of data that can be stored in a single table exceeds
the limit (in V7, thats 16TB), or other situations in which data needs to be separated into
many tables for improved concurrency, data organization, or legal issues. In these situations,
the data is divided into several identical tables by applying some sort of business rule to
determine which data goes into which table. When the data is separated in this manner, it is
more difficult for application programs to access the data when the data required spans more
than one table. Clever finder tables have been invented to direct application queries to the
appropriate table where data is stored, or package switching techniques have been established
to direct SQL statements within an application to the appropriate DB2 table. These techniques
lead to additional database or programming complexity.
If we can now create a view against all of the underlying tables, that view could contain a
reference to each of the appropriate underlying tables, with the appropriate business rule
directing access to the table or tables of interest. This would eliminate any finder tables and
package switching algorithms from our applications, and simplify access to these underlying
tables. Programs and SQL can now access the view as if it were a normal table. Doing this is
very desirable; it simplifies managing these large objects, and the programming required to
access them. However, there are always trade-offs, and users must take care to avoid the
limits and pitfalls associated with UNION in view.
17
Partitioning Tablespaces
Traditionally, our larger database tables have been placed into partitioned tablespaces. A
partitioned tablespace allows us to physically distribute our table data across several physical
datasets. Partitioning is determined by key values of the partitioning index, which is the index
used to determine both the clustering and the partitioning of the data. The database engine to
distribute the data across the partitions uses distinct key value ranges. Partitioning helps with
database management because its easier to manage several small objects versus one very
large object. DB2 utilities can act on individual partitions, and the datasets representing each
of the partitions can be distributed across many DASD units. Tables in partitioned tablespaces
can be read using SQL statements much in the same way as tables in segmented or simple
tablespaces. Updates to tables in partitioned tablespaces can sometimes be more difficult if
the update affects the partitioning key.
There are still some limits to partitioning. For example, each partition is limited to a
maximum size of 64GB, a partitioning index is required (this changes in DB2 for z/OS V8),
and if alternate access paths to the data are desired, then non-partitioning indexes (NPIs) are
required. These NPIs are not partitioned, and exist as single, large database indices. Thus,
NPIs can present themselves as an obstacle to availability (i.e., a utility operation against a
single partition may make the entire NPI unavailable), and as impairment to database
management since it is more difficult to manage such large database objects.
DBAzine.com
those access paths, and those NPIs could be extremely large and hard to manage. If we want
to reorganize a partition, or redistribute the data across the partitions, an outage will probably
be required to do these things. But in our highly available world, such outages may not be
tolerated.
DBAzine.com
19
We can add or remove tables with very small outages, usually just the time it takes to
drop and recreate the view.
We can partition each of the underlying tables, creating still smaller physical database
objects.
NPIs on each of the underlying tables could be much smaller and easier to manage than
they would under a single table design.
Utility operations could execute against an individual underlying table, or just a partition
of that underlying table. This greatly shrinks utility times against these individual pieces,
improves concurrency, and truly gives us full partition independence.
The view can be referenced in any SELECT statement in exactly the same manner as
would be a physical table.
Each underlying table could be as large as 16TB, logically setting the size limit of the
table represented by the view at 64TB.
Each underlying table could be clustered differently, or could be a segmented or
partitioned tablespace.
Predicate Distribution
Each query against our account history view will have predicates distributed across all of the
underlying tables. DB2 creates a separate query block for each underlying table of the view,
and any predicates against the view are distributed to each query block. Note, for example, the
following query:
20
DBAzine.com
SELECT *
FROM
V_ACCOUNT_HISTORY
WHERE ACCOUNT_ID = 12000000;
The predicate will be distributed to all of the query blocks representing access to the
underlying tables. The query is, in essence, rewritten as the following query:
SELECT ACCOUNT_ID, PAYMENT_DATE, PAYMENT_AMOUNT,
PAYMENT_TYPE, INVOICE_NUMBER
FROM
ACCOUNT_HISTORY1
WHERE ACCOUNT_ID BETWEEN 1 AND 100000000
AND
ACCOUNT_ID = 12000000
UNION ALL
SELECT ACCOUNT_ID, PAYMENT_DATE, PAYMENT_AMOUNT,
PAYMENT_TYPE, INVOICE_NUMBER
FROM
ACCOUNT_HISTORY2
WHERE ACCOUNT_ID BETWEEN 100000001 AND 200000000
AND
ACCOUNT_ID = 12000000
UNION ALL
SELECT ACCOUNT_ID, PAYMENT_DATE, PAYMENT_AMOUNT,
PAYMENT_TYPE, INVOICE_NUMBER
FROM
ACCOUNT_HISTORY3
WHERE ACCOUNT_ID BETWEEN 200000001 AND 300000000
AND
ACCOUNT_ID = 12000000
UNION ALL
SELECT ACCOUNT_ID, PAYMENT_DATE, PAYMENT_AMOUNT,
PAYMENT_TYPE, INVOICE_NUMBER
FROM
ACCOUNT_HISTORY4
WHERE ACCOUNT_ID BETWEEN 300000001 AND 999999999
AND
ACCOUNT_ID = 12000000;
The same distribution of predicates can apply to predicates that utilize host variables or
parameter markers, as well as join predicates.
The predicate of this query contains the literal value 12,000,000, and this predicate is
distributed to all four of the query blocks generated. However, DB2 will compare the
DBAzine.com
21
distributed predicate against the predicates coded in the UNION inside our view, looking for
redundancies. In any situations in which the distributed predicate renders a particular query
block unnecessary, DB2 will prune (eliminate) that query block from the access path. So,
when our distributed predicates look like the following, DB2 will prune the query blocks
generated at statement compile time based on the literal value supplied in the predicate:
. . .
WHERE
AND
. . .
WHERE
AND
. . .
WHERE
AND
. . .
WHERE
AND
Although four query blocks would be generated in the previous example, three of them will
be pruned when the statement is compiled. DB2 compares the literal predicate supplied in the
query against the view with the predicates in the view. Any unnecessary query blocks are
pruned. Since three of the four resulting combined predicates are impossible, DB2 eliminates
those query blocks. Only one underlying table will then be accessed.
Query block pruning can happen at statement compile (bind) time, or at run time if a host
variable or parameter marker is supplied for a redundant predicate. So, lets take the previous
query example, and replace the literal with a host variable:
SELECT *
FROM
V_ACCOUNT_HISTORY
WHERE ACCOUNT_ID = :H1
If this statement were embedded in a program and bound into a plan or package, four query
blocks would be generated. This is because DB2 does not know the value of the host variable
in advance, and distributes the predicate amongst all four generated query blocks. However, at
run time, DB2 will examine the supplied host variable value, and dynamically prune the query
blocks appropriately. So, if the value 12,000,000 was supplied for the host variable value,
then three of the four query blocks would be pruned at run time, and only one underlying
table would be accessed.
This dynamic runtime query block pruning gives the UNION in view design an advantage
over a single partitioned database design in that only the necessary data will be accessed at
run time without query re-optimization. If a partitioned table is accessed based on the
partitioning key, as supplied in a predicate via a host variable, then partition range scanning is
not a possible access path unless the REOPT(VARS) bind parameter is utilized. This may
result in increased CPU utilization because every statement will incur an incremental bind.
Thus, the UNION in view design provides a distinct advantage in that the query block pruning
is available at runtime, without any need for re-optimization.
22
DBAzine.com
The join, as well as the local predicates against the person account table, are distributed across
all of the tables in the UNION in the view. Once the join is distributed across all of the tables
in the UNION in view, predicate transitive closure can also be applied. This query is, in
essence, rewritten by the optimizer as the following query:
SELECT *
FROM
PERSON_ACCOUNT A
INNER JOIN
ACCOUNT_HISTORY1 B
ON
A.ACCOUNT_ID = B.ACCOUNT_ID
AND
B.ACCOUNT_ID BETWEEN 1 AND 100000000
AND
A.ACCOUNT_ID BETWEEN 1 AND 100000000
AND
A.PERSON_ID = 55
UNION ALL
SELECT *
FROM
PERSON_ACCOUNT A
INNER JOIN
ACCOUNT_HISTORY2 B
ON
A.ACCOUNT_ID = B.ACCOUNT_ID
WHERE B.ACCOUNT_ID BETWEEN 100000001 AND 200000000
AND
A.ACCOUNT_ID BETWEEN 100000001 AND 200000000
AND
A.PERSON_ID = 55
UNION ALL
SELECT *
DBAzine.com
23
FROM
PERSON_ACCOUNT A
INNER JOIN
ACCOUNT_HISTORY3 B
ON
A.ACCOUNT_ID = B.ACCOUNT_ID
WHERE B.ACCOUNT_ID BETWEEN 200000001
AND
A.ACCOUNT_ID BETWEEN 200000001
AND
A.PERSON_ID = 55
UNION ALL
SELECT *
FROM
PERSON_ACCOUNT A
INNER JOIN
ACCOUNT_HISTORY4 B
ON
A.ACCOUNT_ID = B.ACCOUNT_ID
WHERE B.ACCOUNT_ID BETWEEN 300000001
AND
A.ACCOUNT_ID BETWEEN 300000001
AND
A.PERSON_ID = 55;
AND 300000000
AND 300000000
AND 999999999
AND 999999999
Note that with the previously shown query, the optimizer has distributed the join, and the join
predicate (ON A.ACCOUNT_ID = B.ACCOUNT_ID) to all query blocks. The local predicate
against the person account table (AND A.PERSON_ID = 55) has also been distributed. And, the
range predicate within the view (e.g., WHERE B.ACCOUNT_ID BETWEEN 100000001 AND
200000000 for query block 2) has been applied to the person account table in each query
block via predicate transitive closure.
DBAzine.com
acceptable. Our large database systems tend to be complex, resulting sometimes in large joins.
In one of our recent designs, we began with a large UNION in view design that unioned 51
tables. Complex statements involving joins against this view resulted in SQLCODE 101 or
materialization of the view, and we eventually reduced the number of underlying tables to
eight after several attempts at redesign.
Bind time or run time query block pruning is only available for predicates coded against the
view that utilizes a host variable or literal value. Predicates with literal values against the view
may result in bind time query block pruning, while predicates with host variables or parameter
markers may result in run time query block pruning. Join predicates are not used to prune
query blocks during statement execution.
Predicate transitive closure is applied after joins have been distributed to the view. But DB2
determines if bind time or runtime query block pruning is possible during the distribution of
joins and predicates to the view. So, any predicate derived via predicate transitive closure that
may have resulted in query block pruning does not result in query block pruning. Lets take a
look at our previous join example:
SELECT *
FROM
PERSON_ACCOUNT A
INNER JOIN
V_ACCOUNT_HISTORY B
ON
A.ACCOUNT_ID = B.ACCOUNT_ID
AND
A.PERSON_ID = 55;
If the local predicate against the person table is modified to filter on the ACCOUNT_ID
column:
SELECT *
FROM
PERSON_ACCOUNT A
INNER JOIN
V_ACCOUNT_HISTORY B
ON
A.ACCOUNT_ID = B.ACCOUNT_ID
AND
A.PERSON_ID = 55
AND
A.ACCOUNT_ID = 1204524;
All predicates will be distributed to all query blocks, and the predicate on the ACCOUNT_ID
column will be applied to all of the underlying tables of the view via predicate transitive
closure. However, since predicate transitive closure happens after the join is distributed, none
of the query blocks will be pruned at bind or runtime. All query blocks will be accessed
during the statement execution. So, it is important that when coding such queries, we do so by
applying the predicate that would result in query block pruning explicitly:
SELECT *
FROM
PERSON_ACCOUNT A
INNER JOIN
V_ACCOUNT_HISTORY B
ON
A.ACCOUNT_ID = B.ACCOUNT_ID
AND
A.PERSON_ID = 55
AND
B.ACCOUNT_ID = 1204524;
DBAzine.com
25
Now that the predicate against the ACCOUNT_ID references the column in the view, we can
apply query block pruning.
Because a view containing a UNION cannot be updated or inserted into, any program
processes that need to update or insert into our view will have to update or insert into the
appropriate underlying table. Well have to create application logic to branch to the
appropriate insert or update statement, or we can employ some other technique such as
package switching. To retain the powerful flexibility that UNION in view provides our
design, its prudent to implement a finder table so that applications that make changes to the
underlying tables can utilize flexible routines to do so. For the account history view, the finder
table (ACCOUNT_RANGE, figure 4) would contain the following data:
TABLE NAME
ACCOUNT_HISTORY1
ACCOUNT_HISTORY2
ACCOUNT_HISTORY3
ACCOUNT_HISTORY4
START RANGE
1
100000001
200000001
300000001
END RANGE
100000000
200000000
300000000
999999999
26
DBAzine.com
Considering that the high order column primary key on the person account table is the
PERSON_ID column, it is reasonable to assume that the optimizer will select the person
account table as the outer table of the join. So, after the distribution of the join, and the
predicates across all of the underlying tables of the view, the optimizers rewritten query may
look something like this:
SELECT *
FROM
PERSON_ACCOUNT A
INNER JOIN
ACCOUNT_HISTORY1 B
ON
A.ACCOUNT_ID = B.ACCOUNT_ID
AND
B.ACCOUNT_ID BETWEEN 1 AND 100000000
AND
A.ACCOUNT_ID BETWEEN 1 AND 100000000
AND
A.PERSON_ID = 55
UNION ALL
SELECT *
FROM
PERSON_ACCOUNT A
INNER JOIN
ACCOUNT_HISTORY2 B
ON
A.ACCOUNT_ID = B.ACCOUNT_ID
WHERE B.ACCOUNT_ID BETWEEN 100000001 AND 200000000
AND
A.ACCOUNT_ID BETWEEN 100000001 AND 200000000
AND
A.PERSON_ID = 55
UNION ALL
SELECT *
FROM
PERSON_ACCOUNT A
INNER JOIN
ACCOUNT_HISTORY3 B
ON
A.ACCOUNT_ID = B.ACCOUNT_ID
WHERE B.ACCOUNT_ID BETWEEN 200000001 AND 300000000
AND
A.ACCOUNT_ID BETWEEN 200000001 AND 300000000
AND
A.PERSON_ID = 55
UNION ALL
SELECT *
FROM
PERSON_ACCOUNT A
INNER JOIN
ACCOUNT_HISTORY4 B
ON
A.ACCOUNT_ID = B.ACCOUNT_ID
WHERE B.ACCOUNT_ID BETWEEN 300000001 AND 999999999
AND
A.ACCOUNT_ID BETWEEN 300000001 AND 999999999
AND
A.PERSON_ID = 55;
The predicate on the PERSON_ID column has been distributed to every query block, the join
predicate has been distributed to every query block, and the range predicate from each query
block of the view has been applied to the person account table via predicate transitive closure.
However, since there is no literal value or host variable predicate against the range predicate
of the account history UNION in view, then there is no bind time or runtime query block
pruning. This means that every query block will be executed, and in this case, the person
account table will be accessed redundantly. Figure 5 visualizes the order of the table access
for each query block:
DBAzine.com
27
28
DBAzine.com
Conclusion
UNION in view for very large table design offers significant advantages for concurrency,
design flexibility, and very large virtual tables. As with any advanced database design, and
advanced database features, we must be very careful when weighing the advantages and
disadvantages of the design. And, we must utilize proper consideration and control when
accessing the view. I have deployed UNION in view in some of my very large
implementations, and will continue to do so when it is appropriate. I know that, with proper
considerations, these implementations will be a success.
DBAzine.com
29
30
DBAzine.com
Introduction
Familiarity breeds contempt, or so the saying goes. Many DB2 professionals are certainly
familiar with the DB2 REORG utility, but how often do we really stop and think about
exactly what it is doing and why?
This article will outline the basic operation of the REORG TABLESPACE utility (hereafter
referred to simply as REORG), and provide some simple guidelines on when and how to run
it. This can be quite useful to DB2 rookies and may even be a handy reminder for some of
you battle-hardened veterans out there.
Throughout the article, I am going to concentrate on the basics of the REORG utility, as
supported by IBM and the other major tool vendors such as BMC Software. Although each of
these vendors supports additional advanced functionality within their REORG
implementations, we explore strictly the basics here.
31
have to move it to another location and leave behind a pointer to the new location. In either
case, just like my movie collection, the organisation of the table will gradually degrade over
time and the performance of operations that depend upon that clustering will do likewise.
This performance degradation can be especially noticeable in large batch programs and other
processes that tend to access lots of data sequentially. When the data is highly clustered
according to the processing sequence of the batch program (refer to Figure 1 below), the data
can be very efficiently accessed as DB2 is able to read all of the rows on a page with a single
GETPAGE operation and may use other tricks such as sequential or dynamic Prefetch to
make the process even more efficient.
Batch Process
SELECT .
FROM .
WHERE .
ORDER BY .
Data Pages
32
DBAzine.com
Batch Process
SELECT .
FROM .
WHERE .
ORDER BY .
Data Pages
33
COLUMN (important for performance reasons). With the new dynamic schema change
functionality added in Version 8 allowing DBAs to make many more changes via a simple
ALTER, running REORG is essential to ensure that good performance is maintained
following the change.
Finally, there is the data availability aspect. If you are one of those poor souls asked to
maintain a 24x7 production database, making time for data housekeeping operations such as
REORG can be painful, to say the least. There are two major features offered by REORG that
can help. First, if the tablespace is partitioned you are able to REORG multiple partitions in
parallel to reduce the overall elapsed time of the process, with very little contention1. An even
better option might be to use online REORG, by specifying SHLEVEL REFERENCE or
SHRLEVEL CHANGE on your REORG parms. This allows full-read access (and, or
SHRLEVEL CHANGE, write access) to the data while the REORG is running, with only a
small window at the end of the process in which data is unavailable (see section on How
Does REORG Work? below).
So REORG has many abilities, and can be a valuable aid to the hard-pressed DBA when it
comes to getting a performance boost or rapidly implementing those space allocation changes.
Next we will look at how it actually works.
As you can see from the diagram, if we ignore the usual UTILINIT and UTILTERM phases
used for utility initialisation and cleanup respectively, we are left with four major steps:
The UNLOAD phase will unload the table data into a sequential dataset (using the clustering
index unless the SORTDATA option is specified, in which case a tablespace scan is used and
the data is sorted afterwards).
Prior to Version 8, the existence of non-partitioning indexes (NPIs) on the partitioned table can cause some
contention during the index rebuild phase. Version 8 makes it possible to avoid this issue by introducing Data
Partitioning Secondary Indexes (DPSIs).
34
DBAzine.com
The RELOAD step will then reload the table data back into the tablespace, re-establishing any
FREESPACE and PCTFREE allocations as it goes. As the sequential dataset holds the rows
in clustering sequence, the table will be 100 percent clustered following the reload operation.
Next, the SORT step extracts the keys for any indexes on the table, and sorts them into key
sequence.
Finally the BUILD phase uses the sorted and extracted keys to rebuild the tables indexes.
Table
Index
Table
Data
UTILINIT
UNLOAD
Index
Keys
RELOAD
SORT
BUILD
UTILTERM
Phase
As previously mentioned, both the IBM and the third-party tool vendors implementations of the
REORG utility contain a large number of additional keywords and options to improve the
performance of this process, but these are beyond the scope of this article.
DBAzine.com
35
REORG Frequency: The elapsed time between REORG operations is the final factor. The
more frequently it is run, the less time the data has to become disorganised with a given
amount of update activity.
Table
Volatility
?
Free
Space
Reorg
Frequency
Fix any two of these variables, and the third one can usually be calculated. As we usually
have very limited control over the tables volatility (this is usually a function of the business
processes that use it), it is most common for us to have to juggle the remaining two variables
in order to ensure that application performance remains within acceptable boundaries.
For example, operational issues may make it impossible to reorganise the table more
frequently than once per day. In this case, enough space must be allowed in PCTFREE and
FREEPAGE to accommodate a typical days worth of activity without the clustering
degrading to an unacceptable level.
So what constitutes an unacceptable level? The DB2 catalog contains a wealth of
information that can help, and every DBA has his or her favourite queries for extracting the
relevant information. You will find just a few of the more simple queries immediately
following.
Running this subsequent query will list the total number of rows in the tablespace or partition
(CARD), together with the number of varying-length rows located near to and far from their
original position (NEARINDREF and FARINDREF respectively).
36
DBAzine.com
The higher that NEARINDREF and FARINDREF become as a percentage of the overall
table, the poorer your I/O performance will become as DB2 has to constantly follow row
pointers in order to retrieve the relocated rows.
If you use non-segmented multi-table tablespaces, the PERCDROP column in the same
catalog table will tell you what proportion of the tablespace pages are occupied by dropped
tables. Running REORG will reclaim this wasted space.
The degree to which the table is ordered according to the clustering index can be determined
in a number of ways. Looking at the CLUSTERRATIO in SYSIBM.SYSINDEXES will tell
you the percentage of rows in clustering order; but, if your tablespace is partitioned, this will
only be an average across all partitions. Running the following query against
SYSINDEXPART will give a clear indication on a partition-by-partition basis:
SELECT NEAROFFPOSF, FAROFFPOSF
FROM SYSIBM.SYSINDEXPART
WHERE IXCREATOR = index_creator_name
AND IXNAME = index_name;
This will return the number of rows in each partition, which are near and far from their
optimal positions, respectively. This is due to INSERT operations having insufficient free
space on the target page. Again, as these figures climb, sequential access performance will
degrade.
DB2 Version 6 introduced some REORG enhancements that allowed limits to be specified
for both NEARINDREF/FARINDREF and NEAROFFPOSF/FAROFFPOSF, so that the
REORG will only be executed if these limits are exceeded. This allows you to schedule the
REORG to run on a regular basis, but only have it actually executed when necessary.
Do not forget that the catalog is only as up-to-date as the last time you ran RUNSTATS
against the tablespace, so make sure your stats are current before making any decisions based
upon them. Also, if you are collecting statistics history (using the HISTORY keyword on your
RUNSTATS jobs), then all of these figures can be analysed over time in order to see how
they degrade this can be a big help in determining the sweet spot for REORG frequency
and/or free space allocation.
Conclusion
Reorganising your tablespace can be a critical factor in maintaining the desired performance
levels for your application. Hopefully, this article has prompted you to take a closer look at an
aspect of DB2 performance usually neglected. Now, if youll excuse me, Im off to re-order
my DVD collection!
DBAzine.com
37
38
DBAzine.com
If you use DB2 for z/OS and OS/390 as your database management system, you know that
you need to keep a watchful eye on your memory. DB2 just loves memory. Some might say
that the more memory you throw at DB2, the better. But simply adding more and more
memory to your DB2 buffer pools is not always the best approach. The issue is more about
how to practically manage the memory at DB2s disposal.
Memory Basics
By caching data and internal structures in memory, instead of accessing them from disk every
time they are needed, DB2 optimizes I/O efficiency. A read from memory is much less
expensive than a read from disk.
Whenever we speak of memory, most of us jump to conclusions and think about buffer pools.
Well, buffer pools are just one resource for which DB2 uses memory to manage. There are
others.
One of the more important memory structures used by DB2 is the EDM pool. Proper EDM
Pool allocation is important because DB2 uses this memory to cache structures required by
DB2 application programs as they run.
Every DB2 program requires a plan or a package that contains the optimized access paths for
the SQL it contains. When a DB2 application program is executed, these access paths must be
loaded into the EDM Pool for DB2 to use. This takes the form of Cursor Tables (or CTs) for
plans, and Packages Tables (or PTs) for packages. DB2 also maintains SKCTs and SKPTs,
skeletons for creating additional PTs and CTs when the same program is being run
concurrently.
SKCT/SKPT pages represent cursor table and package table pages. If the plan was bound with
packages, then SKPT pages are allocated. If the plan was bound with DBRMs, then SKCT
pages are allocated. There is only one SKCT page allocated for the plan regardless of the
number of threads executing the plan; the CT/PT pages are allocated for each thread instance
of the plan.
Additionally, DB2 caches DBDs in the EDM Pool. DBD pages are control block structures
that represent databases and their dependent objects. Any time a table is accessed, the DBD
for the database in which that table was created must be loaded into the EDM Pool.
There are other structures cached into the EDM Pool as well, such as an authorization cache
and perhaps cached dynamic SQL mini-plans (if the dynamic statement cache is in use). Also,
DBAzine.com
39
as of DB2 V8, the EDM Pool is broken apart into separate pools for DBDs and CTs/PTs. This
helps so that loading a DBD does not overwrite CTs/PTs/SKCTs/SKPTs, and vice versa.
If the EDM Pool is too small, your system will experience increased I/O to the DB2 Directory
(tables SCT02, SPT01 and DBD01), worse response time as the system loads SKCTs, SKPTs,
and DBDs, and perhaps re-preparation on cached dynamic SQL statements.
DB2 also sets aside memory for the RID Pool. The RID pool is used for all record identifier
(RID) processing. For example, it is used for enforcing unique keys while updating multiple
rows and for sorting RIDs during the following operations:
List Prefetch
Multiple index access
Hybrid joins
If there is insufficient RID pool storage at execution time, it is possible that DB2 will revert
back to a tablespace scan and that can severely impact performance.
Another DB2-memory consumer is the Sort Pool. As might be expected, the Sort Pool is used
by DB2 when data must be sorted. The default Sort Pool size is 1MB, but keep in mind that a
Sort Pool is allocated for every concurrent sort user. So, there is not just one Sort Pool, but as
many as are needed based on your workload. The bigger the Sort Pool, the more efficient
DB2 sorting becomes because more data can be sorted in memory, instead of requiring it to be
stored in the sort work database, DSNDB07.
Which brings us to buffer pools.
DB2 tablespaces and index spaces are assigned to a specific buffer pool when they are
created. If no buffer pool is specified on the CREATE statement, then the object will default
to a buffer pool as specified in the CREATE DATABASE statement. There is both a
tablespace buffer pool default and an index-space buffer pool default that can be specified for
each database. If no buffer pools are specified at the database level, then DB2 will use the
default assigned for buffer pools at install time.
40
DBAzine.com
As a general rule of thumb, though, it makes sense to explicitly specify the buffer pool to be
used for each tablespace and index space. Doing so provides better control and management
of buffering and performance.
But how do you decide which buffer pool to use for which object? This is a much trickier
task, but there are some general guidelines to follow. First of all, it is better to use multiple
buffer pools than to use a single, large buffer pool for everything. In the early days of DB2
(1980s), many shops put everything into BP0ki and let DB2 manage how best to buffer. As
IBM provided additional buffer pools to use, and organizations tested different approaches to
buffer pool management, the general recommendation has changed to use multiple buffer
pools tuned for different types of processing instead of a single big buffer pool.
So, as we build DB2 databases, we need to gather knowledge of how our data will be
accessed by our end users and applications. Therefore, a good, first approach to buffer pool
design is to break apart the objects into different types of usage.
First of all, separate system objects (DB2 Catalog and Directory) from application data
objects (tablespaces and indexes). The system objects will be assigned to BP0 so you can
track and tune access to the system catalog and directory separate from application data. Be
sure to assign only system objects into BP0.
Next, separate your tablespaces and indexes. For example, place the tablespaces in BP1 and
the indexes in BP2. Doing so should help to maintain your index data in memory longer. This
is so because a large sequential Prefetch against a tablespace would not overwrite index data
in the buffer pool since the index and tablespace reside in different buffer pools.
Once you have taken care of this rudimentary separation, you can get down to the hard job of
allocating objects to buffer pools based on workload. Try to put objects that are more
frequently accessed randomly in separate buffer pools from objects that are more often
accessed sequentially. Of course, it is rare for an object to be accessed only randomly or only
sequentially, so you will have to determine which type of access is most frequent. Some
objects may be accessed as a true mix of random and sequential, and you might even have a
separate buffer pool for those.
An additional consideration is tuning buffer usage for DB2 sorting. DSNDB07 is the DB2
database for the sort worktable spaces used by DB2 when sorting data. Consider assigning
these tablespaces to a separate buffer pool tuned for sorting. (BP7 works out well for this
purpose because the 7 at the end of the buffer pool name aligns nicely with the 7 at the
end of the database name.) Then tweak the buffer pool parameters for mostly sequential
access. We will talk about the buffer pool tuning parameters in the next section of this article.
DBAzine.com
41
You might also want to think about pinning some data into a buffer pool. This tactic can be
beneficial for tables or indexes that are accessed very frequently, but are not very large, as are
lookup tables or code tables.
42
DBAzine.com
disposal by learning about the buffer pool thresholds that are monitored and maintained by
DB2.
First of all, DB2 has five adjustable thresholds that can be modified using the ALTER
BUFFERPOOL command. These thresholds are as follows:
Sequential Steal Threshold, or VPSEQT The percentage of the buffer pool that can
be occupied by sequentially-accessed pages. For example, at the default value of 80,
when this threshold is reached, 80 percent of the buffer pool represents pages for
sequential processing. Of course, 80 percent is the default; you can modify this value to
any value (ranging from 0 to 100) that makes sense for your data/processing mix. When
this threshold is reached, DB2 will steal a sequential page first before stealing a page
from a randomly accessed page. So, for data that is accessed mostly sequentially (for
example, through scans and perfecting), consider increasing the value of this parameter;
and for data that is accessed most randomly, consider decreasing the value of this
parameter. A VPSEQT value of zero will prevent any sequential pages from consuming
space in the buffer pool and it will turn off sequential Prefetch. A VPSEQT value of 100
allows the entire buffer pool to be monopolized by sequential pages.
Parallel Sequential Threshold, or VPPSEQT This threshold indicates the amount
of the buffer pool that can be consumed by sequentially-accessed data for parallel
queries. When this threshold is reached, DB2 will cease to steal random pages to store
sequential pages accessed by parallel queries. The default value for VPPSEQT is 50
percent, indicating its size as 50 percent of the sequential steal threshold (VPSEQT). For
example, if the buffer pool is defined as 1000 pages and VPSEQT is set at 80 percent, a
query using I/O parallelism can consume up to 400 sequential pages (that is, 1000 x 80
percent = 800 for the sequential steal threshold and 800 x 50 percent = 400 for the
parallel sequential threshold).
Assisting Parallel Sequential Threshold, or VPXPSEQT This threshold indicates
the portion of the buffer pool that might be used to assist with parallel operations initiated
from another DB2 in the data-sharing group. It is measured as a percentage of VPPSEQT.
Setting VPXPSEQT to zero disallows that DB2 from assisting with Sysplex query
parallelism at run time for queries using this buffer pool.
The final two modifiable DB2 buffer pool thresholds are used to indicate when modified data
is to be written from the buffer pool to disk. Log data is externalized when a COMMIT is
taken, but the two deferred write thresholds control writing of the actual data.
Deferred Write Threshold, or DWQT When DWQT is reached, DB2 starts
scheduling write I/Os to externalize the data pages to disk. By default, the deferred write
threshold is reached when 50percent of the buffer pool is allocated to unavailable pages,
whether updated or in use. The default is probably too high for most larger-sized buffer
pools.
DBAzine.com
43
Vertical Deferred Write Threshold, or VDWQT Basically the same as DWQT, but
for a single page set. By default, VDWQT is reached when 10 percent of the buffer pool
is allocated to one data set. When reached, DB2 will start scheduling write I/Os to
externalize the data pages to disk. Once again, this default is most likely too high for
most shops.
Consider ratcheting the deferred write thresholds down to smaller percentages for most of
your buffer pools. Doing so enables trickle write from the DB2 buffer pools. This means
that the data is written asynchronously to disk regularly, over time, in smaller amounts,
instead of storing up a lot of modified data that must be written all at once when the threshold
percentage is reached. The needs of every shop will vary, but some organizations have had
success by reducing DWQT as low as one.
All of the above thresholds can be changed using the -ALTER BUFFERPOOL command. Of
course, there are additional buffer pool thresholds that must be monitored, but cannot be
changed. Lets look at those three thresholds now:
Prefetch Disabled Threshold Reached when 90 percent of the buffer pool pages are
unavailable. When reached, DB2 will no longer initiate Prefetch operations, and will
cancel Prefetch operations in progress.
Data Manager Critical Threshold Reached when 95 percent of the buffer pool pages
are unavailable. When this threshold is reached, DB2 will access the pages in the virtual
buffer pool once for each row that is retrieved or updated in that page. In other words,
retrieving or updating several rows in one page causes several page access operations.
Immediate Write Threshold Reached when 97.5 percent of the buffer pool pages are
unavailable. When this threshold is reached, DB2 will write updated pages as soon as the
update is complete. The write is performed synchronously as opposed to asynchronously.
This method will potentially cause a large increase in the number of write I/Os performed
by DB2, and should be avoided completely.
And a truncated version of the results will look something like this:
DSNB401I - BUFFERPOOL NAME BP0, BUFFERPOOL ID 0, USE COUNT 20
DSNB402I - VIRTUAL BUFFERPOOL SIZE = 500 BUFFERS 736
44
DBAzine.com
Of course, you can request much more information to be displayed using the DISPLAY
BUFFERPOOL command by using the DETAIL parameter. Additionally, you can request
that DB2 return either incremental statistics (since the last DISPLAY) or cumulative statistics
(since DB2 was started). The statistics in a detail report are grouped in the following
categories: GETPAGE information, Sequential Prefetch information, List Prefetch
information, Dynamic Prefetch information, Page Update statistics, Page Write statistics, and
Parallel Processing Activity details.
A lot of interesting and useful details can be gathered simply using the DISPLAY
BUFFERPOOL command. For example, you can review GETPAGE requests for random and
sequential activity, number of Prefetch requests (whether static or dynamic, or for sequential
or list Prefetch), number of times each of the thresholds were tripped, and much more. Also, if
you are using DB2 V7 or earlier, you will also get hiperpool statistics. Refer to the DB2
Command Reference Manual for a definition of each of the actual statistics returned by
DISPLAY BUFFERPOOL.
Many organizations also have a performance monitor that simplifies the gathering and display
of buffer pool statistics. Such tools are highly recommended for in-depth buffer pool
monitoring and tuning. More sophisticated tools also exist that offer guidance on how to tune
your buffer pools or that automatically adjust your buffer pool parameters according to
your workload. Most monitors also provide more in-depth statistics, such as buffer pool hit
ratio calculations.
The buffer pool hit ratio is an important tool for monitoring and tuning your DB2 buffer
pools. It is calculated as follows:
Hit ratio = GETPAGES - pages_read_from_disk / GETPAGEs
Pages read from disk is a calculated field that is the sum of all random and sequential reads.
The highest possible buffer pool hit ratio is 1.0. This value is achieved when each requested
page is always in the buffer pool. When requested pages are not in the buffer pool, the hit
ratio will be lower. You can have a negative hit ratio this simply means that Prefetch was
requested to bring pages into the buffer pool that were never actually referenced.
In general, the higher the hit ratio, the better, because it indicates that pages are being
referenced from memory in the buffer pool more often. Of course, a low hit ratio is not always
bad. The larger the amount of data that must be accessed by the application, the lower the hit
ratio will tend to be. Hit ratios should be monitored in the context of the applications assigned
DBAzine.com
45
to the buffer pool and should be compared against hit ratios from prior processing periods.
Fluctuation can indicate problems.
Summary
DB2 loves memory. In general, the more memory you can assign to DB2 processing, the
better performance can be. But there are no silver bullets. Everyones DB2 buffer pool set-up
will be different, based on your shops particular processing needs.
Use the tools at your disposal. DB2 provides basic commands that can be used for buffer pool
monitoring and tuning, but a performance monitor can greatly simplify your buffer pool
tuning tasks.
You should align each buffer pool to the workload that DB2 will process using that buffer
pool. And , understand the difference between sequential and random processing and learn
which application processes rely most heavily on each type; tune your buffer pools
accordingly. Separate indexes from table spaces, separate system work from application work,
separate work files from other files, and isolate your code and lookup tables when possible.
Of course, we have only covered the basics of buffer pool management in this article. For
large environments, one could make an entire career out of monitoring and tuning buffer pool
usage to improve performance. Dont stop here keep reading and learning about DB2
buffer pools. Other good resources for learning about buffering techniques include the IBM
DB2 Administration Guide manual, IDUG and IBM Technical conference presentations, DB2
books, and vendor products and Web sites.
46
DBAzine.com
Introduction
We used to think that bigger databases were better. Consequently, many of us have designed
and implemented very large systems that process billions of transactions and SQL statements
daily. The thought was that the big database could contain all of the companys data and serve
multiple purposes. As a consultant, I have seen many issues with these types of large
databases and helped numerous companies fix large data warehousing, Internet and
operational database performance problems I am convinced that bigger databases are not
necessarily better.
DBAzine.com
47
How did business process analysis save this company so much? Partnering with this
companys employees, we worked through their toughest performance issues. First, we
looked at their overall business processes and discovered that the systems were generally I/O
bound. This meant that, regardless of how much CPU we threw at the problem, the processing
would not improve. We looked at the top CPU and I/O processes that existed in their overall
system. These processes were daily, long-running batch jobs that were processing large data
flows; a few large online transactions that were executed millions of times; and some utilities.
Since the utilities showed up in the list, we decided to look more closely at each of the daily,
weekly, and monthly utilities running against their system.
Deep analysis
Long-running batch programs are notoriously symptomatic of overall performance aspects of
the processing within a company. To discover the batch processes performance issues, we
did a deep dive into the programming logic and the business processes. First, we looked at the
input files, the key data elements for the various database tables, and indexing associated with
the processing.
This analysis revealed that the programs were using different keys for different processing,
and that good indexing themes were not available to quickly and uniquely retrieve the data.
For example, we found a database table designed using an IDENTITY data type as its major
processing key. This IDENTITY data type was great for quickly retrieving data throughout its
database indexes, but the key had to be manipulated to relate it to other database systems and
older systems. Because of this additional processing to translate the IDENTITY data type into
another index key along with not being able to join tables, the application had to do multiple
SQL statements instead of SQL Joins. It was also discovered that the program was processing
across and through all the transaction information over six years of data. This database
table was clustered in such a way that the historic data was intermingled with the current data,
causing more I/O to get at the recent transaction data.
Since the input files were not able to match the IDENTITY column key, the SQL retrievals
using other keys had worse cardinality, resulting in more database pages to be referenced and
causing more I/O within the system. Also, since a cursor had to be opened for each of the
batch programs, the system required large sort work areas for cursor result sets triggering
even more I/O issues.
We were able to add additional columns and indexes to the IDENTITY column table. This
provided other indexed columns and complementary keys used outside that particular
database system; these new index keys also allowed the application to quickly focus on its
data replacing the large cursor with a direct SQL statement that did not require sorting.
This business process analysis also exposed a large number of decisions that the program was
making before doing the SQL. Based on the results of the input fields, some of these IF ELSE
conditions were unnecessary and were moved to other sections to be processed only when
required.
48
DBAzine.com
49
every day, causing additional I/Os. Daily, weekly, and monthly batch processes were required
to go through the old data to get to the new transactions. Our analysis showed that cutting
down the database from six years to four years of data (by one-third) would significantly
reduce the number of I/O done by the batch processes. This significant reduction of I/O also
saved bottom-line CPU costs and precious time in the nightly processing window.
Analyzing the data retention requirements and how the data flowed through the business
processes showed that the company had overlapping data across their operational, reporting,
and business intelligence systems. By aligning all of these platforms, departments, and their
systems, we were able to reduce the data overlap and reduce their DASD requirements
significantly across several platforms. This DASD reduction saved processing, downtime
backup time, and CPU across all programs, utilities, and reporting tools.
Conclusion
The next time you are wondering whether your company is going to start that development
effort, buy that software package, or give out bonuses, let me know. You and I might be able
to help the company accomplish these with savings from business process analysis and with
better not bigger databases.
50
DBAzine.com