DBA

REBUILDING INDEXES

WHY, WHEN, HOW?

(SEPT 2004)

Jonathan Lewis, JL Computer Consultancy

OPTIMAL THINKING.
There are many areas of Oracle that are difficult to understand and require careful and subtle testing. Index maintenance is not one of those areas. (Deciding an optimal set of indexes is another matter but thats not the purpose of this presentation). It is easy to create experimental data sets, easy to build indexes, easy to see what the structure of an index looks like. You dont have to catch fleeting activity, or worry about timing critical concurrency components to make some special event occur. Indexes just sit there, waiting to be investigated at leisure. The published purpose of this presentation is to address the issue: "How much effort should you spend trying to 'optimize' the quality of your indexes?", but the underlying approach is really just trying to understand indexes (b-tree indexes in particular) a little better. This white paper supports, and is supported by, the graphic images in the presentation.

OPTIMAL THINKING.
The first (and only) rule of optimization is: "Avoid unnecessary effort". But you have to operate this rule at many levels. In the case of rebuilding indexes, for example, you have four considerations: If an index needs constant care and attention, is this a clue that you really need to be fixing a design error. If an index is required, it should not be allowed to degenerate so far that the optimizer should stop using it. You should not waste resources rebuilding indexes when the performance gain is not worth the effort or risk You should not spend excessive amounts of time trying to work out exactly when each index needs to be rebuilt.

CLASSIFYING INDEXES
Before discussing the possible reasons or methods for doing house-keeping operations on indexes, it is worth noting that there are many variants of indexes that may need to be assessed and handled in different ways. Consider the fact that you have simple B-tree indexes, simple bitmap indexes, index clusters, index-organized tables, B-tree indexes on IOTs, bitmap indexes on IOTs, and bitmap join indexes. Then, to add to the options, you can have partitioned tables with global, globally partitioned, or local indexes and in Oracle 10g you can even have partitioned indexes on nonpartitioned tables. (Ed 7th June: Paul Hancock of HBOS emailed me to point out that you could have partitioned indexes on non-partitioned tables in earlier versions of Oracle the specific new feature for 10g was that the indexes could be hash-partitioned when the earlier versions of Oracle only allowed for range-partitioning). The options for "rebuilding" indexes include deciding whether you want to do online or offline rebuilds, whether you want to mark indexes as unusable first or whether there is scope for doing
Paper #

DBA

a drop and re-create; and perhaps a coalesce would be better than a rebuild anyway. And, alas, the type of index does restrict the options available try doing an online rebuild of one subpartition of a local bitmap index in 9.2 and you'll find it's not possible. Finally, there is the issue of what to do about index statistics when you perform some form of major surgery on an index. This article is going to focus on simple B-tree indexes although some of the points it raises will be equally applicable to the individual segments of a partitioned B-tree indexes, and I may throw in a couple of comments about bitmap indexes.

THE

BENEFITS OF INDEXES

We create indexes because they can reduce the amount of work we have to do to find data. Instead of examining millions of rows from a table to identify the few we need, we might start by examining a "much smaller" data set. This "much smaller" data set may point to exactly the few rows that we are really interested in, or it might perhaps identify a batch of rows that are good candidates for our requirements but still need further (filtering) tests to allow us to identify the exact rows we want. Either way, using an indexing structure should make the task of locating data much less costly.

THE

COSTS OF INDEXES

Every time we change the data in a table, we may have to modify some related index information. The more indexes we create on a table, the more work we may have to do as we modify the table. Clearly we need to be careful when we define the set of indexes we want on a table. If we are too extravagant with creating indexes, we may pay a huge penalty as we load and modify the data and it doesn't matter how quickly our queries can run if we cant get the data into the system in time. (Of course, the bitmap index is an extreme case of resource consumption single row DML on tables with bitmap indexes can result in massive amounts of undo and redo being generated, combined with a huge space wastage inside the index that results in lots of unnecessary I/O as the index is both written or read. This situation has been improved somewhat in 10g). But it's not simply the increased cost of the DML that worries us when we have indexes on our tables. The very act of changing the data could make our indexes less effective, and this is an issue that needs a little consideration. Fortunately, many indexes have a 'natural' steady state, so we often need only look out for special cases.

WHAT'S

THE PROBLEM?

If the way the data is inserted, updated and deleted over a period of time is consistent, then the indexes on that table will usually stabilize at an efficiency of around 70% to 75%. By this, I mean that each leaf block in the index will end up with about 70% 75% of its space used. Some indexes will do better, running at close to 100% packed; some will do significantly worse, and a steady state of 50% space usage is not something I would consider particularly bizarre. Some indexes, of course, degenerate catastrophically and you may occasionally find indexes running with most leaf blocks showing 90% or more empty space. In the worst cases, you may find indexes that appear to contain many completely empty blocks, lots of blocks with just one or two entries, and a few blocks packed full. The important thing about space wastage, though, is that it isn't necessarily a problem. Until version 9i it was quite easy to get a good estimate of how much waste space there was in an index and in version 9i it became possible to get a very accurate idea of how that waste space was distributed through the index. Obviously, an index with a large amount of waste space could have a significant performance impact but if you didn't have the resources to re-pack every
Paper #

DBA

suspect index, how do you identify the indexes where the space wastage really makes a difference. A slightly more surprising feature of space wastage is that it's sometimes a good thing packing an index too well could actually cause a performance problem. So not only could you waste resources re-packing indexes that didn't need to be repacked, you could even cause problems by re-packing indexes that positively should not be re-packed and there is no 'numbers-only' way of deciding which indexes are which. (By "numbers-only", I mean a purely arithmetical way of analyzing the state of the index in isolation from the way it is being used).

CHECK

SPACE USAGE

(1)

Of course, the easy way of checking the (average) efficiency of an index is to validate the index, then check the index_stats view. For example, from an Oracle 9i schema:
validate index t1_pk; -analyze index t1_pk validate structure; execute print_table('select * from index_stats') HEIGHT BLOCKS NAME PARTITION_NAME LF_ROWS LF_BLKS LF_ROWS_LEN LF_BLK_LEN BR_ROWS BR_BLKS BR_ROWS_LEN BR_BLK_LEN DEL_LF_ROWS DEL_LF_ROWS_LEN DISTINCT_KEYS MOST_REPEATED_KEY BTREE_SPACE USED_SPACE PCT_USED ROWS_PER_KEY BLKS_GETS_PER_ACCESS PRE_ROWS PRE_ROWS_LEN OPT_CMPR_COUNT OPT_CMPR_PCTSAVE : : : : : : : : : : : : : : : : : : : : : : : : : 2 128 T1_PK 21666 41 280935 7996 40 1 789 8032 0 0 21666 1 335868 294652 88 1 3 762 12928 2 0

The problem with validate structure is that it tries to lock the table, and fails with Oracle error 54 (resource busy and acquire with NOWAIT specified) if anyone is currently updating the table. Worse, if the validation takes a long time, anyone who then tries to update the table is locked out. The analyze version of the validate command can run online from 9i onwards but unfortunately the online version doesnt populate the index_stats view even in 10g. The validate command really only comes into its own if you regularly clone the production system into a backup or UAT system where interruption of service and poor performance are deemed to be acceptable. I have used Tom Kyte's invaluable print_table procedure to list the contents of the single row that appears in the view index_stats after a validate. The most significant numbers to check are the used_space and btree_space. You can check the meanings of these column by reviewing the
Paper #

DBA

underlying x$ object, but the btree_space is the total space available in blocks that are currently in the index structure, the used_space is the actual space currently in use, and is the sum of the lf_rows_len, br_rows_len and pre_rows_len (leaf rows, branch rows, and prefix rows for compressed indexes respectively). There are two other figures to note when using this view to assess the wastage in this index. The first is the 'percentage use' figure, pct_used: this is just used_space/btree_space and, as with all ratios, if you don't keep an eye on the scale of the underlying figures you may treat it with more significance than it merits. The other is the del_lf_rows_len which is the amount of space that would be available in existing leaf blocks for new insertions if the leaf blocks were cleaned. If this value is a large fraction of the leaf_rows_len, it is a clue that you need to investigate what is unusual about the way the table (or this specific index) is used; it is possible that this index should be regularly coalesced, it could be that you need to play clever games with functionbased indexes. Another number that gets reported from 9i onwards is the 'optimum compression count' opt_cmpr_count with its companion opt_cmpr_pctsave. These tell you the number of columns to compress in the index to get maximum space saving in the leaf blocks, and the percentage reduction in leaf block space used if you applied that compression count. Although it is nice to know these figures, bear in mind that index compression can increase CPU overhead, and the more index entries you have per block the more likely you are to get contention as more processes compete for those blocks. In passing, don't forget that when you validate an index you also populate a view called index_histogram that gives you some idea of the unevenness in the numbers of repetitions of different keys in the index. (There is a hint about this in the index_stats view from the most_reported_key column). From an Oracle 10g schema:
select * from index_histogram; REPEAT_COUNT KEYS_WITH_REPEAT_COUNT ------------ ---------------------0 14097 8 5619 16 4004 24 3398 32 3497 40 3871 48 4140 56 2604 64 838 72 121 80 8 88 1 96 0 104 0 112 0 120 1

In this case, we see that there are 14,097 key values that appear less than 8 times each; 5,619 key values that appear between 8 and 15 time each, and so on until we get to one key value that appears somewhere between 120 and 127 times. Information like this can help us to decide whether there is some odd business-related feature of the index that requires us to make it a candidate for special treatment.

Paper #

DBA

CHECK

SPACE USAGE

(2)

Given the problem of locking that you have with the validate index option, is there an alternative way to get some warning figures about space usage? Yes, but only if you are running version 6 (yes, six) or later. Of course the results you get are going to be approximate, and the options for complicating (or refining) the estimate are numerous, but the method is simple. If you have gathered statistics on the table with the cascade option, then you know: The number of entries in the index (user_indexes.num_rows) The number of rows in the table (user_tables.num_rows)

The average column length of each of the columns in the index (user_tab_col_statistics.avg_col_len) The number of null occurences for each column (user_tab_col_statistics.num_nulls) For each index entry, you need space for: The entry overhead two bytes for the offset, plus one byte for the lock byte and one byte for the flags. The rowid content 6 bytes for a unique index, 7 for non-unique, or 10/11 bytes for a globally partitioned index. For the column content, you don't need to consider index entries individually, you can just add up the total number of column-level entries that must be in the index. This will introduce a little inaccuracy where (for example) an index entry starts with a null column but generally, the precision will be reasonable. For each column, you have the number of nulls, and the average column length; combine this with the number of rows in the table, and the total space in the index for each column is: (user_tables.num_rows user_tab_col_statistics.num_nulls) * user_tab_col_statistics.avg_col_length. Once you've done the tricky bits, you need only allow for the typical index block overhead, and the percentage wastage that you consider to be reasonable, plus a couple of percent for branch blocks, and you've got a reasonable estimate of how big the index should be. For further details check http://www.jlcomp.demon.co.uk/index_efficiency_2.html. (If youre wondering how this could work in Oracle 6 the mechanics were a little messier because there were no statistics before the CBO appeared, but a table scan using the vsize() function, and an index scan to count entries worked perfectly adequately).

PRECISION.
So you've go an idea that an index might be much larger than it should be. How useful is this information ? It can be a clue, but it's far from complete. First, it may have reported indexes that are wasting space but not causing performance problems; more significantly, it could have failed to report indexes that are causing performance problems. The former error may make you waste valuable resources fixing a non-existent problem, the latter may mean you fail to address a worthwhile issue. Consider this scenario. You have decided that you need not examine indexes with an efficiency of 75% or better. But a simple space report fails on the most basic of issues it is a "system wide" (or in this case index-wide) figure that hides details. What if you have an index where 75% of the blocks are packed, and 25% of the blocks are virtually empty ? The 'average' efficiency will be 75% - and yet the near-empty 25% may be the most used, and most problematic part of the index.
Paper #

DBA

In Oracle 9i we can get a very nice report showing the number of index entries per used leaf block, and this could improve the precision of our investigation. We hijack the undocumented function sys_op_lbid() that appeared for use with the dbms_stats package. There are several options built into this function, but one option can be used to count the number of index entries per leaf block. Consider the following SQL statement:
select rows_per_block, count(*) blocks from ( select /*+ no_expand index_ffs(t1,t1_i1) noparallel_index(t,t1_i1) */ sys_op_lbid( {NNNNN} ,'L',t1.rowid) as block_id, count(*) as rows_per_block from t1 where v1 is not null or small_pad is not null group by sys_op_lbid( {NNNNN} ,'L',t1.rowid) ) group by rows_per_block

Table T1 has an index T1_I1 on columns (v1, small_pad). The object number of this index is represented by the {NNNNN} that appears twice in the code. With the 'L' parameter, Oracle returns (effectively) the leaf block address of each entry in the index. If we group by leaf block address, we can report the number of rows per leaf block; if we wrap this in an inline view as we have above, we can produce a report which shows how many leaf blocks hold how many index entries any index which shows extreme variations in the number of rows per leaf block is worth investigating.

PERFORMANCE
But no matter how clever we get with the numbers that describe what's going on inside the index, we still don't know that the index is causing a problem. Of course, we might decide that some "very unusual" distribution of data within the index is a good reason for rebuilding the index. If we can afford the resources, and can't afford the time for further investigation, we may have done enough to capture most of the indexes that need a rebuild and not too many more. Remember the optimization rule sometimes your optimization limit is the time you are allowed to spend investigating issues. The same argument applies if the index is running at (say) 50% of efficient packing even when the distribution of index entries is completely uniform across all leaf blocks. (In both these cases, of course, it is possible that the immediate consequence of the rebuild is that the code 'tries' to get back to the degenerate state in which case we may have caused a performance problem by rebuilding.) However, the real test of whether or not to rebuild is whether there is a performance issue, and whether the cost of rebuilding will be a worthwhile expenditure for the improvement in performance that you will achieve.

Paper #

DBA

Ultimately, we have to know how the indexes are being used, and with understanding of usage, we can get realistic ideas about cost of use and possible reductions in cost that might be available. Let's categorize a few 'general case' examples: Single row table access by unique key Range scan with high-precision index, well-clustered table Range scan with high-precision index, poorly-clustered table Range scan with (relatively) low-precision index, well-clustered table Range scan with (relatively) low-precision index, poorly-clustered table

What does is cost Oracle to acquire the data in each case. Let's assume we have a reasonably sized data set, so that the index has height three (blevel = 2), the root block, one layer of branch blocks, and the leaf blocks. If the index is going to be worth rebuilding, it has to be an index that is well-used (a small inefficiency once per day is probably not something you should be spending much time trying to address, after all). So we can probably assume that any interesting index will have at least the root and branch blocks buffered but we'll be pessimistic about the leaf blocks. We will also be pessimistic about the table blocks if we can't manage to buffer popular index index leaf blocks, we probably won't be buffering the related table blocks very well either. The resulting costs for a 'typical B-tree' index are likely to be something like the following. Scenario Single row, unique key High-precision, well-clustered range scan High-precision, badly-clustered range scan Low-precision, well-clustered range scan Low-precision, badly-clustered range scan Cost 2 Logical I/Os (root, branch) plus 2 physical I/Os (one leaf, one table). 2 Logical I/Os plus 2 physical I/Os (one leaf, one table) 2 Logical I/Os plus a few physical I/Os (one leaf, a few table) 2 Logical I/Os plus a few physical I/Os (one/two leaf, a few table) 2 Logical I/Os plus many physical I/Os (one/two leaf, many table)

So for the general usage B-tree index, what gain could you get from rebuilding an index? As we have seen, a rebuild is probably going to improve the packing of the index entries in the leaf blocks which means a rebuild could reduce the number of leaf blocks that have to be visited by a query. But the only queries that are likely to visit multiple leaf blocks are the relatively large range scans, and even though I have pessimistically suggested that large range-scans on wellclustered tables might visit two leaf blocks, you should remember that a typical 8K leaf block could easily hold a couple of hundred index entries. The most likely reduction in leaf block visits for a typically behaved B-tree index will come only from the large range scan on the badly clustered table in which case the cost of the table visits will be by far the most important portion of the cost that you should be addressing. There are a couple of other arguments that could apply, though one is that you could reduce the blevel of the index by rebuilding it; another is that you could get the entire index permanently buffered using a smaller fraction of the available buffer, thus giving a beneficial side-effect to other objects. Both these arguments have an important element of truth to them.
Paper #

DBA

You have to be quite lucky (or unlucky) to be in the position where an index rebuild reduces the height of a B-tree index and provides a visible performance benefit. But that's why you should predict, then monitor, the effect. If the rebuild does have a beneficial effect, and the effect seems to be due to an index rebuild there may be a better option available to you for example doing the rebuild one more time at the correct pctfree, or (special case only) converting the table into a single table hash cluster. Regularly rebuilding the index could be the 'wrong' thing but be a useful step in finding out the right thing to do. (But the optimization rule still applies the benefit of the rebuild may be sufficient, and you don't have the time (right now) to take the investigation any further). The argument to 'rebuild because it helps everything else' is a little trickier. The problem can be summed up quite nicely by a comment a DBA made to me a little while ago: "We rebuilt all the indexes over the last Christmas break because we had a four-day window (about 250 GB of indexes), and the system didn't run any faster afterwards." It's purely speculative, of course, but I'd be willing to bet that if you analyzed every single process in "the system" most of them would have shown no appreciable change, a few would have got quicker, and a few would have been slower. If you can rebuild exactly one index, and show that "the system" went faster, then that looks like a good index to rebuild. But it would be nice to explain exactly why it had that effect (if you're allowed the time investigate properly). If you rebuild 30 indexes and show that "the system" went faster, how confident are you that you need to rebuild all 30 of them maybe one of them is helping and one of them is slowing things down and waiting to give you a really nasty surprise some time next year when it hits a critical size. It 's the optimization rule again how much effort can you afford to spend making sure that what you do is a good thing? Should you do something that 'does no harm' because in general it seems to be okay, but you don't have time to find out whether "okay" is actually a mix of improvements and damage.

SPECIAL CASES
For a large percentage of B-tree indexes, there is very little to be gained by rebuilding, and there could be a direct penalty you pay for rebuilding. And identifying whether a rebuild will be a good idea or a bad idea is not something that you can prove cheaply as it depends ultimately not on the index, but on the run-time environment. There are, however, special cases where it is very easy to identify the immediate benefits of rebuilds. Some examples will be addressed in the presentation.

IMPLEMENTATION
Once you've decided that an index should be repacked, especially if you decided that it should be repacked on a regular basis, you have to address the problem of how to get the job done. If your aim is simply to repack the index (rather than do some special handling for the duration of a batch load), then you seem to have three primary options:
alter index t1_i1 coalesce; alter index t1_i1 rebuild; alter index t1_i1 rebuild online;

There are pros and cons to consider (summarized in the following table), and supplementary issues to address: Feature Pro Con
Paper #

DBA

Coalesce

Completely "online" process as it doesn't do any table locking. Repacks within existing index structure. Can use the existing index to create the new version. Can be optimized for reduced overheads.

Can generate a lot of redo. Not very aggressive about repacking so only useful for special cases (until 10g). Can 'cause' ORA-01555 errors Locks the table for the duration of the rebuild. "Doubles" space usage temporarily. May require massive sorts. Can "cause" Oracle error 01410. Locks table at start and end of rebuild. Cannot use the index to rebuild the index. "Doubles" space usage temporarily. Adds row-level trigger to table actions. May require massive sorts. Can "cause" Oracle error 01410.

Rebuild ("offline")

Rebuild online

Does not lock table for entire rebuild. Can be optimized for minimal overheads.

The most significant aspects of index rebuilding are the amount of work done which can appear in different places and the consequential errors that can arise. The coalesce command acts on leaf blocks of the current index in a number of small transactions, and a lot of small transactions executed by one session can cause another session with a long-running query or transaction to suffer an Oracle error ORA-01555: snapshot too old. (Less likely 9i with automatic undo retention, of course). The rebuild commands can obviously have side effects due to locking issues so they can cause problems to other processes that are running at the same time. More insidious, though, is that the old copy of the index becomes free space as soon as the build is complete. If you do several consecutive builds, or some other process claims some free space at the wrong moment, then a currently running query that was happily using the old (freed) copy of the index will suddenly crash with an apparently random Oracle error ORA-10410: invalid rowid. In all cases, once the index has been repacked, you have to decide what you are going to do about the statistics. After all, the rebuild will have changed the number of leaf blocks (you hope) and may even have changed the blevel do you want the statistics to reflect the new state, and how much effort are you prepared to put into collecting those statistics? With the rebuild options you can include the directive to compute statistics as the rebuild takes place (and 10g will do this automatically anyway). Interestingly, the mechanism used to compute the statistics is probably the one used by the deprecated analyze command so you may still want to call dbms_stats() even after a 10g rebuild. (Note: if you have a histogram on the first column of an index, there is a bug with this compute option in many versions of Oracle that will delete the histogram).

WRAPPING

IT ALL

UP

B-tree indexes almost always contain some empty space, and some B-tree indexes can end up holding a lot of empty space. Empty space can be eliminated but empty space is not necessarily a performance issue, and attempts to eliminate empty space may be a waste of effort that could be applied more usefully elsewhere.

Paper #

DBA

It is easy to spot indexes with poor space utilization, which makes it easy to identify some of the indexes that may be good candidates for a rebuild it is less easy to decide if the poor space utilization is actually causing a performance problem that is worth fixing. Rebuilding indexes can be expensive, intrusive and risky. In most cases, though, an index that really is a good candidate for rebuilding is likely to be susceptible to the index coalesce feature (especially in 10g). If you think you are going to get some benefit from doing an index rebuild, try to quantify the benefit beforehand, and measure it afterwards. If the rebuild was a good idea, then you have provided a justification for further investigation, and possibly a scheduled rebuild; if the rebuild was a waste of effort or a bad idea you can ensure that it doesn't happen again.

ABOUT

THE

AUTHOR

Jonathan Lewis has been an independent IT specialist for the last 20 years, of which the last 17 years have been spent using and investigating the Oracle database engine. He is the author of one book on Oracle, has contributed to a couple of others, and is busy writing a book that explains the working of Oracle's Cost Based Optimizer. September 2004

Paper #