Вы находитесь на странице: 1из 29

Dynamic Hashing Schemes

FL J. ENBODY
Department of Computer Science, Michigan State University, East Lansing, Michigan 48823

H. C. DU
Department of Computer Science, University of Minnesota, Minneapolis, Minnesota 55455

A new type of dynamic file access called dynamic hushing has recently emerged. It
promises the flexibility of handling dynamic tiles while preserving the fast access times
expected from hashing. Such a fast, dynamic file access scheme is needed to support
modern database systems. This paper surveys dynamic hashing schemes and examines
their critical design issues.

Categories and Subject Descriptors: E.2 [Data]: Data Storage Representation-bosh-t&k


representations; H.2.2 [Database Management]: Physical Design--access methods
General Terms: Algorithms, Design
Additional Key Words and Phrases: Dynamic hashing

INTRODUCTION clearly more desirable but are beyond the


scope of this paper.
Users typically interact with modern data- The basic purpose of access methods is
base systems via a set-oriented, high-level to retrieve and update data efficiently.
language. This database language can be in Access methods fall into three general cat-
the form of a self-contained query language, egories: sequential schemes, with O(n)
or it can be embedded in a host language. retrieval time, where n is the number of
To retrieve the appropriate data, the data- elements in the database; tree-structured
base management system must translate schemes, with O(log n) retrieval time; and
queries into the low-level operations pro- hashing schemes, with O(1) retrieval time.
vided by the operating system. To have a If we are trying to minimize access time,
good response time, it is critical that the hashing schemes are very attractive. The
access system be efficient and support the traditional, static, hashing schemes, how-
high-level database manipulation language ever, did not perform well in a dynamic
well. In addition, the system must work environment. They required that data stor-
efficiently in an environment in which the age space be allocated statically. This
volume of data may grow and shrink con- meant that when a file exceeded the allo-
siderably as a result of insertions and dele- cated space, the entire file had to be re-
tions. Most of the studies on such access structured at great expense. In addition,
methods only consider single-key retrieval overflow handling schemes often increased
operations whereby one record is retrieved the retrieval time as files approached their
at a time. More sophisticated access meth- space limits. To eliminate these problems,
ods that support multikey retrieval and set- dynamic hashing structures have been pro-
oriented associative language facilities are posed. These file structures and their

Permission to copy without fee all or part of this material is granted provided that the copies are not made or
distributed for direct commercial advantage, the ACM copyright notice and the title of the publication and ita
date appear, and notice is given that copying is by permission of the Association for Computing Machinery. TO
copy otherwise, or to republish, requires a fee and/or specific permission.
0 1988 ACM 0360-0300/88/0600-0085 $01.50

ACM Computing Surveys, Vol. 20, No. 2, June 1988


86 l R. J. Enbody and H. C. Du
CONTENTS are adaptations of techniques from static
hashing; other techniques may also be
applicable, but if they have not been studied
in the context of dynamic hashing, we have
INTRODUCTION not included them.
1. WHY DYNAMIC HASHING?
2. DEFINITIONS AND TERMINOLOGY
3. STATIC HASHING 1. WHY DYMMC HASHING?
4. TRIES AS A BASIS FOR DYNAMIC
HASHING In order to understand the need for dy-
5. BASIC DYNAMIC HASHING namic hashing, we present a quick look at
SCHEMES
6. SPACE UTILIZATION static hashing and point out some of its
7. OVERFLOW HANDLING TECHNIQUES
major weaknesses. Hashing is one of many
7.1 Splitting Control
7.2 Recursive Overflow Handling addressing techniques used to find a record,
7.3 Multipage Nodes given its key. A record, for example, could
7.4 Internal storage of overflows be the personal data of an employee with
8. EXPANSION TECHNIQUES the name of the employee as the key. An
8.1 Uniform Key Distribution
8.2 Nonuniform Key Distribution
example of a hash function is a mapping of
9. PHYSICAL IMPLEMENTATION the key (name) to the address of the em-
10. CONCLUSION ployee’s record. The function can be as
ACKNOWLEDGMENT simple as converting the first five charac-
REFERENCES ters of the last name to their ASCII equiv-
alents and using their concatenation as
the address of the record. In practice, how-
ever, these functions are often much more
associated algorithms adapt themselves to complex.
changes in the size of the file, so expensive Consider an example using a simple func-
periodic database reorganization is avoided. tion of a single character of the key as the
In this paper we survey these dynamic address. Figure 1 shows a hashing scheme
hashing schemes, with special emphases on that maps a key with its qth character in
the various design issues. the range a-f into the first slot, g-m into
The goals of this paper are to provide a the second, n-t into the third, and u-z into
basic understanding of dynamic hashing, to the last. If we use the first character as the
outline some of the techniques that have key (T,I= l), keys beth, julie, sue, and wally
been developed, and to show how the var- will map into each of the four slots as shown
ious design parameters relate to perform- in Figure lb. If the keys should be dot, ed,
ance. The techniques can be combined in beth, and duffy, however, all would map
various ways to develop new variations of into the first slot. Mapping to the same
dynamic hashing for specific performance address is called a collision. If there is not
goals. We begin with a short discussion of room in the first slot for four records, these
tries, because they provide a means for collisions cause an overflow, which must be
understanding dynamic hashing, as well as handled. The function could be modified to
a good foundation for the performance consider the last letter of a key that would
analysis that has been done on the dynamic evenly distribute these keys over the four
schemes. Next, we describe two basic dy- slots, as shown in Figure lc. This solution
namic hashing schemes, directory and di- works if these are the only four keys used.
rectoryless, so that the various techniques What happens, however, if we attempt to
can be discussed in a concrete context. The map a fifth record? If each slot can hold
techniques address three major issues: only one record and each is full, there will
space utilization, smooth expansion, and be a collision and an overflow. One solution
physical implementation. There is overlap is to handle overflows by keeping a list of
in the handling of these issues, but we records at each slot, but searching such a
separate them for ease of discussion. Some list takes time. In a dynamic environment,
of the techniques mentioned in this survey the length of that list can increase until

ACM Computing Surveys, Vol. 20, No. 2, June 1988


Dynamic Hashing Schemes l 87

a-e

a-f a-f a-f f-j

g-m g-m g-m k-o

I
n-t n-t n-t P-t

U-7, U-Z
u-z

(4 (b) (c) (d)

Figure 1. Static hashing example.

access time exceeds an acceptable limit. addition, as a file of records changes in size,
Another solution in static hashing is to it should no longer be necessary to remap
develop a new function and remap all the all the records; that is, restructuring, which
records. In this case we could create a new can take a long time, should be kept to a
function that maps into five ranges: a-e, minimum. One way to save restructuring
f-j, k-o, p-t, and u-z. If the function used time is to restructure in an incremental
the first character and the new key mark fashion on the parts of the file.
was added to the original keys (beth, julie,
sue, and wally), the records would be dis-
2. DEFINITIONS AND TERMINOLOGY
tributed evenly, as shown in Figure Id. If
the new key was bob, harry, Sharon, or To make our presentation of dynamic hash-
Wendy, however, we would still have a col- ing more precise, we define some terms.
lision and overflow. A different function Implicit in these definitions is the assump-
would be necessary to handle those keys tion that the volume of data is large and
without overflow; a function to handle all therefore stored in random-access second-
possible name keys would be difficult, if not ary storage devices such as disks.
impossible, to determine.
(1) In this paper we only consider single-
The preceding example shows that if we
key retrieval in which a record is rep-
know the keys in advance, we can often
resented by a key k E K, where K is the
design a static hash function to satisfy our
key domain.
needs. However, in a dynamic environment
in which the keys, particularly the number (2) A file F C K is a collection of records.
of keys, would be unknown, a file of records (3) A bucket (or page) corresponds to one
might need to be restructured many times or several physical sectors of a second-
in order to accommodate the variety of ary storage device such as a disk. Let
keys. The goal of dynamic hashing is to the capacity of a bucket be b records.
design a function and file structure that (4) Since the time for disk accesses is con-
can adapt in response to large, unpredicta- siderably longer than the time for main
ble changes in the number and distribution memory access or internal machine in-
of keys. Tree structures can adapt to such structions, the time taken to respond
changes, but they tend to have longer re- to a query is simply measured in terms
trieval times than hashing schemes. Dy- of the number of distinct disk accesses
namic hashing attempts to be adaptable issued. It is assumed that each disk
while maintaining fast retrieval times. In access will fetch a page (bucket) of data.

ACM Computing Surveys, Vol. 20, No. 2, June 1988


88 l R. J. Enbody and H. C. Du

(5) Space utilization is the ratio between n hash table


,
and m . b, where n is the number of 0 r0
records in the file, m is the number of
pages used, and b is the capacity of the Record R with key k 1 r1
page. h(k) = i
2 *2
.=_
3. STATIC HASHING
We begin with a short look at the tradi-
tional hashing schemes [Severance and
Duhne 19761. As mentioned earlier, hash-
ing takes a record’s key and maps it, using
a hash function, into an addressable space
of fixed size (Figure 2). Hash functions are
chosen so that the records will be uniformly
distributed throughout the table for storage
efficiency. Carter and Wegman [ 19791 have n r,
I
shown that classes of such functions to
choose from exist. It is not necessary for Figure 2. A hashing scheme.
the original key distribution to be uniform,
nor is it necessary for the original order (if
any) to be preserved. The latter is a desir-
able trait, but uniformity and order are which an open slot is found. This technique
difficult to achieve at the same time. Rec- is effective because of its simplicity.
ords are accessed by applying the hash (2) Coalesced chaining. When all slots
function to the key and using the function in a primary bucket (and all currently
value as an address into the address space. chained overflow buckets) are full, store a
If more than one record maps to the same new record in a free slot in any unfilled
bucket, a collision occurs; if more than b (primary or overflow) bucket and link this
records map to the same bucket, an overflow new bucket to the current chain of the
primary bucket. Overflow from a primary
occurs (remember that b is the capacity of
a bucket). bucket A, when stored in another primary
A good hash function will randomize the bucket B, may subsequently precipitate the
assignment of keys over the entire address premature overflow of B and result in the
space. Numerous functions have been pro- coalescing of synonym chains. Counter to
posed, and a good comparison can be found one’s intuition, however, this method can
in a study by Lum et al. [1971]. The func- provide better utilization of storage space
tions include division, folding, midsquare, than linear probing [Severance and Duhne
character and bit analysis, radix transfor- 19761. This improvement is due to the fact
mations, and algebraic coding. As one that, as the storage space fills, the perform-
might expect, if given a fixed set of keys, ance of linear probing drops off much faster
some specific transformation will work than coalesced chaining.
best. However, when we are presented with (3) Separate chaining. The coalesced
an arbitrary set of keys, division performs chaining algorithm is modified to preclude
well, and the divisor need not be prime. the merging of synonym chains. For ex-
In general, we can expect collisions and ample, coalescence can be avoided either
overflows, and these overflowing records by storing overflow records in separate and
need to be stored elsewhere. Often a sepa- independent overflow buckets or by relo-
rate area is reserved for overflows. A cating an overflow record whenever its
number of basic alternatives have been storage slot is subsequently required as a
proposed: primary address. With this technique, one
gains slight retrieval advantage over
(1) Linear probing. An overflow record coalesced chaining at the expense of algo-
is stored in the first successive bucket in rithm complexity and storage overhead.

ACM Computing Surveys, Vol. 20, No. 2, June 1988


Dynamic Hashing Schemes l 89

(4) Rehashing. Multiple pseudoran- trade-off between utilization and access


dom probes generated by a sequence of time.
transformations are used either to locate Before beginning our discussion of dy-
a free slot for an overflow record or to namic hashing, we introduce tries. This
find that record for subsequent retrieval. data structure is important because it forms
Because this method generates random a basis for dynamic hashing. For example,
probes, it can have the greatest access time one of the original dynamic hashing
if records are stored on a secondary storage schemes [Fagin et al. 19791 was presented
device such as a disk. The nonlocality of as a collapsed trie. Dynamic hashing is a
the random accesses can result in large combination of hashing techniques with
rotational delays. trie structure. The tries provide a means of
analyzing the performance of dynamic
In general, the problem is that as more hashing and the intuition for comparing
overflows occur, more records are not where dynamic hashing with tree file structures.
the hash function indicates they should be. The discussion of tries is followed by a
Searching for those records can increase presentation of two basic dynamic hashing
the access time beyond the ideal O(l), pos- schemes, which have been chosen to give
sibly to the undesirable O(n), if an open us a basis for explaining the variety of
addressing method is employed, where n is schemes that have been proposed. We then
the total number of buckets used. address the issues that apply to single-key
The traditional static hashing scheme schemes in this section. We discuss the way
was restricted by the fact that the designer the schemes handle space utilization, ex-
had to fix the size of the address space at pansion, and physical implementation. It
file creation time; that is, the number of should be kept in mind that many of these
buckets and the capacity of each bucket techniques can be combined in various
were parameters. If too large an address ways to adapt a specific implementation to
space was chosen, space would be wasted. particular needs.
If the designer guessed too small, the num-
ber of overflows and the access time would
4. TRIES AS A BASIS FOR DYNAMIC
increase. The only solution was an expen-
HASHING
sive reorganization of the whole file. The
reorganization process first allocated a A trie is a tree in which branching is deter-
larger address space, then rehashed all the mined not by the entire key value but by
records (using a different hash function) to only a portion of it. In addition, branching
the larger address space, and finally is based on consideration of that key alone,
released the old address space. Typically, not on the comparison of a search key with
this reorganization was a time-consuming a key stored inside the node. The keys can
process. be from any character or number set, but
With static hashing schemes, there is a we limit our discussion to keys that are
direct trade-off between space utilization binary numbers for the purpose of intro-
and access time in a dynamic environ- ducing dynamic hashing. Any other key
ment. The smaller the space allocation, representation can be easily converted to
the more likely are overflows and poorer binary.
performance. The dynamic hashing Let the key under consideration be a
schemes attempt to reduce this space binary number with n bits. The bits may
versus access-time trade-off and avoid be chosen in any order, but for simplicity
the expensive reorganization. As we discuss we will let the branching at level i be deter-
dynamic hashing, we shall concentrate on mined by the ith most significant bit. As
the issue of space utilization and access an example consider a key with three bits.
time. In general, these schemes have good Figure 3a shows a trie that addresses three
performance (O(1) disk accesses) with pages A, B, and C. Records beginning with
space utilizations that range from 60 to the bit sequence 00 . . . are placed in page
almost 100%. However, we shall still see a A, those with bit sequence 01 . . - are placed

ACM Computing Surveys, Vol. 20, No. 2, June 1988


90 l R. J. Enbody and H. C. Du

(4 (b)
Figure 3. Tries.

EEZ
00 A

01 B

(a)
10

11 l--l=
I I
(b)
Figure 4. Collapsing a trie to a directory.

in page B, and those with bit sequence bits, but creating a minimum depth trie is
1 . . . are placed in page C. If page A over- NP-complete [Comer and Sethi 19771. Dy-
flows, it can be split into two pages, as namic hashing attempts to address both of
shown in Figure 3b. An additional bit is these problems. To begin with, the search
needed to divide the records that were all is shortened by collapsing the trie into a
in A between A and the new page D. Now directory. This directory can then be ad-
*page A contains records beginning with bit dressed by a hash function. If the chosen
sequence 000 . . . , and those with sequence function is uniform, the implied, collapsed
001 * - * will be mapped to page D. trie will be balanced, which, as we shall see,
The trie shown in Figure 3 is very flexible will tend to keep the directory small.
and can grow and shrink as the number of Collapsing a trie to a directory is illus-
records grows and shrinks. The trie, how- trated in Figure 4. The depth of the trie in
ever, must be searched in order to find a Figure 3a was 2 because 2 bits were needed
desired record. If the trie is skewed owing to decide how to distribute the records
to a skewed key distribution, that can lead among the pages. The second bit was not
to long access times. The trie could be needed for page C in the trie, but it is shown
balanced by choosing a different order of to illustrate how the complete directory is

ACM Computing Surveys, Vol. 20, No. 2, June 1988


Dynamic Hashing Schemes l 91

000

w 001
1
0
0
B 010
1

P 9
011

100

101

110

111

(b)

(a)
Figure 5. Doubling the directory.

formed. Once a complete trie has been Now consider the situation in which
formed as in Figure 4a, the bits on the page A overflows and an additional page D
search path can become indexes to the is added. This solution is similar to what
directory. The trie in Figure 4a can then was done in Figure 3. The trie from
be collapsed into the directory shown in Figure 3b can be filled in, as in Figure 5a,
Figure 4b. The first 2 bits of the keys are and then collapsed into the directory shown
now an index into the directory, so there is in Figure 5b. Note that, since an extra bit
no tree to be searched. There are, however, was required to differentiate between pages
redundant directory entries since two direc- A and D, 3 bits and hence three levels were
tory entries are used for page C. This prob- needed. These extra bits result in a direc-
lem of redundant directory entries is tory with eight entries (23). Of those, many
exacerbated by a skewed trie. If a uniform are redundant. Note that page C has four
key distribution can be found, the trie will directory entries.
be balanced, which will minimize the There is a problem with having a direc-
redundancy. We see later that a hash func- tory: It adds a potentially undesirable level
tion can be used to generate such keys, of indirection that can increase access time.
which in turn can be used as indexes to the A different approach to collapsing the trie,
directory. shown in Figure 6, eliminates the directory.

ACM Computing Surveys, Vol. 20, No. 2, June 1988


92 . R. J. Enbody and H. C. Du
I I
00 A

01 B

3
10 c,

11 G

(4 (b)

Figure 6. Collapsing without a directory.

The directory can be avoided by maintain- basic dynamic file systems. The dynamic
ing the pages in a contiguous address space. hashing systems can be categorized as
Eliminating the directory, however, also either directory schemes or directoryless
eliminates the indirection allowed by the schemes that correspond directly to the trie
pointers in the directory. Again, consider models presented in the previous section.
page C in the directory scheme in Figure 4. Classifying the systems will give us a basis
The two pointers to page C were reproduced for describing the important issues for dy-
in the directory from the lowest level of the namic schemes.
trie. With the directory removed, the search In order to have a hashed file system that
path in the trie forms the address of the dynamically expands and contracts, we
page rather than the index to the directory. need a hash function whose range of values
This addressing method requires that there can change dynamically. Most of the dy-
be as many pages in the contiguous address namic schemes use a function that gener-
space as there were entries in the directory. ates more key bits as the file expands and
The result is that addresses 10 and 11, fewer key bits as the file contracts. One
which through the indirection of the direc- such hash function can be constructed us-
tory both pointed to page C, now must point ing a series of functions hi(k), i = 0, 1, . . . ,
to two different pages. Thus, the contents which map the key space K into the integers
of page C must be split across two pages, as (0, 1, . . . , 2’ - 1) such that for any k E K,
shown in Figure 6. Now when the trie is either
collapsed, no directory is necessary. The 2
hi(k) = hi-,(k)
bits that previously indexed the directory
can now be used to address the pages them- or
selves, as shown in Figure 6b.
As with the directory scheme, there must hi(k) = hi-l(k) + 2’-‘.
be a way to expand (and contract) the file There are many ways to obtain the func-
dynamically. However, when a page over- tions hi(k). We might typically use a func-
flows in the directoryless scheme, the file tion H(k), which maps the key space into
expands one page at a time rather than random bit patterns of length m, for m
doubling. We defer explanation of this ex- sufficiently large. Then hi(k) may be de-
pansion until the addressing technique has fined as the integers formed by the last i
been properly defined. bits of H(k). Thus, if
H(k) = b,-1 . . . bzblb,,,
5. BASIC DYNAMIC HASHING SCHEMES
where each bi is either 0 or 1, then
In this section we present a general descrip-
tion of a dynamic hash function and two hi(k) = bi-1 *** blbo.

ACM Computing Surveys, Vol. 20, No. 2, June 1988


Dynamic Hashing Schemes

(a) (b)

Figure 7. Extendible hashing scheme. (a) Before split. (b) Split a into a and x. (c) Split z into z
and y.

Note that hi defined using the last i bits pages). Consider a directory scheme first.
is a different hi than was implied in At this dth expansion stage, the record with
Figures 3-6. We used the prefix rather than key k will be found in the primary page
suffix because the intuition of dynamic whose address is stored in the directory at
hashing based on tries is easier to explain index hd(k). With a directoryless scheme,
using the first bits of H(k). However, the hd(k) will be the address of the primary
details of the schemes are easier to under- page itself. If a primary page overflows, we
stand if we define hi using the last bits. may have to look further to find the record.
Therefore, we will use the suffix for the The first scheme we consider has a direc-
remainder of this survey. tory (hash table) with 2d entries (extendible
In the dynamic hashing schemes, the file hashing [Fagin et al. 1979]), as shown in
goes through a series of expansions. We can Figure 7. As mentioned above, hd(k) will be
begin with expansion 0, a file with two a d-bit number that maps to a directory
pages addressed by 1 bit (bo). Doubling the entry that points to the primary data page
size of the tile (to expansion 1) will create that contains the record with key k. An
a file with four pages addressed by bits bl important characteristic of this directory is
and bo. At the start of the dth (full) expan- that several directory entries may point to
sion, the file contains s = 2d directory en- the same primary data page. If exactly two
tries or, for a directoryless scheme, primary entries point to the same page, all records
data pages (i.e., not counting overflow in that page will have the same value in

ACM Computing Surveys, Vol. 20, No. 2, June 1988


94 l R. J. Enbody and H. C. Du

bits bd+ - - - bl b,; that is, they differ only is, use hd+l(lz), which maps one more bit,
in bit bdel; for example, in Figure 7a, all doubling the number of directory entries.
records in page a will have 00 as their last If this extra bit differentiates between the
2 bits. If four entries point to the same records in page X, those records can be split
page, the records will have the same value between two pages, x and y. This extra bit
in bits bde3 - - . bl b,,, and so on; for example, in the hashing function may not be enough
in Figure 7a, all records in page c will have to divide the records into both pages. We
a 1 as the last bit. may still hash all the records into one page,
To simplify this introduction, we will not which will still overflow. The only solution
allow overflow pages. As a result, if a page is to split again with the hope that the
overflows, a new primary data page must additional bit will be sufficient to differen-
be added. This page can be added in one of tiate among the pages. The probability of
two ways. Assume that page a, which has this situation occurring is smaller with uni-
two pointers pointing to it, overflows as form distributions.
shown in Figure 7a. A new page x will be One problem with splitting a node is
added so that each of those two pointers knowing whether we need to double the
will point to their own page. The records directory when we make a split. This deci-
that were in page a will now be split be- sion can be aided with the addition of
tween the two pages a and x according to another field in each data page and one
their &(k) values. In effect, they will be additional entry for the whole directory.
split using bit ddvl, as shown in Figure 7b. The directory must keep track of the
Remember that hd is defined as the last bits “global depth” d, that is, the number of bits
of H rather than the first, as was indicated needed for addressing. The “local depth” dl
in earlier figures using tries. After the split, stored within each data page is the number
all records in page a have 000 as the last 3 of bits used in addressing that page. There
bits of the hash function H(h). Similarly, are 2d-di pointers to each data page. When
all records in page x have 100 as their last we split page a in Figure 7a, the local depth
3 bits. As a result of the split, there is an was 2 and the global depth was 3. Hence,
implicit buddy system based on the bits of only 2 of the 3 bits being used in the direc-
the hash values. The records in this pair of tory were significant for accessing this
pages have 00 as the last bits of the function page. We can split page a and create two
H(h) and differ in the third from last bit. pages with local depth 3 without doubling
We say the overflowing page split into two the directory. If, however, the global and
pages. In a similar way those two buddy local depths are the same, then all the
pages can be combined into one page if the directory bits are used to access that page.
file were to shrink. This was the case when page x needed to
Consider the case in which page x over- be split, in Figure 7b. If we split such a
flows. This page has only one directory page, the directory must double to get an
entry pointing to it, so it cannot split to extra bit to distinguish among the records
form two buddy pages. An option at this within the page. Notice in Figure 7c that
point would be to use one of the overflow when we double the directory only the local
techniques from static hashing. For exam- depths within the split pages are changed.
ple, we could use separate chaining and add The other pages can be distinguished using
an overflow page with a pointer to it in page the same number of bits as before, so
X. These links, however, can increase re- their local depths remain the same.
trieval time. The solution taken here is to Another problem emerges here from this
increase the number of directory entries so information.
there will be at most one page for every It is through doubling the size of the
directory entry. To do that, the whole di- directory that the file can expand dynami-
rectory must be split, causing the directory cally. In a similar way the file can contract.
to double in size, as shown in Figure 7c. If pairs of pages with single pointers can
This doubling can be accomplished by using map to and tit in single pages, then the file
the function hd+, (k) in place of hd(k); that can shrink by combining those pages and

ACM Computing Surveys, Vol. 20, No. 2, June 1988


Dynamic Hashing Schemes l 95

contracting the directory to half its size. A order. We will give an example to illustrate
page can be combined with the “buddy” this solution.
page that was formed when a page split. Figure 8 shows an expansion of a direc-
After pages are combined, the directory can toryless scheme. We begin with four pages
be contracted when the local depth of all shown in Figure 8a, that is, d = 2. The 2
pages is less than the global depth. As be- bits needed to address those four pages are
fore, the other data pages will not be af- shown in the figure. In this example the file
fected; there will merely be a decrease in will expand whenever any page overflows.
the number of pointers to their data pages. Let page b be the first page to overflow.
The hash function &(k) will become The file will expand by adding page A and
hdml (lz), so the index will be referenced with splitting the records of page a into pages a
one less bit. The directory in Figure 7a, for and A. As before, in order to differentiate
example, could be contracted since only 2 between records for page a and those for
bits are required to differentiate records in page A, an extra bit is needed. Therefore,
any page. h2(k) is still used to address pages b, c, and
The other basic scheme is a directoryless d, but h,(k) is used for pages a and A. This
scheme. We will describe the logical map- situation is illustrated in Figure 8b. Since
ping assuming that the file will be stored in page a rather than page b split, an overflow
contiguous logical address space, as shown page (w ) is needed for page b. After a num-
in Figure 8. In Figure 8a the file is at the ber of insertions, let page d overflow next
beginning of the second expansion (d = 2), (Figure 8b). It is page b’s turn to split, so a
so the tile contains s = 2d = 4 primary data page B is added to the file and the records
pages. In this scheme the directory has been from page b and its overflow page w are
eliminated, so hd(k) will be the address of split between pages b and B. hz(k) is still
the desired primary data page. used to address pages c and d, but hs(k) is
The expansion of this scheme is similar used for pages a, b, A, and B. Assume more
to the previous one in that overflows can insertions, and let page d overflow again.
cause pages to split. It differs, however, in Page C will now be added, and page c will
that the overflowing page need not be the split as shown in Figure 8~. Finally, let page
one that gets split. Instead, pages are split B overflow causing page D to be added and
in sequential order with no regard for page d to split as shown in Figure 8d. Now
whether the page that is split is the one all pages require 3 bits for their addresses;
that actually overflowed. In this directory- that is, all are using function h,(k). This
less scheme, the hash function directly ad- point marks the completion of expansion 2
dresses the pages, so there must be a page and the beginning of expansion 3. Expan-
for each function value, whereas in the sion continues in the same way; contraction
directory scheme there was a directory en- is done in an analogous manner.
try for each function value. The pointers in There are some issues to consider with
the directory scheme provide a level of “in- this scheme. The first is the splittingpolicy.
direction” in the addressing that, for ex- In our example, we split whenever any page
ample, allows fewer pages than directory overflowed. Splitting could be done after a
entries. If we perform an expansion in a fixed number of insertions or overflows, or
directoryless scheme using hd+l, we will after the pages have reached some average
double the number of pages in the same percentage of their total capacity. Another
way in which a directory doubled in the issue is overflow handling. As in static hash-
previous scheme. However, doubling the ing, there are a number of ways to handle
directory in the previous scheme did not overflows. Finally, expansion can be
double the number of pages. Doubling the smoothed out further. In our example, we
number of pages, as opposed to doubling split one page into two pages-the original
the number of directory entries, does not page and a new one. It is possible to arrange
provide the desired smooth growth of the the expansion so a page is split across more
file. The solution is to not double the file than two pages. We examine these issues
but to add pages one at a time in sequential in detail later.

ACM Computing Surveys, Vol. 20, No. 2, June 1988


96 . R. J. Enbody and H. C. Du

(a) lalb/cldj

00 01 10 11

(4

a b C d A 13 C
I
000 001 010 11 - loo 101 110

I
f
(4
a b C d A B C D

000 001 010 011 100 101 110 111

Figure 8. Directoryless example.

Figure 9 shows the notation used for Since there are s = 2d pages, the new ap-
directoryless schemes, also known as linear pended page will be page s + p. The page p
hushing. It shows a file at the beginning of is split so that the records that previously
the dth expansion, so it will contain s = 2d were in page p will be distributed between
primary data pages and will be addressed pages p and s + p using the hash function
by hd(k). As more records are inserted into hd+l(k). The pointer p will then be incre-
the file, new pages will be added to the end mented by 1. When the file size has doubled
of the file. To determine which page will be (i.e., the split pointer p reaches s), there
split into this new page, we introduce an will be 2d+1 pages in the file. We then
index p. At the start of the dth expansion, increment d and return the split pointer
the pointer p will point to page 0 of the file. to the beginning of the file, and the next
When another primary page is appended to expansion begins.
the file, page 0 (pointed to by p) will be With directory schemes, addressing is
split between page 0 and the new page. trivial. The function hd(k) indexes the

ACM Computing Surveys, Vol. 20, No. 2, June 1988


Dynamic Hashing Schemes 9 97

(4
Figure 9. Linear hashing.

4 c

..

0 8- 1 a+fJ
PM
(b)

directory where the pointer is followed to directory scheme without overflows can
the desired page. Directoryless schemes, guarantee two accesses for every request,
however, use both h&) and hd+l(k) to one for the directory and then one for the
access the primary data page. The proce- desired page. A directoryless scheme, on
dure is as follows: the other hand, can keep many requests to
begin
only one access because the directory fetch
if hd(k) 2 p then page := hd(k) has been eliminated. With a directoryless
else page := hd+l (k); scheme, however, overflows are required
if necessary, chase the overflow chain because the pages are split in some prede-
end fined order. The result of having these ov-
erflows is that some requests will require
The basic idea is that to the left of the two or more accesses. This comparison gets
pointer p each page has been split. For each more complicated if one also allows over-
page to the left of p that has been split, flows in a directory scheme. The distinction
there is a corresponding buddy page to the is that directoryless schemes must allow
right of page s - 1; see Figure 9. In order overflows, whereas the directory schemes
to decide which of the corresponding pair may allow overflows.
of pages to access, an extra bit is needed.
The function hd+l (Iz) provides that discrim- 6. SPACE UTILIZATION
inating bit. For records that hash into the
unsplit pages between the pointer p and In this section we deal with the issue of
page s - 1, the function hd(k) is still used. increasing space utilization in the various
Notice that, since hd(k) only uses d bits, it dynamic hashing schemes. Given n records,
cannot reference to the right of s - 1, that w data pages, and b records per page, we
is, the right edge of the file at expansion d. define utilization to be n/(wb); that is, uti-
At the end of the current expansion d, all lization is the ratio of records to space
pages will be addressed by hd+,(k). When reserved for records. A large utilization ra-
we move the pointer p back to point to page tio is desirable because it means that there
0, we will also increment d, so in the next is little wasted space; that is, each page will
expansion the function hd(k) is the same be close to full. There is, however, a price
as the previous expansion’s hd+l (k). to be paid for increasing utilization. Within
Now that both basic schemes have been any scheme, space utilization can be in-
introduced, one of the fundamental differ- creased by modifying the overflow struc-
ences between them can be pointed out. A ture or by adding overflows if they do not

ACM ComputingSurveys,Vol. 20,No. 2, June 1988


98 l R. J. Enbody and H. C. Du
a
Before Split

After Split
a2

(a) (b)
Figure 10. Overflow pages are used to increase space utilization. (a) Split
without overflow. (b) Split with overflow.

already exist. Therefore, as utilization goes be viewed as having a symmetric binomial


up for a particular scheme, the access time distribution. From that we can expect that
tends to increase also. First we look at the with probability <i”)(f) ‘, j keys will be in the
effect in general of overflow pages on uti- left subtree and (k - j) keys will be in the
lization and then at some of the implemen- right subtree. Therefore, the expected num-
tations that have appeared in the literature. ber of leaf pages in each subtree will be
If there are no overflow buckets in a L(j) and L(k - j), respectively. It follows
dynamic hashing scheme, we can expect the that
utilization to be approximately 69% [Fagin
et al. 1979; Larson 1978; Mendelson 19821. [L(j) + L(k -Al.
The intuition behind that figure is as fol-
lows. Whenever a full page is split, two
roughly half-full pages are created. With b Using this approach, Mendelson [1982]
records (one full page) spread over the two showed that
new pages (with total capacity 2 . b), the k
utilization of the new pages is b/2b = 50%, L(k) =b*
as shown in Figure 10a. After a number of
splittings we expect the utilization over the From the expected number of leaf pages
whole tile to be greater than 50% but less the space utilization can be determined as
than lOO%, since pages will tend to be half- follows:
full (recently split) or somewhat full. Over
time, the utilization of such a scheme will k
approach In 2 = 0.69, or 69%, because of Utilization = b . L(k) = In 2 = 0.69.
the successive splitting in half of the pages.
This space utilization of 69% derives The same results were achieved by Fagin
from analysis of tries associated with dy- et al. [ 19791 assuming keys with a Bernoulli
namic hashing and assumes a uniform key distribution. Larson [ 19781 applied similar
distribution [Fagin et al. 1979; Larson 1978; techniques to linear hashing, but his results
Mendelson 19821. Let L(K) be the expected are complicated by the fact that the trie for
number of leaf pages needed in a trie for k linear hashing may not have all leaves at
records. These leaf pages would correspond the same level during an expansion. His
to pages in a dynamic hashing scheme. results were dependent upon the page size
Clearly L(k) = 1 for k I b, where b is the but were very close to the same 69% utili-
maximum number of records in a page. zation.
Considering a trie with k > b, the number Overflow pages allow the utilization to
of records in each subtree at the root can be increased upon splitting. Let us first

ACM Computing Surveys, Vol. 20, No. 2, June 1988


Dynamic Hashing Schemes 9 99

Before Split

. . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . .._.................
a1

After Split After Split


Q2

(a) lb)
Figure 11. Using overflow pages of smaller size. (a) Split with one small overflow. (b) Split
with two small overflows.

assume that the overflow pages are the which give an indication of the relationship
same size as the primary data pages. If both among utilization, primary page size, and
a primary page and its overflow page are secondary page size for a directoryless
full before splitting, that is, a total of 2b scheme. Two conclusions are apparent
records, then after splitting the utilization from those tables. One is that for low uti-
will be 2b/3b = 66%. This situation is il- lizations (e.g., 80% or less), a wide range of
lustrated in Figure lob. There are three overflow page sizes will work. As the utili-
pages after the split because we expect one zation increases, however, the range of good
of the pages still to be overflowing since an page sizes becomes quite narrow. The other
exact split is not guaranteed. This tech- conclusion is that to achieve high utiliza-
nique yields an improvement over the 50% tion, longer overflow chains will be neces-
per split page without overflows, so we sary. These chains have the undesirable
would expect the overall utilization to im- effect of slowing down access time. How-
prove, too. Further improvement can be ever, a side effect for directory schemes is
expected if we have overflow pages that are that the use of overflows can keep the size
smaller than the primary page. For exam- of the directory down. This improvement
ple, let the overflow page size be b/2. If a is especially noticeable if the distribution
primary page and its overflow page are full of records is not uniform.
with a total of 3 b/2 records, then, when A point to consider with respect to direc-
split, there is a good chance that no over- tory schemes is how large a directory is
flow will be needed, as shown in Figure lla. needed for a given number of keys. Fagin
Those 3b/2 records, spread over two pri- et al. [1979] in their original introduction
mary pages with capacity 2 b, yield a utili- to extendible hashing estimated that 2 .
zation of (3b/2)/2 b = 75%. If an overflow Ilog(k/b In 2)l directory entries would be
were needed after splitting, this utilization needed given n records. They assumed that
could be as low as 60%. If there were two the number of records that fell within an
overflow pages before splitting, however, address range was Poisson distributed and
then, with those 2b records, we would ex- showed that the probability of the local
pect an overflow after splitting with a re- depth differing from the global depth by
sulting utilization of 2b/(5b/2) = 80%, as more than 2 was very small.
shown in Figure llb. A better estimate of the size of the direc-
There are many possible overflow page tory was determined by Flajolet [ 19831. He
sizes, each with a different effect on utili- assumed a Bernoulli model that intuitively
zation. The preceding discussion gives a corresponds to the uniform measure over
feeling for that; more details are available n-tuples of the (infinite) binary sequences
in the tables at the end of Larson [1982], that make up the hash keys. He determined

ACM Computing Surveys, Vol. 20, No. 2, June 1988


100 l R. J. Enbody and H. C. Du
Table 1. Expected Directory Size” so the splitting policy chosen ultimately
b affects utilization.
Rather than split a page as soon as it
n 5 10 20 50 100 200
overflows, splitting can be deferred by uti-
lo3 1.50K 0.30K O.lOK O.OOK O.OOK O.OOK lizing an overflow page. This deferral is
lo4 25.60K 4.80K 1.70K 0.50K 0.20K O.OOK
lo5 424.10K 68.20K 16.80K 4.10K 2.00K l.OOK
implicit in the directoryless scheme because
lo6 6.90M 1.02M 0.26M 62.50K 16.80K 8.10K pages are split systematically, not necessar-
10’ lll.llM 11.64M 2.25M 0.52M 0.26M 0.13M ily when they themselves overflow. Defer-
a From Flajolet [1983]. ring can also be applied to a directory
scheme. In a directory scheme, deferring
the splitting of a page is particularly helpful
if splitting that page would cause the direc-
a probability that a trie built on n keys has
tory to double. Consider Figure 7. In Figure
height less than or equal to h. This proba-
7c many directory entries point to the same
bility is equal to the probability for n num-
page, resulting in a waste of directory space.
bers in [0, l] such that no more than b fall
If page x had been allowed to overflow, the
into any of the intervals determined by
split that would have resulted in the dou-
directory indexes. This turns out to be a
bling of the directory could have been de-
counting problem of permutations with
ferred. If a number of pages were allowed
limited repetitions. Although Flajolet’s
to overflow before splitting, the result
probability formula is too complex to repro-
might be a higher utilization, especially of
duce here in its entirety, we can give his
the directory.
reduced coarse approximation for directory
Deferring splitting is closely related to
size:
the sharing of overflows. The former will
-
e nl+l/b z -
3.92 n 1+1/b
tend to increase the number of accesses
b log 2 b * required because of the increased use of
overflows. This increase in retrieval cost,
In addition, Flajolet provided a table that however, can be partially offset by sharing
displays the expected values of the direc- the overflow pages. Sharing the overflow
tory size using extendible hashing for var- pages decreases accesses by allowing a
ious values of n, the number of records, and given overall space utilization to be reached
b, the page capacity (see Table 1). The using fewer overflow pages. With fewer ov-
entries show the directory size in thousands erflow pages, the average number of disk
(K = 103) and millions (M = 106). accesses should be smaller. This sharing of
overflow space is common in static hashing
where overflows are often kept in one ov-
7. OVERFLOW HANDLING TECHNIQUES erflow area.
7.1 Splitting Control Deferred splitting was introduced by Lar-
son [1978] as a means of improving utili-
When we described the basic schemes in zation in directory schemes. Scholl [1981]
the Introduction, we used a simple splitting investigated the details of implementing
control policy. In both the directory and deferred splitting and added sharing of the
directoryless schemes, a page was split as overflow buckets to improve utilization of
soon as any page overflowed. In this section the overflow pages. Sharing overflow pages
we describe other ways of determining is similar to having overflow pages that are
when to split a page. We give this descrip- smaller than the primary data pages. A
tion here because splitting control has a similar result can be achieved without hav-
direct effect on how much overflow will be ing special overflow pages. Veklerov [ 19851
tolerated. One can delay splitting by allow- logically paired all pages; each page and its
ing pages to overflow; the longer splitting pair were referred to as “buddies.” When a
is delayed, the longer the overflow chains page overflowed, Veklerov proposed put-
become. In the previous section we saw that ting the overflow records into the buddy
overflows can improve space utilization page if the buddy itself was not full. When

ACM Computing Surveys, Vol. 20, No. 2, June 1988


Dynamic Hashing Schemes 9 101
there were too many overflow records or from the previous level, so each level should
the buddy needed the space, the original have fewer records than the previous level.
page would split. Deferring the splitting Over time the file of overflows at any
achieved two goals: It increased utilization level can expand to accommodate an ex-
and by deferring splitting it kept the size panding number of overflows from the
of the directory down. previous level. They can contract in a
Once it has been decided to defer split- similar fashion.
ting, a policy of when to split is needed. An implementation using linear hashing
Unfortunately, in order to get a high utili- at each level is shown in Figure 12 [Rama-
zation more overflow buckets are needed, mohanarao and Sacks-Davis 19841. Unlike
which can increase the access time. A nat- most overflow schemes, the overflows in
ural solution is to split when utilization recursive linear hashing are not explicitly
reaches some threshold. For example, a linked from the primary page. When an
page could be split whenever utilization of attempt is made to insert a record into a
the file became greater than 80%. This full primary page, no explicit overflow page
policy allows coarse control of both the is created. Instead, an attempt will be made
utilization and the length of the overflow to insert the record into the next level.
chains. Thus, a level has the shared overflows from
Since overflows are required for directo- the previous level, but in a dynamic rather
ryless schemes, splitting control policies than static way. A file at any level behaves
were a natural part of their development. exactly like a normal dynamic file and will
Litwin [1980] presented two methods of expand as more records are inserted (or
splitting for linear hashing: controlled attempted to be inserted) into that level.
splitting and uncontrolled splitting. In un- Previously, when a page split, it would col-
controlled splitting the node pointed to by lect records from its own overflow pages
the “split” pointer p is split when any page and divide them between itself and the
overflows (Figure 9). Since there is little added page (p and s + p in Figure 9). In the
correlation between the overflowing page course of the split, one or both of the pri-
and the split page, the utilization of this mary pages might still require overflow
scheme is low, that is, approximately 60%. pages. In recursive linear hashing, a page
Controlled splitting is a deferred splitting collects overflows not from its own overflow
policy where the node pointed to by the pages but from the dynamic file in the next
“split” pointer is split only when the space level. If either primary page still overflows,
utilization is above some threshold, for ex- those overflow records will be put back into
ample, 80%. Pages are allowed to overflow the next level. Although the number of
until the desired space utilization is levels can in theory continue indefinitely,
achieved, and only then is the next page it has been claimed that with a uniform
split. distribution more than three levels are
rarely required [Ramamohanarao and
Sacks-Davis 19841. However, a nonuniform
7.2 Recursive Overflow Handling
distribution could create many levels of
Once the decision has been made to share overflows.
an overflow space, a natural question is how In a normal dynamic scheme, the process
should that space be managed. A static of splitting a page includes collecting over-
overflow space could defeat the flexibility flows to put into the newly created pages.
of a dynamic hashing scheme. One possi- The same is true with recursive hashing.
bility is to manage the overflow space as a When a node at level i is split using some
dynamically hashed file. Since the pages in control policy, overflow records are col-
the overflow space may themselves over- lected from level i + 1 to fill the resulting
flow, this process can be extended in a pair of pages. Overflow records are collected
recursive fashion creating many levels of from successive lower levels (i + 2, . . .) so
dynamic files, one overflowing into the that any nonfull page cannot have records
next. Each level has the shared overflows that overflow into the next level. Since any

ACM Computing Surveys, Vol. 20, No. 2, June 1988


102 ’ R. J. Enbody and H. C. Du

...
Primary Data Pages I
so-1 so+po
O Pf

Overflow Level One


.
S,-1 sI+PI
OPY

Overflow Level Two

M 6-2-l s2+p2
P2

Figure 12. Recursive linear hashing scheme.

lower level is made of overflows from higher Rather than have overflow pages as sepa-
levels, we expect the lower level files to be rate entities, he allowed the primary page
smaller. In a linear hashing implementa- to expand over a number of contiguous
tion by Ramamohanarao and Sacks-Davis pages. The page then became a “multipage
[ 19841, all records stored at level i + 1 node,” as shown in Figure 13. A fixed upper
whose primary page at level i is q will be limit was placed on the size of the directory,
stored in a single page at level i + 1, and and once that limit was reached it was the
the address of this page is determined by q. nodes and not the directory that expanded.
Records are addressed in recursive linear A record was accessed first by consulting
hashing using the following procedure. The the directory and then by searching the
procedure search(p, 1, k, status), which corresponding multipage node.
searches page p of level 1 for a record with The multipage node can be managed in
key k and returns with status either found a number of ways. The simplest would be
or not-found, is assumed to exist. Assume to keep the records in the same order in
that the current number of levels in the file which they were inserted and perform a
is max-level: sequential search. A more complicated ap-
begin
proach would be to manage each multipage
1 := 0; (* initialize level to zero *) node as a dynamically hashed tile. For ex-
repeat ample, the number of pages in the node
I:= 1+ 1; could be kept in the directory, which would
if&(k) 2 p then page := hd(k) allow one to manage the contiguous multi-
else page := hd+l (12); page node as a directoryless hashing
search(page, 1, k, status); scheme. Each multipage node would grow
until status = found or 1= max-level and shrink as a normal directoryless hashed
end file. One could even allow overflows within
The recursive scheme has been imple- the multipage node directoryless file.
mented as a linear hashing scheme, but a A different multipage node proposal was
directory scheme can be implemented in a made by Ramakrishna and Larson [1988].
similar way. Their composite perfect hashing is a static
hashing scheme that can achieve high uti-
lization without overflows. With no over-
7.3 Multipage Nodes
flows, only one disk access is required, thus
Lomet [1983] proposed a different way of the name “perfect” hashing. Using this
handling overflows in directory schemes. perfect hashing with a directory scheme

ACM Computing Surveys, Vol. 20, No. 2, June 1988


Dynamic Hashing Schemes . 103
directory multipage nodes
00

01

10

11

Figure 13. Bounded index exponential hashing.

preserves the two-disk access bound found details of that idea, however, have not been
in extendible hashing. Expansion of com- published.
posite perfect hashing requires rehashing,
but the amortized cost is kept low by keep-
ing the frequency of expansions small. 8. EXPANSION TECHNIQUES
Dynamic hashing schemes can have oscil-
7.4 Internal Storage of Overflows
latory performance. If the hash function
can distribute the hashed keys uniformly
Some static hashing techniques have been over the pages of the file, the pages fill up
directly applied within the dynamic hash- uniformly and become completely filled al-
ing framework. For example, the contig- most simultaneously. Within a small period
uous address space of directoryless dynamic of further file growth, a majority of file
hashing lends itself to open addressing pages overflow, and their entries must be
mentioned earlier. Larson [1985] proposed split over pairs of pages. The result is that
that open addressing be used for storage of utilization increases from 50 to almost
overflows in a directoryless scheme. Open 100% and then suddenly falls to 50% during
addressing searches the file from the over- the short splitting period. In addition, the
flowing page in a linear fashion to find a cost of doing an insertion is comparatively
nonfull page in which to place the overflow- low at low utilizations but is considerably
ing record. In dynamic hashing the problem higher during the splitting period because
is that pages in the unsplit region will tend so many insertions lead to page splitting.
to need the most overflow pages, and few Finally, if overflow records are required by
nonfull pages can be expected until some the technique, then the frequency of occur-
split pages are reached, for example, to the rence of overflow increases as utilization
right of page s - 1 in Figure 9. The solution approaches lOO%, resulting in a sharp in-
is to order the splitting of pages so that the crease in the cost of insertions and searches
split and unsplit pages will be interleaved as overflow page accesses increase.
throughout the file. That will tend to put Fortunately, random variations in hash-
nonfull split pages (with room for over- ing functions, nonperfect uniformity of the
flows) near the full unsplit pages, which functions, and large files all work to keep
will keep the overflows nearby. Access time those oscillations in check. However, a cou-
is kept reasonable by keeping the overflows ple of techniques have been proposed to
close, which prevents the overflow search explicitly smooth out the expansion. First
from getting too long. we look at one that deals with a way to
Chaining can also be used. Mullin [ 19811 smooth out the expansion given a uniform
proposed a directoryless scheme that stores distribution. The second smooths out the
all overflows in the empty space of primary process by changing the distribution to an
data pages that are not full. These over- exponential one. We then look at directo-
flows are then chained together. Vitter ryless and directory schemes that have
[1982] mentioned that coalesced chaining been designed to handle such nonuniform
may be applicable to dynamic hashing. The distributions.

ACM Computing Surveys, Vol. 20, No. 2, June 1988


104 l R. J. Enbody and H. C. Du
8.1 Uniform Key Distribution
In this section we will conside schemes to
improve access time. We ncentrate on
directoryless schemes be / ause directory
schemes without overflows are optimal 0133456789
with respect to access time given their N
structure. P
The problem with our basic directoryless (4
scheme (linear hashing) is that the over-
flow chains get too long near the right
end of the unsplit region, that is, near
page s - 1 = 7 as shown in Figure 14. This
problem occurs because these pages are the
last to be split, so their overflow chains can
become quite long. The recently split pages
are underutilized, and the ones that are last
to be split get overutilized with long chains.
One way to cut down on the length of those
chains is not to wait so long before splitting
the buckets. Larson [1980] proposedpartial P
M
expansion, which allows the file to go
(b)
through its doubling in smaller steps. By
expanding in smaller steps, each page will Figure 14. Uneven distribution of overflow chains.
get a chance to expand partially long before
the whole file is doubled. For example,
consider a basic directoryless file that
splits after L insertions, and let there be of 2b records), then the three pages 0, 4,
s pages. The last page to be split, that is, and 8 (with combined capacity 3b) will be
page s - 1, would be split after sL inser- roughly two-thirds full (2b/3b). If we were
tions. Now assume that the file expanded splitting with only one page in each group,
in two stages, with each page partially that is, our basic directoryless scheme, then
expanded during each stage. Page s - 1 the final utilization would only be one-half
will be partially expanded after only for the two resulting pages.
(sL)/2 insertions. As a result, we would After one partial expansion, we have the
expect the overflow chains to be shorter situation in Figure 16. The pointer p has
and the average access time less. made one pass through the four groups and
Larson’s linear hashing with partial ex- has returned to group 0. We now have
pansion works by dividing tiles into groups. visited each page in the original tile and
We will begin with a look at the concept have added four pages to the file. Thus we
and consider the details of addressing later. are halfway to a full doubling of the file but
Let us examine one partial expansion for have already visited each original page
the case of eight pages (d = 3, s = 2d = 8) once. Group 0 now consists of pages 0, 4,
that are paired in four groups of size 2. and 8. The next page added will be page 12,
The pages are paired so that page i and and the records from pages 0,4, and 8 will
page i + 4 are in the same group, called be spread over pages 0, 4, 8, and 12. If
group i. When we decide to split a page, we the three pages being “split” were full,
actually partially split the pair of pages in then after the “split” the four pages will
a group. Consider group 0; when we add a be 3b/4b = 75% utilized. When this partial
new page (page 8), we shall spread the expansion is finished, there will be 16
records from both pages in group 0 across pages, that is, the file will have doubled and
the three pages (the original pair in one full expansion will have been com-
group 0 plus the new page), as shown in pleted. Now, instead of having a group with
Figure 15. If pages 0 and 4 are full (a total four pages, we return to groups with two

ACM Computing Surveys, Vol. 20, No. 2, June 1988


Dynamic Hashing Schemes l 105

group 0

0 1 2 3 4 5 6 7 8 9 101112131415

(a)
(a)

group 0

0 1 2 3 4 5 6 7 8 9 101112131415
~~

P
b)

(b) Figure 17. Partial expansion, beginning of next full


expansion. (a) d = 3. (b) d = 4.
Figure 15. Partial expansion. (a) Before split. (b)
After split.

The problem is how to address the file as


group 0
it expands. Larson [1980] and Ramamo-
I I 1 hanarao and Lloyd [1982] have each pro-
A.. h.. J posed a scheme for addressing the records.
::::
:::: ::::
:::: ::::
....
:::: :::: .::::
::::
... To confuse the issue they each defined the
::::
:::: ::::
:::: :::: word “group” differently. We used Larson’s
::::
:::: ::::
:::: ::::
:::: definition above. Here is the scheme pro-
0 12 3 4 5 6 7 8 9 1011 posed by Ramamohanarao and Lloyd
[1982]: The address is a function of the
P/“’ record’s key k, the depth of the (full) ex-
(4 pansion d, the partial expansion n, and the
split pointer p. The variables k, d, and p
are as we have defined them earlier. Only
the variable n is new. The address of the
desired record’s primary data pages is
r(k, d, n, p) = group + 2d-’ . offset,

where group is as we defined it above and


offset is the offset (in terms of number of
pages) within the group. Offset will be de-
termined by another hash function ii(k),
(b) which maps into the number of pages
Figure 16. Partial expansion, second pass. (a) Before within a group. The value Zd-l is the dis-
split. (b) After split. tance between pages in a group. For exam-
ple, to address page 8 in Figure 15, we first
go to page 0 (group = 0), then skip over
pages as shown in Figure 17. The number two (offset = 2) sets (of size 2d-’ = 4) of
of groups has now increased to eight groups pages to get to page 8. The value of group
with two pages each and the whole process will be determined by the hash function
will repeat. hd(k) as in the basic directoryless scheme.

ACM Computing Surveys, Vol. 20, No. 2, June 1988


106 l R. J. Enbody and H. C. Du
The function i,(k) is similar to h,&) can then be mapped onto a directoryless
but maps into the size of a group in the cur- scheme.
rent partial expansion (e.g., three pages for Directoryless schemes have a fixed origin
group 0 in Figure 15b), and i,-l(lz) maps and expand by adding pages at one end of
into the size of a group in the previous the file. Spiral storage differs in that, dur-
partial expansion (e.g., two pages for ing expansion, pages are deleted at one end
group 0 in Figure 15a). The address is then of the file and added at the other end. The
determined by result is a contiguous file space that moves
through memory in a way similar to a
dk, d, n, P)
:= if /b(k) 2 p queue. Expansion is achieved by adding
then k,(k) + 2d-1 . i,(k) more pages than are deleted. Records from
else hd+I(k) + 2d-’ . in-~(k). the deleted portion are spread across the
added pages. The essential part of the spiral
The above scheme, presented by Rama- approach is an addressing function that can
mohanarao and Lloyd [1982], moves rec- adapt to the remapping of the records that
ords around within the group as partial are moved. We will begin with a discussion
expansions occur. Larson’s original scheme of the spiral addressing technique and then
is similar, but it only moved the records show how it maps into a directoryless
that were destined for the newly expanded scheme.
page. The reader is referred to Larson Spiral storage works by taking keys
[1980] for details. that initially have a uniform distribution,
giving them an exponential distribution,
8.2 Nonuniform Key Distribution and then assigning logical page addresses.
This technique is illustrated in Fig-
8.2.1 Nonuniform Key Distribution for ure 18. The keys are uniformly distributed
Directoryless Schemes along the x axis between c and c + 1
As explained earlier, with our basic direc- using a function G(c, lz). G takes a key k,
toryless scheme (linear hashing) and a applies a uniform hash function hash(k),
uniform distribution we can expect more 0 5 hush(k) 5 1, and then distributes the
overflow pages to the right of the pointer p values between c and c + 1:
(and to the left of page .s - 1) than the left G = rc - hush(k)1 + hash(k).
(Figure 14). This situation occurs because
those pages to the right are split later than The y values are determined by bG, with
those to the left, so they will tend to get b > 1; the logical page addresses are Lb'J.
more overflows. In other words, we expect As the interval (c, c + 1) is moved to the
the pages to the right to be overutilized and right along the x axis, the size of the inter-
those to the left to be underutilized. A val (b’, bC+l) will increase, causing the
decreasing exponential distribution of keys number of pages to increase. Moving the
can map more records to the left end of the interval (c, c + 1) to the left, causes the
file than the right. This distribution will number of pages to decrease. During expan-
. tend to map more records into pages that sion the parameter c is chosen to eliminate
are already split (i.e., to the left of the the current first logical page completely.
pointerp); that is, more records get mapped The new value of c is obtained from the
to the underutilized pages and fewer to the inverse of the exponential function
overutilized pages. The result is a more new-c = log&irSt + l),
uniform distribution of overflow pages.
An approach for handling such an expo- where first is the current first logical page.
nential distribution of keys in a directory- This process is best illustrated with an
less scheme is spiral storage [Martin 1979; example as shown in Figure 19. Initially,
Mullin 19851. The spiral approach is a the file is as shown in Figure 19a. The file
neat, logical description of a way of hand- is shown on the left, and the mapping on
ling an exponential distribution of keys. the right. All records within the interval
The resulting file, which has no directory, (c, c + 1) map into pages 1 and 2. In

ACM Computing Surveys, Vol. 20, No. 2, June 1988


Dynamic Hashing Schemes 107

bc

bC+l
Figure 18. Graph of bc versus G.

b”

Figure 19b, the file has expanded, deleting actual pages used for the logical pages of
page 1 and adding pages 3 and 4. The Figure 19. The numbers within the pages
mapping on the right shows how all records are the logical numbers of the pages; the
within the interval (c, c + 1) map into pages numbers outside the pages are the actual
2-4. Shifting the interval (c, c + 1) to the numbers of the pages. In the first expan-
right, changes the mapping so logical page sion, when logical page 1, which resides in
addresses fall within pages 2-4. The param- actual page 0, is mapped into logical pages
eter c is selected so that page 1 can be 3 and 4, the actual page 0 is reused for the
eliminated. The formula to determine the logical page 3. Logical page 4 (physical page
new c is c = logb( first + l), where first is 2) is added to the (right) end of the file. In
the leftmost page, that is, the page to the second expansion, when logical page 2
be eliminated. In Figure 19b, first = 1 so is deleted and logical pages 5 and 6 are
c = log,(l + 1) = 1. A further expansion is added, actual page 1 will be reused for log-
shown in Figure 19c, with page 2 deleted ical page 5. Logical page 6 will be added to
and pages 5 and 6 added, the corresponding the end of the file. An additional expansion
mapping is shown on the right with is shown in Figure 20d: Logical page 3 is
c = log,(2 + 1) = log,3. Contraction is deleted, and logical pages 7 and 8 are added.
performed in an analogous fashion, with The procedure to find the actual page
the interval (c, c + 1) sliding to the left. from the logical page is as follows: If the
One apparent problem in what has been logical page resides in the first instantiation
described so far is that the file moves of the actual page, then the actual page is
through the address space. Unless the space the difference between the last logical page
taken up by the “deleted” pages can be and the current first logical page, that
reused, that space can be wasted in most is, the current number of logical pages.
operating systems. The solution is to map For example, the actual page for logical
these logical pages into actual pages. The page 8 is 8 - 4 = 5. To find a reused page,
result is a file structure that looks like our such as where logical page 7 resides, one
basic directoryless scheme. must search back through the logical in-
Let us return to the previous example to stantiations to find when the actual page
illustrate how this mapping can be done. was instantiated. For example, we know
When a page is “deleted” it should be that logical page 7 is in actual page 0.
reused immediately. Figure 20 shows the That actual address can be determined as

ACM Computing Surveys, Vol. 20, No. 2, June 1988


108 . R. J. Enbody and H. C. Du

(a) p 3
a
1 f!
g 2
e

c H-1 G

(b)
p 3
a

g 2
e

c c+l

7
T /G

Figure 19. Spiral storage example.

follows: The records in logical page 7 were remember that logical-page = Lb’J:
previously mapped into logical page 3 be-
FUNCTION actual-page (logical page)
fore the last expansion. Before that expan- (* determine immediate “ancestor” pages *)
sion, the records in logical page 3 had high := L(1 + logical-page)/bl
mapped to logical page 1, which resided in low := Llogical-page/bl
the initial actual page 0. Therefore, logical if low < high
page 7 resides in actual page 0. then
The recursive function to determine the (* reused page; keep searching *)
actual page given the logical page follows; actual-page := actual-page(low)

ACM Computing Surveys, Vol. 20, No. 2, June 1988


Dynamic Hashing Schemes 9 109

consider page c in Figure 7c, which has


(a) 1 2 eight pointers pointing to it. Fully half of
I0 the directory entries point to one page,
which amounts to a tremendous waste of
1
directory space. Bounded Index Exponen-

rbl
tial Hashing (Section 7.3) was proposed as
a way of handling nonuniform distributions
by expanding some nodes more than others;
(b) 3 2 4 Figure 13 shows such a distribution. In that
figure there is an exponential distribution
of pages (and records), yet the directory is
fully utilized.
A different way of handling a nonuniform
distribution with a directory is a scheme
presented by Larson [1978]. His approach
is a variation of our basic directory scheme
in that it does not store its directory as a
simple table. Larson proposed a forest of
index tries whose roots are located by hash-
ing to a table. The data pages are leaf nodes.
Each index trie can grow or shrink accord-
ing to the load on the tile. Whenever a leaf
node’s page gets too full, the node can split
000 001 010 011 100 into two. A page is addressed by first using
a subset of the key to hash into the table
Figure 20. Mapping of logical pages to actual pages. to locate the root of the appropriate trie.
The (binary) trie is then searched using the
bit values to determine which branch to
else take. Larson did not let his directory grow
(* low = high; found first instantiation or shrink, only the index tries did that. His
of page *) scheme, however, can be modified to allow
actual-page := logical-page-low a dynamic directory. This modification is
end possible because the directory can be
Mullin [1984] presented a scheme to viewed as a trie collapsed into a table.
unify linear hashing (our basic directoryless Otto [1988] has proposed a method of
scheme), linear hashing with partial expan- implementing an extendible hashing
sions, and spiral storage. By using a piece- scheme using a balanced m-way tree as a
wise linear curve with a shape that is directory. This results in a linear growth of
similar to the curve in Figure 18, he showed the directory even in the case of nonuni-
how to implement linear hashing and linear form distributions.
hashing with partial expansions using the
spiral approach. 9. PHYSICAL IMPLEMENTATION

8.2.2 Nonuniform Expansion of a Directory


Up to this point we have not considered the
Scheme
issue of physical implementation. For ex-
ample, a directoryless scheme assumes a
As mentioned earlier, a nonuniform distri- contiguous address space. If this space had
bution can smooth out the expansion pro- to be contiguous in physical memory, a
cess in dynamic hashing schemes. With our large piece of physical memory would have
basic directory scheme, however, a nonun- to be reserved, which would remove some
iform hash function would lead to a non- of the advantages of dynamic hashing. The
uniformly utilized directory. For example, easiest solution is to let the operating

ACM Computing Surveys, Vol. 20, No. 2, June 1988


110 l R. J. Enbody and H. C. Du

Figure 21. Physical mapping of a directoryless scheme.

system do most of the work. Let the dy- (size 2q+1) than the block thatp is currently
namic file system map into a contiguous in (size 2q) and store pages that have al-
logical space and let the operating system ready been split, thus the need for twice as
handle the mapping to a dynamic, possibly many pages. The last directory entry (2’) is
noncontiguous, physical address space. a temporary storage area in which pages
In this section we examine a proposal by currently being split can be stored. Pages
Ramamohanarao and Sacks-Davis [1984] from the block currently containing p are
that considers a mapping of a contiguous split into this temporary storage area, thus
directoryless space to a physical address the need for twice as many pages. The
space that is allocated in quanta greater shaded region to the left of p represents
than single pages, but smaller than the pages currently not being used. They have
whole address space of the file. This map- been split and remapped into the unshaded
ping can be done with a fixed table of region in the left part of the temporary
pointers to contiguous blocks of memory. storage page. The shaded part of the tem-
Figure 21 shows a scheme without partial porary storage page contains the pages that
expansions. Quanta of memory are allo- the rest of the pages in the block currently
cated in blocks with 2” pages for various containing p will be split into. A key part
values of x. At the beginning of a full ex- of this technique is that when a page is split
pansion, the physical blocks will all be the the resulting pair of pages will be stored in
same size; as the dynamic hash file ex- physically adjacent pages.
pands, larger blocks of memory will be used. A closer look at how addressing is done
The figure shows two different size blocks will make the structure clearer. The depth
in use, one with 2q pages and the other with will be i + q, where each block at the current
2 q+l pages. The blocks that are adjacent in level is of size 2q, and there are i directory
the table are adjacent in the logical scheme. entries. Let x be the number of pages to the
The pointer p in Figure 21 is the same split left of p, that is, the shaded region in Figure
pointer shown in Figure 9. Blocks above 21. Since p denotes the number of pages
the split pointer p in Figure 21 are larger that have been split during the current

ACM Computing Surveys, Vol. 20, No. 2, June 1988


Dynamic Hashing Schemes l

Figure 22. Physical mapping of a directoryless scheme with partial expansion.

expansion, if hi+,(k) < p and the leftmost i bits choose


directory entry ip
x = p mod 2q, then the primary page has been split and is
in the buffer block 2’ - 1; the offset will be
ip = p div 29 determined by the next q + 1 bits.
end
We form the address by considering the
leftmost i + q bits of H(k) to get hi+,(k). If we are using partial expansions, there
Addressing is then done by the following will be a contiguous block of storage for
procedure: each page in a group. For example, if each
group has two pages, there will be two
begin blocks of storage. When a split occurs and
if hi+,(k) > p
the two pages are split over three pages,
then the leftmost i hits determine the direc-
tory entry and the next q bits determine the this expansion will be done into a third
offset within that block. block of storage. Figure 22 illustrates a
if hi+,(k) < p and the leftmost i bits choose a partial expansion in progress. Once again,
directory entry less than ip p is in the ip region. Some of the records
then the primary page has been split and is have been split into three pages whereas
in one of the newly formed blocks of size others to the right of p are still stored in
2q+‘; the offset will be determined by the two pages. The directory for partial expan-
next q + 1 bits. sions must be able to have as many pointers

ACM Computing Surveys, Vol. 20, No. 2, June 1988


112 l R. J. Enbody and H. C. Du
as there are possible pages within a group. Most schemes assumed a uniform distri-
On successive full expansions the size of bution, but little work has been done on
the contiguous memory blocks will double nonuniform distributions and their effect
in size, as above. on performance. It has been proposed that
Consider the mapping illustrated in Fig- in some cases a decreasing exponential dis-
ure 21. On the left side of the figure is a tribution is desirable. This distribution can
directory, with each entry pointing to a work well with a directoryless scheme,
directoryless file. Compare that with Figure but any skewed distribution can cause a
13, which illustrates Lomet’s Bounded In- directory scheme to generate too large
dex Exponential Hashing approach [Lomet a directory. Also, an increasing exponential
19831. In that scheme each multipage node distribution would amplify the problems
pointed to by a directory entry could behave that a decreasing exponential distribution
like a simple directoryless scheme. The re- attempts to solve in a directoryless scheme.
sult is that both figures look somewhat like The handling of overflows is another im-
a directory of directoryless files. This ob- portant part of the dynamic hashing
servation could prove useful for developing schemes. If one wants space utilization
a combined scheme. above 69%, then overflows are necessary in
either basic scheme. There are many pro-
posals about how to handle overflows, but
10. CONCLUSION
it is not clear what is the best way to
Both directory and directoryless schemes manage them.
should be able to handle dynamic file sys- Dynamic hashing has developed to the
tems better than static hashing schemes. point at which useful implementations can
They can expand and contract gracefully be made. In this paper we only discussed
while maintaining reasonable access times the design issues for single-key dynamic
and space utilization. In this paper we have hashing. Many of these schemes can be
presented a number of variations that let extended to fit multikey associative retriev-
one adapt a scheme to one’s needs. als that are more desirable in many data-
A comparison of performance has not base applications. The design issues for
been done between basic schemes. How- multikey retrieval merit a survey of their
ever, some observations can be made. own.
Directoryless single-key schemes have the
potential for lower average access times ACKNOWLEDGMENT
since they have eliminated the directory.
The partial expansion variations with This research was supported in part by NSF grants
directoryless schemes allow a file to expand DCR-8405498 and DCR-8420935.
with more grace than the doubling that
is necessary with a directory scheme. REFERENCES
Directory schemes also have unique ca-
pabilities. Since overflows are optional in CARTER, J. L., AND WEGMAN, M. N. 1979. Universal
a directory scheme, an upper bound can classes of hash functions. J. Comput. Syst. Sci.
28, 2 (Apr.), 143-154.
be placed on the maximum overflow al-
COMER, D., AND SETHI, R. 1977. The complexity of
lowed and hence the maximum access trie index construction. J. ACM 24, 3 (July),
time. This limit could be important in some 428-440.
performance-oriented applications. Direc- FAGIN, R., NIEVERGELT, J., PIPPENGER, N., AND
toryless schemes cannot place a similar STRONG, H. R. 1979. Extendible hashing-A
bound on the length of overflows. Also, the fast access method for dynamic files. ACM Trans.
Database Syst. 4,3 (Sept.), 315-344.
subtle effects of physical implementations
FLAJOLET, P. 1983. On the performance evaluation
on performance of many of the variations of extendable hashing and trie searching. Acta
presented are not clear. hf. 20,345-369.
There are unresolved issues with respect LARSON, P.-A. 1978. Dynamic hashing. BIT 18,
to the distribution of hashed key values. 184-201.

ACM Computing Surveys, Vol. 20, No. 2, June 1988


Dynamic Hashing Schemes l 113
LARSON, P.-A. 1980. Linear hashing with partial MULLIN, J. K. 1984. Unified dynamic hashing. In
expansions. In Proceedings of the 6th Conference Proceedings of the 10th Conference on Very Large
on Very Large Databases (Montreal). Very Large Databases (Singapore). Very Large Database
Database Foundation, Saratoga, Calif. Foundation, Saratoga, Calif., pp. 473-480.
LARSON, P.-A. 1982. Performance analysis of linear MULLIN, J. K. 1985. Spiral storage: Efficient dy-
hashing with partial expansions. ACM Trans. namic hashing with constant performance.
Database Syst. 7,4 (Dec.), 566-587. Comput. J. 28,3,330-334.
LARSON, P.-A. 1985. Linear hashing with overflow- OTTO, E. J. 1988. Linearizing the directory growth
handling by linear probing. ACM Trans. Database in order preserving extendible hashing. In Pro-
S’yst. IO, 1 (Mar.), 75-89. ceediws of the 4th Data Erxineering Conference.
LITWIN, W. 1980. Linear hashing: A new tool for file IEEE,-New York, pp. 580-588. - .
and table addressing. In Proceedings of the 6th RAMAKRISHNA, M. V., AND LARSON, P.-A. 1988. File
Conference on Very Large Databases (Montreal). organization using composite perfect hashing.
Very Large Database Foundation, Saratoga, ACM Trans. Database Syst. To be published.
Calif., pp. 212-223. RAMAMOHANARAO, K., AND LLOYD, J. W. 1982.
LOMET, D. B. 1983. Bounded index exponential Dynamic hashing schemes. Comput. J. 25, 4,
hashing. ACM Trans. Database Syst. 8, 1 (Mar.), 478-485.
136-165. RAMAMOHANARAO, K., AND SACKS-DAVIS, R. 1984.
LUM, V. Y., YUEN, P. S. T., AND DODD, M. Recursive linear hashing. ACM Trans. Database
1971. Key-to-address transform techniques: A Syst. 9, 3 (Sept.), 369-391.
fundamental performance study on large existing SCHOLL, M. 1981. New file organizations based on
formatted files. Commun. ACM 14, 4 (Apr.), dynamic hashing. ACM Trans. Database Syst. 6,
228-239. 1 (Mar.), 194-211.
MARTIN, G. N. 1979. Sprial storage: Incrementally SEVERANCE, D., AND DUHNE, R. 1976. A practition-
augmentable hash addressed storage. Tech. Rep. er’s guide to addressing algorithms. Commun.
27, Univ. of Warwick, Coventry, U.K. ACM 19,6 (June), 314-326.
MENDELSON, H. 1982. Analysis of extendible hash- VEKLEROV, E. 1985. Analysis of dynamic hashing
ing. IEEE Trans. Softw. Eng. SE-B, 6 (Nov.), with deferred splitting. ACM Trans. Database
611-619. Syst. 10, 1 (Mar.), 90-96.
MULLIN, J. K. 1981. Tightly controlled linear hash- VIPER, J. S. 1982. Implementations for coalesced
ing without separate overflow storage. BIT 21, hashing. Commun. ACM 25, 12 (Dec.),
390-400. 911-926.

Received November 1986; final revision accepted October 1987.

ACM Computing Surveys, Vol. 20, No. 2, June 1988

Вам также может понравиться