LOCAL COLOR AND TEXTURE EXTRACTION AND SPATIAL QUERY John R.

Smith and Shih-Fu Chang

Center for Image Technology for New Media and Department of Electrical Engineering Columbia University New York, N.Y. 10027
{jrsmit h ,sfchang} @ee.columbia.edu

ABSTRACT

In this paper we present a unified system for the extraction, representation and query of spatially localized color and texture regions. The system utilizes a back-projection of binary feature sets to identify and extract prominent regions. The binary feature sets provide an effective and easily indexable representation of color and texture. We also provide a mechanismfor integrating features by combining the binary color and texture feature sets. This enables the extraction and representation of joint color and texture regions. Since a l extracted regions are spatially locall ized, in image database queries the user can specify the locations and spatial boundaries of regions. We present the unified color and texture bat&-projection method and describe its implementation in the VisualSEEk content-based image retrieval system.
1. INTRODUCTION
We propose and evaluate and a unified system for color and texture indexing Ithat provides for automated extraction of local regions, efficient indexing and excellent query performance. The system uses the back-projection of binary color and texture sets to identify salient color and texture regions. By way of iteration over the binary feature sets, localized and arbitrarily shaped regions of color and texture are extracted. These are indexed directly by binary set values, which enables very fast indexing of the image collection. Furthermore, information extracted from the regions such as the size, shape and position, enables a rich variety of queri'es that specify not only color and texture but also the spatial relationships and composition of regions.
1.1. Region and Feature Extraction

Color Region Extraction

Color Extraction

color sets

image

I---colormexture Extraction

fti:ie
sets sets

Extraction

Extraction

Figure 1: Unified system foir color and texture region and feature extraction.

and texture indexing components of the system use the binary feature set back-projection algorithm to extract image regions.

2. FEATURE REPRESENTATION
We now present a unified framework for the extraction and representation of color and texture regions. In both modalities, the image points are initially "resented as multi-dimensional feature points: a color point is defined by values in three color channels and a texture point is defined by values in nine spatialfrequency channels, refer to Table 1. We next transform and quantize the feature points. This gives a manageable number of colors (166) and texture elements (512). Using these feature elements, we utilize two means for representing color and texture - through histograms and binary sets. Histograms, which give a distribution of feature elements, are appropriate for representing the global feature of images. Binary feature sets, which give a selection of feature elemenits, give a compact representation of regional features in the images.

The feature extraction system consists of two stages: (1) region extraction and (2) feature extraction, as illustrated in Figure 1. The goal of region extraction is to automatically identify the prominent regions in the images. We propose a new region extraction process that employs feature set back-projection. The basic idea is that the features within an image are found by back-projecting, candidate feature sets onto the image. In the application of content-based query for image databases, prominent color and f,exture regions are extracted and indexed for fast querying.
1.2. VisualSICEk

I
I

L general I
C
I

color
Vr

featureooint I IL featureelement I binary set

I I

I
I
I

texture 6t
n

I
1
I

channels dimensions

166

512

Table 1: Unified framework for color and texture extraction. The back-projection of the binary feature sets also provides for an efficient automated algorithm for region extraction. By iterating over candidate color and texture binary feature sets for an image, color and texture regions are extracted. This process is described in more detail in Zlection 6. Finally, the quadratic form of the feature set distance function, which is used for computing similarity between color andl texture regions, can be decomposed

We are currently developing the VisualSEEk content-based image retrieval system. VisualSEEk provides tools for querying by color, texture and spatial layout. Other features such as shape, motion and embedded text will be incorporated in the near future. The system emphasizes several research goals - automated extraction of localized regions and features [5][6], querying by both feature and spatial information, feature extraction from compressed images [l]and fast indexing and retrieval. The color

0-7803-3258-X/!36/$5.00 0 1996 IEEE

1011

into terms that are easily indexed, which is describedin Section 5. This allows for fast querying of color and texture regions using their binary feature set values. We next describe in more detail the color and texture set representations.
3. COLORSETS
T-',; Q
3 D Calor Histogram
Binary Color Set

Each image point can be represented by a color point that is defined as follows: Definition 1 Color point, 6 . A color poant as a v e c t o r 6 = , , ( r , g , b ) in RGB color space. of the RGB color The transformation T, and quantization space reorganizes and groups the vectors 6,. Perceptually similar colors are obtained by the sets of vectors that are mapped to same indices m. Definition 2 Color, m. A color is g i v e n b y t h e s e t of color p o i n t s {6c}m t h a t are a s s i g n e d t h e s a m e v a h e m b y m = QpTc6,. In VisualSEEk, T, transforms the RGB colors into their most natural components: hue, saturation and intensity. The quantization Q F provides A = 166 distinct colors in H S V color 4 space [6]. In the H S V color space the similarity between any two colors given by indices mq = ( h q , s q r w qand mt = ( h t , s t , w t ) is ) given by 1 aq,t = 1 - -[(wp - v t ) 2 (sqcoshq- St cosht)2 (1)

Figure 3: Relationship between 3-D RGB color histogram and binary color set.

QF

4. TEXTURE SETS

The texture representation is derived to be symmetric with the color. Texture is represented as a multi-channel spatial/spatialfrequency (s/s-.f) process as follows: by iterating the filter bank twice on the low s/s-f subband, and by discarding the lowest s/s-f subband, a nine channel decomposition is produced, as illustrated in Figure 4.

(sqsinhq - stsinht)2]+.

Figure 2 : Transformation, quantization and binarization of color space to produce the binary color set representation. Next, color sets are obtained by selectingfrom the M colors; the overall process is depicted in Figure 2 . Let B M be the M dimensional binary space such that each axis in BFcorresponds to one color m. Definition 3 Color set, e. A color s e t as a banary v e c t o r in B p a n d c o r r e s p o n d s t o a s e t of c o / o r s { m } . Finally, A distributionof colors produces a color histogram. The histogram is given by, where X and Y are the width and height of the image, respectively,
x-1 Y-1

Figure 4: Representation of texture as a multi-channel process using a wavelet filter bank. The output of channel k in the filter bank is
yk,

(0.. S } . The energy magnitude (& energy) of all points in yk are computed to obtain a measure of s/s-f energy. Each subband

where IC =

is then upsampledback to full size and missing points are filled in with pixel replication using block filters BZ,> obtain Sk. This to produces a channel energy value from each of the nine channels for each image point [5]. This representation is now symmetric with the representation of color. Whereas each image point has values in the three color channels, each image point has values in the nine s/s-f channels.
nine d-fchannels

1 0
3

if QFTc(6c)5, m = otherwise

(2)

o y

Definition 4 Color histogram, h,. A color h z s t o g r a m as a dastrabutzon of colors. Histograms provide a finer resolution of feature content than binary sets. A binary set is equivalent to thresholded histogram. For example, given threshold T~ for color m , a color set is obtained from (see also Figure 3) c[m] =

image I [x, y1

wavelet image { 7,)

if h[m] 2 T~

otherwise.

(3) Figure 5: Representation of texture using wavelet subbands. Since the wavelet expansion of texture gives K = 9 s/s-f channels, a texture point is represented by a 9-D vector. A texture point is the vector 6t = (so, I , .. . , S~c-1)in R",where Sk S

The color set indicates only those colors that are found above threshold levels. However, it works well to represent regional color. If a color is not well represented in a region - for example, if it is below threshold T~ - it is ignored. In this way color content is represented by the set of only the most prominent colors in the region.

1012

gives the energy in s/s-f channel k. Texture elements are generated through transformation Tt and quantization Qf" of the texture points. Similar to Definitions 3 and 4 for color, a texture set is defined as a selection of texture elements and a texture histogram is defined as a distribution of texture elements. The process of generating texture sets is illustrated in Figure 6. The transformation Tt is chosen to compact the feature space by transforming the s/s-f channel outputs. For example, Principal Component Analysis can be used to decorrelate the feature space using a set of training textures. The quantization Qf" can then lbe chosen to provide more h e resolution on the axes that have largest significance. Alternatively, Tt can be derived from Fisher Discriminant Analysis to provide maximum separation among texture classes [4].

Finally, since fq E B L (fq iis a binary vector), (7)

VL, where fq[l]=l

That is, any query for the most similar binary feature set to fq may be easily processed by storing and indexing individually pt and Rt[l],where 1 E 0.. L - 1. Notice also that pq is a constant . of the query. The closest color set f t is given as the one that minimizes pt - 2 vi, where fq[+l Rt[I].We now describe the process for color and texture region extraction using the binary feature sets.
6. FEATURE SIET BACK-PROJECTION

RXY

j-B-i:;w.FB
ch K-1

ch K-1

Figure 6: Formulation of texture as a multi-channel process and generation of the binary texture set representation.

The back-projection extraction system for regions and their features has four stages, as illustrated in Figure 7: (1)transformation, quantization and filtering of the image, (2) back-projection of binary feature sets onto the image, (3) region labeling and thresholding and (4) region feature extraction. The goal of the system is to reduce the insignificant features in the image, and to emphasize and extract ithe prominent features. Through an iteration process over the biinary color and texture sets, the backprojection process (stages 2 - 4) extracts color and texture regions.

5. FEATURE SIMILARITY COMPUTATION AND INDEXING

The similarity of colors or textures is computed from the distance between the feature histograms or binary feature sets. In order to considler the cross-distance between feature elements in f computation of distance a quadratic distance function is used.
5.1. Histogram Quadratic Distance

IImageY1 [x,

& E + q - b
RIx,yl
'-Stage

Stage 2

:
i : :

Stage 3

Stage4Y

The histogramL quadratic distance was proposed by QBIC for color indexing [3]. It measures the weighted similarity between histograms, which provides more desirable results than "like-bin" only comparisons. The quadratic distance between histogramsGq and is given by: (4) where A = [a,,3]and a,,]denotes the similarity between features with indexes i and j . For example, by defining color similarity in H S V color space, a,,]is given by Eq. 2. Since the histogram quadratic distance computes the cross similarity between f e e tures, it is computationally expensive. Therefore, in large applications other strategies, such as pre-filtering [2], are required to avoid exhaustive search.
5.2. Binary Feature Set Distance

Figure 7: The region and feature extraction system, T = transformation, Q = quantization, B P T o = back-projection, NL = ~ non-linear filter, SL = sequential labeling, T = thresholding and FE = feature extraction. In the first stage T and Q reduce and reorganize the feature space, as described in Sections 3 and 4. The next stage extracts regions from the images. This is done by systematically iterating over the binary feature sets. The back-projection of a binary feature set onto the image produces a bi-level image B[s,y]. The levels correspond to the selected and un-selected pixels for the specified feature set. The binary feature set backprojection algorithm is given as follows: given binary feature set f and processed image R[z,y], 1. Generate the bi-level image B[s, y] from back-projection ofbinaryfeatureset f: let k = R[z,y], thenB[z,y] = f[k]. For example, B[s,y] = 1 U the element at R[z,y] is in the binary feature set f[k] = 1. 2. Filter B[z,y] with a majority filter to eliminate spots and connect neighboring regions. Next, a unique label is assigned to each isolated region. Each region is then analyzed in regard to several thresholds. The motivation for these thresholds is to require that regions meet minimum size constraints and to sufficiently represent the elements in the back-projected feature set. The features are extracted from the surviving regions: the back-projected feature set f is recorded and the region area and location is added to a table of image region meta-data.

The histogram quadratic distance equation (Eq. 4) also measures the distance between binary feature sets. The distance between two binary feature sets fq and f t is given by

(5)
Given the binary nature of the binary feature set representation the computational complexity of the quadratic distance function can be reduced. We decompose the binary feature set quadratic formula to provide for efficient cpmputation and indexing. By defining pq = f i A f q p t = ftAft and Rt = Aft, and since A , is symmetric, the quadratic distance is given as
pet
q,t

7. INTEGRATION OF VISUAL FEATURES

The color and texture sets can also be combined to create an even more powerful extraction and representation of image regions by color and texture jointly.

= Pq

+ Pt - 2 f p . t .

(6)

1013

7.1. J o i n t Representation of Color a n d T e x t u r e

A joint representation for color and texture is provided by concatenating the binary feature sets for color and texture as follows:
it = [ E

I i] =

[ E

I o N ] + [ o M I i]

Since the color sets E have A dimensions and texture sets t* have 4 N dimensions, the joint sets i t have N M dimensions. The joint sets indicate a selection from both the colors and texture elements.

7.2. Extraction of J o i n t Color a n d T e x t u r e Regions

To extract joint color and texture regions we analyze the individual color and texture regions. The process involves iteration over the extracted color and texture regions, which are then mapped back onto the image. This produces an intermediate image P [ z ,y], which is processed and analyzed so that the regions with joint color and texture features are extracted. The procedure is illustrated in Figure 8. The extraction algorithm is given as follows: given a set of the color regions {E,} and a set of the texture regions {ik} that have been extracted, Let z iterate over the color sets for the extracted color regions and likewise, 3 and the texture regions. Choose all color regions, that have value E, and texture regions that have value t, and map the regions back onto the image to obtain the intermediate map image P [ z ,y]. Extract joint regionsusing Stages 3 and 4 of the extraction process described in Section 6. Represent the extracted joint coJor and texture region with the feature set ct,, = [ E, I t, 1.
Color Map

Figure 9: VisualSEEk user interface

9. S U M M A R Y A N D F U T U R E WORK

This paper presented a unified system for the extraction, representation and query of spatially localized color and texture regions. A general framework was presented to represent color and texture using binary feature sets. The binary feature sets provide for automated region extraction through feature set backprojection. The binary feature sets also provide for efficient computation of feature similarity and indexing. The unified approach towards color and texture provides for the retrieval of images in the VisualSEEk system by color and texture regions and by the spatial locations, spatial relationships and sizes of regions. 10. R E F E R E N C E S S.-F. Chang. Compressed-domain techniques for image/video indexing and manipulation. In Proceedings, I.E.E.E. International Conference on Image Processing, Washington, DC, Oct. 1995. invited paper to the special session on Digital Library and Video on Demand.

1Texture Map

htcrmcdiate Map

Unified FcaNn Map

Joint Color and Texture Regions

Figure 8: Integration of color and texture. Joint color and texture regions are extracted and are represented by feature set i t .

J. Hafner, H. S. Sawhney, W. Equitz, M. Flickner, and W. Niblack. Efficient color histogram indexing for quadratic form distance functions. IEEE Trans. Pattern Anal. Machine Intell., July 1995.
W. Niblack, R. Barber, W. Equitz, M. Flickner, E. Glasman, D. Petkovic, P. Yanker, and C. Faloutsos. The QBIC project: Querying images by content using color, texture, and shape. In Storage and Retraeval for Image and Video Databases, volume SPIE Vol. 1908, February 1993. J. R. Smith and S.-F. Chang. Transform features for texture classification and discrimination in large image databases. In Proc. Int. Conf. Image Processing, Austin, TX, November 1994. IEEE. J. R. Smith and S.-F. Chang. Automated binary texture feature sets for image retrieval. In Proc. Int. Conj, Acoust., Speech, Signal Processing, Atlanta, GA, May 1996. IEEE.

8 . QUERYING

BY C O L O R , T E X T U R E A N D SPATIAL I N F O R M A T I O N

A significant aspect of discriminating among images depends upon the spatial locations and relationships between objects or regions within the images. The evaluation of spatial locations in a query is accomplished by using a quad-tree spatial index. The evaluation of spatial relationships is accomplished by using a spatial query structure such as the 2-D string. The joint color/texture/spatial queries are formulated graphically by using the VisualSEEk user tools as illustrated in Figure 9. The user sketches regions, positions them on the query grid and assigns them properties of color, texture, size and absolute location. The user may also assign boundaries for location and size. The relationships between regions are diagrammed by connecting regions. In this way, the interface provides for queries that include region features and combinations of both absolute and relative placement of the regions.

J. R. Smith and S.-F. Chang. Tools and techniques for color image retrieval. In S y m p o s z s m o n Electronzc I m a g z n g : Sczence and Technology - Storage 4 Retrreval for Image and Video Databases I V , volume 2670, San Jose, CA, February 1996. IS&T/SPIE.

1014