A Matching Algorithm for Content-Based Image Retrieval

Sue J. Cho Department of Computer Science Seoul National University Seoul, Korea Abstract Content-based image retrieval system retrieves an image from a database using visual information. Among approaches to expressing visual aspects in queries, "query by sketch" is most convenient and expressive. However, the query drawn by the user is typically quite different from the target image. In this paper, a matching algorithm for imperfect queries is presented. The algorithm measures the similarity between the query and each image stored in the database based on their topological structures that are represented by prime edge graphs. Experimental results show that the system retrieves the intended image with a high similarity score even from a partial or shifted query. Keywords: content-based image retrieval, image database, query by sketch, matching, similarity

Suk I. Yoo Department of Computer Science Seoul National University Seoul, Korea user draws a query by placing colored objects in appropriate positions, in which the position of each individual object may not be correct nor may all objects be reproduced. The query is as a form of rough map of colored objects in which the attributes of objects and their positional relationships play an important role. The system matches this rough drawing to each of the image data in the database to retrieve the target image. Since the query image drawn by the user tends to be quite different from the target image, the classical matching algorithm based on pixel values are not very effective in discriminating the target image from the rest of the database. In this paper, a matching algorithm is described, which is to retrieve an image from a partial or shifted query effectively. An image is segmented into a collection of objects. With these objects and their topological structure, the similarity between two images is measured. The rest of this paper gives the brief description of the data model of images, and shows the algorithms and some experimental results.

Introduction

Content-based image retrieval system retrieves an image from a database using visual information such as color, texture, or shape. In most systems, the user queries by presenting an example image that has the intended feature [4,5,6]. Although this approach has advantages in effective query processing, it is inferior in expressive power and the user cannot represent all intended features in his query. To fully reflect user's intention, the alternative approach that accepts a user-drawn sketch as a query has been developed [1,2,3]. In this approach, the type of matching algorithm system employs limits the features that user can represent in his query and vice versa. In the system that uses edge-matching algorithm, the user draws a boundary sketch as a query [1,3]. However, in the system that allows only boundary information in the query, the user cannot represent color information. In the system that uses wavelet-matching algorithm, the user draws a rough painting as a query [2], but even roughly reproducing an image is very difficult. In this paper, it is assumed that the

Image Conversion

An image is segmented into a collection of objects. Here, the meaning of an object is somewhat different from that of what we call an object in real life. An object is a set of pixels that are connected and have the same (quantized) color. Each object is represented as a set of attributes such as color, size, position, major axis, and minor axis. When an image is inserted in the database, it is converted to a collection of objects and these attributes are automatically extracted. Since it is assumed that a user draws a query in object basis, the query image can be recognized as a collection of objects without any segmentation process. This query image is converted to a graph

Query

Complete Digraph

14 10 6 7 9 5 13

11 = 1011(2)
10 : up 01 : down 11 : none 10 : left 01 : right 11 : none

Edge Labeling

11
Prime Edge Graph

Edge Pruning

Figure 1: Query Conversion called prime edge graph, which represents the topological structure of objects in the image. In a prime edge graph, each node represents an object and an edge between two nodes represents the positional relation of corresponding objects.

Figure 2: Edge Labeling Where, ~ denotes a bitwise-AND operator. If the result of bitwise-AND operation between two edge labels L(eij) and L(ejk) is equal to the L(eik), it mean that vi, vj, and vk are placed in one direction in this order. Even though eik is deleted, the positional relation of vi and vk can be still found with L(eij) and L(ejk). For example, if a node u is on the left of v and v on the left of w, it is trivial that u is on the left of w. All such transitivity can be eliminated. Step 4 (Prime edge graph:) Finally, one of two edges eij and eji between two nodes vi and vj is deleted as follows: For every eij, if i > j, then deletet(eij). Steps 3 and 4 are to minimize the number of comparison in later matching process. Figure 3 shows an example of query conversion. The query image in figure 3(a) consists of 4 objects. According to the positional relation of every two objects and figure 2, all edges are labeled (figure 3(b)). Figure 3(c) shows the resulted prime edge graph. Note that e13 is deleted since L(e12) ~ L(e23) = 5 ~ 13 = 5 = L(e13). 12 edges in figure 3(b) is reduced to 3 edges in figure 3(c) in steps 3 and 4, and the topological structure of objects is still remained.
v1 v1 v2 v3 v4 0 10 10 10 v1 5 0 14 10 v2 (b) 5 13 0 10 v3 5 5 5 0 v4 v2 v3 v4 (c)

2.1

Query Conversion Algorithm

The query image is converted to a prime edge graph through several steps a block diagram is shown in figure 1. Step 1 (Generation of a complete directed graph:) From a query image Q, generate a complete directed graph Gq(V,E). Each node vi in V represents an object in the query image Q. An edge eij is inserted between every two nodes vi and vj, where i j. Step 2 (Edge labeling:) Label each edge eij in E a number L(eij) according to the positional relation of two nodes vi and vj. The number assigned to each of eight directions is shown in figure 2. The number for each direction is 4-bit code based on x, y positions of start and end points. The first and second bits of 4-bit code represent the relation of x values of start and end points. If the x value of start point is smaller than that of end point, first and second bits of 4-bit code are 1 and 0. The last two bits represent the relation of y values of start and end points. For example, if vi is to the left of vj, L(eij) = 0111 and L(eji) = 1011, and their decimal representations are 7 and 11 respectively. Step 3 (Edge pruning:) Repeat the following process until no change occurs: if L(eij) ~ L(ejk) = L(eik) then delete(eik)

(a)

Figure 3: Prime edge graph

Image Database

score

Get an image Object Matching accept Prime Edge Matching accept reject

retrieved images

Prime Edge Graph

reject

Figure 4: Image retrieval node in Gp remains. The similarity score St of It is computed by summing and normalizing the match scores of the finally selected candidates. S t = 100 * M ( f k ) / n ,

Image Retrieval

The process of retrieving images with a prime edge graph is shown in figure 4. For each collection of objects that represents an image in the database, similarity score is measured through twostep matching object matching and prime edge matching. Finally, the accepted images are displayed in the order of similarity score.

where fk is the finally selected candidate and n is the number of nodes in Gp.

Experimental results

3.1

Object Matching

A database image It is a collection of objects {c1, c2, , cn} and each object cj is a list of attributes. Let Ak(x) be the function that returns the k-th attribute value. For example, A0(cj) is the color of cj, and A1(cj) is the normalized size of cj. For each node vi in Gp, the candidate objects in an image It are found according to the match score. The match score M(cj) of a candidate object cj is obtained as follows: M (c j ) = wk * Ak (vi ) | Ak (c j ) Ak (vi ) | / Ak (vi ),

The presented matching algorithm is tested on an image database with 400 images. First, 3 queries drawn by different people are tested to retrieve the image in figure 5. Second, to evaluate the algorithm, the similarity scores produced by another algorithms are compared with the presented algorithm. Finally, some interesting tests that are to retrieve images that have certain component are performed.

4.1

Robustness to Imperfect Queries

where wk is the weight of k-th attribute. If M(cj) exceeds some threshold T, cj is marked as a candidate for vi. . Since attributes are compared directly, they must have normalized values. For effective segmentation and matching, colors are quantized and coded so that neighbor colors would produce the high match score. If there exists a node in Gp that has no candidate at all, It is rejected.

The query image Q1 in figure 6(a) contains 2 objects, Q2 in figure 6(b) contains 4 objects, and Q3 in figure 6(c) contains 2 objects whose positions are quite different from the target image in figure 5. The retrieved images for each query and their similarity scores are shown. Even from a partial or shifted query, the system retrieved the intended image with a high similarity score.

4.2 3.2 Prime Edge Matching

The prime edge matching step checks the consistencies between each prime edge eij and the positions of candidate objects of vi and vj. If all candidates of vi and vj are not consistent with L(eij), It is rejected. Finally, one candidate object for each

Retrieval Effectiveness

The presented algorithm has two distinctive characteristics. First, the similarity scores are calculated in object basis, so the user can retrieve an image with a partial query. Second, the topological structure of the objects are compared instead

{{{ {{{ {{{ {{{ {{{ {{{ {{{ {{{ {{{ {{{ {{{ {{{ {{{ {{{ {{{ {{{ {{{

Figure 5: Target image and the objects set

(a) Retrieval results for Q1 (S1 = 92.8, S2 = 90.2, S3 = 76.7)

(b) Retrieval results for Q2 (S1 = 96.4)

(c) Retrieval results for Q3 (S1 = 95.5, S2 = 87.6 , S3 = 74.1) Figure 6: Example queries and retrieved images

of their positions, so even when the query image is shifted or the distance between objects are not correct, target image can be retrieved. To bring out these features clearly, three algorithms are developed. In the first algorithm, the similarity score is calculated as follows: S t1 = 100 * M ( f k ) / m ,

algorithm, respectively.

4.3

Retrieving Unknown Images

where m is the number of objects in It instead of the number of nodes in Gp. Although this algorithm is also based on objects, the similarity score cannot be high if only the small set of objects is represented in the query. In the second algorithm, queries are not converted to prime edge graphs and prime edge matching step is omitted. In object matching step, the position attributes are compared and the final candidate is the one that has the highest match score. The similarity score is calculated as follows: S t2 = 100 * M ( f ' k ) / n ,

Sometimes it is very useful to retrieve images that contain some figures. The user may have not seen the image nor he may know if there is such an image in the image database. For example, a person who is to make a Christmas card may want to retrieve an image of Santa Claus. In this case, he doesn't care where Santa Claus is, nor he knows if there is a Santa Claus image. For this type of query, out matching algorithm shows a good performance with the weight of size attribute minimized. Figure 8 shows some experiments. In figure 8(a), to retrieve a Santa image, a red hat and a white beard are drawn in the query. There are several Santa images in the database, but only the best-matched image is shown in this example.

where f'k is the final candidate for k-th object in the query image Q and n is the number of objects in Q. In the third algorithm, the final candidate is selected as the second algorithm and the similarity score is calculated as follows: S t3 = 100 * M ( f ' k ) / m

In figure 7, S, S1, S2, and S3 are similarity scores of the image in figure 5 with queries in figure 6, calculated by the presented algorithm, the first algorithm, the second algorithm, and the third Similarity Score 100 80 60 40 20 0 S1 S2 S3 Figure 7: Comparison with another algorithms S Q1 Q2 Q3

(a) Santa query: there is a Santa Claus image

(b) Strawberry query: there is no strawberry image Figure 8: Retrieving unknown images

Discussion and Future Work

The content-based retrieval system using the algorithm described in this paper has two important advantages. First, the system can retrieve the intended image with a high similarity score even from a partial or shifted query. Second, the query can be drawn more easily. The proposed method also has some limitations. First, the matching time increases as the database grows. Currently, to speed up the matching, two times of rejection occurs during the matching process. In object matching, if there exits a node in Gp that has no candidate at all, matching of It is terminated immediately. In prime edge matching, if a prime edge with which no candidate object is consistent is found, the image is rejected. However, for very large databases, more powerful speed-up scheme must be devised along with effective database management scheme. Second, for some kind of images such as textured images, the presented method is not so effective. To retrieve such images effectively, supplementary query scheme is needed. It is not difficult to combine a routine that processes the query with global features such as color or texture.

733, July 1996. [6] C. Schmid and R. Morh. Image Retrieval Using Local Characterization. In Proceedings of ICIP-96, pages 781-783, IEEE, 1996.

References
[1] A. D. Bimbo and P. Pala. Visual Image Retrieval by Elastic Matching of User Sketches. IEEE Trans. Pattern Analysis and Machine Intelligence, 19(2), Feb 1997. [2] C. E. Jacobs, A. Finkelstein, and D. H. Salesin. Fast Multiresolution Image Querying. In Proceedings of SIGGRAPH '95, pages 277-286, ACM, New York, 1995. [3] T. Kato, T. Kurita, N. Otsu, and K. Hirata. A Sketch Retrieval Method for Full Color Image Database. In Proceedings of 11th IAPR, pages 530-533, IEEE, 1992. [4] P. M. Kelly, M. Cannon, and D. R. Hush. Query by image example: the CANDID approach. SPIE Vol. 2420 Storage and Retrieval for Image and Video Databases III, pages 238248, 1995. [5] F. Liu and R. W. Picard. Periodicity, directionality, and randomness: Wold features or image modeling and retrieval. IEEE Trans. Pattern Analysis and Machine Intelligence, 18(7):722-