3/23/2013

MMT

Content Based Image Retrieval Report

Name: Daniel Nanton Student Number: 09211280

Daniel Nanton CBIR Report Page 0

Table of contents

Lab 1 - Case Study and Demos.1

Lab 2 - Database Selection and Conceptual Design..3

Lab 3 - Image Processing..4

Lab4 - Database Implementation for Query Image Processing.....7

Lab 5 - Generalisation and Rejection tests...10

Lab 6 - Generalisation and Rejection tests (continuing).10

Lab 7- Performance Analysis..12

Lab 8 - System integration.14

Introduction This report contains a detail description of the work completed throughout the nine lab sessions. The main focus of this report is based on the development of the CBIR database which included tables, queries and forms. This report provides an in-depth discussion on how each stage, using the following softwares; Microsoft Excel, Microsoft word 2010, Photoshop CS6, ImageJ, Microsoft Access 2010 and Microsoft Visual studio were used to complete the project. This report analyses all the problems encountered throughout the project in the evaluation section.

Lab 1 The first lab session was based on understanding the fundamental aspects of a content based image retrieval database. Each individual in the class was provided with an example of a CBIR database created by a previous student. Each individual tested the database and analysed the database to discover if this was a good database or not. Students were given questions and were asked to research into CBIR databases in order to understand how they work. A summary of the questions that were asked were: Demonstrate functional blocks of a CBIR system:

The technical difference in commercial Vs. research systems Commercial: Most commercial CBIR systems use colour histograms as one of the features. In other words commercial systems search, by colour percentages, colour layout, texture, shape/location and keywords. Examples of CBIR commercial systems are: Google Image search (Google), Bing image search (Microsoft), like.com (Product comparison & shopping using CBIR for product images. Previously known as Pixsta) Research Systems: Research Systems is a system that is used to retrieve data more specific. Research systems deal with retrieving data for example medical information. When you think of data such as medical, you get an indication that the system is more specific to the end-user. Examples of CBIR Research systems are: GNU image finding tool, CIRES (developed by the University of Texas at Austin) Each individual was then asked to run the students case study and analyse their database in detail. The type of questions that were asked were: To describe in detail which stages of the students case study was clear To identify the strengths and weaknesses of the students case study To identify what features were used and whether they would be used in the authors design To highlight what changes would be considered when the authors designs he database

Appendix A Full list of questions and answers

Daniel Nanton

CBIR Report

Page 1

Lab 2 - Database Selection and Conceptual Design As homework from lab1, each individual was asked to plan the theme of their database. By the time lab 2 had begun, the author decided to choose Wrestling as their theme. The author had to download images of wrestlers, belts and pants which will be suitable for their content based image retrieval database. Once these Images were downloaded the author started by using the block diagram to plan out their CBIR database. The Author had decided to use a wrestlers head and body size as a key feature extraction Here is a sample of the function blocks: This database would include an end-user inserting an image into my system. The system will then scan through the database for the image based on the feature extractions of the length of the stomach to neck,, the width of the Shoulder to Shoulder and the Length of the feet to head submitted in the database. The system will then find similarity comparisons before retrieving the final result of the actual wrestler. This database would include an end-user inserting an image into my system. The end-user user will insert an image into my system. My system will then scan through the database for the image based on the feature extractions of the colour of the belt, shape of the belt length of the belt, width of the belt and height of the belt submitted in the database. The system will then find similarity comparisons before retrieving the final result of the actual belt. For wrestlers pants, a user will insert an image into my system. My system will then scan through the database for the image based on the feature extractions of the colour of the pants, length of pants, width of pants and the logo submitted in the database. The system will then find similarity comparisons before retrieving the final result of the actual wrestler.

Each individual was then asked to familiarise themselves with Microsoft Access. They were asked to complete the tutorial provided in the lab session and they were asked to use Microsoft word to plan their database. A database needed to consist of Tables, queries, forms and records.

Daniel Nanton

CBIR Report

Page 2

The author had planned to use the following:

- Table o First Name: Text o Last Name: Text o Location Born: Text o Age: Number o Ethnic background: Text o List of colours: Text o Size of character: length, width and height o Size of belt: length, width and height o Size of pants: length, width and height o Type (Belt, pants and wrestlers body size) - Image queries o Belts o Pants o Wrestlers body size - Forms o Search by wrestler name o Search by belt o Search by Wrestlers body size o Wrestlers body pants - Records o Results of current champion o Result of character details (Name, DOB, age etc.) Later on throughout the project these queries and forms changed dramatically and the author decided to only use one function block being the wrestlers body and four feature extractions being the length/width of the wrestlers chest and the length/width of the wrestlers head. The reason why this was decided was because of the development of the database soon became extremely time consuming. The rest of the lab session was based on learning how to use the key tools and features in Microsoft Access. The author was then asked to begin implementing their database

Daniel Nanton

CBIR Report

Page 3

Lab 3 Image Processing For this lesson, each individual was asked to complete the following step: Use ImageJ for Pre-processing an image. Perform several feature extractions from the pre-processed images. Create plot graphs for the data cluster analysis. Export the excel files into ACCESS

The author started this weeks lab by going through the ImageJ tutorial. The author decided to become familiar with the software before pre-processing an image. Once the tutorial was completed the author began Preprocessing the images. The author began by cropping the images of the wrestlers; only the head and chest section of the wrestler was needed. This was important to crop all the images, because the author wanted to erase anything in the background which was not relevant. A folder was named cropped_images and the images were saved to the folder. An example of the original image and cropped image:

Original Image

Cropped Image

After the author had finished cropping the images, the author began to resize the cropped images. It was important to resize the images of all the wrestlers so they were all the same size. The author resized the height of all the images to 270 and resized the width of all the images to 450. A folder was named resized_images and the images were saved to the folder.

Cropped Image
Daniel Nanton CBIR Report

Resized Image
Page 4

After the author resized the images, the author decided to adjust the contrast of the image. The author set the contrast to 255. It was important to change the contrast and to make sure they were all the same. A folder was named contrast_adjusted and the images were saved to the folder.

After the author changed the contrast of the images, the author decided to adjust the illumination of the image. The author set the contrast to 255. It was important to change the illumination and to make sure they were all the same. A folder was named illumination _adjusted and the images were saved to the folder. The next stage was to open up the images from the illumination_adjusted folder and produce several feature extractions. The author started with measuring the width of the chest. This worked well as the measurements were all different. The author made sure I measured the size of the chest rather than the shoulders because some of the wrestler arms were positioned different from others which would affect my measurements. The next stage was to open up the images from the illumination_adjusted folder and produce several feature extractions. The author started with measuring the width of the chest. This worked well as the measurements were all different. The author made sure they measured the size of the chest rather than the shoulders because some of the wrestler arms were positioned different from others which would affect my measurements. The next step was to measure the length of the chest, this didnt work as well as the width because some of the wrestlers had the same length size. The author decided to use this measurement and combine with measurement with the width in order to create a unique feature. The next feature the author decided to measure was the head of the wrestlers. The author decided to measure the length and width of the wrestlers head. The author did this because this was an important and unique feature extraction. After the author had completed the feature extraction, the author saved them all into EXCEL. The author created a table for the image no and the four feature extraction (Length or Width of the head and the Length or Width of the head). From this the author created plot graphs to check if images of the same subject have feature points close to one another and are far away from other subjects.

Daniel Nanton

CBIR Report

Page 5

The author then finally decided to export the excel file into Microsoft ACCESS. The author did this by using the external data and selecting excel. The author then added the filename from my directory and added this into a table.

Daniel Nanton

CBIR Report

Page 6

Lab4 Database Implementation for Query Image Processing For this weeks assignment, the author was asked to understand the process of query image matching based on the features the author extracted in lab3. The author was then asked to design queries in Microsoft Acccess which takes input queries and produces different types of outputs based on the features. The author began this task by opening up Microsoft Excel and copying all the data (Features extracted) from the previous week into an input table. Input Table

As you can see from the table the database consist of all the names below of the wrestlers and across you can see the different types of features and their measurements. Random query image The next step was to input a random query image:

Output table This was implemented by using the unknown query image and subtracting that image by each of the following features (Length of chest (A), Width of chest (B), Length of head (C), Width of head (D)). This table also consisted of a calculation for the sum, total and total/max. Distance measured based on feature ((A) or (B) or (C) or (D)) Each of the four features had to be measured. This was implemented by copying the list of names and pasting them into the database column. The author then copied each of the features from the input feature table into their appropriate table (Length of chest (A), Width of chest (B), Length of head (C), Width of head (D)). The author then used the unknown query image to work out the distance of all the following features. This was done by using the following sample screenshot

Daniel Nanton

CBIR Report

Page 7

It was important to make sure that the query image had a formula; this was because when it came to creating the confusion matrix table because if the author changed the unknown query image value the whole table would change. Combine four Features The author then had to create a table for combining the four features. This was implemented by first pasting the list of wrestlers names into the database area. Using a formula the author worked out how to create the sum of the four combined features. Once this was completed the author worked out the sum similar to how this was done when creating the sum/Max for the output table. Confusion Matrix Once this was done I was then able to fill in the confusion matrix table.

Part 2: MS Access implementation

The second part of this task was to use MS ACCESS to create input queries. The screenshot below shows the types of queries I made:

Daniel Nanton

CBIR Report

Page 8

In this section I will show how each query was made:

Query distance measured based on features separately ((A),( B), (C) and (D)) o In design view adding the Main _Table and using the wrestlers names as well as their features in a field. The author also had to create and expression and a calculation. This was because the query was created to prompt an end user to type in a figure which was then used to find the relevant image in the database. Query distance measured based on (A,B,C and D) o In the design view, this was implemented by inserting a table to add the ID, First Name, Last name along with the four expressions and four calculations. Expressions were written like this: Expr1: [please type input 1] Calculations were written like this: Calculation1: (Abs([please type input 1]-[Length_chest])) Query combing sum of four features o In design view, this was implemented by inserting a table to add the ID, First Name, Last name along with the four calculations. Calculation was written like this: Calculation5: [Calculation1]+[Calculation2]+[Calculation3]+[Calculation4]) Query max combine four features o In the design view, this was implemented by inserting a table to add the four features Sum. Calculation was written like this: MaxOfCalculation_1234: Calculation5 Normalisation o In design view, this was implemented by inserting a table to add the four features Sum and Max four features. Calculation was written like this: calculation6: ([Query Combine four features Sum]![Calculation5]/[Query Max Combine four Features]![MaxOfCalculation_1234]) Here are some screenshots to display the queries:

Daniel Nanton

CBIR Report

Page 9

Lab 5 & 6 Generalisation and Rejection tests For this weeks assignment, the author was asked to repeat lab 2 and download five images of each of my characters (wrestlers). The author was then told to complete the steps taken in Lab-3, which meant using image J to complete the following: crop, resize, contrast and illumination. The reason for doing this again was because the author didnt have enough images to complete the confusion, generalisation and rejection matrix table for lab 5. Once these images were all cropped resized, contrast and illumination the author then began to use imageJ to extract the features of each of the characters. This meant doing the same as I completed in Lab-2 but for all the images. The features extracted were: Length & Width of chest and Length & Width of head. Once all the features were measured the author began to use Microsoft Excel to add the features into the cells. Screenshot below demonstrates one of the wrestlers features The first box (database) represents the inputs of the characters features. The box below (Query image) is used to complete the confusion matrix

The same process in lab 4 which involved calculation the distance measured based on feature ((A), (B), (C) and (D)) had to be implemented. The same process also with working out the sum/max had to be implemented. This was because the confusion matrix for all my wrestlers had to be created. Here is an example of one of the confusion matrix tables

The same process was completed for the following 14 wrestlers. The reason for completing this process was to produce a sufficient validation test. In order to complete a validation test I had to create a table listing all the wrestlers and their five images downwards and across the excel spread sheet. The total number of images used were 70 (14 wrestlers * 5 images). Validation Test The validation test is used to show the 14 confusion matrixs in one table. The importance of this table is to show that the zeros indicate that there is a correct match between the images. For example image Alberto_delRio_1 only matches Alberto_del-Rio_1 (the same image). Generalisation Test From this generalisation table it is important that none of the images are exactly zero however they are really close to zero. The authors generalisation test showed that the images are good enough to represent the characters. Rejection Test This was implemented by using the measurements of one character and trying those features into the image query of another character to determine if any of the images were close to zero. The authors rejection table is a good enough as it clearly shows that the image does not exist. There is no image which is zero and the images are close to 1 so this is a successful rejection table.

Daniel Nanton

CBIR Report

Page 10

PART B importing features into ACCESS

The second part of the task was to implement the work created in Microsoft Excel into Microsoft Access. This was done by first creating several tables: The first table had all the images attached and all the feature measurements of all images stored in the table named full table The second table named validation_data only had the feature measurements of all the images. I then created a table for each of the wrestlers which had the feature measurements of their related mages and their names

This section highlights all the queries that were created Query difference o In design view, adding the Full_table and Validation_data table. The author then had to make several fields and use Full_table to generate those fields. The author then had to create a calculation to work out the difference of each of the features. Query SUM o In design view, using the query difference table and using the calculation of cal16: [cal12]+[cal13]+[cal14]+[cal15] to provide me with the total Query(for each of my wrestlers images, this was a total of 70 queries) o In design view, using the fields full_table and full_name, I used the table sum2 to complete this query. In the criteria the author added the name of the JPEG of each of the images. Query MAX (for each of my wrestlers images, this was a total of 70 queries) o In design view, using the table of the specific query at the time and using the field to obtain the calculation of sum2.cal16. The author had to add the total column to the database also and change the parameter to Max. Query Normalisation (for each of my wrestlers images, this was a total of 70 queries) o In design view, the query and max tables were used. Using the build tool the author found this easier to write the code. The aim of this task was to create a calculation which was able to display a zero for each of the normalisation images. Validation Query o In design view the author began by adding all the norm tables and the validation table to the query. In the fields the author decided to use the Full_Name parameter. Since all the norm queries had the same syntax code nameofvalidation_name the author decided to link these altogether. Once the author had completed this task the author tried to run this task however the author was given an error message. The author researched (online) the reason behind this error message and the author was told the best option would be to split the queries up because there are too many fields in use. The author decided to split the validation query into small validation queries (max 15 images). The author used the SQL code and copied and pasted the appropriate code into each query. Generalisation Query o The author I had the same issue with the generalisation query so the author decided to split the generalisation query up into small queries. The reason for doing this was to show that there shouldnt be a correct match between two different types of images however if they are the same character/object they should have some sort of resemblance (close measurement). This was done by using the validation table and changing the criteria so that this meant it would show the opposite picture to what the author decided to compare it with. Rejection Query o The final task was the rejection query, this task was similar to the generalisation task however this just meant to change the criteria so whatever image(s) the author was using in the database the other image would had to be completely different to the one the author was testing in their database.

Appendix shows all the images

Daniel Nanton

CBIR Report

Page 11

Lab 7 Performance Analysis For this weeks assignment the author was asked to create the GUI interface based on the existing images in my CBIR system. The author was asked to create a search by example and search by type to allow my end users to find the most image(s) based on the wrestler chosen. The author was asked to provide a list of the resulting images to show a set of matches. The author had to use a threshold on the distance measured to produce a result of the highest relevant images based on the wrestler selected. Screenshot of homepage of GUI

List of queries which needed to be created AllTypes o In design view, using the main table (Full_Table) and inserting a new field named wrestlers and ensuring the data type was text. DiffandSum o Having to create a difference and sum query which was very similar to the query difference created in the previous week. The Full_Table was inserted into this query and the Full_Table field and table was used. Using the ([Forms]![Search_by_Example]! for each of the calculations which will later be explained when discussing forms. Get_Image o In order to initiate this query thee author had to create a new table named test table. This table was a copy of my Full_Table. The author created this query by using the Test_Table and used all the fields which were in my table for instance ID,Length_chest, Length_Width etc. In the criteria the Full_Name field I had to add the line of code [Forms]![Search_by_Example]![Text60] Max_Query o The max query was used to find the max from the DiffandSum query. This was done by creating a field named max16, inserting the table DiffandSum and inserting the column table Sum.

Norm_Query
This is regarded as the most important query in my database when using the search by example form. This form was created by using the two previous queries DiffandSum and max. The author had to add all the fields from the DiffandSum query. Search_by_type o The final query the author had to create was the search by type query. This was used to create my search by type form. The author used my Full_Table and added all the fields. In the criteria the author added the code similar to the previous codes. This code was based on the form button which would then use this query to populate the result. o

Daniel Nanton

CBIR Report

Page 12

Creating the Forms All records o The first form created was the All Records form. The author began by using buttons, text boxes, labels and attachments to create this. When creating the buttons the author had to use events in order to code the buttons so they would perform how I would like them to. The GUI displays all the wrestlers with their information on the side along with their features. End users are able to use the buttons to navigate through the images. The author created error events to inform the user when they have reached the final image for example. Search by type o This form was used to display a drop down box of all the wrestlers. Once an end-user clicked on the wrestlers name and selected search you would then be given the image of the wrestler and his other relevant images. This was done by creating a combo box and command button. It was important that in the properties section, the alltypes query was selected in order to show the wrestlers. The search button was created in order to generate the results form. The author had to create a results search by type form and the author had to use the following code in the search button to ensure the select button showed the results. It was important to change the record source from the results form to search by type. The author added the next and previous button so the end user is able to go through the relevant images of this character. Search by Example o This form was created by creating buttons, text boxes, labels, combo box etc. The author began by firstly using photo using photoshop to resize all the images so they could fit on a button. This was because the author was creating an interface which allowed the end user to click on the wrestler and scroll through the threshold to find the relevant match. Once the 70 images were re sized I had to then add the images to a button. This was done by adding a picture to the button. The author then had to use the event tab to code what the author want the button to do. The author had to then manually type in code to ensure the labels and textboxes worked correctly. The author had to do this part 70 times in order to produce my CBIR system. Create a threshold was done by creating a button and using the data tab tying into the raw source all the possible integers between 0 and 1. Extract features button to extract the features once the image was clicked on. Was done by creating an event and using the get image form to generate the information. Performance evaluation o This was done by fully testing the database and trying to discover if any of the images were not functioning correctly. Using Microsoft Excel testing was analysed by using each of my images to see whether my database was finding the most relevant images first or not. I tested each image and all the correct matches were found with no confusion of any other image. My system shows a clear indication of who the wrestler are based on the excellent feature extractions. A precision VS Recall line graph was created to show how well the database operates with the features extracted.

Daniel Nanton

CBIR Report

Page 13

Lab 8 System integration For the final lab session, each individual was asked to recreate their GUI in a website. The author decided to use ASP.NET to create the website. Before the webpages were created, the author had to add the database to the ASP.Net directory as well as the images, tables and queries which were created in MS access. Homepage o The website homepage was created very similar to the GUI MS Access. The Author created buttons, labels and images. When creating the buttons the author had to create an aspx.vb file in order to code the buttons so it would perform how the author would like them to.

All records o The website page created was the All Records. Very similar to the GUI, the author began by using buttons, text boxes, labels and attachments to create this. When creating the buttons the author had to create an aspx.vb file in order to code the buttons so it would perform how the author would like them to. The Website displays all the wrestlers and their images.

Search by type o The website page created was the browse by wrestler. Very similar to the GUI this was created with a dropdown box, which was also coded in aspx.vb. The website was coded to just display the wrestlers name and the image of the wrestler. A home page and refresh page was added at the bottom. To allow the end user to freely navigate around.

Daniel Nanton

CBIR Report

Page 14

Search By Example o The website page created was the search by example. Very similar to the GUI this was created with several image buttons, which was also coded in aspx.vb. The website was coded to just display the feature extraction, image of the feature extracted and the threshold. A home page was added at the bottom, to allow the end user to freely navigate around.

Conclusion Overall each lab was completed successfully and the database was programmed to extract features from an image which was then used to represent an image. There were several stages of the lab which were challenging and very demanding from the author, however each lab was very enjoyable and interesting. The author has learnt the following throughout these labs: - How to create a database - How to create a confusion, validation, generalisation, rejection, precision & recall - How to use MS Access to create a database - How to fully test a database - How to use Visual Studio to create a database - Presentation skills (when presenting work throughout each lab session)

Daniel Nanton

CBIR Report

Page 15