Chapter 1

THE PROBLEM AND ITS BACKGROUND

Introduction
Mobile phones have become a ubiquitous part of our daily lives; they enable us to
talk to our far away loved ones or share a small conversation with them via texting.
Earlier than predicted, smartphones are now expected to rise to account for the majority
of global cellphone shipments in 2013. The increased demand from developed regions for
high-end models and an unexpected strong push from emerging economies for lower-cost
products highly contribute to this trend. (www.isuppli.com)
The holy grail of computer vision research, studied for more than four decades,
object recognition is to tell a story from a single image or a sequence of images (Ullman,
1996; Marr, 1982). Different representation schemes and algorithms aiming at
recognizing generic objects taken under different imaging conditions have taken
substantial efforts. There has been significant progress towards object categorization
from images in the recent years (Ponce et. al., 2006). Object recognition is also related to
content-based image retrieval and multimedia indexing as a number of generic objects
can be recognized.
Gartner, the world's leading information technology research and advisory
company believes that by 2012 Object Recognition (OR) will be a hot trend and apps will
enter the mobile phone market because it has an easy-to-use interface. It assesses that
we'll soon also rely on the camera, as well as other device sensors as a communication
tool and when Object Recognition (OR) features are combined with more-traditional app

2
functions they will give users advanced search options as well as a range of entertainment
and productivity functions. (news.techeye.net).
Augmented Reality (AR) is a term that applies to the enhancing the user
perception of the real world with virtual elements. Unlike Virtual Reality (VR), which
involves simulating a virtual environment with a physical presence, AR does not require
special equipment and can be implemented by means of a simple capturing device e.g. a
digital camera and a means of coating computer designed objects over the cameras
pictures.
Grocery shopping is considered an important activity, as almost everyone shops
for groceries on a regular and frequent basis. The first self-service grocery store was
introduced by Clarence Saunders in 1916 when he opened the Piggly Wiggly in
Memphis, Tennessee (geography.about.com) since then grocery stores became bigger
adding more and more products in its shelves. Because of the importance of grocery
shopping, combined with large business potential it creates an interesting domain for
developing and deploying novel pervasive and mobile computing technologies.
Although grocery shopping may look simple and easy it is actually a relatively
complex and stressful activity. These complexity and stressfulness may arrive from
different sources such as crowding, advertisements, the behavior of other customers, time
and budget constraints (R. Aylott V and W. Mitchell, 1998). Shoppers are also having
trouble finding information on which products are on sale, discount, or other recent
offerings. Other difficulties include finding latest products, conducting price comparisons
and brands, and getting the latest information on activities at the grocery shop. Today

3
most of the grocery shops already provide a terminal where the shopper can get
information about different products. But that the terminal is located only at one place.
And shoppers should look for and go to the location of the terminal. In addition,
information on the existing terminal is still general, and cannot be adapted to individual
needs - individual shoppers.
The researcher intends to solve this problem by making shopping easier for the
shoppers by improving the way the shopper interacts with his surroundings. The real
world problem that the researcher aims to handle is to collect the information from the
surrounding area without or with little effort by the user, in this case the shopper in a
grocery shop.
In this research the researcher will develop and evaluate a Shopping Assistant
using Object Recognition and Augmented Reality for Mobile Phones. A mobile
application will be installed in the users Mobile Phone. Using the mobile phones camera
the application will take image of a selected product. Object Recognition algorithms will
be implemented to recognize the product. Using Augmented Reality information about
the product will then be retrieved and displayed in the mobile phones screen.

Objectives of the Study

The study aimed to develop and evaluate a Shopping Assistant using Object
Recognition and Augmented Reality for Mobile Phones.
Specifically, this study shall seek to attain the following objectives.

4
1. To develop a Shopping Assistant using Object Recognition and
Augmented Reality for Mobile Phones that will recognize different
products from a grocery shop and display relevant information about it.
2. To evaluate the application in terms of:
2.1 Experts evaluation
2.1.1

User Interface;

2.1.2

Performance;

2.1.3

Security;

2.2 Users evaluation

2.2.1

Ease of Use;

2.2.2

Reliability;

Significance of the Study

The findings of the study may provide pertinent information that would be beneficial to
the TSU Community, Future Researchers, People Equipped with Android Mobile Phones and
the Researcher.

To the TSU Community this study will help the community in promoting and
developing a research based school that uses the modern technology.
To the Future Researchers this study could be useful reference for future
researchers engaged in similar study in Object Recognition and Augmented Reality.
To the People Equipped with Android Mobile Phones this study will enhance
their shopping experience by improving the way they interact with their surroundings.

5
To the researcher itself this study will give a great help to the researcher by
enriching his experience and knowledge in creating a Shopping Assistant using Object
Recognition and Augmented Reality for Mobile Phones which could enhance his
performance as an IT Programmer.

Scope and Delimitation

The proposed system in this research will only identify products that can be seen
in a grocery shop.
The users of the proposed system will be the customers in a grocery shop
equipped with Android Mobile Phones with a camera and can connect to the internet.
The proposed system will rely on internet connection for sending and receiving
information.
A mobile application will be created and installed to the users Android Mobile
Phone. The application will provide an interface for capturing objects. It will then extract
its visual features and send it to a server. It will also receive the information forwarded by
the server and display it to the user.
The proposed system will need a server which will act as the retrieval engine. The
mobile phone will only capture the image of an object and extract its visual features.
Object matching will be done on a server computer.

Definition of Terms
The following are technical terms used in the study to aid the reader to better
understand the research.

6
Cameras. Most of the newer mobile phones come along with built-in photo
cameras. Started as a gadget for fun they now become better and reach a quality that is
really usable for taking good pictures.
Client. It is the mobile phone that is used to capture photos of objects and send
visual features of it to the server via WLAN.
Computer Vision. A branch of artificial intelligence and image processing
concerned with computer processing of images from the real world. Computer vision
typically requires a combination of low level image processing to enhance the image
quality (e.g. remove noise, increase contrast) and higher level pattern recognition and
image understanding to recognize features present in the image.
Ease of use. The user friendliness of the developed system and the capability of
the system to be used smoothly with minimum required effort from the user.
Object Recognition (OR). The process of determining the identity of an object
being observed in the image from a set of known labels. Oftentimes, it is assumed that the
object being observed has been detected or there is a single object in the image.
Mobile Application. An application suited and installed to a mobile phone.
Mobile Phone. A telephone which is connected to the telephone system by radio
instead of by a wire, and can therefore be used anywhere where its signals can be
received.
Performance. The rate at on how fast system is able to detect recognizes and gets
information of an object using the vision based Object Recognition (OR) using mobile
phones.

7
Reliability. The rate at which the developed system can detect, recognize and
provide accurate data about an object.
Visual Features. Features that includes the shape, configuration, pattern and
ornamentation of an object.
Security. The rate at which the system can prevent unauthorized changes to the
contents and information encoded in the developed system.
Server. A computer that receives information from users mobile phones and
provides information on the objects recognized using the visual features sent by the
mobile phone.
User interface. A program that controls a display for the user (usually on a
computer monitor) and that allows the user to interact with the system
WLAN (Wireless Local Area Network). A communication system that
transmits and receives data using modulated electromagnetic waves, implemented as
an extension to, or as an alternative for, a wired LAN.

8
Chapter 2
REVIEW OF RELATED LITERATURE AND STUDIES
This chapter presents a brief review of research works, books periodicals and
articles that bear significant relation to the study in view of the researchers desire to get
ample background and establish a solid foundation on the topic which is under
investigation.

Related Literature
Insert literature about mobile phones
A personal shopping assistant is someone who is responsible for identifying a
clients needs and finding products that meet those requirements (www.wisegeek.com).
M. Jalali-Sohi and F. Basakaya (2001) said that a personal assistant on a mobile
device can ease the burden of performing everyday routines, and can hide the complexity
of difficult tasks, by helping the user to perform them. Using mobile phones as a
shopping assistant can greatly ease the shopping experience of a customer.
In the past, shoppers were classified by broad demographics, but soon they will be
able to engage in an augmented reality shopping experience that is customized, and lets
them take advantage of deals and special offers related to products that they likely need
(www.research.ibm.com)
Augmented Reality is changing the world of advertising. AR is much more
stimulating and enticing than a standard video. It allows companies to take their standard
ads and turn them into a highly interactive experience for the customer
(http://www.i95dev.com/).

9
Acording to Azuma (1997) augmented Reality (AR) involves the seamless
overlay of virtual imagery on the real world. In the last five years, a growing number of
researchers have considered how users will interact with AR applications and how to
effectively present information on AR displays. With advances in tracking and increased
computing power, researchers are developing mobile AR systems. These may enable a
host of new applications in navigation, situational awareness, and geolocated information
retrieval.

Related Studies

A. Foreign
Investigations on shopping practices can be a great help when designing grocery
aids. In store interviews and questionnaires about customers shopping habits was
conducted by E. Newcomb et al (2003). The authors found out shopping lists play a
central role in grocery shopping and that customers prefer grocery aids they can use on
their personal devices. Features that facilitate shopping list creation and management
were considered to be the most important. Customers also valued systems that help them
locate products with aisle-level accuracy, provide itemized product or health information
were considered least important.
Using participatory observation with pre and post interviews H. Tamura et al
(2007) analyze customers shopping behavior in Japan. Their study revealed three phases
in grocery shopping; selecting items for the main dish, the side dish and replenishing

10
items that are lacking from home. However, these findings are of limited use in our
setting, as the Japanese shopping and eating habits differ significantly from our own.
G. Roussos and T. Moussouri (2004), in their study, found out that carefully
designed systems that gather and analyze customers shopping data is needed in order not
to raise concerns about the loss of privacy. It is also revealed in the study that
automatically generated shopping list is feared by the participants for it could have a
negative effect on the everyday household routines and responsibilities within the family
unit.
A number of prototypes of pervasive shopping aids for supporting grocery
shopping have been proposed in the literature. These range from instrumented shopping
carts to applications that run on a PDA or a mobile phone.
The MyGrocer system, an example of instrumented shopping carts uses RFIDbased product identification to monitor which products the customer has selected and
provides special offers and information about the total price of the current shopping
basket P. Kourouthanassis and G. Roussos (2003).
An example of applications that run on a PDA or a mobile phone is the EasiOrder system proposed by R. Bellamy et al (2001), which has a list-based personalized
recommendation system, letting the users place grocery orders remotely and fetch them
later on from the supermarket.
Recommendations have also been considered in other systems. For example, R.
D. Lawrence et al (2001) make recommendations by matching the customer with a
cluster of similar shoppers.

11
The iGrocer system proposed by S. Shekar, P. Nair, and A. S. Helal (2003), on the
other hand, adapts its recommendations based on the customers purchase history and
preferences selected from a menu.
Another common feature has been a map that can guide customers to the products
of interest, usually listed in a digital shopping list. In addition to pervasive shopping aids
that support grocery shopping, various pervasive shopping aids that focus on malls or
other retail domains have been proposed in the literature; example is the work of W.
Reitberger et al (2011).
The present study is similar to the studies above in the sense that it will also
provide aid to the shoppers by providing relevant information about the products found in
a grocery store. However the present study will use Object Recognition for detecting and
recognizing the products and Augmented Reality in displaying relevant information about
grocery products which was not used by the studies above.

B. Local
Studies in Object Recognition using Computer Vision have proven its usefulness
in different types of applications.
D. Guzman and E. Peralta (2008) in their study entitled Classification of
Philippine Rice Grains Using Machine Vision and Artificial Neural Networks have
demonstrated the capability and potential of machine vision with well-trained multilayer
neural network classifiers for sizes, shapes, and varietal types identification of rough rice
grain samples grown in the diverse agro-ecological zones in the country.

12
D. Dizon and J. Pabico (2007) in their paper A Low-Cost Machine Vision
System for Real-Time Fire Detection presented methods of identifying fire for a wide
range of local fire and false alarm events using machine vision algorithm that analyzes
color video streams captured from a cheap USB-mounted web camera. Results of their
study shows that their vision-based fire detection system from off-the-shelf hardware can
be a cheap yet flexible alternative to traditional ones
A. De Grano and J. Pabico (2007) in their paper A Neural Network-based Color
Computer Vision for Grading Tomatoes (Lycopersicon esculentum) developed a
computerized color image analysis procedure and a neural network model (NNM) to
automate the classification of maturity of fresh tomatoes it uses a computer color vision
as its artificial eye and an NNM as its artificial brain. Their result shows that an
automatic vision system possesses the same grading accuracy as the human experts but is
more efficient than the manual grading.
Augmented Reality is also used in a number of studies. In the study of F. Santillan
(2010) entitled Interior Design Simulator using Augmented Reality shows how
Augmented Reality tackles the problems of the old system of interior designing.
A study was done to develop a vision system for detecting and counting moving
objects of similar form using model-based object recognition via a web camera E.
Aguilar (2005). The software used the live video feed from a Logitech Quickcam Web
camera. It successfully detected all moving objects but failed to detect which of the
objects have the same form as the model object that affected the actual counting of the
objects of similar forms.

13
Moreover, in a study done by K. Palis and A. Medina (2009), STAR was
developed, which stands for Special Effects in Theatre using Augmented Reality. It
worked by first learning the color in real-time using Model Learning and Histogram
Initialization. After learning the color, Overlay Initialization is done using the Sliding
Window Algorithm that works by defining a square and then sliding it all over the
captured video to get the costume that will be tracked. For keeping track of the object,
Histogram Back Projection was the method used. When the other processes are done the
image sequences are overlaid and synchronized with the video screen and are
concurrently carried out with color tracking.
The present study will use computer vision in performing object recognition
which is similar to the works of D. Guzman and E. Peralta (2008), D. Dizon and J.
Pabico (2007), A. De Grano and J. Pabico (2007) but the present study will identify
products in a grocery store using object recognition which is not done by the studies
above. The study will also use Augmented Reality similar to what F. Santillan (2010), E.
Aguilar (2005), K. Palis and A. Medina (2009) have used however the present study will
use Augmented Reality in displaying the information about grocery products which is not
the case of the studies mentioned above.

14
Conceptual Framework

IT Experts
User interface
Performance
Security
Evaluation
Analysis and
Design

Feedback

Development
Feedback

Evaluation

Users
Ease of use
Reliability

Development and
Evaluation of
Shopping
Assistant using
Object
Recognition and
Augmented
Reality for Mobile
Phones

\
Figure 1 Paradigm of the study

Figure 1 shows the paradigm of study. The Development and Evaluation of

Shopping Assistant using Object Recognition and Augmented Reality for Mobile Phones
will follow five steps in order to create the system.
In the analysis and design phase the researcher will observe and analyze areas in
the field of Information Technology relevant to the study. The most feasible research to
undertake will be selected. Research objectives will be formulated after that the designing
of the software will proceed.
The system will be developed based on the design, followed by the evaluation of
both IT Experts and Users. Based on their recommendations the proposed system will be
redesigned, modified and fine-tuned.

15
Chapter 3
METHODS OF STUDY AND SOURCES OF DATA
This chapter presents the methods of research in developing the system. It also
includes the method of gathering data, the research instruments or analytical tools and the
statistical treatment of the gathered data.

Research Design
The researcher will implement a research and development design wherein the
proposed system will be designed, created, tested and then validated for its significance.
The product will then be pilot tested to determine its usability. It will be revised and
improved based on the users evaluation and feedback. This cycle will went on until the
proposed system addressed the requirements of the Users and IT Experts.
The researcher will use Gantt Chart as the project management tools, and the
Rapid Application Development (RAD) as the software engineering paradigm.
All throughout the RAD phases, the following will be used: the System Structural
Chart (SSC) as the Visual Table of Content (VTOC) for the hierarchy of the functions of
the proposed system, the Entity Relationship Diagram (ERD) as the data modeling tool,
the Context Diagram and Data Flow Diagram (DFD) as the process modeling tools.
The RAD was chosen because it emphasized an extremely short software development
cycle. It is a high speed development paradigm which demands a systematic, sequential
approach to software development applications such as the developed software. Its phases
that will be implemented by the researcher included the business modeling, data
modeling, process modeling, application generation and testing phases.

16
Tools Used in the Development of the System
The Development and Evaluation of Shopping Assistant Using Object
Recognition and Augmented Reality will be done using OpenCV 2.4, Android OS, C++
OpenCV - OpenCV stands for open source computer vision library. Originally
developed by Intel, it runs on both Windows and Linux and is focused mainly towards
real-time image processing. Other areas that the library covers are:

Object Identification

Face recognition

Segmentation and recognition

Motion understanding and tracking

Mobile robotics

Ego-motion

Gesture recognition

Structure from motion (SFM)

Human-computer interface (HCI)

C Programming Language - C is a general purpose, procedural, imperative

computer programming language developed in 1972 by Dennis Ritchie at the Bell

Telephone Laboratories for use with the UNIX operating system. Now it has extended to
a lot of other platforms and is one of the most commonly used programming languages. C
is the programming language used most in the writing of system software and is also
widely used in the writing of applications.
Characteristics of C

17
Here are some of the more important characteristics of the C programming language:

Has facilities for structured programming

Allows lexical variable scope and recursion

Static type system prevents many unintended operations

C has many keywords:

o Declaring data types using struct, union or enum.
o Declaring variables using char, int or double.
o Control flow statements used for conditional or iterative execution such as
if, else, do, while and for.
o Arbitrary jumps are possible with goto.

Parameters of C functions are passed by value

Pass-by-reference is achieved by explicitly passing pointer values

The struct data type allows associated data elements to be linked and controlled as
a unit

Low level admission to computer memory via machine addresses and typed
pointers

Intricate functionality such as mathematical and I/O functions included in library

routines

Pointers allows an object or function to be referenced and recorded in memory by

their address or location

Source text is free-format, using semicolon as a statement stopper

Android OS - an open source platform for mobile development developed by Google

18

Scales Use in Validating the System

To determine that the system was able to satisfy the requirements, the following
scale also known as Likerts Scale was used by IT experts and end users in rating the
system. Pilot testing and trial runs was conducted to let the respondents in rating the
system
Likerts Scale
Mean Value

Weight

Verbal Interpretation

4.51-5.00

Excellent

3.51-4.50

Very Satisfactory

2.51-3.50

Satisfactory

1.51-2.50

Good

1.00-1.50

Poor

IT Experts and End Users who will evaluate the System

The intended users of this study will be the people that shops at grocery stores in
Tarlac. The respondents of the study will be (100) randomly selected grocery shoppers in
Tarlac. Feedbacks and recommendations will be valuable information in greater
improvement of the system.
Five (5) IT experts from the College of Computer Studies and from the outside of
TSU will be selected to evaluate systems user interface, performance and security.

19
Research Instruments
Observation and Interviews will be conducted to gather information and feedback
from the intended users
Library and the internet will also be used to explore studies and literatures related
to the researchers study. This will help the researcher in conceptualizing the study.
Beta testing will also be conducted allowing the users to try-out the system before
the release of the final version. After which, questionnaire will be prepared and
distributed to the respondents for the evaluation of the system.

Statistical Treatment
Tables will be used for the presentation of data to facilitate understanding and to
clearly show the preferences of the respondents.
Frequency distribution a tabular form that ranks the disorganized data from the
highest to lowest and weighted arithmetic mean was used to facilitate analysis of the data.
Responses on the evaluation of the proposed system will be determined and
analyzed using the arithmetic weighted mean (WM):
=

(1 + 2 )

where:
WM = computed weighted mean
= symbol if summation process
f= frequency
x1, x2 xn the value of case being observed

20
N = number of respondents
Quantitative responses will be analyzed using descriptive statistics. Descriptive
statistics will be used to describe the basic features of the data in a study. It will provide
simple summaries about the sample and the measures. Together with simple graphics
analysis, it will form the basis of virtually every quantitative analysis of data. Weighted
arithmetic mean and frequency of the questionnaire items will be calculated.
All computation will be done using computer software statistics.