OBJECT POSITION ESTIMATION AND TRACKING USING AN ACTIVE CAMERA NETWORK

Agus Budi Dharmawan1) 2), Prof. Dr. Karl Kleinmann1), Prof. Dr. Hermann Meuth1), Prof Mauridhi Hery Purnomo2) Master of Science in Electrical Engineering, Fachbereich EIT, University of Applied Sciences Hochschule Darmstadt (h-da), 2011.
2)
1)

Graduate Study Program, Departement Of electrical Engineering, Faculty of Industrial Technology, ITS Surabaya, Indonesia. 2011.

Abstract
This paper presents 3-D estimation of objects position, combining static and moving camera with the Template-Color matching method. The template-Color matching is used to track an object of interest. Color-based tracking method gives more confidence to track an object from many moving objects in arbitrary direction. Template is updated when the same 3-D and 2-D position matching is found. 3-D object position is calculated by merging line of Sight from multiple cameras. Our goal is to develop an object tracking and 3-D estimating system to aid in locating object target within image. networking system. Operator will work with both the systems, either in manual or automatic mode. Operator 1. INTRODUCTION will choose an object among many moving objects to track it. Image processing and Kinematics calculation Machine can see and knows where the object is, will guide the system to view the desired object and something that seems impossible. Many techniques estimate its position. could be adopted to make the system more intelligent. In this project, another solution is proposed to In image processing, camera acts as an eye. System solve the problem by implementing semi-automatic takes input from camera i.e. image pixels such as color active camera system. This project is related to the information. Image processing and networking together previous project. Previous system uses image is helpful in various applications. One of them is used processing in a networked sensor system which to find the objects position using camera networking. consists of several moving cameras. With this Active Multiple cameras can be used to observe the object of Object Tracker (AOT), system can track objects using interest. As a Surveillance system, camera is used to camera. Each camera is connected, and exchanges monitor the objects behavior, activity, or any other information of objects position. If one camera is changing information such as a movement and objects covered by an obstacle, the others can still follow the localization (position). This system can run object. automatically to help human for observation purpose. Camera coordinates and object coordinates are related and can be calculated using Denavit-Hartenberg Calculation. It includes calculation of rotation and translation from the base coordinates to the camera and from camera to the object coordinate system. Position of the object or the camera can be calculated using this calculation. The Idea is to track the object of interest in arbitrary movement. Operator can chose the object by clicking with the mouse pointer. Computer saves the selected object as a template which can be set with different sizes. System will search the object, based on template and color matching method. Afterwards it sends the found objects position to server (Explorer) to get the world coordinate. Tracking will process by combining moving and static cameras. Each client (Backend) will be equipped with one IP camera. Our project will combine Image processing and networking together. Tracking of an object and estimating its position is done in distributed processing. Two static and one moving camera is put in a 2. BACKGROUND

2.1. Camera Model Relationship between image plane in single camera and image plane displayed on the monitor must be discovered to have the proper camera calculation. Combining between image processing and image geometric also become important part to design the algorithms.

Figure 1. Camera Model

Image from same scene will have relationship between two cameras C and C' illustrated in Figure 4. These could be two physically separate cameras or a single moving camera at different points in time [Aghajan, H. et al]. Image point in C (ximage, y image) are corresponding with image point in C' (x'image, y'image) called point correspondence. Relation between C and C' can be projected in equations bellow.

X Y y= f (1) Z Z X Y x = f y = f (2) Z Z f and f are focal length from C and C. Rotation R and translation [tX tY tZ]T are used to find x= f
the C coordinate system.

suitable for tracking objects whose poses do not vary considerably during the course of tracking [Yilmaz et al]. Template matching is a function matches an actual image patch against an input image by sliding the patch over the input image using one of the matching methods [Bradsky G. et al]. RGB Channel is used to have information from the template and find the best likelihood from the image. Matching algorithm will perform scan on the image to get minimum Template-Source Difference (TSD) value.

TSD( x, y ) =

Trows Tcolms i=0 j =0

Diff ( x + i, y + i, i, j )
y =0

(7)

Srows Scolms x =0

TSD( x, y)

(8)

X t x X Y = R Y + t y Z t z Z
With
cos cos + sin sin sin R = cos sin + sin sin cos sin cos cos sin cos cos sin sin cos + cos sin sin sin sin + cos sin cos cos cos

(3)

(4)

Where and are rotation angles around X-, Y-, and Z-axes in C coordinate system. By substituting equation and, relations between two coordinate are:
r11 t f f f y + r13 f + x x + r12 Z f f r31 r32 tz x+ y + r33 + f f Z t f f f y + r23 f + Y x + r22 Z f f r31 r32 tz x+ y + r33 + f f Z

X = x = f Z

(5)

y = f

Y = Z

r21

(6)

Figure 3: Scan A Template Across An Image For Matches The approach is to computes the distance between template and image source and find the minimumdistance. It chooses smallest distance to make decision image [Gonzales R. C. et al]. A template is a small region including a feature to be tracked in a certain image. Using the template, feature positions are estimated in some consecutive images by searching minimum dissimilarity area. Then, the template is replaced by a new image.[Toshimitsu Kaneko] 2.3. Kinematics the serial kinematics, DenavitTo modeling Hartenberg transformation will be used in mathematical description. The description includes rules on applying the coordinate systems of the members in the kinematics chain Relationships between inter-connected joint, position, and orientation in rigid robots can be calculated with the kinematics equation. Forward kinematics is used to find the position and orientation of the end-effectors between individual joint. The variable are angle and distance link that calculated by the rotation and the revolution matrix. In Denavit-Hartenberg convention, each motor is assumed to have degree of foredoom. Each frame will be attached to each link and use transformation to get relationship between those frames. Four basic transformations are two rotation matrix and two translation matrix calculated with homogeneous transformation.

Figure 2: Line Of Sight from Two Cameras 2.2. Image Processing The basic method from template matching is to have a template as a mask and use this mask to search specific feature from the image. Templates are formed using simple geometric shapes or silhouettes [Fieguth and Terzopoulos]. The advantage from template is it carries both spatial and appearance information. However, templates only encode the object appearance generated from a single view. Thus, they are only

T(F1 F ) =Trans(dn) Rot1 (n)Transan) Rot (n) (9) n n zn1 zn xn ( xn

3.

APPROACH & IMPLEMENTATION

1 0 Trans z n1 (d n ) = 0 0

0 0 0 1 dn 0 0 1 0 0 1 0

(10)

cos n sin n Rot z n1 ( n ) = 0 0 1 0 0 1 Trans xn ( an ) = 0 0 0 0

sin n cos n
0 0 0 an 0 0 1 0 0 1

0 0 0 0 (11) 1 0 0 1

(12)

To get more understanding from this system, first we will define the name to be used: Explorer: Denoted for Server. The Form will be designed with GUI Interface for operator. Backend: Denoted for client side. One camera will be attached to each Backend. Two Stepper Motors can also be connected in the backend (moving camera system). Explorer gives the position of object of interest to the backend. Backend will mark the object position using the position value. In Automatic mode, Backend will execute tracking process after having information of the object position from the Explorer. Template and color matching algorithm are running in Tracker class. With this algorithm, Backend is able to perform tracking object. The results from tracking class are new 2-D object positions (xobj, yobj). These results will be sent and calculated in Kinematics class. This class performs LOS from 2-D calculation, 2-D from 3-D calculation and Angle from 3-D calculation.

0 1 0 cos n Rot x n ( n ) = 0 sin n 0 0

sin n
cos n 0

0 0 0 1
(13)

From substitution from equation 9 13 to equation 14 :

cos n sinn cos n sinn sinn an cos n sin cos cos n n cos n sinn an sinn n T(Fn1 Fn ) = 0 sinn cos n dn 0 0 1 0

(14) Figure 4: Use cases overall system

Where: n : joint angle, angle between x in origin n-1 axes to x in origin n axes. n : link twist, angle between z in origin n-1 axes to z in origin n axes. dn : link offset, length between the origin n-1 to the normal n along Zn-1 axes. an : link length, length between the origin n-1 to the normal n along Xn axes. The four parameters are fulfilling each homogeneous transformation for Denavit-Hartenberg convention. Equation 27 can be substituted with formula 20 and the results will be obtained in new matrix distance. Each homogeneous transformation matrix is represented for rotation and translation. Parameter an is distance between Zn-1 and Zn axis and measured along the axis x1. Angle is the angle between the axis Zn-1 and Zn, Xn is measured in the normal plane. Positive sense for is determined from Zn-1 to Zn by the right-hand rule

To perform object tracking and estimating the task, three different classes are implemented on each Backend. They are Tracker System, Kinematics System and Can Communication System. Tracker system handles the image processing algorithms. First process in Tracker System is showing the image merged with template marker. To perform this task, Tracker needs the object position information from the Explorer. To search the objects of interest, template matching method is implemented on the Tracker System. After Backend gives command to tracking the object, this algorithm will scan the image and compare with the template (template matching). When the minimum difference value between image source and template is found, the algorithm will save the position and calculate this value to get 2-D position. Kinematics system will perform The DenavitHartenberg and Matrix calculations. 2-D position from tracker system will be calculated to get Line of sight (start and direction point). The LOS value is sent to the explorer to get 3-D estimate. 3-D estimate is calculated and stored in LOS handling class. 3-D to 2-D calculation for static camera and 3-D to Angle calculation for rotation of the camera is also

processed in the system. In moving camera in the Backend, angle values will be sent to Can Communication system to drive the Stepper Motor. The system uses Can Communication system to drive the Stepper Motor. This system will translate the angle value from kinematics system to step and direction value for driving the Stepper Motor. Can Communication system will send step value to microcontroller and then with driver module, the values will make the Stepper Motor move to the desired position.

4.

TESTING

Test will use robot arm (KUKA) to move the object. The base robot position are (1989, 3201, -930). Robot will move with given position path. System will track the object on the robot arm. The Kuka robot will move the object with two different positions, LIN motion and PTP motion. LIN motion is moving the object along linear paths and PTP is moving the object from point to point until it reaches the desired position. The system is able to find the object. However, some failure also appears. System cannot find the interested object because the object is not as appeared in the image. To solve this problem, moving camera can be used. With motor attached, the camera will try to always put the object in the center of image.

Figure 5: Packet Diagram Of Overall System

Tracker, Kinematics and Can Communication class are implemented on each Backend. Three UseCases were used in this system are LOS from 2-D, 3-D from LOS and 3-D to 2-D-Angle. Use Case 1: Every Backend will perform 2-D to Line Of Sight under Automatic mode. The goal from this use case is to detected image position from Tracker class to produce Line of Sight value. DenavitHatrenberg was used for calculating from 2-D position in camera to get start and direction point of Line of Sight. Use Case 2 is calculating Line Of Sight to 3-D. This process will be done in Explorer. All Line of sight values from every Backend will be sent to the LOS merge class. Explorer will estimate all LOS to get 3-D object positions based on world coordinates and stored in AOT point class. This 3-D estimation will be shown in System control page on GUI and send to every Backend. Use Case 3 performs two different calculations. First is 2-D from 3-D and the second is angles from 3D calculation. First calculation will produce new object position in 2-D coordinate and the second will produce 1 and 2. Result from the first calculation will be used to mark the new object in image plane, and results from the second calculation will be sent to Can Communication Class. In Can Communication Class, the Angle value are Input from Kinematics class which will translate into steps for rotating camera. These steps will drive two Stepper Motors (horizontal and vertical), afterwards moving the camera to the desired position. These movements are being processed to make the objects always appear in the center of image plane. To perform communication, two packages of Discovery and Library class will placed between. AOT_Discovery allow the components in the network to comunicate each other. To recognize all components in the network, AOT_Discovery creates only one discovery transmitter.

Figure 6: Tracking result with Kuka robot Failure also detected in when the system detected the wrong object. This failure appears because the object moves very fast. Since the system is using networking to transfer data and connected with multiple Backend, the time process becomes more critical. To solve this problem, operator can correct the interested object by clicking the actual object position. Different Image results are generated from moving camera. To solve the first problem, moving camera will drive the Stepper Motor using the angle value from Angle from 3-D calculation in Kinematics class.

Figure 6: Tracking result

Hardware testing also performed to find out whether the hardware is working properly or not. This test was observing the Stepper Motor for moving camera. Some movements are produce to get the desired position. From this testing, AOT was perfectly moving the camera with precision angle. In slave mode, operator gives some positions to look at the robot using result value from Tracking Testing. Camera cam perfectly put the object in the middle of image. 5. DISCUSSION and CONCLUSIONS

Q. Cai and J. K. Aggarwal: Tracking Human Motion Using Multiple Cameras. Proceedings of ICPR '96. IEEE (1996). Kaneko, T., Hori, O.: Template update criterion for template matching of image Sequences. In: 16th International Conference on Pattern Recognition, New York, pp.15. IEEE Press, Los Alamitos (2002). L. Velho et al., Image Processing for Computer Graphics and Vision, Text in Computer Science, DOI 10.1007/978-1-84800-193-0-4, Springler-Verlag London Limited (2009). M. Kristan, J. Pers, S. Kovac, A. Leonardis: A Localmotion-based probabilistic model for visual tracking. Pattern Recognition Journal, Elsevier July 17, (2009). Schmitt, Manuel; Liss, Patrick; Schmidt, Steen ; Hackenberger, Bernd: Project Report, Kinematics Software Library. Hochschule Darmstadt, Mrz (2010). Schmitt, Manuel. Entwicklung und Implementierung einer Methode zur Kalibrierung von Schwenk-NeigeKameras in Sensornetzwerken. Master-Thesis. Hochschule Darmstadt, (2011). Wikipedia. Wikipedia Denavit-Hartenberg Parameters. Retrieved from http://en.wikipedia.org/wiki/DenavitHartenberg_Parameters. (2011). Zimmermann, Stephan. Design und Implementierung einer Systemplattform fr ein aktives Kameranetzwerk. Master-Thesis. Hochschule Darmstadt (2010).

Error object position value appears from base value incorrect calculation (D-H parameters). Camera calibration can be performed to solve this problem. However, some problem still appears in this system. Those are data transfer in networking system and performance in tracking algorithm. This problem appears when the system is in automatic mode. Some delay appears in transfer data, especially for image transfer. False Tracking position appears when the moving camera and static camera are unsynchronized. Different loop calculations make the moving camera are late to send the real time LOS. Since data traffic become issues in this project, a better tracking method could help to improve the processing time. Template-Color Matching also has a problem in processing time. Template matching always scans the image in every sequence. This problem can be reducing with adopted the prediction method. For better system, three or more moving camera can be used in this system. With more moving cameras, system can still capture the object while object try to move to outside the camera boundary. More motor can be attached to the system get more degree of freedom in camera movement. Image tracking method can be improved with prediction method. A prediction for object movement such as optical flow and Kalman Filter can be used to reduce the time processes. Other sensor can be used to give another input for the system. This input will provide more confidences to tracking and estimating the object. Improve the speed of Stepper Motor, to provide fast object tracking. Fast image processing also needed to avoid this purpose. Stereo camera can be used and placed on a rigid body to provide moving in three dimensional space (three degrees of freedom) such as robot arms to get the flexibility, speed and high accuracy of movement. 6. REFERENCES

Dharmawan, Agus Budi. Master of Science in Electrical Engineering, Fachbereich EIT, University of Applied Sciences Hochschule Darmstadt (h-da), Germany. Graduate Study Program, Departement Of electrical Engineering, Faculty of Industrial Technology, ITS Surabaya, Indonesia. 2011. Object Position Estimation And Tracking Using An Active Camera Network.

Bradsky G and Kaehler A: Learning OpenCV Computer Vision with the OpenCV Library. OReilly (2008). C. Zhang, J. Eggert, and N. Einecke: Robust Tracking by Means of Template Adaptation with Drift Correction, Springer-Verlag Berlin Heidelberg (2009).