Recent Researches in System Science

FPGA & DSP infrared image processing module for people and objects detection
Snejana Pleshkova, Department of Telecommunications Technical University Kliment Ohridski, 8 Sofia snegpl@tu-sofia.bg
Abstract: Infrared image processing systems are used in the beginning only as visualization systems for observation of people and objects in the night or as night vision systems. Next in these systems are included the blocks and algorithms for thermal image processing and these systems became the base of the todays infrared image processing systems. There are many areas of infrared image processing systems applications like industrial control, medical, military and police systems, etc. One of the topic applications of the infrared image processing systems now are the surveillance and observation infrared system. The goal of this article is to describe the development and of a module with FPGA (Field Programmable Gate Array) and DSP (Digital Signal Processor) for infrared image processing in infrared surveillance and observation system for customs control and combating terrorism. It is proposed the structure of the module suitable for capturing and processing thermal images from an infrared image sensor. Some results of testing the proposed FPGA and DSP infrared image processing module are presented with an algorithm developed for people or objects detection in thermal images. Key-Words: - infrared visual systems; infrared image processing; FPGA and digital signal processors; people and objects detection

1 Introduction
Infrared vision systems are the information systems used in military, police custom traffic control, industrial and other specific applications for collecting and processing visual information from infrared images [1, 2]. They are composed usually with a infrared video camera [3] for capturing the still or moving thermal images. The means or tools for development the methods and application algorithms for infrared captured image processing can be an appropriate hardware or software for implementation, testing and execution of these methods and algorithms [3, 4]. Most of these tools work well for developing and testing tasks, but for real practical implementation of the developed methods and algorithms is necessary to transfer these algorithms in the special developed for the appropriate task fixed or programmable modules. The goal of this article is the development of such module with FPGA (Field Programmable Gate Array) and DSP (Digital Signal Processor) specially designed for infrared image processing in infrared surveillance and observation system for customs control and combating terrorism.

for Objects structure of an intelligent Multimedia Detection

A. The place of the of FPGA and DSPs Infrared Image Processing Module

There are thermo visual systems with image processing modules and embedded digital signal processors for real time object detection [5]. From the schema block of such system (Fig. 1.) is possible to define or determine in the general case the place of the proposed infrared image processing module as a part of an infrared visual system.
IR camera 1 IR camera 2 IR camera 3 IEEE 1394 IEEE 1394 PHY Layer IEEE 1394 Link Layer Digital Signal Processor LCD Display

HOST Controller

USB

FPGA

LAN Ethernet Or Wireless Connection WLAN

Real Time Thermo Vision Module

2 Infrared Image Processing Module Archtecture in a Thermo Vision System

Figure 1. The schema block of a thermo visual system wiith embedded digital signal processor for real time object detection

ISBN: 978-1-61804-023-7

253

Recent Researches in System Science

The important requirements in such existing examples of thermal vision systems are the real time execution of the algorithms for infrared image processing in some practical applications such as objects or people detection and tracking. Here in this article the goal of development of an infrared image processing module is something different - to give the priority of FPGA (Field Programmable Gate Array) over DSP (Digital Signal Processor) in the creating, testing and practical implementations of algorithms for infrared image processing tasks. The differences in implementation of the infrared image processing algorithms between FPGA (Field Programmable Gate Arrays) and DSP (Digital Signal Processor) are the following: - the design style in Field Programmable Gate Arrays (FPGA) is more deeply directed to hardware style of design, opposite to Digital Signal Processors (DSP), where development of the algorithms is realized as a program for the chosen signal processor; - the development of image processing modules with Field Programmable Gate Arrays (FPGA) is accomplished with standard VHDL or Verilog [7] languages for programmable logic arrays design;) many well developed image and signal processing libraries for implementing processing and calculation blocks in Field Programmable Gate Array; - there are for Field Programmable Gate Arrays (FPGA) s (FPGA), which are very easy to use in special cases of infrared image processing modules. The above mentioned differences between FPGA (Field Programmable Gate Arrays) and DSP (Digital Signal Processor) can be used in design of an infrared image processing module, proposed in this article. The general requirements in this case are to design the infrared image processing module with effective using of the hardware and software resources of Field Programmable Gate Arrays (FPGA) and to communicate with the added Digital Signal Processor (DSP), using their capabilities of fats calculations as a supplementary possibility to extend the processing speed.

Infrared Image Sensor Pixel Data Data

FPGA External Memory Expansion Interface Embedded TCP/IP Interface

FPGA
I2C

Clk

JTAG EEPROM DSP Interface

RJ-45 Wi-Fi Display Interface LCD Display

DSP
DSP External Memory

Figure 2. The structure of the proposed infrared image processing module

B. The structure of the proposed FPGA and DSP

Infrared Image Model The structure of the proposed infrared image processing module is show on Figure 2. It can be seen from Figure 2, that these requirements are satisfied, because the FPGA (Field Programmable Gate Arrays) is presented as central part of the proposed structure of the module for infrared image processing.

The FPGA (Field Programmable Gate Array) is connected with the Infrared Image Sensor. This allows FPGA to receive the Pixel Data of the captured images from the Infrared Image Sensor and also to control and adjust the Infrared Image Sensor via I2C Bus. The FPGA is connected also to the DSP (Digital Signal Processor) via DSP Interface to realize the communication with the added Digital Signal Processor (DSP). It is possible to transfer the captured infrared images as Pixel Data or as intermediate processing images from the FPGA to DSP. The DSP can prepare some infrared image processing operations or algorithms and then transfer back the results to the FPGA. The FPGA (Field Programmable Gate Array) and DSP (Digital Signal Processor) both have the external memory block named in Fig.1 as FPGA External Memory and DSP External Memory, respectively. It is possible to use these two external memories as additional memory locations to storing input, intermediate or output infrared images in execution of the algorithms for infrared images processing in the FPGA or DSP blocks of the proposed module for infrared image processing. The rest of the blocks shown in Fig. 1 and included in the proposed module for infrared image processing are not directly connected with the executed algorithms for infrared image processing, but are closely connected and necessary for the module control of operation, testing and their connections and interfacing with other pars of a infrared or thermo visual system with a concrete

ISBN: 978-1-61804-023-7

254

Recent Researches in System Science

practical application. These additional block shown in Fig. 1 in the proposed module for infrared image processing are the following: - Display Interface for connecting the LCD Display or monitor useful for input, intermediate or output infrared images visualization in time of development, testing and execution of the algorithms for infrared image processing in the proposed module; - Embedded TCP/IP Interface with wired or wireless capabilities and two standard connectors RJ-45 for wired local area network and Wi-Fi for wireless networks, both allow the proposed infrared image processing module to send or receive the infrared input, processed or output visual information with establishing the connections with the other parts in an infrared visual system; - JTAG (Joint Test action Group) interface for standardized [8] test procedures in the proposed module for infrared image processing; - EEPROM block for collecting the initial data for all blocks in the proposed module and especially for FPGA and DSP initialization starting an algorithm for infrared image processing and storing the current data in EEPROM finishing the work of the algorithm in the proposed module and using the same stored initial data in the next start procedure; - Expansion Interface for future addition of the supplementary blocks to the developed module for infrared image processing. C. The Implementation of Spartan -3E Xilinx FPGA in Infrared Image Processing Module The proposed structure of the infrared image processing module, shown in Figure 2, is realized with the popular and used in many signal and image processing applications Xilinx FPGA Spartan 3E [9]. The main connections for this concrete implementation of the Spartan 3E FPGA in the proposed infrared image processing module are presented in Figure 3.

The capabilities of using the Spartan 3E FPGA in infrared image processing algorithms and applications are the following [10]. - Real-Time Pixel-by-Pixel Processing and Transformation; - Logical Complexity - around 5,000 Slices or More; - Embedded Memory for Image and look up Table Storage. With these capabilities of Spartan 3E FPGA is possible to realize the following real time characteristics of the proposed infrared image processing module: - real time capturing of thermal images from standard infrared cameras; - real time interfacing between FPGA, DSP (Digital Signal Processor) and infrared camera; - real time infrared image processing in FPGA and DSP (Digital Signal Processor); - real time visualization of input, intermediate and output infrared images. D. The development of an algorithm for testing the proposed structure of the module with implementation of Spartan -3E Xilinx FPGA in infrared image pocessing for object detection The structure of the proposed module for infrared image processing, show as block schema in Fig. 2 and as an implementation with the Spartan 3E FPGA, shown in Fig. 3, is tested with an algorithm for object detection in infrared images. The development of the test algorithm is first created and simulated as Matlab Simulink model shown in Figure 4.

Figure 4. The Simulink model of the algorithm for testing the capabilities for infrared image processing of the proposed structure of the module with Spartan 3E FPGA Figure 3. The Implementation of Spartan -3E Xilinx FPGA in Infrared Image Processing Module

ISBN: 978-1-61804-023-7

255

Recent Researches in System Science

The infrared or thermal image data are presented inside this block in form of binary image data format and are shown as first block Read Binary File in Figure 4. The test algorithm performs the infrared image processing as a filter labeled in Figure 4 as General FIR Block. The coefficients of the FIR filter are presented as features in form of rectangles, are collected as templates and are also show in Fig.ure 4 as Read Binary File 1, Read Binary File 2 and Read Binary File 3, respectively for each of the chosen in [11] features: vertical (ftv.bin), horizontal (fth.bin) and diagonal (ftd.bin) in form of rectangles. In Figure 4 is shown the name of input infrared image loaded from the database with collected infrared images. The infrared input image chosen from the database in this testing example is shown as the block Thermal Image 1.bin in Figure 4. The processing operations are presented as the three Simulink blocks 2-D Correlation, which perform matching in form of two dimensional correlation between chosen for testing Thermal Image 1 and each of the features fth and ftd, respectively. In these 2-D Correlation blocks is applied the following equation to calculate the two dimensional cross-correlation C(i, j ) between the matrix of the tested thermal image and each of the matrix of the features in form of rectangles ftv, fth and ftv:

named Maximum , to calculate the maximal values ft kmax in each of three matrices C k (i, j ), (k = 1, 2, 3) :

ft kmax = max[C k (i, j )] for k = 1, 2, 3 .

(2)

C k (i, j ) =

N y 1N x 1 m = 0 n =0

TI (m, n ).FT (m + i, n + j )
m k

(1)

for 0 i N y + N fyk 1; 0 j N x + N x f k 1; k = 1,2,3 , where

C k (i, j ) is two dimensional cross-correlation for each of three features (k = 1,2,3) ; N x and N y - horizontal and vertical dimensions of
thermal image matrix TI m , respectively;
y Nx f k and N f k - horizontal and vertical dimensions of

the feature matrix FTk , respectively;

k = 1,2,3 - index of three features matrix FT1 , FT2 and FT3 for features ftv, fth and ftd,
respectively; The outputs of three blocks 2D Correlation contain the results of two dimensional cross-correlation in form of matrices C k (i, j ) for k = 1,2,3 . From these three outputs of the blocks 2D Correlation the values in matrices C k (i, j ) are estimated in three Simulink blocks

The calculated in blocks Maximum maximal max max ft1max , ft 2 and ft 3 correspond to values determination existence of features for vertical ftv, horizontal fth and diagonal ftd properties in tested Thermal image 1 max max and ft 3 from the Three outputs values ft1max , ft 2 corresponding outputs Val of the respective blocks Maximum are merged using a Simulink block for multiplexing (the vertical black bar with three inputs in Fig. 4). The joined information for the positions of the max max maximal values ft1max , ft 2 and ft 3 determines and guarantee with some supposition and probability that from these joined feature positions is possible to locate the position of the objects in tested thermal image. To mark the places of these possible object positions is added in Figure 4 the block Constant to define the initial characteristics of Rectangle (rw, rh) with corresponding dimensions rw and rh for width and height, respectively. The initial values or rw and rh are added in a two input multiplex block in Fig. 4 with the max max joined feature ft1max , ft 2 and ft 3 position information. The output of the multiplex block is connected to the input P of the block Draw Shapes in Figure 4. To the input I of this block is entered the visual information for the tested Thermal Image 1. The final result of the operation of this block is to draw the rectangles with the appropriate dimensions (width rw and height rh) in the possible places to mark detection of existent objects. To reach this final operation and to achieve a good exactness of the important objects detection in tested thermal images are prepared the suitable operations for max max estimation of joined feature ft1max , ft 2 and ft 3 to define an object with some preliminary set forms and dimensions. This is a very important and necessary processing step to eliminate from the list of possible detected objects the objects with small or large dimensions (exceeded the preliminary set dimensions) or the objects with an unusual or unexpected form. These restrictions depend from the application of the thermo visual system and here in this article are defined for the goal of detection the objects hidden in dress or baggage of people. The results of detected objects separated and marked with the appropriate rectangles in the tested thermal

ISBN: 978-1-61804-023-7

256

Recent Researches in System Science

image are directed to the Output of General FIR Block, shown in Figure 4.

3 Test and Results

The described in Figure 4 Simulink model of the algorithm for objects detection in infrared images is implemented in the proposed FPGA and DSP infrared image processing module. The tests are performed with the infrared images collected in an infrared image database. The infrared images are prepared with infrared camera type Flir [4, 12] and the preprocessing of these images is made with Flir software [13]. One of these test images used as input infrared image is shown in Fig. 5. The image contain an object (ring) hidden under the gloves of a person. The goal of the algorithm for processing of this infrared image is to find and separate the object existing in the input infrared image.

Figure 6. The output infrared image show the correct object detection (ring separated with a rectangle) after processing with the algorithm for object detection implemented in the proposed FPGA and DSP infrared image processing module The view of the FPGA block used in the proposed FPGA and DSP module is shown in Figure 7 [14] and the view of the DSP block used is TMS320C6416 [15], shown in Figure 8.

Figure 5. An example of infrared image containing an object (ring) hidden under the gloves of a person

The result output infrared image, shown in Fig.6, demonstrate the correct work after execution of the algorithm implemented in the proposed FPGA and DSP infrared image processing module.

Figure 7. The view of the FPGA block used in the proposed FPGA

Figure 8. The view of the DSP block used in the proposed FPGA

ISBN: 978-1-61804-023-7

257

Recent Researches in System Science

In the Table 1 are listed some of the achieved important characteristics of the proposed FPGA module for infrared image processing. Table 1
Property Spartan X3S1500 4 4.379 28% of Logic 40% of Multipliers 17% of Memory

Slides Logic Memory Clock Frequency

75 MHz

Acknowledgment This work was supported by National Ministry of Science and Education of Bulgaria under Contract DDVU 02/04/2011: Thermo Vision Methods and Recourses in Information Systems for Customs Control and Combating Terrorism Aimed at Detecting and Tracking Objects and People. References
Lebold J. Infrared Thermography and Distribution System Maintenance Electricity Today, Volume 3. 2008, 1819 [2] Coon D. D. and Perera A.G. U. Spectral information coding by infrared photoreceptors. International Journal of Infrared and Millimeter Waves Volume 7, Number 10, 15711583 [3] FLIR Application Book. FLIR Company 2010 [4] FLIR Infrared Cameras. http//www.flir.com/ [5] Al.Bekiarski, Sn.Pleshkova, L. Taneva. Thermo vision system with embedded digital signal processor for real time objects detection, The 4th International Congress on Image and Signal Processing, 15-17 October 2011, Donghua University, Shanghai, China (to be published) [6] IEEE Standard VHDL Language. Reference Manual. IEEE Std 1076, 2000 Edition (Incorporates IEEE Std 10761993 and IEEE Std 1076a-2000) [7] IEEE Standard Verilog Hardware Description Language. IEEE Std 1364-2001 (Revision of IEEE Std 13641995) [8] IEEE Standard Test Access Port and Boundary-Scan Architecture IEEE Std 1149.1-2001 (Revision of IEEE Std 1149.1-1990) [9] Spartan-3E FPGA Family: Data Sheet. Xilinx. DS312 (v3.8) August 26, 2009 [10] Tusch M. High-Performances Image Processing with FPGAs. CEO Apical Limited. 2010 [11] Sn.Pleshkova, Al.Bekiarski. Algorithm of feature estimation for real time object detection in thermal images, The 4th International Congress on Image and Signal Processing, 15-17 October 2011, Donghua University, Shanghai, China (to be published)
[1]

ISBN: 978-1-61804-023-7

258