Pattern Recognition Letters 27 (2006) 1948–1956

www.elsevier.com/locate/patrec

A region-based multi-sensor image fusion scheme

using pulse-coupled neural network
a,b,*
Min Li , Wei Cai b, Zheng Tan a

a
Department of Information and Communication Engineering, School of Electrics and Information Engineering,
Xi’an Jiaotong University, Xi’an, Shannxi Province 710049, PR China
b
Xi’an Research Inst. of Hi-Tech Hongqing Town, Xi’an, Shaanxi 710025, PR China

Received 14 June 2005; received in revised form 10 March 2006

Available online 12 July 2006

Communicated by M. Kamel

Abstract

For most image fusion algorithms split relationship among pixels and treat them more or less independently, this paper proposes a
region-based image fusion scheme using pulse-coupled neural network (PCNN), which combines aspects of feature and pixel-level fusion.
The basic idea is to segment all different input images by PCNN and to use this segmentation to guide the fusion process. In order to
determine PCNN parameters adaptively, this paper brings forward an adaptive segmentation algorithm based on a modified PCNN with
the multi-thresholds determined by a novel water region area method. Experimental results demonstrate that the proposed fusion scheme
has extensive application scope and it outperforms the multi-scale decomposition based fusion approaches, both in visual effect and
objective evaluation criteria, particularly when there is movement in the objects or mis-registration of the source images.
 2006 Elsevier B.V. All rights reserved.

Keywords: Image fusion; Pulse-coupled neural network; Image segmentation

1. Introduction Image fusion can be broadly defined as the process of

Extraordinary advances in sensor technology, micro-
electronics and communications have brought a need for
processing techniques that can effectively combine infor-
mation from different sources into a single composite for
interpretation (Hall and Llinas, 1997). As many sources
produce images, image processing has become one of the
most important domains for fusion and has been used in
many fields such as machine vision, medical diagnosis,
military applications and remote sensing (Piella, 2003).
*
Corresponding author. Address: Department of Information and
Communication Engineering, School of Electrics and Information Engi-
neering, Xi’an Jiaotong University, Room 617, Xi’an, Shannxi Province
710049, PR China. Tel.: +86 029 8474 6155; fax: +86 029 8334 3893.
E-mail address: limin@mailst.xjtu.edu.cn (M. Li).
combining multiple input images into a smaller collection
of images, usually a single one, which contains the ‘rele-
vant’ information from the inputs. The aim of image fusion
is to integrate complementary and redundant information
from multiple images to create a composite that contains
a ‘better’ description of the scene than any of the individual
source images. This fused image should increase the per-
formance of the subsequent processing tasks such as seg-
mentation, feature extraction and object recognition. The
different images to be fused can come from different sensors
of the same basic type or they may come from different
types of sensors. The sensors used for image fusion need
to be accurately coaligned so that their images will be in
spatial registration.
A simple image fusion method is to take the average of
the source images pixel by pixel. However, along with sim-
plicity come several undesired side effects including reduced

0167-8655/$ - see front matter  2006 Elsevier B.V. All rights reserved.
doi:10.1016/j.patrec.2006.05.004
 M. Li et al. / Pattern Recognition Letters 27 (2006) 1948–1956
contrast. In recent years, many researchers recognized that
multi-scale transforms are very useful for analyzing the
information content of images for the purpose of fusion.
Various methods based on multi-scale decompositions
(MSD) have been proposed (Zhang and Blum, 1999; Li
et al., 2002). Examples of this approach include the Lapla-
cian pyramid (Burt and Adelson, 1983), the gradient pyra-
mid (Petrovic and Xydeas, 2004), the ratio-of-low-pass
pyramid and the morphological pyramid (Toet et al.,
1989; Matsopoulos and Marshall, 1995). Since wavelet the-
ory has emerged as a well developed yet rapidly expanding
mathematical foundation for a class of multi-scale represen-
tations (Mallat, 1989), it has also been used in multi-sensor
image fusion. At the same time, some sophisticated image
fusion approaches based on multi-scale representations
began to emerge and receive increased attention. Most of
these approaches were based on combining the multi-scale
decompositions (MSD’s) of the source images. The basic
idea is to perform a multi-scale transform on each source
image, then construct a composite multi-scale representa-
tion from these. The fused image is obtained by taking an
inverse multi-scale transform. In general, discrete wavelet
transform (DWT) based fusion method is superior to the
previous pyramid-based methods. Although these methods
often perform satisfactorily, their multi-resolution decom-
positions and the consequent fusion results are shift-variant
because of an underlying down-sampling process. When
there is slight movement or mis-registration of the source
images, their performance will deteriorate quickly.
The traditional algorithms based on MSD techniques
are mainly pixel-based approaches where each individual
coefficient of the MSD decomposition (or possibly the co-
efficients in a small fixed window) is treated more or less
independently. However, in most image fusion applica-
tions, we are not interested in individual pixels but rather
in the objects and regions they represent. It therefore seems
reasonable to incorporate object and region information
into the fusion process.
Because the pulse-coupled neural network (PCNN)
has great advantage in image segmentation (Eckhorn
et al., 1990; Thhomas, 1999), we present a new region-
based image fusion scheme using PCNN, which combines
aspects of feature and pixel-level fusion. The basic idea is
to segment all different input images by PCNN and to
use this segmentation to guide the fusion process. In order
to determine PCNN parameters adaptively, this paper also
brings forward an adaptive segmentation method based on
a modified PCNN, which multi-thresholds determined by
water region area. The implementation of the algorithm
is computationally simple and can be used in real-time
applications.
Extensive experiments with multi-focus image fusion
and different sensor image fusion were performed, all
results show that the proposed method has extensive appli-
cation scope and it avoids some of the well known prob-
lems in pixel-level fusion such as blurring effects and high
sensitivity to noise and mis-registration.
1949
The rest of this paper is organized as follows. PCNN
based image segmentation will be introduced in Section
2. The proposed fusion scheme will be described in Section
3. Experiments will be presented in Section 4, and the last
section gives some concluding remarks.
2. PCNN based image segmentation
2.1. PCNN neuron model and parameters determination
Pulse-coupled neural network (PCNN) is different from
traditional artificial neural networks, models of which have
biological background and are based on the experimental
observations of synchronous pulse bursts in the cat visual
cortex. PCNN can be applied in many fields, such as image
processing, image recognition, and optimization (Ekblad
et al., 2004; Kuntimad and Ranganath, 1999).
Each typical PCNN neuron model has nine parameters
to be determined, including three time decay constants
(aF, aL, ah), three amplification factors (VF, VL, Vh), linking
coefficient b, linking matrix M and W (showed as Fig. 1).
Moreover, to the whole neural network, the iteration times
N should also be decided. The various parameters used in
the PCNN model are of great significance when preparing
the PCNN for a certain task.
The performance of image segmentation based on
PCNN depends on the suitable PCNN parameters. It is
necessary to determine the near optimal parameters of
the network to achieve satisfactory segmentation results
for different images. Up to now, the parameters are most
adjusted manually and it is a difficult task to determine
PCNN parameters automatically for different kinds of
images. During recent years, some work on determining
the optimal values of PCNN parameters has been done.
Some of them are concentrated on optimizing single
parameter while keeping others fixed (Ma et al., 2002;
Liu et al., 2005; Gu et al., 2002). Some of them train the
parameters with desired images to achieve the optimal val-
ues (Karvonen, 2004). However, as Karvonen mentioned, a
very large set of data should be required to optimize PCNN
parameters, which is unfeasible in most applications.
Currently, adopting simplified PCNN model to decrease
the parameters number is an important trend in image
βij 1
αL
… …
Ykl Wijkl
VL Lij
Vθ
αθ
αF
θ ij
Mijkl
step(·)
VF Fij Uij Yij
Sij
Fig. 1. PCNN neuron model.
1950 M. Li et al. / Pattern Recognition Letters 27 (2006) 1948–1956
segmentation applications. There are lots of simplified
PCNN models (Bi and Qiu, 2005; Karvonen, 2004; Aboul
and Jafar, 2004; Ma et al., 2003; Gu and Yu, 2005; Johnson
et al., 1999). These models usually simplify the feeding part
Fij to the normalized pixel intensity Sij and eliminate the
decay factor in linking part Lij.
Based on the research fruits above, we adopt a modified
PCNN model and propose a multi-threshold approach
according to water region area in histogram. Meanwhile,
the adaptive determination method of PCNN parameters
for image segmentation is presented.
The detailed process is given as following.
2.2. Multi-threshold approach using water region area
method
We propose a novel auto-detection algorithm of thres-
holds in image histogram. In accordance with the intuition-
istic features of the histogram, the peaks of the histogram
are considered as watersheds, each valley including two
neighboring peaks and a bottom points. We call the maxi-
mal water capacity in each valley as ‘water region area’.
Step 1. Draw image histogram and smooth it to decrease
noise influence if necessary.
Step 2. Seek all peaks and bottom points in the histogram.
Step 3. Calculate the water region area from the left bot-
tom point. Here, define H as a lower limitation
ranging from 0.01 to 0.03. The smaller the value
of H is, the more threshold points we will get.
When the water region area is larger than H, the
corresponding bottom point will be kept in thres-
hold array Tm. Meanwhile, the corresponding left
side peak point will be kept in peak points array
Pm. Otherwise, the valley will be taken as invalid.
At this situation, compare the two peaks’ values
located in the valley’s two sides:
(1) if the left peak point is larger than the right one, it
will be treated as the new left peak point. While the
next right peak point will be the new right peak one,
the smaller between the current and the next bottom
point will be regarded as the new bottom point.
(2) otherwise, the right peak point, the right bottom
point and the next right peak point will be regarded
as new left peak point, new bottom point and new
right peak point respectively and then the new water
region area will be calculated again.
Step 4. Iteratively execute step 3 until all bottom points
have been processed and then we can get the
threshold array Tm (m = 1, . . . , M and T1 <    <
TM) and the corresponding peak array Pm (m =
1, . . . , M + 1 and P1 <    < PM+1). Hence, a valid
valley Vm includes two neighboring peaks {Pm,
Pm+1} and a threshold Tm (Pm < Tm < Pm+1).
Figs. 5 and 6 in Section 4 show images together with
their water regions and corresponding thresholds.
2.3. Modified pulse-coupled neural network
Considering the applications of multi-sensor image
fusion, the PCNN model we have applied is a modification
of the original PCNN (Eckhorn et al., 1990), adapted
slightly from Karvonen’s PCNN model (Karvonen, 2004;
Ranganath and Kuntimad, 1994), and is implemented by
applying iteratively the equations
X
Lij ½n ¼ W ijkl Y kl ½n  1 ð1Þ
U ij ½n ¼ S ij ð1 þ bLij ½nÞ ð2Þ

1; U ij ½n > T ij ½n
Y ij ½n ¼ ð3Þ
0; otherwise
The indexes i and j refer to the pixel location in the
image, indexes k and l refer to the dislocation in a symmet-
ric neighborhood around a pixel, and n refers to the time
(number of iteration). Lij[n] is linking from a neighborhood
of the pixel at location (i, j), Uij[n] is internal activity at
location (i, j), which is dependent on the signal value Sij
(pixel value) at (i, j) and linking value. b is the PCNN link-
ing parameter, and Yij[n] is the output value of the PCNN
element at (i, j). Tij[n] is a threshold value. We use a set of
fixed threshold values, Tm (m = 1, . . . , M) determined by
water region area method mentioned above.
If Yij[n] is 1 at location (i, j) at n = t, we say that the
PCNN element at the location (i, j) fires at t. The firing
due to the primary input Sij is called the natural firing.
The second type of firing, which occurs mainly due to the
neighborhood firing at the previous iteration, we call the
excitatory firing, or secondary firing.
Starting with the biggest threshold TM, object whose
mean gray value is larger than TM will be picked out at
the first iteration. We keep the threshold TM fixed during
the following iterations until no firing happens. At a certain
threshold, the iteration times differ from image to image
and a suitable amount of iterations in practice is 20–70.
After the first iteration loop, both the natural firing pixels
and excitatory firing pixels are collected, which is the first
level PCNN segmented objects with the largest gray value.
Then the second level objects can be got by the same algo-
rithm using threshold TM1. Repeating this progress until
all thresholds are processed, we will get M + 1 levels of
objects with different intensities at last.
In this PCNN algorithm, we are using the neighborhood
with the radius r = 1.5 (i.e., a usual 3 · 3 neighborhood;
for details see Kuntimad and Ranganath’s paper, with
the linking relative to the inverse of the squared distance
from the midpixel and normalized to one (Karvonen,
2004). Considering those pixels whose intensities are smal-
ler than peak point Pm ought not to be captured at Tm even
if they have the largest linking value 1, so in the iteration
loop at Tm, the value of bm is chosen to be
Tm
bm ¼ 1 ð4Þ
Pm
Because the least peak point P1 may be 0 (showed as Fig.
5(f)), we choose the value of b1 to be 0.1–0.3 at this situation.
 M. Li et al. / Pattern Recognition Letters 27 (2006) 1948–1956
The segmented results based on our algorithm are
shown in Section 4.
3. Multi-sensor image fusion scheme using PCNN
3.1. The basic algorithm
Fig. 2 shows a schematic diagram of the proposed image
fusion scheme based on PCNN. The algorithm consists of
following steps:
Step 1. Segment the registered source images into different
regions by PCNN (details in Section 2).
Step 2. Combine the segmentation results with source
images to determine the region each pixel belongs
to. Suppose the multi-sensor source images are
S1, S2, . . . , Sn, and denote the ith region of image
Sm (m = 1, 2, . . . , n) by DBi(Sm).
Step 3. From each image region DBi(Sm), extract two
features, salience (SA) and visibility (VI), which
reflect its clarity. Denote the feature vector for
DBi(Sm) by (SADBi(Sm), VIDBi(Sm)) (details in Sec-
tion 3.2).
Step 4. Determine the fusion weight of DBi(Sm) accord-
ing to (SADBi(Sm), VIDBi(Sm)). Denote the fusion
weight for DBi(Sm) by WDBi(Sm) (details in Section
3.3).
Step 5. Get the fusion image F by DBi(Sm) and WDBi(Sm).
As we observed before, an important pre-processing step
in image fusion is image registration, which ensures that
the information from each source is referring to the same
physical structures in the real-world. Throughout this
paper, it will be assumed that all sources images have been
registered. Comprehensive reviews on image registration
PCNN
Source image
of sensor 1
…
…
Source image
…
of sensor n
PCNN
Fig. 2. Schematic diagram of the proposed fusion method.
Fig. 3. The multi-focus source images and the reference one: (a) focus on the clock, (b) focus on the student and (c) reference image.
1951
can be found in Barbara, Z. and Flusser’s paper (Barbara
and Flusser, 2003).
3.2. Feature extraction
In this paper, we extract two features, salience and
visibility, from each image region to represent its clarity.
3.2.1. Salience (SA)
The salience proposed in this paper is mainly with a view
to the difference between object and its neighboring back-
ground. The following examples illustrate the necessity
and rationality of SA.
(1) Multi-focus image fusion applications
We consider situations where two or more objects in the
scene are at different distances from the camera. As is typi-
cal with most inexpensive cameras, the image thus obtained
will not be in focus everywhere, i.e., if one object is in focus,
another one will be out of focus. However, by fusing images
with different focus points, an image that is in focus every-
where can be obtained (Seales and Dutta, 1996).
As showed in Fig. 3, each image contains multiple
objects at different distances from the camera. The focus
in Fig. 3(a) is on the clock, while that in Fig. 3(b) is on
the student. We artificially produce an all in focus reference
image showed as Fig. 3(c).
It can be seen from Fig. 3 that Fig. 3(c) is a combination
of the good-focus clock Fig. 3(a) and the good focus student
Fig. 3(b). The left part of Fig. 3(a) is clearer than Fig. 3(b) in
visual effect. Here, we take the clarity of character ‘‘8’’ in the
clock as the symbol of the whole region it belongs to. Take
its edge into account, the intensity difference between the
object and its background, like character ‘‘8’’ and clock
dial, is more obvious in Fig. 3(a) than in Fig. 3(b).
Extract region
Segmentation features
result
Combine Fusion
results image
Segmentation
Extract region
result
features
1952 M. Li et al. / Pattern Recognition Letters 27 (2006) 1948–1956

(2) Different sensor image fusion applications Table 1

Concealed weapon detection is an increasingly impor- SA and VI values for a special region in Figs. 3 and 4
tant topic in the general area of law enforcement, and Selected region SA in SA in VI in VI in
image fusion has been identified as a key technology to Fig. (a) Fig. (b) Fig. (a) Fig. (b)
enable progress on this topic. Fig. 4 shows a pair of visual Fig. 3 clock character 53.002 13.668 0.036175 0.007291
and 94 GHz millimeter-wave (MMW) images. The visual ‘‘8’’ region
Fig. 4 gun region on 0.021392 6.335 16.127 0.024928
image Fig. 4(a) provides the outline and the appearance the right person’s chest
of the people while the MMW image Fig. 4(b) shows the
existence of a gun. From the fused image Fig. 4(c), there
is considerable evidence to suspect that the person on the 3.2.2. Visibility (VI)
right has a concealed gun beneath his clothes. This fused This feature is inspired from the human visual system,
image may be very helpful to police officer, for example, Huang and Li have given its definition (Huang et al.,
who must respond promptly. 1999; Li et al., 2002). We rectify the formula as following:
Let us pay attention to the gun on the right person’s
1 X  1 a jf ðx; yÞ  mk j
chest in Fig. 4(c). For the gun region is much clearly in VIðDBi Þ ¼ P  ð6Þ
Fig. 4(b), while there is nearly no difference between the DBi ðx;yÞ2DB mk mk
i

‘object’ and ‘background’ at the same region in Fig. 4(a), P

where DBi is the total number of pixels in the region
so to this region, it must have much weight in Fig. 4(b)
DBi, mk is the mean gray value of the image region, and
than the same area in Fig. 4(a) during the fusion process.
a is a visual constant ranging from 0.6 to 0.7.
From above analysis, we can see that the clarity of a
We should mention that VI has much significance in
region edge in different source images directly determines
multi-focus image fusion than different sensor image fusion
its fusion weight in corresponding image. The clearer the
from extensive experiments. This is also showed in Table 1.
region edge is, the bigger the fusion weight is. The key
Compared with VI, SA proposed in this paper has much
problem is how to determine a region edge’s clarity.
more significance in different sensor image fusion appli-
Assume DBi(Sm) is the ith region of image Sm, p is
cations.
any point of DBi’s edge and N8(p) is its eight neighboring
pixels, we expand the edge by N8(p) to get an edge
3.3. Fusion weight of the region
band about three pixels width. Take the edge band into
account, we can work out the mean gray value of those
Considering the different contributions to the fusion
pixels belong to DBi and those not. The salience of DBi
result between various source images, we use region
is defined as
fusion weight to denote these. Assume WDBi(Sm) is the
  fusion weight of region DBi(Sm), we use visibility and
1 X m
1X n 
  salience of the region as two main factors to determine it.
SA ¼  Sj  Sk  ð5Þ
m j¼1 n k¼1  Based on extensive experiments, we define it as
W ðDBi Þ ¼ eVIðDBi ÞW VI þ eSAðDBi Þ ð7Þ
where Sj {j = 1, . . . , m} and Sk {k = 1, . . . , n} are the gray
values of pixels in the edge band, while the former belongs where WVI is a visibility constant.
to DBi and the latter does not.
In order to demonstrate the effectiveness of formula (5), 4. Experimental results
we choose the region of ‘‘8’’ in Fig. 3, and calculate the val-
ues of SA in Fig. 3(a) and (b) separately. In Fig. 4, the gun 4.1. Setup for quantitative evaluation
region is selected, the values of SA in Fig. 4(a) and (b) are
also calculated. From Table 1 we can see that the values of To evaluate the performance of the proposed fusion
SA coincide with visual effect very well. The clearer the method, three objective evaluative criteria are used. They
region is, the bigger the value SA is. are the root mean squared error (RMSE)

Fig. 4. The different sensor images and fused one: (a) visual image, (b) MMW image and (c) fused image.
 M. Li et al. / Pattern Recognition Letters 27 (2006) 1948–1956 1953
vffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
u and peak signal noise ratio (PSNR) (Eskicioglu and Fisher,
u 1 X M 1 X N 1
RMSE ¼ t ½Rði; jÞ  F ði; jÞ2 1995; Xydaes and Petrovic, 2000)
MN i¼0 j¼0
L1
PSNR ¼ 20  log10 :
between the reference image R and the fused image F, RMSE
cross-entropy (CEN) Here, M, N are the dimensions of the images, R(i, j) and
X
L1 F(i, j) are the pixel values at position (i, j) of R and F, L
pðgÞ
CENðP : QÞ ¼ pðgÞlog2 is the number of gray levels, the gray level distributions
qðgÞ
g¼0 of two images are P = {p(0), p(1), . . . , p(g), . . . , p(L  1)}

Fig. 5. The ‘‘Pepsi’’ source images (256 level, size of 512 · 512) and fusion results: (a) focus on the Pepsi can, (b) focus on the testing card, (c) reference
image (all in focus), (d) fused image using LPT (level = 3), (e) fused image using DWT (db8, level = 3); (f) water regions and thresholds of (a), (g) PCNN
segmentation result of (a), (h) water regions and thresholds of (b), (i) PCNN segmentation result of (b), (j) fused image using PCNN, (k) difference between
DWT fused image (e) and reference image (c), (l) difference between PCNN fused image (j) and reference image (c).
1954 M. Li et al. / Pattern Recognition Letters 27 (2006) 1948–1956

and Q = {q(0), q(1), . . . , q(g), . . . , q(L  1)}. Notice that cian pyramid transform based (LPT), discrete wavelet
CEN measures the difference of corresponding pixels in transform based (DWT), are applied to fuse the same
two images, and a small CEN means getting more informa- images. The wavelet basis ‘‘db8’’, together with a decompo-
tion from the source images through the fusion method. sition level of 3, is used. Similar to Zhang and Blum’s
method (Zhang and Blum, 1999), we employ a region-
4.2. Comparison with MSD-based fusion schemes based activity measurement for the active level of the
decomposed wavelet coefficients, a maximum selection
For comparison purpose, besides the fusion scheme pro- rule for coefficient combination, together with a window-
posed in this paper, another two fusion algorithms, Lapla- based (5 · 5) consistency verification scheme, the value of
a is 0.7.
Table 2
Performance of the different fusion methods on processing Fig. 5
4.3. Subjective evaluation
Objective criteria LPT DWT This method
RMSE 6.1505 3.9775 2.7531 To evaluate the performance of the proposed fusion
CEN 0.0226 0.0239 0.0092 method, extensive experiments with multi-focus image
PSNR 32.3527 36.1385 39.3342
fusion and different sensor image fusion have been per-

Fig. 6. The multi-focus source images (256 level, size of 640 · 480) and fusion results: (a) focus on the clock, (b) focus on the student, (c) reference image
(all in focus) (d) fused image using LPT (level = 3), (e) fused image using DWT (db8, level = 3), (f) water regions and thresholds of (a), (g) PCNN
segmentation result of (a), (h) water regions and thresholds of (b), (i) PCNN segmentation result of (b), (j) fused image using PCNN, (k) difference between
DWT fused image (e) and reference image (c), (l) difference between PCNN fused image (j) and reference image (c).
 M. Li et al. / Pattern Recognition Letters 27 (2006) 1948–1956
formed. Here, we give three experiments (part of the source
images are provided by Lehigh University).
The first experiment is performed on one set of perfectly
registered multi-focus source images. As showed in Fig. 5,
each image contains multiple objects at different distances
from the camera. The focus in Fig. 5(a) is on the Pepsi
can, while that in Fig. 5(b) is on the card. We artificially
produce an all in focus reference image showed as
Fig. 5(c). The resultant fused images are shown in
Fig. 5(d)–(j). A clearer comparison can be made by exam-
ining the differences between the fused and reference
images (showed in Fig. 5(k) and Fig. 5(l)). Quantitative
comparisons of their performance are shown in Table 2.
From Table 2, we observe that different fusion methods
appear to provide different image fusion performance. The
proposed scheme provides best performance and outper-
forms the discrete wavelet transform approach in terms
of the smaller value of RMSE and CEN, together with
the larger PSNR quality. It can be seen from Fig. 5(l) that
the fused image produced by PCNN is basically a combina-
tion of the good-focus Pepsi can and the good-focus card.
The objective evaluation results coincide with the visual
effect very well.
Table 3
Performance of the different fusion methods on processing Fig. 6
Objective criteria LPT DWT This method
RMSE 7.1126 4.6622 3.5002
CEN 0.3642 0.2273 0.3329
PSNR 31.0903 34.7590 37.2489
Fig. 7. The medical source images (256 level, size of 256 · 256) and fusion results: (a) CT image, (b) MRI image, (c) fused image using DWT (db8,
level = 3), (d) PCNN segmentation result of (a), (e) PCNN segmentation result of (b), (f) fused image using PCNN.
1955
The second experiment is performed on one set of mis-
registration multi-focus source images in Fig. 6. The focus
in Fig. 6(a) is on the clock, while that in Fig. 6(b) is on the
student. Notice that there is a slight movement of the stu-
dent’s head in Fig. 6(a) and Fig. 6(b). We artificially pro-
duce an all in focus reference image showed as Fig. 6(c).
The resultant fused images are shown in Fig. 6(d)–(j).
Again, a clearer comparison can be made by examining
the differences between the fused and reference images
(showed in Fig. 6(k) and (l). Quantitative comparisons of
their performance are shown in Table 3.
In comparison, the proposed scheme outperforms the
DWT-based approach at multi-sensor source images mis-
registration situation. From Table 3, we can see the fusion
method presented in this paper is superior to MSD-based
method in terms of the smaller value of RMSE, together
with the larger PSNR quality. Because there is slight move-
ment of the student’s head in Fig. 6, we can see this causes
little differences between the fused image by PCNN and the
good-focus student (Fig. 6(l)), while a lot of differences
between the DWT-based fused image (Fig. 6(k)). More-
over, we can infer that the use of region information for
the combination of the approximation images as well as
for the detail images improves the fusion effect consider-
ably. The objective evaluation results coincide with the
visual effect very well.
The third experiment is performed on source images
come from different types of sensors. Fig. 7(a) is computed
tomography (CT) image and Fig. 7(b) is magnetic reso-
nance image (MRI) image. The resultant fused images
are shown in Fig. 7(c)–(f).
1956 M. Li et al. / Pattern Recognition Letters 27 (2006) 1948–1956
From Fig. 7, we can see that the proposed method can
preserve more useful information and the distinct extent
of the fusion image is greatly improved compared with
MSD based fusion methods. So the proposed fusion
scheme also shows significant improvement over the
MSD-based method in the applications of source images
come from different types of sensors.
5. Conclusion
This paper describes an application of pulse-coupled
neural network to multi-sensor image fusion problem
based on the use of image regions, which combines aspects
of feature and pixel-level fusion. In order to determine
PCNN parameters adaptively, this paper also brings for-
ward an adaptive segmentation algorithm based on a
modified PCNN with the multi-thresholds determined by
water region area method.
Lots of experiments on evaluating the fusion per-
formance have been made and the results show that the
proposed method has extensive application scope. It
outperforms the MSD based fusion approaches, both in
visual effect and objective evaluation criteria. Furthermore,
the proposed approach has such advantages that the fusion
process becomes more robust and avoids some of the well-
known problems in pixel-level fusion such as blurring
effects and high sensitivity to noise and mis-registration.
The implementation is computationally simple and realiz-
able in real-time.
In addition, this paper expands the application range of
PCNN.
Acknowledgements
The authors would like to thank the anonymous review-
ers for their constructive comments on an earlier version of
this paper, and also to thank Prof. Rick S. Blum for pro-
viding the multi-focus images.
References
Aboul, E.H., Jafar, M.A., 2004. Digital mammogram segmentation
algorithm using pulse-coupled neural networks. In: The 3rd Internat.
Conf. on Image and Graphics, Hong Kong, China, pp. 92–95.
Barbara, Z., Flusser, J., 2003. Image registration methods: A survey.
Image Vision Comput. 21 (11), 977–1000.
Bi, Y.W., Qiu, T.SH., 2005. An adaptive image segmentation method
based on a simplified PCNN. Acta Electron. Sinica 33 (4), 647–650.
Burt, P.J., Adelson, E.H., 1983. The Laplacian pyramid as a compact
image code. IEEE Trans. Comm. 31 (4), 532–540.
Eckhorn, R., Reitboeck, H.J., Arndt, M., Dicke, P., 1990. Feature linking
via synchronization among distributed assemblies: Simulations of
results from cat visual cortex. Neural Comput. 2, 293–307.
Ekblad, U., Kinser, J.M., Atmer, J., Zetterlund, N., 2004. The intersecting
cortical model in image processing. Nucl. Instrum. Methods Phys. Res.
A 525, 392–396.
Eskicioglu, A.M., Fisher, P.S., 1995. Image quality measures and their
performance. IEEE Trans. Comm. 43 (12), 2959–2963.
Gu, X.D., Yu, D.H., 2005. Image shadow removal using pulse coupled
neural network. IEEE Trans. Neural Networks 3, 692–695.
Gu, X.D., Guo, Sh.D, Yu, D.H., 2002. A new approach for automated
image segmentation based on unit-linking PCNN. In: The First
Internat. Conf. on Machine Learning and Cybernetics, Beijing, China,
pp. 175–178.
Hall, D.L., Llinas, J., 1997. An introduction to multisensor data fusion.
Proc. IEEE 85 (1), 6–23.
Huang, J.W., Shi, Y.Q., Dai, X.H., 1999. A segmentation-based image
coding algorithm using the features of human vision system. J. Image
Graphics 4 (5), 400–404.
Johnson, J.L., Taylor, J.R., Anderson, M., 1999. Pulse coupled neural
network shadow compensation. In: SPIE Conf. on Applications
and Science of Computational Intelligence, Orlando, Florida,
pp. 452–456.
Karvonen, J.A., 2004. Baltic sea ice SAR segmentation and classification
using modified pulse-coupled neural networks. IEEE Trans. Geosci.
Remote Sens. 42 (7), 1566–1574.
Kuntimad, G., Ranganath, H.S., 1999. Perfect image segmentation using
pulse coupled neural networks. IEEE Trans. Neural Networks 10 (3),
591–598.
Li, Sh.T., Kwok, J.T., Wang, Y.N., 2002. Multifocus image fusion using
artificial neural networks. Pattern Recognition Lett. 23 (8), 985–997.
Liu, Q., Ma, Y.D., Qian, Zh.B., 2005. Automated image segmentation
using improved PCNN model based on cross-entropy. J. Image
Graphics 10 (5), 579–584.
Ma, Y.D., Dai, R.L., Li, L., 2002. Automated image segmentation using
pulse coupled neural networks and image’s entropy. J. China Inst.
Comm. 23 (1), 46–51.
Ma, Y.D., Shi, F, Li L., 2003. Gaussian noise filter based on PCNN. In:
IEEE Internat. Conf. on Neural Networks & Signal Processing,
Nanjing, China, pp. 149–151.
Mallat, S.G., 1989. Multifrequency channel decomposition of images and
wavelet models. IEEE Trans. Acoust. Speech Signal Process. 37 (12),
2091–2110.
Matsopoulos, G.K., Marshall, S., 1995. Application of morphological
pyramids: Fusion of MR and CT phantoms. J. Visual Comm. Image
Represent. 6 (2), 196–207.
Petrovic, V.S., Xydeas, C.S., 2004. Gradient-based multiresolution image
fusion. IEEE Trans. Image Process. 13 (2), 228–237.
Piella, G., 2003. A general framework for multiresolution image fusion:
From pixels to regions. Inform. Fusion 4 (4), 259–280.
Ranganath, H.S., Kuntimad, G., 1994. Image segmentation using pulse
coupled neural networks. In: IEEE Internat. Conf. on Image Process-
ing, pp. 1285–1290.
Seales, W.B., Dutta, S., 1996. Everywhere-in-focus image fusion using
controllable cameras. Proc. SPIE 2905, 227–234.
Thhomas, L., 1999. Inherent features of wavelets and pulse coupled neural
networks. IEEE Trans. Neural Networks 10 (3), 607–614.
Toet, A., Van Ruyven, L.J., Valeton, J.M., 1989. Merging thermal and
visual images by a contrast pyramid. Opt. Eng. 28 (7), 789–792.
Xydaes, C., Petrovic, V., 2000. Objective image fusion performance
measure. Electron. Lett. 36, 308–309.
Zhang, Z., Blum, R.S., 1999. A categorization of multiscale-decomposi-
tion-based image fusion schemes with a performance study for a digital
camera application. Proc. IEEE 87 (8), 1315–1326.