A Novel Multi-frame Super Resolution algorithm for

Surveillance Camera Image Reconstruction

Aunsia Khan, Muhammad Aamir Khan,Faisal Obaid, Sultanullah Jadoon,
Mudassar Ali Khan and Misba Sikandar
{aunsiakhan,mudaser,aamir,sultan,misbasikandar}@uoh.edu.pk, faisalobaid74@gmail.com
Department of Information Technology, University of Haripur, Khyber Pakhtunkhwa, Pakistan
Abstract This paper gives a loom towards the
growing spatial resolution necessary to beat the limitations
of the imaging technology in surveillance and security
disciplines. It has been observed that metropolis cities
worldwide invest huge sum of money in surveillance
camera system but few are closely observing the benefits
and the costs of those investments and to measure the
overall impact of surveillance cameras on crime rates. The
low resolution coupled with poor quality optics is not be
enough to identify the subject of interest in crowd, from a
distance, in bad weather and any other limiting factor. In
this paper we have introduced multi-frame superresolution technique that does not require explicit motion
estimation and will be useful for producing imagery
evidence that the police might reasonably accept as proof
of someone's identity. Mostly the research is done in this
area by taking a SR image and then after adding their own
noise patterns where as our algorithm are working on
actual LR images of surveillance camera and getting a SR
image while removing the original blur and noise. Our
algorithm requires the training set of Low resolution (LR)
images from a still camera to produce High resolution
(HR) image data and enhances it using anisotropic
Diffusion and De-noising. In the image based
representations, this technique of super resolution provides
a great step towards resolution independence. The
application of this method was successfully demonstrated
for the restoration from a short low resolution set of
images into a super resolved image. This super resolution
algorithm works best when the Diffusion is applied and
noise reduction filters are applied.
KeywordsSuper Resolution, Surveillance camera
images, Multi frame super resolution, Gray images, Low
Resolution images, Training Set.

I. INTRODUCTION
Image-based models intended for computer graphics
are resolution dependent, they cannot be zoomed a lot
beyond the pixel resolution on which they were sampled
at without causing degradation of overall quality.
The process of obtaining a high resolution image by
increasing the number of pixels is known as Super
resolution. Vande walle et al defined higher resolution
image as an image constructed by up- sampling and
interpolation having more resolution and large number
of pixels than actual low resolution image but have no

978-1-4799-7620-1/15/$31.00 2015 IEEE

extra details. Capel's proposed definition of resolution

enhancement depicts that the low resolution image
should result in an improved detailed contents in super
resolution image by restoring the high frequency
contents. Which causes the overall increase in the
number of pixels [10].
Super Resolution image reconstruction (SRIR) is one of
the capable techniques of digital image processing in
which challenges are made by reconstructing a High
Resolution imagery by blending some of the information
enclosed inside countable numbers of under sampled
low-resolution images of a particular scene. This
technique engage many groups by filtering out the blur
and noise and Up sampling the under sampled images.
In contrast with diverse image enhancement techniques,
SRIR technique filter outs the distortions and increasing
their spatial resolution can enhance the worth of such
low resolution and under sampled images [1]. Two
approaches are used to perform super-resolution
enhancement: that are single frame and multi-frame
enhancement. Since low resolution images inherently
contain less information than higher resolution images,
the process of constructing high resolution images form
an input low resolution images requires the missing high
resolution data to be calculated.
The target of SRIR method is to recuperate an image of
high resolution from low resolution images from a
surveillance camera provided as an input. Two families
of methods are used to classify the SR i.e. i) Classical
multiple image super resolution and ii) Super resolution
Based on Example. In the classical method (e.g., [12, 5,
8]) a group of images of a stationary scene having lowresolution are taken. All of the low resolution images
inflict a set of linear constraints on the unidentified high
resolution intensity values. If considerable number of
low-resolution images are taken from the source, then
high resolution image can easily be reconstructed.

In this paper we focus on multi-frame image

reconstruction in which LR images are captured
simultaneously by a surveillance camera. As one of
constrain with image reconstruction is the missing data.
So in the multi-frame super-resolution reconstruction,
Observed images are registered to a specific high
resolution reference frame in order to formulate the
fusion problem.

high resolution image which is similar to the image

which is trying to be enhanced. The multi-frame
approach on the other hand uses the data which is
contained in the multiple input frames to interpolate the
HR image pixel values, this allows the multi-frame
approach to be applied to any input image, even if a
similar image was not used during the design of the
algorithm.

The major processing steps that characterize the Super

resolution consist of these following steps:
1. Low Resolution images acquisition: LR images are
acquired from the same scene with non-integer (in terms
of inter-pixel distances) geometric displacements
between any two of the images.
2. Motion compensation and Image registration: For the
reference to high resolution desirable grid, the sub-pixel
geometric transformation is estimated for every source
image.
3. High resolution image reconstruction: constructing a
high resolution output image as a solution to the actual
problem.
4. Noise Reduction: Reducing blur and noise from the
final image.

At first, as we don't have high resolution data so this

task may seem impossible. However, we know that the
edges of the low resolution images should be kept sharp
in the next resultant resolution level. It is extremely
difficult to estimate the arbitrary motion without
guarantee of the estimators performance in the image of
real world scenes. The Super resolution performance can
have disastrous implications if the motion estimation is
incorrect.

One would expect that the affluence of the real- world

images seem difficult to capture analytically. This
encourage an approach based on learning: that is; in a
training set, The pixel details of different images at a
low resolution are learnt; then the details of final image
are predicted based on the learned relationships. For the
past several years [2, 6], researchers have been
exploring this approach for enlarging images. To
motivate why this approach should work at all, we noted
that a set of completely random variables have much
high variability as compared to the set of image pixels.
Many research studies have been conducted on
mammalian visual system for their early processing
stages [4, 15]. We take advantage of these regularities of
registration in our algorithms, as we use multiple images
of one scene, with a still camera, to create an image
having information more visible then LR images.
Correct high resolution images cannot be generated
without a specific training data set.

We aim of generating visually plausible details of

images e.g. sharp edges and credible texture. The set of
data is then used to create a new image having increased
number of pixels and, on the same time, increase in
resultant image size.
This paper is to be considered of having the following
steps:
1. Set of low resolution images captured via still
Surveillance camera.
2. Mapping pixels of each image of low resolution
to a grid of high resolution.

Fig. 1. LR images to HR image

3.
4.

Anisotropic diffusion applied to the HR image.

Noise Reduction using Noise Reducing Filter.
III. IMPLEMENTATION

II. ALGORITHM
CONCEPTUAL IMAGE RESAMPLING
After reading over the different approaches to perform
super-resolution, we have decided to focus on multiframe. We selected the multi frame approach because
we believe it will produce more robust results above
wider collection of input images. One of the main
limitations of the single-frame approach is that the
process will only be effective if the database contains a

A. Generation of Training set

We started the generation of training set by collecting
low resolution images and arranged the images that we
process all in way for up gradation that we planned. On
average, low resolution image with original number of
pixels is created by sub-sampling and blurring. But we
collected a cluster of same low resolution images to
reconstruct an up-graded image.

Fig. 2. Single LR image

B. Mapping LR pixels to HR grid

We apply an initial step of mapping tthe pixels of lowresolution image. We needed to store tthe different pixel
values of different low resolution imagees to form the
true image of high resolution. We supppose a HR image
having grids as the number of pixels in the LR image.
The size of each HR grid be the same size as that of the
number of LR images used. We put alll the pixels of the
LR images in the HR grid respectivelyy, as shown in the
figure

Fig. 4. Grid of HR image having LR images Pixels

Fig. 3. LR image to HR grid

We want to store each pixel that corrresponds to every
possible pixel of the low-resolution image; here the
pixels are taken from set of 25 imagees, respectively as
shown in fig. (3) whereby interlacing the four images,
we get that the desired resolution is oobtained, and thus
perfect reconstruction is guaranteed. Due to limited
camera resolution, we sample using aan insufficient 2D
grid. We restricted ourselves to reasonnable information
of the image, there is still a large numbber of information
that we have to stock up, and therefore we formulated
generally applicable training sets. Fig. (4) shows the HR
mapping of pixels.
image which is created by the said m
As per our believe, for the prediction oof very fine details
present in the original image, the com
mponents of low
resolution image with highest resoluttion play a very
important role.

Fig. 5. HR image

C. Image Enhancement using Diffusiion

We applied anisotropic diffusion on
n the LR image and
image obtained as HR image so thaat we dont have the
mosaic pattern in our resultant imaage. The anisotropic
diffusion helped in favoring the high
h contrast edge over
the low contrast edge. We believe that the local image
contrast affects the overall relationship between the low
and high resolution image pixels, and essentially the
anisotropic diffusion helps in reetaining the overall
contrast resultant image of high resolution.

Fig. 6. Diffused LR image

Fig. 9. De-Noised Diffused HR im

mage

IV. RESULTS

Fig. 7. Diffused HR imagee

D. Noise Reduction
We filter out the Diffused HR image byy a noise reducing
filter in the frequency domain, to reducce the noise in the
resultant image. There was no noise iin the LR image;
the overall degradation is caused onlyy because of less
number of pixels.

Fig. (9) shows results of the algorith

hm we applied to our
LR images. Training set was taken
t
from a still
surveillance camera. A significantly clear and acceptable
resulting image was generated that was highly
acceptable than the original LR images, the image
details and sharp edges were presserved. The fig. (5)
depicts that our low level training seet is not sufficient to
produce an acceptable HR accurate image data. The
algorithm interprets that resultant HR
R image is enhanced
more via diffusion. The fig. (7) show
ws that the diffusion
upgrades the HR image. In the mean
n while the de-noised
diffused HR image fig. (9) using thee filter shown in fig.
(8) showed sharper and better resu
ult than the diffused
LR image in fig. (6). Nonetheless; the
t algorithm proved
its best to depict the low-resolution image, resulting in a
high resolution detailed image, fig. (9).
There is a surprising regularity acro
oss images, such that
a training set made from the imagees of low resolution
can be used to invent missing detaills in many other LR
images. While training set of the images to be processed
works best, a training set of generic images can handle a
very broad class of inputs. Finally, the algorithm works
best when the Diffusion is applied and noise reduction
filters are applied.
TABLE I

Fig. 8. Applied Filter in Frequency D

Domain

Sampling rate

10 samp
ples/sec

Image array dimensions

721 X 542

No. of Samples

25

Zoom in Factor

17 X

Table I enumerate the sampling rate of dataa set, the total number of
samples taken for generating the output an
nd the zoom in factor by
which pictures were zoomed for comparison.

V. CONCLUSIONS

TABLE II
Figure

Algorithm applied

Result

No

Poor

Wiener filter

Deficient

Bicubic interpolation

Good

Our algorithm

Best

Table II enlists the application and result of different algorithms when

applied on the same input and the sub figures shows the zoomed
images by factor of 17.

In our this paper, we have introduced multi-frame

super-resolution technique that does not require explicit
motion estimation and will be useful for producing
something that the police might reasonably accept as
proof of someone's identity. Our algorithm was inspired
by the algorithms [2, 4]. It is unclear in many studies
that appear to show success whether surveillance
cameras had a positive impact in combination with
improved lighting, or whether the improved lighting
might accomplish the positive outcome on its own.
Studies vary on the degree to which they take
confounding factors into account. The computational
burden being a difficult task prevents the iteration
scheme to produce improved results. Since there are
many parameters to be considered, it seems that a fair
comparison with other super resolution algorithms
cannot be made, resulting in no comparisons presented
in this paper. Experiments with a number of sets of
parameters suggest that our algorithm yields results
which are quite comparable if not superior to some of
the algorithms [17, 16] especially when the training set
images are of very low resolution.
We have constructed a super-resolution algorithm that
is based on training set [7], and introduced a faster,
simpler, and, we believe, better super-resolution
algorithm of single pass. The algorithm requires the
training set of LR images from a still surveillance
camera to produce HR image data and enhances it using
anisotropic Diffusion and De-noising. In the image
based representations, this technique of super resolution
provides a great step towards resolution independence
The application of this method was successfully
demonstrated for the restoration from a short low
resolution set of images into a super resolved image.
RECOMMENDATIONS

Fig. 10. Comparing our SR with the wiener filter and Bicubic
Interpolation constraints. Note that the bicubic interpolation, when
applied to LR patches (a) , results in a high-resolution image (c)
which is sharper and cleaner than the wiener filtered image (b), but is
not able to recover the fine and plausible details. In contrast, our highresolution image (d) produced these fine details while maintaining the
edges and line sharpness.

The approach we described here possibly will be very

useful in image processing techniques and graphics
applications (see [6, 13]). A recent Home Office study
found that 80 per cent of images from CCTV cameras
were of such poor quality that they were worthless as
evidence. For the enlargement of images, removal of
noise, estimation of shapes of 3-D surfaces and for
attacking the other imaging application, Training sets
can be built and used. We assumed the gray images that
were registered and were spatially static for our
restoration of images in multi frame super resolution.
Reviewing previous work on image re-sampling theory
draw from the literature of computer, we formed a
relationship between contrasting research areas. In our
research, we demonstrated that Resampling theory of

images can be handy to imaging systems that are

aligned. The supremacy of this approach is fabricated in
the larger training sets, proper noise reduction models,
allowing rendering models and rich prior probabilities
and, allowing efficient scene inference. This algorithm
of super resolution provided the state of the art results
when applied. Furthermore, this work can be expanded
by using colored images of low resolution as a data set
hence producing comparable results.
ACKNOWLEDGMENT
This paper is made possible through the help and
support from everyone, including: parents, teachers, and
friends. The product of this research paper would not be
possible without all of them.
REFERENCES
[1] Vivek Bannore, 2009, Volume 195/2009 "IterativeInterpolation Super-Resolution Image Reconstruction,Studies"
in Computational Intelligence.
[2] W. T. Freeman and E. C. Pasztor MA, 1999. MIT Press"
Learning to estimate scenes from image".
[3] S. Baker and T. Kanade. 2000. " Limits on superresolution and how to break them". In IEEE Conf.
Computer Vision and Pattern Recognition,.
[4] D. J. Field 6:559601, 1994. "What is the goal of sensory
coding". Neural Computation.
[5] Dallas County, Texas: "Repeat offender shows
surveillance cameras' limitations," by Tanya Eiserer, The
Dallas Morning News.
[6] W. T. Freeman, E. C. Pasztor, and O. T. Carmichael.
40(1):2547, 2000 " Learning lowlevel vision". Intl. J.
Computer Vision.
[7] J. S. DeBonet and P. Viola 1998. "Texture recognition
using a non-parametric multi-scale statistical model". In Proc.
IEEE Computer Vision and Pattern Recognition.
[8] S. Farsiu, M. Robinson, M. Elad, and P. Milanfar T-IP,
(10), 2004. "Fast and robust multiframe super resolution".
[9] William T. Freeman, Thouis R. Jones, and Egon C.
Pasztor_ MERL (A MITSUBISHI ELECTRIC RESEARCH
LABORATORY)
http://www.merl.com
Example-Based
Super-Resolution
[10]
Denis
Chekhlov
http://www.cs.bris.ac.uk/~chekhlov/sr.html
[11]M. Irani and S. Peleg CVGIP, (3), 1991. "Improving
resolution by image registration".

[12]H. Sidenbladh and M. J. Black. Learning image

statistics for Bayesian tracking. In Intl. Conf. on
Computer Vision, 2001.
[13] D. J. Heeger and J. R. Bergen ACM SIGGRAPH, pages
229{236, 1995. "Pyramid-based texture analysis/synthesi". In
Computer Graphics Proceedings, Annual Conference Series.
[14]E. P. Simoncelli. Statistical models for images:
Compression, restoration and synthesis. In 31st Asilomar
Conf. on Sig., Sys. and Computers, Pacific Grove, CA,
1997.
[15] 8. S. Farsiu, D. Robinson, M. Elad, and P. Milanfar
August 2004, Volume 14, no 2, pp. 47-57, " Advances and
Challenges in Super-Resolution", International Journal of
Imaging Systems and Technology.
[16]S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, Fast and
Robust Multi-Frame Super-Resolution, IEEE Transactions on
Image Processing, October 2004, Volume 13, no 10, pp. 13271344.
[17] E. P. Simoncelli Paci c Grove, CA, 1997. "Statistical
models for images: Compression, restoration and synthesis". In
31st Asilomar Conf. on Sig., Sys. and Computers,.
[18] S. C. Zhu and D. Mumford. Prior learning and Gibbs
reaction-dilusion. IEEE Pattern Analysis and Machine
Intelligence, 19(11), 1997.