2018 Ural Symposium on Biomedical Engineering, Radioelectronics and Information Technology (USBEREIT)
Machine Learning in Video Surveillance for Fall
Detection
Lesya Anishchenko
Bauman Moscow State Technical University
Moscow, Russia
anishchenko@rslab.ru
Abstract—The present paper considers the usage of deep
learning and transfers learning techniques in fall detection by
means of surveillance camera data processing. As a dataset,
an open dataset gathered by the Laboratory of Electronics and
Imaging of the National Center for Scientific Research in Chalon-
sur-Saone was used. The architecture of the CNN AlexNet, which
was used as a starting point for the classifier, was adapted to
solve fall detection problem. The proposed method was tested on
a dataset of 30 records containing a single fall episode each.
We achieved Cohen’s kappa of 0.93 and 0.60 for the fall –
non-fall classification for the known and unknown for classifier
surrounding conditions, respectively.
Index Terms—deep learning, machine learning, intelligent
video analysis, fall detection
I. I NTRODUCTION
At present, surveillance cameras are widely used. They are
utilized to identify suspicious persons, as well as persons
in a state of alcohol and drug intoxication, which may be
dangerous to a life and health of others. The developers of the
closed-circuit television system (CCTV systems) offer com-
plexes consisting of IP cameras for the automated detection of
suspicious persons. These systems can be based on biometric
identification (for example, the NeoFace system, smart glasses
R7) [1], [2], as well as on the recognition of emotions or
facial expressions (for example, DeepFace ) [3]–[6]. The main
disadvantage of this type of systems is that information about
the possible trespasser may be absent in the database used for
biometric identification.
It should be noted that in the majority of cases, data from
CCTV cameras are used only for the creating archives of
video records, and the possibilities of intelligent video analysis
of data from remote objects are practically not used. Such
situation is determined in particular by technical difficulties
associated with the use of intelligent video surveillance sys-
tems in practice. For example, classification algorithms are
usually extremely sensitive to lighting conditions [7].
As a specific problem of intelligent video analysis, we can
define the problem of detecting an abnormal situation when
a person is in danger and there are no passers around, who
can help (for example, a person has fallen and can not get
up or call for help). If such situation takes place in the cold
The research was supported by the grant of Russian Foundation for Basic
Research (17-20-03034).
978-1-5386-4946-6/18/$31.00 ©2018 IEEE
season, it is essential to provide first aid as soon as possible
in order to prevent the negative consequences associated with
the hypothermia. Therefore, the up-to-date task is to design
technical means that will detect an emergency situation in an
automated mode and inform the operator if help is needed.
In the present work, the problem of developing such a
system for recognizing human movements through the records
of CCTV cameras and detecting falls has been considered. The
fall of a person can be caused by many factors: loss of balance
due to lack of cerebral blood supply, muscle weakness, etc. In
any case, a situation when a person has fallen and cannot get
up without assistance is dangerous and requires an immediate
response.
The problem of detecting falls using video analysis has been
studied in a large number of works [8], [9], which were based
on analyzing the shape and position of the person in the frame,
gradients in the vertical and horizontal directions, and changes
in images in the time domain.
In the majority of papers on CCTV cameras records analysis
the effectiveness of the fall event classifiers is artificially
overestimated because of the limitations of the datasets used
for testing and training procedure. These limitations may be
described as follows:
• Dataset is usually formed by the data recorded in un-
changed conditions (most often in the laboratory condi-
tions, rather than realistic ones), and for uniform illumi-
nation of the entire analyzed scene.
• One and the same person acts as test subjects.
• Movement artifacts of the ”fall” type are similar in pre-
ceding actions of the subject and her/his relative position
toward the camera at the time of the fall.
• In almost all cases, falls are performed on a cushioning
mat, which most often has a significant color contrast
with respect to the clothing of the subject.
All these factors result in a significant overestimation of the
classifiers presented in papers based on the analysis of datasets
with the above-listed disadvantages.
The purpose of this paper was to evaluate the applicability
of deep learning and transfer learning techniques in automated
detection of falls by analysis of surveillance cameras data
gathered in realistic conditions.
99
 2018 Ural Symposium on Biomedical Engineering, Radioelectronics and Information Technology (USBEREIT)
Fig. 1. CNN architecture.
II. M ETHODS
As it is known, deep learning technique is based on the
use of convolution neural networks (CNN) [10], which are
biologically-inspired variants of multilayer perceptrons. The
main drawback of this technique is that its training requires
a huge amount of data (more than 100,000 examples) for the
successful implementation in practice. As a result, the training
process can last for weeks and months, which is not always
acceptable.
Transfer learning [11] is the technique which allows over-
coming disadvantages of deep learning. It may be described
as following: the researcher selects for solving a new problem
CNN already trained for another similar problem. That allows
transferring knowledge obtained as a result of solving one task
(for example, recognition of images of various animal species)
to solve another (for example, recognition of interior objects
on the image). This approach allows reducing significantly the
time required for CNN training on a new dataset comparing
to training a neural network from scratch (when initializing
the neural network connection with random weights).
In present paper transfer learning procedure was done utiliz-
ing MATLAB2017b. We used pre-trained CNN AlexNet [12]
which is available by installing Neural Network ToolboxTM
Model for AlexNet Network support package. This CNN was
trained by its authors on 1.2 millions of images to classify
1000 different classes and significantly outperformed in 2012
all earlier CNN versions by utilizing more filters per layer that
previously proposed LeNet-5 [13], which was a pioneering
CNN designed by Yann LeCunn in 1998.
In order to use this network to recognize only two classes of
978-1-5386-4946-6/18/$31.00 ©2018 IEEE
images (fall and non-fall classes), we need to make changes
to the AlexNet CNN architecture. Namely, replace the 23d
layer with 2 ”fully connected layers”, since there are two
classes, and replace the 25th layer in accordance with the class
names for ”fall” and ”non-fall”. After making the appropriate
changes, the network architecture has the appearance shown
in Fig. 1. Layers that are different from the original AlexNet
are underlined. After CNN is designed supervised training
procedure should be done by using a dataset with images for
both classes (fall and non-fall).
III. E XPERIMENTAL DATA
In present work, an open database of video recordings
provided by the Laboratory of Electronics and Imaging of the
National Center for Scientific Research in Chalon-sur-Saone
was used [14]. In Fig. 2 and 3 examples of frames extracted
from the video records both corresponding to fall and non-fall
episodes are presented. The merits of this database include the
following factors:
• Video signals are recorded for various environmental
conditions.
• The illumination of the experimental scene is irregular.
The dataset contains records for which contrast of the
human over the background objects is rather low due to
the limits of the camera dynamic range and presence of
regions with high brightness (for example, window area
in Fig. 3).
• 4 different subjects (3 men and 1 woman) participated in
the experiments.
• Falls were performed at different viewing angles, both
from the standing position and from the sitting position.
100
 2018 Ural Symposium on Biomedical Engineering, Radioelectronics and Information Technology (USBEREIT)
Fig. 2. Frame classified as fall episode.
Fig. 3. Frame classified as non fall episode.
• Volunteers were falling both on a specially prepared
cushioning base, and directly on the floor.
30 records from the dataset [14] were used in this paper
to train and to test classifier. All records contain at least one
fall episode. For each record operator visually detected fall
episodes, and provide the number of the frame corresponding
to the start and the end of the fall episode. The duration
of fall episodes was estimated as 22±9 frames (mean±SD).
Taking into account the sampling frequency of 25 frames/s,
the duration of fall artifacts was 0,7±0,3 s.
IV. R ESULTS AND DISCUSSION
Each record from the dataset [14] was divided into separate
frames with known class (fall or non-fall), which were used
as a training data for the classifier. After that, each frame was
re-sampled to make it compatible with CNN AlexNet, namely
277 by 277 pixels. Thereupon the dataset was randomly
divided into training and testing sets in the ratio 80:20%.
As transfer learning technique was used to train the CNN,
the parameters of the input layer were remained unchanged.
That allows using values of connections weights available after
CNN pre-training. Moreover, initial learning rate parameter
was set to be equal 0.001 to make changes of the initial values
of weights small. As it is known, an epoch is a full training
cycle for the whole training dataset. The maximum number of
training epochs parameter was set to be 20, and the batch size
978-1-5386-4946-6/18/$31.00 ©2018 IEEE
Fig. 4. Classification accuracy.
Fig. 5. Classification error.
64. The classifier performance during the training process was
evaluated on validation dataset.
As a result of CNN training, we had a graph of classification
accuracy increasing with each iteration of the CNN training
(Fig. 4), while the classification error for a batch with each
iteration became smaller, as it is shown in Fig. 5.
After completing the training process, the efficiency of the
neural network was evaluated on the test sample. The results
of the classification in a form of a confusion matrix are shown
in Fig. 6.
Since the number of images corresponding to the fall class
are more than ten times smaller than those corresponding to
the not falling class (88 and 1345, respectively) the present
dataset is an unbalances one. Therefore, the accuracy of the
classification (in our case it is 99.23%) overestimates the
efficiency of the trained classifier. In such case, it is also
useful to use the sensitivity, specificity, precision or positive
predictive value and negative predictive value, as well as
measures of inter-rater agreement (Cohen’s kappa [15]), as
additional estimates of the classifier’s performance. For the
trained neural network these parameters are given below:
• Cohen’s kappa 0,93
• Accuracy = 0.99
• Sensitivity = 0.93
• Specificity = 0.99
• Positive predictive value = 0.94
• Negative predictive value = 0.99
We have also used trained classifier for new surrounding
conditions (another 30 records subset of the dataset [12]),
to understand if the CNN will need re-training in case the
101
 2018 Ural Symposium on Biomedical Engineering, Radioelectronics and Information Technology (USBEREIT)
Fig. 6. Confusion matrix (known surrounding conditions).
Fig. 7. Confusion matrix (unknown surrounding conditions).
environmental conditions changed. When testing a trained
classifier on a new dataset the classification results are sig-
nificantly lower. Although the accuracy of the classification is
still high (0.98), Cohen’s kappa is 0.6. The confusion matrix
is shown in Fig. 7, all estimates of classifier performance for
unknown conditions are given below:
• Cohen’s kappa 0,60
• Accuracy = 0.99
• Sensitivity = 0.61
• Specificity = 0.99
• Positive predictive value = 0.61
• Negative predictive value = 0.99
Such differences in the effectiveness of the classification
may be due to the over-training of the classifier, or to the fact
that the training sample was not representative, and enriching
of the dataset will help to solve this issue.
978-1-5386-4946-6/18/$31.00 ©2018 IEEE
Also, it should be noted that the classification was carried
out for each frame regardless of the antecedent and consequent
frames classes. In other words, relative position of the frames
in time was not taken into account. Usage of this information
and the additional heuristics should improve the accuracy of
the final classification, removing, for example, false positives
errors (for example, if the classifier detects a potential episode
of a fall that lasts less than a certain a priori predetermined
threshold).
V. C ONCLUSION
In this paper, we presented the feasibility study of deep
learning and transfer learning techniques usage in fall detec-
tion by means of surveillance camera data processing. The
proposed method allows detecting fall events on a video
records. We achieved Cohen’s kappa of 0.93 and 0.60 for the
fall – non-fall classification for the known and unknown for
classifier surrounding conditions, respectively.
The future activity will consider enriching the dataset.
Moreover, we are planning to improve the classifier per-
formance by using additional heuristics considering relative
positions of the frames in time and knowledge about typical
duration of fall episodes.
The results might be used while creating new noncontact
fall detectors both for outdoors and indoor applications.
R EFERENCES
[1] www.necam.com/docs/?id=6c812b4d-2a12-40ed-9fea-fae81550c7aa
[2] www.osterhoutgroup.com/pub/static/ version1515417478/ frontend/ In-
fortis/ ultimo/ en US/ pdf/ R-7-TechSheet.pdf
[3] Yaniv TaigmanMing YangMarc’Aurelio RanzatoLior Wolf, DeepFace:
closing the gap to human-level performance in face verification, Con-
ference on Computer Vision and Pattern Recognition (CVPR), June 24,
2014.
[4] www.robots.ox.ac.uk/ vgg/publications/2015/Parkhi15/parkhi15.pdf.
[5] Mohammadian, A., Aghaeinia, H., Towhidkhah, F. Video-based facial
expression recognition by removing the style variations in Image Pro-
cessing, IET, 2015, vol. 9, no. 7, pp. 596–603.
[6] Iosifidis A., Tefas A., Pitas, I. Class-specific reference discriminant
analysis with application in human behavior analysis, IEEE Transactions
on Human-Machine Systems, 2015, vol. 45, no. 3, pp. 315–326.
[7] Rice, D., Evaluating camera performance in challenging lighting
situations, 2014. www.sdmmag.com/articles/90525-evaluating-camera-
performance-in-challenging-lighting-situations
[8] Rougier, C., Meunier, J., St-Arnaud, A., Rousseau, J. Fall detection from
human shape and motion history using video surveillance, Proc. 21st Int.
Conf. AINAW, 2007, vol. 2, pp. 875–880.
[9] Lee, T., Mihailidis, A. An intelligent emergency response system:
Preliminary development and testing of automated fall detection, J.
Telemed. Telecare, 2005, vol. 11, no. 4, pp. 194–198.
[10] https://en.wikipedia.org/wiki/Convolutional neural network
[11] https://en.wikipedia.org/wiki/Transfer learning
[12] Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. ”ImageNet
classification with deep convolutional neural networks.” Advances in
neural information processing systems. 2012.
[13] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning
applied to document recognition. Proceedings of the IEEE, november
1998.
[14] Charfi, I., Miteran, J., Dubois, J., Atri, M., Tourki, R., Optimised spatio-
temporal descriptors for real-time fall detection: comparison of SVM
and Adaboost based classification, Journal of Electronic Imaging (JEI),
Vol.22. Issue.4, pp.17, October 2013.
[15] Cohen, J., A coefficient of agreement for nominal scales, Educational
and Psychological Measurement, 1960, 20 (1), pp. 37–46.
102