52

Int. J. Mechatronics and Automation, Vol. 4, No. 1, 2014

Augmented reality-based RehaBio system for

shoulder rehabilitation
Yee Mon Aung* and Adel Al-Jumaily
School of Electrical, Mechanical and Mechatronic Systems,
Faculty of Engineering,
University of Technology,
Sydney, 15 Broadway, Ultimo, NSW, 2007, Australia
E-mail: yee.m.aung@student.uts.edu.au
E-mail: adel.al-jumaily@uts.edu.au
*Corresponding author
Abstract: This paper presents the development of rehabilitation with biofeedback (RehaBio)
system for upper-limb rehabilitation that can be used to restore the upper-limb lost functions of
patients who suffer from traumatic brain injury (TBI), spinal cord injury (SCI) or cerebrovascular
accident (CVA), which generally result in paralysis on one side of the body. The system aims to
close the gap in the requirements of one-to-one attention between physiotherapist and patient, to
replace boring traditional upper-limb rehabilitation exercises and to reduce high healthcare cost.
RehaBio is made up of three major modules: database module, rehabilitation exercise module and
biofeedback simulation module. Database module provides the information of the patients and
their rehabilitation progress while rehabilitation exercise module provides with effective and
motivated exercises based on augmented reality approach. Biofeedback simulation module in
RehaBio serves two purposes: from physiotherapist point of view, it provides the tracking of
biofeedback information of patients muscle performance and activities. From the patients point
of view, it serves as a visual reflection of current activated muscles that create as an additional
motivation during training process. The effectiveness of the RehaBio system was evaluated by
performing the experiments and provided with promising results.
Keywords: rehabilitation; augmented reality; biofeedback; mechatronics.
Reference to this paper should be made as follows: Aung, Y.M. and Al-Jumaily, A. (2014)
Augmented reality-based RehaBio system for shoulder rehabilitation, Int. J. Mechatronics and
Automation, Vol. 4, No. 1, pp.5262.
Biographical notes: Yee Mon Aung is currently a PhD student in the School of Electrical,
Mechanical and Mechatronics Systems at University of Technology Sydney. She received her BE
in Mechanical Engineering and MSc in Biomedical Engineering both from Nanyang
Technological University, Singapore in 2008 and 2010, respectively. Her current research
interests include upper limb rehabilitation robotics and augmented reality-based rehabilitation
system. She is an IEEE student member.
Adel Al-Jumaily holds a PhD in Electrical Engineering (AI). He is an Associate Professor in
Faculty of Engineering and Information Technology, University of Technology, Sydney (UTS),
Australia. His research interests include biomechatronics, artificial intelligent systems, healthcare
and biomedical engineering.
This paper is a revised and expanded version of a paper entitled Shoulder rehabilitation with
biofeedback simulation presented at IEEE ICMA 2012 Conference, Chengdu, China,
58 August 2012.

Introduction

The human nervous system is made up of specialised cell

networks called neurons. It controls the physiological
activities and movements of the human body. It consists of
two main parts: central nervous system (CNS) and
peripheral nervous system (PNS). CNS consists of brain and
spinal cord which are responsible for receiving and
interpreting signals from PNS. PNS is made up of nerves
that connect with limbs and organs. The nerves can be either

Copyright 2014 Inderscience Enterprises Ltd.

afferent or efferent sensory nerves. Afferent sensory nerves

convey the information towards the CNS while efferent
motor nerves carry the signals from CNS to the cells
(Noback et al., 2005). If there is some disturbance between
these connections, the physiological activities and
movements will be severely affected. There are several
conditions that can disturb these connections such as
traumatic brain injury (TBI), spinal cord injury (SCI) or
cerebrovascular accident (CVA). TBI occurs when the

Augmented reality-based RehaBio system for shoulder rehabilitation

external force is strongly impacted to the brain such as
vehicular accident or violence. SCI refers to the injury
caused to the spinal cord due to trauma such as vehicular
accidents, falls or violence like in TBI. CVA normally
refers to stroke which occurs where the flow of the blood
is disturbed due to the blockage or haemorrhage in blood
vessel that supply to the brain. As a result, the brain cannot
get enough oxygen through the blood and lost the brain
functions that control the physiological activities of human
body. All of these injuries or accident will result in the loss
of receiving or interpreting signals from PNS and this will
prevent the communications between CNS and PNS. This
loss of communication will result in the failure of
controlling in human physiological activities or limb
movements depending on which part of the brain or nerves
is affected.
In general, TBI, SCI and CVA patients suffer from
paralysis either in half of the body or the whole body.
Paralysis can be defined as loss of muscle functionality for
one or more muscles. This loss will cause the patients to
prevent from performing daily life activities such as eating,
drinking, dressing and these cause to reduce the patients
quality of life. To recover from such loss, rehabilitation
exercises are essential and conducted in healthcare sector as
there is a proof that the intense use of active movements in
repetitive tasks and task-orientated activities will improve
motor skills and muscular strength (Riener et al., 2005).
Generally, rehabilitation requires a one-to-one guidance
between physiotherapist and patient to perform traditional
exercises that leads easily to boredom and unmotivating for
patients (Pedretti, 1985). This requirement effects directly
from the shortage of physiotherapists in the healthcare
sector and increases the labour cost. Furthermore, the cost
of such rehabilitation in healthcare sectors is very
expensive. In Australia, it costs about 13 billion per annum
in healthcare sectors (Waimr, 2012). To overcome such
problems, rehabilitation with biofeedback (RehaBio) system
is developed with low cost and motivational approach.
RehaBio is a shoulder rehabilitation system that is
integrated with augmented reality (AR)-based rehabilitation
exercises and biofeedback system to retrain the plasticity of
the brain for fast recovery. The detailed development of
theRehaBio system will be presented in the latter part of this
paper.
The rest of the paper is constructed as follows:
Section 2 describes the related work. Section 3 explains the
development of RehaBio system that consists of database
module, AR-based rehabilitation module and biofeedback
simulation module. Section 4 depicts the experimental
results of RehaBio system. Finally, in Section 5, the
conclusion and perspective for future work are presented.

Related work

2.1 Upper-limb rehabilitation system

There are numerous developments for upper limb
rehabilitation system. These include robotic approach,

53

virtual reality (VR) approach and AR approach. Generally,

robotic approach aims to rehabilitate severe impairments
and is classified as an expensive approach due to its
hardware development cost. This approach can be either
end-effector or fixed-based type as in Krebs et al. (2007)
and Masia et al. (2007) or exoskeleton or wearable
rehabilitation robot type (Ying and Agrawal, 2012; Perry et
al., 2007). The selection of robot type depends on the
rehabilitation goal such as range of motions, kinematics
criteria of human limb, ambulatory and portability
(Hasegawa et al., 2008); while the exoskeleton provides a
wider range of movements compared to that of end-effector
robot. As for VR or AR systems approach, they aim to
rehabilitate the minor impairment or later stage of
rehabilitation training with low cost as these approaches do
not require heavy hardware development. Moreover,
VR or AR approach provides better encouragement and
motivation because these approaches employ games like
exercise as a training platform. The VR approach is the
computer-simulated environments approach that can
simulate virtual objects created on the computer display.
However, VR approach requires additional communication
device to communicate between real world and virtual
world. Although integrating with VR in rehabilitation has
provided with positive result (Da Silva Cameiro et al.,
2011), the requirement of bulky communication device
leads very inconvenient for the paralysed patients. Recent
developments of such VR-based rehabilitation system can
be found in Harley et al. (2011), Kaluarachchi et al. (2011)
and Song et al. (2011). Some of the robotic systems were
also integrated with VR environment to motivate the
patients for fast recovery (Guidali et al., 2011). To
overcome the inconvenient problem of VR approach, AR
approach was introduced for better interactive training
environment in recent years. AR is the combination of real
and virtual environment where virtual objects lay on top of
the real environment on the display screen. Virtual objects
are developed with computer programme while real world
video image is imported as a background via webcam. The
communication between these two environments can be
completed with just a light weight marker such as colour or
pattern marker. Such AR-based rehabilitation systems can
be found in Dinevan et al. (2011), Alamri et al. (2010) and
Burke et al. (2010). However, to authors knowledge there
are no AR-based rehabilitation systems that are integrated
with biofeedback system to monitor the training
performance and improvement of patients condition.
Therefore, our group have been developing the
rehabilitation systems to improve the upper-limb
movements with biofeedback in AR environment (Aung and
Al-Jumaily, 2011, 2012a, 2012b).

2.2 EMG biofeedback

Biofeedback is the learning of changes in human
physiological activity by means of brainwaves via
electroencephalograph (EEG), muscle activity via
electromyograph
(EMG),
heart
function
via
electrocardiograph (ECG), breathing via pneumograph and

54

Y.M. Aung and A. Al-Jumaily

skin temperature via feedback thermometer with thermistor

to improve health and performance (Association for Applied
Psychophysiology and Biofeedback, 2012). Among these
available methods, EMG biofeedback is mostly employed in
the development of physical rehabilitation system. EMG is
the electrical signals associated with the contraction of a
muscle. These signals provides the basic information of
muscle activities such as contracting of the muscles, rate of
tension build-up, fatigue, and reflex activities. Therefore, it
becomes very important information in many clinical and
biomedical applications. The EMG signals can be obtained
via two ways: surface EMG where sensor is attached to the
user skin and intramuscular EMG where needle electrode is
inserted via skin into muscle tissue. In both ways, the
detection of the abnormalities and activation level of human
movement is done by analysing the shape, size, and
frequency of the motor unit action potentials generated by
muscle fibres. The normal amplitude of the raw sEMG
signal, that is used here, can range from 010 mV with
usable energy of 0500 Hz. However, it is difficult to utilise
the information of raw sEMG signal. Therefore, useful
information from the sEMG signal is extracted from raw
signal by several feature extraction methods. George
Whatmore (Whatmore and Kohli, 1974) stated that in the
use of surface electromyography (sEMG) as a biofeedback
tool highlighted how problems could emerge when an
individual muscle performing efforts were too high or too
low. The sEMG biofeedback techniques fall roughly into
three clinical entities: down-training, up-training and
coordination training. The down-training techniques are
used to facilitate a reduction in muscles which are
overactive. Up-training is to learn how to turn on a
particular muscle or muscle group. It is commonly
conducted when working with inhibited muscles, or muscles
that have been weakened due to disuse or injury such as
TBI, SCI or CVA. Coordination training is considered an
advanced level of training and usually follows successful
up- or down-training. This will teach the patient on how to
obtain the correct balance of agonists or antagonists. There
are quite a number of EMG-based applications such as
controlling of prosthesis (Ho et al., 2011), controlling of
exoskeletons (Kiguchi and Hayashi, 2012), detection of user
intended movement (Qichuan et al., 2011) and controlling
of virtual models (Sha et al., 2010) for up-training.
Although the proof of concept for those developments had
achieved, clinical trial for most of the developments are yet
to be done.

RehaBio system

In this paper, RehaBio system is proposed and it is made up

of three modules:
1

the database module where patient profile and training

information are stored

rehabilitation exercise module for upper limb

rehabilitation exercises that were developed based on
AR technologies

biofeedback simulation module to monitor and

visualise the patients muscle performance and
activities.

Details of each module will be explained in following

section.

3.1 Database module

RehaBio database module is developed in MATLAB
platform. This module is made up of two interfaces namely
physiotherapist interface and patient interface as shown in
Figure 1. As physiotherapist interface is only allowed to be
manipulated by physiotherapists, staff login interface will
display upon access as shown in Figure 2. When staff ID is
entered wrongly, the warning box will display and request
the physiotherapist to re-enter the ID again. After successful
log-in with staff ID, physiotherapist will be able to access
the patients profile, performance result and patient profile
as depicted in Figure 3. Patients profile allows the
physiotherapists to add new patients together with their
particulars such as EMG threshold level for individual
muscles and training session or delete the old patients that
successfully finished their trainings and for those who had
already discharged. The patient profile interface can be
found in Figure 4. As for performance result,
physiotherapist will be able to select the desired patient
from the drop down menu of patients names. Once the
patient name is selected, the related EMG threshold value
will display under last EMG threshold value as illustrated in
Figure 5. Based on this value, physiotherapist will define
the new threshold level to increase the patients muscles
performance. The new EMG threshold value entered by
physiotherapist in this interface, will update the EMG
threshold value in biofeedback module for muscle
simulation display and also be updated in the patients
profile.
Figure 1

Main GUI of RehaBio system (see online version

for colours)

Augmented reality-based RehaBio system for shoulder rehabilitation

Figure 2

Staff login GUI for physiotherapist (see online version

for colours)

patients to manipulate. Four rehabilitation exercises are

developed under rehabilitation exercises module and more
exercises can be added later. Biofeedback simulation
module provides patients muscle performance in real-time.
Figure 6

Figure 3

55

Patient interface (see online version for colours)

Physiotherapist interface (see online version

for colours)

3.2 Rehabilitation exercise module

Figure 4

Patients profile interface

Figure 5

Performance result interface (see online version

for colours)

Under patient interface as shown in Figure 6, rehabilitation

exercises module and biofeedback simulation module are
available. This interface permits both physiotherapists and

This module is based on AR approach and implemented in

Adobe Flash Professional platform where ActionScript API
is utilised to capture the video scenes, display the virtual
objects, search for suspected marker and detect collision
between real marker and virtual objects. It only requires PC
with cheap webcam (or laptop with built in webcam) and
colour marker to complete this module. As AR is the
combination of real world and virtual world, the virtual
objects in this module will lay on top of the video image
which is fed via webcam. The virtual objects will then
communicate with real colour marker via collision
detection. In additional to that, setting up of game display
screen, selecting of virtual objects colour and display of
graphical user interface (GUI) are carefully considered
according to the game design principle mentioned by Salen
and Zimmerman (2003). RehaBio system provides four
rehabilitation exercises: ping-pong rehabilitation (PPR),
balloon collection rehabilitation (BCR), transfer object
rehabilitation (TOR) and feeding animal rehabilitation
(FAR) as depicted in Figure 7. These exercises aim to
promote the shoulder range of motions and strengthen the
associated muscles used. They are also aiming to replace the
boring traditional rehabilitation exercises which are
normally conducted in rehabilitation sectors (Pedretti,
1985). To increase the patients motivation, the system
provides the scoring, audio feedback and countdown-timer.
There are two countdown timers: one is just before the
training and another is during the training which serves as a
training duration. The duration of the training for individual
exercise is different from one another and so does for the
score. The procedures for all the rehabilitation exercises are
the same as follows:
1

Physiotherapist or patient is expected to choose the

appropriate rehabilitation exercise from AR
rehabilitation exercises GUI.

Subsequently, the intro page of the exercise will display

and let the user to choose either left or right hand to be

56

Y.M. Aung and A. Al-Jumaily

trained. On the same page, How to Play instruction is
presented to the user.

After choosing left or right arm to be trained, the

system will ask for the permission to access the
webcam.

Once access to the webcam is granted, the webcam will

feed the video image on the display screen and the
instruction Please Click on the Marker will be
displayed on top of that video image.

When the user clicked on the colour marker which is

worn by the users thumb, five seconds of countdown
timer will start. This countdown will allow the user to
be ready to start the rehabilitation exercise.

When the five seconds are up, the chosen rehabilitation

exercise will display and the patient will need to start
the rehabilitation exercise.

Once the exercise is started, the second timer will start

counting down which is the duration of the training. At
the same time the score will increase and audio
feedback will activate if the user movement is correct.
There will be no penalty for wrong movement.

When the second timer is up, the summary page will

display with times up notification and total score of
the user achievement.

Figure 7

AR rehabilitation exercise interface (see online version

for colours)

arm movement by playing PPR exercise. Rehabilitation

purpose of PPR exercise is to provide wider range of
movement in shoulder flexion (SF) as well as strengthen the
associated muscle.

3.2.2 BCR exercise

The aim of BCR exercise is to catch a dropping balloon at a
time from the display screen with colour marker attached to
the users thumb and place into the box which is located at
the centre of the screen as depicted in Figure 8(b). Once the
colour marker is intersected with the dropping balloon, the
user is required to bring that balloon to the box and place it.
As the balloons are dropping randomly on the screen, user is
expected to move his/her arm in various directions to catch
the dropping balloon. Therefore, this exercise will provide
wide range of shoulder movements such as flexion,
abduction, horizontal adduction (HAD) and horizontal
abduction (HAB). By performing this exercise, the muscles
involved in those movements will be strengthened.

3.2.3 TOR exercise

The TOR exercise is developed with five different solid
virtual objects located at the bottom row of the screen. At
the top row of the screen, the same shape but hollow virtual
objects are located with different order. The user is expected
to pick up the blinking solid object that indicates which
object the user is to pick up from bottom row of the screen.
Then, the user is expected to place on the correct hollow
shape at the top row of the screen as depicted in Figure 8(c).
By collecting and placing the virtual objects, users SF,
abduction and HAD motion will be achieved and associated
muscles will be strengthened.
Figure 8

3.2.1 PPR exercise

PPR exercise is designed to maintain the bouncing ball
within the display screen by moving users arm up and
down as shown in Figure 8(a). The ball moves within the
display screen with the limitation of upper and lower
boundary. One side of the display screen border is limited
by moving block controlled by the system according to the
ball movement direction to restrict the ball from out of
display. The other side of the stage is to be controlled by the
marker attached to users thumb and prevent the ball from
out of the display. Therefore, user will perform the specific

AR-based rehabilitation exercises (a) PPR (b) BCR

(c) TOR and (d) FAR (see online version for colours)

(a)

(b)

(c)

(d)

Augmented reality-based RehaBio system for shoulder rehabilitation

3.2.4 FAR exercise

The objective of the FAR exercise is similar to TOR
exercise as illustrated in Figure 8(d). The food plate is
placed all over the display screen and user is expected to
collect the food which is located at the bottom row of the
screen. The food which is indicated with red colour arrow
need to be picked by the user with colour marker and placed
in the food plate which is indicated with green colour arrow.
The different height of the placement of food plate will train
the shoulder range of motion which is normally conducted
based on traditional way by therapists. By collecting the
food and placing into the food plate as instructed by the
system will provide the user to move their shoulder in
flexion, abduction and HAD motions and the associated
muscles will be strengthened.

3.3 Biofeedback simulation module

In biofeedback simulation module, BioGraph Infiniti system
from Thought Technology (2012) is used. The pre-amplified
sEMG sensor permitting input range of 02,000 V and
channel bandwidth of 10 Hz1 kHz were used to record the
patients sEMG signal and extracted with time domain
feature extraction method named root mean square (RMS)
by calculating the amplitude. Although there are many other
methods such as integration EMG (iEMG), autoregressive
coefficients (AR) and wavelet coefficients are available, the
RMS is employed as it provides good real-time information.
The RMS value can be calculated with equation (1).
EMGrms =

1
N

Plot data: All the processed data will then stream and
plot in real-time by this sub-module.

Arm muscle simulation: This sub-module is responsible

for simulation according to the EMG amplitude. The
activation of this module is based on predefined EMG
threshold value (amplitude) which is defined by
physiotherapist according to the patients muscle
performance from performance result GUI (Figure 5).
Real-time simulation of muscle is represented by
changing the colour of the muscles.

Stop connections: This sub-module will release the

handle to ActiveX objects and then stop the connection
from all the channels and save the sEMG data of
muscles performance.

The screen shot of biofeedback simulation module is

portrayed in Figure 9. In this figure, there are two muscle
windows for biofeedback simulation and four line graphs
representing anterior deltoid, posterior deltoid, biceps
brachii and pectoralis major muscle signals. The recorded
sEMG signals are employed as the input data for simulation
of the muscle in muscle windows. Simulation will activate
when recorded sEMG signals are above the predefined
threshold value. When the recorded sEMG signals are above
predefined activation level, the muscle colour will change
so that patient and therapist can observe the current active
muscle during specific exercise movement.
Figure 9

sEMG(i)

57

Biofeedback simulation module (see online version

for colours)

(1)

i =1

where sEMG(i) is the amplitude of the signal in ith sampling,

N is the number of samples. The sampling rate of 256 Hz
with the frequency of 128Hz was used in this work. Four
sEMG signals are recorded for shoulder movements while
performing the rehabilitation exercise. This module is
developed in MATLAB platform to read the live data of
EMG, extract EMG features by RMS, display the real-time
signals and perform the simulation of active muscles in
real-time. When the user presses the start button in
biofeedback simulation module, the recording of the data
will begin and display the recorded data in real-time. The
stop button will terminate the recording and displaying of
data and simulation. Six sub-modules were developed to
complete this module as follows:
1

Initialisation: This sub-module is responsible to

initialise the activeX object and return the handle to
global workspace.

Setups: This sub-module is in-charge of setting up the

encoder and channels properties, in this work one
encoder and four channels are utilised.

Read data: This sub-module provides the collecting of

all the four sEMG data and process.

RehaBio experiments

The experiments were carried out to present the

effectiveness of the RehaBio system. There are two phases
of experiments. The first phase was carried out to verify the
significant contribution of the muscles for shoulder

58

Y.M. Aung and A. Al-Jumaily

movements to increase shoulder range of motions. The

second phase of the experiment is to determine the
effectiveness of developed database module, AR-based
exercise module and biofeedback simulation module.

4.1 Experimental setup

Experiments were performed with healthy subjects. The
signals were received from anterior deltoid, posterior
deltoid, biceps brachii and pectoralis major muscles of
upper left arm. Before starting the experiment, hardware set
up was performed. Standard personal computer (PC),
webcam and Biograph Infiniti system were utilised in this
experiment. Four MyoScan sensors from Thought
Technology were connected to four channels of FlexComp
Infiniti encoder. The encoder was then connected to
TT-USB interface unit via fibre-optic cable and the interface
connects to a USB port of the computer. Four MyoScan
sensors were attached to the four arm muscles as shown in
Figure 10. The raw sEMG signals of anterior deltoid,
posterior deltoid, biceps brachii and pectoralis major muscle
are detected via channel A, channel B, channel C and
channel D of FlexComp Infiniti encoder, respectively.
These raw signals were then amplified, filtered and
extracted by means of RMS method. To conduct the first
phase of experiment, user was asked to perform four types
of arm movements: SF, shoulder abduction (SAB), HAD
and HAB as shown in Figures 11(a) to 11(d). In the second
phase of experiment, user was expected to follow the
instructions from user graphical interface (GUI) that display
on the PC to verify the effectiveness of three modules.
Figure 10

sEMG electrode positions (see online version

for colours)

were significantly contracted as depicted in Figure 12(d).

Therefore, to summarise the relationship between motion
and muscle contribution, anterior deltoid and biceps brachii
muscles are trained during SF motion while SAB and HAB
motions train anterior deltoid, posterior deltoid and biceps
brachii muscles. HAD motion will strengthen the anterior
deltoid, biceps brachii and pectoralis major muscles.
Although same muscles are contributed to achieve different
movements, the degrees of sEMG amplitude are different
according to the different type of shoulder movements.
Figure 11

Types of upper limb movements (a) shoulder

flexion-SF (b) SAB (c) HAD (d) HAB (see online
version for colours)

(a)

(b)

(c)

(d)

In the second phase of experiment, the evaluations and

results of manipulation of database, AR-based rehabilitation
exercises and biofeedback simulations are reported.
Figure 12

Experimental results of (a) SF (b) SAB (c) HAD

(d) HAB motions (see online version for colours)

4.2 Result and discussion

From the first phase of experiment, the sEMG signals from
each arm movements were recorded. According to the
recorded EMG data during SF motion, anterior deltoid and
biceps brachii muscles are contracted more than the rest of
the muscles as shown in Figure12(a). This shows that
performing SF motion will increase the strength of anterior
deltoid and biceps brachii muscles mostly. In the
performance of SAB motion, it was found that anterior
deltoid, posterior deltoid and biceps brachii muscles were
contracted more than pectoralis major muscles as portrayed
in Figure 12(b). In Figure 12(c), anterior deltoid, biceps
brachii and pectoralis major are the most contracted to
achieve HAD motion. HAB motion was attained due to
anterior deltoid, posterior deltoid and biceps brachii muscles

(a)

(b)

Augmented reality-based RehaBio system for shoulder rehabilitation

Figure 12

Experimental results of (a) SF (b) SAB (c) HAD

(d) HAB motions (continued) (see online version
for colours)

Figure 14

59
Survey results of (a) PPR exercise (b) BCR exercise
(c) TOR exercise (d) FAR exercise (see online
version for colours)

(c)

(a)

(d)
Figure13 TOR rehabilitation exercise with biofeedback
simulation module (see online version for colours)
(b)

The user data under RehaBio database can be easily saved

and deleted via patient profile GUI while physiotherapists
are able to retrieve the patients performance information
via performance result. Rehabilitation exercises under
patient interface GUI allows the choices of AR-based
rehabilitation exercises easily recommended by the
physiotherapist. The START and STOP button from
biofeedback
simulation
interface
will
let
the
physiotherapists and users to easily manipulate for reading
and
stopping
of
sEMG
data, displaying
of
real-time sEMG amplitude data and muscle simulation as
shown in Figure 13.

(c)

(d)

60

Y.M. Aung and A. Al-Jumaily

Figure 15

Experimental results of (a) PPR exercise (b) BCR

exercise (c) TOR exercise and (d) FAR exercise
(see online version for colours)

(a)

(b)

To evaluate the effectiveness of AR-based rehabilitation

exercises with biofeedback simulation module, it was
tested with healthy subject and the results are depicted in
Figures 15(a) to 15(d). According to the results, during PPR
exercise, anterior deltoid muscle is the most contracted as
the exercise trains only for SF motion. Therefore, anterior
deltoid will be trained to strengthen in PPR exercise. In
BCR exercise, all of the four muscles are trained to increase
its strength as this exercise required to move the shoulder in
SF, SAB, HAD and HAB movements. In the result of TOR
exercise, all the EMG muscle activation levels are higher
than that of BCR or PPR exercises since the user is required
to put more effort to accomplish the TOR rehabilitation
exercise. In this exercise, anterior deltoid and biceps brachii
muscles are mainly trained for their strength. In the results
of FAR exercise, anterior deltoid, biceps brachii and
pectoralis major muscles are significantly trained.
According to the results, it proved that all of four muscles
are trained to increase their strength by performing
developed rehabilitation exercises. Therefore, the developed
exercises are not only to increase the range of motion at the
shoulder joint, but also train effectively in muscle strength.

(c)

(d)

The evaluation of developed rehabilitation exercises were

conducted by answering the questionnaire, which describes
in Appendix, at the end of each exercise. The score ranking
from 1 to 4 where 1 refers to strongly disagree and 4
refers to strongly agree is required to answer for each
question which is stated in the questionnaire. Ten healthy
subjects participated in the evaluation process and the
feedback is as shown in Figures 14(a) to 14(d). According
to the result, participants found the exercises are interesting
and they enjoyed the exercises without any major
discomfort. Almost all participants felt that the tracking of
the colour marker is good as well as the exercise is very
easy to understand with motivated feedbacks.

Conclusions and future work

In this paper, the development of RehaBio system for upper

limb rehabilitation system is presented. The system is made
up of database module, AR-based rehabilitation exercise
module and biofeedback simulation module. At this stage of
developing the rehabilitation system, our system is
successfully integrated with AR-based rehabilitation
exercises and biofeedback simulation that provide the
physiotherapists and users to monitor the trained muscles
performance not only in real-time signal displays but also in
real-time muscle simulation display while performing
exercises. In addition, the users are able to interact with the
rehabilitation exercises, is user-friendly and the
development of database for patients provides a complete
stand alone RehaBio system. The results from the
experiment proved that RehaBio system provides effective
rehabilitation system for shoulder joint movements with
motivational approach. As far as future work is concerned,
we will improve our development of database and
interaction under the guidance of physiotherapists
requirements and to carry out the clinical trial at Port
Kembla Rehabilitation Hospital.

Acknowledgements
The authors would like to acknowledge the physiotherapists
from Port Kembla Rehabilitation Hospital for their advice
and feedback.

Augmented reality-based RehaBio system for shoulder rehabilitation

References
Alamri, A., Jongeun, C. and El Saddik, A. (2010) AR-REHAB: an
augmented reality framework for poststroke-patient
rehabilitation, IEEE Transactions on Instrumentation and
Measurement, Vol. 59, No. 10, pp.25542563.
Association for Applied Psychophysiology and Biofeedback
(2012) What is Biofeedback? [online]
http://www.aapb.org/i4a/pages/index.cfm?pageid=1 (accessed
10 August 2013).
Aung, Y.M. and Al-Jumaily, A. (2011) Rehabilitation exercise
with real-time muscle simulation based EMG and AR, 11th
International Conference on Hybrid Intelligent Systems (HIS),
pp.641646.
Aung, Y.M. and Al-Jumaily, A. (2012a) AR based upper limb
rehabilitation system, 4th IEEE RAS & EMBS International
Conference on Biomedical Robotics and Biomechatronics
(BioRob), pp.213218.
Aung, Y.M. and Al-Jumaily, A. (2012b) Shoulder rehabilitation
with biofeedback simulation, 2012 International Conference
on Mechatronics and Automation (ICMA), pp.974979.
Burke, J.W., Mcneill, M.D.J., Charles, D.K., Morrow, P.J.,
Crosbie, J.H. and Mcdonough, S.M. (2010) Augmented
reality games for upper-limb stroke rehabilitation, 2nd
International Conference on Games and Virtual Worlds for
Serious Applications (VS-GAMES), pp.7578.
Da Silva Cameiro, M., Bermdez I Badia, S., Duarte, E. and
Verschure, P.F.M.J. (2011) Virtual reality based
rehabilitation speeds up functional recovery of the upper
extremities after stroke: a randomized controlled pilot study
in the acute phase of stroke using the rehabilitation gaming
system, Restorative Neurology and Neuroscience, Vol. 29,
No. 5, pp.287298.
Dinevan, A., Aung, Y.M. and Al-Jumaily, A. (2011) Human
computer interactive system for fast recovery based stroke
rehabilitation, 11th International Conference on Hybrid
Intelligent Systems (HIS), pp.647652.
Guidali, M., Duschau-Wicke, A., Broggi, S.,
Klamroth-Marganska, V., Nef, T. and Riener, R. (2011) A
robotic system to train activities of daily living in a virtual
environment, Medical and Biological Engineering and
Computing, Vol. 49, No. 10, pp.12131223.
Harley, L., Robertson, S., Gandy, M., Harbert, S. and Britton, D.
(2011) The Design of an Interactive Stroke Rehabilitation
Gaming System Human-Computer Interaction. Users and
Applications, Vol. 6764, Springer, Berlin/Heidelberg.
Hasegawa, Y., Mikami, Y., Watanabe, K., Firouzimehr, Z. and
Sankai, Y. (2008) Wearable handling support system for
paralyzed patient, IEEE/RSJ International Conference on
Intelligent Robots and Systems, IROS 2008, pp.741746.
Ho, N.S.K., Tong, K.Y., Hu, X.L., Fung, K.L., Wei, X.J.,
Rong, W. and Susanto, E.A. (2011) An EMG-driven
exoskeleton hand robotic training device on chronic stroke
subjects: Task training system for stroke rehabilitation,
IEEE International Conference on Rehabilitation Robotics
(ICORR), pp.15.

61

Kaluarachchi, C., Aung, Y.M. and Al-Jumaily, A. (2011) Virtual

games based self rehabilitation for home therapy system,
11th International Conference on Hybrid Intelligent Systems
(HIS), pp.653657.
Kiguchi, K. and Hayashi, Y. (2012) An EMG-based control for an
upper-limb power-assist exoskeleton robot, IEEE
Transactions on Systems, Man, and Cybernetics, Part B:
Cybernetics, Vol. 42, No. 4, pp.10641071.
Krebs, H.I., Volpe, B.T., Williams, D., Celestino, J., Charles, S.K.,
Lynch, D. and Hogan, N. (2007) Robot-aided
neurorehabilitation: a robot for wrist rehabilitation, IEEE
Transactions on Neural Systems and Rehabilitation
Engineering, Vol. 15, No. 3, pp.327335.
Masia, L., Krebs, H.I., Cappa, P. and Hogan, N. (2007) Design
and characterization of hand module for whole-arm
rehabilitation following stroke, IEEE/ASME Transactions on
Mechatronics, Vol. 12, No. 4, pp.399407.
Noback, C.R., Strominger, N.L., Demarest, R.J. and
Ruggiero, D.A. (2005) The Human Nervous System Structure
and Function, 6th ed., New York City, USA.
Pedretti, L.W. (1985) Occupational Therapy: Practice Skills for
Physical Dysfunction, 7th ed., Maryland Heights, Missouri,
USA.
Perry, J.C., Rosen, J. and Burns, S. (2007) Upper-limb powered
exoskeleton design, IEEE/ASME Transactions on
Mechatronics, Vol. 12, No. 4, pp.408417.
Qichuan, D., Xingang, Z., Anbin, X. and Jianda, H. (2011) A
novel motion estimate method of human joint with
EMG-driven model, 5th International Conference on
Bioinformatics and Biomedical Engineering, pp.15.
Riener, R., Frey, M., Bernhardt, M., Nef, T. and Colombo, G.
(2005) Human-centered rehabilitation robotics, 9th
International Conference on Rehabilitation Robotics,
pp.319322.
Salen, K. and Zimmerman, E. (2003) Rules of Play: Game Design
Fundamentals, 1st ed., Cambridge, Massachusetts, USA.
Sha, M., Varley, M., Lik-Kwan, S. and Richards, J. (2010) EMG
biofeedback based VR system for hand rotation and grasping
rehabilitation, 14th International Conference Information
Visualisation, pp.479484.
Song, Z., Guo, S. and Fu, Y. (2011) Development of an upper
extremity motor function rehabilitation system and an
assessment system, International Journal of Mechatronics
and Automation, Vol. 1, No. 1, pp.1928.
Thought Technology (2012) BioGraph Infiniti [online]
http://www.thoughttechnology.com/software.htm (accessed
10 August 2013).
Waimr (2012) Neurotrauma Statistics [online]
http://www.nrp.org.au/statistics.html (accessed 10 August
2013).
Whatmore, G. and Kohli, D. (1974) The Physiopathology and
Treatment of Functional Disorders, 1st ed., New York City,
USA.
Ying, M. and Agrawal, S.K. (2012) Design of a cable-driven arm
exoskeleton (CAREX) for neural rehabilitation, IEEE
Transactions on Robotics, Vol. 28, No. 4, pp.922931.

62

Y.M. Aung and A. Al-Jumaily

Appendix

The surveys were conducted as below. The users responded

to the following survey questions according to the scale 1 to
4, where 4 represents strongly agree and 1 represents
strongly disagree.

The present of feedback such as timer and scoring

system are motivating.

Tracking of the colour marker is good.

It can feel the arm muscles fatigue.

I have tried augmented reality games before.

It is comfortable throughout the exercise.

The game is motivated and interested.

The given information and guide are easy to

understand.

The training duration for healthy/stroke user is

appropriate.

10 Please provide other feedbacks and suggestions if any.

It is comfortable to wear the marker.