2019 International Conference on Robotics and Automation (ICRA)
Palais des congres de Montreal, Montreal, Canada, May 20-24, 2019
A Real-Time Interactive Augmented Reality Depth Estimation
Technique for Surgical Robotics
M. Kalia1 N. Navab2 and T. Salcudean3
Abstract— Augmented reality (AR) is a promising technology
where the surgeon can see the medical abnormality in the
context of the patient. It makes the anatomy of interest
visible to the surgeon which otherwise is not visible. It can
result in better surgical precision and therefore, potentially
better surgical outcomes and faster recovery times. Despite
these benefits, the current AR systems suffer from two major
challenges; first, incorrect depth perception and, second, the
lack of suitable evaluation systems. Therefore, in the current
paper we addressed both of these problems. We proposed a
color depth encoding (CDE) technique to estimate the distance
between the tumor and the tissue surface using a surgical
instrument. We mapped the distance between the tumor and
the tissue surface to the blue-red color spectrum. For evaluation
and interaction with our AR technique, we propose to use
a virtual surgical instrument method using the CAD model
of the instrument. The users were asked to reach the judged
distance in the surgical field using the virtual tool. Realistic tool
movement was simulated by collecting the forward kinematics
joint encoder data. The results showed significant improvement
in depth estimation, time for task completion and confidence,
using our CDE technique with and without stereo versus other
two cases, that are, Stereo-No CDE and No Stereo-No CDE.
I. INTRODUCTION
Augmented reality (AR) and Mixed reality (MR) naviga-
tion systems have seen a surge in pre- and intra-operative
minimally invasive surgical paradigm in recent years [1]
,[2]. The AR/MR visualizations facilitate in rendering the
pre-operative medical data directly onto the patient, hence
providing the ease of faster and more precise mental mapping
between the pre-operative medical data and the patient’s
body. This can result in potentially better hand-eye coordi-
nation, better surgical outcomes and faster patient recovery
times.
With these realized benefits, the last decade has witnessed
a remarkable increase in the efficacy and adaptability of
surgical guidance and robotics systems [3], [4], [5], [6],
[7], [8], [9]. Although promising, lack of correct depth
perception has been reported as a major issue in flawless
integration of AR systems in intra-operative frameworks
[10]. In addition to this, finding a robust evaluation technique
for medical AR systems is challenging as different medical
systems have their own unique set of properties. Moreover,
1 M. Kalia is a joint PhD candidate with Faculty of Applied Science,
Electrical and Computer Engineering, University of British Columbia,
Canada and Computer Aided Medical Procedures, Technical University of
Munich, Germany mkalia@ece.ubc.ca
2 N. Navab is with Chair of Computer Aided Medical Procedures,
Technical University of Munich, Germany nassir.navab@tum.de
3 T. Salcudean is with Faculty of Applied Science, Electrical
and Computer Engineering, University of British Columbia, Canada
tims@ece.ubc.ca
978-1-5386-6027-0/19/$31.00 ©2019 IEEE
Authorized licensed use limited to: Technische Hochschule Deggendorf. Downloaded on May 10,2020 at 06:12:35 UTC from IEEE Xplore. Restrictions apply.
the AR visualizations find their place at the very end of the
surgical pipelines in most of the cases, thus accumulating
the systemic and calibration errors from the previous steps
[11]. This further limits the development of good error-free
AR evaluation systems.
Therefore in the current paper we tried to address these
two major problems in medical AR research, i.e. the lack of
correct depth perception and the lack of robust evaluation
methods for surgical AR. We present a depth estimation
technique where we mapped the distance between the tumor
surface and the surgical instrument tip to the red-blue color
spectrum. We called our technique Color Depth Encoding
(CDE). Second, to bypass the very many calibrations steps
to evaluate interactive AR techniques, we propose to use a
virtually rendered surgical instrument using surgical robot’s
forward kinematic joint encoder data for a depth reaching
task. The users were asked to take the virtual tool to the
position of the judged distance.
To our knowledge a virtual tool has not been used for
evaluation of surgical robotics AR/MR techniques. Although
in the current paper we have evaluated the technique on
real radical prostatectomy surgical scenes, we believe that
the evaluation technique can easily be used for evaluating
other interactive AR methods for other surgical robotics
procedures.
Results showed a significant improvement in estimating
depth using our CDE technique when compared to no
CDE. Significant reduction in time to perform the task was
also observed. Furthermore, the subjects gave much higher
confidence scores to our technique in estimating depth.
II. BACKGROUND AND RELATED WORK
The advent of new medical imaging technologies over the
last few decades has resulted in magnanimous amount of
data generation. This data gave the medical professionals
an ability to see the medical abnormality inside the body,
behind many layers of tissues, which otherwise was not
accessible. This possibility to look inside the human body
also gave birth to many new possibilities to use the data
in more meaningful ways. This set the footing for medical
AR realm. Many researchers foresaw the benefits of medical
augmented reality in early 90’s and scientific community
started to witness some early works in the area [12] [13] [14]
[15]. Since then, the medical AR research has progressed
leaps and bounds with the parallel improvements in surgical
robotics, virtual reality and computer vision. However, many
problems have emerged as deterrent in disruption of medical
AR/MR technology in intra-operative surgical paradigms and
8291
two of them are: inaccurate depth perception and dearth of
good evaluation methods [11].
Trivially rendered medical data on top of the surgical
camera feed, even at the correct spatial location doesn’t give
right depth perception. The data seems like pasted over the
real surgical scene and looks like floating due to conflicting
depth cues [16]. The culprit is the cue of occlusion, which
is the strongest among all the depth cues [17]. Thus if
not handled properly the occlusion cue can give the wrong
ordinal depth perception. Interestingly, on the other hand,
partial occlusion can resolve the issue of misleading depth
perception. Partial occlusion helps in displaying the anatomy
of interest with the relevant contextual information and
perceiving it at correct spatial location.
Therefore, partial occlusion has been implemented in
numerous ways by various research groups to have percep-
tually correct visualizations. Bajura et al., 1992, rendered
the ultrasound image in the context of the patient using
a synthetic hole [12]. Although the method conveyed the
correct spatial location of the ultrasound data, it hindered
the natural intuitive surgical view. Similarly, Sielhorst et
al., 2006, used a virtual window to partially occlude the
Computer Tomography (CT) spinal cord data in one of
the visualizations [10]. The visualization can enhance depth
perception by introducing the motion-parallax cue but has
the potential to cause hindrance for the surgeon. On the
other hand, the work by Bichlmeier et al., 2007, can be of
interest in the context of surgical robotics [16]. In the work,
to get the final opacity of each fragment, curvature value,
distance falloff and the angle of incidence was considered.
Furthermore, Lerotic et al., 2006, used a pq-space (slope of
the surface along x and y axes) based visualization method
for minimally invasive surgical robotics [3]. In a more recent
work, the Sobel filter was used to get the edges from the
surgical scene and create a partial occlusion-like effect [18].
Similarly, in our work we implemented partial occlusion by
extracting edges from the camera feed using a Frei Chen edge
filter and rendered it on top of the tumor extracted from the
pre-operative Magnetic Resonance Imaging (MRI) data of a
real patient.
In addition to resolving the wrong ordinal depth informa-
tion, quantitative depth information can also be useful for
a surgeon. However, presentation of such an information in
AR, without distracting a surgeon is crucial. Using color as a
distance cue is interesting because of availability of various
color hues that we can distinguish from one another and
then exploit it to understand the quantitative information like
distance. Winne et al., 2011, has used alternatively colored
guiding lines for distance visualization [19]. Furthermore,
Pseudo Chromadepth is a well known technique for depth
perception in 3D angiography datasets, where the color
gradient corresponds to the distance value at each point
[20], [21]. Although, the technique enhances the relative
distance perception between various points in cluttered an-
giography dataset, it doesn’t give a real-time feedback for
depth perception which is important during a real surgery.
Real-time visual feedback to estimate depth can aid a surgeon
8292
Authorized licensed use limited to: Technische Hochschule Deggendorf. Downloaded on May 10,2020 at 06:12:35 UTC from IEEE Xplore. Restrictions apply.
Fig. 1. (Left) The image showing the surgical instrument tool-tip interacting
with the visualization. (Right) The zoomed-in image elaborating the problem
of mismatch between the actual tool-tip and tool-tip rendered based on the
forward kinematic data from the robot. The difference between the actual
tool tip and kinematics data is marked in orange
in taking a critical decision like resectioning a tumor. One
such technique is proposed by Schoob et al.,2016, where
they used the color gradient to give the real-time visual
feedback for focus positioning in laser microsurgery [22].
Our approach is similar to theirs where we are giving real-
time feedback using color gradient for estimating the depth
of tumors or other anatomy of interest for robotic assisted
and laparoscopic surgery intra-operatively.
Moreover, evaluation is another challenge in interactive
medical AR for surgical robotics. Many groups have used
optical or Infrared (IR) tracking systems using fiducial mark-
ers to evaluate interactive AR techniques [16] [23]. Such an
evaluation system might not be ideal for minimally invasive
surgical robotics as the working area is very small. On the
other hand, for minimally invasive surgical robotics, Lerotic
et al., 2007, used a pointer of Omni Phantom device (The
SensAble Technologies, Woburn, MA, USA) [3]. Although
the device can be used for AR evaluations, it might not
be suitable where interaction is an integral part of the
visualization, like ours. In such cases, the mode and ease of
interaction might also affect the depth perception. Therefore,
with the aim to use a real surgical tool for evaluation,
we conducted an initial experiment. In this experiment we
prepared a prostate gelatin phantom where the cast for the
phantom was prepared from the pre-operative MRI of a
prostate of a real patient. We rendered a tumor on the
endoscopic camera image of the phantom. In physical space
the tumor is underneath the phantom surface, simulating the
real surgical scenario where a tumor to be taken out is under
the tissue surface. Therefore, it is impossible to reach the
tumor physically without destroying the phantom. Therefore,
a virtual extension of 4mm was added to the tool tip in the
form of three red dots (figure 1). Then the users were asked to
touch the tumor with the extended surgical tool from different
orientations. As can be seen in the figure 1, the visible
position of the tool tip in the camera image differs from
the projected tool tip (the first red dot) based on kinematics
data. Moreover, the orientation of the projected tool changed
drastically from the initial orientation, due to the error in
the hand-wrist motion of the surgical instrument. Therefore,
all the subjects reported discomfort while interacting with
the system. Thus, we used a virtual tool to evaluate our
technique, replacing the need for a real tool.
 III. METHODS AND MATERIAL
A. Virtual Tool
In the following paragraphs to render the virtual tool at the
right position various transformation matrices are calculated
to make the virtual setup similar to the endoscope setup.
Virtual Tool was rendered using CAD model of Intuitive
Surgical Da Vinci (DV) Prograsp instrument. To simulate the
realistic tool movement, the joint encoder data was obtained
from Da Vinci clinical API. We collected the position and
orientation information of the various joints of the surgical
tool by reading Denavit–Hartenberg (DH) tables provided by
the surgical API. The forward kinematics data thus obtained Fig. 2. The figure showing the set-up used for the hand-eye calibration for
the current work. The surgical instrument image was captured in the right
was used to estimate the model-view matrices (E TR ) in and left camera frame and the corresponding tool-tip position in the Da
OpenGL to model a virtual camera. This transformation Vinci endoscope frame was also collected. The 3D position of the tool-tip,
brought the points from the robot base coordinate frame to P, in the left camera coordinate system was estimated by stereo triangulation.
Finally, the hand-eye transformation, CL TE , was estimated using a least-
robot endoscope coordinate system. Furthermore, to bring the square method.
points from the robot endoscope camera frame to the actual
right and left camera frames, the hand-eye transformation TABLE I
matrix, CL TE , was estimated. T HE LIST OF THE CASES EVALUATED
To estimate CL TE , we collected approximately 40 tool tip case 1 Stereo + Color Depth Encoding
positions in Da Vinci endoscope frame. The corresponding case 2 No Stereo + Color Depth Encoding
right and left images were also collected. The tool was visible case 3 Stereo + No Color Depth Encoding
in all the right and left collected images. Then these points case 4 No Stereo + No Color Depth Encoding
were manually selected in the right and left images and
the 3D positions of these points were estimated by stereo
triangulation. All the estimations were done with respect to depth information. This can result in perceptually disturbing
the left stereo camera center. The transformation between visualization.
the corresponding endoscope to the actual camera frame was B. Color Depth Encoding
estimated using a least square approximation method [24].
To give an intuitive measure of depth, the distance, di ,
The entire procedure is shown in figure 2.
between the surgical instrument tip and the tumor surface
Additionally, to have a convincing rendering of the tool
was mapped to the hue value of the HSV color space. Since
in the rendered surgical scene, it is important to have a
mapping the entire array of colors to the distance can cause
correct projection matrix, K. If incorrect, the wrong camera
additional cognitive load, we restricted the color mapping
parameters can cause conflict in disparities and hence, visual
from the red to blue color range, where red meant closer
discomfort. Projection matrix is the transformation to bring
to the tumor and blue meant the farthest. In other words,
the points from the world to the image space. Therefore, the
the hue component H(d) was defined as the function of the
stereo endoscope was calibrated using the method mentioned
distance, di . The colors were then normalized between the
in [25]. Then the estimated intrinsics were used to model
red to blue color range. If di was greater than the distance
the OpenGL projection matrix. The re-projection error for
between the tissue and the tumor surface, Dt , then the tumor
images of size 1920 × 1080 came out to be 0.29 pixels.
was rendered in blue color. On the other hand, if the tool tip
Thus all the calibrations used to bring the CAD tool model
was beyond the position of the tumor then di was given a
from the robot base to the right camera (i TR ) image frame
negative sign and the tumor was rendered in red.
are as follows:
Thus, the distance to color mapping was done as follows:
i
TR = K ×CR TCL ×CL TE ×E TR (1)
( d >D i t h0.66, 1.0, 1.0i
Hhh,s,vi = h(−0.4.di + 1.0), 1.0, 1.0i
0 < di < Dt
In addition to the virtual tool we also rendered a 4mm di < Dt h0.0, 1.0, 1.0i
virtual cylindrical surgical instrument extension to the virtual (2)
tool as shown in the images in figure 3. This extension was Then for the final visualization, Hhh,s,vi value was con-
added because in a real surgery the tumor or the anatomy verted to the RGB color space [26]. All calculations were
of interest can’t be reached as it is behind the physical done on GPU using GLSL shading language in fragment
intact tissue surface. Thus if the tumor is rendered at the shader.
correct position, it won’t be possible to reach the tumor. For
perceptually coherent visualization, the virtual tool was made C. Partial Occlusion
semi-transparent by adjusting the alpha channel in fragment To make the visualization perceptually consistent, partial
shader. This step is important as the conflict between the occlusion effect was used and two more AR techniques were
occlusion and the disparity cue can give different ordinal utilized; first, virtual window [10] and second, the edge

8293

Authorized licensed use limited to: Technische Hochschule Deggendorf. Downloaded on May 10,2020 at 06:12:35 UTC from IEEE Xplore. Restrictions apply.
extraction using Frei Chen edge detector [27]. To estimate
the final alpha mask in an OpenGL texture, the alpha masks
generated from both the techniques were combined. The final
alpha value was equal to the value of the edge alpha, if the
value of edge alpha was greater than virtual window alpha.
Otherwise, the value was equal to virtual window alpha.
Moreover, the final position of the alpha window in the
scene was chosen based on the 3D-tool tip position for a
consistent disparity. Surgical instrument was taken to the
position where the alpha window was to be rendered and
the position of the tool was recorded. Then this position was
projected in right and left camera image frames and that was
taken as the center of the alpha window.
D. Experimental Set-Up
Windows 7 platform was used for the visualizations with
3.60-GHz Intel(R) Core i7. OpenGL version 4.5 and C++
programming language was used for the renderings. Intuitive
Surgical Da Vinci Clinical API was used to collect the
forward kinematics joint encoder data from the surgical
robot. The stereo video stream with graphics was sent in
through Intuitive surgical Da Vinci surgical console for the
evaluation.
For evaluation, subjects were shown the real surgical scene
from a pre-recorded real prostate surgery. At all times they Fig. 3. Stereo image pairs showing the renderings from the evaluation
saw two views: on the top half of the screen, surgical scene task. (A-B) Stereo image pair with virtual tool and no visualization. This
and virtual tool, on the bottom half of the screen, the surgical view was visible in all the 4 evaluation cases on top half of the screen.
(C-D) Stereo Image pair showing the color depth encoding technique when
scene, virtual tool and the tumor with our rendered CDE the tool is the farthest from the tumor. (E-F) Images showing the change in
technique. This was done to mimic Da Vinci Tile-Pro view, color of tumor when the tool tip is the mid-way between the tissue surface
where the surgeon sees the real surgical scene and the scene and the tumor. (G-H) Stereo Image pair showing the change in color of
the tumor when the tool tip is the nearest to the tissue surface. Virtually
with the augmented medical data simultaneously, on the top rendered surgical tool can be seen on the right-hand side of all the images.
and the bottom half of the screen, respectively.
IV. E XPERIMENTAL D ESIGN AND VARIABLES
C. Tasks
A. Within-Subject Experimental Design
1) Training Task:
We chose a similar within-subjects experimental design
Since the proposed technique is an interactive depth es-
as in [28], [29]. We evaluated 4 conditions i.e. Stereo (S) +
timation technique for surgical robotic system, there is a
Color-Depth Encoding (CDE), Stereo + No CDE, No Stereo
requirement for a minimal dexterity in handling the master
+ CDE and No Stereo + No CDE. A list of all the cases is
side manipulator of Da Vinci surgical system to control the
given in table I. Furthermore, to counter-balance any carry-
virtual surgical tool. Keeping this in mind, we designed one
over effects we presented the cases according to the Latin
training task and two evaluation tasks. We designed a game
square used by Jones et al., 2008 [28].
In the current design the dependent variables are : judged where the user was shown randomly appearing spheres in
distance, time for completion and absolute depth judgement the surgical scene. The user was asked to poke the spheres
error. A complete list of dependent and independent variables with the extension of the virtual surgical tool. As an instant
is given in table II. feedback the spheres disappeared when the tool tip extension
intersected the bounding box of the sphere. The spheres also
B. Subjects changed colors based on the distance between the tool-tip
12 subjects participated in the study in total, out of which extension and the bounding box. The task continued till the
8 were male and 4 female. The participants were in the age user felt completely comfortable using the system.
group of 22 - 34 with the mean age of 27. The subjects 2) Evaluation Task:
were screened for vision or depth related abnormalities by In the evaluation task, for each case, the subjects were shown
verbal reporting. 11 participants were university students and tumors and were asked to judge two ordinal distances: first,
one was employed at the university. Out of 12 subjects, 4 the position of the mid point between the tissue surface and
were well versed with Da Vinci surgical system whereas the tumor and, second, the position of the tumor from the
rest 8 were novice with no prior experience with the sur- tissue surface. We called these cases in the rest of the paper
gical system. All participants were asked for verbal consent as ’mid’ and ’touch’, respectively. Each subject was asked to
beforehand and participated voluntarily in the study. stop the tool tip extension at the two ordinal distances. For

8294

Authorized licensed use limited to: Technische Hochschule Deggendorf. Downloaded on May 10,2020 at 06:12:35 UTC from IEEE Xplore. Restrictions apply.
 TABLE II
D EPENDENT AND I NDEPENDENT VARIABLES
INDEPENDENT VARIABLES
Variable Number Description
Observer 12 random-variable
Viewing Condition 2 Stereo and No stereo
Color Depth Encoding (CDE) 2 present and absent
Ordinal Distance 2 mid, touch
Repetitions 5 -
DEPENDENT VARIABLES
Judged distance 480 mid, touch
Time 480 - likeability, ease of use and visual comfort as 4.67, 4.08,
Absolute Depth Judgement Error 480 - 4.33, 4.42, 4.12 , respectively. All scores were given out
both the judged distances, the subjects verbally said ’start’
before starting the task and ’stop’ after reaching the judged
ordinal depth. The start and stop tool positions were recorded
by pressing a button. The time for the task completion was
also recorded. For each single ordinal distance, the variation
of tool trajectories with time was also monitored.
For each case, 5 tumors were shown for which the subjects
were asked to judge two ordinal distances. Thus in total
480 (4 cases × 5 tumors × 2 ordinal distances × 12
subjects) data points were collected. The total evaluation
task (including the training task) took about 30 minutes for
each subject.
At the end, the subjects were asked to fill a Likert scale
type evaluation test where they were asked to rate the
technique on the scale of 1 - 5 (1 being the least and 5 being
the most) in the following categories: usefulness, confidence
in depth estimation, general likeability, ease of use and visual
comfort using our CDE technique when compared to no CDE
in depth estimation.
V. RESULTS
The data points were analysed for judged distance, time re-
quired for task completion and tool trajectories. A histogram
of the data and Shapiro-Wilk test revealed that the data is not
normally distributed. The absolute error in depth estimation
was calculated for each case and subject as shown in figure
4 (A and B). In the figure, 95% confidence interval is also
plotted for each subject. The analysis was done on these
errors in depth estimation. The results of non-parametric
Wilcoxon signed rank test revealed a significant reduction
in perceived depth error between Stereo-CDE and Stereo-
No CDE cases (p < 0.0001, N = 240) and Stereo-CDE and
No Stereo-No CDE cases (p = 0.000, N = 240). Moreover,
No Stereo-CDE case also showed significant improvement
on Stereo-No CDE case (p = 0.002, N = 240). The analysis
of two ordinal depths separately showed significant depth
judgement improvement of No Stereo-CDE case on No-
Stereo-No CDE case (p = 0.002, N = 120) for the tumor
touch case.
Significant reduction in time of completion of task was
observed between Stereo-CDE and Stereo-No CDE case (p
= 0.029, N = 240). Average time in completing the task
for each subject, for all the four cases in shown in figure
4, B and C, for touch and mid cases, respectively, along
8295
Authorized licensed use limited to: Technische Hochschule Deggendorf. Downloaded on May 10,2020 at 06:12:35 UTC from IEEE Xplore. Restrictions apply.
with the corresponding box plots in figure 4, J and K. . The
absolute error in depth judgement, averaged over for all the
five tumors per evaluation case, is shown in figure 4, E and
F, for the touch and mid cases, respectively, along with the
corresponding box plots in figure 4, H and I.
The analysis of tool trajectories revealed less variations
with time in cases 1 and 3 versus cases 2 and 4, respectively.
Subjects rated our color CDE technique as compared to
No CDE in usability, confidence in depth estimation, general
of 5. Additionally 66.7% subjects reported maximum visual
comfort with the Stereo-CDE technique, while 8.7% , 8.7%
and 16.7% users felt most comfortable using No Stereo-CDE,
Stereo-No CDE, No Stereo-No CDE cases, respectively.
VI. CONCLUSION AND DISCUSSION
In the current work we presented a depth estimation
technique by mapping colors to the depth values in the
context of the surgical scene. The aim was to present the
depth information to the surgeon in such a way that there is
no need to look away from the surgical scene. Furthermore,
to evaluate our technique we used an innovative virtual
tool method where we rendered a virtual tool replacing the
need for a real tool in the evaluation task. To simulate the
realistic tool movement, forward kinematics data was used
from the robot. Unlike a previous experiment where the
subjects interacted with the visualization using a real surgical
tool, none of the subjects mentioned visual discomfort. Due
to compliance of the surgical instrument wrist mechanism,
there is an error in orientation value output by the robot.
Thus the subjects in a previous experiment experienced a
mismatch between where they intend to move the tool and
where the tool was actually moving. Such an error in the
mode of interaction with the visualization can cause serious
bias in the reported depth judgements. Thus, such a bias was
eliminated using a virtual tool instead.
In depth estimation also, our technique observed sig-
nificantly improved results. The subjects also scored our
technique very high in confidence in estimating depth in
the Likert scale questionnaire. Without our technique, the
estimation of depth, solely based on realtive size and partial
occlusion, can be affected by prior experience [30]. But
providing a quantitative depth measure might have helped
the subjects in taking the decision more confidently. In all
the cases, the depth judgement error in the mid ordinal depth
case is higher than the touch case (Figure 4 (B, F, H, I)). This
might be due to the variation in the correct estimation of the
color corresponding to the mid distance, across all subjects.
On the other hand, all the subjects observed the same
color, red, in the touch case. Thus, consistently less error
is observed for all the subjects in the touch case. Moreover,
the time taken for the task completion in Stereo+CDE case
is less than the Stereo+No CDE case thus showing that CDE
helped in making the depth judgement faster in stereo case.
There are two main contributions of this work: first,
presenting a quantitative depth estimation technique in the
Fig. 4. (A) The bars showing the absolute distance error in judged distance for all 4 cases for the near ordinal depth. (B) The bars showing the absolute
distance error in judged distance for the mid ordinal depth. (C) The bars showing the time taken for the task completion in near ordinal depth judgement
(D) The bars showing the time taken for the task completion in mid ordinal depth judgement task. In all the cases, blue, green, red and purple bars show
the Stereo-CDE, No Stereo-CDE, Stereo-No CDE and No Stereo-No CDE cases, respectively. The 95 % confidence interval is plotted in each plot for
each subject. (E) Average absolute error in depth judgement for 12 subjects for the near ordinal depth. (F) Average absolute error in depth judgement for
12 subjects for the mid ordinal depth. (G) The comparison of tool trajectories of subject 1, in Stereo-CDE (red) versus Stereo-No CDE (blue) cases. For
clarity only case 1 and case 4 are shown. (H-I) Box-plot of absolute error in judged distance in mid and near ordinal depth judgement tasks, respectively.
(J-K) Box-plot of time taken for the task completion in mid and near ordinal depth judgement tasks, respectively.

context of the surgery and, second, a new evaluation method
for medical AR for minimally invasive surgical robotics.
Our technique helped subjects in estimating the depth of the
tumor better, faster and with more confidence. The pattern of
tool trajectories also showed less variations with time using
CDE when compared to No CDE cases.
For future work, it will be interesting to design exper-
iments to study the affect of combination of CDE with
other depth cues like motion parallax or inter-sensory cue
like auditory information. Additionally, for evaluation, in
the current work we modelled the patient side manipulator
(PSM) information to render the virtual tool. It can be of
interest to use the MTM joint encoder data to render the
virtual tool and see its effects on the interaction. Using
MTMs can open gates for more innovative evaluation tasks
using a live video feed and thus replacing the real surgical
tool from the camera endoscopic view.

8296

Authorized licensed use limited to: Technische Hochschule Deggendorf. Downloaded on May 10,2020 at 06:12:35 UTC from IEEE Xplore. Restrictions apply.
 R EFERENCES [21] J. Wang, P. Fallavollita, L. Wang, M. Kreiser, and N. Navab, “Aug-
mented reality during angiography: integration of a virtual mirror for
[1] P. Pratt and A. Arora, “Transoral robotic surgery: Image guidance and improved 2d/3d visualization,” in 2012 IEEE International Symposium
augmented reality,” ORL, vol. 80, no. 3-4, pp. 172–180, 2018. on Mixed and Augmented Reality (ISMAR). IEEE, 2012, pp. 257–264.
[2] K. Abhari, J. S. Baxter, E. C. Chen, A. R. Khan, T. M. Peters, [22] A. Schoob, D. Kundrat, S. Lekon, L. A. Kahrs, and T. Ortmaier,
S. de Ribaupierre, and R. Eagleson, “Training for planning tumour “Color-encoded distance for interactive focus positioning in laser
resection: augmented reality and human factors,” IEEE Transactions microsurgery,” Optics and Lasers in Engineering, vol. 83, pp. 71–79,
on Biomedical Engineering, vol. 62, no. 6, pp. 1466–1477, 2015. 2016.
[3] M. Lerotic, A. J. Chung, G. Mylonas, and G.-Z. Yang, “Pq-space based [23] F. Bork, B. Fuers, A.-K. Schneider, F. Pinto, C. Graumann, and
non-photorealistic rendering for augmented reality,” in International N. Navab, “Auditory and visio-temporal distance coding for 3-
Conference on Medical Image Computing and Computer-Assisted dimensional perception in medical augmented reality,” in Mixed and
Intervention. Springer, 2007, pp. 102–109. Augmented Reality (ISMAR), 2015 IEEE International Symposium on.
[4] M. Kersten-Oertel, I. Gerard, S. Drouin, K. Mok, D. Sirhan, D. Sin- IEEE, 2015, pp. 7–12.
clair, and D. L. Collins, “Augmented reality in neurovascular surgery: [24] S. Umeyama, “Least-squares estimation of transformation parameters
first experiences,” in Workshop on Augmented Environments for between two point patterns,” IEEE Transactions on Pattern Analysis
Computer-Assisted Interventions. Springer, 2014, pp. 80–89. & Machine Intelligence, no. 4, pp. 376–380, 1991.
[5] A. Meola, F. Cutolo, M. Carbone, F. Cagnazzo, M. Ferrari, and [25] Z. Zhang, “A flexible new technique for camera calibration,” IEEE
V. Ferrari, “Augmented reality in neurosurgery: a systematic review,” Transactions on pattern analysis and machine intelligence, vol. 22,
Neurosurgical review, vol. 40, no. 4, pp. 537–548, 2017. 2000.
[6] M. Kersten-Oertel, S. J.-S. Chen, and D. L. Collins, “An evaluation [26] A. R. Smith, “Color gamut transform pairs,” ACM Siggraph Computer
of depth enhancing perceptual cues for vascular volume visualization Graphics, vol. 12, no. 3, pp. 12–19, 1978.
in neurosurgery,” IEEE transactions on visualization and computer [27] C.-C. Chen et al., “Fast boundary detection: A generalization and a
graphics, vol. 20, no. 3, pp. 391–403, 2014. new algorithm,” IEEE Transactions on Computers, vol. 100, no. 10,
[7] M. Kersten-Oertel, P. Jannin, and D. L. Collins, “The state of the art pp. 988–998, 1977.
of visualization in mixed reality image guided surgery,” Computerized [28] J. A. Jones, J. E. Swan II, G. Singh, E. Kolstad, and S. R. Ellis, “The
Medical Imaging and Graphics, vol. 37, no. 2, pp. 98–112, 2013. effects of virtual reality, augmented reality, and motion parallax on
[8] N. Navab, T. Blum, L. Wang, A. Okur, and T. Wendler, “First egocentric depth perception,” in Proceedings of the 5th symposium on
deployments of augmented reality in operating rooms,” Computer, Applied perception in graphics and visualization. ACM, 2008, pp.
vol. 45, no. 7, pp. 48–55, 2012. 9–14.
[9] C. Hansen, J. Wieferich, F. Ritter, C. Rieder, and H.-O. Peitgen, [29] J. Swan, M. A. Livingston, H. S. Smallman, D. Brown, Y. Baillot,
“Illustrative visualization of 3d planning models for augmented reality J. L. Gabbard, and D. Hix, “A perceptual matching technique for depth
in liver surgery,” International journal of computer assisted radiology judgments in optical, see-through augmented reality,” in Virtual Reality
and surgery, vol. 5, no. 2, pp. 133–141, 2010. Conference, 2006. IEEE, 2006, pp. 19–26.
[10] T. Sielhorst, C. Bichlmeier, S. M. Heining, and N. Navab, “Depth [30] M. Kalloniatis and C. Luu, “The perception of depth,” 1995.
perception–a major issue in medical ar: evaluation study by twenty
surgeons,” in International Conference on Medical Image Computing
and Computer-Assisted Intervention. Springer, 2006, pp. 364–372.
[11] C. A. Linte, K. P. Davenport, K. Cleary, C. Peters, K. G. Vosburgh,
N. Navab, P. Jannin, T. M. Peters, D. R. Holmes III, R. A. Robb,
et al., “On mixed reality environments for minimally invasive therapy
guidance: systems architecture, successes and challenges in their
implementation from laboratory to clinic,” Computerized Medical
Imaging and Graphics, vol. 37, no. 2, pp. 83–97, 2013.
[12] M. Bajura, H. Fuchs, and R. Ohbuchi, “Merging virtual objects with
the real world: Seeing ultrasound imagery within the patient,” ACM
SIGGRAPH Computer Graphics, vol. 26, no. 2, pp. 203–210, 1992.
[13] R. T. Azuma, “A survey of augmented reality,” Presence: Teleoperators
& Virtual Environments, vol. 6, no. 4, pp. 355–385, 1997.
[14] P. Edwards, D. Hawkes, D. Hill, D. Jewell, R. Spink, A. Strong, and
M. Gleeson, “Augmentation of reality using an operating microscope
for otolaryngology and neurosurgical guidance,” Journal of image
guided surgery, vol. 1, no. 3, pp. 172–178, 1995.
[15] E. M. Friets, J. W. Strohbehn, J. F. Hatch, and D. W. Roberts, “A
frameless stereotaxic operating microscope for neurosurgery,” IEEE
Transactions on Biomedical Engineering, vol. 36, no. 6, pp. 608–617,
1989.
[16] C. Bichlmeier, F. Wimmer, S. M. Heining, and N. Navab, “Contextual
anatomic mimesis hybrid in-situ visualization method for improving
multi-sensory depth perception in medical augmented reality,” in
Mixed and Augmented Reality, 2007. ISMAR 2007. 6th IEEE and ACM
International Symposium on. IEEE, 2007, pp. 129–138.
[17] J. E. Cutting and P. M. Vishton, “Perceiving layout and knowing
distances: The integration, relative potency, and contextual use of
different information about depth,” in Perception of space and motion.
Elsevier, 1995, pp. 69–117.
[18] M. Kersten-Oertel, S. J.-S. Chen, S. Drouin, D. S. Sinclair, and D. L.
Collins, “Augmented reality visualization for guidance in neurovascu-
lar surgery.” in MMVR, 2012, pp. 225–229.
[19] C. Winne, M. Khan, F. Stopp, E. Jank, and E. Keeve, “Overlay
visualization in endoscopic ent surgery,” International journal of
computer assisted radiology and surgery, vol. 6, no. 3, pp. 401–406,
2011.
[20] B. Behrendt, P. Berg, B. Preim, and S. Saalfeld, “Combining pseudo
chroma depth enhancement and parameter mapping for vascular sur-
face models.” in VCBM, 2017, pp. 159–168.

8297

Authorized licensed use limited to: Technische Hochschule Deggendorf. Downloaded on May 10,2020 at 06:12:35 UTC from IEEE Xplore. Restrictions apply.