3D RANGE MAP USING STRUCTURED LIGHT AND THREE-POINT

EPIPOLAR CONSTRAINTS

J.M. VALIENTE GONZÁLEZ; A. RODAS JORDÁ; G. ANDREU GARCÍA

Departamento de Ingeriería de Sistemas, Computadores y Automática (DISCA)

Universidad Politécnica de Valencia. Camino de Vera s/n, 46071 Valencia
SPAIN
E-mail: jvalient@disca.upv.es

ABSTRACT other hand, passive techniques only use information

3D visual information is one of the primary
requirements for the robot operation in complex
applications. It is especially necessary in tasks that involve
detecting, inspecting or manipulating objects. Structured
Light is an active technique based on projecting a pattern
of light onto the workspace. The depth informati on can
be recovered by taking an image of the scenario and by
establishing the proper correspondences among the points
obtained in the image and those of the projected pattern.
In this paper we present an approach based on
structured light that tries to obtain a range map using a
grid of light. The term “grid-coding” is often used in the
literature to refer this kind of techniques. Our work is
focused on establishing correspondences between a set of
three connected points in the image and some triangle
pattern in the projected grid. We propose a triangle-
labelling problem by using the epipolar constraint applied
to the three-point case. So, it has been become into a
minimizing problem for which a least-squares solution
have been obtained.
To check the solution a simulation tool has been
developed. The experiences performed showed us the
extreme sensitivity of the 3D solutions obtained with
respect to the relative camera-projector displacements. We
conclude that less than 1,5 % of error in the 3D point
coordinates can be obtained for orthogonal camera
configurations.
obtained from camera images without introducing an
artificial light.
Structured light is an active technique that uses an
artificial light source from which some pattern of light can
be projected over the workspace. The light projecting
device can range from complex laser scanners to simple
slide projectors. Besides this, the used light pattern could
be dots, lines or more complex patterns. Once the light
pattern has been projected, an image is taken from a
camera. In this image the light structure appears deformed
by the object shape. Then, the image is pre-processed by
using thresholding and thinning algorithms that permits to
obtain a clean version of the image grid. At this point, the
goal pursued is similar as in stereo vision one, which is to
solve the correspondence problem. To accomplish this, it
is necessary to determine feature points in the image so as
to identify the corresponding pattern in the projector and
finally to recover depth information using triangulation.
The advantage of using structured light is that light pat-
terns can be chosen in order to facilitate this identification.
Several approaches using colour [2], M-arrays [3], or
codified binary patterns [4] have been reported in the
literature.
In this paper we present an approach based on
structured light that tries to obtain a 3D range map by
using a grid of light. Although the grid does not include
any kind of spatial codification, the term “grid-coding” is
often used in the literature to refer this sort of techniques.

Keywords: 3D Range map, Structured light, Grid-
coding, Three-point epipolar constraints.
1. INTRODUCTION
There are many approaches to the problem of
recovering 3D information by using 2D input obtained
from vision systems. They can be divided into two broad
categories [1], active and passive. Active techniques use
some kind of energy projected onto the workspace. On the
Grid coding has been used in [5], [6] to recover depth
information in different contexts. In these approaches,
either the z coordinate or the surface orientation of 3D
points is obtained by using projective geometry and by
imposing geometrical and topological constraints. These
constraints comprise both global 3D constraints (object
geometry, surface smoothness, scenario limits) and local
2D image constraints (grid point connectivity).
To simplify the mathematical formulation, a parallel
projection scheme is always used. Recent studies try to
improve the results using two cameras [7] or rectifying the  i− j n ⋅ i − n1 ⋅ j 
grid junctions [8]. G (i , j ) =  ( x , y ) / x = ⋅p ; y= 2 ⋅ p
 n2 − n1 n2 − n1 
In our work, we only rely on projective geometry ∀i, j = 0,±1,±2,....,± k (1)
properties of a reduced set of grid points. The minimum where n1 = tg α1; n2 = tg α2 and p are the grid
number of points used is three, even though four, five or parameters. These parameters are pre-defined in some real
more points could also be used. We formulate the context. It is supposed that the grid is centred in the origin
underlying perspective equations to triplets of connected of the projector coordinate system.
points that produce some triangle pattern extracted from
the image. To solve this equation system, we introduce a
method based on epipolar geometry constraints that,
applied to three points, permit us to obtain a least-squares
solution of this system. Finally, a dense range map can be
obtained by repetitively applying this technique to every
triangle patterns in the image.

2. THE SENSING ENVIRONMENT

Figure 1 shows the two devices that compose the used
sensing environment, the structured light projector and the
CCD camera. A mathematical model has been defined for
each one.
We suppose a projector model based on a point light
Figure 2. – The grid pattern.
From the grid model we can define the basic patterns
formed by three connected points (triangle patterns).
Given a grid point p(i0,j0), the coordinates of some triangle
pattern formed by three connected points p0, p1 and p2, can
be obtained as follows:

 x0   x0   0   x 0   h0 
      p    
x =
 1  0  1 0
x + ( i − i ) − ( j − j 0 
) ⋅ =  x 0  +  h1  (2)
−
1
 x   x   (i − i ) − ( j − j )  2 n n1  x  h 
 2  0  2 0 2 0   0  2
 y0   y0   0   y0   v0 
      p    
 y1  =  y 0  +  n 2 (i1 − i0 ) − n1 ( j1 − j0 )  ⋅ =  y 0  +  v1 
 y   y   n (i − i ) − n ( j − j )  n 2 − n1  y   v 
Figure 1. The sensing environment  2  0  2 2 0 1 2 0   0  2
source where, for convenience, the global coordinate
system is placed. The projector plane, that includes the In this context, The term “connected” means that
grid pattern, is located at fp on the z-axis, where fp is the there are light segments joining the three points, and that
focal length of the projector lens. With regard to the these unions can be checked in the image. It is remarkable
camera, we suppose a camera model having an image that Eq. (2) relates every point coordinates with those from
coordinate system OXcYcZc that has its origin at the focal the first point (the base point), and that they only depend
point. The front image plane (the CCD plane) is at fc on on index differences, no matter about the absolute value of
the z-axis, where fc is the focal length of the camera lens. It these indexes.
is supposed that all the intrinsic and extrinsic camera
parameters are known in advance. In real conditions, these 3. THE PROBLEM STATEMENT
parameters should be estimated using the suitable
calibration technique. The grid of light is projected over the workspace and
an image is taken from the camera. Obviously, the image
A mathematical grid model of the grid pattern is also grid has deformations due to the objects in the scenario,
necessary to define the relations among the grid points. and it is not continuous because it could be broken by
Two sets of lines uniformly spaced and with known angles object edges or by occlusions.
form a grid. These lines, that we call i-lines and j-lines,
are defined by an index number, as it is shown in figure-2. We have assumed that the objects in the scene have a
The intersecting points of these lines are uniquely defined smooth surface and their dimensions are greater than the
by a 2-tuple (i,j), in such way that the grid pattern can be units of grid space. From the image grid, all the
completely defined by the set of intersecting points as intersecting points are drawn, grouping them by triplets of
follows: connected points (deformed triangles). At this point, a
matching procedure has to find out which pattern triangle
on the projector corresponds to the deformed triangle in
the image. Since all triangle patterns are identical, only
differing in their position on the grid, the implicit
correspondence problem becomes a triangle-labelling
problem.
respective epipolar line?. Using a minimum square point-
line distance criterion, a linear equation system is obtained
which has only one least-squares solution within the grid
pattern scope.

As stated before, many authors have used some kind

of spatial codification [4][5][6], to simplify the labelling
problem. However, in this case there is not such kind of
codification, then we only have to rely on geometrical
considerations to obtain a solution. Unfortunately, the use
of perspective projection assumptions is very difficult,
because it usually produces non-linear equation systems.
Therefore, we propose an alternative approach based on
epipolar constraints.

4. EPIPOLAR GEOMETRY Figure 3. - Triangle fitting to the pattern grid.

Let p (r,s) be an image point. All the 3D points lying
on the line formed by the Oc (the origin of camera
coordinate system) and the point (r, s, fc) have the same
projection on the image plane. The projection of this line
over the projector plane is called the epipolar line of p. It
can be shown that the equation of this line, referred to
projector coordinate system, is given by:
Let p0(i,j) = (x0,y0), p1(i+1,j) = (x1,y1) and p2(i+1,j+1)
= (x2,y2) be three connected points in the grid. Their order
implies a specific triangle orientation. Let d0, d1 and d2 be
the distances between these points and their epipolar lines.
If we use the Eq. (2), we can write these distances as
d 0 = ( K 0 + m 0 x 0 − y 0 ) / m12 + 1

y = m⋅ x +b (3) d 1 = ( K 1 + m1 x 0 − y 0 ) / m12 + 1 (5)

t w −t w t 2 w1 − t 1 w2 d 2 = ( K 2 + m 2 x 0 − y 0 ) / m 22 + 1
m= 3 2 2 3 b = fp ⋅
t 3 w1 − t 1 w3 t 3 w1 − t 1 w3 K 0 = b0 + m 0 h0 − v 0 K 1 = b1 + m1 h1 − v1
 w1  r  K 2 = b2 + m 2 h2 − v 2
   
 2
w = R ⋅  s t = (t 1 t 2 t 3 ) T follows:
w  f 
 3  c
Minimizing Ω = Σdi2 with respect to the two
unknown parameters (x0,y0), we set ∂Ω/∂x0 = 0 and
where R is the rotation matrix and t the displacement
∂Ω/∂y0 = 0 to yield
vector of the camera with respect to the projector.
 a0 − a1   x 0   B 0 
 ⋅  =   (6 )
Let p0(r0,s0), p1(r1,s1) and p2(r2,s2) be the three  − a1 a 2   y 0   B1 
extreme points of some deformed triangle in the image.
2 2 2
Let the corresponding epipolar lines in the projector plane
be given by
a0 = ∑m D
0
i
2
i
2
a1 = ∑m D
0
i i
2
a2 = ∑D
0
i
2

L0 : y = m0 ⋅ x + b0
2 2
( 4) B0 = − ∑m K D i i i
2
B1 = ∑K D i i
2
D = 1 /(mi2 + 1)
i
2

L1 : y = m1 ⋅ x + b1 0 0

where
L2 : y = m2 ⋅ x + b2
The solution of this 2x2 normal equation system
yields the least-squares estimates for the unknown triangle
Then, under ideal conditions, the three corresponding
a B + a1 B1 a B + a 0 B1
points of the triangle pattern have to lye exactly on these x0 = 2 0 y0 = 1 0 (7)
lines (Epipolar constraint). Obviously, this condition is a 0 a 2 − a1
2
a 0 a 2 − a12
not hold under real conditions. This is due to data errors position
that lead to some displacements in the epipolar lines. However, the triangle patterns on the projector can
only have discrete positions, whereas the solution obtained
Thus, a minimising problem can be stated as follows: can be situated anywhere as it is shown in figure 3. A final
Given three lines (the epipolar lines converging on the step is introduced to move the triangle solution to the
same point) and a triangle pattern (with a known size and nearest grid triangle pattern.
orientation), which is the "best" triangle position having
each of its extreme points lying on (or near to) their
5. EXPERIMENTS AND RESULTS
Several experiments were carried out by computer
simulation. A virtual workspace was generated including a
base plane (worktable) and several geometric objects like
cubes, cylinders or cones on it. Afterwards, a ray tracing
procedure was carried out simulating the light scattering
from the projector, with the rays passing through the grid
pattern and intersecting with the objects. From the camera
point of view, an image was generated taking into account
the possible occlusions. Finally, the Eq. (8) was applied to
each of the triangles obtained from the image grid.
The simulation parameters were chosen to
approximate to real conditions. A ½” CCD camera was
used having a 758x570 pixel resolution and a 20 mm focal
length. Two 33-fringes patterns with α1=30º and α1=120º
compose the grid of light. This grid was projected onto a
2x2 m scenario by using a 20 mm projector.
Three camera configurations were proposed to the
simulation tool: (1) A parallel-configuration with the
camera located at distance (300,700,10) millimetres and
without rotation respect to the projector coordinate system.
(2) An orthogonal configuration with displacement
(1400,1400,1400) mm and rotation (60,60,0) degrees. (3)
Finally, an intermediate configuration with displacement
(700,1400,100) and rotation (40,-10,0).
The results showed that using exact data points, the
3D-triangle pose solution was successful. In practice,
exact camera point coordinates are not possible. Errors
due to lens distortion and signal digitisation are always
present. Moreover, the most important source of error lies
on detecting the grid point intersection in the camera
image [8]. Rather than pursuing an analytical error
characterisation, we examined the effect of point location
errors by adding a Gaussian random error N (0,σ) to the
exact point locations. Random pixel displacements were
assigned to both, the xi and the yi co-ordinates of the three
image points. Then, a root mean squared error of the 3D-
point position was computed, by repetitively applying
(5000 trials) this random displacement to each image
point. Several experiments were carried out using this
approach with different objects on the workspace.
The results showed in figure 4 reveal an extreme
sensitivity of the 3D solutions obtained with respect to the
relative camera-projector displacements. More accurate
results were obtained by using the orthogonal
configuration than the parallel one. Less than 1,5 % of
errors in 3-D point coordinates were obtained in the
former case.
Another experiences also showed that errors in the
camera displacement parameters affect proportionally to
the obtained RMS 3D-position error. Instead, the errors in
the rotation parameters were negligible due to the
invariance of epipolar lines to camera rotations.
Figure 5. - Relative 3D position error vs. camera configuration
6. CONCLUSIONS
The structured light approach presented is
appropriate in industrial contexts where the lighting
conditions and the camera-projector configuration can be
controlled. The precision obtained seems adequate to
generate 3D range maps suitable for robot applications.
More accuracy can be achieved including constraints
between neighbour triangles as in [5]. Moreover, the
solution proposed can be used as an initial estimation for
some more complicated and robust methods.
REFERENCES
[1] D. Nitzan. Three-dimensional vision structure for robot
applications. IEEE Trans. Patt. Anal. Machine
Intelligence, 10(3), 1988, 291-309.
[2] K.L. Boyer and A. C. Kak. Color-encoded structured light
for rapid active ranging. IEEE Trans. Patt. Anal. Machine
Intelligence., 9(1), 1987, 14-28.
[3] S. Kiyasu, H. Hoshimo, K. Yano and S. Fujimura.
Measurement of the 3-D shape of specular polyhedrons
using an M-array coded light source. IEEE Trans. On
Instrument and Measurement., 44(3), 1995.
[4] P. Vuylsteke and A. Oosterlinck Range image acquisition
with a single binary-encoded light pattern. IEEE Trans.
Patt. Anal. Machine Intelligence., 12(2), 1989, 148-164.
[5] G. Hu and G. Stockman. 3D surface solution using
structured light and constraint propagation. IEEE Trans.
Patt. Anal. Machine Intelligence, 11(4), 1989, 390-410.
[6] N. Shrikhande and G. Stockman,. Surface orientation from
a projected grid IEEE Trans. Patt. Anal. Mach. Intell.,
11(6), 1989, 650-655.
[7] C. Chen and Y.F. Zheng, Passive and active stereo vision
for smooth surface detection of deformed plates. IEEE
Trans. On Industrial Electronics, 42(3), 1995, 300-306.
[8] T.L. Chia, Z.Chen, C-J Yueh. A method for rectifying grid
junctions in grid-coded images using cross ratio. IEEE
Trans. On Image Processing, 5(8), 1996, 1276-1281.