Hongxin Zhang

2007-06-28
State Key Lab of CAD&CG, ZJU


z
z

Shape from shading

Kernel methods

Shape from X
z
z

Many cues can be used for inferring object shapes from images.
Based on the number of images used, there are two categories of
methods:
z
z

methods using multiple images

methods using a single image.

A list of so called shape-from-X algorithms are shown in the following

table.

Shape from texture

(a)
(b)

image of a plane covered by a deterministic texture.

the same texture on a curved surface

Shape from shading

z

Shape can be recovered from a single image by human visual system based on
the shading information

To estimate the shape from a single image, some assumptions have to be made,
e.g. constant albedo (surface color)

This technique is very useful for reconstructing surfaces of planets from

photographs acquired by spacecrafts

two images of the same Lambertian surface seen

from above but illuminated from different directions

3D rendering of the surface

Shape from shading - Example

Reflectance of a Lambertian surface

z

The radiance at a 3D point is proportional to the cosine of the angle

between the surface normal and the direction of the illuminant

L(P ) = R ,i = i in
z

is called the effective albedo(: the real albedo times the

intensity of illuminant

Illuminance
Luminance
Intensity

Fundamental equation for shape from shading

z

The pixel intensity at

p=[x, y]T is

d
E (P ) = L(P) cos 4
f
z

Assumptions:
z

If we neglect the constant term and if we assume the optical system has
been calibrated to compensate the cos4 effect, and in addition,

if we assume that all the visible points of the surface receive direct
illumination, we have the fundamental equation for shape from shading

E (P ) = R , i (n)
z

Notice that the image intensity is determined only by the surface normal vector

Reflectance map is a viewer-centered

representation of reflectance
n = [1, 0, p ] [ 0,1, q ] =
T

[ p, q,1]
2

1 + p2 + q

Surface
Orientation
Z

(0,1,Zy )
(1,0,Zx )
x y
Z = Z(x, y)

Depth
X
Y

dy

dx
IMAGE PLANE

Representing normal vectors

z

Assume that
z
z

the scene is far away from the camera, we can use a weakperspective camera model to describe the projection.
the average depth of the scene is Z0.

The weak-perspective projection can be written as

X
Y
x= f
y= f
Z0
Z0

Therefore, with some rescaling, the scene surface can be

thought of as a function of the (x, y), or
Z
Z
Z = Z ( X , Y ) = Z ( x 0 , y 0 ) = Z ( x, y )
f
f

Then the slopes along the x axis and y axis are

T

Z
Z x
Z f
Z f

1, 0, X = 1, 0, x X = 1, 0, x Z and 0,1, y Z

0
0

Representing normal vectors

z

We denote the above vectors as [1, 0, p]T and [0, 1, q]T . The
normal vector is the cross product of these two vectors, therefore

n = [1, 0, p ] [ 0,1, q ] =
T

[ p, q,1]

1+ p + q
2

Now we can rewrite the fundamental equation as

E ( x, y ) = R , i ( p, q ) =

1+ p + q
2

i [ p, q,1]
T

The problem of shape from shading

z

Assumptions
z
z
z
z

The imaging system is calibrated so that the cos4 effect is compensated

All the visible surface receive direct illumination
The surface is imaged under weak-perspective projection
The surface can be parameterized as Z=Z[x, y]

Shape from shading

z

Given the reflectance map of the surface R , i ( p, q ) , and full knowledge of the
parameters and i relative to the available image, reconstruct the surface slopes, p
and q, for which

E ( x, y ) = R ,i ( p, q )
z

and the surface Z=Z[x, y] s.t.

Z Z 0
Z Z 0
p=
and q =
x f
y f

To simplify the formulation, we will reconstruct a scaled version of the original

surface so that

p=

Z
Z
and q =
x
y

Finding albedo and illuminant direction

z

Assumption
z
z
z

Albedo is the same for all surface points

The surface is Lambertian
The direction of the surface normal vectors are uniformly
distributed in the 3D space

A normal vector is represented by two angles,

and , where [0,2] and [0, /2]

As seen from the image plane, the

probability of a normal vector with

Finding albedo and illuminant direction

z

If we also represent the illuminant direction using two angles

[0,2] and [0,/2]. As the result, the illuminant vector is

The image brightness of a surface point with normal angles

and is

The average image intensity becomes

Finding albedo and illuminant direction

z

Similar derivation gives us

From above two equations, we can recover the

albedo and the angle

Therefore

Finding albedo and illuminant direction

z

To estimate the angle , we compute the average image spatial

gradient and estimate

Algorithm_Approximate_Albedo_Illuminant
z Compute the average of the image intensity <E> and of its
square, <E2>
z Compute the average spatial image gradient <Ex, Ey>
z Estimate the albedo and the illuminant angle as

Shape from shading

z

The fundamental equation of shape from shading is

E ( x, y ) = R , i ( p, q ) =

1 + p2 + q2

iT [ p, q,1]

The problem of shape from shading

z Given an image E(x,y),
z and full knowledge of the parameters and i relative to the
available image,
z reconstruct the surface slopes p and q, and the surface Z(x,y)

Shape from shading

z
z

For each pixel,

z there are two unknown variables and one known quantity.
Shape from shading is a highly under-constrained problem
Solution
z To make the problem tractable, we add a constraint that a
smoother surface is preferred. This soft constraint can be
translated into a regularization term in an optimization problem
z Variational method for shape from shading: we will find the
smoothest solution that satisfies the fundamental equation. In
other words, the solution should minimize the following energy
function

Euler-Lagrange equation
z

If J is defined as
where q = dq / dt
The derivative of J vanishes if the Euler-Lagrange
equation

is satisfied. If the time-derivative q is replaced by

space-derivative qx, the equation becomes

Euler-Lagrange equation
z

In our case, the equation is

Using the Euler-Lagrange equation with two independent

variables, the stationary value can be achieved when

where

Therefore,

The discrete form

can be simplified as

The discrete version of these equations are

An iterative algorithm
z

The previous discrete equations can be

rewritten as

An iterative algorithm
z

The previous discrete equations can be rewritten as

pij and qij can be solved by starting from some initial solution at
step 0, and advancing from step k to step k+1 using the updating
rule

(1)

Enforcing integrability
z

pij and qij are solved independently, there may not

exist surface Z so that Zx=p, Zy=q

Solution:
z

After each iteration, we find a revised solution p and q, so

that they are integrable and are closest to p and q
Suppose IFFT of p and q are

Enforcing integrability
z

Solution:

(2)

Enforcing integrability
z

Then the integrable solution that are closest p

and q are

It can be verified by starting from p and q

and using the above equation to obtain the
same p and q

Shape from shading algorithm

z

Given the effective albedo, the illuminant direction,

and the image, initialize the surface slopes p and q
to 0.

Until a suitable criterion is met, iterate the following

two steps
z
z

Update p and q using (1)

Compute the FFT of the updated p and q, estimate Z and p
and q using (2)

Output Z, p, q

Experimental results

After 100 iterations

After 1000 iterations

After 2000 iterations


z
z

Shape from shading

Kernel methods
main idea : x

( x)

The stages involved in the

application of kernel method
Kernel ?

Data

Nonlinear ?

K(X,Z)

Kernel
Function

Kernel
Matrix

SVM ?

PA
Algorithm

f ( x) = i k ( xi , x)

Pattern
Function

Linear regression in a feature

space
z

Primal linear regression:

z

Find a homogeneous real-valued linear function

n

g ( x ) = w , x = w ' x = wi xi
i =1

That best interpolate a given training set

S = {( x1 , y1 ), ( x2 , y2 ),
z

( xl , yl )}

Or create a pattern function that should be

approximately equal to zero

f ( x, y) = y g ( x ) = y w, x 0

Primal linear regression

z

We would like to find a function for which all

of these training errors are small
l

i =1

i =1

L( g , S ) = L( w , S ) = ( y g ( xi )) 2 = L( g , ( xi , yi )) 2
Least squares approximation

Hence the lose function can be written as

L( g , S ) =

2
2

= ( y Xw )( y Xw )

We have
L( w , S )
= 2 X ' y + 2 X ' Xw
w

Primal linear regression

z

Normal equations
X ' Xw = X ' y
z

w = ( X ' X ) 1 X ' y

The cost is O(n)3

Dual representation
w = ( X ' X ) 1 X ' y = X ' X ( X ' X ) 2 X ' y = X '
l

g ( x ) = w , x = w ' x = ' Xx = i xi , x
i =1

Ridge regression
z

Ridge regression corresponds to solving the

following optimization:
l

min L ( w , S ) = min w + ( yi g ( xi )) 2
2

i =1

Primal:
X ' Xw + w = ( X ' X + I n ) w = X ' y

w = ( X ' X + In ) X ' y
1

Ridge regression
z

The resulting prediction function is

g ( x ) = w , x = yX ( X ' X + I n ) 1 x

w = 1 X '( y Xw ) = X '
= 1 ( y Xw )
G = XX ' or Gij = xi , x j
= ( y XX ' )
( XX '+ I l ) = y
= (G + I l ) 1 y

Dual:

g ( x ) = w, x =

i =1

i =1

x
,
x
=

x
,
x
=
y
'(
G
+

I
)
k
i i
i i
n

Ridge regression
z

The dual solution:

g ( x ) = w, x =

i =1

i =1

x
,
x
=

x
,
x
=
y
'(
G
+

I
)
k
i i
i i
n

ki = xi , x
z

The ridge regression can be solved in a form

that only requires inner products between
points

Nonlinear feature mappings

z

An embedding map
: x

S = {( x1 , y1 ), ( x2 , y2 ),

( x) F

( xl , yl )}

S = {( ( x1 ), y1 ), ( ( x2 ), y2 ),

( ( xl ), yl )}

f ( x, y) = y g ( x ) = y w, ( x ) 0

G = XX ' or Gij = ( xi ), ( x j )

ki = ( xi ), ( x )

Kernel function
z

A kernel is a function k that for all x, z satisfies

( x, z ) = ( x ), ( z )
z

where is a mapping from X to an (inner

product) feature space F
:x

( x) F

Kernel Example
z

The feature map

: x = ( x1 , x2 ) ( x ) = ( x12 , x22 , 2 x1 x2 ) F = 3
The hypothesis space of linear functions in F
g ( x ) = w x + w x + w12 2 x1 x2
2
11 1

2
22 2

The inner product

( x, z ) = ( x ), ( z ) = x12 z12 + x22 z22 + 2 x1 x2 z1 z2

= x1 z1 + x2 z2

= x, z

The kernel trick

z

More generally: x , z

x, z
=

= xi zi
j =1

j1 , j2 ,

jd =1

x j1 x j2

,d

x jd z j1 z j2

z jd = ( x ), ( z )

Characterization of kernels
z

A function

:XX
which is either continuous or has a countable domain, cab be
decomposed

( x, z ) = ( x ), ( z )
into a feature map into a Hilbert space F applied to both its
arguments followed by the evaluation of the inner product in F if
and only if it satisfies the finite positive semi-definite properties

Kernel PCA
z

Input: data S = { x1 , x2 , , xl } , dimension k

Process:

K ij = ( xi , x j ), i, j = 1,...l
K K 1l jj'K 1l Kjj '+ l12 ( jKj ') jj '
[V , ] = eig ( K )

j =

normalization
Eigen-analysis

v j , j = 1,...k
k

j
xi = i ( xi , x )
i =1
j =1
Output: transformed data S = { x1 , x2 , , xl }
l

Kernel matrix

Dual principle component

regression
z

Input: data S = { x1 , x2 , , xl } , dimension k and target

s
y
, s = 1,..., m
output vectors

Process:

K ij = ( xi , x j ), i, j = 1,...l
K K 1l jj'K 1l Kjj '+ l12 ( jKj ') jj '
[V , ] = eig ( K )
k

=
s

j =1

(v j y s )v j , s = 1,...m

Output: regression functions

l

f s ( x ) = is ( xi , x ), s = 1,..., m
i =1