Three-Dimensional CMOS Image Sensor

for Pedestrian Protection and Collision

Mitigation

Peter Mengel Siemens AG

Co - Authors:
L. Listl, B. König, C. Toepfer, M. Pellkofer, Siemens AG
W. Brockherde, B. Hosticka, O. Elkhalili, O. Schrey, W. Ulfig, Fraunhofer IMS

peter.memgel@siemens.com AMAA 2006 1

Content

• Introduction and motivation

• 3D sensor technology
• Road safety applications
• Car evaluation of first prototype
• Conclusion & outlook

peter.memgel@siemens.com AMAA 2006 2

Environmental Sensing Overview

1) Near Distance Range

! Radar (24 GHz)
Lane Change Assistant
! Video
Intersection
! 3D-Kamera

Blind Spot Detection

Pre- Crash
Night Vision

Back-up Parking Parking

LDW ACC
Aid Aid Aid
Lane Keep
Stop Assistant
& Go

Blind Spot Detection

Intersection
2) Medium Distance Range
Lane Change Assistant ! Video

3) Far Distance Range

! Radar (77/79 GHz)
! Lidar

peter.memgel@siemens.com AMAA 2006 3

TOF Principle
Multiple Double Short Time Integration (MDSI)

U Integrator
Transmitted
3D-Scene Laserpulse
Laserpulse U2

Reflected
U1 Laserpulse
d Trigger

d
CMOS Sensor T0 T1 T2
With High
Speed Shutter
vC & U1 #
d = TPW $$1 ' !!
2 % U2 "
VC = Velocity of light
TO = Time of Flight
TPw= Laser Pulse Width
peter.memgel@siemens.com AMAA 2006 4
3D CMOS Image Sensor Evolution

• Available technology at start of PReVENT

• Sensor used for first prototype evaluation

64 x 4 Pixel ToF Line Sensor

peter.memgel@siemens.com AMAA 2006 5
3D CMOS Image Sensor Evolution

New ToF 64 x 8 Pixel Array

Pixel pitch:
130!m!300!m

Chip size:
10.58 ! 9.4mm_

Sensitive area:
8.39 x 2.40mm_

Process:
IMS 0.5!m CMOS

peter.memgel@siemens.com AMAA 2006 6

ToF 64 x 8 Pixel Array

Key Specification Data

– Pixel count/size 64x8 = 512 / 130µm x 300µm
– Pixel principle photo diode, single buffer type
– NEP (= smallest 2,3 W/m2
detectable signal)
– Minimum shutter 30 ns
– Dynamic range 75 dB (distance & reflectance range)
– Image acquisition 50…200 fps (1…128 pulses/frame)
non-destructive readout
– Background light suppr. > 40dB
– Operating wavelength 850….910nm
– Chip size < 100mm2 (IMS 0.5µm CMOS process)
peter.memgel@siemens.com AMAA 2006 7
Microsystems Architecture

12
12VV
Flash RAM ADC Power
Power
Flash RAM ADC
Supply
Supply(Car)
(Car)

DC/DC-
DC/DC-
3D-
3D- Converter
Converter
FPGA CMOS
CMOS
RS-232
RS-232 FPGA (not included into
(not included into
sensor
sensor camera housing)
camera housing)

• Sensor Control
LAN
LAN 100
100 64 x 8 Pixel Array
• Data Acquisition
• Distance Calculation
User
User I/O
I/O • Application Software Laser
Laser
Source
Source
Interface
Interface
Board Sensor Laser
LaserModule
Board SensorBoard
Board Module

peter.memgel@siemens.com AMAA 2006 8

Customized Laser Modules

Modular design depending on application

– High Output energy up to 400 !Ws (Multilayer design)
– Selectable pulse width 40 - 200 ns, 40 kHz repetition rate
– Laser class 1 (module + illumination optics)

Image of completed base module Optical output pulse

peter.memgel@siemens.com AMAA 2006 9
Road Safety Application- Objectives

One Active 3D Range Sensor for a Variety of Different

Functions and Applications:

• The near to intermediate side

and front distance range of
cars up to 25 m
Side View
• Blind area surveillance
• 3D- algorithms for traffic
Front view
object detection and
classification
Blind Area Surveillance
• Performance investigation
with car/truck demonstrators

peter.memgel@siemens.com AMAA 2006 10

Requirements and Specifications

• Front View Perception (BMW & CRF):

– Pedestrian and rear end car collision mitigation in
urban areas

– Protection by means of autonomous and semi-

autonomous braking and/or conditioning of other
active components

peter.memgel@siemens.com AMAA 2006 11

Requirements and Specifications

Side Crash Perception Blind Area Surveillance

for Cars (Renault) for Trucks (VTEC)

peter.memgel@siemens.com AMAA 2006 12

3D Test System in Experimental Car

peter.memgel@siemens.com AMAA 2006 13

Scenarios for Vehicle Tests

Application Camera Laser

collision frontal view, 1 module (75 µWs),
mitigation mounted at hood isotropic illumination

frontal view, 2 modules (2 x 75 µWs),

mounted at hood anisotropic illumination
pedestrian
protection
frontal view, 2 modules,
mounted at hood anisotropic illumination

lateral view,
side crash 2 modules,
mounted at
prediction isotropic illumination
side window

blind area lateral and 2 modules,

surveillance frontal view, isotropic illumination

peter.memgel@siemens.com AMAA 2006 14

Distance Calibration in a 20 m Stage

20 m track (computer controlled position) plywood target

2020
1818
1616
1414
1212
calibration
1010 function
88
66
44
22
000 2 4 6 8 10 12 14 16 18 20
0 2 4 6 8 10 12 14 16 18 20
true distance [m]
1

0.80.8
spatial
0.6

0.40.4
standard
0.2
deviation
0 0
= ~0.3m
-0.2

-0.4
-0.4

-0.6

Target (reflector, dark felt, plywood) -0.8

2 4 6 8 10 12 14 16 18 20 22

2 4 6 8 10 12 14 16 18 20 22
AMAA 2006 15
peter.memgel@siemens.com
true distance [m]
Test Scenario: 2 cars

range data

Mercedes at 11 m Renault at 5 m

! both cars (Renault and Mercedes) are visible to the camera

peter.memgel@siemens.com AMAA 2006 16

Test Scenario:
Approaching Pedestrian

• Pedestrian detection from 18 m to 4 m

m true distance = 12m

/ true distance 12 m
e 20
c
n 18
a
t 16

measured distance [m]

s
i
d 14

d 12
e
r 10
u
s
8
a
e
m 6

0
0 10 20pixel30number
40 50 60
pixel
Image scene with a pedestrian Range data with 64 pixel line sensor
peter.memgel@siemens.com AMAA 2006 17
Test Scenario:
Laterally Passing Pedestrian

camera mounted
on side window

12000
12000

10000
10000

# pixel pedestrian 8000

8000

0
6000
6000

4000
4000

test 2000
2000

30 car
00 0 10 20 30 40 50 60
0 10 20 30 40 50 60
# pixel
range data for the marked (x)
63 birds-eye view
pedestrian position
peter.memgel@siemens.com AMAA 2006 18
Test Scenario: Side Crash Protection - Tree
at Lane Side

range data
12000
12000

10000
10000 tree at 5 m
8000
8000

6000
6000

4000
4000

2000
2000

00 0 10 20 30 40 50 60
0 10 20 30 40 50 60
# pixel

! tree trunk is well visible at 5 m distance

peter.memgel@siemens.com AMAA 2006 19

Test Results

" The sensor performance was successfully

investigated in different test scenarios,
including moving objects
" !"#$%#&'()*(+*&,#++'-*.-/)#,'(.*.0(1
%./+%$*,#)2/*3#&#*%4*&(*56*7
" 89.&/7*-#)*-(4/*1'&0*(%&3((,:.'&%#&'().;
')-$%3')2*<,'20&*.%)*$'20&
" =(,*')&/,'(,*7(%)&')2*>?:&,#).4#,/)&
1')3(1.*#,/*)/-/..#,9

peter.memgel@siemens.com AMAA 2006 20

Ongoing Camera Enhancements

"Improve detection and enable classification:

increase image resolution to 64*8 pixels
"For detection of dark obstacles at higher distances:
increase sensitivity by lowering NEP
">)-,/#./*3'.&#)-/*,#)2/*<9*7/#).*(+*#3#4&'"/
.'2)#$*#--%7%$#&'()
">74$/7/)&*$#./,*7(3%$/.*1'&0*0'20/,*(%&4%&
/)/,29*#)3*0'20/,*,/4/&'&'()*+,/@%/)-9*AB6*CDEF
#)3*-%.&(7'E/3*(4&'-.*1'&0*$(1*+:)%7</,
"!)0#)-/*./).(,*4#-C#2/*+(,*-#,*/)"',()7/)&

peter.memgel@siemens.com AMAA 2006 21

Principles of Image Processing

Depth Image
• Goal: Reliable and predictable 3D
position and velocity estimation for
traffic objects (VRU’s, vehicles,
obstacles)
• Sources of information:
– Depth image:
Intensity image • Segmentation
• Object classification
• 3D position and motion prediction
(Kalman filter)
– Intensity image:
• Confidence estimate
(images composed of
line measurements)
peter.memgel@siemens.com AMAA 2006 22
Discussion

Thanks for your attention

peter.memgel@siemens.com AMAA 2006 23