HYPERSPECTRAL

IMAGING
2015 ACTIVITY UPDATE

Introduction
January 2015

ASPIRE | INVENT | ACHIEVE

PUBLIC IMEC 2015

Organic
solar
200mm
cell
line
imager
Silicon
line
solar
cell line

IMEC MISSION

NERF
lab

300mm
pilot
line

Nano
Bio labs

450mm
ready

Netherlands

World-leading research in
nano-electronics

>1B cumulated investment

on-site in leading-edge
semiconductor factories

Scientific knowledge with

innovative power of global
partnerships in ICT,
healthcare and energy

Focus on industry-relevant
technology solutions

International top talent in

a unique high-tech
environment committed to
provide building blocks for a
better life in a sustainable
society

Belgium
China

office
USA

office Japan
India

ASPIRE | INVENT | ACHIEVE

Taiwan

PUBLIC IMEC 2015

HISTORY
Established by state
government of
Flanders in 1984
Independent non-profit
organization
Worldwide reach
Initial investment: 62M

Initial staff: ~70

ASPIRE | INVENT | ACHIEVE

PUBLIC IMEC 2015

FACTS & FIGURES

Total revenues in 2014 of
330M, growth of 5%
- 80% industry vs. 20% funded

132 patents awarded & 133

patents submitted in 2011
Collaboration with 600
companies & 208 universities
600 residents on-site from
industrial partners

ASPIRE | INVENT | ACHIEVE

PUBLIC IMEC 2015

HEADCOUNT

Top 5 foreign employees

1. The Netherlands (# 159)

May 2013:

2. China (# 87)

2,100+ employees

4. India (# 76)

3. France (# 86)
5. Japan (# 61)

1,077 Belgians
72 foreign nationalities

159

160

140

120

100

87

86

76

80

59 61

60
46
40

43
31

30
18

16

20
1 1 3 1 2 2

7 7 5

1 1

4 4

13
1 1 2

13
3

28

25

1 1 2

4
4
1 3 1 1 1
1

10

10
2

15
8

5 6

12
1 2

4 2

1 1

ASPIRE | INVENT | ACHIEVE

PUBLIC IMEC 2015

200mm
line

Silicon
solar cell
line

Nano
biolabs

Organic
solar cell
line

NERF
lab

300mm
pilot line

450mm
ready

STATE-OF-THE-ART
Research to Production FACILITIES
ASPIRE | INVENT | ACHIEVE

PUBLIC IMEC 2015

IMEC TOWER
Expansion of 14,208 m2
16 floors capacity of 450 people & lab space
ASPIRE | INVENT | ACHIEVE

PUBLIC IMEC 2015

GLOBAL NETWORK OF SEMICONDUCTOR PARTNERS

Memory IDM

ASPIRE | INVENT | ACHIEVE

Logic IDM

Foundries

PUBLIC IMEC 2015

Fablite

Fabless

IMEC ECO-SYSTEM FOR NANOTECH

Core CMOS (22nm and below)
Lithography

Devices

Interconnects
GO OPTICAL

CMORE

Organic

Photonic, MEMS,
3D integration

electronics

Sensor
systems

Si PHOTONICS

Energy

Life-Sciences

Green Radio

Imaging

Photovoltaic
Power devices
LEDs

Microfluidic
Body Area
Networks

Low power RF
communications

Image sensors
& Vision systems

ASPIRE | INVENT | ACHIEVE

PUBLIC IMEC 2015

for industrial
applications

IMAGE SENSOR & VISION SYSTEM ACTIVITIES

Line scan

TDI in
CMOS

Snapshot
Highspeed

Large
Area

HYPER
SPECTRAL

LIDAR

IMAGING
IMAGING

UV / EUV

CMOS
IMAGE
SENSOR

THZ/MMW

PLATFORMS
ultra-sound
(MEMS based)

NON
VISIBLE
IMAGING

Colony
inspection

LENS-FREE
IMAGING

ORGANIC
IMAGERS

Polymer
photo-diodes
& full imagers

ASPIRE | INVENT | ACHIEVE

STEREO
VISION
IMAGING

3D stereo
vision

Cell
sorter

Industrial

PUBLIC IMEC 2015

10

WHY DO WE NEED HYPERSPECTRAL IMAGING?

1
2

ASPIRE | INVENT | ACHIEVE

PUBLIC IMEC 2015

to improve vision
and discrimination
power ...

... by using
spectral
dimension of
objects being
imaged

11

SPECTRAL IMAGING OPEN ONE NEW DIMENSION

COLOR IMAGING

SPECTROSCOPY

Accurate spectral analysis

of one spatial pixel only
ASPIRE | INVENT | ACHIEVE

R
y

HYPERSPECTRAL IMAGING

Seeing RGB colors

of one image only
PUBLIC IMEC 2015

spectral signature images revealing

objects chemical composition
12

wavelength dimension for each pixel ...

x

ASPIRE | INVENT | ACHIEVE

PUBLIC IMEC 2015

13

CONVERGENCE OF IMAGING & SPECTROSCOPY

SPECTROSCOPY

High discrimination power

Single point
with highest
spectral
resolution

IMEC approach: spectral filters on multiple pixels

Mosaic pattern OR

Line pattern

RGB filters
on multiple pixels

IMAGING
SPECTROSCOPY

High speed, high spatial resolution

ASPIRE | INVENT | ACHIEVE

Tiled pattern OR

PUBLIC IMEC 2015

IMAGING
14

SPECTRAL FILTERS APPLIED TO IMAGE SENSOR

WAFERS BY SEMICONDUCTOR PROCESS

IMEC hyperspectral filter structures processed at wafer-level on top of commercial

CMOS image sensor wafer (here on CMOSISs CMV2000 & CMV4000 sensors)

ASPIRE | INVENT | ACHIEVE

PUBLIC IMEC 2015

15

We design & manufacture spectral filters directly onto

image sensor pixels ...

ASPIRE | INVENT | ACHIEVE

... based on CMOS compatible

semiconductor process

PUBLIC IMEC 2015

16

IMEC APPROACH FABRY-PEROT SPECTRAL FILTERS

L

Wavelength selection depends on cavity length L

k 2nL cos

Different cavity heights = different

spectral wavelengths captured!

Narrow-band & high transmission

efficiencies spectral filters
Tx (%)

FWHM ~
5-20nm

ASPIRE | INVENT | ACHIEVE

PUBLIC IMEC 2015

17

CUSTOM HYPERSPECTRAL IMAGING SOLUTIONS

Filter response

Spectral filters can be integrated

onto commercial image sensor wafers

FWHM
OD

Optical filters & imager can

be customized to match final
application requirements

Fill the sensor ONLY with

selected bands of interest
Continuous
Line-scan
ASPIRE | INVENT | ACHIEVE

PUBLIC IMEC 2015

Custom
Line-scan

Snapshot
Tiled

Snapshot
Bayer Mosaic
18

OFF-THE-SHELF SPECTRAL IMAGING SENSORS

RGB + NIR

RGB

70+
commercial
vendors
ASPIRE | INVENT | ACHIEVE

B NIR

(in development)

Multi Spectral

400nm

750nm

Snapshot mosaic & tiled

16 / 25 / 32 bands

PUBLIC IMEC 2015

Hyper Spectral

400nm

750nm

Linescan
100+ bands

19

ROADMAP FOR OFF-THE-SHELF HSI SENSORS

Snapshot
RGB+NIR

Snapshot
MOSAIC

NIR
VIS
GEN1 470-630nm
(4x4 =16bands)

Snapshot
TILED

VNIR

GEN2 600-1000nm
(5x5 = 25bands)

GEN3 420-1000nm
(XX bands)

VNIR

NIR

GEN2 420-1000nm
(XX bands)

GEN1 600-1000nm
(32bands)

LINESCAN
Wedge

NIR

VNIR

GEN1 600-1000nm
(100+bands)

2012
ASPIRE | INVENT | ACHIEVE

2013

GEN2 470-900nm
(150+bands)

2014
PUBLIC IMEC 2015

2015

VNIR

GEN3 420-1000nm
(XX bands)

2016+
20

LINE-SCAN HSI SENSOR DESIGN (GEN 1)

CMOSIS CMV2000
8 pixels 1000nm
8 pixels 996nm
8 pixels 992nm

.................
...

1088 pixels

...

100+ spectral bands

of 2048x8 spatial pixels each

8 pixels 608nm
8 pixels 604nm
8 pixels 600nm

2048 pixels

Key specifications
-

Spectral resolution: 128 bands in 600-1000nm

FWHM: ~ 10-12nm

Spatial resolution: 2048 pixels x length of scan

Speed: up to 340 fps (full sensor frame)

ASPIRE | INVENT | ACHIEVE

PUBLIC IMEC 2015

21

SNAPSHOT TILED HSI SENSOR DESIGN

CMOSIS CMV2000

1088 pixels

1 tile = 256x256 pixels

792nm

582nm

775nm

761nm

841nm

874nm

630nm

899nm

820nm

833nm

725nm

688nm

801nm

865nm

920nm

885nm

700nm

715nm

644nm

595nm

895nm

911nm

606nm

905nm

676nm

856nm

656nm

621nm

557nm

907nm

737nm

600nm

2048 pixels

Key specifications
-

Spectral resolution: 32bands in 600-1000nm with 12nm incremental steps (optical duplicator needed)

FWHM: ~ 10-15nm

Spatial resolution: 256x256 for each spectral band

Speed: up to 340 data-cubes / s (max sensor limit)

ASPIRE | INVENT | ACHIEVE

PUBLIC IMEC 2015

22

SNAPSHOT MOSAIC HSI SENSOR (GEN1)

4x4 mosaic = 16 bands

1088 pixels

CMOSIS CMV2000

465
nm

546
nm

586
nm

630
nm

474
nm

534
nm

578
nm

624
nm

485
nm

522
nm

562
nm

608
nm

496
nm

510
nm

548
nm

600
nm

4x4 mosaic

2048 pixels

Key specifications
-

Spectral resolution: 4x4 mosaic (1filter / pixel) = 16bands in 470-630nm

FWHM: ~ 15nm

Spatial resolution: from 512x272 (RAW per band)

Speed: up to 340 data-cubes / s (max sensor limit)

ASPIRE | INVENT | ACHIEVE

PUBLIC IMEC 2015

23

OFF-THE-SHELF HSI SENSORS OFFERING

LINESCAN

Snapshot TILED

wedge design

chess-board design

Or

SCANNING
needed
HIGHEST spatial &
spectral resolution

ASPIRE | INVENT | ACHIEVE

Snapshot MOSAIC
per-pixel design

Or

NO SCAN

NO SCAN

Spatial vs. Spectral

resolution trade-off

Spatial vs. Spectral

resolution trade-off

Optical duplicator
needed

NO optical duplicator

PUBLIC IMEC 2015

24

bringing hyperspectral
imaging ...

... to machine
vision industry!
ASPIRE | INVENT | ACHIEVE

PUBLIC IMEC 2015

25

COMMERCIAL PARTNERS FOR HSI SENSOR CAMERA INTEGRATION

Intelligent
camera platform

High speed camera

Cameralink or CoaXPress interface
Compact optical design with ZIFF
socket to swap different HSI sensors
Robust camera implementation for
use in industrial, medical and global
security

Intelligent vision module

Smart camera

Giga/Ethernet
interface
Focus on remote
sensing, UAV &
machine vision

Giga/Ethernet interface
Reprogrammable FPGA
pre-processing hypercube
data inside camera
Focus on industrial &
machine vision

Giga/Ethernet interface
Motorized optics
Embedded HSI data
processing include storage
Multi-camera head option

Handheld mobile device

USB 3.0 vision camera

High-speed USB3.0 camera

interface
Ultra compact camera:
26.4 x 26.4 x 21.2 mm
27grams weight only!

ASPIRE | INVENT | ACHIEVE

PUBLIC IMEC 2015

First proof-of-concept of
IMECs HSI sensors
integration to mobile
Android OS open platform
USB2.0 interface

26

ASPIRE | INVENT | ACHIEVE

PUBLIC IMEC 2015

27

NEW EVALUATION KITS BASED ON USB3.0 CAMERA

ASPIRE | INVENT | ACHIEVE

PUBLIC IMEC 2015

465nm

510nm

548nm

600nm

474nm

522nm

562nm

608nm

485nm

534nm

578nm

624nm

496nm

546nm

586nm

630nm

28

DRIVING APPLICATIONS FOR IMEC HSI

Remote sensing
UAV/drones & nano-sattelites for:
Precision agriculture
Environment monitoring
Terrestrial / maritime earth
observation

Machine vision /
Optical sorting

Life-science / spectroscopy
instrumentation

Imaging spectroscopy analyzers

DNA sequencers / flow cytometers
Water monitoring analyzers
Blood / urine analyzers

Automotive & Transport

Food sorting / quality grading
Pharmaceutical defect inspection
Industrial inspection (plastic,
ceramic, glass, etc...)
Robotic machine vision
Mining / Mineralogy
Print quality inspection

Night vision systems

Fuel monitoring systems

Security / Surveillance

Medical imaging

ASPIRE | INVENT | ACHIEVE

Surgery-guided imaging
Fluorescence microscopy
Endoscopy
Ophthalmology / retina imaging
Wounds imaging

PUBLIC IMEC 2015

Industrial gas leaks

monitoring
Intrusion detection /
authentication
Rescue
Forensics

29

portable | compact | ready-to-go!

ASPIRE | INVENT | ACHIEVE

PUBLIC IMEC 2015

30

... integration path

into UAS / UAVs

ASPIRE | INVENT | ACHIEVE

PUBLIC IMEC 2015

31

Vegetation monitoring
applications
Genotyping
Phenotyping
Biodiversity
Invasive species
Environmental stress
Climate change

RGB | Hyperspectral (4x4 mosaic VIS)

Cropping area under the Field Phenotyping Platform (FIP) of the Group of Crop Science
at the ETH Research Station for Plant Sciences Lindau-Eschikon, ETH Zrich
ASPIRE | INVENT | ACHIEVE

PUBLIC IMEC 2015

32

LINESCAN SENSOR IN PRECISION AGRICULTURE

NIR

RAW image

862 nm
2.5cm ground
resolution!

ASPIRE | INVENT | ACHIEVE

spectral image

NDVI (vegetation index)

vegetation
index map

PUBLIC IMEC 2015

management
zone map

33

snapshot hyperspectral
imaging in medical
ASPIRE | INVENT | ACHIEVE

PUBLIC IMEC 2015

34

ATM / BANK NOTE INSPECTION

Hyperspectral (5x5 mosaic NIR)

___________

RGB

Courtesy of Perception Park Studio

ASPIRE | INVENT | ACHIEVE

PUBLIC IMEC 2015

35

LINESCAN SENSOR FOR FOOD CONTAMINATION

raspberry
plastic back
plastic front
teflon back
teflon front
rubber
dark rubber
staples
dish bg

2.5

x 10

1.5

teflon_with_powder

teflon_front

teflon_back

staples

rubber

raspberry

1
plastic_front

plastic_back

0.5
dish_bg

dark_rubber

50

100

150

200

250

300

350

400

450
black_bg

Producer of fruit pieces for the global snack food market

Hyperspectral imaging allowed to detect various contaminants
ASPIRE | INVENT | ACHIEVE

PUBLIC IMEC 2015

36

LINESCAN HSI SENSOR FOR RICE INSPECTION

Classification Training Set

Training Set Hyperspectral signatures

500

dark

good

black

450
400

good
samples

350

black

300
250

dark

200
150
black_sample
dark_sample
background_dish
good_wholegrain

100
50
0
600

Objectlevel classification
+ standard vision software processing

650

700

750
800
850
wavelength (nm)

900

950

1000

Principal Component Analysis

black spot

good_wholegrain

good

dark

dark_sample

1800
1600

2000
black_sample

1400

2200

background

2400
background_dish

1000

2600
2800

800
3000

ASPIRE | INVENT | ACHIEVE

PUBLIC IMEC 2015

1200

600

37

HANDHELD APPLICATIONS TARGETED BY IMEC HSI

Cosmetic / Skin tone measurement

Food quality grading (e.g. sugar content

(e.g. make-up advice, etc...)

in fruits, freshness of food, calories content, etc...)

teflon_with_powder

teflon_front

teflon_back

staples

rubber

raspberry

plastic_front

plastic_back

dish_bg

dark_rubber

black_bg

Source LOREAL

Anti counterfeiting
spectral tag readers
Skin-care / personalized
medicine (e.g. melanoma, diabetic
ulcers, wound care...)

Colors white balancing

(e.g. display screen, camera pictures)

ASPIRE | INVENT | ACHIEVE

PUBLIC IMEC 2015

38

HSI SOFTWARE IMAGE PROCESSING PIPELINE

HSI
cube

Camera calibration,
registration, synchronization,
reflectance calculation...etc

Pre-Processing

Spectral unmixing,
smoothening, PCA, SVM, kmeans clustering, ...etc

Feature
Extraction &
Selection

Camera/Sensor
Specific

Feature
Classification/
Recognition

Post-Processing

Application Specific

Application Specific

IMEC can support these

SOFTWARE layers

ASPIRE | INVENT | ACHIEVE

Visualization, GUI... etc

(to be developed for each platform)

PUBLIC IMEC 2015

39

ASPIRE | INVENT | ACHIEVE

PUBLIC IMEC 2015

40