Northeastern University, Fall 2005

CSG242: Computational Photography

Ramesh Raskar Mitsubishi Electric Research Labs Northeastern University

Course WebPage : http://www.merl.com/people/raskar/photo/course/

CSG242: Computational Photography

Course WebPage
http://www.merl.com/people/raskar/photo/course

Google phrase Northeastern Computational Photography

Welcome
Understanding Film-like Photography Film Parameters, Nonlinearities, Ray-based concepts Ray-

Image Processing and Reconstruction Tools

Multi-image Fusion, Gradient domain, Graph Cuts Multi-

Improving Camera Performance

Better dynamic range, focus, frame rate, resolution

Future Directions
HDR cameras, Gradient sensing, Smart optics/lighting

Welcome
Topics: Digital Imaging Beyond Film-like Photography Film Computational Aspects of Lenses, Image Sensors and Processing Algorithmic Solutions for Camera Sensor, Lens and Lighting Limitations Adaptive Fusion of Multiple Images for Impossible Photos Image Reconstruction from Coded Images Samples Future Directions in Smart Lighting, Optics and Sensors

Goals
Understand computational aspects of cameras
Discuss issues in traditional cameras Explore alternative imaging methods Learn vision and optics techniques

Discuss image processing and reconstruction tools Review of 30+ recent papers

Goals
What we will not cover
Composition, Aesthetics, Color Film Cameras, film issues Lighting equipment Color issues Traditional image processing/editing (Photoshop)
Histogram, Artistic filters

Digital camera user manual, Storage Novel view rendering (IBR)

Instructor: Ramesh Raskar

Ramesh Raskar is a Senior Research Scientist at MERL.
His research interests include projector-based graphics, computational photography and non-photorealistic rendering. He has published several articles on imaging and photography including multi-flash photography for depth edge detection, image fusion, gradient-domain imaging and projector-camera systems. His papers have appeared in SIGGRAPH, EuroGraphics, IEEE Visualization, CVPR and many other graphics and vision conferences. He was a course organizer at Siggraph 2002, 2003 and 2004. He is a panel organizer at the Symposium on Computational Photography and Video in Cambridge, MA in May 2005. He is a member of the ACM and IEEE. http://www.merl.com/people/raskar/raskar.html

Opportunities
Unlocking Photography
How to expand camera capabilities Digital photography that goes beyond film-like photography film-

Opportunities
Computing corrects for lens, sensor and lighting limitations Computing merges results from multiple images Computing reconstructs from coded image samples Cameras benefit from computerized light sources

Think beyond post-capture image processing post Computation well before image processing and editing

Learn how to build your own camera-toys camera-

Traditional Photography
Detector Lens

Pixels

Image

Computational Photography: Photography:

Optics, Sensors and Computations
Generalized Sensor

Computations
Ray Reconstruction Upto 4D Ray Sampler

Generalized Optics
4D Ray Bender

Picture

Computational Photography
Novel Cameras
Generalized

Sensor
Generalized

Processing

Optics

Computational Photography
Novel Cameras
Generalized

Novel Illumination
Light Sources

Sensor
Generalized

Processing

Optics

Computational Photography
Novel Cameras
Generalized

Novel Illumination
Light Sources

Sensor
Generalized

Processing

Optics

Scene: 8D Ray Modulator

Computational Photography
Novel Cameras
Generalized

Novel Illumination
Light Sources

Sensor
Generalized

Processing

Optics

Display

Recreate 4D Lightfield

Scene: 8D Ray Modulator

Computational Photography
Novel Cameras
Generalized

Novel Illumination
Light Sources Modulators Generalized Optics

Sensor
Generalized

Processing
Ray Reconstruction Upto 4D Ray Sampler

Optics
4D Ray Bender

4D Incident Lighting

4D Light Field Display

Recreate 4D Lightfield

Scene: 8D Ray Modulator

A Teaser: Dual Photography

Projector Photocell

Scene

A Teaser: Dual Photography

Projector Photocell

Scene

A Teaser: Dual Photography

Projector Photocell

Scene

A Teaser: Dual Photography

Projector Photocell Camera

Scene

The 4D transport matrix:

Contribution of each projector pixel to each camera pixel
projector camera

scene

The 4D transport matrix:

Contribution of each projector pixel to each camera pixel
projector camera

scene

Sen et al, Siggraph 2005

The 4D transport matrix:

Which projector pixel contribute to each camera pixel
projector camera

scene

Sen et al, Siggraph 2005

Dual photography from diffuse reflections

the cameras view

Sen et al, Siggraph 2005

Are BOTH of a photograph?

http://research.famsi.org/kerrmaya.html
Rollout Photographs Justin Kerr: Slide idea: Steve Seitz

New Ways of Seeing the World

Multiple-Center-ofMultiple-Center-of-Projection Images Rademacher, P, Bishop, G., SIGGRAPH '98

What rays are most expressive?

Andrew Davidhazy, RIT: RIT:
http://www.rit.edu/~andpph/

Thick photography: interaction

What other ways better reveal shape to human viewers? (Without direct shape measurement? )
Time for space wiggle. Gasparini, 1998.

Can you understand this shape better?

Topics: Digital Imaging Beyond Film-like Photography Film Computational Aspects of Lenses, Image Sensors and Processing Algorithmic Solutions for Camera Sensor, Lens and Lighting Limitations Adaptive Fusion of Multiple Images for Impossible Photos Image Reconstruction from Coded Images Samples Future Directions in Smart Lighting, Optics and Sensors

A Brief History of Images

1558

Camera Obscura, Gemma Frisius, 1558

A Brief History of Images

1558 1568

Lens Based Camera Obscura, 1568

A Brief History of Images

1558 1568

1837

Still Life, Louis Jaques Mande Daguerre, 1837

A Brief History of Images

1558 1568

1837
Silicon Image Detector, 1970

1970

A Brief History of Images

1558 1568

1837

Digital Cameras

1970 1994

Dream of A New Photography

Old
People and Time Each photo Lighting External Sensors Stills / Video Exposure Settings Exposure Time Resolution/noise HDR range ~Cheap Precious Critical No Disjoint Pre-select Pre-select Pre-select Pre-select

New
Precious Free Automated* Yes Merged Post-Process Post-Process Post-Process Post-Process

The digital photography marketplace

90+% of American households own a camera, 20% own a digital camera, 3% own only digital 57 million film cameras sold last year (down 10%) + 100+ million disposable film cameras (up 8%) + 53 million digital cameras (up 15%) + 53 million camera phones (skyrocketing) Surveillance camera, robotics, Webcams ..

Marc Levoy

Digital cameras are boring

roughly the same features and controls as film cameras
zoom and focus aperture and exposure shutter release and advance one shutter press = one snapshot

but things are changing

Marc Levoy

Digital camera technology #1: real-time in-camera processing

fast auto-focus systems
sharp photographs of moving objects

optical image stabilization

long handheld exposures

automatic object recognition

adaptive metering (e.g. of faces)

helps film cameras as well

Marc Levoy

Digital camera technology #2: unusual acquisition protocols

continuous auto-focusing
sequences of sharp photographs of moving objects

large DRAM buffer

16MB on Canon EOS-D30 = 10 JPEG images permits burst-mode photography

free and plentiful permanent storage

2 GB microdrive = 1000 JPEG images permits extended shooting

replacing film cameras for newsgathering

Marc Levoy

Digital camera technology #3: digital post-processing

Photoshop
replacing traditional darkroom techniques also replacing exposure compensation, color filtering, and other specialized shooting techniques

second generation tools

warping images, stitching panoramas will eventually replace the view and panoramic camera

around the bend

high-X imaging (resolution, dynamic range, focus, etc.) techniques based on multiple images
Marc Levoy

Survey
How many of you are photographers ? How many of you are photo-artists ? photo How many of you are digiphoto-artists ? digiphoto How many do active programming ? Field of work: work: Academics? Industry ? Research ? Art ? Brief Introductions

Credit (Tentative)
Assignments: Five project-oriented assignments projectRequires programming in Matlab 8 points each MidMid-term Exam 20 points Term Paper Individual or a group of 2, 8 to 10 pages, 15 minute presentation, demo encouraged 15 points Final Project Individual or in groups of up to 3 20 points Discretionary credit 5 points Each student is expected to prepare notes for one lecture These notes should be prepared and emailed to the instructor no later than the following Sunday (midnight EST). Revisions and corrections will be exchanged by email and after changes the notes will be posted to the website before class the following week. Discretionary credit will be given for this. No late submissions

Structure of class: class:

1st half 6:00PM 7:20PM break 7:20PM 7:40PM 2nd half 7:40PM 9:00PM

Please be on time. Please keep your cell-phones switched off during class hours. cellPlease do not email/IM during class. Every class has a break of 20 minutes for your convenience, please use this time to take/make calls, check emails, etc. Course mailing list: Please make sure that your emailid is on the course mailing list list: csg242@ccs.neu.edu. csg242@ccs.neu.edu. Send email to raskar@merl.com

Instructor: Instructor: Ramesh Raskar, 617-621-7533, raskar@merl.com 617-621Email is the best way to get in touch with me. Office hours: -hours: Location: Location: --