Digital Refocusing with Incoherent Holography |
Figure 1:
Light field propagation vs. electric field propagation:
(a) Incoherent light
propagation through free space is represented by 4D light fields (in
some literature, propagation is expressed as a shearing of the 4D
light field, which can be represented as a 4D convolution). Light
field cameras capture 4D light fields on 2D sensors, resulting in
loss of spatial resolution. (b) Coherent light wave propagation is
represented by 2D electric fields. The 2D field can be propagated
between parallel planes by a 2D convolution. The coherent blur
encodes depth information and has broadband frequency response. This
enables digital refocusing without loss of spatial resolution.
|
Project Description
Light field cameras allow us to digitally refocus a photograph after
the time of capture. However, capturing a light field results in a
significant loss in spatial resolution. In this paper, we propose
incoherent holography for digital refocusing without loss of spatial
resolution. The main idea is to capture 2D coherent holograms of the
scene instead of the 4D light fields. The key properties of coherent
light propagation are that the coherent spread function (hologram of a single
point source) encodes scene depths and has a broadband spatial frequency
response. These properties enable digital refocusing with 2D
coherent holograms, which can be captured on sensors without loss of
spatial resolution. Incoherent holography does not require
illuminating the scene with high power coherent laser, making it
possible to acquire holograms even for passively illuminated scenes.
We provide an in-depth performance comparison between light field
and incoherent holographic cameras in terms of the
signal-to-noise-ratio (SNR). We show that given the same sensing
resources, an incoherent holography camera outperforms light field
cameras in most real world settings. We demonstrate a prototype
incoherent holography camera capable of performing digital refocusing from only 3 acquired images.
We show results on a variety of scenes that
verify the accuracy of our theoretical analysis.
|
Publications
"Digital Refocusing with Incoherent Holography,"
O. Cossairt, N. Matsuda, M. Gupta,
IEEE International Conference on Computational Photography, 2014.
[PDF]
|
Images
 |
|
Digital Refocusing with a Coherent Spread Functions (CSF):
Here we provide a simple simulation demonstrating how refocusing a hologram is carried out simply as a convolution with a 2D CSF.
The scene consists of three playing cards
placed at different depths. The coherent spread functions (CSF) for
the cards vary in size according to their depths. The CSFs have the
shape of a sinusoidal zone plate. Consequently, high spatial
frequencies are preserved in the hologram even though the maximum blur diameter is nearly half the image width.
In contrast, the incoherent PSFs in conventional imaging act as low-pass filters.
|
|
|
|
 |
|
Prototype Incoherent Holography Camera:
Our prototype camera utilizes a Michelson interferometer setup.
A cube beam splitter divides incoming light from the scene into two beams.
Two mirrors then reflect the beams back towards an objective lens
that images the wavefront onto the sensor.
One of the mirrors has a small amount of curvature to shift the focus of the beam
relative to the other path. A piezo-actuated mirror creates sub-micron
displacements that are used for phase shifting.
The mirrors were both 25.4mm diameter and the objective lens was an
50mm focal length lens (Edmunds Optics 59-873),
resulting in an aperture setting of approximately F/2.
For the sensor, we used a 1/2" monochrome
10 Mpix sensor with 1.67um pixels (Edmunds Optics 86-749).
For the color filter, we used a 10nm bandwidth interference filter
with center frequency 632.8nm (Thorlabs FL632.8-10).
|
|
|
|
 |
|
Incoherent Holography Geometry:
The figure shows the geometric parameters of an incoherent holographic camera. A Michelson
interferometer setup is unfolded and the two paths are shown independently. The key property
is that each point source is divided into two point sources with slightly different radii
of curvature. The two point sources are then recombined and interfere on the sensor plane,
reproducing a Point Spread Function (PSF) very similar to the optical propagator for a coherent optical field (i.e. the CSF).
|
|
|
|
 |
|
Incoherent Holography Capture Decomposition:
The incoherent holography PSF consists of a sum of four different PSFs.
As a result, the captured image is a sum of four different blurred images. Two of the images
are simply blurred by the incoherent PSF from the two point sources in the
Michelson interferometry setup. The other two images are blurred by the CSF. Phase shifting is used
to remove the undesired blurred images.
|
|
|
|
 |
|
Noisy Simulations for Digital Refocusing with Incoherent Holography:
We show that The MTF of an incoherent holography system is inversely proportional to the maximum blur size.
In this simulation, the scene consists of three playing cards placed at
different depths. Gaussian noise is added to captured images with a std-deviation of &sigma=.005.
The camera is focused on the front card (Depth 1).
The scene depths are chosen so that the maximum blur size is 32
pixels (top row) and 64 pixels (bottom row). (a) The three input
phase-shifted images. (b) The recovered holograms. The front card is
in focus, and the other two cards are blurred. (c, d) Images
refocused on the middle card (Depth 2) and the back card (Depth 3),
respectively. Image noise is higher for the larger blur (bottom
row).
|
|
|
|
 |
|
RMSE for Digital Refocusing with Incoherent Holography:
We derive analytic expressions showing that The MTF of an incoherent holography system is inversely proportional to the maximum blur size.
Since the MTF of is flat over the
passband, this means that SNR falls off exactly in the same way, producing significantly greater
SNR than a full resolution light field camera.
To verify this result, we perform simulations with a variety of blur sizes and
add Gaussian noise with &sigma=.005.
As shown in this plot, the increase in noise
is linear, where it is evident that the ratio of RMSE
deblurring error is approximately equal to the ratio of blur sizes.
|
|
|
|
 |
|
Experimental results for
digital refocusing:
A Playing cards scene. Depth range is approximately 75cm and the maximum
blur size is approximately 1200 pixels.
The scene was captured at full 10 Mpix resolution. The exposure time was 1/3 sec
due to the amount of light blocked by the interference filter. The small pixels have a
very small dynamic range, and we averaged 100 frames and used denoising software to increase the SNR to a reasonable level.
The right image shows the coherent field recovered after phase shifting, which is focused on the King in
the background. The center image shows the results after refocusing on the Jack in the foreground.
The Jack is seen to come into crisp focus while the King becomes significantly blurred.
|
|
|
|
 |
|
Experimental results for
digital refocusing:
A Diner scene. The depth range is approximately 50cm and the
maximum blur size is approximately 600 pixels. The right image shows
the coherent field recovered after phase shifting, which is focused on
the diner in the background, while the fence is severely blurred. The high frequencies in the fence
are still visible even with an extremely large blur size. The center
image shows the the fence come into
focus while the diner becomes blurred.
|
|
|
|
|
Acknowledgements
Oliver Cossairt and Nathan Matsuda acknowledge support through
a Samsung GRO grant.
|
Slides
Refocusing Videos for Experimental Scenes
|
When Does Computational Imaging Improve Performance?
Flexible Depth of Field
Shape From Focus
Focal Sweep Camera (Fosera)
|
|
|