ResearchResearch Interests | Current Research Questions | Research Projects | Students Research Interests - Computational Photography & Illumination
- Computer Vision and Interactive Computer Graphics
- High Dynamic Range (HDR) Images and Image Processing
- Digital Archives of Visual Appearance for Museum Collections
- Human Visual Perception, Psychophysics, and Physiology
Current Research Questions - How can we enrich digital photography? How can we attach machine-readable meaning to pixels?
- What does direct visual examination of an object give us that current photographs lack?
- What could we do to capture most of a photo’s visually meaningful contents in machine-readable form?
- What is the best way to ‘lash together’ multiple photos of an object to form a unified visual archive?
- How can we make a quick, low-cost, but complete archival record of a museum object?
- Can we make such archives expandable? device-independent? View independent? Lighting-independent?
- Can these archives help us find hidden object features? Search for similarities among large sets of objects?
- Can they help us detect subtle long-term change & degradation of objects? Identify lost or stolen objects? Detect fakes?
- What digital tools will help museum officials protect, share, and explore their collections with visitors?
Research Projects ![](images/projects/1-gradientcam.jpg)
| "Why I Want A Gradient Camera” Can a spatio-temporal gradient-sensing camera with Nystagmus provide high-dynamic-range sensing, super-resolution, velocity compensation, and variable-time video, all at once? Though it would require substantial computation (Poisson Solver, etc.) to convert its output to a displayed image, the apparent advantages appear huge, and commodity programmable graphics hardware can perform these calculations in a fraction of a second. Let's simulate such a sensor to explore its advantages, difficulties, and feasibility. (Early collaboration interest from Ramesh Raskar and others at MERL, and Fredo Durand, MIT LCS/AI) | ![](images/projects/2-4dloop.gif)
| 4-D Loop Textures for Metric Light Fields and IBMR Archives (Jack Tumblin with Prasun Choudhury and Sangwon Lee) Problem: The 2-plane parameterization of the 4D light field has elegant storage and rendering properties for hardware, but its field of view is limited, angularly non-uniform, and has lowest angular resolution 'straight ahead'. Sphere-based alternatives (e.g. Camahort 98, others) offer uniformity, but conversions between storage address and 4D angles & positions is complex. Multiple-baseline stereo reconstruction for either form can require extensive interpolation. Solution: A uniform grid of torii offers a much simpler, uniform and complete parameterization of the 4D set of rays that pass through a planar rectangular window. With its intrinsic epipolar geometry, simple interpolation, and full spherical field of view, 4D loop textures are well suited for optical shape description as well as 'surface light fields'. Assembled from interleaved orthographic images (instead of perspective images for the 2-plane method) it offers direct measurements of objects for museum curators. (Collaboration interest from Dr. Nancy Pollard, CMU for use in robotic grasp planning, and Dr. Phil Dutre' at KU Leuven (Belgium) for use in 4D lighting/6D reflectance field capture). | ![](images/projects/3a-trilateral.jpg) ![](images/projects/3b-trilateral.jpg) ![](images/projects/3c-trilateral.gif) ![](images/projects/3d-trilateral.gif)
| The Trilateral Filter for High Contrast Images and Meshes Eurographics Symposium on Rendering 2003 pp 286-296, pg 310 (Jack Tumblin with Prasun Choudhury) Problem:How can we keep perceptually important 'sharp edges' or 'boundaries' in a signal, yet smooth away noise or 'small details'? How can we apply it to a high contrast image, a volumetric data set, a surface (2D manifold embedded in 3D)? Solution:We present a new, single-pass nonlinear filter for edge-preserving smoothing and visual detail removal for N-dimensional signals in computer graphics, image processing, and computer vision. Built from two modified forms ofTomasi & Manduchi's 'bilateral filter'(1998), the 'trilateral' filter smoothes signals towards a sharply-bounded piecewise linear approximation. We show it provides excellent results for many visual applications including contrast reduction, motion prediction, and de-noising polygonal meshes. | ![](images/projects/4-tonemapping.jpg)
| Artistic Guidance for Tone Mapping Algorithms (Jack Tumblin with Elaine Tam and Dan Cogswell) (UNPUBLISHED) Problem: Computer graphics solutions to 'tone mapping' (rendering high contrast scenes on low contrast displays) have focussed on artifact-free detail-preserving contrast reduction. But what other algorithmic methods can artists teach us for beautiful renderings of high contrasts? Solution: We made a high-contrast test scene and a comfortable environment for artists to make their best renderings in several media. They discussed their work with us, and we made quantitative comparisons against floating-point images. We described 5 new methods that can improve existing or new tone mapping algorithms. | ![](images/projects/5a-bixels.gif) ![](images/projects/5b-bixels.gif)
| Bixels: 8-Bit Encoding of Sharp Image Boundaries (Jack Tumblin with Prasun Choudhury) Problem: Boundaries and pixels are almost antithetical--humans see sharp edges positioned with subpixel accuracy, but extracting them from a grid of pixels is difficult. Humans make partial geometric estimates from sensed edges, but no geometry is available in a grid of pixels. Boundaries are needed for image enlargement, typography, disambiguation of transparency and partial occlusion, and more. Solution: 'Bixels' directly encode infinitely sharp discontinuities in intensities and/or gradients in an image with sub-pixel precision, adding just 8 bits per pixel, as if boundaries were a color. Using a switched bilinear kernel, bixel images can be enlarged endlessly without blurring boundaries. A novel 2-stage interpolation scheme converts bixels to pixels without unnecessary blurring, and a pixel image with known boundaries can be converted to bixels iteratively. OpenGL texture mapping hardware can display bixel images. | ![](images/projects/6a-super4d.jpg) ![](images/projects/6b-super4d.jpg)
| Smart 4D Lighting For Museum Curators' IBMR Archiving Tasks (Jack Tumblin with Xing Hu, Ankit Mohan, Pin Ren) Problem:Good lighting doesn't happen by accident, and can be quite troublesome and time consuming to achieve. Deeply shadowed regions, highly specular surfaces, and self-occlusions can easily hide historically important details such as the hand engraved annotations on historical instruments. Solution: Computer-controlled lighting devices that interactively adjust the 4D incident light field can quickly achieve good lighting for any camera placement; allowing museum curators to spend their time selecting important features. When combined with measurements of the controllable lights, these images also reveal partial information about the object's shape and surface reflectances to accompany the photos. Partial Image of globe:"The Universe Unveiled: Instruments and Images through History", Stephenson et al., AdlerMuseum, ChicagoIL (c)2000, pg 14 | ![](images/projects/7-robust.jpg)
| Robust Matting: Drifting Cameras, Lights and Backgrounds (Jack Tumblin with Pin Ren) (UNPUBLISHED) Problem: For good results, two-color matting (Smith,Blinn, SIGGRAPH'96) and environment mattes (Zongker et. al SIGGRAPH'99,2000) requires at least 2 pairs of perfectly matched photographs taken with and without the foreground object. Mismatches are often hard to avoid if photo pairs are not taken immediately; temperature change, camera or background movement, camera changes from power cycling, 'auto' white-balance, etc. Solution: Estimation of changes across all images, not just image pairs, can provide robust separation of foreground, background, and alpha channels. Background Image Credit:"Pyramid" (c)1995 VJacobsen, from http://www.afh.com/co220/assign6.html | ![](images/projects/8-residues.jpg) | "Residues" for Antialiasing for Pixel Intensity Distortions (Jack Tumblin with Josh Anon) Journal of Graphics Tools Problem: Antialiased images are delicate; almost any localized or non-linear change in pixel intensities can cause new aliasing . For example, distorting pixels by a threshold function (e.g. f(I) = 0.2 if I<0.5, else f(I) = 0.8) will add jagged edges to even the most smoothly rounded shadings. Conventional antialiasing methods can remove these artifacts, but require two filtering steps that blur the image everywhere, even in regions where no aliasing occurred. Solution: We suppress aliasing by adding "residues" to distorted pixels, remainders that never blur needlessly: alias-free pixel distortions produce zero-valued residues. Residues can improve the final appearance of any nonlinear intensity-modifying operation on images, including gamma correction, NPR processes, and global or local tone mapping. | ![](images/projects/9-exact2d.gif)
| Exact 2D Integration inside Quadrilateral Boundaries (Jack Tumblin) Accepted, Journal of Graphics Tools Problem: Integrating any arbitrary 2-D polynomial over arbitrary quadrilateral boundaries can be messy to implement in imperative languages such as C/C++. The obvious approach requires substitutions and symbolic manipulations that are more trouble than they're worth. Solution: If we write these polynomials in matrix form, we can compute the integrals by convolution, element-by-element multiply, and sum operations. The solution is straightforward to implement, suitable for parallel implementation on graphics hardware (Cg, etc.) and can aid in many graphical tasks. Extensions to tensors handle a limited class of curved boundaries as well. | ![](images/projects/10a-ibmr.jpg) ![](images/projects/10b-ibmr.jpg)
| IBMR 'Starter' Code for Windows (Jack Tumblin) Problem: Image-based Modeling and Rendering techniques often have high entry barriers--student projects can get lost in user-interface, file I/O, and texture map details instead of the IBMR topics they wish to study. Solution: Working C++ code that encapsulates most common components used in IBMR work, including 'arcball' camera positioning, animation timing, encapsulated lighting, texture-mapped 3D meshes and transparency, file I/O, on-screen text, labeled 3D axes, rays, planes. etc. | ![](images/projects/11-pointimage.gif)
| FPview: A Data-Flow Testbed for Floating Point Image Computing (Jack Tumblin) Problem: Existing image editing software (PhotoShop, etc.) designed for fixed-point (8- or 16-bit) cannot manipulate high dynamic range (HDR) images accurately. Long, repeatable sequences of numerical manipulations (+/-, log/exp, median,diffusion, etc.) are also awkward especially with adjustable constants within the sequence. Solution: FPview interactively computes a cascade of floating-point images (32 bits/color). Each window holds one floating-point image result, either loaded from a file or computed from the contents of other windows. Changing any window contents triggers re-computation of all other windows that depend on it. New window-computing functions are easily added for students' experimental work. | ![](images/projects/12-implicit.jpg) | Implicit Surface modeling with 'Sticky and Repulsive' Primitives (Jack Tumblin with Alison Ortony, Ankit Mohan) Problem: Until recently, many implicit surface primitives were blob-like expansions of points or central skeletons; blobs placed too close together would stretch towards each other, merge, and change topologies. 'Variational Implicits' offer implicit surfaces that exactly interpolate given points with a global minimum-curvature surface. However, local shape editing was difficult--analogous to squeezing a balloon into the shape of a horse. Also, variational (and related radial basis fcns and partition-of-unity) implicits require matrix solvers to determine coefficients for the surface, setting a practical limit on the number of points in the surface. Solution: we developed a nonlinearly-combined implicit surface primitive that exactly interpolates points, offers local control of shape, gradient, and curvature suitable for interactive adjustment, requires no matrix solver, and has both repulsive and attractive('sticky') properties that allow modeling of complex folded shapes such as the surface of the human brain. | ![](images/projects/13-highcontrast.jpg)
| High Contrast Reference Display Device (with Steve Gordon, Dai Kim, Sameer Mathur, and Mike Worth) Problem: Recreating a repeatable, reliable high-contrast image or scene (>5000:1) can be difficult, even in a lab setting. Conventional displays have limited contrast (<100:1 typ.), and high contrast scenes aren't portable and may change due to interreflections with surroundings. Solution: Inspired by Greg Ward (Larson)'s high-contrast viewer, students designed this fully enclosed, recessed multi-layer back-lit film display with stablized cooling to stop temperature drift. Provides high contrast images for unlimited time (hours), with contrasts measured at >10,000:1. An ordinary color printer produces layered display transparencies. | Students - Amy Gooch: Ph.D. scheduled 2006
- Ankit Mohan: Ph.D. scheduled 2007
- Eric Russell: MS scheduled 2006
- Chi-Yin Cheung: MS scheduled 2007
- Xing Hu: MS received 2004
- Alison Ortony: 2001-2002
|