Antony Lam

We robustly estimate heart rate from the face using a consumer camera under changing light spectra. We show conditions under which linear ICA can be used to extract the cardiac pulse using pairs of face regions.

We consider the sparsity across the spatial-spectral domain, high correlation across spectra, and non-local self-similarity over space to denoise hyperspectral images, resulting in state-of-the-art performance.

We separate fluorescent and reflective components in the spectral domain using only a single hyperspectral image. We first mathematically designed optimal illumination spectrum for the task and found that an off-the-shelf lamp is both cheap and effective. In addition, a fast linear separation algorithm was developed. We validate the proposed system in simulation and with real images.

We make use of a ring of complementary lights to achieve photometric stereo for non-Lambertian surfaces.

We exploit fluorescence to allow for single shot interreflection removal on object surfaces. We show complex shapes can be painted with consumer fluorescent paint to allow for direct and indirect lighting to be separated in scenes with complex dynamic motions.

Separation of Direct Illumination from the Light Source and Indirect Lighting for a Moving Piece of Cloth

In previous work, we used specialized cameras and lighting, which allows for high accuracy. This work aims to make reflective-fluorescent spectral imaging for scenes more accessible by using a conventional RGB camera with colored active lighting. We show that although accuracy was slightly degraded, we still achieved accurate recovery of all three spectral components from our conventional RGB camera.

We use light spectra that are complementary to each other and have a high frequency in the spectral domain to extract three types of hyperspectral images from reflective-fluorescent scenes. These correspond to the reflectance spectra and fluorescence emission and absorption spectra. (All crucial components for fully modeling the color characteristics of the scenes.)

The spectral reflectance of a scene provides a wealth of information for tasks ranging from color relighting to recognition. In this paper, we present an active lighting based spectral imaging method that is accurate and highly robust to unknown ambient light. This is achieved through the use of "basis lights" (light spectra that are analogous to basis vectors). We show what kinds of basis lights are optimal and that they can be well approximated in real life (even when negative values are present in a basis).

The spectral data of a scene allows for highly accurate color relighting of a scene. However past methods for capturing spectral data do not consider flourescent effects. This is despite the presence of fluorescence in many everyday objects. We present a method for efficiently capturing reflective and fluorescent spectral data of entire scenes and show our method produces color relighting results that are very close to ground truth.

Try Our Matlab Demo

Hyperspectral images are useful for a wide variety of tasks ranging from satellite imaging to medical imaging. Unfortunately, the narrowband images that make up such images can be noisy. We show that by explicitly considering the statistics of the spectral domain, hyperspectral images can be denoised with state of the art results using a very simple method.

Code

Flowchart of Our Method

Noisy and denoised images from the 500 nm band of a hyperspectral image.

We developed a system for automatic counting of fly spot cards (a tool used in entomology studies for measuring fly populations). Fly spot cards are placed in farms and as flies land on them, they leave their waste on the cards. By observing the amount of waste, one can infer trends in fly populations. (More waste means more flies were present.) Our automatic fly spot card counter was found to have a high correlation to human expert counts in an extensive field test on large dairy operations.

Example images of scanned cards with fly waste and annotation of spots found by our algorithm. (Click to enlarge.)

Farms where tests were conducted.

Relevance feedback (RF) allows a user to more precisely define what it is that they want to retrieve from a database. In this work, we reduce the amount of user feedback needed through the use of transfer learning. Our main contribution is the difficult problem of deciding what source knowledge to use in order to make transfer learning effective for RF. We test our method using a combination of YouTube videos of our own and the YouTube Action Dataset.

For example, if the user wanted to use RF to teach the computer what basketball videos are, we could automatically select source knowledge about volleyball. This would reduce the number of training examples needed to learn what basketball videos are.

Some sample clips of videos for testing retrieval of activities.

Through the use of SVMs, we learned the relations between face patches over pose. We were then able to perform face recognition over rotations as large as 90 degrees. In addition, we also propose a face alignment method based on SVMs.

Example patches used in our system.

Face Poses in the CMU PIE Database

This work was mainly an application for textiles research. The goal was to predict certain charateristics of how a piece of fabric given its mechanical properties would drape over a rigid surface. The particular characteristics were the "drape coefficient" and "circularity" which are measurements obtained from a device called the Cusick Drape meter. Two neural network architectures were compared in their ability to learn the drape coefficient and circularity given seven mechanical properties.

Cusick Drape Meter

Image of a piece of fabric being measured with a Cusick Drape meter.