Research

Sample projects from our research.

Dynamic 3D Gaze from Afar

Dynamic 3D Gaze from Afar

Gaze estimation from eye-head-body coordination without relying on the eye appearance.

Project Page

Multimodal Material Segmentation

Multimodal Material Segmentation

Road-scene per-pixel material recognition with multimodal imaging.

Project Page

Polarimetric Normal Stereo

Polarimetric Normal Stereo

Binocular stereo of polarization cameras for fine detail surface geometry recovery as per-pixel surface normals.

Project Page

Shape from Sky

Shape from Sky

We show that per-pixel surface normals can be estimated from the reflected sky polarization pattern.

Project Page

Consistent 3D Human Shape from Repeatable Action

Consistent 3D Human Shape from Repeatable Action

Reconstructing a clothed human body shape from videos capturing a few instances of a repeatable action.

Project Page

Dense Video Annotation

Non-Rigid Shape from Water

We introduce a novel underwater 3D sensing method for recovering a consistent, dense 3D shape of a dynamic, non-rigid object.

Project Page

Dense Video Annotation

Dense Video Annotation

We introduce a novel dense video annotation method that only requires sparse bounding box supervision.

Project Page

Invertible Neural BRDF for Object Inverse Rendering

Invertible Neural BRDF for Object Inverse Rendering

We introduce a novel BRDF model based on a conditional normalizing flow and a deep illumination prior for single-image joint estimation of reflectance and illumination.

Project Page

Appearance and Shape from Water Reflection

Appearance and Shape from Water Reflection

We show that you can recover 3D geometry with HDR appearance and even calibrate the camera from a single image a scene with water reflection.

Project Page

Surface Normals and Shape from Water

Surface Normals and Shape from Water

We introduce a novel method for reconstructing surface normals and depth of dynamic objects in water.

Project Page

Variable Ring Light Imaging

Variable Ring Light Imaging

We show, for the first time, that transient subsurface light transport can be recovered from the steady-state outer appearance of a surface captured with ordinary imaging components.

Project Page

Wetness and Dry Appearance Recovery

Wetness and Dry Appearance Recovery

We derive an analytical spectral appearance model of wet surfaces and a method for recovering wetness and dry appearance from a single multispectral image.

Project Page

Shape from Water

Shape from Water

We introduce a novel depth recovery method based on light absorption in water and develop a co-axial camera system that achieves real-time shape recovery of objects with complex appearance in water.

Project Page

Radiometric Scene Decomposition

Radiometric Scene Decomposition

We introduce a novel method for radiometric decomposition, the recovery of reflectance, illumination, and geometry, of real-world scenes from a few RGB-D images.

Project Page

Perceptual Visual Material Attributes

Perceptual Visual Material Attributes

We derive a framework that allows us to discover locally-recognizable material attributes from crowdsourced perceptual material distances. We show that the discovered attributes better delineate material categories and capture semantic material properties, despite being recovered without any semantic supervision.

Project Page

Visual Material Traits

Visual Material Traits

We propose to represent material categories with their inherent properties exhibited in their looks. These visual material traits, such as shiny and smooth can be recognized accurately at each pixel and provides a discriminative representation for recognizing material categories.

Project Page

Multiview Shape and Reflectance from Natural Illumination

Multiview Shape and Reflectance from Natural Illumination

We derive a probabilistic geometry estimation method that fully exploits the rich signal embedded in multiview images of objects of unknown intricate reflectance captured under known but complex natural illumination.

Project Page

Shape and Reflectance from Natural Illumination

Shape and Reflectance from Natural Illumination

We introduce a method to jointly estimate the complex reflectance and geometry of an object from a single image taken under known, but uncontrolled, natural illumination.

Project Page

Reflectance and Illumination from Natural Illumination

Reflectance and Natural Illumination from a Single Image

We introduce a method to jointly estimate the complex reflectance of an object of known geometry and the surrounding natural illumination from a single image.

Project Page

Single Image Multimaterial Estimation

Single Image Multimaterial Estimation

We derive method for estimating the reflectances and a single point source while segmenting the surface into different material regions from a single image of an object made of multiple materials.

Project Page

Directional Statistics BRDF Model

Directional Statistics BRDF Model

We introduce a novel parametric BRDF model that can accurately encode a wide variety of real-world isotropic BRDFs with a small number of parameters by devising a novel hemispherical directional statistics distribution.

Project Page

3D Geometric Scale Space

3D Geometric Scale Space

We derive a sound scale-space representation surface geometry and derive novel scale-dependent/invariant features and descriptors for range image registration and 3D object recognition.

Project Page

Reassembling Thin Surface Geometry

Reassembling Thin Surface Geometry

We present a novel 3D reassembly method for fragmented, thin objects with unknown geometry based on a novel scale-space representation of contour geometry and photometry.

Project Page

Tracking People in Crowds

Tracking People in Crowds

We derive a novel Bayesian framework for tracking pedestrians in videos of crowded scenes using a space-time model of the crowd flow.

Project Page

Anomaly Detection in Crowds

Anomaly Detection in Crowds

We introduce a novel statistical framework for modeling and detecting anomalous local spatio-temporal motion pattern behavior of crowded scenes (i.e., crowd flow).

Project Page

Eyes for Relighting

Eyes for Relighting

We show how we can compute the illumination surrounding a peron from her eyes and use it to replace the face with another one, while ensuring that the appearance of the new face is consistent with the lighting of the scene.

Project Page

The World in An Eye

The World in An Eye

We show that we can recover a wide-angle view of the surrounding, a view of exactly what the person is seeing, and the 3D geometry of the object the person is looking at just from the person’s eye(s) in an image.

Project Page