Research
Our research mainly centers around human modeling and behavior analysis; appearance modeling, inverse rendering, and material recognition; and physics-based vision and computational photography. Our research finds applications in AR/VR, graphics, HCI, and robotics, and our current focus is autonomous driving, driving assistance, elderly care, and assisted living. Some of the projects from our research can be found here.
Diffusion Reflectance Map
Single-Image Stochastic Inverse Rendering of Illumination and Reflectance.
Fresnel Microfacet BRDF
THE BRDF model that unifies Polari-Radiometric Surface-Body Reflection.
nLMVS-Net : Deep Non-Lambertian Multi-View Stereo
Multi-view stereo for joint shape and reflectance estimation of complex surfaces.
RGB Road Scene Material Segmentation
Road-scene per-pixel material recognition of ordinary RGB images.
Extrinsic Camera Calibration from A Moving Person
A person is a moving calibration target consisting of easily associative oriented points.
Dynamic 3D Gaze from Afar
Gaze estimation from eye-head-body coordination without relying on the eye appearance.
Multimodal Material Segmentation
Road-scene per-pixel material recognition with multimodal imaging.
Polarimetric Normal Stereo
Binocular stereo of polarization cameras for fine detail surface geometry recovery as per-pixel surface normals.
Shape from Sky
We show that per-pixel surface normals can be estimated from the reflected sky polarization pattern.
Consistent 3D Human Shape from Repeatable Action
Reconstructing a clothed human body shape from videos capturing a few instances of a repeatable action.
Non-Rigid Shape from Water
We introduce a novel underwater 3D sensing method for recovering a consistent, dense 3D shape of a dynamic, non-rigid object.
Dense Video Annotation
We introduce a novel dense video annotation method that only requires sparse bounding box supervision.
Invertible Neural BRDF for Object Inverse Rendering
We introduce a novel BRDF model based on a conditional normalizing flow and a deep illumination prior for single-image joint estimation of reflectance and illumination.
Appearance and Shape from Water Reflection
We show that you can recover 3D geometry with HDR appearance and even calibrate the camera from a single image a scene with water reflection.
Surface Normals and Shape from Water
We introduce a novel method for reconstructing surface normals and depth of dynamic objects in water.
Variable Ring Light Imaging
We show, for the first time, that transient subsurface light transport can be recovered from the steady-state outer appearance of a surface captured with ordinary imaging components.
Wetness and Dry Appearance Recovery
We derive an analytical spectral appearance model of wet surfaces and a method for recovering wetness and dry appearance from a single multispectral image.
Shape from Water
We introduce a novel depth recovery method based on light absorption in water and develop a co-axial camera system that achieves real-time shape recovery of objects with complex appearance in water.
Radiometric Scene Decomposition
We introduce a novel method for radiometric decomposition, the recovery of reflectance, illumination, and geometry, of real-world scenes from a few RGB-D images.
Perceptual Visual Material Attributes
We derive a framework that allows us to discover locally-recognizable material attributes from crowdsourced perceptual material distances. We show that the discovered attributes better delineate material categories and capture semantic material properties, despite being recovered without any semantic supervision.
Visual Material Traits
We propose to represent material categories with their inherent properties exhibited in their looks. These visual material traits, such as shiny and smooth can be recognized accurately at each pixel and provides a discriminative representation for recognizing material categories.
Multiview Shape and Reflectance from Natural Illumination
We derive a probabilistic geometry estimation method that fully exploits the rich signal embedded in multiview images of objects of unknown intricate reflectance captured under known but complex natural illumination.
Shape and Reflectance from Natural Illumination
We introduce a method to jointly estimate the complex reflectance and geometry of an object from a single image taken under known, but uncontrolled, natural illumination.
Reflectance and Natural Illumination from a Single Image
We introduce a method to jointly estimate the complex reflectance of an object of known geometry and the surrounding natural illumination from a single image.
Single Image Multimaterial Estimation
We derive method for estimating the reflectances and a single point source while segmenting the surface into different material regions from a single image of an object made of multiple materials.
Directional Statistics BRDF Model
We introduce a novel parametric BRDF model that can accurately encode a wide variety of real-world isotropic BRDFs with a small number of parameters by devising a novel hemispherical directional statistics distribution.
3D Geometric Scale Space
We derive a sound scale-space representation surface geometry and derive novel scale-dependent/invariant features and descriptors for range image registration and 3D object recognition.
Reassembling Thin Surface Geometry
We present a novel 3D reassembly method for fragmented, thin objects with unknown geometry based on a novel scale-space representation of contour geometry and photometry.
Tracking People in Crowds
We derive a novel Bayesian framework for tracking pedestrians in videos of crowded scenes using a space-time model of the crowd flow.
Anomaly Detection in Crowds
We introduce a novel statistical framework for modeling and detecting anomalous local spatio-temporal motion pattern behavior of crowded scenes (i.e., crowd flow).
Eyes for Relighting
We show how we can compute the illumination surrounding a peron from her eyes and use it to replace the face with another one, while ensuring that the appearance of the new face is consistent with the lighting of the scene.
The World in An Eye
We show that we can recover a wide-angle view of the surrounding, a view of exactly what the person is seeing, and the 3D geometry of the object the person is looking at just from the person’s eye(s) in an image.