Yoshiki Fukao, Ryo Kawahara, Shohei Nobuhara, and Ko Nishino
Kyoto University
We introduce a novel method for recovering per-pixel surface normals from a pair of polarization cameras. Unlike past methods that use polarimetric observations as auxiliary features for correspondence matching, we fully integrate them in cost volume construction and filtering to directly recover per-pixel surface normals, not as byproducts of recovered disparities. Our key idea is to introduce a polarimetric cost volume of distance defined on the polarimetric observations and the polarization state computed from the surface normal. We adapt a belief propagation algorithm to filter this cost volume. The filtering algorithm simultaneously estimates the disparities and surface normals as separate entities, while effectively denoising the original noisy polarimetric observations of a quad-Bayer polarization camera. In addition, in contrast to past methods, we model polarimetric light reflection of mesoscopic surface roughness, which is essential to account for its illumination-dependency. We demonstrate the effectiveness of our method on a number of complex, real objects. Our method offers a simple and detailed 3D sensing capability for complex, non-Lambertian surfaces.
Polarimetric Normal Stereo
Y. Fukao, R. Kawahara, S. Nobuhara, and K. Nishino,
in Proc. of Conference on Computer Vision and Pattern Recognition CVPR’21, Jun., 2021.
[ paper ][ supp. material ][ project ][ talk ]
Talk
Overview
From a pair of polarimetric images from which AoLP and DoLP can be computed for each pixel, we construct a polarimetric cost volume, in addition to a regular RGB cost volume, that measures the Stokes vector discrepancy between that computed from the surface normal estimate and the two observations for a given disparity. We filter these cost volumes, while effectively denoising the input polarimetric observations to estimate the surface normal as well as diffuse and specular albedo values at each pixel.
Our key idea is to formulate simultaneous estimation of per-pixel depth-independent normal and albedo as RGB-polarimetric cost volume filtering. First, let us assume we are given a correspondence, in other words a hypothesized disparity, and introduce a polarimetric cost for the corresponding left right pixel pair. We define the polarimetric loss as the difference between the observed Stokes vectors in the two views and the Stokes vector computed from the surface parameters. By minimizing the polarimetric loss, we can obtain the optimal surface parameters, and define the polarimetric cost as the residual of the polarimetric loss for both views. Given hypothesized disparities for each pixel, we compute the optimal surface parameters and the polarimetric cost for each pixel and disparity value to construct the polarimetric cost volume.
We now have a polarimetric cost volume in addition to a regular RGB cost volume that encodes the color discrepancy for hypothesized disparity values for each pixel. Our goal is to filter these cost volumes to simultaneously arrive at optimal disparity and surface normals. We achieve this with belief propagation while also denoising the noisy quad-Bayer polarimetric observations. Remember that the surface normals are computed from polarimetric observations, not the recovered depth, which is why our method can recover fine detailed surface geometry.
Results
Surface normal and albedo recovery of various real objects with reflection and geometry. The results demonstrate that our method is able to recover the fine geometry of these objects accurately regardless of material composition. As the input AoLP images show, the polarimetric observations are quite noisy. Our method is able to robustly recover the surface normals and albedo values thanks to the denoising integrated in cost volume filtering. Please see supplementary video for more results.
