Shape from Sky

Polarimetric Normal Recovery Under The Sky
Tomoki Ichikawa^*, Matthew Purri^*, Ryo Kawahara, Shohei Nobuhara, Kristin Dana, and Ko Nishino
Kyoto University, Rutgers University
(^*Equal contribution)

The sky exhibits a unique spatial polarization pattern by scattering the unpolarized sun light. Just like insects use this unique angular pattern to navigate, we use it to map pixels to directions on the sky. That is, we show that the unique polarization pattern encoded in the polarimetric appearance of an object captured under the sky can be decoded to reveal the surface normal at each pixel. We derive a polarimetric reflection model of a diffuse plus mirror surface lit by the sun and a clear sky. This model is used to recover the per-pixel surface normal of an object from a single polarimetric image or from multiple polarimetric images captured under the sky at different times of the day. We experimentally evaluate the accuracy of our shape-from-sky method on a number of real objects of different surface compositions. The results clearly show that this passive approach to fine-geometry recovery that fully leverages the unique illumination made by nature is a viable option for 3D sensing. With the advent of quad-Bayer polarization chips, we believe the implications of our method span a wide range of domains.

• Shape from Sky: Polarimetric Normal Recovery Under The Sky
  T. Ichikawa^*, M. Purri^*, R. Kawahara, S. Nobuhara, K. Dana, and K. Nishino,
  in Proc. of Conference on Computer Vision and Pattern Recognition CVPR’21, Jun., 2021. (^*Equal contribution)
  [ paper ][ supp. material ][ project ][ talk ]

Talk

Overview

We introduced a novel method for recovering surface normals from polarimetric images captured under the sky. The sky, on a clear day, has an angular polarization distribution uniquely defined by the latitude and longitude centered around the sun. The Rayleigh sky model conveniently predicts both the angle of polarization (AOP) and the degree of polarization (DOP) across the sky. We show that this everyday, but special angular polarization pattern readily gives us sufficient incident cues that are modulated by the unknown reflectance and unknown surface normals so that we can robustly decode it to recover spatially varying reflectance and per-pixel surface normals. We conduct a number of experiments using real images taken outdoors with a linear polarizer at the camera or with a quad-Bayer polarization camera. The results clearly demonstrate that our method can recover accurate fine geometry of objects with complex reflectance from a single or a few polarimetric images taken completely passively. The implication of this work is far-reaching. With the advent of polarization cameras using quad-sensor chips, the proposed work gives the ability to perform 3D reconstruction under natural lighting without point matching. Furthermore, while specular surfaces are typically challenging to reconstruct with geometric methods, these surfaces are particularly well-suited for shape-from-sky.

Results

We applied our method to real objects with different material compositions, including homogeneous and spatially varying diffuse albedo as well as different combinations of specular and diffuse reflection. For each object, we used up to 3 polarimetric images (black fish: 1, turtle, cup: 2, and clownfish: 3) captured at different times of the day (e.g., 2 hours apart). Quantitative analysis is shown with the three numbers under each surface normal map representing mean, median, and standard deviation of the angular errors from ground truth in degrees. Our method achieves quantitatively and qualitatively accurate geometry reconstructions of real objects with both homogeneous and spatially varying materials.

We also quantitatively evaluate the effect of varying magnitudes of diffuse reflection. The mean/median/standard deviation of the normal errors in degrees are 14.69/14.44/6.10, 16.38/16.22/5.72, and 14.48/14.49/5.28, respectively. These results demonstrate the robustness of our method to different combinations of diffuse and specular reflection magnitudes.

We reconstruct different objects made of various materials ranging from natural to man-made, and strong diffuse to predominantly specular. The results show that our method can achieve fine geometry recovery for various types of real objects.