SPIDeRS: Structured Polarization for Invisible Depth and Reflectance Sensing

Tomoki Ichikawa, Shohei Nobuhara, and Ko Nishino
Kyoto University

Can we capture shape and reflectance in stealth? Such capability would be valuable for many application domains in vision, xR, robotics, and HCI. We introduce structured polarization for invisible depth and reflectance sensing (SPIDeRS), the first depth and reflectance sensing method using patterns of polarized light. The key idea is to modulate the angle of linear polarization (AoLP) of projected light at each pixel. The use of polarization makes it invisible and lets us recover not only depth but also directly surface normals and even reflectance. We implement SPIDeRS with a liquid crystal spatial light modulator (SLM) and a polarimetric camera. We derive a novel method for robustly extracting the projected structured polarization pattern from the polarimetric object appearance. We evaluate the effectiveness of SPIDeRS by applying it to a number of real-world objects. The results show that our method successfully reconstructs object shapes of various materials and is robust to diffuse reflection and ambient light. We also demonstrate relighting using recovered surface normals and reflectance. We believe SPIDeRS opens a new avenue of polarization use in visual sensing.

• SPIDeRS: Structured Polarization for Invisible Depth and Reflectance Sensing
  T. Ichikawa, S. Nobuhara, and K. Nishino,
  in Proc. of Conference on Computer Vision and Pattern Recognition CVPR’24, Jun., 2024.
  [ arxiv ][ paper ][ supp. PDF ][ supp. video ][ project ]

Video

Overview

In this paper, we introduce Structured Polarization, a first-of-its-kind novel depth and reflectance sensing method using structured light of polarized light, which we refer to as SPIDeRS. The key idea is to project structured light patterns of varied angle of linear polarization (AoLP). The use of polarization immediately gives us the following three advantages that none of the past structure light methods can achieve.

• Polarization is invisible to the naked eye and a regular camera. By using structured polarization patterns and an RGB-polarimetric camera, we can realize invisible depth and texture capture.
• The surface normals can be directly recovered providing dense fine details of the target surface. This is in sharp contrast to regular depth sensing where the normals can only be obtained from the triangulated depth, which is inevitably noisy due to the differentiation.
• We can also estimate the reflectance properties of the target surface from the AoLP patterns in the object appearance. This enables joint depth and reflectance sensing with a single set of invisible structured light patterns.

To accurately decode the reflected AoLP structured pattern, we derive a novel method that can extract the pattern from the captured polarization image while accounting for specular and diffuse reflection, which encodes and is independent from the thrown AoLP, respectively, and ambient light. Diffuse reflection and ambient light alter the AoLP throw in the reflected polarimetric object appearance and cause decoding errors (left). Our extraction method robustly extracts the true projected pattern (right).

Experimental Setup

We implement SPIDeRS by assembling a projector-camera system with a polarization projector whose AoLP can be controlled in the throw at each pixel and an RGBpolarimetric camera. For this, we leverage the fact that liquid crystal acts as a polarizer. We use a liquid crystal spatial light modulator (SLM) to modulate the polarization orientation of incident polarized light by altering the per-pixel voltage. This can be understood as an LCD projector without an analyzer that modulates the amplitude such that the AoLP instead of the intensity is modulated. As shown in this figure, our polarization projector consists of an LED telecentric illuminator with a polarizer, SLM, and convex lens.

Results

Reconstructed depth maps of real objects of various materials and colors. The scene of the first and second rows do not have ambient light and the third row does. The numbers below each depth map are the mean and median of the depth errors in millimeters. Our method successfully reconstructs object shapes of different materials and is robust to diffuse reflection and ambient light while simultaneously capturing the surface appearance. The results also show that the accuracy of our method is comparable with classic structured light while retaining the surface appearance and being totally invisible to the naked eye.

The estimated surface normal maps of real objects and those computed from the reconstructed depth maps by Principal Component Analysis (PCA). The numbers below each normal map in the quantitative evaluation are the mean and median of angular errors in degrees. Our method successfully exploits polarimetric reflection to reconstruct detailed surface normals unlike past methods that can only produce them as differentiated byproducts of the measured depth maps.

Relighting under a novel illumination with the estimated BRDF and surface normal. Illuminations are white directional lights from the left (Relighting1) and red, green, and blue ones from different directions (Relighting2). Our method can estimate the BRDF accurately which results in qualitatively plausible relighting. This is possible as our method can simultaneously capture the texture and polarimetric reflection in the visible spectrum. Past structured light methods cannot estimate the BRDF and surface normals as it only acquires radiometric images and the estimation is ill-posed.