Spatial Polarization Multiplexing:
Single-Shot Invisible Shape and Reflectance Recovery

Tomoki Ichikawa, Ryo Kawahara, and Ko Nishino
Kyoto University

We propose spatial polarization multiplexing (SPM) for joint sensing of shape and reflectance of a static or dynamic deformable object, which is also invisible to the naked eye. Past structured-light methods are limited to shape acquisition and cannot recover reflectance as they alter scene appearance. Our key idea is to spatially multiplex a polarization pattern to encode the incident ray and also densely sample the reflected light. We derive a quantized polarized light pattern that can be robustly and uniquely decoded from the reflected Angle of Linear Polarization (AoLP) values. It also enables single-shot disentanglement of polarimetric diffuse and specular reflections for accurate BRDF estimation. We achieve this spatial polarization multiplexing (SPM) with a constrained de Bruijn sequence. We validate this novel invisible single-shot shape and reflectance method with real static and dynamic objects. The results demonstrate the effectiveness of SPM for accurate shape and BRDF measurement which opens new avenues of application for 3D sensing thanks to its invisibility and ability to jointly recover the radiometric properties.

• Spatial Polarization Multiplexing: Single-Shot Invisible Shape and Reflectance Recovery
  T. Ichikawa, R. Kawahara, and K. Nishino,
  [ arXiv ][ video ][ project ]

Video

Overview

Spatial Polarization Multiplexing (SPM) recovers the shape and reflectance of a target in a single shot with a novel polarimetric structured light pattern. We design an SPM pattern with quantized stripes of AoLP values. Per-pixel polarization control enables robust decoding based on neighboring pixels for shape reconstruction and decomposition of captured polarimetric image into polarimetric diffuse and specular reflections for BRDF reconstruction. SPM enables joint sensing per-frame of deforming objects without altering the appearance (invisible to the naked eyes).

Pattern detection and decoding from an observed polarimetric image using quantized AoLP values. As shown in the inset, the observed AoLPs are corrupted due to diffuse reflection. “Detected”: the quantized AoLPs are incorrect where the corruption exceeds tolerance of quantization. “Decode”: decoding by dynamic programming corrects these mislabelings.

Results

Shape and reflectance reconstruction results of real-world objects from a single polarimetric image. Our method robustly reconstructs shapes and reflectances regardless of surface textures and materials in a single shot by fully exploiting polarimetric reflections of the SPM pattern. We can relight real-world objects by using recovered shapes and BRDFs.

We apply our method to a dynamic surface by capturing continuous frames and using each frame. Our method can capture changes in both shape and reflectance and relight the dynamic deformable surface at each frame. Top: a plush toy pushed down with a finger. Botton: a soft loaf expanding after compression.

Shifting the smoothed spatial polarization multiplexing pattern allows for high-resolution reconstruction for static scenes. Our method for shifted patterns can capture the detailed shape and texture at the resolution of the polarimetric camera.

Our shifted SPM patterns enable adaptive reconstruction resolution depending on whether objects are dynamic or static. A static clay figure is reconstructed in high resolution. A camouflage-printed rubber duck is reconstructed in low resolution when moving and in high resolution after stopping.