Meng-Yu Jennifer Kuo, Ryo Kawahara, Shohei Nobuhara, and Ko Nishino
Kyoto University
We introduce a novel 3D sensing method for recovering a consistent, dense 3D shape of a dynamic, non-rigid object in water. The method reconstructs a complete (or fuller) 3D surface of the target object in a canonical frame (e.g., rest shape) as it freely deforms and moves between frames by estimating underwater 3D scene flow and using it to integrate per-frame depth estimates recovered from two near-infrared observations. The reconstructed shape is refined in the course of this global non-rigid shape recovery by leveraging both geometric and radiometric constraints. We implement our method with a single camera and a light source without the orthographic assumption on either by deriving a practical calibration method that estimates the point source position with respect to the camera. Our reconstruction method also accounts for scattering by water. We prototype a video-rate imaging system and show 3D shape reconstruction results on a number of real-world static, deformable, and dynamic objects and creatures in real-world water. The results demonstrate the effectiveness of the method in recovering complete shapes of complex, non-rigid objects in water, which opens new avenues of application for underwater 3D sensing.
Non-Rigid Shape From water
Meng-Yu Jennifer Kuo, Ryo Kawahara, Shohei Nobuhara, and Ko Nishino,
in IEEE Trans. on Pattern Analysis and Machine Intelligence (Special Issue of ICCP 2021), May, 2021.
[ IEEE Online First ][ supp. material ][ project ][ talk ]
Talk
Overview
We show that the full (fuller) 3D shape of a dynamic non-rigid (deformable) object in water can be recovered from an image sequence captured with a near-infrared point source and a perspective camera. Given the near infrared attenuation characteristics in water, the working range of the method is in the sub-meter range.
We first derive a practical calibration method that estimates the point source position with respect to the camera by using the absorption difference of two near-infrared wavelengths along the same refractive underwater light path. We then show that underwater dense 3D scene flow can be estimated across time by combining 2D optical flow and depth estimates. We derive a canonical method for non-rigid registration and shape refinement using these 3D scene flow estimates. We also account for light scattering induced by suspended particles in real-world water (e.g., river, lake, and ocean) by devising an alternating optimization that descatters and estimates 3D geometry. Furthermore, we derive a principled means to accumulate directionally varying incident light observations to refine each estimated surface point. We achieve these by formulating refractive structure-from-motion and depth-dependent light absorption with surface shading in water.
We implement our method with an imaging system consisting of a point source and a custom-built multi-wavelength camera for video-rate capture. To capture texture information, we optionally use additional three light sources each placed with a RGB bandpass filter in front of it. We thoroughly evaluate the effectiveness of our method with extensive real experiments using rigid, non-rigid, and dynamic objects and creatures in water. To our knowledge, our method is the first to achieve consolidated 3D geometric reconstruction of non-rigid underwater objects in motion. We believe these results open new avenues for three-dimensional analysis of underwater objects and scenes in a wide range of domains.
Results
Per-frame and rest-shape 3D recovery of swimming fish in various real-water samples that were taken from an actual lake, river, and ocean. For each fish, from top to bottom, each row shows captured RGB image, per-frame 3D shape reconstructions, and the rest-shape reconstruction. These results clearly show that our method can robustly and accurately reconstruct fuller shapes of dynamically moving, non-rigid objects in water. Please see supplementary video for results.
