Ryo Kawahara, Meng-Yu Jennifer Kuo, Shohei Nobuhara, and Ko Nishino
Kyoto University
We introduce single-image geometric and appearance reconstruction from water reflection photography, i.e., images capturing direct and water-reflected real-world scenes. Water reflection offers an additional viewpoint to the direct sight, collectively forming a stereo pair. The water-reflected scene, however, includes internally scattered and reflected environmental illumination in addition to the scene radiance, which precludes direct stereo matching. We derive a principled iterative method that disentangles this scene radiometry and geometry for reconstructing 3D scene structure as well as its high-dynamic range appearance. In the presence of waves, we simultaneously recover the wave geometry as surface normal perturbations of the water surface. Most important, we show that the water reflection enables calibration of the camera. In other words, for the first time, we show that capturing a direct and water-reflected scene in a single exposure forms a self-calibrating HDR catadioptric stereo camera. We demonstrate our method on a number of images taken in the wild. The results demonstrate a new means for leveraging this accidental catadioptric camera.
Appearance and Shape from Water Reflection
R. Kawahara, M-Y. J. Kuo, S. Nobuhara, and K. Nishino,
in Proc. of Winter Applications on Computer Vision WACV’20, Mar., 2020. (Best Paper Finalist)
[ paper ][ supp. material ][ project ]
Overview
For the first time to our knowledge, we show that a single image capturing both the direct and reflected observation through water reflection of a scene results from a self-calibrating high-dynamic range catadioptric imaging system. That is, we show that a high-dynamic range appearance and 3D shape of the scene can be recovered without any knowledge about the image formation a priori. We first consider the case where the water surface is calm and can be modeled as a planar mirror. The main challenge lies in the fact that water-reflected scene radiance is compounded with environmental light scattered in the water medium and also reflected by the bottom surface. Scene radiance must be sifted out from this superposition in order to match against the direct observation for triangulation. We derive a canonical iterative method to recover scene geometry from the direct and water-reflected observations. We also show that high-dynamic range scene radiance can be estimated in the process and water reflection even enables calibration of the camera. That is, we do not need to know anything about the camera; its intrinsic parameters can be recovered by seeking agreements in angular-dependent Fresnel effects in the reflected observation, and its extrinsic parameters can be estimated by identifying the water surface. The water surface in the image is not always calm and can have waves that lead to noticeable displacements in the reflected observation. We show that this can be modeled as surface normal variations of the water surface and derive an iterative approach to simultaneously recover the shape of both the scene structure and the water surface. For this, we introduce a principled method for incorporating realistic prior knowledge such as piecewise planarity for scene geometry and a Fourier-domain wave representation.
Results
We apply our method to various images either taken by our phone cameras or found on the Internet. The supplementary video contains more results. The results show that the 3D scene structure can be recovered despite waves and complex water surface reflection. It is also interesting to see how the method applies to a wide range of scene scales, ranging from a small bird to a large architecture. Images taken with long focal length tend to result in flatter surface with limited depth variation as one expects. The results also include various types of water reflection, ranging from a puddle to a lake demonstrating its successful application to images truly taken in the wild.
