John Novatnack, Prabin Bariya, Gabriel Schwartz, and Ko Nishino
Drexel University
Despite their ubiquitous presence, little has been investigated about the scale variability – the relative variations in the spatial extents of local structures – of 3D geometric data. We present a comprehensive framework for exploiting this 3D geometric scale variability in range images that provides rich information for characterizing the overall geometry. We derive a sound scale-space representation, which we refer to as the geometric scale-space, that faithfully encodes the scale variability of the surface geometry, and derive novel detectors to extract prominent features and identify their natural scales. The result is a hierarchical set of features of different scales which we refer to as scale-dependent geometric features. We then derive novel local shape descriptors that represent the surface structures that give rise to those features by carving out and encoding the local surface that fall within the support regions of the features. This leads to scale-dependent or scale-invariant local shape descriptors that convey significant discriminative information of the object geometry. We demonstrate the effectiveness of geometric scale analysis on range images, and show that it enables novel applications, in particular, fully automatic registration of multiple objects from a mixed set of range images and 3D object recognition in highly cluttered range image scenes.
3D Geometric Scale Variability in Range Images: Features and Descriptors
P. Bariya, J. Novatnack, G. Schwartz, and K. Nishino,
in Int’l Journal of Computer Vision, vol. 99, no. 2, pp232-255, Sept., 2012.
[ paper ] [ Springer ][ code ][ project ]
Scale-Hierarchical 3D Object Recognition in Cluttered Scenes
P. Bariya and K. Nishino,
in Proc. of IEEE International Conference on Computer Vision and Pattern Recognition CVPR’10, Jun., 2010.
[ paper ][ code ][ project ]
Scale-Dependent/Invariant Local 3D Shape Descriptors for Fully Automatic Registration of Multiple Sets of Range Images
J. Novatnack and K. Nishino,
in Proc. of Tenth European Conference on Computer Vision ECCV’08, Oct., 2008.
[ paper ][ code ][ project ]
Scale-Dependent 3D Geometric Features
J. Novatnack and K. Nishino,
in Proc. of IEEE Eleventh International Conference on Computer Vision ICCV’07, Oct., 2007.
[ paper ][ code ][ project ]
Geometric structures that characterize the geometry captured in a range image reside on the surface. For this reason, we must construct a representation that faithfully encodes the scale variability on the surface, i.e., its surface geometry and not its embedding (point coordinates), and allows us to analyze geometric structures across different scales. We choose to represent the geometry of a given surface with its surface normals. By deriving and applying a scale-space operator that accounts for the geodesic distances on the surface, we build a scale-space of this surface normal field, which we refer to as the geometric scale-space. The figure shows the geometric scale-space of a range image of a Buddha model (a). As the standard deviation increases from (b)-(d), finer details of the surface structures are smoothed away. The red, green and blue color channels, in (b)-(d), encode the direction of the surface normals in the x, y and z directions respectively.
Given this geometric scale-space representation of a range image, we may detect salient features in the geometric scale-space that characterize the underlying 3D geometry across different scales. For this, we derive the first- and second-order partial derivatives of the normal map of a range image. A novel corner and edge detector can then be derived using these partial derivatives. We then devise an automatic scale-selection algorithm to identify the natural scale of each feature and unify the features detected at each scale into a single set of scale-dependent geometric features. The figure shows scale-dependent geometric corner and edge detection results on a disc topology model (Caesar) and two genus zero models (Buddha and armadillo). The corner and edges are accurately detected across different scales. The resulting scale-dependent geometric feature set encode the scale variability of the underlying surface geometry resulting in a unique representation of each model.
Once we detect scale-dependent features via geometric scale-space analysis, we can carve out and encode the local region of the surface that characterizes the local geometric structure surrounding the feature, in particular, a representative corner point, in the form of a compact local shape descriptor. The associated inherent scale of each scale-dependent corner directly tells us the natural spatial extent (the support size) of the underlying local geometric structure. This information can then in turn be used to identify the size of the neighborhood of each corner that should be encoded in any local shape descriptor. This figure shows scale-dependent corners and scale-dependent local 3D shape descriptors computed based on geometric scale-space analysis of two range images. The range images are depicted with their normal fields. The scale-dependent corners are colored according to their inherent scales, with red and blue corresponding to the coarsest and finest scales, respectively. The scale-dependent local 3D shape descriptors capture local geometric information in the natural support regions of the scale-dependent features. Here, the colors in the descriptors represent the direction of the normals encoded in the descriptors.
The novel scale-dependent and scale-invariant local 3D shape descriptors contain rich discriminative information regarding the local geometric structures. As a practical example, we show the effectiveness of these descriptors in range image registration, one of the fundamental steps in geometry processing. We show that the scale-dependent and scale-invariant descriptors can be used to register a set of range images both with and without global scale variations, without any human intervention. Most importantly, we show that we can register a mixed set of range images corresponding to multiple 3D models (with each range image containing a single view of a single model) simultaneously and fully automatically. This figure shows automatic registration of a set of range images of multiple objects:total 42 range images shown on the left most, consisting of 15 views of the Happy Buddha model, 12 views of the Armadillo, and 15 views of the Dragon model. The scale-dependent local 3D shape descriptors contain rich discriminative information that enables automatic discovery of the three disjoint models from the mixed range image set. Note that, to show the accuracy of our registration, the results shown here have not been post-processed with a global registration algorithm.
We show how the hierarchy induced by the scale variation of local geometric structures may be employed in aiding accurate 3D object recognition. The goal of 3D object recognition is to correctly identify objects that are present in a 3D scene, usually in a depth/range image, and to estimate the location and orientation of each object. 3D object recognition has been of interest for industrial automatic assembly but with the availability of portable laser range scanners and especially consumer range scanners such as the Microsoft Kinect, scene understanding of cluttered range image scenes is an important problem to address. We show that by fully leveraging the additional information provided by the scale variability in the matching phase in addition to employing our scale-dependent/invariant local 3D shape descriptors that encode the natural support region for each feature, we can achieve accurate recognition results even in highly cluttered range image scenes. This figure shows scale-dependent recognition results on four scenes from the Queen’s dataset. All objects that have been recognized are replaced with their 3D models in different colors. Only the Kid in (c) and Zoe in (d), the yellow and brown colored models in (a-b) respectively, were not recognized. In both these cases, significant regions of the scene corresponding to these models were lost during the surface generation process. All other models were recognized and localized accurately despite the significant clutter and occlusion.
