RatBodyFormer: Rodent Body Surface from Keypoints

Ayaka Higami¹, Karin Oshima², Tomoyo Isoguchi Shiramatsu²
Hirokazu Takahashi² Shohei Nobuhara³, and Ko Nishino^1
1Kyoto University
²The University of Tokyo ³Kyoto Institute of Technology

Rat behavior modeling goes to the heart of many scientific studies, yet the textureless body surface evades automatic analysis as it literally has no keypoints that detectors can find. The movement of the body surface, however, is a rich source of information for deciphering the rat behavior. We introduce two key contributions to automatically recover densely 3D sampled rat body surface points, passively. The first is RatDome, a novel multi-camera system for rat behavior capture, and a large-scale dataset captured with it that consists of pairs of 3D keypoints and 3D body surface points. The second is RatBodyFormer, a novel network to transform detected keypoints to 3D body surface points. RatBodyFormer is agnostic to the exact locations of the 3D body surface points in the training data and is trained with masked-learning. We experimentally validate our framework with a number of real-world experiments. Our results collectively serve as a novel foundation for automated rat behavior analysis and will likely have far-reaching implications for biomedical and neuroscientific research.

• RatBodyFormer: Rodent Body Surface from Keypoints
  A. Higami, K. Oshima, T.I. Shiramatsu, H. Takahashi, S. Nobuhara, and K. Nishino
  [ arXiv ][ video ][ project ]

Video

RatDome Dataset

We learn to estimate the rat body surface from its detected keypoints. For this, we will need a sufficiently large-scale dataset of paired sets of 3D keypoints and 3D body surface points. This is challenging as the rat body is completely textureless. We overcome this challenge by simply endowing the rat with body texture. We attach color beads to the body as well as paint markers in the areas where beads cannot be attached. It is important to note that these markers are only used for training data capture and the rats are completely in their natural form when their behavior is observed in actual experiments.

We build a novel multi-camera system which we refer to as RatDome to passively observe the color-beaded rat and reconstruct the 3D coordinates of individual markers. By capturing freely moving rats of different ages (7, 9, 11 week-old) in RatDome, we collect multiview videos and the paired sets of 3D keypoints and 3D markers. We refer to this first-of-its-kind large-scale rat body surface dataset as the RatDome Dataset.

RatBodyFormer

We derive a transformer-based network that takes the keypoint 3D coordinates as input and outputs sampled body surface point 3D coordinates. We refer to this network as RatBodyFormer. The model is trained with the RatDome Dataset. We manually selected a standing-on-two-feet pose as the reference pose such that almost all of the whole body surface is visible from the cameras. We use it as the basis to model the deformation. We first compute the as-rigid-as possible deformation of the keypoints in the reference pose to the observed keypoints. The encoder takes the displacements of these two as input tokens. The decoder queries are the displacement of the body surface points in the reference pose before and after they are ARAP deformed to the target pose using the keypoints. The decoder outputs are the estimated displacements of the body surface points from their coordinates in the reference pose.

Results

We evaluate the accuracy of RatBodyFormer with a color-beaded rat. The average 3D L2 error was around 6mm, roughly the size of the beads. The estimated body surface points are accurate regardless of the rat’s pose.

RatBodyFormer also generalizes across different ages of weeks. RatBodyFormer can be applied to a variety of rats ranging from 5 to 14 week-olds. This covers the age range typically used in biomedical and neuroscientific experiments.

RatBodyFromer can predict future rat 3D body surface by just predicting detectable keypoint coordinates. This is possible as RatBodyFormer effectively reduces the necessary points for forecasting to those that are well-defined. We believe this forecasting scheme can benefit a wide-range of scientific experiments, especially that concern multiple rats.