Publications / 2023

Mono-Hydra: Real-Time 3D Scene Graph Construction from Monocular Camera Input with IMU

U.V.B.L. Udugama, G. Vosselman, F. Nex

ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, volume 1, pages 439 to 445, 2023

A monocular camera and an IMU can build a layered 3D scene graph while the robot is still moving, without a depth sensor or LiDAR.

Context

Scene graphs represent an environment as a layered graph running from mesh geometry up through objects, places, rooms, and buildings. Systems that build them in real time, Hydra and Kimera and S-Graphs+, all assumed RGB-D or 3D LiDAR input. Removing the depth sensor was largely unexplored, and it is exactly what small aerial platforms need.

Contribution

Learned depth and learned semantics feed a robocentric square-root visual-inertial odometry front end, and the resulting semantic mesh drives the Hydra scene graph layers. The contribution is the system rather than any single network. It was the first demonstration of this construction from a monocular camera and an IMU.

System design diagram for monocular 3D scene graph construction, showing the perception networks, visual-inertial odometry, and the scene graph layers. — The original Mono-Hydra design: two independent networks for depth and semantics, feeding a robocentric VIO and the Hydra scene graph layers.

A 3D scene graph over the ITC building second floor. A single building node at the top connects down to a room node, then to a dense band of place nodes following the corridor loop, then to object nodes scattered along the walls, and finally to the reconstructed mesh of the floor. — The layered scene graph over the ITC second floor. It runs from one building node down through a room node, a band of place nodes following the corridor loop, and object nodes along the walls. Counts by category are in the legend.

Results

0.19 m and 0.21 m mesh mean error on the ITC building 2nd and 3rd floors, against a LiDAR backpack point cloud DistDepth front end, 848x480, RTX 3080 laptop. The Lite-Mono front end was roughly twice as bad, at 0.39 m and 0.36 m.
15 fps end-to-end real-time system rate 848x480 input, RTX 3080 8GB laptop GPU
0.03 m to 0.28 m mesh mean error against Hydra on uHumans2 Three synthetic scenes, DistDepth 0.05 to 0.28 m, Lite-Mono 0.03 to 0.23 m with higher variance

Datasets: uHumans2, ADE20k and NYUv2 (fine-tuning), ITC building (own capture, RealSense D435i), LiDAR backpack point cloud (ground truth). Hardware: Fine-tuning on TITAN Xp 12GB. System on Alienware m17 laptop, RTX 3080 8GB.

Limitations

The semantic quality of the generated 3D semantic mesh was never quantified, so no accuracy measure for it exists.
Hierarchical loop closure, room detection, and temporal consistency of the mesh are all named as unsolved.
The abstract rounds the mapping result to sub-20 cm, though the 3rd floor measured 0.21 m.
Deployment on embedded hardware needed further network optimisation, which the later work addressed.

Links

arXiv Code Video LinkedIn BibTeX