Bavantha Udugama 0000-0002-1932-692X

Publications / 2026

Mono-Hydra++: Real-Time Monocular Scene Graph Construction with Multi-Task Learning for 3D Indoor Mapping

U.V.B.L. Udugama, George Vosselman, Francesco Nex

ISPRS Journal of Photogrammetry and Remote Sensing, 2026

arXivCodeLinkedInBibTeX
Status

Under review at ISPRS Journal of Photogrammetry and Remote Sensing. Manuscript number PHOTO-S-26-02536.

A single camera and an IMU build a hierarchical metric-semantic 3D scene graph of an indoor building in real time, and the map is accurate to 0.08 m.

Context

Metric-semantic mapping systems that produce usable scene graphs have almost always assumed RGB-D or LiDAR input, because reliable geometry is the hard part and depth sensors supply it directly. That assumption rules out the platforms where mapping is most useful, since payload and power are what limit small drones. Mono-Hydra++ removes the depth sensor and closes the loop between perception and state estimation instead.

Contribution

Predicted sparse depth is fed back into the VIO as metric anchor factors, semantic masks exclude dynamic and unreliable regions from the estimator, and the estimated poses align the dense predictions in time. Each direction of that loop is ablated separately. The components are measured inside the running system, so the comparison does not isolate what any single one contributes on its own.

Depth-backed visual-inertial odometry on the drone's Jetson, walked down and back up the stairs at home. The label switches between MOVING and ZUPT STATIONARY: when the platform is still, zero-velocity updates hold the estimate in place instead of letting it drift. Predicted depth, tracked features, and predicted semantics are the panels on the left.

Architecture / Mono-Hydra++

Trace a camera frame through the mapping stack.

RGB and IMU measurements become depth, semantic labels, motion estimates, refined poses and a hierarchical map.

Mono-Hydra++ system overview: RGB and IMU feed perception and odometry; aligned depth supports local graph refinement and volumetric fusion; Hydra builds a hierarchical scene graph.
Mono-Hydra++ system overview. Scene and map views are illustrative. Open full-size ↗

01 / Inside Mono-Hydra++

Align predictions before fusing the map.

A pixel in the previous frame does not refer to the same scene point after the camera moves. RVIO2 relative poses warp historical depth and labels into the current view. The nearest sample is retained at each pixel; accepted history is fused with the current prediction using depth medians and label voting.

The output is aligned depth and labels for local graph refinement and volumetric fusion.

Previous depth and labels are back-projected, transformed by relative camera pose and reprojected into the current image; validity checks precede depth median and label voting.
Pose-aware temporal alignment. Illustrative scene views show how history is aligned and combined with the current prediction. Open full-size ↗

02 / Inside Mono-Hydra++

Turn selected depth samples into pose constraints.

Aligned depth is sampled at tracked keypoints. Depth validity, motion, reprojection and semantic checks select the samples that can become depth factors. These enter local graph refinement alongside visual tracks, RVIO2 motion and IMU constraints.

Refined poses and loop-closure proposals are passed to Hydra, which optimises the map and updates the scene graph.

Tracked keypoints sample aligned depth; validity, motion, reprojection and semantic checks select depth factors for local graph refinement.
Sparse depth-factor selection. Scene views are illustrative. Open full-size ↗

03 / Inside Mono-Hydra++

Constrain optical-axis depth in the local graph.

The depth measurement is camera-frame distance along the optical axis. A depth factor connects a camera pose and a tracked landmark; visual, motion and IMU factors provide complementary constraints.

The illustration omits velocity and bias variables to make the pose–landmark relationship easier to read.

Optical-axis depth relates a camera pose to a landmark; the local factor graph combines visual, IMU, motion and depth constraints.
Depth-factor geometry and local factor graph. Velocity and bias variables are omitted in this illustration. Open full-size ↗
Three trajectory plots on the uHumans2 dataset comparing ground truth against a baseline and against the system with depth sparse factors and a semantic mask, marking accepted, available, and missed loop closure candidates.
Loop closure behaviour on uHumans2. Of 233 candidates the baseline finds 212 and misses 21; adding depth sparse factors and the semantic mask shifts which candidates survive rather than simply finding more.
Three floor plans side by side for ScanNet scene 0054. The first shows ground truth object centres as coloured dots, the second shows predicted object centres as squares, the third overlays them showing matched, missing, and unmatched objects.
Object placement on ScanNet scene 0054, scored by radius match at 0.5 m. Green is matched, orange is a ground-truth object the system missed, and a red cross is something predicted that is not there. The aggregate figure for this metric is 42.59 percent.
Semantic mesh reconstruction compared against ground truth, showing small object classes absorbed into larger neighbouring surfaces.
A failure case. Small classes are absorbed into the dominant surface next to them, which the global mIoU of 44.96 percent does not show on its own.

Results

Datasets: ScanNet, 7-Scenes, uHumans2, ITC building (own capture). Hardware: RTX 4080 Super for the full system. Jetson Orin NX 16GB for the exported perception model only.

Limitations

Links

arXivCodeLinkedInBibTeX