Skip to content

Publications / 2026

M2H-MX: Multi-Task Semantic and Geometric Perception for Real-Time Monocular 3D Scene Graph Construction

Also circulated as: M2H-MX: Multi-Task Dense Visual Perception for Real-Time Monocular Spatial Understanding

U.V.B.L. Udugama, George Vosselman, Francesco Nex

IEEE International Conference on Robotics and Automation (ICRA) 2026, SRRA Workshop, Vienna

DOI arXiv Code LinkedIn BibTeX

Better per-frame depth and semantics measurably improve the map and the trajectory, not just the benchmark score.

Context

Dense prediction papers usually stop at per-frame metrics, which leaves open whether a better model actually produces a better map. M2H-MX is evaluated twice: once as a benchmark predictor, and once as the perception front end inside a running monocular SLAM pipeline that is otherwise unchanged.

Contribution

A DINOv3 backbone with LoRA adaptation, a register-gated Mamba decoder, and directed cross-task refinement between depth and semantics. In the ablation, replacing DINOv3 with DINOv2-L costs 8.81 mIoU, while removing the cross-task mixer and the spatial attention costs 2.07. Most of the gain therefore comes from the backbone features rather than from the new decoder blocks.

The whole pipeline on ScanNet scene 0000. M2H-MX predicts depth and semantics from the image stream, Mono-Hydra++ consumes those predictions inside the visual-inertial odometry and the mapping backend, and the hierarchical scene graph is built in real time. Playback is 16x.Watch the full clip

An indoor living room: a person seated on a patterned sofa, a table lamp, a wooden side table, an armchair, and framed pictures on the wall. — Indoor frame, NYUDv2. Metric depth and semantic labels from one RGB image. Pick a layer, or drag to blend between them.

Predicted depth for the same living room, as a smooth colour field where the near armchair and sofa stand out from the far wall. — Indoor frame, NYUDv2. Metric depth and semantic labels from one RGB image. Pick a layer, or drag to blend between them.

A city street seen from a vehicle: parked and moving cars, a yellow DHL lorry, road markings, traffic signs, trees, and buildings. — The same model outdoors, Cityscapes: 82.28 mIoU, 3.15 points over the strongest baseline. Pick a layer, or drag to blend between them.

Predicted depth for the same street, with the road surface graded from near to far and vehicles resolved as separate depth bodies. — The same model outdoors, Cityscapes: 82.28 mIoU, 3.15 points over the strongest baseline. Pick a layer, or drag to blend between them.

M2H-MX architecture diagram. A DINOv3 backbone with LoRA adaptation feeds token reassembly and a feature pyramid, then register-gated Mamba blocks at each scale, task adaptors, a cross-task mixer, multi-scale convolutional attention, and finally the task heads. — M2H-MX. The backbone stays mostly frozen and LoRA adapts the later blocks, so the compute goes into the decoder and the cross-task exchange rather than into retraining the features.

The register-gated Mamba block. A global register vector passes through a linear layer and sigmoid to produce a channel-wise gate that modulates reshaped feature tokens before a Mamba block. — The register-gated Mamba block. One global vector gates every decoder scale, so global scene layout reaches local decoding without the cost of full attention.

Cross-task mixer followed by multi-scale convolutional attention, showing how auxiliary task features modulate a target task before spatial refinement. — Cross-task mixing, then spatial refinement. Removing both costs 2.07 mIoU; replacing the backbone costs 8.81.

Results

65.60 mIoU, 0.3800 RMSE NYUDv2 semantic segmentation and depth M2H-MX-L, RTX 4080 Super. This is +4.06 mIoU points and 9.4 % lower RMSE than M2H.
6.91 cm average ATE on selected ScanNet sequences Mono-Hydra stack with M2H-MX, monocular. Down from 17.59 cm for monocular Go-SLAM, a 60.7 % reduction. RGB-D baselines in the same table sit at 7.02 to 7.15 cm.
82.28 mIoU Cityscapes semantic segmentation M2H-MX-L, against MTMamba++ at 79.13

Datasets: NYUDv2, Cityscapes, ScanNet (ScanNet25k subset for the deployed model). Hardware: RTX 4080 Super, 25 to 30 Hz asynchronous perception-to-mapping loop at 640x480.

Limitations

M2H-MX-L with 4 heads is 353.53 M parameters and 491.91 GFLOPs, which is large for a perception front end.
The gain comes mostly from backbone quality rather than from the proposed decoder blocks.
The ScanNet comparison uses selected sequences, not the full benchmark.
Monocular ATE is close to the RGB-D baselines but does not beat the best of them.

Links

DOI arXiv Code LinkedIn BibTeX