Skip to content

Publications / 2025

M2H: Multi-Task Learning with Efficient Window-Based Cross-Task Attention for Monocular Spatial Perception

U.V.B.L. Udugama, George Vosselman, Francesco Nex

IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2025

arXiv Code Video LinkedIn BibTeX

Four dense prediction tasks share one small model and make each other better, fast enough to sit inside a mapping loop.

Context

Mono-Hydra ran an independent depth network and an independent semantic network side by side. That duplicates compute and produces two representations that do not agree with each other. M2H replaces both with one multi-task model in which depth, semantics, surface normals, and edges exchange information through window-based cross-task attention.

Contribution

A window-based cross-task attention module that lets tasks exchange structured features while keeping task-specific detail, on a DINOv2-small backbone. In the ablation both the local attention and the global fusion module matter: removing either degrades every task.

The scene graph forming over one loop of the ITC second floor, from the first frame to loop closure. Playback is 10x real time. Live RGB, predicted semantics, and predicted depth run down the right.Watch the full clip

A corridor in the ITC building seen by the monocular camera: doors on the right, artwork on the walls, an orange floor receding into the distance. — One frame from the ITC corridor sequence: camera image, predicted depth, predicted labels. Pick a layer, or drag to blend between them.

Predicted depth for the same corridor frame, rendered as a colour field where blue is near and red is the far end of the corridor. — One frame from the ITC corridor sequence: camera image, predicted depth, predicted labels. Pick a layer, or drag to blend between them.

Thumbnail for Mono Hydra with M2H for Monocular 3D Scene Graph Construction — Mono Hydra with M2H for Monocular 3D Scene Graph Construction. The full length version, loaded from YouTube only when you press play.

Results

61.54 mIoU, 0.4196 RMSE NYUDv2 semantic segmentation and depth RTX 3080 laptop. Best prior compared: SwinMTL 58.14 mIoU, MTMamba++ 0.4818 RMSE.
0.11 m at 30 fps ITC building 2nd floor mesh mean error, in the Mono-Hydra loop 224x224, RTX 3080 laptop. The Mono-Hydra baseline with separate networks reached 0.19 m at 15 fps.
+5.65 mIoU Hypersim semantic segmentation over EMSANet 52.31 against 46.66, Table II. Depth RMSE 3.19 against ScaleDepth-NK at 4.825.

Datasets: NYUDv2, Hypersim V1, Cityscapes, ITC building (own capture). Hardware: Trained on A40. Tested on RTX 3080 laptop, 30 fps.

Limitations

M2H is 81 M parameters and 488 GFLOPs. The parameter count is moderate but the GFLOP count is high.
M2H-small is far cheaper at 33.7 M parameters and 59 GFLOPs, and gives up 3.49 mIoU (58.05 against 61.54).
Boundary detection on NYUDv2 is the one task where the small variant falls behind prior work, at 74.44 odsF.

Links

arXiv Code Video LinkedIn BibTeX