Skip to content

Research / University of Twente

Research.
From sensing to autonomy.

The PhD is lightweight perception for aerial robots: one camera, one network predicting depth, semantics, surface normals and edges together, and those predictions driving mapping and autonomous exploration on the drone's own compute, in real time.

System sketch

From sensing to spatial understanding

Perception and mapping run onboard. ATLAS is trained and tested in simulation, with current drone transfer underway and earlier iterations flight-tested.

Conceptual exploded view of a scene graph: building, rooms, connected places, objects and a geometric mesh.

Read the map at five levels

Turn geometry into a map a robot can reason about.

A scene graph connects geometric detail to objects and larger spaces. The illustration shows the hierarchy; the experimental reconstructions appear below.

BuildingThe top-level environment.
RoomsRegions that organise the building.
PlacesConnected locations through the space.
ObjectsSemantic entities in the map.
MeshReconstructed geometry carrying semantic labels.

Conceptual thesis illustration, not an experimental reconstruction.

Follow the mapping pipeline ↗

Prologue. The same question, ten years earlier

The same question was asked at Peradeniya in 2017: an autonomous exploration planning strategy for a robot mapping an unknown space, using a laser range finder fused with stereo cameras. It is being asked again 10 years later with a learned model and a single camera.

2017 · Peradeniya

Autonomous exploration planning for a reconnaissance agent

An autonomous exploration planning strategy for a reconnaissance robot mapping an unknown space, with a laser range finder fused with stereo cameras. I was the SLAM developer, one of three students on the project.

The 2017 Peradeniya project video. A holonomic robot with a laser range finder and stereo cameras explores a plywood maze it has never seen, builds the occupancy map in the inset, and finishes with the stitched 3D reconstruction of the arena. The localisation and control segment, which was mine, starts at 0:59.Watch the full clip

The ICIIS 2017 paper

2026 · ITC, University of Twente

Monocular 3D scene graphs, built in real time

The same job with the scanner removed. One camera and an IMU produce a hierarchical 3D scene graph of the building in real time, and the exploration policy that chooses where to go next is being learned rather than written.

A 3D scene graph over the ITC building second floor. A single building node at the top connects down to a room node, then to a dense band of place nodes following the corridor loop, then to object nodes scattered along the walls, and finally to the reconstructed mesh of the floor.

The mapping system

Part I. Structuring the seen

Building metric-semantic hierarchical 3D scene graphs from a single camera and an IMU, in real time.

Pipeline diagram. Monocular RGB enters M2H-MX on the GPU, producing depth and semantics. Those feed pose-aware temporal alignment and a CPU-side VIO that uses predicted depth, with IMU input. The result is a metric semantic mesh and a 3D scene graph with building, room, place, object, and mesh layers. — One camera and one IMU in, a layered scene graph out. Two arrows return from odometry into the perception block: the estimated pose is used to align the next dense prediction, not only to build the map.

One building, four generations

One building, four system generations, same measurement. Hardware and input resolution change across rows, so this is a system progression, not a controlled ablation.

Mapping error by system generation
System	Year	Mean error	Input	Hardware
Mono-Hydra	2023	0.19 m	848x480	RTX 3080 laptop
M2H	2025	0.11 m	224x224	RTX 3080 laptop
M2H-MX-B	2026	0.10 m	640x480	RTX 4080 Super
M2H-MX-L	2026	0.08 m	640x480	RTX 4080 Super
M2H-MX-L, embedded	2026	0.22 m	256x192	Jetson Orin NX 16GB

Metric-semantic mapping systems that produce usable scene graphs have almost always assumed RGB-D or LiDAR input, because reliable geometry is the hard part and a depth sensor supplies it directly. That assumption rules out the platforms where the mapping is most useful, since payload and power are exactly what limit a small drone. Removing the depth sensor was the question, and four papers answer it in sequence.

Mono-Hydra, 2023. Proving it is possible

Learned depth and learned semantics, running as two separate networks, feed a robocentric square-root visual-inertial odometry front end, and the resulting semantic mesh drives the layered scene graph. Measured against a LiDAR backpack point cloud of the ITC building it reached 0.19 m and 0.21 m mean error on two floors, at 15 fps. Read more.

M2H, 2025. One model instead of two

Two independent networks duplicate compute and produce two representations that do not agree with each other. M2H collapses them into a single multi-task model in which depth, semantics, surface normals, and edges exchange information through window-based cross-task attention. The benchmark numbers improved, and so did the map: 0.11 m on the same building, at twice the frame rate. Read more.

M2H-MX, 2026. Testing whether the map actually cares

A better benchmark score is not the same thing as a better map, and dense prediction papers usually stop before finding out. M2H-MX is evaluated twice, once as a predictor and once as the front end inside an otherwise unchanged SLAM pipeline. Average trajectory error on selected ScanNet sequences fell from 17.59 cm to 6.91 cm, which puts a monocular system close to the RGB-D baselines in the same table. The ablation attributes most of the gain to the backbone features rather than to the new decoder blocks. Read more.

Mono-Hydra++, under review. Closing the loop backwards

Everything so far runs perception forwards into mapping. Mono-Hydra++ sends it back the other way as well: predicted sparse depth enters the estimator as metric anchor factors, semantic masks keep dynamic regions out of it, and the poses that come back out temporally align the next dense predictions. 0.08 m on the ITC second floor, 0.033 m calibrated trajectory error on 7-Scenes. Read more.

A 3D scene graph over the ITC building second floor. A single building node at the top connects down to a room node, then to a dense band of place nodes following the corridor loop, then to object nodes scattered along the walls, and finally to the reconstructed mesh of the floor. — The layered scene graph over the ITC second floor. It runs from one building node down through a room node, a band of place nodes following the corridor loop, and object nodes along the walls. Counts by category are in the legend.

Part II. Exploring the unseen Simulation validated · Drone transfer underway

A learned agent over a persistent belief graph of places, open ends and unfinished obligations, choosing where to look next in a building it has never seen.

Status

Trained and tested in simulation. Current drone transfer underway; earlier iterations flight-tested.

The robot enters an unmapped indoor space with one forward-facing camera and has to cover it under a time budget without collision. Motion is treated as solved: a geometric planner proposes only routes it has verified against the occupancy map the robot has built. The open problem is choice, not motion, so the learned layer decides among certified options and may abstain, but never commands motion itself. ATLAS is built in PyTorch and fully trained and tested in simulation. Current work transfers the policy to the drone, building on earlier flight-tested iterations.

Predict consequences, not appearances.

The learned component sits between two deterministic layers and has no path to the actuators except by selecting an option the planner above it has already certified as safe. It may re-order those options and it may abstain. It may not invent one. The requirements that follow come from the shape of the task rather than from the hardware:

Options are judged against one another, so two routes leading through the same opening are not treated as genuine alternatives.
Actions are temporally extended, because passing through a doorway is locally expensive and locally uninformative, and everything of value happens afterwards. The critic credits them by their duration.
Unfinished work is a node type rather than a heuristic: an obligation stays in the graph until it is discharged, so regions get completed and exhausted ones get abandoned on evidence rather than on a timer.
Every edge is keyed to a certified route and its certificate hash, so the graph cannot express a way through that has not been verified.
The reward is registered before the experiments, so the objective cannot be tuned to the result.
No learned component is permitted to command motion. It re-orders certified options or refuses them.

More on the exploration work.

Thesis

The two parts above take their names from the thesis, Structuring the Seen, Exploring the Unseen, which is expected to be completed in 2026 at the Faculty ITC, University of Twente, supervised by George Vosselman and Francesco Nex.