Watching a Robot Fleet

In plain words

What the project is, and what problem it exists to solve.

One robot is easy to watch. You look at it, or you open a terminal and read its messages. Ten robots in a building are not: nobody is looking at any individual one, and the interesting question stops being what is this robot doing and becomes which of these robots needs a human right now.

This project builds the plumbing that answers the second question. Robots publish their position continuously. That stream is collected, stored, and watched by a rule. When a robot has not moved for a while, the system files an alert with the robot's identifier and its last known position. The alert stays open until either a person closes it or the robot starts moving again, at which point it closes itself.

The thing that makes it more than a demo

Nothing here is hand-wired. The fleet is described once, in config/fleet.yaml. The simulation world, the visualiser layout, the launch file, the message producers, the driving patterns and the tests all read that same file. Adding a fourth robot is five lines of configuration and one regeneration command — no code changes anywhere. A continuous-integration job refuses the change if the generated files and the configuration have drifted apart.

Why it sits alongside the learning projects

The other projects on this site are about making a model that works. This one is about the layer underneath that: the part that has to keep running while nobody is watching it, notice something is wrong, and still be inspectable afterwards. It is deliberately the infrastructure slot rather than the main line of work, and it is built the way that layer has to be built — every service restartable, every failure visible, every claim checked by a test.

Terms used on this page

odometryA robot's own running estimate of where it is and how fast it is going, computed from its wheels and sensors.

message busA service that takes messages from producers and hands them to any number of independent consumers. Here, Apache Kafka.

topicA named channel on the bus. Producers write to it, consumers read from it, and neither knows the other exists.

time-series databaseA database built for append-only data stamped with a time. Fast to write, fast to query over a window, bad at changing rows after the fact.

dead-letter queueA separate topic where messages that could not be processed are parked, with the reason, instead of being dropped or crashing the reader.

sliding windowThe last N samples, with the oldest falling off as each new one arrives. The detector's memory.

edge-triggeredActing on the moment a condition changes, not on every sample while it holds. One alert per event, not one per message.

healthcheckA command a container runs on itself so the orchestrator can tell "started" apart from "actually ready to serve".

TF frameA named coordinate system in ROS 2. Two robots publishing the same frame name in one scene collide; prefixes keep them apart.

The data path

One position reading, from the simulated robot to the alert on a dashboard.

stage 1Gazebo Harmonic Three TurtleBot3 models drive in the generated world. ros_gz_bridge carries odometry, velocity commands, joint states, transforms and the clock across to ROS 2. /tb1/odom · /tb2/odom · /tb3/odom

stage 2Producer node A ROS 2 node subscribes to every robot's odometry topic and re-publishes each reading as one JSON message. kafka_producer.py · 79 lines

stage 3Kafka The bus. Two topics: the odometry stream, and a dead-letter topic for anything unreadable. robot_odom · robot_odom_dlq

stage 4aStorage consumer Reads the stream, writes each valid reading into QuestDB over the line protocol, and updates the robot's heartbeat in Postgres at most once every five seconds. kafka_consumer.py · 104 lines

stage 4bDetector Reads the same stream independently, keeps a sliding window of positions per robot, and opens or resolves alerts on transitions. anomaly_detector.py · 124 lines

stage 5Grafana + CLI Dashboards load themselves from the repository. A command-line tool lists robots and alerts and closes them. localhost:3000 · fleet_cli.py

Why two independent readers

Storage and detection are separate Kafka consumer groups reading the same topic. Neither blocks the other, either can be restarted alone, and the detector can be changed without any risk to the data being recorded. That separation is the entire reason a bus is in the design rather than a direct database write from the robot.

Every file

The whole repository, with line counts. Generated files are marked.

File	Lines	What it does
config/fleet.yaml	46	The single source of truth: robots, spawn poses, colours, drive patterns, endpoints, detector thresholds.
scripts/generate_fleet.py	284	Reads the yaml, writes the simulation world and the visualiser layout.
worlds/fleet.sdf	420	Generated. The Gazebo world — ground plane, lighting, one model per robot at its spawn pose.
config/fleet.rviz	127	Generated. RViz layout with one display group per robot.
launch/multi_robot.launch.py	104	Starts the simulator, the bridge, one state publisher per robot, the static transforms, and optionally RViz.
pipeline/common.py	181	Shared plumbing: config loading, endpoint resolution, retrying connections, the QuestDB writer, the Postgres store.
pipeline/kafka_producer.py	79	ROS 2 node. Every robot's odometry topic to one Kafka topic, as JSON.
pipeline/kafka_consumer.py	104	Kafka to QuestDB, with the dead-letter path and the registry heartbeat.
pipeline/anomaly_detector.py	124	Sliding-window stuck detection and the full alert lifecycle.
pipeline/mock_producer.py	77	Synthetic fleet emitting the identical message schema, so the pipeline runs with no simulator at all.
scripts/smoke_test.py	96	Starts the three services, runs them for 25 seconds, then asserts four things about the databases.
scripts/fleet_cli.py	91	List robots, list alerts, acknowledge one, resolve one.
scripts/drive_fleet.py	56	Publishes each robot's configured drive pattern to its velocity topic.
dashboard/monitor.py	138	Terminal dashboard, querying QuestDB over the Postgres wire protocol.
docker-compose.yml	106	Kafka, QuestDB, Postgres and Grafana, each with a healthcheck.
docker/postgres/init.sql	27	Two tables: the robot registry and the alert lifecycle, with their indexes.
docker/grafana/	166	Datasources and the Fleet Overview dashboard, provisioned from the repository.
.github/workflows/ci.yml	45	Lint, a drift check on the generated files, then the smoke test against the real container stack.
Makefile	51	The short commands: up, sim, rviz, pipeline, mock, drive, test, down, clean.

Roughly 1 330 lines of Python across the pipeline, scripts, dashboard and launch file. The two highlighted service files are where all the interesting behaviour is.

One file defines the fleet

Why a configuration file generates the world instead of a human editing XML.

A robot in this system exists in six places at once: as a model in the simulation world, as a display group in the visualiser, as a state publisher and a static transform in the launch file, as a topic subscription in the producer, as an entry in the synthetic fleet, and as a target in the driver. Keeping six hand-written places in agreement is the kind of job that quietly stops being done.

So one file holds it, and the two file formats nobody wants to hand-edit are generated from it:

robots:
  - id: tb1
    model: turtlebot3_burger
    color: "0.9 0.5 0.0 1"
    spawn: {x: 0.0, y: 1.0, yaw: 0.0}
    drive: {linear: 0.15, angular: 0.3}    # drives a circle
  - id: tb2
    spawn: {x: 0.0, y: -1.0, yaw: 0.0}
    drive: null                            # never moves — the alert case
  - id: tb3
    spawn: {x: 0.0, y: 3.0, yaw: 1.57}
    drive: {linear: 0.1, angular: -0.5}    # circles the other way

generate_fleet.py

Emits worlds/fleet.sdf (420 lines) and config/fleet.rviz (127 lines). Both carry a "do not edit" header.

launch, pipeline, driver

Read the same yaml directly at start-up. No generation step, no duplication.

detector thresholds

Window size, variance threshold and minimum sample count live in the same file, under detector:.

endpoints

Kafka, QuestDB and Postgres addresses resolve environment variable first, yaml second — so a container overrides them without a code change.

The check that makes it hold

Generated files drift the moment somebody edits the yaml and forgets to regenerate. So CI regenerates them and runs git diff --exit-code on the results. If the committed world does not match the committed configuration, the build fails with the command needed to fix it. The guarantee is not "we remembered" — it is "the build will not pass if we did not".

One robot in the fleet has drive: null. That is not an oversight — it is the test fixture. A robot that never moves is exactly the condition the detector exists to catch, so the default configuration always contains one, and the smoke test refuses to run if none is present.

Two databases, on purpose

Telemetry and state are different problems, so they get different stores.

	QuestDB	PostgreSQL
Holds	Every position reading, and every alert event as it happened	The robot registry and the current state of each alert
Shape	Append-only, time-stamped, never updated	Rows that change: `open → acknowledged → resolved`
Written by	Line protocol over a raw TCP socket, port 9009	`psycopg2`, port 5432
Read by	Grafana and the terminal dashboard, over the Postgres wire protocol on 8812	Grafana and `fleet_cli.py`

The split is the point. A time-series database is built to accept a very high rate of immutable rows; asking it to update one row from open to resolved is using it against its design. A relational database handles that update trivially but is the wrong shape for tens of readings per second per robot, forever.

So the alert exists twice, deliberately, in two different senses. QuestDB records that a stuck event happened at a particular instant, and that record never changes. Postgres records that an alert is currently open, and that row is the one a human acts on.

CREATE TABLE alerts (
    id              BIGSERIAL PRIMARY KEY,
    robot_id        TEXT NOT NULL REFERENCES robots (robot_id),
    alert_type      TEXT NOT NULL,
    status          TEXT NOT NULL DEFAULT 'open'
                    CHECK (status IN ('open', 'acknowledged', 'resolved')),
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    acknowledged_at TIMESTAMPTZ,
    resolved_at     TIMESTAMPTZ
);

The CHECK constraint is doing real work: the set of legal states is enforced by the database, not by the Python that writes to it. A future service with a bug cannot put an alert into a state the rest of the system does not understand.

Grafana reads QuestDB through its Postgres wire protocol on port 8812, which means one datasource type covers both stores and no plugin is needed. Datasources and the dashboard are provisioned from files in the repository, so bringing the stack up produces working dashboards with nothing clicked.

The stuck detector

One rule, one state machine, and the two database writes it triggers.

The rule is deliberately simple, and simple is the right choice here: a robot that is not moving has a position that is not changing, and "not changing" is measurable as variance.

10window sizeThe last ten positions per robot, oldest dropped as each arrives.

5minimum samplesBelow this, the detector returns nothing — a robot that just appeared is not yet stuck.

0.001variance threshold, m²Population variance of x plus variance of y. Below it, the robot is considered stuck.

xs, ys      = positions in the window
variance    = ( Σ(x − x̄)² + Σ(y − ȳ)² ) / n
is_stuck    = variance < 0.001

moving → stuck      : QuestDB event + open an alert row in Postgres
stuck  → moving     : QuestDB event + resolve that alert row
otherwise           : nothing at all

Edge-triggered, not level-triggered

The detector keeps a set of robot identifiers it currently believes are stuck. An alert fires on the transition into that set and a resolution fires on the transition out of it. A robot that stays stuck for an hour produces exactly one alert, not one per message. This is the difference between a system somebody watches and a system somebody mutes.

Two layers of duplicate protection

The in-memory set survives only as long as the process. If the detector restarts while a robot is still stuck, it will see the transition again and try to open a second alert. So the database refuses:

INSERT INTO alerts (robot_id, alert_type, severity, pos_x, pos_y)
SELECT %s, %s, %s, %s, %s
WHERE NOT EXISTS (
    SELECT 1 FROM alerts
    WHERE robot_id = %s AND alert_type = %s AND status <> 'resolved');

One unresolved alert per robot per alert type, enforced in the write itself rather than by a read-then-write that two processes could interleave. Resolution is the mirror image — a single UPDATE over every unresolved alert for that robot, returning the number of rows it actually changed, which is what gets logged.

A detail that shows in the code

The detector skips malformed messages with a bare continue and a comment saying the consumer service dead-letters them. That is correct while both services run — but it does mean that running the detector alone, without the storage consumer, silently discards bad messages instead of parking them. The division of responsibility is real, and it is a coupling worth knowing about.

When something breaks

Four failure modes that were designed for rather than discovered.

A malformed message arrives

Parsing is a single function that raises on anything unexpected. The caller catches it, forwards the raw bytes plus the error text to robot_odom_dlq, logs a running count, and carries on. The poison message is inspectable afterwards; the consumer does not die.

This one is not hypothetical. The synthetic producer once emitted different key names for velocity and timestamp, and it crashed the consumer. The dead-letter path is the fix that came out of it.

A database or the bus is not up yet

Every connection helper retries instead of failing: Kafka thirty times at two-second intervals, QuestDB and Postgres twenty times at three. Services can be started in any order, and a slow container start is not a crash.

The QuestDB socket dies mid-run

The writer holds a raw TCP socket for the line protocol. A write that raises BrokenPipeError or OSError triggers a reconnect and one retry of that same line, so a database restart costs at most a reading rather than the process.

The stack is "started" but not ready

Every container declares a healthcheck and the stack is brought up with --wait, which blocks until they pass. Nothing in the scripts or CI sleeps for an arbitrary number of seconds and hopes.

Also worth naming

Both consumers use auto_offset_reset="latest". That is the right choice for live monitoring — a service that restarts should show what is happening now, not replay an hour of history — but it does mean a consumer started after a producer misses whatever went past in between. The smoke test starts its consumers before its producer for exactly this reason.

What CI actually proves

The four assertions, the numbers they produced, and what they leave untested.

Every push brings up the real container stack — Kafka, QuestDB, Postgres, Grafana — runs the synthetic fleet for 25 seconds at five messages per second per robot, then queries both databases and asserts four things.

Assertion	Query	Result
Odometry was stored	`count() FROM robot_odom`	369 rows
A stuck event was recorded	`count() FROM robot_alerts WHERE event = 'stuck'`	≥ 1
An open alert exists for the idle robot	`alerts WHERE status = 'open'`	≥ 1
Every robot registered itself	`count(*) FROM robots`	3 of 3

Three robots at five hertz for 25 seconds is 375 expected readings; 369 landed. The shortfall is the start-up and shutdown edges of a timed run, and it is the reason the assertion is greater than zero rather than an exact count — a test that demands 375 would fail on a slow runner for no useful reason.

The job also runs ruff over every Python directory, regenerates the world and RViz files and fails on any diff, dumps the last hundred lines of every service log if anything fails, and tears the stack down with its volumes whatever the outcome. Fifteen-minute timeout.

What it does not prove

CI exercises the Gazebo-free path only: synthetic producer to Kafka to both databases. The simulator, the bridge, the real ROS 2 producer node and the visualiser are not in CI and are not covered by any assertion. The end-to-end claim this project can make is about the data pipeline, not about the robots.

What this does not do

The honest boundary of the project, stated before anyone finds it themselves.

The simulator has never been watched running

The launch file has been validated by inspection only — the three-robot fleet has not been brought up in Gazebo and RViz and observed. Every claim on this page about the data pipeline is backed by a test; the claims about the simulation layer are backed by code review and nothing else.

Nothing is measured for speed

There is no latency figure from robot to database, no throughput ceiling, and no answer to how many robots this handles. The Kafka topic is single-partition with one consumer per group; partitioning by robot identifier is designed but not built, and the "50+ robots" question is open.

One detector, one rule

Position variance catches a robot that has stopped. It does not catch a robot moving wrongly, a velocity spike, or a robot that has gone silent — dropout detection needs a timer over absence of messages, which is a different structure and is not written.

Three simulated robots, no hardware

No physical robot has ever connected to this pipeline. The message schema is what a ROS 2 odometry topic provides, so a real robot would fit — but that has not been demonstrated here.

The producer is Python

The odometry bridge is a Python ROS 2 node. For a node in the hot path of every robot's sensor stream, C++ with librdkafka is the correct language, and rewriting it with a latency comparison against the Python version is the first item on the project's own list.

Running it

Two paths: the full simulator, or the pipeline alone on any machine.

Full stack

./scripts/start_all.sh          # everything

# or step by step
make up                         # Kafka, QuestDB, Postgres, Grafana — waits for healthchecks
make rviz                       # Gazebo Harmonic + RViz with the generated fleet
make pipeline                   # producer, consumer, detector
make drive                      # drive the robots per their configured patterns

No simulator needed

make up
make mock                                 # synthetic fleet into Kafka
python3 pipeline/kafka_consumer.py        # separate terminal
python3 pipeline/anomaly_detector.py      # separate terminal

This second path is the one CI uses and the one that runs anywhere Docker does. It needs no ROS 2 installation and no simulator.

Watching and closing alerts

python3 scripts/fleet_cli.py robots              # registry and online status
python3 scripts/fleet_cli.py alerts --status open
python3 scripts/fleet_cli.py ack 3
python3 scripts/fleet_cli.py resolve 3

Grafana

localhost:3000 — the Fleet Overview dashboard loads itself.

QuestDB console

localhost:9000 — raw SQL over the telemetry.

make test

The same four assertions CI runs, locally.

make clean

Stops everything and deletes the volumes for a full reset.

The short version

What to take away in four sentences.

Three simulated robots publish their position; a message bus carries it to two independent readers, one that stores every reading and one that watches for a robot that has stopped, and an alert opened by the second can be acknowledged by a human or closed automatically when the robot moves again.

The fleet is described in one configuration file that generates the simulation world and the visualiser layout, and continuous integration fails the build if the generated files and the configuration have drifted apart.

Telemetry and state live in different databases because they are different problems, malformed messages are parked in a dead-letter topic rather than dropped, and every service reconnects rather than dying when something it depends on is not ready.

What is proved is the data pipeline, by four assertions against the real container stack on every push. What is not proved is the simulator layer, the latency, or anything about scale — and those are stated on the page because a reviewer would find them anyway.