Tabletop Clutter Detector

A repair applied to something that is not broken is damage

bgsub alone leaves a bimodal error distribution: 47.3 % of objects recovered at IoU ≥ 0.9, and 38.6 % below 0.5 because touching objects merge into one region. The obvious fix — distance-transform watershed — made things worse when applied to every region.

Running the watershed everywhere redraws the boundary of regions that were already correct: the near-perfect tier collapsed from 47.3 % to 6.3 % while the merged tier only fell from 38.6 % to 25.0 %. Counting distance-transform seeds inside each connected region first, and leaving single-seed regions untouched, keeps the good tier and fixes the merges. A parameter sweep would never have found this — the sweep converged towards "split less", which was the wrong axis entirely.

A wrong prior still returns an answer

Global Otsu thresholding on greyscale is a reasonable-looking method: objects and table differ in brightness, so split the histogram. Under easy it scores 0.446. Under hard, where table value is uniform(0.25, 0.85) and object value uniform(0.45, 1.0), the two distributions overlap and the split lands inside the objects — mAP 0.143, a 3.1× collapse, detection rate 0.232. It never raised an error. It returned a mask every time, and its per-class numbers read like a weak detector rather than a broken one.

Classification is the bottleneck, not localisation

On hard, class-agnostic AP50 is 0.823 against class-aware AP50 0.698. The boxes are in the right place; the label on them is wrong 15 % of the time. Per class: sphere 0.695, cylinder 0.510, box 0.391. A sphere's silhouette is a circle from every direction, so nine shape features describe it completely. A box's silhouette under a tilted camera changes with yaw, and at 28 px there is not enough of it left for a linear rule. That gap is the specific thing a learned feature extractor should close — and it is now measured rather than assumed.

Occlusion is where the classical pipeline stops, not degrades

Recall at IoU 0.5 by visible fraction: 0.785 above 0.9 visibility, 0.348 between 0.5 and 0.9, and 0.000 below 0.5. Not a slope — a cliff. A partially hidden object has the wrong silhouette, so every shape feature it produces is wrong at once.

The classification gap closed — which the baseline predicted

Per-class AP: box 0.391 → 0.914, cylinder 0.510 → 0.910, sphere 0.695 → 0.908. The spread across classes collapses from 0.304 to 0.006. The learned extractor does not just score higher, it scores evenly — the class that was hardest for hand-designed features is now indistinguishable from the easiest. That was a stated prediction before training.

The occlusion cliff is gone — with a caveat that matters more than the number

Recall at IoU 0.5 by visible fraction: ≥0.9 → 0.785 becomes 1.000; 0.5–0.9 → 0.348 becomes 0.999; <0.5 → 0.000 becomes 0.910. A centre heatmap has no coupling to the silhouette — the centre of a half-occluded object is still a centre, and the size head regresses the full extent from partial evidence.

The caveat: n = 53. That is 0.6 % of the dataset, and 0.910 on 53 samples carries about ±0.04 at one sigma. The honest claim is narrow — the specific failure mode that stops the classical pipeline does not appear here. It is not evidence of robustness to heavy occlusion in general.

Localisation is essentially exact, and AP75 is where it shows

AP75 0.9896 against AP50 0.9899 — a gap of 0.0003. The baseline lost 0.134 between the same two thresholds. The metric's shift table gives the reading: a uniform 4 px error leaves AP50 at 1.000 while mAP has already fallen to 0.464, so AP50 alone cannot see localisation. The offset head is what buys this; it is measured at 2.00 px of worst-case centre error, which at stride 4 is exactly the quantisation it exists to undo.

The learned detector is slower, not faster

3.87 ms against 2.02 ms, single image, PyTorch eager, decode included. Worth stating plainly because the convenient story would be that the network wins on every axis, and it does not. Step 5 was set up to find out whether that was the architecture or the runtime.

Small objects still cost, in the expected direction

By size tercile: 0.805 (<25.5 px), 0.854 (25.5–31.9 px), 0.884 (>31.9 px). IoU is scale-relative, so a fixed pixel error costs a small box more. Output stride 4 means centres quantise to 4 px cells, which is 16 % of a 25 px object and 9 % of a 43 px one. The offset head removes most of that; the residual 0.079 spread is what is left.

The gap is appearance, and it is a cliff, not a slope

A detector trained on fixed appearance scores 0.90 on what it saw and 0.15 when the table colour and the light move. Detection rate falls to 0.29 — it does not misclassify seven objects in ten, it fails to see them. Nothing about the geometry, the camera, the shapes or the sizes changed. This is the number the whole track is about: a policy trained in an un-randomised simulator meets the real world in exactly this way, and the failure is silent — no error, just boxes missing.

Photometric augmentation recovers a third of the gap and stops

0.147 → 0.521, still 0.35 short of simply training on hard (0.874). The mechanism is visible in what each one varies. Augmentation perturbs the whole image with one brightness, one contrast, one gamma, one channel gain — a global nuisance model. The randomiser gives every object its own hue and moves the light source, which changes the shading direction on every face. Augmentation is a hand-written model of the nuisance; the randomiser samples the nuisance itself. A model cannot be augmented towards a variation nobody wrote down.

Geometric augmentation does nothing across the gap

0.147 → 0.181, within the noise of a single seed. Flip and translation add placement diversity, and placement was never the problem — the two regimes share one camera and one placement distribution. An augmentation only buys invariance along the axis it perturbs. Worth stating because the default recipe applies everything at once and then cannot say which part worked.

On top of the randomiser, photometric augmentation buys nothing

hard/none 0.8735 against hard/photo 0.8742: +0.0007, for 33 % more training time. The randomiser already covers the axis photometric augmentation would add. Every hour spent on an augmentation pipeline for a randomised simulator is an hour spent on nothing.

Geometric augmentation does add +0.016 within hard. Single seed, so suggestive rather than proven, but the direction is expected for a 6000-image dataset: flip and shift are extra placements, which the randomiser only samples 6000 times.

The randomised model transfers to the fixed regime for free

hard/none loses 0.005 going to easy. easy/none loses 0.75 going the other way. That asymmetry is the entire argument for domain randomisation: the randomised distribution contains the fixed one, so a model trained on it has already seen the fixed regime as a special case. Nothing was tuned for easy and nothing needed to be.

More data does not close an appearance gap

The full-budget easy/none model saw 75 % more images and 67 % more epochs and gained +0.027 on easy — and only +0.033 on hard, from 0.147 to 0.180. Coverage of the distribution is what transfers, not the number of samples drawn from the wrong one. When a model fails out of distribution, "collect more data" is only the right answer if the new data comes from somewhere new.

The runtime is worth 2.7× at eight threads — and 1.4× at one

The previous project found ONNX Runtime on one thread beating PyTorch eager on eight. That does not repeat here. The difference is the model: that network was 27 k parameters with a 16×16 output, so per-op dispatch overhead was most of the time and a runtime that removes overhead removes most of the time. This one is 14× larger with a 48×48×7 output; the convolutions themselves are the cost, and they cost the same arithmetic in either runtime. The refined rule: the runtime dominates while the model is small enough to be overhead-bound; once it is compute-bound, the thread budget matters as much as the engine. The earlier number was true and its generalisation was not.

Four threads is the knee, and eight is worse for eager

ONNX Runtime: 3.69 → 1.12 → 1.01 ms for 1 → 4 → 8 threads. 3.3× from the first four, 10 % from the next four. PyTorch eager on 8 threads (2.75) is slower than on 4 (2.54) — thread oversubscription on a batch-1 forward costs more than the parallelism returns. A node given 4 cores gets essentially everything this model can deliver; a node given 1 core is 1.8× slower than the classical pipeline it was meant to replace. The deployment question is not "how fast is the detector" but "how fast is the detector on the cores it will actually be given" — and that answer can land on either side of the baseline.

Decode is 15 % of the end-to-end latency at the best setting

0.18 ms against 1.01 ms of model, and it does not shrink with the runtime. On a small, fast model the post-processing stops being free — any deployment budget that lists only the forward pass is under-reporting by that much.

The first measurement was discarded, and why is the most reusable finding here

Step 5 first ran unattended immediately after the 96-minute ablation and recorded ONNX-8t 1.225 ms and eager-8t 3.773 ms. Same weights, same code, same box, re-run idle the next morning: 1.014 and 2.750 — 17 % and 27 % faster. The single-threaded classical baseline moved only 5 % in the same comparison.

WSL exposes no temperature sensor, so the cause cannot be read directly. But the box had just finished 1.5 hours at sustained 8-thread load, and the multithreaded numbers moved while the single-threaded one barely did. Three rules adopted for the rest of the track: a latency number is only reported with the conditions it was taken under; nothing timed directly after a training run is trusted; and both measurements stay in the repository, because a number that was silently replaced is worse than a number that was wrong.