Microduck on a CPU

In plain words

Training a walking policy for a small biped on a laptop CPU — after checking that it is possible at all.

The Microduck is a small open-source walking robot. Its authors publish the physics model, the meshes, and a training setup — which requires a CUDA GPU, because that is how modern robot-learning frameworks run thousands of simulated robots in parallel.

There is no GPU here and no hardware to buy. So this project runs the same physics model in plain CPU MuJoCo, parallel across processes instead of GPU threads, inside a budget of eight of the machine's fourteen cores, on a laptop that has other work to do.

Two steps are finished, and neither of them trains anything. That is deliberate.

What the first two steps are for

Step 1 is a feasibility gate: is this machine fast enough to train a policy at all, and what is the honest number to budget against? Step 2 is the environment contract: what does the policy see, what does it emit, when does an episode end, and what number does it have to beat? Both were written before any learning code, because a training run that fails is uninterpretable if you cannot rule out the machine and the environment first.

Why "sim-to-sim" is the real target

The robot's authors ship four physics variants of the same robot — including one with a passive backlash joint inserted in series with every actuated joint, which is a much more realistic model of a cheap hobby servo's slop. All four expose the same fourteen actuators, in the same order, under the same names.

That means a policy trained on one variant can be run on another with no remapping at all. The held-out physics is supplied by the robot's own authors rather than invented here — a far stronger position than randomising some parameters and then testing on the same randomisation, which is the standard and much weaker version of this experiment.

Terms used on this page

policyThe function mapping what the robot senses to what it commands. This is the thing being learned.

observationThe vector the policy reads each control step — here 48 numbers: joint positions and velocities, the previous action, the gyro, and gravity in body frame.

action scaleHow far one policy output is allowed to move a joint away from its neutral position. Too large and the robot thrashes.

PD controllerA fixed control law: command proportional to position error plus a term proportional to how fast that error is changing. No learning.

env-steps/sSimulation throughput. It is the number that decides whether training takes an hour or a week.

domain randomisationVarying physics and appearance during training so the policy is not tuned to one exact model of the robot.

system IDMeasuring the real actuator to find out how it actually behaves, which then sets an honest range to randomise over.

sim-to-realThe gap between a policy that works in simulation and one that works on the hardware. The whole point of the exercise.

The five steps

The five steps of the project, and which two are finished.

step 1 · doneFeasibility gate Asset fetch, model inspection, CPU throughput scaling, sustained-load decay, drop tests. PASS · budget ≈13 300 steps/s

step 2 · doneEnvironment contract 48-dim observation, 14-dim action, 50 Hz, fixed-length episodes with seeded pushes, multiprocess vector env. 27 tests · PD baseline 108.7

step 3PPO against the baseline Reusing the implementation from the previous project. Metric: recovery rate under randomised pushes, n = 100. opens on the reward decision

step 4Domain randomisation Over the four measured actuator classes, evaluated on the backlash variant as held-out physics. Report the gap. not started

step 5ONNX export Single-thread latency, verified against PyTorch two ways. not started

Every file, and what it is for

Every source file, how big it is, and the single job it owns.

File	Lines	What it does
common.py	203	The shared contract. The four variant filenames; the fourteen actuator names written out rather than read from the model, so a test can catch an upstream reordering that would silently permute every action a trained policy emits; the 500 Hz / 50 Hz timing; the four measured actuator classes; and — critically — `actuated_qpos_index()`, which computes the strided joint index instead of assuming a contiguous slice.
env.py	323	The environment. The 48-dim observation layout and its per-group scaling, the residual action mapping, the six-term reward, the push schedule, episode bookkeeping, and the `mj_forward` that keeps every observation channel on one timestamp.
vec_env.py	190	A hand-written multiprocess vector environment. Forks N workers over pipes, one environment each, capped at the eight-thread budget. Environment i is seeded `seed + i`, and a test asserts that N workers reproduce N sequential environments bit for bit — so a result cannot quietly depend on how it was parallelised.
baseline.py	180	Runs the shipped PD controller commanded to hold the standing pose over twenty seeds of the full task, and writes `runs/baseline.json`. Also measures the three open risks step 3 inherits.
bench.py	466	The CPU benchmark and the most defensive file here. Brackets every configuration with a single-process reference window and refuses to certify a table if that reference drifts more than 10 %; samples `/proc/<pid>/stat` twice to measure what else is on the box right now; refuses more than eight processes without an explicit flag; and tests a sustained curve for a plateau in both directions rather than only checking whether it fell.
bench_wrapper.py	97	Measures the Python environment wrapper's cost against a bare `mj_step` loop doing the same substeps, back to back in one process so the comparison is clean.
drop_test.py	159	Two physics sanity tests — a limp ragdoll drop and a commanded pose hold — with a settle detector and a rendered filmstrip.
inspect_model.py	92	Dumps what each MJCF variant actually contains: degree-of-freedom counts, joints, actuators, sensors, contact masks.
test_model.py	161	The MJCF contract. Catches a reordered action space, a passive joint shifting the state vector, a contact bitmask that drops the robot through the floor, and an upstream refit of a servo class. None of those raise on their own.
test_env.py	263	Twenty-seven environment checks, including the one that rebuilds the observation from its five sources and asserts a bit-for-bit match, and the one that runs the observation builder on the held-out backlash model against the obvious shortcut.
fetch_assets.sh	53	Pulls the MJCF and 86 STL meshes from a pinned upstream commit. Nothing under `assets/` is committed: the 3D files are Creative Commons BY-SA-NC, and a pinned fetch script makes clear where the robot came from in a way that vendoring 23 MB of someone else's binaries would not.
verify.sh	97	Six tiers. The first five are cheap and check every structural claim in a few minutes; tier 6 is the benchmark and is the only part that needs the machine to itself. Prints the resolved interpreter, because `nice` reports a missing interpreter as `No such file or directory`, which reads like a missing script.
runs/ · out/ · assets/	—	Benchmark JSON (including both the contaminated and the clean latency runs), the baseline file, drop-test filmstrips, and the fetched upstream model.

Step 1 — the feasibility gate, and four wrong answers

The throughput measurement that decides whether the project is feasible, and the four wrong numbers before the right one.

The unit, and the gate

One environment step is one 50 Hz control decision, which is ten physics steps at the model's 500 Hz timestep. Reinforcement-learning budgets are quoted in environment steps, so the gate is written in them, and it was set before measuring:

The gate, fixed in advance

Below 5 000 environment-steps per second at eight processes, walking leaves the scope and the project becomes stand-only.

MuJoCo's step for a sixteen-body model is single-threaded and small enough to stay in cache, so parallelism has to come from processes. Every worker sets OMP_NUM_THREADS=1 before MuJoCo loads, or the workers each spawn a thread pool and fight each other.

Scaling, 20 s per configuration

Processes	env-steps/s	Per process	Speed-up	Efficiency
1	6 521	6 521	1.00×	100 %
2	12 555	6 278	1.93×	96 %
4	21 424	5 356	3.29×	82 %
8	28 749	3 594	4.41×	55 %

Gate: PASS, 5.7× over. The single-process reference window between configurations held to within 6 % across the whole sweep, so the four rows were measured on the same machine as each other. Run twice, seven minutes apart: the eight-process row came back 28 960 and 28 749 — 0.7 % apart.

The first set of numbers, and why they are the interesting ones

Step 1 originally certified a different table: 6 666 / 9 801 / 14 275 / 20 474 — same code, same variant, same thread budget, same laptop on mains power. The difference is a host state the guest cannot read.

Read the shape rather than the totals, because the shape is the diagnosis. Single-process is flat within run-to-run spread; every multi-process row is roughly a third lower. Thermal throttling decays with time under load and would have moved the reference windows within the sweep. Another process competing for CPU depresses every row, the single-process one included. A sustained all-core power limit does exactly this and nothing else — single-core turbo never depended on the all-core budget, every additional core did.

The tell that should have caught it at the time

Two processes on a fourteen-core machine returned 74 % efficiency. There is no core-count explanation for that — two workers cannot contend for cores when twelve are idle. It was read as a property of the chip, and a scaling story was built on top of it. It was a property of the machine's power state that afternoon.

There was a deeper excursion in the same window: the single-process rate fell to 3 069 env-steps/s — 46 % of certified — with the box idle. The benchmark's reference bracket called that run stable, and it was right to: the tool detects a reference that moves during a sweep, and this was a machine that was uniformly slow for the whole sweep. That is a real limitation of the method — a drift guard cannot catch a bias that is already in place when the first window opens. The only defence is an absolute expectation, which is what the certified table now provides.

Both recovered after a check on the Windows side. Which setting changed is not recorded, and that is the finding rather than a hole in it: a WSL2 guest cannot read power mode, charger wattage, package power or core frequency, so host state is unrecoverable after the fact and has to be written down at measurement time or lost.

What a training run actually gets — three reasons it is not 28 749

First, the sweep pauses and a training run does not. The sweep brackets every configuration with a reference window, which lets the processor package cool. Holding eight processes flat out for four minutes:

Elapsed	env-steps/s	vs peak
20 s	33 349	—
60 s	≈30 900	−7 %
120 s	27 313	−18 %
180 s	≈24 600	−26 %
240 s	19 256	−42 %

Monotonic in all twelve windows, and the final step is −1.0 %, so the curve has just about flattened. Run in the reduced-power state, the same test read 27 905 in its first window and then eleven windows at 20 152 ± 287 — a 5 % spread with no trend. The two runs are worth more together than separately: their floors agree to 3.9 %. The host power state changed how much burst the machine had to spend, not where it ended up.

Second, the benchmark does not measure the thing that gets trained. Its worker is a bare mj_step loop on the walk variant. The task runs the Python environment wrapper on groundcontact, and both of those cost:

Single process, 10 s windows	Bare `mj_step`	Through the wrapper	Wrapper cost
`walk`	7 505	5 692	32 %
`groundcontact`	5 223	4 175	25 %

About seven points of that 25 % is a correctness fix rather than overhead — see the defect list below — and it is explicitly not an optimisation target.

Third, groundcontact does not scale like walk. The variant with five times the ground-collidable geometry loses efficiency earlier and ends thirteen points lower at eight processes: 42 % against 55 %. That is the signature of a shared resource outside the core — cache or memory bandwidth — and against a performance-core/efficiency-core split, which would not care how much contact solving each process does.

The number a training run should be budgeted on

Measured directly, on the variant that gets trained, under the load it gets trained at, over eighteen windows. The shape is not the walk shape: groundcontact undershoots at window 11 and then climbs back for six windows, ending 11 % above its minimum and still rising +0.6 % at the last window. So this is not a steady state either, and eighteen windows found the dip rather than the floor.

Window 11, the global minimum, is contaminated — a ten-second two-process benchmark of the author's own overlapped it — and is excluded. Excluding it, the last nine windows run 15 806 to 17 266, a 9 % band, mean 16 590. That band is the honest uncertainty, and the budget uses its middle rather than its top.

16 590 × 0.799 (wrapper) = ~13 300 env-steps/s
range 12 600 – 13 800 across the band

The correction that went the other way — and it is the most reusable finding here

The tally for this one number is 28 749 → 18 400 → 8 000 → 13 300. Three corrections downward from unmeasured optimism, and then one upward, because the 8 000 figure had been built by multiplying an already-depressed measurement by a sustained derate borrowed from a different variant — the derate was counted twice.

Being conservative is not free and it is not automatically honest. An unmeasured pessimistic assumption is the same error as an unmeasured optimistic one, and it costs real scope. The fix in both directions was identical: measure the thing itself instead of composing estimates of its parts.

Training budget	Wall time at ≈13 300 steps/s
10 M — hyperparameter probe	13 min
50 M — stand + push recovery, expected	1.0 h
100 M — stand, generous	2.1 h → 2 chunks
400 M — walking gait, upstream-scale	8.4 h → 5 chunks

Scope conclusion: stand-and-recover is one comfortable sitting; walking is five chunks. That was the question the gate had to answer, and it survived all four revisions of the number — which is the only reason the revisions were tolerable.

Why efficiency falls to 55 %, and what could not be determined

The chip is an Intel Core Ultra 5 225H: fourteen cores, no hyperthreading, heterogeneous — performance cores alongside efficiency cores. The obvious hypothesis is that workers 5–8 land on the slower ones. Pinning four workers to CPUs 0–3 and then to CPUs 10–13 gave a 15 % difference under load and 6 % single-threaded — far too small for a performance/efficiency split, which would show roughly 2×.

The explanation is that CPU affinity inside WSL2 does not pin to a physical core. sched_setaffinity binds the process to a virtual CPU, and the hypervisor schedules virtual CPUs onto physical cores on its own. So that experiment cannot answer the question, and neither can any other experiment run from inside the guest.

Pinning was the wrong instrument. Changing the workload worked. The groundcontact sweep loses efficiency earlier and ends thirteen points lower on the same cores in the same session, and a core-type split cannot produce that — which physical core a process lands on does not depend on how much contact solving it does. That points at cache and memory bandwidth over the other candidates. It is one comparison between two variants, not a proof, and it is recorded as unresolved rather than resolved with the plausible-sounding answer. What matters operationally is settled anyway: eight processes is the right choice, because it delivers the most total throughput even at 55 % efficiency.

Three things about the model that changed the plan

Three properties of the robot model that forced the plan to change.

On the `walk` model, only the feet can touch the ground

MuJoCo lets two geometries collide only if one's contype shares a bit with the other's conaffinity. In the walking model the floor is 1/1 and the two foot geometries are 1/1 — but every other body and limb geometry is 2/2. Two and one share no bits, so the trunk, head and legs cannot touch the floor at all.

Variant	Geometries that can touch the floor	Trunk at rest
`walk`	2 (the feet)	−10.5 cm — through the floor
`groundcontact`	10	+3.2 cm — resting on it

This is deliberate upstream: a walking task ends the episode the instant the robot falls, so what happens afterwards never has to be physical, and dropping the collision geometry makes the simulation faster. It makes walk the wrong model for anything involving lying on the ground or getting back up. The failure is silent — nothing raises, no contact is reported, the robot just sinks, and a reward function reading trunk height would have been quietly rewarded for falling through the world. Decision: the task uses groundcontact.

The backlash variant renumbers the state vector — and it is the evaluation model

Variant	nq	nv	Passive joints	Action space
`walk`	21	20	0	identical
`groundcontact`	21	20	0	identical
`rollers`	25	24	4	identical
`walk_backlash`	35	34	14	identical

That the action space is identical across all four is what makes the sim-to-sim step possible. But walk_backlash inserts a passive joint in series with every actuated joint, and those passive joints interleave: the actuated joint angles sit at state indices 7, 9, 11 … 33, not the contiguous 7…20 they occupy elsewhere.

Any observation builder that hard-codes qpos[7:21] reads a mixture of joint angles and backlash deflections on the evaluation model — plausible numbers, wrong meaning, no error. Measured, the shortcut is wrong on 13 of 14 joints, by up to 0.938 rad. Everything here goes through common.actuated_qpos_index() instead, and a test asserts the stride.

Joint friction is the parameter the hardware people disagree about by 6.7×

The upstream properties file ships four fitted actuator classes for the same servo — different people, different benches, all left in the file.

Class	Damping	Friction loss	Armature	kp	Force limit
`chosen_actuator`	0.053	0.0048	0.0018	0.550	0.96
`chosen_actuator_old`	0.048	0.0060	0.0020	0.520	0.91
`chosen_actuator_new`	0.041	0.0320	0.0020	0.386	0.67
`chosen_actuator_antoine`	0.044	0.0130	0.0017	0.430	0.75
spread (max/min)	1.29×	6.67×	1.18×	1.42×	1.43×

Damping, armature, gain and torque limit agree to within about 40 %. Friction loss disagrees by a factor of seven. This is a measured parameter uncertainty, not a guess, and it is the honest place to get a domain-randomisation range from — rather than the usual ±20 % around a nominal, chosen because it sounds reasonable. It also says where to spend the randomisation budget: wide on friction, narrow on everything else. A test parses the XML and fails if upstream refits a servo.

Do the physics behave? Two drop tests

Two physics checks confirming the model falls, settles and collides the way it should.

Limp — a ragdoll from 25 cm

Actuator gains and biases zeroed so the robot has no muscle at all. Note that commanding ctrl = 0 does not do this: a MuJoCo position actuator produces gain·ctrl + bias₁·qpos + bias₂·qvel, so a zero command is a stiff hold at the zero pose. All three coefficients have to go.

It falls, lands, and comes to rest at trunk height 3.0 cm with 14 contacts and a maximum joint speed of 0.09 rad/s. No sinking, no jitter, no explosion.

Hold — commanded to the standing pose

The actuators are commanded to the STAND keyframe, which is where every training episode will begin. It holds for 0.79 s, then topples, and settles face-down at 4.30 s.

Why that number matters

The shipped PD controller at kp = 0.55 does not hold the pose. So STAND is an unstable equilibrium, standing is a real balancing problem rather than a pose hold, and the baseline the learned policy has to beat is easy to state: falls over in 0.79 s.

The same test on walk topples at the identical 0.79 s — the dynamics match until the body reaches the floor — and then sinks to −10.5 cm, which is the contact-mask finding again, visible as a number.

Step 2 — the environment contract

The 48-dimensional observation, the action scaling, and the six reward terms.

The observation is 48-dim. Step 1 said 61, and 61 was wrong

61 is everything the simulator knows about the robot. 48 is everything the robot knows about itself. The difference is thirteen numbers that are free in MuJoCo and do not exist on hardware.

14joint positionservo present-position

14joint velocityservo present-velocity — real, and noisy. Included because the servos report it, not because it is clean

14previous actionthe policy's own last output; it is in RAM

3gyroIMU rate gyro

3projected gravityworld −Z in the trunk frame — the observable part of attitude

3IMU linear velocitya velocimeter. There is no state estimator on the robot, so base linear velocity is not measurable

3trunk positionworld-frame xyz — same problem, and no external tracking rig in the loop

4full orientation quaternionyaw is not observable from a gyro and an accelerometer; the model declares no magnetometer, so heading can only be integrated, and drifts

3root angular momentuma MuJoCo computation over the body tree, not a sensor

Why cut them before training rather than after

A policy that reads those learns to depend on them and then has nothing to run on. Since the only claim this project can honestly make is about transfer, the observation is cut down to sensors the robot carries before any training happens. It costs nothing now and cannot be retrofitted later.

One further detail: the gyro is read from the angular-velocity sensor rather than its twin imu_ang_vel. Both sit on the same site and read identically today, because MuJoCo applies a sensor's declared noise only when a flag is set and it is not set here. But angular-velocity is the one upstream put noise="0.005" on — so step 4 flips the flag and gets the noise magnitude the robot's authors chose, instead of one invented to look reasonable.

A test rebuilds the observation from those five sources and asserts the environment's output matches bit for bit. That is the only way to prove nothing simulator-only leaked in — and it separately checks that the excluded velocimeter is reading a live non-zero signal, so the exclusion is a real one rather than a channel that happens to be zero.

Actions, and a trap in the model file

Fourteen outputs in [−1, 1], applied as residuals around the standing pose:

target = STAND_pose + 0.35 * action        then clipped to the joint limits

A saturated action moves a joint 0.35 rad — about 22 % of the median joint range — in one 20 ms decision. That number is a hyperparameter, not a derived quantity, and step 3 is set up to report what happens at 0.2 and 0.5.

The clip is not decoration

ctrlrange on all fourteen actuators is [−10, 10] rad, while the tightest joint limit — hip roll — is ±0.384 rad. Handing a position actuator a 10 rad target raises nothing: it saturates against the joint stop and spends the entire force range holding there. Nothing in the model prevents this and nothing reports it.

Episodes and pushes

250 steps at 50 Hz — five seconds — fixed length with no early termination. The usual locomotion setup ends the episode the moment the robot falls. That is right for walking and wrong here: recovery is half the task, and terminating on a fall makes falling unrecoverable by construction. It also keeps returns comparable — every episode is the same length, so a baseline that topples early gets a low return rather than a short episode that looks cheap.

Three pushes per episode, drawn from the episode seed: magnitude uniform in 0.15–0.45 m/s applied to the trunk's linear velocity, direction uniform in azimuth, timing uniform but held half a second clear of both ends. A push at t = 0 is an initial condition rather than a disturbance, and one at the buzzer is never recovered from inside the episode.

The reward — six terms, all logged separately

Term	Weight	Shape
upright	+1.0	trunk z-axis against world up, floored at 0
height	+1.0	Gaussian on trunk height about 0.12 m, σ = 3 cm
posture	−0.10	mean squared joint deviation from STAND
action rate	−0.05	mean squared change in action
joint velocity	−2e−4	mean squared joint velocity
effort	−0.02	mean squared actuator force

A perfectly held stand scores 2.0 per step, so 500 is the episode ceiling. Each term is logged separately in the step info so step 3 can show which one is doing the work, rather than reporting a single scalar and calling it tuned.

Observation scaling, and the vector environment

Each group is divided down so no input dominates by unit choice alone: joint positions and projected gravity are already order 1, joint velocity reaches ~20 rad/s in a fall and is scaled by 0.05, and the gyro by 0.25. Whether those scales are right is one of the three open risks below — they are not.

vec_env.py forks N workers over pipes, one environment each, capped at eight. Environment i is seeded seed + i, and a test asserts that N workers reproduce N sequential environments bit for bit — so a run is reproducible at any worker count, and a result cannot quietly depend on how it was parallelised.

The number a learned policy has to beat

The hand-tuned PD controller that any learned policy has to beat.

The shipped PD controller commanded to hold STAND — which is exactly what a zero action emits — over twenty seeds of the full task, pushes and initial-state noise included.

Measurement	Value
return	108.7 ± 2.9 of a 500 ceiling
fraction of the episode on the floor	54 %
starts to tilt (upright cos < 0.9)	0.65 s
trunk reaches the floor (height < 4 cm)	2.28 s

The last two reconcile the drop test with this one: 0.79 s was the toppling threshold, 2.28 s is ground contact. Same event, measured at two points on the way down.

Why this file exists at all

The first version of these numbers — 108.9 ± 2.8, with a ground-contact time of 2.54 s against a threshold that was never written down — came from an ad-hoc script that was not kept. The return survived re-measurement to within the noise it already reported. The contact time did not, and there is now no way to tell whether that is the correctness fix or a different threshold. That is the argument for a tracked script and a JSON file rather than a number in a notebook.

Five silent defects the tests did not catch

Defects the test suite passed over, and what each would have cost during training.

Step 2 closed by reviewing the environment against the thing that is about to consume it — a PPO loop — rather than against its own tests. Twenty-seven contract tests had passed. Five defects came out of that review, all silent, all now fixed with a test that fails without the fix.

The observation mixed two instants

mj_step integrates position and velocity to t+1 but leaves sensor data and body positions at t — so the gyro was 2 ms older than the joint angles beside it. Every number was plausible. Measured, it was 0.054 rad/s on the gyro and 1.4 mm on trunk height against a 30 mm reward sigma. It attacks the one claim this project makes — that every observation channel is one real hardware could produce — and no IMU disagrees with its own encoders by a timestep. Fixed by calling mj_forward before reading anything, which costs 7 % of throughput and is a correctness cost, not overhead.

The reset clamp never executed

np.clip(qpos[idx], lo, hi, out=qpos[idx]) with an integer index array writes into a temporary copy — fancy indexing does not produce a view. Latent at the default initialisation noise, because the tightest joint sits 0.297 rad clear of its limit. At the larger noise step 4 will use, two joints of fourteen start 0.086 rad outside their range and MuJoCo applies a limit impulse at t = 0.

The done flag carried no truncation information

Every episode ends on the step limit and none on a terminal state. A stock GAE loop zeroes the bootstrap at done, which at γ = 0.99 corrupts the value target about a hundred steps back into a 250-step episode. The training curve still goes up. This is the same bug the previous project in this track designed its environment specifically to make impossible.

A dead worker reported nothing useful

A worker that died gave the parent a bare EOFError naming neither the worker nor the cause, and bad environment arguments failed the same way out of the constructor. The child's traceback does reach stderr — but in a redirected training log it is nowhere near the failure.

Stepping before resetting produced a valid-looking, push-free episode

And the vector environment's constructor raising a validation error emitted an unrelated AttributeError from its destructor on top of it. One test in this repository was doing exactly the first thing and discarding the result.

The point of listing these

Twenty-seven contract tests say the environment does what it claims. A review against its consumer found five things the tests did not. Neither can say the task is learnable — that is what step 3 is for.

Three open risks — decisions, not bugs

Choices still open, listed as decisions rather than as bugs.

1 — Three of the four reward penalties are numerically dead

Term	Weight	PD baseline	Random actions
upright	+1.0	70.0	67.9
height	+1.0	38.8	40.8
action_rate	−0.05	0.00	−8.33
posture	−0.10	−0.15	−0.22
effort	−0.02	−0.01	−0.07
joint_vel	−2e−4	−0.00	−0.04

action_rate is zero for the PD baseline only because a constant action has no rate — it is live. The other three are under 0.25 % of the return under both policies. joint_vel's weight was chosen for the ~20 rad/s of a fall, and measured joint speed is under 1 rad/s — 400× smaller once squared. So the plan to report which penalty is doing the work would report three zeros, and effort and velocity regularisation, which is usually what governs whether a policy transfers, is absent in practice. Reweighting changes the baseline, so it is a step-3 decision rather than a step-2 edit.

2 — The reward is informative about standing and nearly flat about getting up

Per-step reward while upright (cos > 0.9) is 1.92 ± 0.05; while down (cos < 0.3) it is 0.12 ± 0.04. The level gap is 16×, so the return clearly prefers standing and the task is not degenerate. But upright is floored at 0 past 90°, and height is a 3 cm Gaussian worth about 1e−3 at floor level — so inside the fallen region the reward barely says which way is up. Expect exploration, not reward shape, to decide whether recovery is learned, and expect fall slowly and stay tilted as the competing local optimum.

3 — The observation scaling is a guess, and the measurement says it guessed wrong

Group RMS under random actions: projected gravity 0.577, previous action 0.575, gyro 0.211, joint position 0.094, joint velocity 0.044. A 13× spread — with the 28 dimensions that describe the body's configuration carrying the least variance of all, so an unnormalised first layer attends mostly to the policy's own previous output. Step 3 needs a running observation normaliser rather than a better fixed guess.

Scope, now that the gate has been measured

	Item
In	Stand and recover from pushes, on `groundcontact`, ~50 M environment steps — 1.0 h at the measured rate. One comfortable sitting, checkpointed anyway.
In	Domain randomisation over the four measured actuator classes, evaluated on `walk_backlash` as held-out physics.
In (was deferred)	A walking gait, 400 M steps — 8.4 h in five chunks of ≤2 h, against the 36 h that had put it out of reach.
Out	Anything requiring mjlab, MuJoCo Warp, or a GPU.

What steps 1 and 2 do not prove

Nothing here trains anything. Throughput is measured with random actions around STAND. A real training loop adds policy forward passes, advantage computation and optimiser steps on top, so the measured rate is an upper bound on the rollout half only.
The training-rate figure has one inherited factor. The wrapper cost was measured single-core on an idle box, not under eight-process load.
Neither sustained run reached a steady state. One was still falling 1.0 % per window after twelve; the other bottomed out at window 11 and was still climbing 0.6 % after eighteen. The 9 % plateau band is the honest uncertainty on the budget rate.
The throughput figure has been wrong four times. Treat any number here as provisional until a script in the repository reproduces it.
The 0.79 s fall time is one deterministic rollout from one keyframe. It is a baseline to beat, not a distribution — the 108.7 ± 2.9 return over twenty seeds is the distributional version.
The reward has never had anything optimise against it. Three of its four penalties are measurably inert and the fallen region is nearly flat; both are written up rather than quietly retuned, because changing either invalidates the baseline they would be measured against.
No observation noise, no domain randomisation, no actuator variation. One nominal physics model. Step 4 adds the spread.
Run-to-run spread is about 1 % from two runs seven minutes apart — not enough samples to call it a distribution, and it says nothing about spread across days, where the host power state is the dominant term and has already moved these numbers by 40 %. Two significant figures is all any of this supports.

Running it

Setup, the commands in order, and the runtimes to expect on a CPU.

git clone https://github.com/AungKaung1928/microduck-rl-cpu.git
cd microduck-rl-cpu
python3 -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt

./fetch_assets.sh        # pinned upstream commit, ~24 MB, gitignored
./verify.sh              # tiers 1-5 are cheap; tier 6 loads the box

The cheap checks, individually

python3 test_model.py     # the MJCF contract: variants, actuator order, classes
python3 test_env.py       # 27 environment contract checks
python3 inspect_model.py --all
python3 drop_test.py --mode limp --z0 0.25 --seconds 5
python3 baseline.py --seeds 20

The benchmark — needs the machine to itself

nice -n 10 python3 bench.py --seconds 20 --ref-seconds 10 --tag main
nice -n 10 python3 bench.py --sustained 8 --seconds 20 --windows 12 --tag sustained
nice -n 10 python3 bench_wrapper.py --seconds 12        # 1 core, no load

The only hard dependencies are mujoco and numpy. Rendering is needed only by the drop-test filmstrips and goes through whatever MUJOCO_GL names.

bench.py refuses to run more than eight processes without an explicit flag, and refuses to certify a table if its single-process reference drifts more than 10 % during the sweep.

Licence

Upstream code is Apache-2.0; the 3D model files are Creative Commons BY-SA-NC. assets/ is therefore gitignored and reproduced by fetch_assets.sh from a pinned commit. Code in the repository is the author's own.

Measured environment

Intel Core Ultra 5 225H, 14 cores, no hyperthreading, 21 GB available to WSL2. Windows 11 + WSL2, Ubuntu 22.04, Python 3.10, MuJoCo 3.12, software rendering through WSLg. No CUDA anywhere.

The short version

The whole project compressed into one paragraph.

A balance-and-recover task for a 14-servo open-source biped, built to run entirely on a laptop CPU because the upstream training stack requires CUDA. Step 1 is a feasibility gate whose answer arrived four times — 28 749, then 18 400, then 8 000, then 13 300 environment-steps per second — and the write-up keeps every revision, including the one that was wrong in the conservative direction because a sustained derate had been counted twice. Along the way it found that the walking model's contact bitmask lets the robot fall through the floor, that the backlash variant renumbers the state vector so the obvious joint slice is wrong on thirteen of fourteen joints, and that the robot's own authors disagree about joint friction by 6.7× — which is where the domain-randomisation range will come from. Step 2 defines the environment: a 48-dimensional observation cut down from 61 so that every channel is one real hardware could produce, 14 residual actions with a clip that exists because the model's control range is 26× wider than the tightest joint limit, fixed-length episodes with seeded pushes so recovery stays in scope, and a six-term reward with a measured PD baseline of 108.7 ± 2.9 out of 500. A review of the environment against its future consumer found five silent defects that twenty-seven passing contract tests had not, and three open decisions — three dead penalties, a nearly flat reward in the fallen region, and an observation scaling that measurement says is wrong — are documented rather than quietly retuned, because changing them invalidates the baseline they would be measured against.