PPO Against LQR

The reason

PPO will fail to learn several times before step 5 is finished. Every time it does, the first question is algorithm or environment. That question is only cheap to answer if the environment was verified before any reinforcement learning existed. This one was.

The dynamics

Barto, Sutton & Anderson (1983), appendix. θ is measured from upright; positive θ leans the pole toward +x.

temp      = (F + m·l·θ̇²·sin θ) / (M + m)
θ̈         = (g·sin θ − cos θ · temp) / (l·(4/3 − m·cos²θ/(M+m)))
ẍ         = temp − m·l·θ̈·cos θ / (M + m)

Two conventions hide inside those three lines and both are classic bug sources. l = 0.5 is the half-length, not the full pole length. And the 4/3 is 1 + I_cm/(m·l²) for a uniform rod, whose moment of inertia about its own centre is m(2l)²/12 = m·l²/3. Getting the half-length convention wrong is the most common bug in a hand-written cart-pole, so one check exists purely to catch it.

Three decisions worth arguing with

Complete list of deviations from the standard CartPole-v1: observations are float64 not float32; stepping a finished episode raises rather than warning; the two done flags are never merged. Everything else — constants, equations, integrator and its ordering, initial distribution, thresholds, reward, 500-step cap — matches.

What LQR is

Linear-Quadratic Regulator. If a system is linear and the cost you want to minimise is quadratic, the optimal controller is a matrix multiplication: u = −Kx. The gain K is computed from the model, not learned from data. Cart-pole is not linear, but near upright it is close enough — so linearise, solve, and use the result.

Linearising

Dropping second-order terms about the origin, with D = l(4/3 − m/M):

θ̈ = (g/D)·θ − (1/(M·D))·F
ẍ = F/M − (m·l/M)·θ̈

Ac[3,2] = g/D = 15.7756 s⁻², and √15.7756 = 3.9719 s⁻¹ is the growth rate of the free cart-pole — slightly faster than the pinned-cart 3.834058 s⁻¹ measured in step 1, because a free cart lets the base slide out from under the pole. Two related numbers, both derivable, both matching.

Discretisation is exact here — and that is the payoff for step 1's integrator choice

Every acceleration in euler_step() is evaluated at the old state, so the plant genuinely is s' = s + τ·f(s,u) and therefore

Ad = I + τ·Ac        Bd = τ·Bc

with no approximation at all. A semi-implicit or exact zero-order-hold plant would need a matrix exponential here. It was verified against a finite-difference Jacobian of the actual step function anyway: max absolute difference 7.57e−14.

Riccati by iteration, not by library call

P_{k+1} = Q + Adᵀ P_k Ad − Adᵀ P_k Bd (R + Bdᵀ P_k Bd)⁻¹ Bdᵀ P_k Ad,   P₀ = Q
K       = (R + Bdᵀ P Bd)⁻¹ Bdᵀ P Ad,      u = −Kx

P_k is the cost-to-go matrix with k steps of horizon remaining, so this is literally value iteration for a quadratic value function. It is the same object PPO's critic estimates by regression — the difference is that here it can be computed exactly, which is the whole reason to write the recursion out longhand.

Converged in 866 sweeps / 22 ms. The theory says the convergence rate is ρ(Ad−BdK)² = 0.96810 per sweep, predicting about 923 sweeps to reach 1e−13 relative. Measured 866. That agreement is the check that the recursion is the textbook one and not something that merely happens to converge.

K = [−0.910126, −2.132488, −30.594563, −7.841506]
ρ(Ad − Bd·K) = 0.983919

Bridging a continuous controller to two discrete actions

The environment offers ±10 N. LQR returns a real number. The bridge is a design decision, not a detail:

action = 1 if −Kx > 0 else 0

Bang-bang control with an LQR-derived switching surface. Only the sign of −Kx reaches the plant, so the policy is invariant to any positive rescaling of K, and therefore to scaling the whole cost — which is exactly what check L6 confirms.

Return is a saturating metric — measured, not asserted

An early claim in the repository was that the cost weight R cannot matter because it only rescales K. That is false — the Riccati equation is not linear in R — and sweeping R from 0.01 to 100 rotates the normalised gain direction by 0.130. The check caught it. The interesting part is what happened next: that rotation moved the measured return by 0.0. Every one of those controllers scores 500.0.

So return cannot rank controllers on this task once they are all merely good enough. Steps 4 and 5 therefore rank on steps-to-threshold, and this repository does not contain a single reward curve.

What can tell controllers apart is the basin — how large a disturbance they can recover from — found by bisecting on the initial state one axis at a time:

Bang-bang has the larger basin than the same law with a proportional actuator of equal strength. Not a paradox: recovering from a large disturbance is a time-optimal problem, and the time-optimal solution to a bounded-input problem is bang-bang. The linear law under-commands near the edge because it is minimising a quadratic cost, not maximising survival. Angle finds nothing — every angle inside the termination limit is recoverable, which is why the sweep runs on velocity.

The four ingredients

PPO is policy gradient plus four specific additions. Each is written longhand here, and each can be switched off from the command line — which is what step 5 measures.

Architecture: separate 64-64 tanh actor and critic trunks, orthogonal initialisation (gain √2, 0.01 on the policy head, 1.0 on the value head), Adam with eps = 1e−5, linear learning-rate anneal, gradient-norm clipping at 0.5, and 8 environments × 128 steps = 1024-step batches.

The truncation bootstrap — what step 1 was for

At the 500-step cap the pole has not fallen. The episode was cut by the clock. Bootstrapping V(s_final) there is correct; treating it as terminal teaches the critic that surviving 500 steps is worth zero future return. That is the single most common silent bug in a hand-written PPO, and it is only avoidable here because step 1 refused to merge the two done flags.

Implemented by folding the bootstrap into the reward of the truncated step, r_t += γ·V(s_final), and then treating that step as an episode boundary for GAE — which it is, since the next observation comes from a fresh reset.

Cost, and a prediction confirmed

Step 1 predicted the environment would not be the bottleneck — 374 k steps/s standalone against 13 k inside training. Confirmed: the environment is about 3 % of the wall clock. The vectorised rewrite was never needed and was never written.

Both controllers score 500.0/500. Step 2 already showed return saturates and cannot rank anything competent, so the comparison runs on the basin instead — the same bisection and the same widened initial-condition sweep, same plant, same termination test, LQR against all sixteen trained checkpoints.

Two things change from steps 1–5 at once — the physics engine and the action space — so both are pinned down separately before anything is concluded.

The plant is identical, verified to 1.4e−14

cartpole_mj.xml sets the inertias by hand rather than letting MuJoCo derive them from a geometry primitive. A capsule adds hemispherical caps and a cylinder adds a 3r²/12 term; the 4/3 in the numpy plant is exactly the uniform-thin-rod assumption. Deriving inertia from the geometry would have made the two plants nearly the same object, which is the worst possible state to be in.

Check V1 compares MuJoCo's qacc against the numpy accelerations over 5000 random states: max absolute difference 1.42e−14. The two simulators describe the same rigid body. Anything that differs from here on is the integrator or the action space, and cannot be blamed on the model.

The Gaussian policy, and a bias that is reported rather than hidden

Only the head changes: Categorical(logits) becomes Normal(μ(s), exp(log σ)) with log σ a free parameter, log-probability summed over action dimensions, and Gaussian differential entropy. The entropy coefficient defaults to 0.0 here against 0.01 in the discrete version, because Gaussian differential entropy is unbounded below and can go negative, so an entropy bonus pushes σ up with no floor — the same coefficient does not mean the same thing in the two files.

The search table makes the case for why initialising σ matters on a bounded action space. At init_log_std = 0, σ = 1 on a ±1 action range means 37 % of sampled actions are outside the actuator range — the policy spends most of its probability mass on actions the plant cannot execute, and it costs roughly 50 % more environment steps. The adopted setting, −1.0, ends at σ = 0.133 with 0.5 % saturation.

• No real exploration problem. Cart-pole rewards every timestep and a random policy already scores 22. The ablation ordering would very likely change on a sparse-reward task.
• The ablations are one-at-a-time. Interactions were not measured.
• One hyperparameter configuration was searched over eight candidates on four seeds. A different configuration could change the ablation magnitudes — though the clipping result is large enough that it is unlikely to flip.
• no_clip failing is measured at this learning rate and epoch count. Clipping matters because of the ten epochs; at one epoch it would matter much less.
• The basin comparison uses the greedy policy. The stochastic policy used during training has a different, smaller basin.
• Step 6 changes the action space, not the task. The plant, the reward and the horizon are the same, so it tests whether the conclusion is an artefact of the categorical head — not whether it holds anywhere else.