Quadraped robot: Stabilising a CPG Trot on Soft Ground
My undergraduate dissertation - Whether a residual PPO policy could improve the stability of a quadruped gait on soft ground
From walking to stable locomotion
At the beginning of the project, the problem appeared straightforward: construct a gait for Bittle, command a forward velocity, and evaluate whether the robot could walk successfully. But once I started examining the motion rather than just the displacement, I realised that “the robot moves forward” was not a particularly useful definition of successful locomotion. A controller can achieve reasonable forward velocity while the body gradually rotates away from its intended heading, roll and pitch oscillate between support transitions, touchdown introduces vertical bouncing, or a swing leg contacts the ground earlier than intended.
These effects became more noticeable when I moved away from ideal rigid-ground contact. On compliant ground, the phase commanded by the gait generator and the actual mechanical support phase are not necessarily identical. A controller may mathematically consider a leg to have entered stance while, in the physics simulation, the foot is still settling into the surface and the corresponding support force is developing gradually. A small timing difference at one touchdown may initially appear insignificant, but if one diagonal pair repeatedly develops support differently from the other, the error can accumulate across gait cycles and appear as heading drift, increased yaw rate, body tilt or inconsistent vertical motion.
If the robot already has a reasonable walking controller, what exactly should reinforcement learning be responsible for?
One option would have been to remove the handcrafted controller entirely and train an end-to-end policy that maps observations directly to joint targets. I deliberately did not take that route. The coordination pattern of a diagonal trot is already something I can describe explicitly: I know which legs should move together, how the two diagonal pairs should be phase-shifted, and approximately how swing and stance should progress. Asking a neural policy to rediscover all of this would reduce interpretability without necessarily addressing the part of the problem that I actually found difficult.
Instead, I treated the problem as a separation of responsibilities. The CPG defines the nominal gait structure, while the learned policy corrects deviations from that nominal behaviour. This also changed how I defined performance. Forward velocity remained important, but it was no longer sufficient on its own; I also cared about velocity-tracking error, heading drift, yaw motion, roll-pitch tilt, vertical bouncing and consistency of foot-ground interaction. Reward was useful during training, but I did not want the final argument for success to be simply that cumulative return increased. The controller needed to look physically better as well.
That reasoning eventually led to the central design of the project: first construct a structured and interpretable CPG gait that can walk on its own, and then allow PPO to learn only a bounded residual correction on top of it.
Why I started from a CPG
A Central Pattern Generator provides a compact way of representing rhythmic locomotion. For Bittle, I used a diagonal trot in which the left-front and right-back legs form one pair, while the right-front and left-back legs form the other. The two diagonal pairs are separated by half of a gait cycle.
For leg i, I represent its normalised phase conceptually as:
where f is the gait frequency and φi determines the phase relationship between the legs.
I chose a trot partly because the coordination is simple enough to reason about. For a project involving reinforcement learning, having a baseline that I could understand both visually and mathematically was important; I did not want the starting point to already be a black box. However, it became clear quite early that a simple sinusoidal CPG was not enough. Walking is periodic, but contact events are not merely smooth periodic curves: liftoff, forward swing, touchdown and loading all play different physical roles.
I therefore divided each gait cycle into several physically meaningful stages.
Swing Liftoff → Swing Transfer → Pre-Touchdown → Touchdown Buffer → Stance Sweep
The touchdown buffer was particularly important to me. Immediately switching from swing to full stance creates a fairly abrupt change in the role of the leg. On a purely mathematical gait this transition may appear harmless, but in simulation the resulting support dynamics can disturb the body. Rather than expecting PPO to repair an unnecessarily harsh baseline later, I preferred to make the nominal gait physically more reasonable first.
This became a recurring principle throughout the project: RL should compensate for uncertainty, not for problems that I could remove from the baseline controller myself.
Getting the simulation trustworthy enough to train
Before doing any reinforcement learning, there was a less glamorous but essential problem: making sure the simulated robot itself behaved correctly. The Bittle model was imported into Isaac Sim from an existing URDF-based asset, but the collision geometry needed to be corrected and checked before I trusted any contact-based result.
This mattered more than I initially expected. If a collider is wrong, a foot may appear to touch the ground visually while the physics engine reports something different. That error then propagates through the entire pipeline: contact timing becomes unreliable, scuff detection becomes unreliable, reward terms become unreliable, and eventually the RL policy learns from information that does not represent the motion I think I am observing.
It changed how I approached the project. It is easy to think of the RL algorithm as the difficult part, but in practice I found that getting the robot model, coordinate conventions, joint directions, contacts and logging correct was at least as important. A sophisticated optimiser cannot rescue a badly defined environment.
The baseline had to be good before RL was allowed to help
I spent a significant part of the project examining the CPG with the residual policy completely disabled. If the baseline already contained severe asymmetry, large heading drift or unrealistic joint trajectories, allowing PPO to correct everything would make the final controller much harder to interpret. I therefore wanted the zero-residual controller to produce a recognisable diagonal trot on its own.
This meant checking shoulder and knee trajectories, mirrored joint signs, stance progression and touchdown behaviour before training. The important question was not whether the baseline was perfect, but whether its remaining errors were genuinely the type of errors that justified feedback adaptation.
This gave me a clearer boundary between the two parts of the controller:
- CPG: gait structure and nominal motion.
- PPO: state-dependent correction when the physical motion departs from that nominal assumption.
Residual reinforcement learning
The central control structure can be written as:
Here, qCPG is the nominal joint target generated by the structured gait, ΔqPPO is the learned residual correction, and qfinal is the target ultimately passed to the robot after the relevant limits are applied.
I preferred this structure to an end-to-end locomotion policy mainly because of interpretability and control authority. If a final shoulder target changes, I can separately inspect the original CPG target and the PPO residual, which tells me what the robot would have done without learning and what the learned policy decided to modify.
At the same time, I did not want the policy to have unrestricted authority over complete joint trajectories and then rely purely on reward shaping to discourage undesirable behaviour. Instead, I deliberately reduced the learning problem. The policy receives body state, joint state, gait phase, contact information, velocity command and previous-action information. Its action consists of eight joint residuals together with three gait-level residuals: touchdown-buffer ratio, shoulder amplitude and knee-lift amplitude.
I liked this action structure because it allows two levels of adaptation. PPO can make a local correction to an individual joint, but it can also slightly reshape a physically meaningful component of the underlying gait. In other words, the policy can respond both to instantaneous state errors and to aspects of the nominal gait that may need to become slightly more or less aggressive.
Why the residual is deliberately restricted
Simply maximising reward is not enough. If PPO discovered that a violent shoulder correction improved velocity tracking for several steps, that behaviour might be numerically useful while being completely unsuitable for the physical robot. I therefore treated action limits as part of the controller design rather than an afterthought.
The raw PPO actions are scaled, clipped and rate-limited before reaching the robot. Final joint targets are also constrained by joint limits and a conservative workspace.
I thought about this as two different workspaces. One is the set of configurations that are geometrically possible, while the other is the smaller set of configurations that I would actually be comfortable asking the controller to use repeatedly.
Residual learning should operate inside the second one. These constraints are not a formal safety proof, but they prevent the policy from exploiting some of the most obvious unrealistic solutions and, more importantly, preserve the intended hierarchy of the controller: PPO remains a correction layer rather than becoming a second gait generator competing with the CPG.
Simulation and training setup
The full environment was implemented using NVIDIA Isaac Sim and Isaac Lab, with RSL-RL used for PPO training. The main experimental settings are summarised below.
| Parameter | Setting |
|---|---|
| Simulation platform | NVIDIA Isaac Sim + Isaac Lab |
| Robot | Corrected Petoi Bittle articulated model |
| Controller | CPG baseline + residual PPO |
| Task | Straight-line body-frame velocity tracking |
| Forward command | 0.05–0.30 m/s |
| Yaw command | 0 rad/s |
| Policy action | 8 joint residuals + 3 gait-parameter residuals |
| Training terrain | Compliant-contact ground |
| Ground stiffness | 3500 |
| Ground damping | 180 |
| Static / dynamic friction | 1.15 / 1.05 |
| Parallel environments | 512 |
| Physics timestep | 1/120 s |
| Control timestep | 1/30 s |
| Control decimation | 4 |
| Episode duration | 15 s |
| PPO iterations | 1500 |
| Actor / critic | MLP 256–256–128 |
| Activation | ELU |
| Learning rate | 1 × 10⁻³ |
| Discount factor γ | 0.99 |
| GAE λ | 0.95 |
| PPO clip parameter | 0.2 |
| Final evaluation | 16 environments, 32 episodes, 360 steps per episode |
One distinction here is important: I call the terrain compliant ground, not a true simulation of mud, sand or soil. Adjusting stiffness, damping and friction changes how support develops during contact and creates a less ideal interaction than rigid flat ground, but it does not model granular flow, permanent deformation, local compaction or realistic sinkage.
I deliberately kept this distinction because it would otherwise be easy to overstate the result. My goal at this stage was not to claim that I had solved locomotion on mud, but to create a controlled environment in which the nominal gait experiences contact uncertainty and then examine whether feedback-based residual correction improves the resulting locomotion.
Designing the reward: what should PPO actually care about?
Reward design required more thought than I initially expected. A naive locomotion objective could make forward velocity dominant, but that was not necessarily what I wanted. If the robot reaches the requested speed while continuously drifting in yaw or bouncing heavily at touchdown, I would not describe the controller as successful.
The reward therefore combines positive terms for forward tracking, survival and support continuity with penalties related to heading error, yaw rate, roll-pitch tilt, vertical bouncing, action magnitude, action rate, swing scuffing and diagonal-support imbalance. The exact weights are implementation details, but the conceptual choice was more important: I wanted the optimisation target to correspond reasonably well to what I would visually describe as a better gait.
That sounds obvious, but reward design and human judgement can diverge surprisingly easily. This is also why I later evaluated the trained policy using individual physical metrics instead of relying on cumulative return as the final evidence.
What does convergence mean in this project?
The word convergence needs some care in reinforcement learning. I cannot claim that PPO converged to a mathematically optimal locomotion policy. What I can observe is empirical convergence during training.
The return improves rapidly during the early stage of training and later becomes substantially flatter. However, I did not want to infer convergence from reward alone. Several physical quantities evolved in the same direction: yaw drift decreased, tilt RMS fell, vertical-bounce RMS was reduced, and swing scuffing became less frequent. Eventually these quantities entered comparatively stable regions rather than continuing to change dramatically between checkpoints.
That is what I mean here when I say that training converged: not convergence to a proven global optimum, but a stage in which continued PPO updates produced relatively small changes and the principal stability metrics had settled substantially below their initial values. There was still variation between later checkpoints, particularly in heading-related quantities, so I think of the end of training as a stable policy region rather than a single uniquely optimal iteration.
This also affected policy selection. I was more interested in a checkpoint whose behaviour was consistently reasonable across multiple physical metrics than in blindly choosing whichever checkpoint happened to have the largest scalar return.
Looking at the learned motion instead of just the reward
One of the questions I cared about most after training was: did PPO actually stabilise the existing gait, or did it quietly invent a different one?
A scalar reward cannot answer that, so I compared the baseline CPG targets with the final targets after PPO correction.
This was probably the result that made the residual formulation most convincing to me. The trained targets still follow the overall phase structure of the CPG and remain smooth and recognisably periodic rather than becoming high-frequency neural control signals. The largest corrections appear around swing, touchdown and stance transitions, which is physically reasonable because these are exactly the parts of the gait where small differences in contact timing or support development matter most.
PPO did not need to discover a new locomotion cycle. It mainly changed how the existing cycle behaved near the regions where the nominal assumptions were weakest, which I found much more interesting than simply observing a higher reward.
Contact timing told a similar story
I also compared representative contact timelines.
The alternating diagonal structure remains visible after training, meaning the learned policy does not fundamentally change which legs support the robot together. Instead, the differences appear mainly around contact transitions and overlap.
I am cautious about interpreting this too strongly because the contact signal comes from simulation and includes proxy assumptions. Nevertheless, together with the joint-target plots, it supports the interpretation that the residual policy modifies support-transition behaviour while preserving the underlying diagonal trot.
Final quantitative comparison
The final experiment compares the original CPG with zero residual against the selected trained checkpoint under the same compliant-ground setup.
The cumulative return increases from 240.6 to 390.8, corresponding to an improvement of approximately 62.4%. More importantly, the physical locomotion metrics improve at the same time. Velocity RMSE decreases from 0.1376 to 0.0506, a reduction of approximately 63.2%, while heading drift falls from 0.0244 to 0.0104, approximately 57.4% lower.
Yaw-rate magnitude decreases from 0.0199 to 0.0060, a reduction of approximately 69.8%, and tilt RMS decreases from 0.0162 to 0.0049, also approximately 69.8% lower. Vertical-bounce RMS improves more modestly, from 0.1198 to 0.1026, corresponding to approximately 14.4%.
I think the differences between these improvements are themselves informative. The controller did not improve every aspect of locomotion by the same amount: heading and attitude stability improved substantially, while vertical bouncing proved harder to eliminate. That is more believable than a result in which every metric improves uniformly, and it also suggests where the present control formulation still has room to improve.
What I actually learned from the project
Before doing this project, I tended to think about reinforcement-learning controllers and classical controllers as two competing approaches: either I design the controller, or the policy learns it. I no longer think that distinction is particularly useful. A more useful question is: which part of the control problem is already understood well enough that I should not ask a neural policy to relearn it?
For Bittle, diagonal gait coordination is not mysterious. I know which legs should move together, approximately how swing and stance should progress, and how to encode those relationships directly in a controller that I can inspect. What is harder to specify manually is the exact correction required when touchdown happens differently from expected, when the body begins to yaw, or when support develops asymmetrically. That is where learning becomes useful.
Residual RL therefore became less of a compromise between “traditional control” and “AI” and more of a decomposition of the problem: encode what I understand; learn what I do not model well.
The project also made me more cautious about evaluating RL systems. Watching a reward curve rise is satisfying, but it is not enough. I found joint trajectories, contact timelines, yaw behaviour and body motion much more informative because they forced me to ask what the policy was physically doing. Similarly, fixing colliders and checking contact semantics initially felt separate from the interesting control problem, but in reality they determine what the policy observes and what the reward means. The RL algorithm only optimises the world I give it; if that world is wrong, optimisation can be perfectly successful for the wrong problem.
Limitations
The most important limitation is that all of these results remain simulation-based. The compliant-contact environment provides a useful controlled disturbance to the nominal gait, but it should not be confused with realistic deformable-terrain mechanics. A real Bittle introduces additional uncertainties including servo backlash, actuator delay, limited torque, battery-voltage variation, structural compliance, sensing uncertainty and real impact dynamics.
Another limitation is the lack of a full ablation study. The complete policy can modify both individual joint targets and several gait parameters, and the final results show that this combined controller improves on the baseline. However, they do not establish how much of that improvement comes from each part. A useful future experiment would therefore compare CPG only, CPG with joint residuals, CPG with gait-parameter residuals, and the complete residual controller.