Humanoid Training · Part of The Humanoid Group

Learning to Walk

Reinforcement learning lets a robot work out how to walk through repeated attempts that are scored automatically, instead of following hand-tuned equations. This guide follows that process from reward design and simulated terrain through to the first steps on real hardware.

By Arjun Rao · Updated October 2026

Rewards, observations and actions

An RL locomotion controller is a neural network policy that reads the robot's state and sends joint commands at a high, fixed rate. Its observations usually include joint angles and velocities, body orientation from the inertial sensor, and the command it should follow, such as a target walking speed. The reward scores every moment of the attempt: tracking the commanded velocity earns credit, while falling, slipping feet, jerky joints and wasted energy cost it. Engineers spend much of their time shaping these terms, because a policy will exploit any loophole it finds. Give too little reward for lifting the feet, for example, and it may learn to shuffle along the floor instead of stepping.

Learning to fall where it costs nothing

An untrained policy flails and topples long before anything resembling a gait appears, which no lab could afford on a real humanoid. Tang and colleagues' 2025 survey in the Annual Review of Control, Robotics, and Autonomous Systems traces the difficulty of real-world RL to the complexity and cost of interacting with the physical world. Simulators sidestep this by resetting the robot instantly and generating terrain on demand, often through a curriculum that adds slopes, steps and rough ground as the policy improves. Siekmann and colleagues at Oregon State University showed how far terrain variety alone can go: adding randomised stair-like terrain to an existing set-up, with no change to the reward, let the Cassie biped climb real stairs using only its own body sensing.

Crossing over to real hardware

Moving a policy onto the real robot is usually the hardest step. Motors heat up and lose torque, gears have backlash, sensors report late and the floor rarely has the friction the simulator assumed. Teams prepare for this by randomising those properties during training, adding random shoves, and modelling the actuators as closely as they can. Google DeepMind's soccer study, published in Science Robotics by Haarnoja and colleagues, combined high-frequency control, targeted dynamics randomisation and perturbations during training to move a policy onto small real humanoids with no further tuning. Against a scripted baseline, the authors report that the learned controller walked 181% faster, turned 302% faster and took 63% less time to get up after a fall.

What to ask before relying on it

If you are choosing a humanoid whose walking was trained with RL, ask the maker practical questions and expect specific answers. Find out which surfaces, slopes and step heights the policy was tested on, and whether those tests ran on the real robot. Ask how it behaves when pushed, when a foot lands on something unexpected and when the battery runs low. Check whether the walking policy can be updated separately from the rest of the software, and how each new version is tested before release. Ask what happens when balance is lost, and whether the robot has a fallback such as a controlled crouch. Research teams should also confirm that they can send commands directly to the joints.

Sources and further reading

Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes — Annual Review of Control, Robotics, and Autonomous Systems.
Peer-reviewed 2025 survey of where deep reinforcement learning has worked on real robots and what still holds it back.
Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning — Google DeepMind (arXiv; published in Science Robotics).
Small humanoids learn to walk, turn, kick and get up in simulation, then play on real hardware with no extra tuning.
Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learning — Oregon State University (arXiv).
Adding stair-like terrain to simulated training let the Cassie biped climb real stairs using only its own body sensing.

Common questions

How long does it take to train a robot to walk with reinforcement learning?

Training time varies widely with the robot, the simulator and the computing hardware. In most projects the compute is not the slow part: shaping rewards, modelling actuators and testing on the real robot usually take far longer than the training run itself.

Is reinforcement learning better than model predictive control for walking?

Neither wins everywhere. Model-based methods such as model predictive control are easier to analyse and tune for known conditions, while learned policies can cope well with rough ground and disturbances. Many current robots blend both, so ask makers which approach their controller uses and why.

Learning From Demonstration — How humanoids learn skills from human demonstrations and teleoperation data.
Simulation & Sim-to-Real — Why humanoids train in simulation first, and what it takes to cross the gap to the real world.
Evaluation & Readiness — How to judge whether a humanoid is safe, reliable and ready for real work.
Preparing Your Team — The roles, skills and habits your organisation needs before humanoids arrive.
Humanoid Training Services — When to outsource robot data collection, teleoperation and evaluation, and what to ask a provider.
How to Train a Humanoid — A plain-language walkthrough of teaching a humanoid a new task, from demonstration to testing.
Teleoperation for Data — VR, leader-arm and exoskeleton rigs for recording humanoid demonstrations, and what makes a run worth keeping.
Learning to Walk — How reinforcement learning teaches humanoids to walk and balance in simulation before they step out for real.
Training Data and Privacy — Recording people at work or at home to train robots: consent, data protection law and careful data handling.
Choosing First Tasks — How to pick a humanoid's first skills, set pass marks and budget for the demonstrations each one needs.
Safety While Training — Zones, stops and supervision that keep people safe while a humanoid is being taught, tested and debugged.
Latest updates — Teaching robots now takes data operations, simulation and careful testing. These figures and dated updates show where training data comes from, how fast robots learn new tasks and how results are checked.
About us — What we cover, how we source every figure, and who publishes this site.

Visit The Humanoid Group

People also search for reinforcement learning locomotion, humanoid walking reinforcement learning, RL walking controller, legged robot reward design, sim-to-real locomotion and terrain curriculum.