What Sim-to-Real Means
Practice in a simulator, prove it on real hardware — the reality gap is the part that decides if it works.
Sim-to-real means training, testing, or improving a robot in a simulated world, then moving that behavior to a real robot.
It sounds like a shortcut. It is not that simple. Simulation helps because real robots are slow, expensive, and easy to damage. A robot can fall thousands of times in simulation. A real humanoid may break a motor, damage a floor, or injure someone after one bad fall.
But simulation has a problem: it is not the real world.
Can a behavior learned or tested in simulation still work when the robot enters the messy physical world?
The reality gap
The reality gap is the difference between the simulated world and the real one.
- In simulation, the floor may be flat. In real life, it may flex, slope, or have dust on it.
- In simulation, a box may be a perfect shape. In real life, it may be bent, taped, wet, or half-open.
- In simulation, a joint may move exactly as commanded. In real life, the motor may lag, heat up, or wear down.
- In simulation, a camera may see clearly. In real life, it may see glare, motion blur, or strange reflections.
Sim-to-real is not “train in simulation, then you are done.” It is “train in simulation, then prove the behavior survives reality.”
How sim-to-real usually works
- 011. Build a simulated task
Robot body, joints, sensors, objects, floor, lighting, basic physics. The real world is too detailed to copy perfectly — teams must choose what matters.
- 022. Train or test the behavior
Often reinforcement learning. The robot tries actions, gets a score, and learns. Simulation lets it fall, fail, reset, and try again far more times than real life would allow.
- 033. Randomize the fake world
Domain randomization. Change floor friction, object weight, lighting, motor strength, sensor noise. Do not make one perfect fake world — make many imperfect ones.
- 044. Move to the real robot
Sometimes called zero-shot transfer. The behavior moves to physical hardware without extra real-robot training. Impressive when it works, but tied to that task and setup.
- 055. Measure what breaks
The first real-world test is often humbling. The failures show what the simulator missed. Sim-to-real is a loop, not a one-way trip.
Why sim-to-real matters for humanoid robots
Humanoids have many moving parts. They must balance. They may use both hands. They may walk while carrying weight. They may work near people. They may touch objects that move in unexpected ways. That creates many chances for small errors to grow into large failures.
If a wheeled robot makes a small navigation mistake, it may stop. If a humanoid places a foot wrong while carrying a bin, it may fall.
Walking has made strong progress. Manipulating messy everyday objects is still much harder.
What people often misunderstand
- Mistake 01
Simulation replaces the real world.
It reduces real-world testing — it does not remove it. The real world is still the final test. A simulated fall costs almost nothing; a real fall has a bill.
- Mistake 02
A better-looking simulator solves the problem.
Realism is not just visuals. The physics matter — friction, weight, contact, motor delay, battery voltage, sensor noise, cable drag, heat. Hidden parts matter more than pretty parts.
- Mistake 03
Domain randomization is magic.
Useful if it includes the right kinds of variation. Useless if it misses the real problem. Randomizing colors will not fix bad finger contact.
- Mistake 04
Zero-shot transfer means general intelligence.
It means the robot did not need extra real-world training for that transfer. It does not mean the robot understands the world like a person.
- Mistake 05
A demo is the same as deployment.
Deployment needs how often the skill works, how often a person steps in, what happens after hours of use, and whether it survives normal mess.
Real examples: what is proven, and what is claimed
- Domain randomization for vision (Tobin et al., 2017)
Randomized simulated images transferred to real robot grasping in clutter. Research result — proves the approach works for some perception, not general autonomy.
- Dynamics randomization (OpenAI, 2017)
Sim-to-real transfer for robotic object pushing by randomizing physics. Research result — supports randomized control transfer for a specific task.
- ANYmal locomotion (Hwangbo et al., 2019)
Neural policy trained in sim, transferred to real quadruped. Strong evidence for legged locomotion sim-to-real.
- OpenAI Rubik's Cube hand (2019)
Trained entirely in simulation with Automatic Domain Randomization. Solved the cube 60% of the time — 20% for hardest scrambles. Impressive milestone, narrow setup.
- Humanoid locomotion (Radosavovic, Humanoid-Gym, Berkeley Humanoid)
Zero-shot sim-to-real walking on real humanoids over outdoor terrain. Real progress — bounded to walking and balance.
- Figure 02 walking (2025)
Company claim of zero-shot transfer using domain randomization and high-frequency torque feedback. Fits the trend — not proof of long-running commercial autonomy.
What is still hard
- Contact — friction, softness, shape, pressure, speed all change the outcome.
- Soft objects — clothes, bags, cables, half-open boxes are still harder than rigid blocks.
- Sensors — real ones drift, blur, fail on shiny or transparent objects.
- Motors — delay, heat, wear, limits, battery effects do not behave like clean math.
- Long tasks — even high single-step success rates compound across a real shift.
- Overfitting to the simulator — a policy may exploit a physics bug that does not exist in reality.
- Honest metrics — how many real trials, what counted as success, what failed.
A billion bad simulated trials are still bad data. The point is data that transfers.
The simple test for any sim-to-real claim
If the simulator went away tomorrow, would the robot still do this on a real floor?
- What skill transferred?
- What changed between simulation and reality?
- How much real-world tuning was used?
- How many real trials were reported?
- What failed?
- Sim-to-real means learning or testing in simulation, then moving the result to a real robot.
- The main problem is the reality gap.
- Simulation helps robots fail safely and cheaply — it does not replace real-world testing.
- Domain randomization helps by training across many fake worlds, not one perfect fake world.
- Zero-shot transfer is useful but tied to a specific task and setup.
- Evidence is strongest in selected skills, especially legged locomotion.
- Humanoid manipulation in messy human spaces is still much harder.
- Simulation
- A computer-made version of a robot, task, or environment.
- Sim-to-real
- Moving a behavior, model, or test result from simulation to a real robot.
- Reality gap
- The difference between the simulated world and the real world.
- Domain randomization
- Training in many varied simulated worlds so the robot is less surprised by the real one.
- Dynamics randomization
- Randomizing physical properties — friction, mass, motor strength, delay, weight.
- Reinforcement learning
- A way for a system to learn by trying actions and getting rewards or penalties.
- Policy
- The part of the robot system that chooses what action to take next.
- Zero-shot transfer
- Moving a learned behavior to the real robot without extra training on that real robot first.
- Synthetic data
- Data made by a simulation rather than collected from the real world.
- Digital twin
- A simulated copy of a real machine, place, or process.
- System identification
- Measuring a real system so the simulator can be adjusted to match it better.
- Hardware-in-the-loop
- Testing where real hardware is connected to a simulated environment.
Sources and evidence notes
What this essay leans on
| Claim | Evidence | Strength | Note |
|---|---|---|---|
| Policies can overfit simulator errors; randomized dynamics can help. | OpenAI, dynamics randomization for object pushing, 2017. | Strong | Research result; specific task. |
| Domain randomization can transfer perception from sim to real. | Tobin et al., 2017. | Strong | Foundational paper. |
| Sim-to-real works well for legged locomotion. | Hwangbo et al. on ANYmal, 2019; Radosavovic et al. on Digit, Science Robotics 2024. | Strong | Real robot research evidence. |
| Dexterous manipulation is harder to transfer. | OpenAI Rubik's Cube, 2019 — 60% / 20% success rates. | Strong | Major milestone with reported limits. |
| Humanoid sim-to-real is an active research area. | Humanoid-Gym 2024; Berkeley Humanoid 2024. | Strong | Bounded to walking and balance. |
| Simulation is now part of the standard robotics stack. | NVIDIA Isaac Sim / Isaac Lab. | Medium | Toolchain, not proof of any specific robot skill. |
| Companies are using sim-to-real for humanoid walking. | Figure AI walking post, March 2025. | Medium | Company claim; not independent deployment proof. |