Contents
Here's the uncomfortable truth every simulation-trained RL project eventually confronts: a policy that performs beautifully in MuJoCo can fall over, misjudge distances, or outright fail the instant it touches real hardware. All that training — the reward shaping, the millions of timesteps, the carefully tuned PPO hyperparameters — can evaporate against the gap between "physically accurate simulation" and "actual physics."
This is genuinely the article that ties together threads from across this whole series — MuJoCo's contact modeling, the grasping arm's HER training, domain randomization mentioned in passing during the hardware-kits article. Sim-to-real transfer is the actual finish line for all of that work, and it deserves to be treated as its own discipline rather than an afterthought tacked onto "and then deploy it."
By the end of this guide, you'll understand exactly why the reality gap exists, the major techniques for closing it, and a practical checklist before you ever plug a trained policy into real hardware. IMO, this is the single most humbling and most important stage of any robotics RL project :)
What the "Reality Gap" Actually Is
Sim-to-real transfer means deploying a policy trained entirely (or mostly) in simulation onto a corresponding real-world system. The core obstacle is what's universally called the reality gap — the discrepancies between simulated and real dynamics that cause policy behavior to diverge, sometimes badly, between the two.
Physical dynamics mismatches: simulated friction, mass, and joint damping are approximations, never perfect replicas of your actual hardware's physical properties. Sensory input mismatches: simulated cameras and sensors lack the noise, latency, and imperfections real sensors introduce. Environmental variability: real-world lighting, surface textures, and unpredictable disturbances rarely match a simulation's cleaner, more controlled conditions.
Even with a perfect physics engine, abstractions and approximations inevitably creep in — this is one of the most pressing, actively researched challenges in applied robotics, not a solved problem you can shortcut around.
Our MuJoCo tutorial covers the physics simulation foundation where sim-to-real transfer begins — the same contact dynamics, friction modeling, and joint physics that need to approximate reality closely enough for policies to survive deployment.
Figure 1: Sim-to-real transfer bridges the reality gap — the discrepancy between simulated and real physics that causes policies to fail on hardware
Two Fundamentally Different Approaches
Sim-to-real transfer in the RL literature generally splits into two strategies, and understanding which one you're actually attempting matters for how you structure your whole project.
Zero-Shot Transfer
Here, you train entirely in simulation and deploy the resulting policy directly onto real hardware, with no additional real-world fine-tuning at all. This is the more ambitious goal — genuinely "train once, deploy anywhere" — and it's what most domain randomization techniques are specifically built to enable.
Domain Adaptation
Here, you train in simulation first, then fine-tune using a smaller amount of real-world data to close whatever gap remains. This is generally more forgiving than pure zero-shot transfer, since you're not asking simulation alone to anticipate every real-world quirk — you're giving the policy a final calibration pass against actual reality.
Zero-shot is the harder, more elegant target. Domain adaptation is the more pragmatic fallback when your simulation genuinely can't capture everything that matters for your specific hardware.
Domain Randomization: The Dominant Technique
If there's one idea that shows up across nearly every sim-to-real success story, it's domain randomization (DR) — deliberately varying simulation parameters during training so the resulting policy generalizes across a distribution of conditions, rather than overfitting to one exact simulated setup.
Dynamics randomization: varying physical parameters like mass, joint damping, and friction coefficients across training episodes, so the agent never gets to over-rely on one exact set of physical constants. Visual randomization: randomizing textures, lighting, and object appearance — this genuinely matters for vision-based policies, where a network trained on one lighting condition can fail completely under slightly different real-world lighting. Sensing and actuation perturbations: injecting noise into simulated sensor readings and actuator responses, so the policy doesn't assume the pristine, noise-free signals simulation naturally provides.
The logic here is genuinely elegant: if a policy performs well across a wide range of simulated physical parameters, reality — which is just one more point in that range — has a much better chance of falling within what the policy already handles competently.
Our curriculum learning guide covers the broader training methodology that domain randomization fits within — both techniques exist because random exploration alone can't find solutions in unstructured, hard-to-explore problem spaces.
Why This Works (And Its Real Limits)
Domain randomization is popular specifically because simulator data is cheap to sample — you're not paying any real-world cost to generate a huge diversity of training conditions. But it comes with a genuine design challenge: finding the right randomization ranges is difficult. Randomize too little, and you haven't actually covered reality's true variation. Randomize too much, and the policy might learn an overly conservative, needlessly robust-but-mediocre behavior that sacrifices performance for generality it didn't need.
DROPO: Letting Data Choose Your Randomization Ranges
Here's a genuinely clever fix for that "how much should I randomize" problem: instead of guessing randomization ranges by hand, estimate them from real data.
DROPO (Domain Randomization Off-Policy Optimization) uses a limited, precollected offline dataset of real robot trajectories to estimate which randomization distributions would actually best explain the observed real-world behavior. This explicitly models parameter uncertainty using a likelihood-based approach, rather than a human guessing plausible-sounding ranges. Demonstrated results include compensating for unmodeled phenomena — dynamics effects the simulator doesn't explicitly capture at all get partially absorbed into the estimated randomization distribution anyway.
This matters because hand-tuned randomization ranges are genuinely a weak point in the classic DR approach — you're relying on intuition about what "enough" variation looks like. Data-driven methods like DROPO replace that guesswork with something empirically grounded.
Tactile Sensing: Closing the Gap Beyond Vision and Proprioception
Most sim-to-real work historically focused on visual input and joint-position (proprioceptive) feedback. Tactile sensing is a genuinely underexplored piece of this puzzle, and it turns out to matter a lot for contact-rich manipulation tasks.
Researchers have modeled tactile sensor arrays directly in simulation, processing the response signals to account for the inconsistent sensitivity and discrepancy between simulated and real tactile signals. Adding tactile feedback to an RL policy — trained via zero-shot sim-to-real transfer with domain randomization — meaningfully improved grasping stability on a real door-opening manipulation task. This is a genuinely instructive example of the broader pattern: closing the reality gap isn't only about randomizing physics parameters — it's also about making sure your simulated sensory modalities actually correspond to what your real robot perceives.
If your manipulation task involves genuine contact-rich behavior (grasping, insertion, assembly), don't assume vision and proprioception alone will transfer cleanly. Tactile feedback might be the missing piece.
Our robot arm grasping tutorial covers the simulation training side of manipulation — understanding HER and sparse-reward handling makes the sim-to-real challenges here feel like the natural next step rather than a separate problem.
Beyond Domain Randomization: Other Techniques Worth Knowing
Domain randomization dominates the conversation, but it's not the only tool in this space.
Real-to-sim transfer — rather than randomizing simulation broadly, this approach works to make the simulation itself more accurately reflect the specific real-world system you're targeting, reducing the gap at its source rather than training around it. State and action abstractions — simplifying what the policy actually perceives and controls (recall the Fetch grasping task's compact 4-dimensional action space) can make transfer easier, since there's less surface area for sim-real mismatch to hide in. Sim-real cotraining — training jointly on both simulated and real data simultaneously, rather than treating simulation as a pure pretraining phase followed by a separate real-world fine-tuning step. Meta-learning approaches — training policies specifically to adapt quickly to new conditions, aiming for fast real-world calibration rather than assuming zero-shot transfer will work perfectly out of the box. Continual domain randomization — since a single randomization distribution risks catastrophic forgetting of skills relevant to conditions outside its current range, this approach adapts the randomization itself over time rather than fixing it once at the start of training.
Why You'd Even Bother With Simulation At All
Given how much effort sim-to-real transfer requires, it's worth being explicit about why anyone starts in simulation in the first place rather than just training directly on real hardware.
Training RL agents from scratch on real robots requires the physical robot for the entire training duration — genuinely impractical for most projects given hardware availability and cost. Early training is dangerous for hardware. An untrained policy makes mistakes constantly, and those mistakes can damage expensive equipment or require constant human supervision to reset failed episodes. Simulation removes both constraints entirely — training runs in parallel, at accelerated speed, with zero risk to physical hardware, which is exactly the appeal that makes sim-to-real transfer worth the added complexity.
This is genuinely the same tradeoff that motivated the "simulate first, buy hardware second" advice from our robotics simulation hardware guide — the sim-to-real gap is the price you pay for training somewhere safe and cheap.
For physical robot arms you'd deploy trained policies on, the MyCobot Pro 630 offers 6-DOF with ROS compatibility — the exact kind of hardware that benefits from simulation-first training followed by domain-randomized transfer.
A Practical Pre-Deployment Checklist
Before plugging a simulation-trained policy into real hardware, a few concrete steps genuinely reduce your risk of a failed (or damaging) first attempt.
Confirm your simulation models the physically relevant parameters — mass, friction, joint damping, actuator response — with reasonable accuracy for your specific hardware, not just generic defaults. Apply domain randomization across the parameters you're least confident about, rather than randomizing everything uniformly and hoping for the best. Test in simulation against out-of-distribution conditions before ever touching hardware — if your policy is fragile to reasonable simulated perturbations, it will likely be fragile to real-world variation too. Budget for a domain adaptation phase rather than assuming pure zero-shot transfer will work — a small amount of real-world fine-tuning is often the pragmatic difference between "close" and "actually works." Start deployment tests with safety limits in place — reduced speed, restricted range of motion, human oversight — treating the first real-world runs as validation, not confident production use.
Common Mistakes People Make
I've referenced pieces of this across other articles in this series, but worth naming these specifically as sim-to-real failure patterns.
Assuming a well-trained simulated policy will transfer with no additional work. The reality gap is a genuinely active research problem, not a minor technicality — expect some transfer effort regardless of how good your simulation training looked. Randomizing simulation parameters arbitrarily, without grounding ranges in anything real. Guessed randomization ranges can either undershoot real-world variation or introduce needless conservatism — data-driven approaches like DROPO exist specifically to fix this. Ignoring sensory modalities beyond vision and joint position. Contact-rich manipulation tasks specifically benefit from tactile sim-to-real work that pure visual/proprioceptive approaches miss entirely. Treating this as a robotics-only concern. The same reality gap principles are increasingly relevant to any RL system deployed from a training environment into a genuinely different production environment, not just physical robots. Deploying directly to full-speed, full-range real hardware on the first attempt. Always test with safety margins first — an imperfect transfer is far less costly to discover at reduced speed than at full operational parameters.
Wrapping This Up
Sim-to-real transfer is the genuine finish line for essentially every robotics RL project covered across this series — all that MuJoCo training, reward shaping, and HER-based grasping work exists in service of eventually running on real hardware, and the reality gap is the honest obstacle standing between simulated success and that goal. Domain randomization remains the dominant technique for closing that gap, with newer data-driven approaches like DROPO addressing its biggest weakness: guessing the right amount of randomization.
Remember that zero-shot transfer is the more ambitious target while domain adaptation with real-world fine-tuning is the more pragmatic fallback, and that closing the reality gap increasingly involves more than just visual and proprioceptive randomization — tactile sensing and dynamics-parameter estimation genuinely matter for contact-rich tasks. FYI, if you've followed this series from CartPole through MuJoCo through robot grasping, this article is genuinely the payoff stage all that groundwork was building toward :)
Now go back to whatever MuJoCo or grasping policy you trained earlier in this series and ask honestly: would this survive contact with real hardware, or does it only work because the simulation is being unrealistically kind to it?