Contents
Figure 1: Train it in simulation a million times, then let it wobble in the real world once
Robots rarely learn graceful movement on the first try. They wobble, fall, fling objects into strange corners, and somehow discover every possible way to miss a target. PyBullet lets them make those mistakes inside a fast physics simulation instead of on an expensive real robot.
That alone makes PyBullet a great place to start with robot learning. You can load robot models, simulate gravity and collisions, control joints, collect camera images, and train reinforcement learning agents — all from Python. No lab full of broken motors required. Nice.
PyBullet provides a Python interface for the Bullet physics engine and supports robotics, machine learning, games, and visual effects. It uses a client-server-style API, and it can run with an interactive graphical interface or headlessly without a display.
Why Use PyBullet for Robot Learning?
Robot learning needs a safe place for agents to explore. In reinforcement learning, exploration means trying actions the robot has not mastered yet. In real hardware, that can mean dropped objects, overheated motors, damaged joints, and someone asking why the robot arm attempted a cartwheel.
PyBullet gives you a simulated world where failure costs almost nothing. You can run thousands of episodes, reset the world instantly, and change physical conditions without rebuilding hardware.
PyBullet helps with several common robot-learning tasks:
- Robot arm control for reaching, grasping, pushing, and placing.
- Legged locomotion for quadrupeds, humanoids, and walkers.
- Drone control for hovering, navigation, and multi-agent flight.
- Manipulation for moving blocks, opening drawers, and stacking objects.
- Vision-based learning with simulated RGB, depth, or segmentation camera data.
- Reinforcement learning experiments with Gymnasium-compatible environments.
Gymnasium lists PyBullet-based third-party environments for robot-arm manipulation and quadcopter dynamics, which shows how developers commonly combine PyBullet physics with RL-friendly environment APIs.
Install PyBullet and Start a Simulation
Install PyBullet with pip:
pip install pybullet
Then create a simple Python file and import the library:
import time
import pybullet as p
import pybullet_data
PyBullet offers two main connection modes:
| Mode | What it does | When to use it |
|---|---|---|
p.GUI |
Opens a 3D window so you can watch the simulation | Building and debugging scenes |
p.DIRECT |
Runs without a visible window | Headless training on servers |
The PyBullet quick-start documentation describes GUI and DIRECT as built-in physics-server modes. Both run the simulation in the same process, but DIRECT skips graphics and returns results directly.
Start with GUI Mode
Use GUI mode while you build and debug your scene:
physics_client = p.connect(p.GUI)
p.setAdditionalSearchPath(
pybullet_data.getDataPath()
)
p.setGravity(0, 0, -9.81)
plane_id = p.loadURDF("plane.urdf")
robot_id = p.loadURDF(
"r2d2.urdf",
basePosition=[0, 0, 0.1]
)
for _ in range(10_000):
p.stepSimulation()
time.sleep(1 / 240)
p.disconnect()
This script connects to the simulator, loads a ground plane, loads an R2-D2 model, applies Earth-like gravity, and advances physics one simulation step at a time.
At 240 steps per second, the simulation runs smoothly enough for many robotics examples. You can change the time step later, but start with the default configuration while you learn the basics.
Switch to DIRECT for Training
Once your simulation works, swap this line:
p.connect(p.GUI)
for this:
p.connect(p.DIRECT)
That one change removes rendering overhead. For reinforcement learning, headless mode often makes a major difference because agents need many environment interactions.
Watching a robot fall looks entertaining for about 30 seconds. Watching it fall 10 million times through a GUI does not make training better.
Understand the PyBullet Simulation Loop
Every PyBullet project follows a version of the same workflow:
- Connect to the physics server.
- Configure gravity, time step, and search paths.
- Load the ground, robot, objects, and obstacles.
- Read robot state.
- Send motor-control commands.
- Advance the simulation with
stepSimulation(). - Calculate rewards or task results.
- Reset and repeat.
The physics engine handles contacts, gravity, collision response, joint constraints, and rigid-body dynamics. You decide the task logic around those physical events.
Here is the basic pattern:
while True:
observation = get_observation()
action = choose_action(observation)
apply_action(action)
p.stepSimulation()
reward = calculate_reward()
if episode_finished():
reset_world()
You can write choose_action() as a hand-coded controller, a neural network, or a reinforcement learning policy.
Load Robot Models with URDF
PyBullet commonly loads robots through URDF files, short for Unified Robot Description Format. A URDF describes the robot's links, joints, masses, collision shapes, visual geometry, and limits.
You can load a URDF with:
robot_id = p.loadURDF(
"kuka_iiwa/model.urdf",
basePosition=[0, 0, 0],
useFixedBase=True,
)
The useFixedBase=True option anchors the robot base to the world. That setting makes sense for a table-mounted robot arm, while a legged robot usually needs a free base. For locomotion examples, the bipedal walker tutorial shows a free-base agent in action.
PyBullet supports URDF and MJCF model loading, which lets you import many common robot descriptions and simulated locomotion models — the same model formats covered in our MuJoCo physics simulation tutorial.
Inspect the Robot's Joints
A robot arm can contain several joints, and PyBullet identifies each one with an index.
num_joints = p.getNumJoints(robot_id)
for joint_index in range(num_joints):
joint_info = p.getJointInfo(
robot_id,
joint_index,
)
joint_name = joint_info[1].decode("utf-8")
joint_type = joint_info[2]
print(joint_index, joint_name, joint_type)
Run this once after loading a model. It helps you identify which joint indices control the parts you care about.
Do not guess joint indices. A wrong index can produce confusing behavior, such as moving a gripper when you intended to rotate a shoulder. The robot will still move, technically, but it may look like it has received deeply questionable instructions.
Control Robot Joints
PyBullet supports several control modes. For beginner robot learning projects, position control offers the easiest starting point.
| Control mode | Command | Best for |
|---|---|---|
| Position control | Target joint angle | Reaching and first manipulation tasks |
| Velocity control | Target joint speed | Regulating motion speed and timing |
| Torque control | Direct joint force | Maximum realism and learned dynamics |
Position Control
Position control tells a joint to move toward a target angle:
p.setJointMotorControl2(
bodyUniqueId=robot_id,
jointIndex=2,
controlMode=p.POSITION_CONTROL,
targetPosition=0.5,
force=200,
)
This command asks joint 2 to move toward 0.5 radians, with an upper force limit of 200.
You can command several joints at once:
joint_indices = [0, 1, 2, 3, 4, 5, 6]
target_positions = [
0.0,
0.3,
0.0,
-1.2,
0.0,
0.8,
0.0,
]
p.setJointMotorControlArray(
bodyUniqueId=robot_id,
jointIndices=joint_indices,
controlMode=p.POSITION_CONTROL,
targetPositions=target_positions,
forces=[200] * len(joint_indices),
)
Position control works well for reaching and initial manipulation tasks because it gives the policy a stable, intuitive action target.
Velocity Control
Velocity control asks a joint to move at a target speed:
p.setJointMotorControl2(
bodyUniqueId=robot_id,
jointIndex=2,
controlMode=p.VELOCITY_CONTROL,
targetVelocity=1.0,
force=100,
)
This mode can help when you want a controller to regulate motion speed. It also creates a more dynamic task because the agent must account for momentum and timing.
Torque Control
Torque control gives the agent lower-level motor authority:
p.setJointMotorControl2(
bodyUniqueId=robot_id,
jointIndex=2,
controlMode=p.TORQUE_CONTROL,
force=5.0,
)
Torque control offers more realism and more challenge. The agent must learn how applied forces create movement, balance, and contact behavior.
Start with position control if you are new to PyBullet. Move to torque control after you can build a stable environment. Throwing raw torques at a complex robot on day one usually creates a physics-based modern dance performance, not a trained policy.
Read Robot State
A reinforcement learning agent needs observations. For a robot arm, useful state information often includes:
- Joint positions.
- Joint velocities.
- End-effector position.
- Target-object position.
- Object velocity.
- Distance from the gripper to the object.
- Gripper state.
- Contact information.
Read a joint's position and velocity like this:
joint_state = p.getJointState(robot_id, 2)
joint_position = joint_state[0]
joint_velocity = joint_state[1]
joint_reaction_forces = joint_state[2]
joint_torque = joint_state[3]
Read an end-effector position through forward kinematics:
link_state = p.getLinkState(
robot_id,
linkIndex=6,
)
end_effector_position = link_state[0]
end_effector_orientation = link_state[1]
For a reaching task, you might build an observation vector like this:
observation = [
*joint_positions,
*joint_velocities,
*end_effector_position,
*target_position,
]
Keep the observation focused. Give the policy enough information to solve the task, but avoid noisy or irrelevant values that make learning harder.
Build a Robot Reaching Task
A reaching task gives you a clean first robotics environment. The goal: move the robot's end effector close to a randomly placed target.
Set Up the Scene
import pybullet as p
import pybullet_data
import numpy as np
p.connect(p.GUI)
p.setAdditionalSearchPath(
pybullet_data.getDataPath()
)
p.setGravity(0, 0, -9.81)
plane_id = p.loadURDF("plane.urdf")
robot_id = p.loadURDF(
"kuka_iiwa/model.urdf",
basePosition=[0, 0, 0],
useFixedBase=True,
)
target_position = np.array([
0.45,
0.10,
0.55,
])
target_id = p.loadURDF(
"sphere2.urdf",
basePosition=target_position,
globalScaling=0.05,
)
The robot arm starts at the origin. The target sphere sits inside a reachable workspace. In a full RL environment, you would randomize the target position at every episode reset.
Calculate a Distance-Based Reward
The reward should encourage the end effector to approach the target:
def compute_reward(
end_effector_position,
target_position,
):
distance = np.linalg.norm(
np.array(end_effector_position)
- np.array(target_position)
)
return -distance
This reward gives less-negative values as the end effector gets closer. You can add a success bonus:
if distance < 0.05:
reward += 10.0
A simple reward function works well at the beginning:
- Reward closeness to the target.
- Give a bonus for success.
- Penalize unsafe contacts or joint-limit violations if needed.
- End the episode after success or a fixed number of steps.
Reward the outcome you want, not a vague approximation of effort. If you reward arm motion alone, your agent may wave enthusiastically while ignoring the target. Cute, but not useful. The reward function design guide covers more of these traps.
Build a Gymnasium Wrapper
PyBullet controls the physics, but Gymnasium provides a clean interface for training RL agents. You can combine both by creating a custom environment — the custom Gymnasium environment tutorial explains the underlying contract.
Here is a compact structure:
import gymnasium as gym
from gymnasium import spaces
import numpy as np
import pybullet as p
import pybullet_data
class PyBulletReachEnv(gym.Env):
def __init__(self, render_mode=None):
super().__init__()
self.render_mode = render_mode
self.action_space = spaces.Box(
low=-1.0,
high=1.0,
shape=(7,),
dtype=np.float32,
)
self.observation_space = spaces.Box(
low=-np.inf,
high=np.inf,
shape=(20,),
dtype=np.float32,
)
connection_mode = (
p.GUI
if render_mode == "human"
else p.DIRECT
)
self.client_id = p.connect(connection_mode)
p.setAdditionalSearchPath(
pybullet_data.getDataPath(),
physicsClientId=self.client_id,
)
self.max_steps = 200
self.step_count = 0
def reset(self, seed=None, options=None):
super().reset(seed=seed)
p.resetSimulation(
physicsClientId=self.client_id,
)
p.setGravity(
0,
0,
-9.81,
physicsClientId=self.client_id,
)
p.loadURDF(
"plane.urdf",
physicsClientId=self.client_id,
)
self.robot_id = p.loadURDF(
"kuka_iiwa/model.urdf",
useFixedBase=True,
physicsClientId=self.client_id,
)
self.target_position = self.np_random.uniform(
low=[0.3, -0.3, 0.3],
high=[0.7, 0.3, 0.8],
)
self.target_id = p.loadURDF(
"sphere2.urdf",
basePosition=self.target_position,
globalScaling=0.05,
physicsClientId=self.client_id,
)
self.step_count = 0
return self._get_observation(), {}
def step(self, action):
target_positions = np.asarray(
action,
dtype=np.float32,
)
p.setJointMotorControlArray(
bodyUniqueId=self.robot_id,
jointIndices=list(range(7)),
controlMode=p.POSITION_CONTROL,
targetPositions=target_positions,
forces=[200] * 7,
physicsClientId=self.client_id,
)
for _ in range(8):
p.stepSimulation(
physicsClientId=self.client_id,
)
self.step_count += 1
observation = self._get_observation()
distance = observation[-1]
reward = -distance
terminated = distance < 0.05
truncated = self.step_count >= self.max_steps
if terminated:
reward += 10.0
info = {"distance_to_target": float(distance)}
return (
observation,
reward,
terminated,
truncated,
info,
)
def _get_observation(self):
joint_states = [
p.getJointState(
self.robot_id,
joint,
physicsClientId=self.client_id,
)
for joint in range(7)
]
joint_positions = [
state[0] for state in joint_states
]
joint_velocities = [
state[1] for state in joint_states
]
link_state = p.getLinkState(
self.robot_id,
6,
physicsClientId=self.client_id,
)
end_effector_position = np.array(link_state[0])
distance = np.linalg.norm(
end_effector_position
- self.target_position
)
return np.array(
[
*joint_positions,
*joint_velocities,
*end_effector_position,
*self.target_position,
distance,
],
dtype=np.float32,
)
def close(self):
p.disconnect(
physicsClientId=self.client_id,
)
This example gives the agent seven joint target positions as actions. It returns joint data, end-effector position, target position, and distance as observations.
For a production environment, also add joint-limit handling, self-collision checks, object-reset logic, camera setup, action scaling, and validation with Gymnasium's environment checker.
Train a Robot with PPO
For continuous robot actions, PPO offers a practical starting algorithm. PPO handles continuous Box action spaces and works well for many simulation-based control tasks.
A typical Stable-Baselines3 setup looks like this:
from stable_baselines3 import PPO
env = PyBulletReachEnv()
model = PPO(
policy="MlpPolicy",
env=env,
learning_rate=3e-4,
n_steps=2048,
batch_size=64,
gamma=0.99,
verbose=1,
)
model.learn(
total_timesteps=1_000_000,
)
PPO learns a policy that maps observations to continuous joint commands. It collects simulation rollouts, estimates which actions improved return, and updates the policy gradually. Our Stable-Baselines3 tutorial covers the training workflow in more detail.
You can also explore SAC for continuous control. SAC often uses environment interactions efficiently because it learns off-policy and reuses replay-buffer experiences. PPO usually offers an easier first baseline when you want stable, straightforward training behavior.
Add Cameras for Vision-Based Learning
PyBullet can render camera images with getCameraImage(). That feature lets you train a robot from visual observations instead of privileged state values.
A basic camera call looks like this:
view_matrix = p.computeViewMatrixFromYawPitchRoll(
cameraTargetPosition=[0.4, 0, 0.4],
distance=1.2,
yaw=45,
pitch=-30,
roll=0,
upAxisIndex=2,
)
projection_matrix = p.computeProjectionMatrixFOV(
fov=60,
aspect=1.0,
nearVal=0.1,
farVal=3.0,
)
width, height, rgba, depth, segmentation = (
p.getCameraImage(
width=128,
height=128,
viewMatrix=view_matrix,
projectionMatrix=projection_matrix,
)
)
PyBullet supports camera images, visual-shape information, and texture utilities through its rendering API.
For vision-based robot learning, you can use:
- RGB images for normal camera input.
- Depth maps for distance information.
- Segmentation masks for object identity.
- Frame stacks for motion-aware policies.
Start with low-dimensional state observations first. Visual RL needs more training data, more compute, and more patience. Save the camera pipeline for after your robot can reach a target using joint and position data.
Improve Simulation Realism
A simulation helps, but no simulator perfectly matches the real world. When you later transfer a policy to hardware, small gaps in friction, motor strength, latency, sensor noise, and object mass can break a policy that looked brilliant in simulation.
Use domain randomization to reduce that gap. Randomize selected properties at every reset:
- Object mass.
- Surface friction.
- Joint damping.
- Motor strength.
- Camera angle.
- Lighting.
- Sensor noise.
- Target position.
- Action delay.
For example:
p.changeDynamics(
object_id,
linkIndex=-1,
lateralFriction=float(
self.np_random.uniform(0.3, 1.2)
),
)
Randomization prevents the policy from memorizing one perfect simulated setup. It forces the agent to learn behavior that survives small changes.
Do not randomize everything immediately. Begin with a stable, deterministic environment. Then add variation once the basic task works. Our sim-to-real transfer guide details the full pipeline.
Common PyBullet Mistakes
The Robot Falls Through the Floor
Check that you loaded a plane, set gravity correctly, and advanced physics with p.stepSimulation(). Also inspect collision shapes in the robot and object URDF files.
The Robot Does Not Move
Confirm that you use the right joint indices and motor-control mode. Print joint information before you issue commands, and ensure your force limit is large enough to move the joint.
Training Never Improves
Inspect the reward first. Then check action ranges, target reachability, observation scaling, termination conditions, and episode length.
Most RL failures start with environment design rather than algorithm choice. A policy cannot solve a target that spawns outside the robot's workspace, no matter how fancy the neural network looks.
Training Runs Too Slowly
Use p.DIRECT for headless training, reduce rendering calls, simplify collision geometry where appropriate, and run vectorized environments if your RL framework supports them.
Simulation Works but Real Hardware Fails
Expect this problem. Add domain randomization, model realistic delays and noise, calibrate physical parameters, and begin real-robot testing with strict safety limits.
A virtual robot can recover from a bad torque command by resetting the simulation. A real robot may offer a more expensive interpretation of "learning opportunity."
Recommended Books
- Robotics: Modelling, Planning and Control by Bruno Siciliano, Lorenzo Sciavicco, Luigi Villani and Giuseppe Oriolo — the rigorous foundation for the kinematics, dynamics, and joint control this tutorial puts into code.
- Robotics, Vision and Control: Fundamental Algorithms in Python by Peter Corke — a hands-on companion with runnable Python for transforms, kinematics, and camera pipelines like the ones used here.
- Grokking Deep Reinforcement Learning by Miguel Morales — covers the reward design, exploration, and PPO/SAC algorithms your simulated robots learn with.
Want to watch your arm actually learn to reach? Grab the GPTAstra full course at https://cutt.ly/5yviN6qd — it builds the PyBullet-to-Gymnasium-to-PPO pipeline end to end.
Frequently Asked Questions
What is PyBullet?
PyBullet is the Python interface for the Bullet physics engine. It supports robotics, machine learning, games, and visual effects through a client-server-style API that runs either with an interactive GUI or headlessly in DIRECT mode.
Should I use GUI or DIRECT mode?
Use GUI mode while building and debugging your scene so you can watch the simulation. Switch to DIRECT mode for training: it removes rendering overhead, which matters because RL agents need millions of environment interactions.
Which joint control mode should a beginner use?
Start with position control: it gives the policy a stable, intuitive target angle. Move to velocity control when you want speed regulation, and to torque control only after your environment works, since raw torques demand much more from the policy.
How do I combine PyBullet with Gymnasium?
Subclass gymnasium.Env, connect to PyBullet in __init__, load the scene in reset(), apply actions and call stepSimulation() in step(), and return the five-value Gymnasium tuple. The full reaching environment in this tutorial shows the pattern.
PyBullet or MuJoCo for robot learning?
Both are excellent physics engines with strong RL track records. PyBullet is notably easy to install with pip and friendly for beginners; MuJoCo is widely used in research. Try either first, then compare if your task demands it.
How do I transfer a simulated policy to a real robot?
Use domain randomization: vary mass, friction, motor strength, delays, lighting, and sensor noise during training. Model realistic latency, calibrate physical parameters, and start hardware tests with strict safety limits.
Final Thoughts
PyBullet gives you a practical sandbox for robot learning. You can load URDF models, apply realistic physics, command joints, read sensor-like state, build Gymnasium environments, and train reinforcement learning policies without risking hardware on every experiment.
Start with a simple reaching task. Use GUI mode to inspect the scene, switch to DIRECT mode for training, give your agent clear observations and rewards, and test the environment before launching a million-step PPO run.
Once that foundation feels comfortable, move on to grasping, locomotion, camera-based control, multi-robot tasks, and sim-to-real transfer. The robot may still fall over at first — but in PyBullet, it can try again before you even finish your tea.