Simulation for Physical AI: A Review

The main challenge in building physical AI systems is data availability. Unlike large language models (LLMs) and vision-language models (VLMs), which are trained on internet data, robots need to learn the consequences of interacting with the physical world (e.g., a cup slipping, a cable bending, a gripper contacting an object). Collecting such data in the real world is slow, expensive, and dangerous. Simulation solves this problem by allowing the generation of thousands of hours of robot experience using GPU parallelism. Today, simulation has become part of the model development pipeline: it is used for creating perceptual datasets, reinforcement learning (RL), collecting demonstrations, augmenting real data, benchmarking, and testing rare scenarios. Simulation engines differ in their support for RL, batch simulation, contact physics, photorealistic rendering, and sensor simulation. Popular engines include: MuJoCo (precise dynamics of articulated systems), MuJoCo Warp (GPU-accelerated version for RL), NVIDIA Isaac Sim (high-precision PhysX physics, RTX rendering, sensor simulation, support for OpenUSD), and Isaac Lab (a lightweight multi-backend environment for robot training with support for RL and imitation learning). Modern robot simulation frameworks use GPU-accelerated numerical physics. To ensure speed, stability, and scalability, the Newton engine has been developed—an open, GPU-accelerated, extensible, and differentiable physics engine from NVIDIA, Google DeepMind, and Disney Research. Newton integrates MuJoCo Warp as a key physics backend and provides multiple solvers (SolverMuJoCo, SolverFeatherstone, SolverSemiImplicit, SolverXPBD, SolverKamino, SolverVBD, SolverImplicitMPM, SolverStyle3D) for different types of tasks. In Isaac Lab, Newton is an active integration direction, focusing on high-performance experiments. Other engines (PyBullet, DART, ODE, Drake) remain niche. The authors note that simulation has become a key layer in the stack of modern embodied AI, and a significant portion of the infrastructure is developed in open source.