Waymo is introducing the Waymo World Model, a frontier generative model that drives its next generation of autonomous driving simulation. The system is built on top of Genie 3, Google DeepMind’s general-purpose world model, and adapts it to produce photorealistic, controllable, multi-sensor driving scenes at scale.
Waymo already reports nearly 200 million fully autonomous miles on public roads. Behind the scenes, the Driver trains and is evaluated on billions of additional miles in virtual worlds. The Waymo World Model is now the main engine generating those worlds, with the explicit goal of exposing the stack to rare, safety-critical ‘long-tail’ events that are almost impossible to see often enough in reality.
From Genie 3 to a driving-specific world model
Genie 3 is a general-purpose world model that turns text prompts into interactive environments you can navigate in real time at roughly 24 frames per second, typically at 720p resolution. It learns the dynamics of scenes directly from large video corpora and supports fluid control by user inputs.
Waymo uses Genie 3 as the backbone and post-trains it for the driving domain. The Waymo World Model keeps Genie 3’s ability to generate coherent 3D worlds, but aligns the outputs with Waymo’s sensor suite and operating constraints. It generates high-fidelity camera images and lidar point clouds that evolve consistently over time, matching how the Waymo Driver actually perceives the environment.
This is not just video rendering. The model produces multi-sensor, temporally consistent observations that downstream autonomous driving systems can consume under the same conditions as real-world logs.
Emergent multimodal world knowledge
Most AV simulators are trained only on on-road fleet data. That limits them to the weather, infrastructure, and traffic patterns a fleet actually encountered. Waymo instead leverages Genie 3’s pre-training on an extremely large and diverse set of videos to import broad ‘world knowledge’ into the simulator.
Waymo then applies specialized post-training to transfer this knowledge from 2D video into 3D lidar outputs tailored to its hardware. Cameras provide rich appearance and lighting. Lidar contributes precise geometry and depth. The Waymo World Model jointly generates these modalities, so a simulated scene comes with both RGB streams and realistic 4D point clouds.
Because of the diversity of the pre-training data, the model can synthesize conditions that Waymo’s fleet has not directly seen. The Waymo team shows examples such as light snow on the Golden Gate Bridge, tornadoes, flooded cul-de-sacs, tropical streets strangely covered in snow, and driving out of a roadway fire. It also handles unusual objects and edge cases like elephants, Texas longhorns, lions, pedestrians dressed as T-rexes, and car-sized tumbleweed.
The important point is that these behaviors are emergent. The model is not explicitly programmed with rules for elephants or tornado fluid dynamics. Instead, it reuses generic spatiotemporal structure learned from videos and adapts it to driving scenes.
Three axes of controllability
A key design goal is strong simulation controllability. The Waymo World Model exposes three main control mechanisms: driving action control, scene layout control, and language control.
Driving action control: The simulator responds to specific driving inputs, allowing ‘what if’ counterfactuals on top of recorded logs. Devs can ask whether the Waymo Driver could have driven more assertively instead of yielding in a past scene, and then simulate that alternative behavior. Because the model is fully generative, it maintains realism even when the simulated route diverges far from the original trajectory, where purely reconstructive methods like 3D Gaussian Splatting (3DGS) would suffer from missing viewpoints.
Scene layout control: The model can be conditioned on modified road geometry, traffic signal states, and other road users. Waymo can insert or reposition vehicles and pedestrians or apply mutations to road layouts to synthesize targeted interaction scenarios. This supports systematic stress testing of yielding, merging, and negotiation behaviors beyond what appears in raw logs.
Language control: Natural language prompts act as a flexible, high-level interface for editing time-of-day, weather, or even generating entirely synthetic scenes. The Waymo team demonstrates ‘World Mutation’ sequences where the same base city scene is rendered at dawn, morning, noon, afternoon, evening, and night, and then under cloudy, foggy, rainy, snowy, and sunny conditions.
This tri-axis control is close to a structured API: numeric driving actions, structural layout edits, and semantic text prompts all steer the same underlying world model.
Turning ordinary videos into multimodal simulations
The Waymo World Model can convert regular mobile or dashcam recordings into multimodal simulations that show how the Waymo Driver would perceive the same scene.
Waymo showcases examples from scenic drives in Norway, Arches National Park, and Death Valley. Given only the video, the model reconstructs a simulation with aligned camera images and lidar output. This creates scenarios with strong realism and factuality because the generated world is anchored to actual footage, while still being controllable via the three mechanisms above.
Practically, this means a large corpus of consumer-style video can be reused as structured simulation input without requiring lidar recordings in those locations.
Scalable inference and long rollouts
Long-horizon maneuvers such as threading a narrow lane with oncoming traffic or navigating dense neighborhoods require many simulation steps. Naive generative models suffer from quality drift and high compute cost over long rollouts.
Waymo team reports an efficient variant of the Waymo World Model that supports long sequences with a dramatic reduction in compute while maintaining realism. They show 4x-speed playback of extended scenes like freeway navigation around an in-lane stopper, busy neighborhood driving, climbing steep streets around motorcyclists, and handling SUV U-turns.
For training and regression testing, this reduces the hardware budget per scenario and makes large test suites more tractable.
Key Takeaways
Genie 3–based world model: Waymo World Model adapts Google DeepMind’s Genie 3 into a driving-specific world model that generates photorealistic, interactive, multi-sensor 3D environments for AV simulation.
Multi-sensor, 4D outputs aligned with the Waymo Driver: The simulator jointly produces temporally consistent camera imagery and lidar point clouds, aligned with Waymo’s real sensor stack, so downstream autonomy systems can consume simulation like real logs.
Emergent coverage of rare and long-tail scenarios: By leveraging large-scale video pre-training, the model can synthesize rare conditions and objects, such as snow on unusual roads, floods, fires, and animals like elephants or lions, that the fleet has never directly observed.
Tri-axis controllability for targeted stress testing: Driving action control, scene layout control, and language control let devs run counterfactuals, edit road geometry and traffic participants, and mutate time-of-day or weather via text prompts in the same generative environment.
Efficient long-horizon and video-anchored simulation: An optimized variant supports long rollouts at reduced compute cost, and the system can also convert ordinary dashcam or mobile videos into controllable multimodal simulations, expanding the pool of realistic scenarios.
Check out the Technical details. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Michal Sutter is a data science professional with a Master of Science in Data Science from the University of Padova. With a solid foundation in statistical analysis, machine learning, and data engineering, Michal excels at transforming complex datasets into actionable insights.
