New NVIDIA Cosmos 3 Open Models for Interactive World Simulation
2026-11-08 · Hugging Face
NVIDIA Releases Cosmos 3 Open Models for Interactive World Simulation
NVIDIA has added four open-weight checkpoints to its Cosmos 3 model family within the Cosmos world-model development platform. The models target interactive world simulation and multi-sensor, multi-view driving scenes. All four are post-trained examples of Cosmos 3 Nano, have 16B trainable parameters, and use the OpenMDW1.1 license, permitting commercial and non-commercial use under its terms. NVIDIA also points readers to downloadable checkpoints, a technical report, and demos.
The four checkpoints
Cosmos3-Nano-Transfer-Auto is a bidirectional model for offline synthetic driving-data generation. It generates synchronized video across up to 11 camera views, including fisheyes, and supports joint RGB–LiDAR generation. Per-camera World Scenario Map (WSM) controls specify scene structure and traffic motion, while text prompts guide appearance. Developers can vary weather, lighting, and time of day; optional initial RGB images anchor scene appearance. Joint LiDAR generation also uses aligned HD-map range-map controls. Transfer-Auto can serve as a foundation for adapting Cosmos to a user’s own sensor rig and driving domain: post-train on user data for camera layout, LiDAR configuration, and target environment, then use the adapted model directly for offline data generation or apply causal training and few-step distillation for fast streaming simulation—the path used to create Sim-Auto.
Cosmos3-Nano-Sim-Auto is the causally distilled streaming variant of Transfer-Auto. It generates successive small chunks of synchronized camera video and LiDAR in a few denoising steps, using generated history and updated driving controls to support fast streaming and long interactive rollouts. Users provide per-camera captions and synchronized WSM controls, optionally with initial RGB images; LiDAR generation requires HD-map range controls aligned with camera controls. Closed-loop interaction requires a simulation runtime and WSM renderer to update controls from driving inputs. For a new sensor rig or domain, NVIDIA recommends adapting Transfer-Auto first and then distilling it into Sim-Auto.
Cosmos3-Nano-Sim-Transfer turns simulator controls and a text prompt into photorealistic RGB video. The simulator computes motion, contacts, and physical state; Sim-Transfer uses spatial controls and text to render color observations that a policy can consume. Its loop is action → simulator update → spatial controls → generated RGB → next policy action. Spatial controls include depth (relative distance from the camera), segmentation (regions and object boundaries), edges (scene contours), and blur (coarse appearance and layout). Text guides appearance, while physics and ground-truth state remain with the simulator.
Cosmos3-Nano-Sim-Bimanual predicts video from robot or camera actions and simulates how an environment evolves in response to a bimanual robot’s actions. It supports interactive two-arm simulation for YAM and AgiBot embodiments, controlled by a policy or human-operated joysticks. Starting from an image, users move the robot or camera and generate the next observations; one control sequence can switch between viewpoint changes and manipulation. NVIDIA notes that a mixed-action example uses 59 control values, padded to 64 during training, and that its schema defines coordinate frames and related control fields.
Why the checkpoints matter
NVIDIA frames these checkpoints as complementary ways to build environments for robotics and autonomous driving. For robotics, environments must answer what actions produce across changing scenes and viewpoints: Sim-Bimanual predicts visual consequences from actions, while Sim-Transfer converts a physics simulator’s controls into generated observations. For autonomous driving, appearance and geometry must agree; synchronized camera and LiDAR generation connects visual observations with 3D sensing, and scene and traffic-signal controls let researchers vary road layouts and signal states and study how a driving policy responds. Streaming generation connects video prediction to ongoing interaction, giving world-model researchers a way to explore how actions and observations unfold together.
NVIDIA positions open weights and post-training recipes as a shared starting point for adapting models, comparing methods, and contributing improvements. It also states that hard problems remain: measuring where the models faithfully support policy learning and evaluation, understanding their limits, and improving where they fall short.