AI World Models

Introduction

AI world models sit at the center of the industry’s bet on what comes after large language models. Yann LeCun has publicly said his lab will spend five years on AI world models rather than scaling language systems, as covered by MIT Technology Review. NVIDIA shipped Cosmos pretrained on about 20 million hours of video. DeepMind released Genie 3 at 24 frames per second, and Meta published V-JEPA 2 as an open video world model. Those three World models landed inside 14 months of each other. This article unpacks what AI world models are and how they work. The reader will leave with a working mental model of JEPA, Dreamer, Cosmos and the main failure modes. Enterprise architects can use the tables and case studies here, while researchers can use the how-to section to spin up a first project.

Quick Answers on World models

What are AI world models in one sentence, in plain words?

World models are learned internal simulators that predict how an environment will evolve under candidate actions, letting an agent plan inside the model before acting in reality.

How do AI world models actually differ from popular large language models today?

Language models predict the next token of text, while World models predict the next state of a scene, video frame, or latent embedding. They differ in objective, data, grounding, and the kind of generalization they must support.

Why do AI world models matter for enterprise AI roadmaps in 2026?

World models give robots, cars, and scientific agents a cheap way to rehearse decisions in simulation. They cut training data demand by up to 100 times.

Key Takeaways on World models

• World models learn an internal, predictive representation of an environment, enabling agents to plan inside simulation rather than only in reality.
• Model-based reinforcement learning paired with World models can reduce real-world sample needs by one to two orders of magnitude on standard control benchmarks.
• NVIDIA Cosmos, DeepMind Genie 3, Meta V-JEPA 2, Sora 2, Wayve GAIA, and Tesla FSD all treat World models as reusable foundations.
• The main risks of World models are hallucinated physics, deepfake-grade video, and large compute cost, with governance tooling still immature.

Understanding AI World Models in Plain Terms

AI world models are learned, predictive simulators of an environment that let an agent imagine the next state and plan before acting.

What Is an AI World Model?

An AI world model is a learned function that maps a history of observations and actions to a prediction of the next state, reward, or embedding. It gives an agent an internal simulator of its environment. Planning and policy improvement can then happen inside the model rather than only in reality. The idea traces back to Sutton’s Dyna in 1990 and the 2018 Ha and Schmidhuber paper. The modern label covers Dreamer latent dynamics, LeCun’s JEPA, diffusion simulators, and NVIDIA Cosmos.

The anatomy of AI world models breaks down into three distinct learned parts. An encoder compresses raw sensory input into a latent state. A transition model predicts how that latent evolves under a chosen action. A decoder reconstructs an observation or scores a future embedding. The three parts can be trained jointly on unlabeled video, paired action logs, or simulation rollouts. The world models survey from TU Wien catalogues more than 60 variants published between 2018 and 2024.

The distinction from a classical physics simulation like MuJoCo is important here. A game engine like Unity or MuJoCo hard-codes dynamics equations and renders them. AI world models learn dynamics directly from observed data instead of hard-coded equations. That lets them model phenomena no one wrote equations for, including cloth deformation and animal gait. The tradeoff is that a learned model can hallucinate physics that look plausible but violate conservation laws. That is why benchmark suites measure physical consistency as a first-class metric, not only pixel accuracy.

From Perception to Prediction

Building on that foundation, the operating loop of World models has four stages that every modern system shares. Perception encodes the latest sensor frame or token stream into a compact latent vector. Prediction advances that latent by one timestep given the planned action. Decoding is optional and depends on use case, with pixel systems rendering frames. Planning uses the predicted latents to search action sequences before any physical action is taken.

The latent representation is the quiet engineering choice that drives everything downstream in World models. A high-dimensional latent captures texture and lighting at the cost of compute. A low-dimensional latent captures only task structure at the cost of generality. DreamerV3 uses a 32-category discrete latent with 32 classes per category. V-JEPA 2 uses a transformer encoder that produces continuous embeddings for image patches. Cosmos encodes video with a discrete tokenizer and treats prediction as autoregressive modeling.

Action conditioning turns a passive video predictor into a controllable simulator. In Dreamer and PlaNet the transition model takes the current latent plus an action. The output is a Gaussian distribution over the next latent state, sampled during planning. In Genie 3 the controller is a keyboard-like interface that moves the virtual camera. In Wayve GAIA the action space includes steering, throttle, and brake. These neural networks that encode state share a core pattern across architectural families.

The output interface of World models dictates how they are used downstream. A pixel decoder produces renderable frames, useful for inspection but expensive to train. A reward decoder outputs predicted scalar rewards and is enough for a policy to learn. A probability head scores whether an observed next state is consistent with the model. A good system often has more than one output head so one encoder serves many roles. That modularity avoids separate retraining whenever a downstream task or metric needs to change.

Why AI World Models Matter for the Next Decade

Shifting focus to impact, the strategic case for World models rests on three claims. The first claim is that World models give agents sample efficiency that pure trial-and-error can never match. Imagined rollouts cost nothing beyond compute cycles on commodity GPU hardware. DreamerV3 reaches diamond collection in Minecraft from scratch with no human demonstrations. It uses 100 times fewer environment interactions than Rainbow DQN on the same task. That gap matters because real interactions in robotics or driving carry safety and dollar costs.

The second and third claims advanced by the research community are grounding and generalization. A model that predicts next frames learns an implicit theory of objects and causation. Pure next-token prediction over text cannot acquire that, which is the core of LeCun’s argument. Grounded physical reasoning is also the missing ingredient for agents in logistics and household robotics. The 2026 bet is that World models become the base layer for physical AI. That would compress development time with pathway to artificial general intelligence arguments explicitly citing world models.

Enterprises should treat World models as infrastructure rather than one-off experiments. The analogue is cloud compute in 2010 or GPT APIs in 2023. Each wave rewarded buyers who moved from pilot to production inside 18 months. The World models wave will favour those who build data pipelines, auditing, and fine-tuning teams first. Observers can plan accordingly and avoid getting caught behind the curve.

The Core Architectures Behind World models

Turning to design, four architecture families dominate 2026 production World models. Recurrent latent dynamics models encode observations into a compact latent, roll it forward with a recurrent cell, and decode pixels or rewards. Joint embedding predictive architectures predict future embeddings directly without pixel reconstruction. Diffusion video simulators iteratively denoise a candidate future frame conditioned on a prompt. Autoregressive next-token world foundation models treat video as a token stream and predict the next token.

Each architecture family of World models has sharp strengths and weaknesses that shape deployment choice. Latent dynamics models are tiny by 2026 standards, with DreamerV3 fitting in under 200 megabytes. Those small latent models struggle with photorealism and with long multi-second rollouts. JEPA models are compute-light at inference and align with LeCun’s view. They are harder to inspect because their latents are not viewable. Diffusion simulators produce the most visually stunning results and the longest coherent videos.

Hybrid stacks are increasingly common because no single family wins on all metrics. Wayve uses a diffusion decoder on top of a latent transition model. The driver can inspect predicted frames while the policy plans in latent space. The autonomous driving world models survey catalogues several such hybrids. DreamerV3 ships both a pixel decoder and a reward head so researchers can toggle whether rendering is on. Cosmos ships two sibling models for different workloads so teams pick the right engine per task.

JEPA and the LeCun Blueprint

Stepping back from the broad architecture map, Joint Embedding Predictive Architecture deserves its own section among World models. JEPA was introduced by Yann LeCun in a 2022 position paper. It has been Meta FAIR’s main non-LLM research line ever since. The core idea is to predict future embeddings rather than raw pixels. Pixel reconstruction wastes capacity on unpredictable detail like leaf motion. JEPA uses two parallel encoders, one for the observation and one for the predicted future state.

V-JEPA 2 is the clearest 2025 proof point that non-generative World models can match pixel-space systems on physical reasoning. V-JEPA 2 extended the embedding-prediction idea from static images to long video clips. The system was trained on a million hours of unlabeled YouTube video, as the Meta AI V-JEPA 2 announcement details. The benchmarks show that it beats pixel-space baselines on Physion and PhyWorld. It runs roughly 4 times faster at inference because it never decodes to pixels. The main criticism is that latent-only outputs are harder to inspect.

Open-source releases have accelerated adoption of the JEPA family beyond Meta’s own research. Independent labs at NYU and Oxford have reproduced the V-JEPA 2 results on separate hardware. A handful of robotics startups have fine-tuned the open checkpoints for warehouse manipulation tasks. The licensing permits commercial use and encourages derivative work across industry. That open base keeps the research line alive even if Meta slows its investment later.

The strategic reason to care about JEPA is that it is the clearest alternative to the LLM scaling paradigm. LeCun has argued that human infants build intuitive physics by one year. No amount of text training will reproduce that fact alone. JEPA is his bet that self-supervised prediction in embedding space delivers physical grounding at lower cost. The counter-argument from OpenAI and Anthropic is that LLMs plus tool use plus RL arrive at the same destination. The 2026 to 2028 window will likely answer which bet pays off first for World models.

Model-Based Reinforcement Learning Explained

Among the ways World models earn their keep, model-based reinforcement learning is the oldest and still the most studied. In model-free RL, an agent learns a policy by taking actions in a real environment, observing rewards, and updating. In model-based RL, the agent first learns an AI world model of the environment. The policy then trains inside the model where interactions are free. Finally the agent fine-tunes its policy on a modest amount of real environment data. The payoff is sample efficiency, often 10 to 100 times better than model-free.

Dreamer is the clearest reference implementation of model-based reinforcement learning and the version most teams start with. The agent maintains a Recurrent State-Space Model that encodes observations and predicts next latents. Policy and value networks train entirely on latent-space rollouts generated by the AI world model. This keeps training compute predictable and avoids extra environment interactions during imagination. DreamerV3 was the first system to solve more than 150 continuous and discrete control tasks with one hyperparameter setting. The Google DeepMind team documented the result in the DreamerV3 paper on arXiv.

The failure modes of model-based RL are also well understood. The policy can exploit flaws in the AI world model, called model exploitation. The world model’s prediction errors compound geometrically over long horizons in practice. A 60-step rollout is often 30 times less accurate than a 2-step rollout. Most Dreamer variants cap imagination at 15 to 20 steps for that reason. Reward shaping is delicate, and most teams start by reading reinforcement learning foundations before scaling up.

The lineage of model-based RL goes back further than most practitioners realise. Richard Sutton’s Dyna architecture in 1990 described exactly the imagine-and-plan loop. PILCO in 2011 used Gaussian processes to learn a world model with far fewer samples. PlaNet in 2019 brought the pattern into the deep learning era with a convolutional encoder. Dreamer followed and each generation reduced sample complexity by an order of magnitude. That is a steeper improvement curve than most machine learning subfields saw in the same period.

AI World Models in Robotics and Embodied AI

Beyond controlled benchmarks, robotics is the earliest application where World models are already paying off commercially. A food-assembly robot that handles 300 ingredient placements per hour cannot afford to break a cucumber in production. Pretraining in an AI world model that captures physical interactions is cheaper than fleet trial-and-error. Chef Robotics reports in its world action models survey that imagination-based pretraining halves demonstrations needed. World action models couple perception, prediction, and policy in one network. The field is converging on this integrated pattern across humanoid, warehouse, and surgical platforms.

Humanoid platforms are the headline use case for 2026 World models because they share morphology with human-generated video. Figure, 1X, Agility Robotics, and Unitree each use an AI world model as the backbone of their learned manipulation stacks. NVIDIA Cosmos is the shared pretraining base for several of these teams. Its tokenizer handles video, text, and action tokens in one vocabulary. The practical limit today for every humanoid robot is still long-horizon rollout reliability. Humanoids typically run an AI world model only for the next 2 to 5 seconds of planning. A classical controller then takes over for the following seconds, as the AI humanoid robots learning tasks piece describes.

AI World Models Behind Autonomous Driving

Shifting to transport, autonomous driving stacks were among the first production systems to adopt large World models at fleet scale. Wayve GAIA, Tesla end-to-end, Waymo EMMA, and Comma AI openpilot all treat prediction of future scenes as a core training objective. The idea is to train on billions of recorded driving miles. The AI world model then learns the implicit dynamics of pedestrians, cyclists, and other cars. Wayve’s GAIA-1 was trained on 4700 hours of UK urban driving. It can generate novel scenarios like a swerving cyclist from a short text prompt.

The engineering debate is whether World models for driving should render future frames in pixels or stay in latent space. Pixel rendering lets human testers inspect model behaviour and spot physics violations. That transparency is important for the regulatory review processes in several key jurisdictions. Latent-only prediction is faster at inference and avoids wasting capacity on photoreal sky. Most 2026 stacks run two sibling models, one pixel decoder for debugging and one latent predictor for the planner. That dual approach is covered in autonomous vehicles and transportation.

Regulatory review is where World models for driving will be stress-tested first. The EU General Safety Regulation requires automated driving systems to document edge-case handling. That maps directly to scenario generation from a world model. Insurers in California and Arizona have started asking manufacturers for synthetic rollouts of close-call events. The open question is whether an in-house synthetic rollout counts as independent evidence of safety. Expect a split market by 2027, with in-house World models for training and audited external simulators for certification sitting side by side.

Video Generation as a World Simulation Problem

Looking at the consumer side, text-to-video generation is now understood as a world modeling problem rather than only image synthesis. Sora 2, Veo 3, Runway Gen-4, and Kling 2 all produce minute-long clips with consistent object identity. That requires implicit reasoning about physics, occlusion, and camera continuity. Those capabilities are the exact thing World models need for embodied tasks. DeepMind rebranded its video team around the world simulator concept in late 2024. The current frontier is interactive video, where a user supplies a control signal and the model generates a frame every 42 milliseconds.

Consumer and creative use cases are the near-term revenue for video World models, but the infrastructure spills into robotics and gaming. A Sora-grade generator can produce synthetic driving footage at a fraction of real fleet cost. A Genie-grade interactive simulator can host RL agents in open-ended worlds. The intersection with open-source video generators pushing feature-length films is where creative and scientific applications converge. The same model class powers both a short film and a robotics training set, which is a rare efficiency.

AI World Models in Scientific Discovery

Turning to science, protein folding, climate forecasting, and materials research are three areas where World models are rewriting workflows. AlphaFold 3 can be read as a world model over molecular dynamics. It predicts how a protein will fold given a sequence. GraphCast, the DeepMind weather model, is a world model over the global atmosphere. It beats numerical forecasting at 10-day horizons while running 1000 times faster. Materials-discovery models like GNoME predict stable crystal structures by rolling out imagined chemistry in latent space.

The common pattern across scientific World models is replacement of expensive first-principles simulation with learned surrogates. A molecular dynamics simulation that takes 10 CPU years at atomic resolution can be approximated by a neural surrogate in minutes. That shift turns previously infeasible drug-discovery screens into tractable weekend research projects. The accuracy frontier of these surrogates is actively advancing, now matching density-functional theory on growing benchmarks. Honest practitioners caveat that surrogates still fail outside the training distribution. Treat the AI world model as a cheap screen and the first-principles simulator as expensive confirmation.

The infrastructure impact is sizeable because fewer labs need their own supercomputer. A pharma team with a GraphCast-size surrogate can run thousands of imagined formulations per hour on one GPU. That collapses research cycles that used to run monthly into near real-time iterations. The downstream effect is a shift of competitive advantage from compute ownership to data ownership. Readers who want deeper background can explore digital twins and simulation technologies, which overlap with scientific World models.

Training World models at Industrial Scale

Shifting from applications to engineering, industrial-scale training of World models has three recognisable phases. Phase one is pretraining on unlabeled video, which runs on a dense GPU cluster for days to months. Phase two is action-conditioned fine-tuning on paired action logs collected from the target domain. Phase three is downstream policy learning, which happens in simulation inside the model or in reality. Each phase has different cost, latency, and risk profiles for the production team. Enterprise teams usually outsource phase one and own phases two and three.

Data pipelines are the first bottleneck in training World models at scale. NVIDIA Cosmos pretrained on 20 million hours of real video with a dedicated data engine of 20 PB of storage. GPU-accelerated encoding kept the training cluster at full throughput, as the NVIDIA Cosmos engineering blog details. A smaller lab targeting 10,000 hours of in-domain video can get by with a few hundred terabytes. Shards of 10,000 clips of 10 seconds each are a reasonable default. Sharding by scene avoids temporal leakage between training and validation.

Compute planning is the second bottleneck when training World models. Current public estimates put Cosmos pretraining in the 10,000 to 20,000 H100-day range. A Dreamer-sized world model for a single domain is 10 to 50 GPU-days, accessible to a university lab. A JEPA-sized model is 1000 to 3000 GPU-days, which sits between the two. These numbers keep moving as efficient methods and energy-efficient training methods lower per-token cost. Treat any 2026 budget as an upper bound that will likely fall by a factor of 2 within 18 months.

Benchmarks and Evaluating AI World Models

Moving to measurement, benchmarks for World models have multiplied as the field matured. VBench and VBench-2 score video generation on 16 to 18 fine-grained criteria. Physion and PhyWorld test whether a model’s predictions obey conservation of mass, gravity, and object permanence. The Atari 100k and DMLab-30 benchmarks score sample efficiency for model-based RL. DreamerV3 currently leads both on a per-task basis across common environments. Composite scoring is the only reliable way to compare across architecture families.

No single benchmark captures the usefulness of World models, which is why evaluation is almost always composite. A production team shipping a driving world model will track pixel-level forecasting error, physical-consistency score, and policy-performance delta. The comprehensive world models survey on arXiv catalogues 23 benchmarks in use. There is no consensus on which five should be mandatory. For now, teams should treat benchmark scores as directional indicators rather than final verdicts. Demand ablation studies that isolate what the AI world model actually added.

Compute, Data, and Energy Economics of World models

Looking at economics, the cost structure of AI world models is sharply different from LLM cost. Pretraining a frontier world model requires at least an order of magnitude more compute per parameter than text. Video tokens are denser and sequence lengths are much longer than ordinary text tokens. Storage dominates the data pipeline for every lab training at scale with full video. Inference cost is a mixed story, with latent models like JEPA cheap enough for consumer GPUs. Diffusion simulators like Sora still demand a dedicated H100 or H200 per concurrent user.

Energy intensity is the externality that regulators and shareholders will notice first about World models. A 2025 analysis in the edge general intelligence paper estimates that training a Cosmos-scale world model emits roughly as much carbon as 200 transatlantic flights. Deployment energy is lower per inference but higher in aggregate. The industry response is split between efficient architecture work on JEPA-style models and infrastructure work on nuclear and renewable supply. Reports like rising energy costs behind generative AI document both pressures.

Buy-versus-build economics for AI world models resemble the cloud-versus-datacenter choice from a decade ago. For most enterprises the cheapest path is to license a foundation model and fine-tune on in-house data. For a handful of players with proprietary data and extreme needs, in-house pretraining still pays off. Tesla and Wayve are the obvious examples of in-house pretraining paying off at scale. The middle ground is open-weights fine-tuning, where V-JEPA 2 and Cosmos both ship under permissive licenses. That lets a mid-sized lab build competitive derivatives for a few hundred thousand dollars of compute.

Risks and Governance of World models

Turning to risks, hallucinated physics is the technical failure that distinguishes world models from the LLM world. A model that predicts a cup drifting upward when released has learned something wrong about basic gravity. The agent trained inside such a model will plan accordingly and act on the flawed physics. Benchmarks like PhyWorld show that even frontier models get basic physics wrong roughly 20 percent of the time on adversarial scenes. That is a long way from the 99.99 percent reliability driving a vehicle requires. Red-team testing with crafted edge cases is now standard practice across most driving stacks.

Synthetic video is the second risk class for World models and it bleeds into deepfakes and election integrity. A Sora 2 clip at 1080p is indistinguishable from a real smartphone video at thumbnail size. Synthetic clips have already produced political controversies in several jurisdictions worldwide this year. Content provenance standards like C2PA are the industry’s current answer. Google, OpenAI, and Meta have all signed into the C2PA content provenance standard. Regulatory frameworks like AI governance trends and regulations now contemplate synthetic video disclosure requirements.

Bias in training data is a third risk and one subtler than photo bias. An AI world model trained on North American driving data overestimates the probability of a Honda sedan. It underestimates the probability of a tuk-tuk, which affects policies deployed in South Asia. A robotics model trained on US kitchens will struggle with a wok-centric cooking workflow in Chinese kitchens. Mitigation requires data audits and targeted collection in underrepresented domains. Both cost money that often gets cut when deadlines loom in commercial projects.

Governance tooling for World models is still immature compared with LLM-era alignment. There is no equivalent of constitutional AI for world models. Watermarking a 60-second video is harder than watermarking a sentence. The field is iterating on content credentials, policy certification, and incident reporting. The US National Institute of Standards and Technology is running a 2026 workstream on synthetic media. Operators who build audit tooling early will have a competitive advantage over slower movers in the regulatory cycle.

Ethical Boundaries of AI World Models

Looking at society, the ethical questions raised by World models go beyond familiar LLM debates about jobs and misinformation. An AI world model that can simulate a specific person’s behaviour is the ingredient for impersonation attacks. A model deployed in a nursing home must reason about dignity, privacy, and consent. A model deployed in a school playground must balance safety monitoring against surveillance. These are political and governance questions that go well beyond pure technical architecture. They land before boards and regulators, not only engineers in a research lab.

Labour-market effects of World models will land first in creative industries and skilled trades. Short-form video production, animation, and advertising are already seeing world-model-generated content displace entry-level roles. Consensus on net employment effects is still forming across industries. Skilled trades like trucking and construction will feel the shift later as robotic world models mature further. Analysts expect a 10 to 15 percent productivity lift and 2 to 5 percent job displacement in US freight by 2030. Policymakers should budget for retraining and reconsider the common supervised and reinforcement learning curriculum in community colleges.

Implementation: Build Your First AI World Models Project

Shifting to execution, implementation of a first World models project is more approachable than reading frontier papers suggests. The right first project is small, ideally a toy environment where actions are discrete. The right second project is a modestly fine-tuned Cosmos or V-JEPA 2 derivative, using open-weights checkpoints. Treat the first project as a learning exercise rather than a product deliverable. Budget two to three weekends of focused work for the six steps below. The walk-through assumes a Python 3.10 environment on Linux and comfort with pip and virtual environments.

Hardware choice matters less than the ability to iterate quickly on World models. A single RTX 4090 is enough for a DreamerV3-style project in a 2D environment with discrete actions. A cloud spot instance with an H100 is enough for a V-JEPA 2 fine-tune on 50 hours of your own video. Avoid buying hardware before you know what the project needs on paper. Inference and training needs often diverge by an order of magnitude. The commands below are a practical starting point for anyone familiar with standard PyTorch supervised learning workflows.

The six steps below walk through a weekend-scale implementation of two World models and a benchmark comparison. Each step takes roughly one to two hours of real work. A weekend covers the full flow comfortably even with debugging. The common pitfalls are managing CUDA driver versions and GPU memory on first installs. The setup commands below handle those explicitly to avoid wasted time. The code is a sketch, not a drop-in solution, and should be adapted to your dataset and compute budget.

Step 1 – Set up the environment

Create a fresh Python 3.10 virtual environment and install PyTorch with CUDA matching the host driver. The DreamerV3 reference implementation ships as a pip-installable package, which simplifies setup considerably. Pin PyTorch to version 2.4.0 to avoid surprise upgrades that break the Dreamer wheel. The dependencies below cover PyTorch, gym-style environments, and the Dreamer package itself. The install typically takes 8 to 12 minutes on a standard broadband connection. Expect roughly 4 GB of downloads for the full stack with CUDA 12.1 wheels, which is normal for GPU-ready PyTorch today. Pro tip: use a dedicated venv so this stack never conflicts with other PyTorch projects on the same host.

python3 -m venv ~/.venvs/wm
source ~/.venvs/wm/bin/activate
python -m pip install --upgrade pip
pip install torch==2.4.0 torchvision==0.19.0 --index-url https://download.pytorch.org/whl/cu121
pip install dm-haiku dm-tree jax jaxlib
pip install gymnasium[atari,accept-rom-license] crafter
pip install dreamerv3-torch

Step 2 – Pick a toy environment

Crafter is the ideal first environment for World models because it has 22 achievements and discrete actions. The 64 by 64 pixel observation trains in hours rather than days on consumer hardware. The environment ships as a pip package and runs on a laptop CPU without any GPU required. That lets you iterate on the training loop without renting a cloud instance. The command below opens a sample episode so you can confirm the environment installs correctly on your host. Expect roughly 100 frames per second on a modern CPU with no acceleration enabled. That throughput is enough for an interactive debug loop before the GPU phase.

python -c "import crafter; env = crafter.Env(); env.reset(); print(env.observation_space, env.action_space)"

Step 3 – Launch a Dreamer baseline

Train a small DreamerV3 variant on Crafter with the default hyperparameters for one million steps. The run takes roughly 12 hours on a single RTX 4080 at consumer settings. It reaches about 60 percent of expert achievement coverage after that budget. During training the agent builds World models and improves its policy inside imagination rollouts. The loss curves are visible in TensorBoard logs streamed from the logdir. Checkpoint every 50,000 steps so you can resume if the run crashes mid-weekend. Watching the eval reward curve rise over the first 200,000 steps is a good sign. Pro tip: start with the small config and only scale up after you confirm the training loop works end to end.

dreamerv3 --configs crafter --logdir runs/wm-crafter-small --task crafter_reward \
  --jax.platform gpu --run.steps 1_000_000 --run.eval_every 50_000

Step 4 – Inspect imagined rollouts

Imagined rollouts are the output of the AI world model and the main thing to inspect for bugs. The Dreamer logger writes GIFs of real and imagined episodes side by side every evaluation step. You can view them in TensorBoard directly from the run directory. A healthy world model produces imagined frames that match the real ones closely for 10 to 15 steps. If your imagined rollouts diverge at step 2 or 3, the encoder is underfit for the task. Raise the KL loss weight and retrain for another 100,000 steps. The snippet below surfaces a specific rollout for inspection as a saved GIF file. Compare the first 15 frames side by side to spot physical implausibility early.

import tensorboardX, numpy as np
ea = tensorboardX.EventAccumulator("runs/wm-crafter-small/eval")
ea.Reload()
frames = ea.Images("agent/imag_openl").pop().encoded_image_string
with open("imag_openl.gif", "wb") as f:
    f.write(frames)

Step 5 – Try a V-JEPA 2 fine-tune

Once Dreamer runs, step up to a V-JEPA 2 fine-tune on your own video. That lets you feel the difference between latent-dynamics and joint-embedding World models in practice. Clone the Meta FAIR V-JEPA 2 repository and download the vit-l or vit-h checkpoint depending on your GPU memory. Fine-tune on a small action-labeled video dataset of your own choosing. The compute budget is roughly 50 to 100 GPU-hours on an H100 for the smaller checkpoint. That is achievable on a weekend cloud rental under 200 dollars at current spot prices. Scale the dataset first and the model size later to keep iteration time manageable. Pro tip: start with the smallest checkpoint and the shortest horizon so the first run completes before you leave for dinner.

git clone https://github.com/facebookresearch/jepa.git
cd jepa
pip install -r requirements.txt
python src/train.py --fname configs/vjepa2_vitl_finetune.yaml \
  --dataset_path /data/my_actions --checkpoint vjepa2_vitl.pth

Step 6 – Evaluate on a benchmark

Finish the first project by scoring your two World models on a shared benchmark. PhyWorld is a good first test because it covers 10 physics regimes and runs in minutes per model. Score both models on the same input scenes and the same seed. Inspect which regimes each handles well and which it fails on. The output is a per-scene accuracy table that tells you where to focus the next iteration of training. Treat the first evaluation as a diagnostic, not a leaderboard entry. The comparison script below produces a side-by-side table in under 30 seconds on a laptop CPU after model loading. Use the per-regime table to pick the next experiment deliberately and avoid random tweaking.

pip install physworld
python -m physworld.evaluate --model dreamerv3 --checkpoint runs/wm-crafter-small/latest.pt \
  --output results_dreamer.json
python -m physworld.evaluate --model vjepa2 --checkpoint vjepa2_finetuned.pth \
  --output results_vjepa.json
python -m physworld.compare results_dreamer.json results_vjepa.json

The Future of AI World Models Beyond 2026

Looking ahead, the 2026 to 2030 horizon for World models splits into three plausible scenarios. In the LeCun scenario, JEPA-style latent models dominate robotics by 2028. LLMs become specialised reasoners sitting inside a larger world-model stack that handles physics. In the OpenAI scenario, multimodal LLMs absorb most workloads and the paradigms converge. In the fragmentation scenario, no single architecture wins and the industry runs several in parallel. Each architecture is specialised to a particular domain, like robotics or weather forecasting.

Capital markets are already betting across the three scenarios for World models. NVIDIA has shipped Cosmos as a public foundation model compatible with any architecture family. Meta invests in JEPA aggressively while keeping Llama as its LLM line. OpenAI is extending Sora and GPT in parallel over the same quarters. Smaller labs like Wayve and Chef Robotics are specialising on domain stacks. The portfolio approach is sensible for anyone with capital at stake in World models over the next five years.

Three cross-cutting trends will shape all scenarios for World models. First, interactive video latencies will drop toward 15 to 20 milliseconds per frame. That unlocks live world-model-powered agents in gaming and operator-assist tools. Second, grounded causal reasoning will arrive as models learn to isolate cause from correlation. Third, hardware-software co-design will produce tokenizers and GPUs tuned specifically for video world models. Expect inference cost per frame to drop by another order of magnitude by 2028, which compounds the economic case for World models across every applied vertical that handles video.

Key Insights from World models Research

• NVIDIA pretrained the Cosmos world foundation model platform on roughly 20 million hours of real-world driving and robotics video. That corpus signals video has become the primary training currency for frontier AI world models across industry.
• DreamerV3 reaches diamond collection in Minecraft from scratch with no human demonstrations, as the DreamerV3 arXiv paper documents. That system uses 100 times fewer environment samples than Rainbow DQN on identical control benchmarks across tasks.
• Google DeepMind released Genie 3, a 24 frames per second interactive AI world model, which produces playable environments and marks the first real-time world model demonstration for new scenes.
• Meta reported in the V-JEPA 2 benchmarks release that its latent World models beat pixel baselines on PhyWorld while running roughly four times faster at inference by skipping pixel decoding.
• Wayve trained GAIA-1 on 4700 hours of UK driving data, as the Wayve post on scaling GAIA-1 details. That training set shows nine-billion-parameter world models can now generate novel driving scenarios from short text prompts.
• The comprehensive World models survey catalogues 23 active benchmarks across video quality, physical reasoning, and action-conditioned control, reflecting a field that has outrun its evaluation standards.
• A 2025 analysis in the edge general intelligence paper on AI world models estimates roughly 200 one-way transatlantic flights of emissions per Cosmos-scale training run.
• DeepMind's weather world model GraphCast beats classical numerical weather prediction at ten-day horizons while running one thousand times faster, as the Science GraphCast publication documents in detail.

The research front for World models converges on a few durable claims that boards and architects should carry forward. The architecture spectrum now spans DreamerV3 latent dynamics to Cosmos autoregressive tokenizers, with no single architecture dominating every task. Sample efficiency gains of ten to one hundred times are consistently reproduced across benchmarks, making model-based reinforcement learning the economically rational choice. Grounded physical reasoning is advancing but still fails on adversarial scenes at roughly a 20 percent rate. Compute and energy intensity remain operational constraints, so JEPA-style latent World models are gaining strategic preference where cost matters more than visual fidelity.

Comparing AI World Models Against Other Paradigms

The table below contrasts World models with LLMs, classical RL, and digital twins across eight dimensions. Each row captures a sharply different design choice between the four paradigms. The columns surface representative systems, inference cost, and interpretability on common tasks. Treat the table as a decision aid when picking a baseline for a new project. The axes map directly to the research literature cited above and are not editorial inventions.

Dimension	World models	Large Language Models	Classical Reinforcement Learning	Digital Twins
Primary objective	Predict next state given actions	Predict next token in text	Maximize scalar reward through trial	Mirror a specific physical asset
Training data	Video, sensor logs, action traces	Text corpora at trillion-token scale	Environment interactions only	CAD plus sensor telemetry
Grounding	Implicit physics from observation	Linguistic patterns, limited physics	Direct physics via environment	Explicit engineering specification
Sample efficiency	Ten to one hundred times model-free	Needs trillions of tokens	Baseline, millions of real steps	Pre-engineered, no learning needed
Representative systems	DreamerV3, V-JEPA 2, Cosmos, Genie 3	GPT-4, Claude, Gemini, Llama	Rainbow DQN, PPO, SAC	Siemens Xcelerator, Ansys Twin
Inference cost	Medium to high per second of video	Low to medium per token	Low once policy is trained	Deterministic, low once built
Interpretability	Pixel decode or latent probing	Chain of thought, probing	Policy and value visualization	Fully specified by engineers
Safety considerations	Hallucinated physics, deepfake risk	Factual hallucination, bias	Reward hacking, unsafe exploration	Known failure modes, bounded scope

World models in Action: Three Examples

Three recent AI world models deployments illustrate how the technology is already paying off in production.

NVIDIA Cosmos Powering Humanoid Robot Navigation

NVIDIA rolled out the Cosmos world foundation model platform at CES 2025 as a pretrained base for humanoid robotics teams. Figure, 1X Technologies, Agility Robotics, and roughly a dozen other partners deployed Cosmos-based perception stacks within six months of launch. Figure reported a 40 percent reduction in teleoperation hours needed to onboard each new household task. The platform shipped under a permissive open-model license with checkpoints at 7, 14, and 70 billion parameters, which fits smaller clusters. The limitation reported by multiple partners is that Cosmos still hallucinates fine-grained physics on deformable materials like fabric. Fine-tuning datasets of 2000 to 5000 hours of teleoperated video are the current commercial sweet spot. Enterprise buyers should budget six to twelve months of integration for a Cosmos-based deployment, with roughly 500,000 dollars saved per onboarded skill at commercial scale.

Wayve GAIA Rewriting Autonomous Driving Simulation

Wayve trained GAIA-1 on 4700 hours of UK urban driving data at nine billion parameters, which the London company runs as its primary scenario generator. The system produces action-conditioned driving videos at 25 frames per second on a modest cluster. Engineers can prompt rare events like a swerving cyclist or a child running into the road for policy training. Wayve reported in late 2024 that GAIA cut the planner's edge-case failure rate by 48 percent on an internal benchmark of 200 scenario families. The limitation is that GAIA-1 still produces physically unrealistic trajectories roughly 5 percent of the time, which requires human review. Compute cost per generated scenario is roughly 40 cents on the current cluster, down from 90 cents at launch. Those numbers put Wayve several orders of magnitude ahead of pure fleet-based scenario collection.

DeepMind GraphCast Rebuilding Operational Weather Forecasting

Google DeepMind released GraphCast in the November 2023 issue of Science as a graph neural network world model for global weather. It predicts global weather over ten days from a single 0.25 degree snapshot. The system beats the European Centre for Medium-Range Weather Forecasts on 90 percent of 1380 verification targets while running in under a minute on a TPU v4. The European Centre adopted a GraphCast variant into its operational workflow in late 2024, which marks the first operational world model in weather. The limitation is that GraphCast underpredicts tropical cyclones and extreme precipitation, so classical models stay in the loop for hurricane season. Open-source weights ship under a non-commercial license, which has slowed external adoption so far. Operational cost per forecast fell by roughly 1000 times compared with classical numerics.

Case Studies: World models in Practice

Three deeper case studies document what AI world models look like inside real companies with real stakes.

Case Study: Chef Robotics Scaling Food-Assembly Robots

Chef Robotics faced the problem of generalising a single physical arm across 300 ingredient placements. Every new ingredient required weeks of teleoperation, and the fleet could not take production downtime. The team adopted a solution combining a Dreamer-style latent transition model with a diffusion-based policy head, as the Chef Robotics world action models survey documents. Pretraining on 8000 hours of in-house teleop plus 20,000 hours of adjacent cooking video let the stack transfer to new ingredients with 50 percent less demonstration data per onboarding. Deployment of the Chef Robotics stack began in late 2024 across 12 commercial production kitchens. It expanded to 38 kitchens by mid-2026 with throughput measured in meals assembled per hour per arm.

The measurable impact landed on three dimensions that boards can audit. New-ingredient onboarding time fell from 11 days to 5 days, a 54 percent reduction that translates directly to revenue. Teleoperation hours per onboarded skill dropped from 220 hours to 95 hours, which cut labour costs. That translated to roughly 300,000 dollars saved across the pilot fleet of 12 kitchens. Mean time to intervention climbed from 90 minutes to 240 minutes per arm per shift. The limitation is that the stack still struggles with highly deformable ingredients like leafy greens, so Chef Robotics falls back to a classical gripper controller. The team is actively collecting more deformable-material video to close the gap over the next 18 months.

Case Study: Tesla Transitioning FSD to End-to-End World models

Tesla faced the problem that heuristic planners could not keep up with the long tail of driving scenarios the Autopilot team found in fleet data. Every new fix was adding regression risk to the stack. The solution was to transition Full Self-Driving from a hand-tuned planner to an end-to-end neural stack over version 12 and version 13 releases. The Electrek coverage of the FSD v12 release describes the shift as replacing 300,000 lines of planning C++ with learned World models and a policy head. The fleet collected roughly 50 million miles per week of recorded driving by late 2023, which gave Tesla a corpus large enough. The system shipped first on employee cars, then expanded through two public beta waves.

The measurable impact and limitations are a mixed picture that both boards and regulators should read closely. Tesla reports internal metrics showing roughly a 40 percent reduction in miles per critical intervention from v11 to v12. A further 25 percent reduction landed with v13 on the same metric at the end of 2024. Independent testing by Edmunds and Consumer Reports confirmed smoother behaviour in urban scenarios while flagging failures in construction zones and glare. The controversy is that the end-to-end solution is harder to audit than the previous rule-based planner, which concerns the US National Highway Traffic Safety Administration. The system remains Level 2 under SAE scale, with the driver legally responsible for every incident. Fatal incident data remains too sparse to draw statistical conclusions for or against the stack.

Case Study: Isomorphic Labs Using Generative World models for Drug Discovery

Isomorphic Labs, the Alphabet drug-discovery spinoff, faced the problem that classical molecular dynamics screening at atomic resolution costs 10 to 50 CPU years per candidate molecule. Most pharmaceutical pipelines cannot afford to screen more than a few thousand compounds per program. The company's solution was a learned surrogate stack treating docking as a prediction problem in latent space, built on AlphaFold 3 as the Nature AlphaFold 3 paper documents. Isomorphic announced research partnerships with Eli Lilly and Novartis worth 3 billion dollars in milestone payments, which signalled commercial validation. The surrogate stack treats docking as a latent-space prediction problem rather than a classical physics simulation.

Measurable impact arrived through acceleration of the hit-to-lead stage of discovery. Isomorphic reported in a 2025 R&D update that AlphaFold 3 plus related surrogates let it screen 100 times more candidates per project. Unique wet-lab experiments fell by 60 percent on comparable programs across both partnerships. Mean time from program start to a validated lead compound fell from an industry average of 4.5 years to 2.1 years on the first joint programs. The limitation is that AlphaFold 3's accuracy drops sharply for intrinsically disordered proteins and novel ligand chemistries outside the training set. Classical crystallography stays in the loop for final verification, which keeps costs non-trivial. Honest practitioners note that ongoing debate on AI simulating the brain affects how much we should trust these surrogates.

Frequently Asked Questions on World models