World Action Models and the Shift Towards Predictive Robotics

Over the past few years, robotics has made meaningful progress on a problem that once looked unsolvable: training a single model to interpret a visual scene, understand a natural-language instruction, and turn that understanding into physical action. Vision-language-action models, or VLAs, are the clearest expression of this shift. Systems such as RT-2, OpenVLA, and Physical Intelligence’s π0 build on large vision-language models and extend them into the action space, allowing a robot to generalize across a wider range of objects, instructions, and environments than earlier task-specific systems. This is an important development, but it still leaves open a more basic question about physical intelligence. Knowing what action is appropriate in a given situation is not the same as understanding what that action is likely to cause.

An interesting example is a simple household task such as placing a mug in a sink. A capable VLA may correctly identify the mug, understand the instruction, and generate a sequence of motor commands that usually works. The difficulty is that physical environments are rarely as clean as the demonstrations used to train the policy. The mug may be partially blocked, the handle may face the wall, another object may be in the way, or the first grasp may slip. At that point, the robot needs more than semantic knowledge and a learned mapping from images to actions. It needs some representation of how the scene is likely to change as a result of what it does. This is the basic motivation behind the recent interest in world action models.

From world models to world action models

The term “world action model,” or WAM, is relatively new and the category is still fairly loose. A 2026 paper uses the term to describe models that combine predictive state modeling with action generation, although several closely related systems use different terminology. For now, it is probably more useful to focus on the underlying idea than on where exactly the category begins and ends.

A VLA is primarily trained to map an observation and an instruction to an action. A world model tries to represent how the environment evolves, often conditioned on a possible action. A world action model brings those two together, so that a prediction of the future informs the action the robot ultimately takes.

This distinction also helps clarify why a strong video model is not automatically a useful world model for robotics. A generative model can produce a visually reasonable sequence showing an object being picked up or moved without having a reliable model of which robot command would produce that outcome in the current scene. The generated motion may look right while getting the geometry, timing, or contact dynamics slightly wrong. In robotics, those differences are critical. A plausible video of a glass moving across a table is not the same as understanding how a particular gripper, at a particular pose, should interact with that glass without knocking it over.

Exhibit: Conceptual comparison of the input-output formulations of vision-language-action models, world action models, and standard world models, highlighting WAM’s capability to jointly predict actions and future observations.

The broader idea has a much longer history. Richard Sutton’s Dyna work in 1990 combined direct experience with planning using a learned model of the environment. Later research extended this idea to high-dimensional environments. World models showed that an agent could learn a compressed representation of visual dynamics and train a controller entirely inside that learned environment. PlaNet learned latent dynamics directly from images and planned within that latent state. Dreamer used imagined trajectories inside a learned world model to train its policy and value function, while MuZero showed that useful planning does not necessarily require reconstructing every detail of an observation. A model can be valuable even if it represents only the aspects of the environment that are relevant to the decision.

That last idea is particularly important for robotics. A robot does not need to predict the exact texture of a countertop or the reflection on the side of a stainless-steel sink. It may, however, need a good estimate of whether a mug will collide with another object, whether a grasp is likely to remain stable, or whether a partially completed action has moved the scene into a state that requires a different plan. The central design question is therefore not simply whether a robot should predict the future, but which representation of the future is sufficient to improve control.

What changed to make world action models possible

Recent robot foundation models emerged from a different research lineage. RT-1, RT-2, OpenVLA, π0, and related systems were designed primarily as generalist policies. They combine visual observations, language, and large-scale robot data to produce actions directly. RT-2 was particularly notable because it demonstrated that concepts learned from internet-scale vision-language pretraining could transfer into robotic behavior, while OpenVLA showed that a large open model could be trained on nearly a million robot demonstrations. These systems significantly improved the semantic side of robotic control by allowing policies to reuse representations learned far beyond the narrow set of objects and tasks present in an individual robot dataset.

What they do not automatically provide is an explicit model of physical consequences. A model may recognize an object as a mug and understand that mugs can be placed in sinks, yet still have only a weak representation of how a particular grasp will affect that mug. The distinction is subtle because a sufficiently capable policy may implicitly learn some dynamics as part of predicting good actions. There is no sharp boundary between a reactive policy with strong internal representations and a system with a dedicated world model. The current interest in WAMs is partly an attempt to make this predictive component more explicit and to use the enormous amount of available video as a source of information about how scenes evolve over time.

Several recent systems illustrate different ways of doing this. One family of approaches predicts an explicit future image or video and then converts that forecast into robot actions. This is conceptually attractive because the model’s prediction can be inspected directly. If a robot is trying to fold a towel, a generated intermediate image can reveal whether the model expects the towel to move toward the desired configuration. The downside is that generating pixels is computationally expensive and may force the system to model visual details that have little relevance to control. It can also create a misleading sense of correctness, because a video may look coherent while getting a small but physically important aspect of the interaction wrong.

Another family avoids rendering the future in pixel space and instead predicts a compressed representation. Meta’s V-JEPA 2 is a useful example. The model was pretrained on more than one million hours of internet video and then adapted into an action-conditioned world model using less than 62 hours of robot video from the DROID dataset. Rather than generating photorealistic future frames, it predicts how a learned representation of the scene should change. This is harder to inspect visually, but it follows the logic that a robot may only need to model the variables that matter for action. Depending on the task, those variables could include object motion, depth, geometry, contact, or some intermediate notion of task progress.

A third approach is to learn future prediction and action generation jointly. DreamZero, for example, starts with a pretrained video diffusion model and trains it to predict both future video and robot actions. NVIDIA’s Cosmos Policy similarly adapts a pretrained video model so that it can produce robot actions alongside representations of future states and expected reward. The appeal of these models is that prediction and control are learned together rather than being connected after the fact. The tradeoff is that they place more work inside the control loop, and the computational requirements of video-scale models are substantial.

Prediction and control may operate at different speeds

This systems constraint may prove just as important as the underlying model architecture. Robots need to react continuously to changing sensor inputs, while large predictive models can be relatively slow. DreamZero’s authors report closed-loop control at 7 Hz from a 14-billion-parameter model after substantial inference optimization. Physical Intelligence’s π0.7 takes a different approach by allowing a separate model to generate visual subgoals while the lower-level policy continues to execute actions asynchronously. Google DeepMind’s Gemini Robotics 2 family also separates higher-level embodied reasoning from lower-level VLA control, though Google does not describe the system as a world action model.

These examples suggest that prediction and control do not necessarily need to happen at the same frequency. A robot might update a high-level prediction or subgoal relatively infrequently while running a faster policy for local control. That structure is not especially net new; classical robotics has long separated planning from control because the two operate on different timescales. The new question is how much of each layer should be learned, and whether large pretrained predictive models can provide useful guidance without introducing too much latency or instability.

There is also a more fundamental problem. Prediction introduces another possible source of error. A direct policy can select the wrong action from the current observation, but a predictive system can first form an incorrect model of the future and then choose an action based on that mistaken forecast. The outputs of modern generative models can be persuasive even when they are wrong. A world model therefore has to be judged less by whether its predictions look convincing and more by whether those predictions are causally useful for control.

Learning from video only gets you so far

One reason world models are receiving so much attention is that they potentially change the data equation. Robot demonstrations are difficult and expensive to collect. They require real hardware, calibrated sensors, human operators, and a control interface specific to the robot. Video, by contrast, is abundant. The internet contains enormous amounts of footage showing people manipulating objects, using tools, opening containers, folding clothes, cooking food, and moving through physical environments. This makes video an attractive source of information about object motion and scene dynamics.

The limitation is that most video records what happened without recording the action that caused it in a form a robot can directly use. A human hand opening a drawer provides evidence about how the drawer moves, but it does not provide the motor command for a multi-finger gripper. Human and robot embodiments differ in kinematics, sensing, contact surfaces, and other factors. A model may therefore learn a useful prior about physical change from video while still requiring robot-specific data to connect that prior to executable actions. V-JEPA 2 and other recent systems provide early examples of this strategy, but the extent to which broad video pretraining can reduce the need for expensive robot data remains an open question.

This embodiment gap is likely to remain one of the central problems in the field. A video model can learn that pulling a handle opens a drawer, but successful execution depends on where the robot grasps, how the wrist is oriented, how much force is applied, and what the robot does if the drawer sticks halfway. Those details are specific to the body performing the task. World models may help robots learn more from observation, but they do not eliminate the difference between watching an action and being physically capable of performing it.

An architecture that is still taking shape

It is still too early to know whether “world action model” will become a durable category or simply a temporary thought experiment for a broader convergence between world modeling and robot policies. The research landscape currently includes VLAs, models that predict visual subgoals, latent world models, joint video-action models, and hierarchical systems that separate reasoning from control. There is no clear evidence yet that one of these approaches dominates across tasks. In some settings, a strong reactive policy may be sufficient. In others, explicit prediction may become much more important, particularly when the robot needs to handle unfamiliar states, reason over longer horizons, or recover after its original plan fails.

The more consequential change may be conceptual. The first generation of robot foundation models focused heavily on connecting language, perception, and action. The newer work is exploring whether a general-purpose robot also needs a learned representation of consequences, not only what action tends to follow a given observation, but how the world itself is likely to respond. That question reaches back to some of the oldest ideas in model-based control, but foundation models, large video datasets, and modern generative architectures have made it possible to revisit it at a very different scale.

Robots may ultimately need rich video-based predictors, compact latent models, lightweight subgoal generators, or some mixture of all three. Different tasks may require different levels of prediction, and the right architecture may depend as much on latency and embodiment as on model quality. What is becoming clearer is that simply recognizing the world and choosing an action is only part of physical intelligence. A capable physical intelligence system also needs some way of representing what its actions are likely to do to the world around it.