# JEPA: From Language Models to World Models

> Why Yann LeCun's JEPA bet makes the hidden state the training target instead of the next token, and what that shift means for vision-language models, robots, and long-horizon planning.

Category: Engineering
Published: 2026-07-07

---

A language model can tell you what usually comes next. A world model should tell you what happens if you act. That is the core of Yann LeCun's JEPA bet.

## Why next-token prediction got us this far

Before getting into JEPA, it is worth being fair to LLMs. Next-token prediction is brutally scalable: every document, code file, transcript, and forum thread becomes labeled training data. It also works because language is already compressed human experience. Text contains physics, social behavior, software conventions, recipes, plans, contracts, emotions, and arguments.

That is why LLMs became useful general interfaces. The mismatch appears when we ask them to become agents. Token prediction can still produce useful hidden states, but the training signal only checks whether the next visible token was likely.

It does not directly check whether the model represented the physical state, the future consequences, or the variables needed for control. For an agent that has to perceive, predict, act, and correct itself over time, that gap can become the bottleneck.

## Token prediction can be locally right and globally weak

A language model can generate the right next token while still having a weak internal trajectory for what comes next. That sounds subtle, but it matters. A token is a local target. A world state is a stronger constraint.

Say the model produces the right word while its hidden state drifts into a bad part of representation space. The immediate output may look fine. The system is still poorly positioned for planning, consistency, or future action.

The Semantic Tube Prediction paper is useful here. It separates two things that are easy to collapse: producing the right next token, and maintaining a useful hidden-state trajectory. The paper argues that a model can land in the correct token region while drifting away from the representation path that would support later predictions [1]. One of its core assumptions is:

> "The trajectory of x≤t is locally linear almost everywhere." [1]

The narrow point is that the next-token loss can reward the right next word without guaranteeing that the hidden trajectory remains useful for later prediction.

*A model can land in the right token region while its hidden state drifts away from a coherent future. Adapted from Semantic Tube Prediction [1].*

This is the first intuition behind JEPA: the target should not only be the next visible thing. The target should shape the internal state that makes future prediction useful.

If the goal is a better autocomplete system, predicting tokens is natural. If the goal is an autonomous system that can handle changing environments, future consequences, and multi-step plans, that is not enough. The hidden state itself probably needs to become a first-class training target.

![A model landing in the right token region while its hidden-state trajectory drifts](https://videodb.io/assets/blog-media/jepa-from-language-models-to-world-models/right-token-wrong-trajectory.webp)

*A model can land in the right token region while its hidden state drifts away from a coherent future. Adapted from Semantic Tube Prediction [1]. ([animated version](https://videodb.io/assets/blog-media/jepa-from-language-models-to-world-models/right-token-wrong-trajectory.mp4))*

## JEPA changes the object of prediction

A JEPA-style model predicts the representation of another view, missing part, or future state. It does not need to reproduce every pixel or emit every word. It learns an encoder and a predictor so that one latent representation can predict another. A simple version looks like this:

```
z_c = E(x_context)
z_t = E_target(x_target)
L_JEPA = D(P(z_c), stopgrad(z_t)) + R(E)
```

Here, E encodes the context and E_target encodes the target view or target state. P predicts the target representation, D measures distance in latent space, and R(E) keeps the representation from collapsing. That anti-collapse term is not a detail. It is the difference between learning a useful latent space and mapping everything to the same vector.

A naming note before going further: I use JEPA for the broad joint-embedding predictive idea. LeJEPA is a specific JEPA variant from a separate paper [3]. Its main difference is the anti-collapse mechanism: it adds SIGReg, a regularizer that pushes embeddings toward an isotropic Gaussian.

I use LeJEPA only when discussing that specific regularization and the theory built around it. It is not a synonym for every JEPA model.

This is the core shift:

| Architecture | Main prediction target | What it encourages |
| --- | --- | --- |
| LLM | next token | linguistic continuation |
| VLM | next token conditioned on visual context and prompt | visual-language alignment through generated answers |
| JEPA | latent target representation | representations shaped for prediction |

The important part is not that JEPA uses embeddings. Everyone uses embeddings. The important part is that JEPA trains the embedding space to be predictive.

V-JEPA 2.1 defines JEPA as a framework that learns representations "by making predictions in a learned latent space, rather than directly in the observation (input) space" [4]. That sentence is the architectural pivot.

*JEPA trains a predictor to match target embeddings rather than reconstructing pixels. Adapted from V-JEPA 2.1 [4]. Robot-arm photo source: Shixart1985 / Wikimedia Commons, CC BY 2.0 [12].*

For LLMs and most VLMs, the training check is still token-level: did the model assign enough probability to the next answer token? JEPA changes the check. Given a context view, can the model predict the target embedding for another view, masked region, or future state?

That does not solve agency by itself. But it puts pressure on the representation to carry information that survives across views and time. For a robot or long-running visual agent, that can include object position, pose, contact, reachable surfaces, and whether the scene is moving closer to the goal.

![A JEPA training loop comparing context and target embeddings](https://videodb.io/assets/blog-media/jepa-from-language-models-to-world-models/jepa-training-loop.webp)

*JEPA trains a predictor to match target embeddings rather than reconstructing pixels. Adapted from V-JEPA 2.1 [4]. ([animated version](https://videodb.io/assets/blog-media/jepa-from-language-models-to-world-models/jepa-training-loop.mp4))*

## What changes for vision-language models

This is where the JEPA argument becomes more concrete. Most VLMs turn perception into text. The model sees an image or video, takes a question or instruction, and produces tokens. That is useful, but it puts language generation in the loop even when the system mainly needs an updated state.

A vision-language JEPA changes the default object being predicted. Instead of mapping every visual question into an autoregressive text sequence, it can predict the semantic target embedding directly. VL-JEPA makes this argument clearly. The paper says:

> "Instead of autoregressively generating tokens as in classical VLMs, VL-JEPA predicts continuous embeddings of the target texts." [8]

That matters because the world is underdetermined at the surface. Ask a model what happens if a switch is flipped down. "The lamp turns off," "the room goes dark," and "the light shuts off" can all be correct.

In token space, those answers are different strings. In semantic embedding space, they should be nearby states. VL-JEPA uses the same example: "the lamp is turned off" and "room will go dark" are separate in raw token space, but ideally close in embedding space [8].

The implication is bigger than parameter efficiency. A VLM can become a continuous perception system. It can watch, update a semantic state, compare that state to a query or goal, and decode text only when language is actually needed.

VL-JEPA reports roughly 50% fewer trainable parameters in a controlled comparison. It also reports about 2.85x fewer decoding operations with selective decoding, while output quality stays similar [8]. The exact numbers will change with models and tasks. But the direction is important: language becomes an interface to a visual state, not the only form the state can take.

That changes what VLMs are good for. Instead of treating every frame as more context for a token generator, a JEPA-style VLM can maintain a compact, queryable representation of what is happening.

For video systems, that is a major shift. A model watching a long stream should not need to narrate everything. It should keep track of the meaningful state changes and speak when the task requires it.

## A world model is more than an embedding

Calling JEPA a world-model approach raises the bar. In his 2022 position paper, LeCun framed common sense as "a collection of models of the world" that tell an agent what is likely, plausible, and impossible [11].

That standard shifts attention from whether a model has embeddings to what those embeddings preserve. A representation can be useful for retrieval or classification and still drop the state variables needed for control. For agents, the latent space has to preserve the variables that matter for prediction and action.

A useful world model needs at least four things: a compact state representation, a way to predict future states, a way to condition those predictions on actions, and a planner that can search over possible futures. The strongest theoretical idea here is linear identifiability. If z is the true latent state of the world, a good learned representation h(z) should recover it up to a simple transformation:

```
h(z) = Qz
```

Here Q can be a rotation or another simple linear transform. The exact coordinates may change, but the geometry should remain usable. This matters because planning is geometry.

Suppose the real world has a smooth path from state A to state B. If the learned latent space twists that path into something warped, a planner working in latent space will make confident mistakes. It may optimize the wrong distance. It may choose a straight line in embedding space that becomes a bad trajectory in the real world.

A theory paper on LeJEPA makes this precise: under Gaussian latent variables, OU-style transitions, alignment, and Gaussian regularization, the learned representation can recover the true latent state up to rotation [2].

That is a narrow theorem about a specific setup, not a universal claim about all JEPAs. But it gives the right test: a world model is not an embedding that "looks semantic." It is an embedding whose geometry preserves the world well enough for prediction and planning.

This is the cleanest way to state the embedding-space objection: JEPA's biggest advantage is also its biggest risk, because everything important happens in latent space. If the latent space is faithful, JEPA gives agents a compact substrate for prediction. If it is collapsed, over-compressed, or geometrically wrong, the system can fail while looking mathematically elegant.

## The anti-collapse problem is the whole game

A naive latent-prediction model has an easy way to win: make every embedding the same. If E(x) = c for every input, then predicting the target embedding is trivial. The loss can look good while the representation is useless.

This is why JEPA methods care so much about stop-gradients, target encoders, exponential moving averages, whitening, variance constraints, contrastive losses, or explicit distributional regularizers. They are what prevent the model from cheating.

The LeJEPA paper's answer is SIGReg: push the distribution of embeddings toward an isotropic Gaussian [3]. The intuition is that a good latent space should be spread out, statistically well-conditioned, and difficult to collapse. The paper emphasizes that LeJEPA combines prediction with a regularizer that makes the embedding distribution behave well, reducing reliance on a bag of heuristics [3].

This also explains why "everything happens in embedding space" should not be dismissed as hand-waving. Embedding space is not magic. It has to be engineered, constrained, and tested against whether it preserves the right variables.

JEPA is powerful only if the learned space is shaped by objectives that make prediction, planning, and control possible. A bad latent space is worse than a bad image. At least a bad image can be inspected. A bad latent space may fail silently.

## Video makes the argument obvious

Language hides this problem because text is already an abstraction. Video makes it obvious. A video is not just a list of frames. It is a stream of state changes: objects move, hands interact with objects, cameras shift, and actions create consequences.

A pixel generator can learn to produce plausible future frames. That is useful, but it forces the model to spend capacity on surface detail.

LeCun's dashcam example is perfect. A generative video model may waste resources predicting the random motion of leaves beside the road. Those leaves occupy many pixels, even though they are mostly irrelevant to driving [9]. A JEPA-style video model asks a different question: how should the representation move?

```
ẑ_{t+1} = z_t + Δz_t
```

That is the right kind of abstraction for agents. The model does not need to render every texture to understand that an object moved left, a hand approached a cup, or a door is now more open.

V-JEPA 2.1 pushes in this direction by predicting masked or future visual representations, making features spatially dense, and preserving temporal consistency [4]. The paper's target is not just semantic recognition. It tries to make the latent state "spatially structured, semantically coherent, and temporally consistent" [4].

That phrase matters. A robot does not only need to classify a scene. It needs to know where things are, how they move, which surfaces matter, and which changes persist across time.

If a model sees a hand move toward a cup, the most important prediction is not the exact next pixel color of the hand. It is the evolving state: distance to cup, grasp possibility, object pose, likely contact, future occlusion, and maybe the intention implied by the motion. Those are latent variables, and this is where JEPA starts to look like a bridge from perception to action.

## What changes for vision-language-action models

VLMs connect perception to language. VLAs connect perception, language, and action. A VLA can read an instruction, look at a workspace, and output motor commands or action tokens. That is already a large step beyond captioning.

The question is whether the action system has a reusable model of consequences. It may instead be learning a direct mapping from observation and instruction to action.

JEPA points to a different middle layer for VLAs. The stack becomes: perceive the scene, encode the current latent state, predict how candidate actions change that state, then choose actions that move the world toward a goal. Language still matters because it specifies goals, constraints, and explanations. But the control loop needs a state space where consequences can be predicted before the robot acts.

This is the difference between a VLA that reacts and a VLA that can plan. If the instruction is "put the cup in the drawer," a direct VLA policy may learn useful behaviors from demonstrations.

A JEPA-style world model should also represent intermediate states: the cup is visible, the gripper is aligned, the cup is graspable, the drawer is open, the cup is above the drawer, the cup is released. Those are not just words. They are latent states the system should recognize, predict, and reach.

The implication for multimodal AI is practical. Bigger context windows help a VLM remember more frames. They do not by themselves give the model action-conditioned dynamics.

More demonstrations help a VLA imitate more behaviors. They do not by themselves give the system a compact space for counterfactual search. JEPA is interesting because it tries to make the hidden state itself predictive enough to support planning.

## Action is the line between representation and agency

A model that predicts what happens next from observation alone is still missing a key question for agency: what changes if the system chooses an action? For a world model, the core equation is:

```
z_t = E(o_t)
ẑ_{t+1} = F(z_t, a_t)
```

The model observes o_t, encodes it into latent state z_t, then predicts the next latent state after action a_t. That changes the question from "what comes next?" to "what happens if I do this?" This is the line LeCun keeps drawing around real agency: "I do not understand how you can even think of building an agentic system without the ability to predict the consequences of its actions" [10].

LeWorldModel is a concrete example of this direction. It learns from pixels, predicts next latent states conditioned on actions, and plans by rolling out candidate actions in latent space [5]. The paper states the JEPA instinct clearly:

> "Instead of attempting to model every aspect of the environment, JEPA focuses on capturing the most relevant features needed to predict future states." [5]

The planner can optimize toward a goal embedding:

```
a*_{1:H} = arg min_{a_{1:H}} D(ẑ_{t+H}, z_goal)
```

*[Video illustration of action-conditioned world model]*

This is still early. It is bounded, short-horizon, and evaluated in controlled settings. But architecturally it is the right shape.

The model does not need to generate a full video of the future. It needs to predict the state variables that make action selection possible.

This is why JEPA might replace language models as the core of some future AI systems. Language is still useful for instructions, tools, and explanation. But an agent that lives in the world needs a predictive state engine.

![The same current state leading to different predicted next scenes depending on the action](https://videodb.io/assets/blog-media/jepa-from-language-models-to-world-models/action-conditioned-world-model.webp)

*The same current state leads to different predicted next scenes depending on the action. Adapted from LeWorldModel [5]. ([animated version](https://videodb.io/assets/blog-media/jepa-from-language-models-to-world-models/action-conditioned-world-model.mp4))*

## Long-horizon autonomy is a decomposition problem

If a task requires many steps, a flat planner has two problems. Prediction error compounds with every rollout step, and the search space grows with the horizon. The Hierarchical World Models (HWM) paper [6] says single-level planning fails in two regimes: non-greedy tasks and long-horizon tasks where "prediction errors compound over autoregressive rollouts and the action search space grows exponentially with horizon."

Hierarchical latent world models attack this directly. The high-level model plans over coarse future states or macro-actions. Its first predicted future state becomes a subgoal. The low-level model then plans primitive actions to reach that subgoal. The system replans repeatedly as new observations arrive.

```
z_subgoal = F_high(z_t, l_t)
a*_{1:h} = arg min_{a_{1:h}} D(F_low(z_t, a_{1:h}), z_subgoal)
```

The high-level model handles long-horizon direction while the low-level model handles short-horizon precision. LeCun's version is simple: low levels make "short-term prediction with a lot of details," while longer-term prediction has to throw away detail so it does not diverge from reality [10].

*[Video illustration of hierarchical latent planning]*

This is one of the strongest arguments for JEPA-style world models. In the HWM paper, hierarchy improves planning across latent world-model backbones and task suites. One reported robot result is especially sharp: under the evaluated setup, flat VJEPA2-AC gets 0% success on Franka pick-and-place, while the hierarchical version reaches 70% for the cup task [6].

The failure is not only representation quality. Manual subgoals can rescue flat planners, which means the bottleneck is often decomposition.

This is also where language-only planning starts to feel brittle. A chain-of-thought can describe substeps, and a task list can store a plan. But an embodied agent needs subgoals in the same space where it predicts consequences.

LeCun puts the point bluntly: "your cat can do hierarchical planning, and your cat does not know language" [10]. The point is not that cats are the benchmark, but that hierarchical planning does not have to be expressed first as language. A useful agent should be able to represent "the cup is in a graspable pose" not merely as a sentence, but as a latent state it can recognize, predict, and reach.

![High-level latent subgoals decomposing one long rollout into short plans](https://videodb.io/assets/blog-media/jepa-from-language-models-to-world-models/hierarchical-latent-planning.webp)

*Hierarchy turns one long brittle rollout into repeated short plans toward latent subgoals. Adapted from Hierarchical Planning with Latent World Models [6]. ([animated version](https://videodb.io/assets/blog-media/jepa-from-language-models-to-world-models/hierarchical-latent-planning.mp4))*

## The risk: elegant latent spaces can lie

This is the part that should make us cautious. JEPA only works if the latent space preserves the right structure. There are many ways to fail:

- Collapse: everything maps to the same point.
- Over-compression: useful details disappear.
- Wrong variables: the representation captures texture instead of state.
- Warped geometry: latent distances do not match controllable changes.
- Off-manifold planning: the planner searches states the model never learned.
- Weak action coverage: the model cannot predict actions outside its data.
- Missing hierarchy: short-horizon predictions do not compose into long-horizon behavior.

This is also where human-like intelligence comparisons should be handled carefully. Humans do not merely compress the world into minimal vectors. We preserve messy, useful detail. We keep context, remember exceptions, and carry "inefficient" structure because it helps us adapt.

The *From Tokens to Thoughts* paper makes a related point about LLM embeddings: they "broadly align with human category boundaries, yet fall short on fine-grained semantic distinctions" [7]. A JEPA-style system that compresses too aggressively may be efficient and still miss what matters.

The core question becomes: which constraints make embeddings preserve the world variables needed for action? That is where the field is heading. Better objectives, better anti-collapse methods, better temporal structure, action-conditioned prediction, hierarchy, memory, and meta-control.

## The likely future stack

I do not think the future is a giant JEPA that simply replaces every language model use case. A more plausible architecture is simpler: language remains the interaction layer, while a JEPA-style world model holds and predicts state.

At a high level:

- Encoders map observations and instructions into latent state.
- A JEPA-style world model predicts how that state changes.
- A planner searches over future states and selects a path.
- Decoders turn the result into human-readable language or action-readable commands.

The important point is where language sits in the system. Language remains how humans instruct, inspect, and coordinate with machines. It is also how the system explains itself back to us.

If the latent space becomes faithful enough, future agents may use language as the interface over a predictive world model. The loop is simple: encode observations into state, predict how the state changes, plan over that state, and decode the result into words or actions.

## References

1. *Semantic Tube Prediction: Beating LLM Data Efficiency with JEPA*. https://arxiv.org/pdf/2602.22617
2. *When Does LeJEPA Learn a World Model?* https://arxiv.org/pdf/2605.26379
3. *LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics*. https://arxiv.org/pdf/2511.08544
4. *V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning*. https://arxiv.org/pdf/2603.14482
5. *LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels*. https://arxiv.org/pdf/2603.19312
6. *Hierarchical Planning with Latent World Models*. https://arxiv.org/pdf/2604.03208
7. *From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning*. https://arxiv.org/pdf/2505.17117
8. *VL-JEPA: Joint Embedding Predictive Architecture for Vision-language*. https://arxiv.org/pdf/2512.10942
9. Welch Labs, JEPA interview / explainer with Yann LeCun, part 1. https://www.youtube.com/watch?v=kYkIdXwW2AE
10. Welch Labs, JEPA interview / explainer with Yann LeCun, part 2. https://youtu.be/v_jDvpEGTIg
11. Yann LeCun, *A Path Towards Autonomous Machine Intelligence*, 2022. https://openreview.net/forum?id=BZ5a1r-kVsf
12. Shixart1985, *Robotic arm...*, Wikimedia Commons. https://commons.wikimedia.org/wiki/File:Robotic_arm_at_work_lifting_a_box_during_a_technology_exhibition.jpg

*First published on [VideoDB Labs](https://labs.videodb.io/research/jepa-from-language-models-to-world-models), July 7, 2026. This is the canonical home.*