JEPA Explained: Why Robot AI Predicts Features, Not Pixels

Why JEPA is interesting

You've probably seen tons of headlines about how data hungry the transformer LLM architecture is in order to learn. More data, more compute, and the model gets smarter. However, one massive problem with robotics is that there is no where near the amount of collectable data to train models like LLMs. LLM coding agents can just be fired up a ton of sandboxed reinforcement learning data; essentially getting the LLM to code for millions of hours. After all code is just bytes. But robots don't have this luxury. So we search for architectures that can learn from less data, and JEPA is one of the most promising.

If you build mobile or web apps, LLM APIs probably shaped your mental model of AI: send text in, stream text out. That model serves you well right up until you get curious about robots, and then it stops helping. The models moving robot arms today make different bets, and the loudest of those bets is JEPA. We came to robotics from app development ourselves at Wendy, and this post is the guide we wish we had.

The short version of the story: Yann LeCun, one of the three researchers who shared the Turing Award for deep learning, published a position paper in 2022 arguing that the path to machines with common sense runs through architectures that predict features, meaning compact numeric summaries of a scene, rather than generating raw data. Meta then shipped a series of models built on that idea, ending with a video model that can drive a real robot arm in a lab it has never seen. In late 2025 LeCun left Meta to chase the idea full time, and by March 2026 his new company, AMI Labs, had reportedly raised just over a billion dollars before releasing a single product. You do not have to agree with the bet to want to understand it.

What JEPA is not

A little honesty before we go any further. JEPA models are not chatbots and will not answer questions unless you bolt a language model onto it. The robot results near the end of this post, though genuinely impressive for how little robot data they needed, are far from solved manipulation: grasp success drops to 25 percent on box-shaped objects, you currently have to hand it goals as images, and planning horizons are short. And the generative camp isn't much worse for simple tasks so we can't just dismiss it in favor of JEPA. NVIDIA's Cosmos models power a competing strategy that generates synthetic robot training data offline, where rendering cost matters far less, and both approaches are getting results. The field has not picked a winner, which is precisely why it's a good time to understand both.

So let's understand it. The whole thing starts with a question that sounds almost silly: when a model fills in a missing piece of the world, where should you grade its answer?

Autocomplete worked for text, so why not for video?

LLMs learn by playing autocomplete by taking a mountain of text, hiding the next word, and training the model to guess it. Each token the LLM guesses is just a short chunk of text, roughly a word. This works beautifully for language partly because the menu of possible next tokens is small and fixed. There are tens of thousands of tokens, and spreading a bet across a few plausible ones is a perfectly good answer.

Now let's try the same trick on a robot's camera feed. Hide the next frame and ask the model to paint it, every pixel. Suddenly the menu of possible answers is astronomically large, and most of the differences between them are noise nobody could ever predict: which way each leaf on a tree flutters, exactly where the sunlight sparkles on water, the precise grain of a carpet. As LeCun puts it, "the real world is not entirely predictable." Punish a model for every wrong pixel and you force it to spend most of its capacity on exactly the details that matter least.

And here is the painful part: when many sharp answers are all equally plausible, the safest pixel-level answer is their average, and the average of many sharp images is a blur. We built the scene below so you can watch that failure happen, played out on an impressionist beach painting, and then watch the JEPA fix. It grades the same prediction two different ways.

Grade on pixelsGrade on featuresReplay

The input the model sees

The painting is chopped into patches and one patch is hidden. The model must fill it in from context. It clearly belongs to the crowd on the shoreline, but the exact figures could be painted many different ways.

Equally correct realities

What a pixel-graded model converges to

pixel error:36%

Every sharp guess is heavily punished, because its brushstrokes and figures never land exactly where they do in the painting.

This is the core JEPA argument. Grading predictions pixel by pixel forces the model to waste capacity on unpredictable detail. Grading them on a feature summary lets it focus on what is actually knowable. And yes, one of the four candidates above really is the painting's hidden patch, but nothing in the visible context can tell you which.

If you have ever maintained screenshot tests, you have already lived this problem. Pixel-perfect screenshot diffs are notoriously flaky, because antialiasing, timestamps, and cursor blinks change pixels without changing meaning. So teams reach for a steadier comparison: diff the DOM tree, or the view model, or an accessibility snapshot. It's the same interface, but the comparison happens at the level of what it means rather than how it renders. JEPA makes exactly that move, but for learning itself: stop grading the model on pixels, and grade it on a compact summary of meaning instead.

Before we look at the architecture, we should pin down the vocabulary, because the papers lean on a few words constantly.

Jargon break: four words the papers will not stop using

• Encoder. A function that takes raw input (an image, a video clip, a sentence) and returns a fixed-size array of numbers. If it helps, think of the signature as encode(image) -> number[1024]. In modern AI the function happens to be a large neural network, but the contract really is that simple.
• Embedding. The array that comes out of the encoder. The papers also call it a representation, sometimes a latent, and informally the features of the input. All of these names mean the same thing: a list of floats that summarizes the input. Useful embeddings have a lovely property, which is that similar inputs land near each other, so the distance between two embeddings measures how similar two inputs are in meaning.
• Embedding space. The set of all possible embeddings, treated like a map. Also called representation space, latent space, or feature space. When someone says a model "predicts in latent space," they mean its outputs are these arrays, not pixels or words.
• Self-supervised learning. Training without human labels. Nobody tags millions of images by hand. Instead, the data grades itself: hide part of it, predict the hidden part from the rest, score the guess, repeat. LLM autocomplete is self-supervised learning on text, so you already knew this one by a different name.

What a JEPA actually is

A JEPA is three networks wired together, and the whole trick lives in where we measure the error. There is a context encoder that sees the visible part of the input, a target encoder that sees the hidden part, and a predictor that tries to guess the target encoder's output from the context encoder's output. Notice that the predictor produces embeddings, never pixels. The training error is simply the distance between the predicted embeddings and the target embeddings, two short arrays of numbers.

Step through it below. The scene follows one training example through all three parts.

1Mask2Encode3Predict4CompareReplay

1. Mask.The input is split into patches. Blue patches are the context the model is allowed to see. The two orange patches are hidden and become the targets.

That's the whole architecture, and the name suddenly makes sense: both the visible context and the hidden target get embedded, jointly, and one embedding is predicted from the other. Because the error lives in embedding space, the target encoder is free to discard unpredictable detail. If we can't predict the exact arrangement of leaves, the encoder learns to stop encoding it and wastes no capacity chasing it. A generative model never gets that option, because its training grades every pixel, including the hopeless ones.

The cheating problem, and the slow copy that fixes it

At this point the sharp-eyed reader might notice that this setup has an exploit. If both encoders learn freely, the laziest way to win is for every encoder to output the same constant array for every input. The prediction is then trivially perfect, the error is zero, and the model has learned nothing at all. The papers call this representation collapse, and it's the reason joint-embedding methods took years to tame.

The fix I-JEPA and V-JEPA use is delightfully simple: nobody ever trains the target encoder directly. Instead, its weights (the numbers inside the network that training adjusts) are an exponential moving average of the context encoder's weights, which is just a smoothed, slowly trailing copy, the same rolling-average idea you would use to smooth a noisy sensor reading or debounce a jittery value in a UI. The context encoder learns, and the target encoder drifts along behind it on a delay. Because the target keeps moving on its own schedule, the constant-output cheat stops being a stable solution, which pushes the system toward embeddings with real content. As a bonus, this is cheap because there is no second training process, only a weighted copy of the weights.

Watch the delay below. Every gradient step jolts the blue context encoder to new weights, while the orange target encoder only ever glides toward wherever the blue one is now. Flip the momentum to see the trail tighten or stretch.

Momentum 0.9Momentum 0.99Gradient step 0 of 8: the context jumps, the target glides.Replay

Every gradient step jolts the blue weights to a new value. The orange weights never train; each moment they move a small fraction of the way toward wherever the blue weights are right now, which is all an exponential moving average is. Flip the momentum to see the trail tighten or stretch: with high momentum the target barely flinches at any single step, and that steadiness is what makes it a goalpost worth predicting.

Researchers found the trick a little unsatisfying, since it works better than anyone can fully explain. In November 2025, LeCun and Randall Balestriero published LeJEPA, which replaces the moving-average machinery with a single extra penalty term in the training error, plus a proof of why that penalty prevents collapse. The details are beyond this post, but the direction is worth knowing: the architecture is getting simpler and better understood.

The family tree: I-JEPA, V-JEPA, V-JEPA 2, VL-JEPA

The idea shipped as a series of models, each pointing the same architecture at a bigger slice of the world.

I-JEPA proved the idea on still images. Hide a few regions of a photo, predict their features from the rest. The headline was efficiency: Meta trained a 632 million parameter vision model (parameters are the adjustable numbers inside a network, the same things we called weights a section ago) on ImageNet, the classic million-image research dataset, with 16 GPUs in under 72 hours. The paper reports that as roughly 2 to 10 times fewer GPU hours than comparable methods, while learning stronger features. This efficiency is extremely notable since the architecture didn't spend unnecessary GPU resources on unpredictable details.

V-JEPA moved to video, hiding large chunks of each clip across both space and time. Meta reported it reached the same quality as pixel-based training from 1.5 to 6 times less data and compute, and the features it learned, reused as-is with no further training (the papers call this frozen), turned out to be very good at questions like "what action is happening here," which is precisely the kind of understanding a robot needs before it can act.

V-JEPA 2 scaled the recipe to a 1.2 billion parameter model trained on over a million hours of internet video. It set new marks on motion-understanding benchmarks, and, more importantly for us, it became the foundation for the robot work in the next section. Meta released the code and weights for commercial and research use, and the encoder loads straight from Hugging Face.

VL-JEPA is the newest branch, and it points the same trick at language. A standard vision-language model answers a question about a video by generating text one token at a time. VL-JEPA instead predicts a single embedding of what the whole answer should mean, and only runs a small text decoder when someone actually needs to read it. In the paper's controlled comparison it beat token-by-token training while training half as many parameters, and in streaming settings it can skip nearly two thirds of decoding work by staying silent until the meaning of the scene actually changes. As of this writing it's a paper without released weights, but it completes the pattern: images, video, action, and now language, all learned by predicting meaning instead of raw output.

There is also active work running the connection in reverse: LLM-JEPA adds a JEPA-style feature-prediction objective on top of ordinary LLM training and reports consistent gains. The two worlds are converging from both directions.

The robotics payoff: a model that can imagine outcomes

Everything above is perception: understanding what is in front of the camera. Robotics needs one more ingredient, a world model, which is any model that answers "if I take this action, what happens next?" Give a robot that, and planning stops being guesswork: it can audition actions in imagination and only perform the winner.

This is exactly what Meta built on top of V-JEPA 2, called V-JEPA 2-AC (action-conditioned). They froze the big video encoder and trained a much smaller predictor, about 300 million parameters, that takes the current scene's features plus a candidate arm movement and predicts the features of the resulting scene. The startling part is the training data: under 62 hours of robot recordings from DROID, a public dataset of arm demonstrations, with no rewards and no task labels. They dropped the result onto Franka robot arms in two labs it had never seen, with no practice runs in those rooms (the papers call this zero-shot), and it picked and placed novel objects with 65 to 80 percent success.

How does a feature predictor become action? Watch the loop. One note about the little arm below: the planner only ever picks where the gripper should go, and a separate layer called inverse kinematics turns each target into joint angles, which is exactly the division of labor we explored in the inverse kinematics post.

Step 1 of 3Imagining three candidate gripper targets in feature space...Replay

This loop is called model-predictive control. V-JEPA 2-AC runs exactly this pattern: encode the current camera frame and the goal image into features, imagine candidate movements forward in feature space, score each imagined future by its distance to the goal features, execute the best first move, and re-plan from what the camera actually sees next. The little arm here follows each executed target with a small inverse kinematics solver, built the same way as the solvers in our inverse kinematics post.

The loop has a name, model-predictive control, but the logic fits in a sentence: propose many candidate movements, imagine each one's outcome in feature space, score each imagined future by its distance to the goal image's features, execute the best first move, look at the camera, and re-plan. The loop never renders anything. The imagination is pure embedding math, which is why it's fast: V-JEPA 2-AC plans each action in about 16 seconds on a single consumer GPU, where a comparable generative world model (NVIDIA's Cosmos, in the paper's comparison) took around 4 minutes per action, because it has to paint video frames for every future it considers.

The same pattern keeps appearing elsewhere. DINO-WM, from LeCun's and Lerrel Pinto's groups at NYU, froze an off-the-shelf image encoder and trained only a small predictor over its features, then planned exactly the same way, and it beat much heavier world models on pushing and maze tasks. And VLA-JEPA, now shipped in Hugging Face's LeRobot library, uses a V-JEPA 2 world model purely as an extra training signal for a robot policy, which is the network that maps what the robot sees to what it does next: the world model never runs on the robot at all, yet the policy it shaped scores about 96.5 percent on the LIBERO manipulation benchmark. The JEPA recipe in the wild is a frozen big encoder, a small trainable predictor, and planning or learning in feature space.

Why this matters for Local AI

Here is the part we care about most at Wendy. Look at what actually has to run on the robot in the planning loop above: an encoder pass over a camera frame, then a few hundred imagined futures, each one a forward run of a 300 million parameter predictor over arrays of numbers. There is no video generation and no cloud round trip. That workload is in the same weight class as models developers already run on Jetson-class boards today (we keep a running guide to which LLMs fit on Orin Nano and Thor, and a 300M predictor is modest by comparison). Fair warning on the numbers: the paper measured the 16 seconds per action figure on a desktop RTX 4090, so treat on-robot latency as an argument from model size rather than a published benchmark, for now.

The rest of the stack is the unglamorous part we build for. A JEPA planner outputs gripper poses, and the inverse kinematics layer still has to turn those poses into joint angles. The camera pipeline, the policy, the planner, and the safety logic all have to ship to a fleet of devices, update without bricking anything, and roll back when an experiment goes wrong. On WendyOS, each of those pieces is just a containerized app you deploy with wendy run, whether the model inside it is a YOLO detector or a V-JEPA 2 predictor.

Where to start as a developer

Three practical on-ramps, in increasing order of commitment. First, load the V-JEPA 2 encoder from Hugging Face, feed it video frames, and use the embeddings for search or classification: the experience is close to calling an embeddings API, except it runs on your own machine. Second, install LeRobot and run its VLA-JEPA policy on the LIBERO benchmark to see the training-signal version in action. Third, put a Jetson on your desk, flash it, and make the deployment problem real, which is the part our quickstart exists for. However far you go, you now know what the papers mean when they say a robot is predicting the future: it measures the distance to the goal in feature space instead of painting pixels.

Sources

• Yann LeCun. A Path Towards Autonomous Machine Intelligence. Position paper, OpenReview, June 2022.
• M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y. LeCun, and N. Ballas. Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture (I-JEPA). CVPR 2023.
• A. Bardes, Q. Garrido, J. Ponce, X. Chen, M. Rabbat, Y. LeCun, M. Assran, and N. Ballas. Revisiting Feature Prediction for Learning Visual Representations from Video (V-JEPA). 2024.
• Meta AI. Introducing V-JEPA 2. Blog post and paper, June 2025. Code and weights at facebookresearch/vjepa2.
• D. Chen, M. Shukor, T. Moutakanni, et al. VL-JEPA: Joint Embedding Predictive Architecture for Vision-Language. December 2025.
• G. Zhou, H. Pan, Y. LeCun, and L. Pinto. DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning. ICML 2025.
• Sun et al. VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Model. February 2026. Shipped in Hugging Face LeRobot v0.6.0.
• H. Huang, Y. LeCun, and R. Balestriero. LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architectures. September 2025.
• R. Balestriero and Y. LeCun. LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics. November 2025.
• Fortune. Coverage of AMI Labs' $1.03 billion seed round. March 2026.