Proving and Improving the Platonic Representation Hypothesis with JEPAs

Proving and Improving the Platonic Representation Hypothesis with JEPAs

Can independently trained models for images, text, video, and other modalities learn essentially the same representation of the world? The Platonic Representation Hypothesis, or PRH, says that they may converge toward a shared description as models and datasets scale. Randall Balestriero and Yann LeCun argue that scale is not the only—or even the best—lever. In their paper, they connect cross-modal alignment to the way self-supervised models define positive training pairs.

From a hypothesis to a conditional theorem

The authors study Joint-Embedding Predictive Architectures (JEPAs), which train an encoder to give related inputs similar representations while using an anti-collapse mechanism to prevent every input from receiving the same embedding. Their central result is the Platonic Representation Theorem.

Under an idealized setup, separately trained JEPAs learn the same representation of a shared world, up to a rotation of the embedding space, when their positive pairs connect the same world states. The modalities do not need to see one another during training. Images and captions, for example, can be encoded independently if their training objectives identify the same underlying scenes.

The qualification matters. The theorem assumes a finite world, injective renderings, matched positive-pair distributions, population-level optimization, and minimizers that are unique up to rotation. A spectral analysis shows why: with whitening and a squared-distance objective, the optimal representation is related to the leading eigenvectors of the graph induced by positive pairs. If two modalities construct the same graph, their solutions share the same geometry, differing only by an orthogonal transformation.

Good views versus bad views

The paper distinguishes between “good views” and “bad views.” Good views are separate observations of the same world state or scene, such as two photographs of an event or two captions describing one image. Bad views are corruptions of one observation, such as masked versions of the same sentence or crops of one photograph.

This distinction is important because corruptions often preserve modality-specific nuisance information. A caption may retain its author’s wording, while an image crop preserves viewpoint, lighting, or composition. The authors show theoretically that an encoder can spend embedding dimensions on these nuisances. Those dimensions may be useful within a modality, but they cannot be aligned with another modality that does not observe the same details.

The paper also presents a bound on misalignment when two modalities use different positive-pair graphs. In that analysis, alignment worsens as the graphs diverge and improves when the relevant eigenvalues are separated by a larger spectral gap. This gives a practical interpretation: the quality and consistency of positive pairs shape which factors become stable, shared directions in the representation.

Why next-token prediction is inefficient for this goal

Balestriero and LeCun recast next-token prediction as a form of JEPA with poor views. Its positive pair is a prefix and its continuation within a document. These fragments may share words, style, or document-level regularities, but they do not necessarily describe the same world state. The authors present this as a theoretical explanation for why language models can align with vision, yet often require very large scale to do so.

This interpretation is partly a formal result and partly a heuristic extension. The paper proves the structural connection between the softmax loss and a JEPA-like objective, but the nuisance argument is not claimed as a direct corollary for ordinary language models because the full spectral assumptions do not apply to next-token prediction.

Evidence across datasets and model sizes

The experiments compare good-view JEPAs, bad-view JEPAs, and next-token models using the same or closely matched text, encoders, and training conditions. On Flickr30k, caption-view training consistently produces stronger alignment with a separately trained image encoder. In one reported comparison, the good-view model achieves mutual 10-nearest-neighbor alignment of .062, compared with .047 for masked-span training and .042 for next-token prediction. After fitting one rotation, image-to-caption retrieval at rank one is .044 for the good-view model, versus .027 and .021 for the other two objectives.

At larger scale, the authors train text encoders on the texts associated with up to 940 million web images. They report that a good-view JEPA reaches the alignment obtained by next-token prediction with 42 times fewer training samples on the same corpus. A 109-million-parameter text JEPA scores .184 on the paper’s WIT-1024 alignment benchmark with DINOv2, above every evaluated language model, including models up to 72 billion parameters. These comparisons measure alignment with particular vision encoders and benchmarks; they do not establish that the smaller model is broadly more capable than those language models.

The authors also fit an empirical scaling law using the fraction of good views and the number of samples seen. It predicts 85 held-out runs with an error comparable to the variation between random seeds within the tested range. However, the relationship breaks down when training is extended far beyond the fitted range, especially with repeated data. The authors describe the law as an empirical summary of one recipe, not a consequence of their theorem.

Applications and practical implications

Once two embedding spaces differ mainly by a rotation, they can be composed with a small alignment layer. The authors demonstrate image captioning without training an image encoder on image-caption pairs: a frozen visual encoder is mapped into a text JEPA space, and a decoder trained on text alone generates captions. Their strongest reported configuration reaches CIDEr .838 on COCO, though the rotation is fitted using paired data in that experiment.

They also test pair-free alignment using shared geometry. On a benchmark that aligns unpaired image and caption collections, the 109-million-parameter good-view JEPA reaches 64.4% top-5 accuracy on ImageNet-100, compared with at most 56% among the reported language-model settings they evaluate. This result depends on the particular unpaired alignment procedure and data, so it is evidence for a promising pipeline rather than a general guarantee of zero-shot performance.

For practitioners, the message is direct: if the goal is a representation shared across modalities, construct positives that correspond to the same underlying state. Group multiple descriptions, frames, or sensor readings of an event; remove details that are not shared; and evaluate alignment separately from task accuracy. More data can help weak positives, but the authors’ results suggest that better positives can be substantially more efficient than simply increasing model size.

Limits of the claim

The study’s theory does not cover the full complexity of real multimodal data. Real observations are lossy rather than injective, finite-sample optimization may not reach population optima, and the uniqueness result is established most cleanly for spectral or second-order objectives rather than the full LeJEPA constraint. Most image experiments use crop-based objectives, which the paper itself classifies as bad-view training.

There is also an important data caveat. Many large-scale good views come from web image texts, including recaptions generated by a vision-language model. The authors report that removing those recaptions reduces alignment, so the strongest results partly rely on information distilled from image-text systems. Even with these limitations, the paper offers a useful design principle: shared representations are shaped by shared views of the world, not by parameter count alone.


Sources

AI-assisted research summary. Check the original paper for complete methods, results, and limitations.

Avatar photo

Tech News

Jeff is a passionate blog writer who shares clear, practical insights on technology, digital trends and AI industries. With a focus on simplicity and real-world experience, his writing helps readers understand complex topics in an accessible way. Through his blog, Jeff aims to inform, educate, and inspire curiosity, always valuing clarity, reliability, and continuous learning.

↑ ↓ navigate Enter open Esc close