V-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations

V-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations

Video-language models often appear to understand motion while relying on shortcuts that work from a single frame. A caption may mention a ball and a hand without requiring the model to determine whether the ball moves left or right. A question may ask what is on a desk, even though the answer never depends on how the scene changes.

The paper by Keuntae Kim, Dongjun Kim, and Yong Suk Choi examines this gap between having dynamic information and actually using it. Their central claim is that a video encoder can represent speed, direction, order, and state changes, while the language model connected to it learns to ignore those signals unless training examples make the change itself necessary for the answer.

The problem: dynamics can be present but unread

The authors compare predictive video encoders such as V-JEPA with frame-wise, text-aligned encoders such as SigLIP. V-JEPA is trained to predict masked spatiotemporal regions in latent space, rather than to match frames directly to captions. This objective is intended to produce features that capture how scenes evolve.

Yet conventional video-language training can waste that capability. In the paper’s controlled speed-ratio test, a linear probe reads speed from frozen V-JEPA features with 86.7% accuracy, while the corresponding V-JEPA language model answers at 33.7%, close to the three-way chance level of 33.3%. The information survives the visual-to-language bridge, but the model’s final answer does not use it. The authors also report that the model gives the same answer to an original clip and a re-timed version 89% of the time.

This is an important diagnostic distinction: probing a representation demonstrates that information is decodable, not that the deployed model will use it when answering a question.

Where the dynamic signal lives

Kim and colleagues analyze a video and a transformed version of that video—for example, a reversed, mirrored, or re-timed clip. They decompose the encoder output into a shared content component and a change component. The question is whether the change component points in a consistent direction across different videos.

The measurements suggest a meaningful difference between encoder families. For V-JEPA ViT-L, the cross-video consistency measure for speed is 0.080, compared with 0.0016 for SigLIP so400m. For mirror changes, the corresponding values are 0.031 and 0.0014. The paper reports that V-JEPA’s change directions are roughly 20 to 100 times more consistent across videos than those of SigLIP in the tested settings.

Frame-wise encoders can still support temporal reasoning by comparing tokens at different positions. However, the authors argue that this is less useful for a compact, shared readout: time reversal may simply reorder frame representations rather than change the content of any individual token. Predictive spatiotemporal encoders, by contrast, can place aspects of the transformation into a common feature subspace.

Counterfactual supervision makes change matter

The proposed remedy is counterfactual supervision. The researchers transform public videos at training time and generate questions whose answers are determined by that transformation. A clip can be reversed so that opening becomes closing, mirrored so that left becomes right, re-timed to alter apparent speed, or rearranged to change event order.

The original and transformed clips can receive the same question but different answers. Because the two versions share most of their scene content, content alone cannot reliably solve the pair. The model must read the difference introduced by the transformation.

The generator uses fourteen constructions across six families, including sign flips, event order, temporal position, magnitude, counting, and completion or physical plausibility. The resulting counterfactual examples account for about 6% of the sampled training mixture. They are generated from public video labels and transformations, so the authors do not add new manual annotations or use a language model to write the labels.

V-JEPA-LM combines this supervision with a frozen V-JEPA 2.1 encoder and Qwen3 or Llama language models. The released models range from 1B to 8B parameters. Training uses one vision-language stage, 4.53 million sampled rows, 32 visual tokens per frame, and a bridge that reduces each clip to 1,024 visual tokens.

Results: strong temporal gains with less training data

Across twelve scores from eleven video benchmarks, the authors report that V-JEPA-LM leads the compared open models in every size class. Their 4B Qwen3 model reaches an average score of 72.14, compared with 61.08 for the best listed open 4B model and 64.79 for the best listed open 8B model. The 2B model reaches 66.02.

The largest improvements appear on motion-heavy evaluations. In the controlled 2B comparison, adding counterfactual supervision to V-JEPA raises the average score by 4.5 points at the end of training and improves the grouped speed ability by 38.1 points. On six held-out construction sets, the final 2B model averages 95.0, while the same V-JEPA setup without counterfactual examples averages 64.3. The authors also test unseen speed ratios and a successive-playback construction not used during training, reporting high performance on both.

Internal interventions support the proposed mechanism. Removing eight learned speed directions reduces unseen speed performance from 98.7% to 91.5% and lowers the TOMATO score from 50.9 to 46.4. A random subspace with similar energy has much smaller effects, suggesting that the removed directions are not merely arbitrary capacity.

Limitations and practical implications

The evidence is promising but scoped. All models use 32 frames, and the experiments focus on short clips rather than hour-long video. The authors do not target image question answering, OCR, documents, or charts; V-JEPA’s video-centric training is not designed for those tasks. They also acknowledge that open models trained on more data retain advantages on knowledge-heavy benchmarks.

The counterfactual generator depends on valid transformations and source annotations. Some speed labels rely on assumptions about the source motion, although the paper includes an audit of apparent motion. Benchmark overlap is also not zero: the authors identify 147 shared evaluation items, while reporting that removing them changes the twelve-score average by at most 0.04 points.

For practitioners, the main lesson is not simply to select a larger video encoder. First test whether the encoder contains a stable signal for the behavior that matters. Then design paired or answer-balanced examples that make ignoring that signal costly. This approach could reduce dependence on massive generic video-QA mixtures, but whether it transfers to long-form video, real-world deployment, or tasks outside the tested transformations remains an open question.


Sources

AI-assisted research summary. Check the original paper for complete methods, results, and limitations.

Avatar photo

Tech News

Jeff is a passionate blog writer who shares clear, practical insights on technology, digital trends and AI industries. With a focus on simplicity and real-world experience, his writing helps readers understand complex topics in an accessible way. Through his blog, Jeff aims to inform, educate, and inspire curiosity, always valuing clarity, reliability, and continuous learning.

↑ ↓ navigate Enter open Esc close