JVT: Just Video Transformer makes pixel-space video diffusion simple
Video diffusion has largely moved into latent space because raw pixels are expensive to model. A pretrained autoencoder compresses frames before denoising, reducing computation, but it also introduces reconstruction error and adds another system to train and run. The authors of JVT: Just Video Transformer makes pixel-space video diffusion simple ask whether video can instead be generated directly in pixel space with a comparatively ordinary Transformer.
Their answer is JVT, or Just Video Transformer: a DiT-style architecture that operates on noisy RGB video without a pretrained autoencoder. The model does not use U-Net skip connections, cascaded stages, auxiliary detailers, or a separate reconstruction module. The paper presents this as a deliberately simple baseline and studies which components are actually needed to make pixel-space video diffusion practical.
What JVT changes
JVT follows a clean-data prediction and rectified-flow training formulation. Rather than predicting noise directly, the network receives a noisy video and predicts the clean video, with a velocity loss used for supervision. Large spatiotemporal patches reduce the number of visual tokens that the Transformer must process.
The central architectural investigation concerns video patch embedding. The authors compare standard tubelet embedding with two factored alternatives: patch-first embedding, which extracts spatial features before aggregating across time, and frame-first embedding, which reverses that order. They also test a channel bottleneck, allowing the embedding to pass through a smaller intermediate representation before expanding to the Transformer width.
JVT additionally experiments with register tokens. These learnable tokens provide extra computational slots and can be retained or removed during sampling to create register guidance, an analogue to classifier-free guidance. Standard adaptive layer normalization, or AdaLN, remains the main route for injecting timestep and class information.
Evidence from video-generation benchmarks
The authors evaluate JVT on class-conditioned UCF-101 and unconditional UCF-101, FaceForensics, SkyTimelapse, and Taichi-HD. The experiments use 16-frame clips at 256 by 256 resolution and report Fréchet Video Distance, or FVD, using 2,048 generated samples.
On the paper’s reported comparisons, JVT outperforms the listed competing methods across these benchmarks. Its class-conditioned UCF-101 score is 136.5 without register guidance and 129.2 with it. For unconditional generation, the reported scores are 413.5 on UCF-101, 32.86 on FaceForensics, 46.58 on SkyTimelapse, and 119.1 on Taichi-HD. Register guidance improves UCF-101 substantially, reaching 215.9 in the unconditional setting, but does not consistently help the other datasets.
Those results support the authors’ main claim: a direct pixel-space DiT-style video model can be competitive with the latent diffusion baselines included in the study. They do not establish that pixel-space modeling is universally superior. FVD depends on the evaluation protocol, sample count, dataset, training recipe, and baseline implementation, and the paper itself reports a separate UCF-101 FVD at 10,000 samples to show that comparisons using different sample counts are not interchangeable.
What the ablations reveal
The ablation results point to patch-first embedding with a bottleneck width of 32 as the preferred quality-efficiency compromise. Bottlenecked embeddings generally achieve lower training loss than the plain tubelet baseline, while some configurations also reduce embedding cost and improve FVD. However, the relationship is not monotonic: an overly narrow bottleneck can discard useful spatiotemporal information, while a wider one can add computation without delivering a clear quality benefit.
AdaLN proves harder to replace. Removing it worsens class-conditioned UCF-101 FVD from the baseline’s 136.5 to 156.3 in one comparison. Register-based conditioning also fails to provide a reliable substitute. The authors therefore retain standard AdaLN and treat registers primarily as a sampling-time guidance mechanism rather than as a replacement conditioning pathway.
The register results are notably dataset-dependent. The paper observes stronger visual-token interaction with registers on UCF-101 than on FaceForensics, SkyTimelapse, or Taichi-HD. The authors hypothesize that UCF-101’s broader action and semantic diversity gives registers more useful global information, but they present this as an interpretation rather than a demonstrated causal explanation.
Adding geometry to video generation
JVT’s pixel-space representation also makes it straightforward to generate aligned modalities alongside RGB. The authors demonstrate this with camera-conditioned RGB-disparity generation on RealEstate10K. Given an initial frame and a target camera trajectory, the model jointly predicts future RGB frames and disparity. Disparity can then be converted to depth and unprojected with camera parameters into colored point clouds, producing an explicit scene representation without an additional reconstruction model at inference.
In the reported experiment, RGB-only JVT achieves an FVD of 27.0, while the RGB-disparity version scores 27.5. The joint model improves relative translation accuracy at both reported thresholds and raises the combined AUC from 84.1 to 88.6. Rotation accuracy is already high for all compared methods, while translation remains the harder problem. These findings suggest that aligned geometry can help a generator follow camera movement, although the evaluation relies on Pi3X-generated depth targets during training because RealEstate10K lacks dense depth annotations.
Practical implications and limitations
For researchers, JVT offers a useful argument for testing simpler pixel-space designs before adding specialized architecture. Its common pixel grid also provides a natural interface for depth, surface normals, segmentation, optical flow, or other aligned signals. That could be valuable for world-model research where appearance and geometry need to remain synchronized.
There are important boundaries to the evidence. The experiments focus on short clips, 256-by-256 resolution, and selected datasets rather than long-form, high-resolution generation. The model is still computationally substantial, and guidance can require multiple denoiser evaluations. The strongest register gains appear on one dataset, while camera translation remains imperfect. Finally, the study compares against the baselines and protocols it selected; broader compute-matched comparisons would be needed to determine how JVT scales against current large latent video systems.
Overall, the authors show that pixel-space video diffusion does not necessarily require a complicated U-Net or a multi-stage pipeline. JVT’s contribution is less a finished production system than a clear experimental recipe: use large patches, preserve effective conditioning, measure efficiency carefully, and exploit the pixel grid when aligned geometry matters.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.