Unbundling Multimodal Vision Encoding
Multimodal language models need a way to turn pixels into representations that a language backbone can use. The standard solution is a separate vision encoder, often pretrained before multimodal training. Newer systems have questioned that design, training the encoder jointly with the language model or removing it entirely and feeding image patches directly into the backbone.
In Unbundling Multimodal Vision Encoding, Ellis Brown, Chris Hoang, and colleagues argue that these recipes combine several different decisions, making comparisons difficult. A conventional encoder contributes three things at once: a visual prior learned during upstream training, parameters dedicated to vision, and computation performed before image and text tokens interact. The authors’ central question is which of those ingredients actually matters.
A controlled comparison of vision architectures
The authors designed experiments to vary one choice at a time. Rather than starting with externally pretrained language or vision models, they trained the components from scratch on common data using a shared dense Transformer backbone and a common optimization recipe. They compared models at matched lifetime compute, which includes all training stages rather than counting only multimodal training.
The study evaluated 1.7B and 4B backbones, plus a scaling ladder from 0.6B to 6.4B. Visual ability was measured with an average across 16 visual question-answering benchmarks, including general understanding, knowledge, OCR and chart reading, and vision-centric tasks. Text performance was measured across 12 language benchmarks. All models received the same downstream instruction-tuning treatment before visual evaluation.
The main recipes were a separately pretrained encoder, an encoder trained jointly from scratch, an encoder-free model, and an “in-fusion” design. In-fusion models add vision-only attention and feed-forward modules inside the earliest layers of the shared backbone, while retaining shared processing and cross-modal attention.
Separate visual pretraining was not necessary
The first result challenges a common assumption about vision encoders: that their value comes primarily from a separately learned visual prior. At matched lifetime compute, the scratch-trained encoder performed on par with encoders pretrained using contrastive, self-supervised, or captioning objectives.
In fact, allocating a larger share of the total budget to encoder pretraining reduced performance once that allocation became substantial, because it took compute away from multimodal training. The authors also tested a setting where encoder pretraining was treated as free. Longer pretraining improved ImageNet recognition, but it did not improve downstream VQA; the scratch encoder remained competitive.
This does not show that all external vision encoders are useless. The experiments used the same image source for pretraining and multimodal learning, and did not test encoders trained on substantially larger or different datasets. The narrower conclusion is that, under the study’s controlled conditions, a visual prior was not required.
Dedicated vision capacity mattered
The strongest architectural distinction was whether the model had parameters reserved for visual processing. The encoder-free model projected patches directly into the shared backbone and consistently trailed the scratch encoder on visual understanding, while remaining broadly comparable on language benchmarks.
Adding the encoder’s parameter count to the shared backbone did not close the visual gap. The deficit was especially pronounced on OCR and chart tasks, suggesting that raw parameter totals are a poor substitute for capacity explicitly allocated to image tokens. Richer patch embeddings helped in some higher-resolution settings, but did not generally replace a dedicated visual module in the main comparison.
The scaling experiments reinforced this finding. Across backbone sizes, the authors report that encoder-free models required roughly three times as much compute as the scratch-encoder recipe to reach comparable VQA or held-out caption-loss performance. That is an empirical result from the tested scaling range, not a universal law for every encoder-free architecture.
Dedicated capacity can move inside the backbone
The paper’s most practical result is that dedicated vision capacity does not have to be placed in a separate pre-fusion encoder. In-fusion modules added vision-only branches to the earliest backbone layers, reserving those parameters for vision tokens while allowing all tokens to use the shared backbone.
Placement mattered. Modules in the first eight layers matched the scratch encoder, while the same budget spread across all layers or placed in the final layers performed much worse. Giving the extra parameters to every token also underperformed the vision-only routing design on VQA. The authors interpret this as evidence that image tokens benefit from specialized processing while they are still close to raw patch features.
At both 1.7B and 4B, in-fusion was comparable to the scratch encoder across training budgets. The result also held when starting from a pretrained language model: early in-fusion and a scratch encoder led encoder-free and a free pretrained MetaCLIP encoder in the reported midtraining experiment.
Why the result matters in practice
For researchers designing a multimodal model from scratch, the findings favor joint training with a dedicated visual pathway rather than automatically importing a pretrained vision tower. A team that wants to remove a separate encoder could redirect its parameters into early, vision-routed backbone modules instead of simply widening the shared Transformer.
There may also be serving advantages. In the authors’ H200 measurement, in-fusion reduced time to first token relative to a separate encoder at comparable VQA, because image processing occurred within the backbone rather than as a serial preprocessing stage. The authors report that in-fusion trained about as fast as encoder-free, while their encoder reference was modestly slower; they caution that implementation and optimization choices affect these timing comparisons.
Limitations and open questions
The evidence is substantial but bounded. The experiments used dense backbones, images that were 224 by 224 during multimodal training, a fixed data mixture, and models no larger than 6.4B parameters. The pretraining data was deliberately controlled rather than drawn from the much broader datasets often used for released encoders. The paper does not establish how the conclusions transfer to mixture-of-experts systems, video, heavily interleaved image-text data, or substantially higher-resolution training.
Overall, Brown and colleagues’ study separates the vision encoder’s bundled features and finds that only one consistently survives: dedicated visual capacity. The authors’ evidence favors training that capacity jointly with the multimodal backbone and placing it early, whether in a conventional encoder or inside the backbone itself.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.