Weekly Paper Notes — one of the top picks from the 2026-11-14 CS paper digest. Area: AI / ML.
Authors: Ellis Brown, Chris Hoang (equal contribution), Shengyi Qian, Shang-Wen Li, Mike Lewis, Marjan Ghazvininejad, Luke Zettlemoyer, Lucas Beyer, Mengye Ren, Jakob Verbeek, Saining Xie, Hu Xu (Meta FAIR, New York University) arXiv: 2611.17334 · PDF · Project page
TL;DR
Most vision-language models bolt a separately pretrained image encoder (CLIP, SigLIP and relatives) onto a language model. Over the past year that recipe has started to come apart. Some labs now train the encoder from scratch together with the language model, and “encoder-free” models skip it and feed linear patch embeddings straight into the transformer. Each camp has published results that seem to back its choice, but the models differ in data, compute and architecture, so nobody could say which part of the encoder actually matters.
This paper splits the encoder into three separate choices: a visual prior from upstream pretraining, dedicated vision capacity (parameters only image tokens use), and pre-fusion placement of that capacity (it runs before image and text tokens meet). The authors train every model from scratch on the same data, with the same backbone and optimizer, and compare at matched lifetime compute, which charges encoder pretraining to the same budget as everything else.
The results are tidy. Pretraining the encoder buys nothing. Dedicated vision parameters are what matter: an encoder-free model needs roughly 3× the compute to match an encoder on VQA, and giving its backbone the encoder’s 0.4B parameters doesn’t close the gap. Those parameters don’t have to sit before the backbone, though. Vision-only attention and FFN branches added to the first eight backbone layers match a pre-fusion encoder at every size from 0.6B to 6.4B, and they cut time to first token for image requests.
Three choices that always shipped together
Figure: The three questions the paper asks, one per panel. Every recipe shares one backbone (“B”) and is trained from scratch on the same data. Source: arXiv:2611.17334, Figure 1.
The standard recipe runs Flamingo → LLaVA → PaliGemma → Qwen-VL. A ViT is pretrained on billions of image–text pairs, then attached to a language model through a projector and either frozen or fine-tuned. That encoder brings three things at once, and earlier comparisons always changed several of them together:
| Recipe | Pretrained prior | Dedicated capacity | Pre-fusion |
|---|---|---|---|
| Pretrained encoder | ✓ | ✓ | ✓ |
| Scratch encoder | ✗ | ✓ | ✓ |
| Encoder-free | ✗ | ✗ | ✗ |
| In-fusion (new) | ✗ | ✓ | ✗ |
The authors point to two confounds in prior work. Controlled studies such as Cambrian and MM1 compared which pretrained encoder to use, never whether to pretrain one. And they counted only multimodal training compute, treating encoder pretraining as free, even though SigLIP 2 alone saw 40B image–text samples (about 10 trillion image tokens). Encoder-versus-encoder-free studies then removed all three properties at once and gave the credit to pretraining. The paper also takes on Shukor et al. (2025), who trained both designs from scratch and found encoder-free matching or beating the encoder. It traces that result to two setup choices: a learning rate too low for the encoder, and a scaling-law fit on a loss average that hides the vision signal.
The protocol
Everything is held fixed except how pixels reach the backbone. The backbone is a Qwen3-style dense decoder (RMSNorm, RoPE, SwiGLU, GQA with QK-norm) at 1.7B and 4B for the main experiments, plus a 0.6B–6.4B ladder for scaling. The encoder is a 27-layer, 0.4B-parameter ViT modelled on Kimi K3’s MoonViT-V2. The “pretrained” variants use the same architecture, pretrained by the authors on the same MetaCLIP 1.2 images, so the prior they carry is controlled too. Text comes from Nemotron-CC, images from MetaCLIP 1.2 with captions regenerated by Qwen2.5-VL-32B, mixed 70:30 text to image–text. Training is at 224px (256 tokens), and evaluation instruction-tunes each checkpoint on Cambrian-7M at 448px before scoring 16 VQA benchmarks and 12 text benchmarks.
The accounting unit is lifetime compute, the sum of training FLOPs over every stage that produced the model:
C_lifetime = Σ_{s ∈ S} C_s
A warmup–stable–decay schedule lets the authors decay any checkpoint and read off a result, so one run produces a whole compute curve from 6.5×10²⁰ to 2.9×10²² FLOPs.
Finding 1: the pretrained prior doesn’t help
Four pretraining objectives were tested: MetaCLIP contrastive, WebSSL self-supervised, and Cap/CapPa captioning. Spending up to 15% of a 1.04×10²² FLOP budget on encoder pretraining comes out even with a scratch encoder. Spending 60% clearly hurts. Across backbone sizes the scratch encoder matches or beats MetaCLIP at every budget.
The pretrained encoder still loses when its pretraining is free. With the multimodal stage held at 2.7×10²¹ FLOPs, the authors grew MetaCLIP pretraining from 1.4 to 9.0×10²¹ FLOPs. ImageNet linear-probe accuracy rose from 80.4% to 84.4%, while average VQA peaked at the shortest pretraining run, and the scratch encoder matched it. A better recognizer doesn’t make a better VQA model. Delaying vision hurts as well: training on text alone for the first 25–75% of the run, with the same total image–text examples, clearly lowers VQA, consistent with the “vision laziness” other groups have reported.
Finding 2: dedicated capacity is the part that matters
Remove the encoder and VQA falls across both backbone sizes and every budget, while text scores stay level. The scaling ladder puts the price at about 2.9× compute for encoder-free to reach the encoder’s VQA and held-out caption loss, at every size from 0.6B to 6.4B.
The obvious objection is that the encoder-free model simply has 0.4B fewer parameters. Table 2 tests that directly on the 1.7B backbone at 2.6×10²¹ FLOPs, adding the encoder’s 0.4B parameters to the encoder-free backbone by widening, deepening, or both:
- Scratch encoder: 44.4 average VQA
- Encoder-free: 39.0
- Encoder-free, wider / deeper / both: 38.6 / 38.7 / 38.8
None of the three variants moves VQA, and all of them gain on text, because text tokens use the extra parameters too. Richer patch embeddings (a Gemma-4-style two-layer projection, a hierarchical MLP as in Inkling) don’t close the gap at 224px either. The deficit is largest on OCR and chart questions, the ones that need fine visual detail. The paper also notes that Inkling, released as encoder-free, shipped its successor Inkling 2 Small with a 292M-parameter vision encoder.
Finding 3: the capacity can live inside the backbone
Figure: (a) In-fusion adds vision-only modules whose outputs sum with the shared ones; (b) mixture-of-transformers gives each modality its own weights and shares only attention. Source: arXiv:2611.17334, Figure 4.
The new recipe, in-fusion, moves the encoder’s parameter budget into the backbone as additive, modality-routed experts. For token i with vision indicator mᵢ ∈ {0, 1}, every attention projection gets a vision-only delta:
P(z_i) = W_P z_i + m_i ΔW_P z_i, P ∈ {Q, K, V, O}
and the FFN gets a parallel vision-only branch reading the same normalized input:
h'_i = h_i + F(N(h_i)) + m_i ΔF(N(h_i))
The deltas and the output projection of ΔF are zero-initialized, so training starts from the plain shared model. Text-only inputs run through exactly the computation they would have without the vision modules. This differs from mixture-of-transformers (MoT), where vision tokens get their own full copy of the weights in every layer. Here every token still passes through the shared weights, and image tokens get an addition on top.
Where the budget goes matters a great deal. With the same 0.4B parameters as FFN-only branches on the 1.7B model (Table 3):
- A narrow branch in every layer: 39.4
- Eight wider branches spread across depth: 37.6
- Layers 21–28: 35.1
- Layers 1–8: 45.0 (scratch encoder: 44.4)
Only early placement works; the other three placements do no better than encoder-free. Layer-wise similarity analysis shows in-fusion models change image tokens mostly in the first eight blocks and little after, which is what an encoder does. Reserving the capacity for image tokens matters too. Giving the same layers-1–8 parameters to all tokens scores 40.1 (widened shared FFN) or 42.3 (separate all-token branch), behind the vision-only branch even with twice the training. The default variant adds full QKVO deltas to layers 1–8 (about 0.1B parameters at 1.7B) plus narrower FFN branches. It ties FFN-only at 1.7B and edges ahead at 4B.
Figure: Over the whole training run, in-fusion tracks the scratch encoder on VQA at both sizes, while encoder-free trails; text accuracy is comparable across recipes. Source: arXiv:2611.17334, Figure 6.
Beyond the controlled setting
Two follow-up experiments check that the findings aren’t an artefact of training everything from scratch.
Midtraining from a text-only LM. The authors pretrain their own 1.7B LM on 1T text tokens and then continue it to 1.33T token-equivalents with each recipe. In-fusion, MoT restricted to layers 1–8, and the scratch encoder come out comparable, and all beat encoder-free. Full all-layer MoT is worse despite 3.5× more vision-only parameters, which matches the finding that late placement doesn’t help. A free pretrained MetaCLIP encoder trails the scratch encoder here, mostly on OCR, chart and vision-centric questions. The authors’ explanation is that a scratch encoder learns features suited to the language model from its first update, while contrastive features that miss fine detail have to be reshaped by a model that already leans on text.
Serving. On one H200 with 16 concurrent requests (448px image, 256 text tokens), a separate encoder adds a serial step before the backbone can start. In-fusion and its FFN-only variant have clearly lower time to first token than the encoder at the same VQA, and the advantage grows with the vision budget. Training is also slightly cheaper: in-fusion trains as fast as encoder-free, while the encoder recipe is 3% slower at 4B. The authors add one caveat. Under continuous batching, in-fusion’s vision modules run inside forward passes shared with text-only requests, while a separate encoder adds latency only to image requests, so the serving win depends on the traffic mix.
Limitations the paper states
The encoders here are pretrained on the same images as the multimodal stage. The study can’t rule out that a released encoder trained on far more or different data (SigLIP 2’s 40B pairs, say) helps. Training is at 224px on MetaCLIP, so high-resolution and text-heavy data are left open. Backbones are dense and top out at 6.4B; the authors name sparse MoE backbones as the next step, since modality-routed experts fit naturally there. The text-to-image data mix was tuned on the scratch-encoder recipe and not re-tuned per recipe. Interleaved and video data aren’t covered.
Why this matters
The practical takeaway is a concrete architecture recommendation for anyone training a VLM from scratch or midtraining an LM into one. Don’t spend compute pretraining a separate vision tower, and don’t drop dedicated vision parameters either. Put them in the first few layers of the backbone as vision-only deltas, which is simpler to serve than a separate encoder and about as accurate.
It also clears up an argument that has run through 2026 releases. Labs going encoder-free were right that the pretrained prior isn’t needed, but wrong to conclude the parameters weren’t needed. The paper explains why encoder-free results looked competitive in some earlier studies (their losses averaged away the vision signal) and gives a specific fix, the 3× compute penalty with a design that removes it.
The method is also worth noting. Lifetime-compute accounting plus WSD checkpoints made it possible to compare recipes that spend compute in different stages. The same approach would work for other “bundled” design questions, such as separate speech encoders or pretrained tokenizers for audio and video.
Read alongside
- Shukor et al., Scaling Laws for Native Multimodal Models (2025): the from-scratch study whose encoder-free result this paper disputes.
- Tong et al., Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (2024): the benchmark suite and Cambrian-7M instruction data used here.
- Beyer et al., PaliGemma (2024): the high-resolution instruction-tuning stage the protocol borrows.
- Liang et al., Mixture-of-Transformers (2025): the modality-expert design in-fusion is compared against.
- Adept AI, Fuyu-8B (2023): the original encoder-free VLM.
- Hägele et al., Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations (2024): the WSD schedule that makes one run yield many compute points.
Links
📄 arXiv abstract · 📄 PDF · 🌐 unbundling.github.io
Part of the Weekly CS Paper Digest series. Summary written from a close read of the preprint; figures cropped from the arXiv PDF and reproduced here under fair use for educational commentary.