Seen but Unsaid
Locating and Repairing Cross-Image Comparison in Vision–Language Models
Zhaoning Zhang · University of Toronto
Show a vision–language model two images, mark a point in the first, and ask which of four points in the second is the same one. It often gets this wrong, even when the right answer can be read from its own hidden states. I follow the signal from the pixels to the answer letter in Qwen3-VL (2B to 32B), InternVL3.5-8B and Gemma-3-12B to find where, between seeing and saying, the match goes missing.
probe .64, model .43visual tokens +36 pts over the answer probe16–32 heads that compare−17 to −22 pts when they are scrambled+28.3 pts from 0.02% of the parameters21–51% of the gain by transplant
The thesis
Seeing is not saying.
- I
Two failures, not one. Exact copies are compared and then lost before the answer. Semantic matches are seen but never compared.
- II
One pathway. A few dozen attention heads carry the comparison; scramble them and accuracy collapses.
- III
A lever training finds. Tuning only those heads repairs semantic matching, and free adapters route through them anyway.
Exhibit 01 · Follow the signal
Where does the match get lost?
I measure the signal at four points along the way, with three tasks that get harder: exact copies of synthetic shapes, the same physical point from a new viewpoint (HPatches), and the same part on a different object (SPair-71k).
Switch tasks and watch where the × moves.
Exhibit 02 · Across models
Two different failures
For exact copies the comparison works, but the answer doesn't come out. A linear probe at the answer position beats the model's own calibrated answer by 15 to 25 points in every family except the smallest Qwen. On real correspondence the two agree, so the gap isn't something the probe invents.
Semantic matching breaks earlier. A readout over the visual tokens beats the answer-position probe by 14 to 26 points on SPair-71k: the model sees the matching part and never compares it.
Exhibit 03 · Decide
Beside the output path
Maybe the decoded answer is just too faint to win? I tested that directly. Removing or amplifying the probe's subspace changes accuracy by at most 0.2 points, no more than a random subspace.
The layer itself isn't inert: adding the output direction of a letter the model didn't pick does flip answers. The information sits beside the path to the output, so more gain at that layer won't bring it back.
Exhibit 04 · Compare
The heads that compare
Each image is encoded on its own, so the comparison must happen inside the language model. A small set of heads does it: tokens of the right option look back at the reference far more than the distractors do.
To test whether they are needed, I redistribute their cross-image attention without changing its total. Synthetic accuracy falls 17 to 22 points and HPatches 15 to 32. Random or attention-matched heads never cost more than 7.
Exhibit 05 · Repair
A lever training finds
If these heads carry the comparison, they should be where to fix it. Training only their query and key projections, about 0.02% of the parameters, lifts SPair-71k by 28.3 points in Qwen3-VL-8B and 25.1 in InternVL3.5-8B, and it helps on HPatches and BLINK too, which it never saw.
Location matters. The same recipe on layer-matched or attention-matched heads does worse. Click a point to compare.
When nobody points the way
of a free all-layer adapter's gain returns when only the comparison heads' attention is transplanted into the base model. Random heads: at most 7%.
of that effect still works with the heads' base weights. Training changes what they read, not how they compare.
is all that is lost when the heads are reset inside the adapter. The pathway is sufficient, and redundant.
Cite this paper
@article{zhang2026unsaid,
title = {Seen but Unsaid: Locating and Repairing Cross-Image
Comparison in Vision--Language Models},
author = {Zhang, Zhaoning},
journal = {arXiv preprint arXiv:2611.21539},
year = {2026}
}