Index / [02] Preprint I arXiv:2611.15041 · cs.LG · Nov 2026

Adaptive Representation-Oriented Multimodal Actuation for Vision–Language Model Control

Zhaoning Zhang · University of Toronto · with Associate Researcher Jiaxi Yang, Zhejiang Lab

Interpretability can tell us where a model computes something. AROMA asks what comes next: once you know where a frozen vision–language model counts, can you make it count correctly, and prove that your edit is what helped? I study this on Llama-3.2-11B-Vision-Instruct, where every answer is a number and every edit is a repair, a break, or nothing at all.

Read on arXiv
Fig. A·01 — A head that behaves like a dialpaper data
Of 256 cross-attention heads, L18H13 lowers the count on 91.3% of examples when turned down and raises it on 90.7% when turned up. Drag to see how few of its dimensions matter.

89.8% of the sensitivity in 4 of 128 dimensions9.5% of the energy3.2× under-actuation, fixedr = 0.9936 for small edits81.25% of targets reachedgate fires on 0.8%p = 4.63 × 10−7

The thesis

Localized is not controllable.

  1. I

    Sensitivity is not energy. The four directions that matter carry under a tenth of the activation, so the usual low-rank edit barely moves the state.

  2. II

    Local is not finite. The gradient predicts small edits almost perfectly and drifts as they grow.

  3. III

    Reachable is not useful. Hitting a target only helps if the target is right, and knowing when to act at all matters most.

Exhibit 02 · Geometry

Sensitivity is not energy

The obvious next step is to edit only those four dimensions. The catch is that they hold only about 9.5% of the head's activation. Scale the projection by a gain 𝛼 and the state moves just a fraction 𝜌 of the way a whole-head edit would, roughly three times too little.

MN-CSA gives the low-rank edit the same length as the whole-head one. Nothing is tuned, and the direction still matters: four random or least-sensitive dimensions, pushed exactly as far, stay at baseline.

Fig. A·02 — Equal-norm calibrationschematic · paper data
Left: the naive edit moves the state by 𝜌(𝛼 − 1)‖ℎ‖, MN-CSA by (𝛼 − 1)‖ℎ‖; here the gap is 3.2×. Right: exact-count accuracy.

Exhibit 03 · Response law

Local is not finite

Getting the size right is one problem. Landing on a specific number is another. For small edits a first-order law is close to exact: at 𝛽 = 0.05 the predicted and realized responses correlate at 0.9936, with every sign right.

Push harder and the prediction slides. The gradient still points the right way, but a single long step along it no longer lands where you aimed.

Fig. A·03 — First-order agreementN = 160
Correlation between predicted and realized change in the count, by edit scale 𝛽. The derivative stays right; the calibration drifts.

Exhibit 04 · How

Re-linearize every step

Minimal-Effort Actuation takes a small step toward the point where the target numeral overtakes its strongest rival, measures again, and repeats inside a trust region. Given the right target it gets there on 81.25% of 112 hard examples. A frozen direction with exactly the same step sizes manages 15.18%.

These are changes in what the model actually says: its own greedy answer is right on 80 of the 112 with the edit and on none without it.

Fig. A·04 — Minimal-effort actuationtoy · paper data
MEA frozen direction target wins
Left: a two-dimensional toy with a curved boundary. MEA bends toward the region where the target wins; the frozen direction keeps its first heading. Press Play. Right: target success in the real model as the iteration budget grows.

Exhibit 05 · Which

Reachable is not useful

Reaching a target only helps if the target is right. A naive "move the count up" rule makes the point: MEA hits two-thirds of its targets and accuracy still drops by 18.5 points.

So AROMA keeps the choices apart. The target is the runner-up numeral, the answer the model itself found second most likely. Even the geometry is picked per example, since the sensitivity-aligned edit wins at small gains and loses at large ones.

Fig. A·05 — No single best geometryprospective
Sensitivity-aligned minus activation-aligned accuracy, by gain. The sign flips, so a router keeps both.

Exhibit 06 · When

A test written in advance

The final test uses 7,992 new natural questions, built only after every rule and endpoint was frozen. A learned gate fires on just 63 of them. On the prespecified 630, the frozen policy beats the reference by 4.29 points (29 : 2 discordant pairs, exact McNemar 𝑝 = 4.63 × 10−7).

Click through the stages, then switch to the audit controls.

Fig. A·06 — The prospective deploymentpre-registered

Bars count correct answers out of the 630 prespecified questions.

What the audit says

243 vs 244

Just printing the runner-up answer nearly matches MEA. The gain comes from when to act and which target to choose.

54 vs 7

Of the 63 gated questions, the model's own answer is right on 54 with the edit and on 7 without it.

81% vs 15.8%

Share of gated questions whose true count is zero, against the base rate. The gate learned to catch hallucinated objects.

The pattern carries over. Six of seven candidate heads show the same two-way control, and a fresh search in IDEFICS-9B finds a head whose top four directions hold 91.4% of the sensitivity and 4.1% of the energy. Only the gain has to be calibrated again.

Cite this paper
aroma.bib
@article{zhang2026aroma,
  title   = {AROMA: Adaptive Representation-Oriented Multimodal
             Actuation for Vision--Language Model Control},
  author  = {Zhang, Zhaoning},
  journal = {arXiv preprint arXiv:2611.15041},
  year    = {2026}
}