[00] — Index Toronto · 43.66° N, 79.40° W

Zhaoning Zhang

I study how vision–language models represent what they see — and how to control those representations, precisely and verifiably.

Study
Computer Science Specialist, University of Toronto
Lab
Research Assistant, Zhejiang Lab
Fig. 00 — Steering field Live
Move to apply a local, bounded intervention.
[01] — Research Mechanism → Control

Models already contain the mechanisms behind their behaviour. My work is to find them, steer them, and prove that the steering is real.

A · Locate

Find the causal component.

Activation intervention and gradient-based sensitivity analysis to pinpoint where a behaviour lives — down to a single cross-attention head and a low-dimensional subspace inside it.

B · Control

Design the intervention.

Magnitude-normalized subspace actuation, iterative trust-region controllers, and selective gates that decide when to act — all on frozen models, without retraining.

C · Verify

Hold every claim to account.

Paired exact McNemar tests, bootstrap confidence intervals, rank-matched controls, and untouched evaluation cohorts frozen before any result is seen.

Interests
  • Vision–Language Models
  • Multimodal Machine Learning
  • Large Language Models
  • Representation Learning & Model Control
  • Efficient Adaptation & Inference
  • Deep Learning Systems
  • Data-Centric & Reliable AI
[02] — Preprint I · Exhibit arXiv:2611.15041 · cs.LG · Nov 2026

Adaptive Representation-Oriented Multimodal Actuation for Vision–Language Model Control

Zhaoning Zhang · University of Toronto · with Associate Researcher Jiaxi Yang, Zhejiang Lab

Interpretability can tell us where a model computes something. AROMA asks what comes next: once you know where a frozen vision–language model counts, can you make it count correctly, and prove that your edit is what helped? I study this on Llama-3.2-11B-Vision-Instruct, where every answer is a number and every edit is a repair, a break, or nothing at all.

Fig. A·01 — A head that behaves like a dialpaper data
Of 256 cross-attention heads, L18H13 lowers the count on 91.3% of examples when turned down and raises it on 90.7% when turned up. Drag to see how few of its dimensions matter.

89.8% of the sensitivity in 4 of 128 dimensions9.5% of the energy3.2× under-actuation, fixedr = 0.9936 for small edits81.25% of targets reachedgate fires on 0.8%p = 4.63 × 10−7

[03] — Preprint II · Exhibit arXiv:2611.21539 · Nov 2026

Seen but Unsaid

Locating and Repairing Cross-Image Comparison in Vision–Language Models

Zhaoning Zhang · University of Toronto

Show a vision–language model two images, mark a point in the first, and ask which of four points in the second is the same one. It often gets this wrong, even when the right answer can be read from its own hidden states. I follow the signal from the pixels to the answer letter in Qwen3-VL (2B to 32B), InternVL3.5-8B and Gemma-3-12B to find where, between seeing and saying, the match goes missing.

Fig. B·01 — Four readouts, Qwen3-VL-8Bpaper data
Visual tokens—region readout over answer probe, 𝑃 − 𝑅3
Comparison—selectivity of cross-image heads, 𝜌ref
Answer position—input-matched probe, 𝑅3
Output—calibrated answer, 𝑅4aff
Solid borders mark a signal that is carried, dashed ones a signal that is lost.

probe .64, model .43visual tokens +36 pts over the answer probe16–32 heads that compare−17 to −22 pts when they are scrambled+28.3 pts from 0.02% of the parameters21–51% of the gain by transplant

[04] — Experience Research · Industry

Where the work happened.

  1. May 2026 — Present

    Research Assistant · Zhejiang Lab

    Supervised by Associate Researcher Jiaxi Yang. Built AROMA end to end: reproducible Python, PyTorch and Hugging Face pipelines for multimodal inference, activation intervention, gradient-based sensitivity analysis and statistically controlled candidate evaluation.

    Remote
  2. Jun — Aug 2026

    Machine Learning Intern · Hundsun Technologies

    Enterprise LLM-RAG customer-service platform: domain-aware semantic retrieval, agent–human handoff for ambiguous and long-tail queries, parameter-efficient fine-tuning for local deployment, and multidimensional metrics for conversational quality. Worked across ML, software and DevOps teams to ship GPU-accelerated services in containers.

    Hangzhou
  3. May 2026

    Machine Learning Systems Intern · Xiong’an Xingyuan Technology

    Reproduced the KTransformers heterogeneous inference framework with DeepSeek-MoE-16B under Q2-K quantization on consumer GPUs; tuned expert offloading and placement and profiled throughput, VRAM and host-memory trade-offs.

    Xiong’an
[05] — Selected projects Systems that ship

Built, measured, deployed.

P.01Jun — Aug 2026

FineX Intelligent Customer-Service System

Hundsun Technologies

Modular AI service backend with Redis SETNX concurrency control, persistent dialogue state and streaming responses; an end-to-end RAG pipeline over ChromaDB; and QLoRA fine-tuning of Qwen2.5-7B-Instruct on 17,946 dialogues with 4-bit NF4 — containerized with GPU passthrough.

  • FastAPI
  • Redis
  • ChromaDB
  • QLoRA
  • Docker
P.02May — Jun 2026

AI-Powered KYC Compliance Agent

York ABMP Career-Fit Program

Agentic-RAG, multi-agent prototype for identity verification, financial-risk assessment and watch-list screening, with an auditable TCP Double-Lock for high-risk transactions under CIRO / FINTRAC constraints. Mentored by IBM AI SME Utkarsh Pandey; selected as an outstanding participant.

  • watsonx Orchestrate
  • Agentic RAG
  • Multi-agent
P.03Sep — Dec 2025

Account-Based Online Store

Full-stack web application

Account-driven e-commerce platform covering authentication, cart, checkout, promotions and order processing, with object-oriented domain modelling and validated REST APIs for addresses, coupons, transaction state and insufficient-fund edge cases.

  • Java
  • REST
  • HTML/CSS
  • Xano
[06] — Toolkit Daily instruments

From hypothesis to GPU.

ML & multimodal research

PyTorch · Hugging Face Transformers · NumPy · pandas · scikit-learn · causal intervention · activation steering · statistical testing

LLM training & inference

LoRA · QLoRA · PEFT · TRL · BitsAndBytes NF4 · MoE inference profiling · GPU inference optimization

Retrieval & agents

ChromaDB · semantic retrieval · retrieval-augmented generation · agentic RAG · multi-agent orchestration

Systems & languages

FastAPI · Pydantic · Redis · MySQL · WebSocket · Docker · Python · Java · C · JavaScript · Linux

[07] — Education & honours 2024 — 2028

University of Toronto

Bachelor of Computer Science, Computer Science Specialist

Aug 2024 — Expected Jun 2028 · Toronto, Ontario

  • AwardUniversity of Toronto International Scholar Award — Faculty of Arts & Science
  • HonourDean’s List — University of Toronto
[08] — Contact Toronto · Eastern Time

Let’s talk.

Interested in mechanistic interpretability, controllable multimodal models, or research collaboration — I’d be glad to hear from you.