Teaching Small AI Models to Look Before They Speak: LaViT Closes the Perception Gap in Multimodal Reasoning
LaViT is a training framework that makes compact vision language models inherit not just what a large teacher model says about an image, but where it looks. By distilling gaze and visual semantics into continuous latent visual reasoning tokens, a 3B model gains up to +16.9% on difficult spatial and geometric reasoning tasks. The result matters because students no longer fake visual understanding by leaning on language habits.
LaViT is a training framework that makes compact vision language models inherit not just what a large teacher model says about an image, but where it looks. By distilling gaze and visual semantics into continuous latent visual reasoning tokens, a 3B model gains up to +16.9% on difficult spatial and geometric reasoning tasks. The result matters because students no longer fake visual understanding by leaning on language habits.
![![Side by side diagram contrasting text only mimicry, which copies the teacher's words and answers wrongly, with LaViT latent visual thought alignment, which reproduces the teacher's visual gaze and answers correctly, on a kitchen depth question]](/news/img/lavit.JPG)
Why it matters
Multimodal reasoning models are increasingly used for diagrams, medical images, screenshots, charts and robots that must act on what they see. A model that answers fluently but never actually looks at the right pixels is a reliability risk, invisible in standard accuracy scores and dangerous in deployment. LaViT starts from a striking diagnostic finding: a distilled student can match its teacher's words almost perfectly while its internal visual attention wanders to entirely different parts of the image. The team calls this the Perception Gap, and shows that standard supervised fine tuning trains students to guess from language priors rather than observe.
The fix is elegant and model agnostic. Instead of matching static image embeddings, LaViT forces the student to autoregressively generate a short sequence of latent tokens that reconstruct the teacher's contextualised visual semantics and its attention trajectory, all before it writes a single word. A curriculum sensory gating schedule first closes the direct image path so the student must compress vision into the latent tokens, then reopens it as a residual perception channel so training matches inference. Because this only needs white box access to an open weight teacher such as Qwen2.5 VL 32B, any lab can adopt the recipe. The payoff: a 3B model outperforms Qwen2.5 VL 7B on five of seven benchmarks, edges out GPT 4o on relative depth (78.23 vs 64.52) and IQ Test (32.0 vs 30.0), and even shows steadier visual focus than the larger teacher it learned from.
Key findings
- Headline gain LaViT lifts Relative Depth accuracy by +16.94% over the base 3B model, reaching 78.23% on the BLINK benchmark.
- Geometry jump Relative Reflectance improves by +15.67% over the 3B baseline, pointing to stronger structural visual grounding.
- Beats GPT 4o The compact 3B model scores 78.23% on Relative Depth versus GPT 4o's 64.52%, and 32.0% on IQ Test versus 30.0%.
- Cross scale LaViT 3B outperforms the larger Qwen2.5 VL 7B on five of seven evaluated benchmarks.
- Fine grained perception MMVP accuracy rises 5.00 points to 67.33%, well ahead of DMLR at 61.33% and PAPO at 50.0%.
- Language prior check MMStar robustness rises to 54.07% from the 50.2% baseline, showing gains come from seeing, not from language habits.
- Perception Gap measured In the pilot study, correct answers maintained an average visual focusing score of 18.78% versus 9.18% for wrong ones, a gap of roughly 104%.
- Attention drift Under plain fine tuning the student's cosine distance to the teacher stays steady at 0.52 to 0.55, yet attention divergence peaks at a KL mean of about 1.39 on attribute words.
- Sharper focus Attention entropy on Relative Depth drops from 4.870 to 4.686, moving toward the teacher's 4.284 and away from a diffuse gaze.
- Steadier than teacher Variation in salient region count falls from 0.392 in the 32B teacher to 0.102 in the 3B student, a calmer and more decisive viewing strategy.
- Compact design Only four latent visual tokens are generated per response, and the 15K training set is filtered in three stages for correctness, difficulty and grounding.
- Ablation proof Removing curriculum sensory gating drops MMVP to 59.33%, and single stage training drags Relative Reflectance to 38.81% versus 45.52%.
A small model that looks where it should
- Up to +16.9% on complex spatial reasoning benchmarks
- 3B parameter student beats Qwen2.5 VL 7B on 5 of 7 tasks and edges out GPT 4o on two
- Lower attention variance than its own far larger teacher (0.102 vs 0.392)
What it means for practitioners
Teams building vision language model products on a budget can take the LaViT recipe seriously. It needs no proprietary data, no external tool calls and no reinforcement learning loop; just white box access to an open weight teacher, a small filtered distillation set of 15K tuples, and a two phase training schedule totalling roughly 1,000 steps. The paper reports that removing either the trajectory alignment loss or the semantic reconstruction loss causes consistent drops across every benchmark, and that masking the latent tokens at inference time degrades results sharply. That is a welcome sign of genuine reliance on the learned visual thoughts rather than a cosmetic add on, and it makes the method easy to validate in your own evaluation harness.
The engineering detail most worth copying is the attention bias gate. A permanent mask between response tokens and image patches creates a train and test mismatch; no mask at all lets the model shortcut around the latent tokens. Curriculum sensory gating interpolates between the two with a cosine schedule inside the attention bias, which removes the need for multi stage pipelines and dynamic loss weighting. The project has released code, models and data publicly, so practitioners can adapt the latent thought container idea to domain specific visual tasks such as document charts, defect inspection or surgical imagery, where the cost of fluent but ungrounded answers is highest.
Related reading
Get the paper & cite it
Official citation: Linquan Wu, Tianxiang Jiang, Yifei Dong, Haoyu Yang, Fengji Zhang, Shichang Meng, Ai Xuan, Linqi Song, Jacky W. Keung (2026). LaViT: Aligning Latent Visual Thoughts for Multi-modal Reasoning. Lecture Notes in Computer Science. DOI: 10.48550/arXiv.2601.10129.