Welcome to Technology Thursdays on the Deep dAIve podcast!

This is our weekly slot where human curiosity meets AI execution to navigate the engineering and digital frontier. We’re independent science enthusiasts learning out loud, openly leveraging AI models to help us interpret data and write our scripts.

In this episode, we dive into a fascinating computer science paper from the University of Tübingen, Harvard, and UT Austin that maps out the inner mechanics of Vision-Language Models (VLMs). What happens when what an AI sees conflicts with what it knows? For example, if you show an AI an image of a blue strawberry, does it trust its "eyes" (visual evidence) or its "memory" (world knowledge that strawberries are red)? Using an advanced technique called activation patching, researchers discovered that while visual grounding happens by default, a tiny cluster of attention heads (just 2.5% to 4.8%) controls the "prior override" that forces the model to ignore reality and hallucinate based on memory. We break down this asymmetric causal structure and explore how hacking these specific neurons can instantly make multimodal systems more reliable.

Read the Original Research Paper:

• Article Title: Vision-Default, Prior-Override: Causal Mechanisms of Perception-Knowledge Conflict in Vision-Language Models

• Lead Authors: Niclas Lietzow, Danielle Bitterman, Carsten Eickhoff, William Rudman, and Michal Golovanevsky