An image is not worth a thousand words - it's worth an indefinite number of them. So why do the metrics we use to evaluate AI-generated image descriptions still assume there's one correct answer?

In this episode of Women in AI Research, I talk with Elisa Kreiss (Assistant Professor of Communication at UCLA, director of the Coalas Lab) about what happens when you actually test the metrics the field relies on, and why CLIPScore, one of the most widely used measures for scoring image descriptions, stops correlating with human judgment the moment you introduce context. We also get into why longer descriptions aren't necessarily more informative, what happens when you just ask a model to "be concise," and whether AI models trip over charts and graphs the same way humans do.

Elisa's research sits at the intersection of linguistics, accessibility, and multimodal AI, and this conversation covers the full arc of her work, from the theoretical question of why humans never describe images the same way twice, to the practical question of what that means for building systems that actually work for blind and low-vision users.In this episode:

• Why "context matters" is more radical than it sounds for image description evaluation
• The hidden reason CLIPScore breaks down once context enters the picture
• Why length is a bad proxy for information density - and what to use instead
• What happens when you prompt a model to just "be concise"
• Why charts and photos need completely different evaluation approaches
• Whether AI models make the same mistakes as humans when reading data visualizations
• What NeurIPS's Top Reviewer Award taught her about writing a genuinely useful peer review

Resources & Links:

• Context Matters for Image Descriptions for Accessibility: Challenges for Referenceless Evaluation Metrics
• When More Words Say Less: Decoupling Length and Specificity in Image Description Evaluation
• CHART-6: Human-Centered Evaluation of Data Visualization Understanding in Vision-Language Models

🎧 Subscribe to stay updated on new episodes spotlighting brilliant women shaping the future of AI.

Follow WiAIR at:

• ⁠⁠LinkedIn⁠⁠
• ⁠⁠Bluesky⁠⁠
• ⁠⁠X (Twitter)⁠⁠
• ⁠⁠⁠WiAIR website⁠

Only 19% of Americans say AI has actually improved their productivity - so why the gap between the hype and reality?

In this episode of Women in AI Research, Dr. Malihe Alikhani (Northeastern University, Contextual AI Lab) unpacks the hidden failures in how we build and deploy AI: why sycophancy is really a collapse of alignment, why bigger models aren't better aligned, and why "thin" alignment breaks down in the real world.

Key topics

• The impact of moving across different AI contexts on system design
• The role of language as performative and active in shaping reality
• Interactive inference and uncertainty in AI systems
• The importance of context in meaning and system design
• AI policy, transparency, and societal impact
• Sycophantic behavior in large language models
• Measuring AI alignment: thin vs. thick
• AI adoption across sectors and demographic groups
• The role of policy in AI development and safety
• Ethical considerations in AI research and deployment

Resources & Links:

• Breaking the AI Mirror: Sycophancy, productivity, and the future of collaboration
• Hype and harm: Why we must ask harder questions about AI and its alignment with human values
• How are Americans using AI? Evidence from a nationwide survey

Connect with Dr. Malihe Alikhani:

• https://x.com/malihealikhani

🎧 Subscribe to stay updated on new episodes spotlighting brilliant women shaping the future of AI.

Follow WiAIR at:

• ⁠LinkedIn⁠
• ⁠Bluesky⁠
• ⁠X (Twitter)⁠
• ⁠⁠WiAIR website⁠

Do large language models truly understand language—or are they sophisticated pattern matchers?

In this conversation, Dr. Anna Ivanova (Asst. Prof. at Georgia Tech) explores one of the important questions in AI: the relationship between language, thought, and intelligence. Drawing from neuroscience, cognitive science, and AI research, Anna explains why language understanding is harder to define than most people realize, why reasoning and language are not the same thing, and what today's LLMs can and cannot tell us about human cognition.

Key Topics:

• Do LLMs understand language or merely generate convincing text?
• The difference between formal and functional linguistic competence
• What LLMs can learn from language alone—and what they cannot
• Why human cognition and AI cognition may be fundamentally different
• Theory of mind, reasoning, and common misconceptions about AI capabilities
• How cognitive scientists evaluate the "thinking" abilities of LLMs
• What neuroscience can teach AI researchers about interpretability
• Why understanding AI requires studying both behavior and internal representations
• The future of multimodal models and AI cognition

Resources & Links:

• What does it mean to understand language?
• Dissociating language and thought in large language models
• How to evaluate the cognitive abilities of LLMs
• How Do LLMs Use Their Depth?
• True Lens

Connect with Dr. Anna Ivanova:

https://bsky.app/profile/neuranna.bsky.social

https://x.com/neuranna

🎧 Subscribe to stay updated on new episodes spotlighting brilliant women shaping the future of AI.

Follow WiAIR at:

• LinkedIn
• Bluesky
• X (Twitter)
• ⁠WiAIR website⁠

What actually happens when AI systems fail in the real world?

In this final part of our conversation with Saadia Gabriel (UCLA), we unpack one of the most urgent challenges in modern AI: why even the most advanced models remain vulnerable to manipulation - and what that means for safety, fairness, and society.

From multi-turn jailbreaking attacks with near 100% success rates to misinformation shaping human beliefs, this conversation goes beyond surface-level concerns and dives into how harms actually emerge in deployed systems.

We explore:

• Why current guardrails are not enough
• How realistic attack scenarios differ from academic benchmarks
• The connection between model vulnerabilities and societal harm
• What AI can (and cannot) do about misinformation and persuasion
• The open research problems that still don’t have solutions

Resources & Links:

• Generative AI in the Era of 'Alternative Facts'
• ModelCitizens: Representing Community Voices in Online Safety
• Translation as a Scalable Proxy for Multilingual Evaluation

Connect with Dr. Saadia Gabriel:

• https://x.com/GabrielSaadia
• https://bsky.app/profile/skgabrie.bsky.social

What does it mean to build AI systems we can actually trust?

In this first part of our conversation with Saadia Gabriel (UCLA), we explore the deeply personal and technical journey behind her work on AI safety, misuse, and responsible NLP.

From experiencing targeted hate speech firsthand to receiving a best paper nomination, Saadia shares how her lived experience shaped her research — and why language models must be designed with both capability and risk in mind.

🧠 In this episode, we cover:

• How personal experiences influence AI research directions
• The intersection of NLP, security, and privacy
• Why LLMs can be both powerful and dangerous
• What it means to build trustworthy AI systems
• Lessons from working across multiple research paradigms
• How to pursue high-impact research as a PhD or early-career scientist

Resources & Links:

• X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents

Connect with Dr. Saadia Gabriel:

• https://x.com/GabrielSaadia
• https://bsky.app/profile/skgabrie.bsky.social

What does it actually mean for a model to understand audio

Paper: https://arxiv.org/abs/2601.19673

In this episode, I talk with Iwona Christop, a PhD student at Adam Mickiewicz University, about her recent EACL paper introducing ART (Audio Reasoning Tasks) — a new benchmark designed to evaluate whether multimodal LLMs can truly reason over audio, not just transcribe or classify it.

Most existing benchmarks test audio skills in isolation (like ASR or classification). But real-world intelligence requires something deeper: combining signals, comparing sounds, tracking context, and making decisions.

This work takes a different approach:

• No text-only shortcuts — tasks can’t be solved via transcription alone
• Reasoning-first design — models must combine multiple audio cues
• No expert knowledge required — anyone can verify correctness

We also dive into the diverse task design, including:

• Audio arithmetic (counting and comparing sounds)
• Cross-recording speaker & language identification
• Sound-based reasoning (e.g., inferring properties from audio)
• Speech feature comparison (accents, variations)
• Multimodal reasoning across text and sound

The dataset includes 9 tasks, 9,000 samples, and 30+ hours of audio — all generated in a scalable way using templates and TTS.

👉 If you care about multimodal reasoning, evaluation, or the limits of current LLM capabilities, this conversation is for you.

Iwona Christop:

https://www.linkedin.com/in/iwona-christop/

👍 Like & subscribe for more deep dives into cutting-edge AI research

🔔 New episodes from EACL 2026 coming soon

#WiAIR #EACL2026

LLM-based agents are everywhere, but most research focuses on just one step: getting the model to call the right tool. What happens after that?

Paper: https://arxiv.org/abs/2510.15955

In this talk, Kiran Kate (IBM Research) presents new findings from their EACL 2026 paper on a largely overlooked problem:👉 Can LLMs actually understand and use the outputs returned by tools?

As tool-augmented systems become more complex, this question becomes critical. The work dives into how current models handle non-trivial, real-world tool responses, and where they break down.

💡 Key ideas covered:

• Why tool calling is only half the story in LLM agents
• The challenge of processing complex tool outputs
• Failure modes in current LLM-based systems
• What this means for building robust, real-world AI agents

This talk is especially relevant if you're working on:

• LLM agents and tool use
• Evaluation of LLM capabilities
• Real-world deployment of AI systems
• Agentic workflows and reasoning pipelines

Kiran Kate:

• https://www.linkedin.com/in/kiran-kate-8b98672/

👍 Like & subscribe for more deep dives into cutting-edge AI research

🔔 New episodes from EACL 2026 coming soon

In this episode of #WiAIRpodcast, we dive into a subtle but critical question: Does adding reasoning actually make LLMs safer and more reliable?

Paper: https://arxiv.org/abs/2510.21049

Atoosa Chegini (University of Maryland, Apple) presents Reasoning's Razor (EACL 2026), where she and her collaborators examine how reasoning impacts high-stakes binary classification tasks, including safety filtering and hallucination detection.

Their findings highlight an important nuance:

• While reasoning can improve overall accuracy, it may degrade performance at low false positive rates -- exactly where real-world systems need to operate.

This conversation covers:

• Why accuracy is a misleading metric for safety-critical LLM applications
• The importance of evaluating models at fixed false positive rates (FPR)
• How two models with identical accuracy can behave completely differently in deployment
• The impact of "think-on" (with reasoning) vs "think-off" (no reasoning) settings
• Practical implications for RLHF, SFT, and post-training pipelines

If you're working on:

• LLM evaluation & reliability
• AI safety or hallucination detection
• Production deployment of language models

— this discussion offers a perspective that is both technically grounded and immediately actionable.

Atoosa:

• https://www.linkedin.com/in/atoosa-chegini-6713741a3/
• https://scholar.google.com/citations?user=5nY9tagAAAAJ&hl=en&oi=ao

👍 Like & subscribe for more deep dives into cutting-edge AI research

🔔 New episodes from EACL 2026 coming soon

LLMs are notoriously overconfident, but can we teach them to admit uncertainty?

In this episode, Maor Juliet Lavi (Tel Aviv University) presents her EACL 2026 paper on Detecting Unanswerability in Large Language Models with Linear Directions.

Paper: https://arxiv.org/abs/2509.22449

We cover:

• Why prompt-based fixes for hallucinations aren’t enough
• How “unanswerability” emerges inside model representations
• A simple but powerful idea: linear directions in hidden states
• Why this method generalizes better across datasets
• What this reveals about where abstract concepts live inside LLMs

👉 Watch to see how far we can push models toward knowing when they don’t know.

Maor:

• https://www.linkedin.com/in/maor-juliet-lavi-07494a155

👍 Like & subscribe for more deep dives into cutting-edge AI research

🔔 New episodes from EACL 2026 coming soon

LLMs often give different answers to the same question, just phrased differently. But how do we measure and understand this behaviour rigorously?

In this episode of the #WiAIRpodcast, Cléa Chataigner (Mila, McGill) presents AUGMENT, a user-grounded, controlled paraphrasing framework for auditing prompt sensitivity in large language models, accepted as an oral at EACL 2026.

Paper: https://arxiv.org/abs/2505.03563

Instead of relying on noisy, unconstrained paraphrasing, AUGMENT introduces:

• Structured, linguistically grounded paraphrase types (e.g., voice, style, dialect)
• A guided generation + automated quality control pipeline
• Fine-grained analysis of how specific linguistic variations impact model behaviour

Key insights:

• Different paraphrase types can shift model performance in opposite directions
• Standard baselines can hide critical failure modes
• Even strong LLMs struggle with certain transformations (e.g., voice changes)

Evaluated across:

• Bias benchmarks (BBQ)
• Knowledge tasks (MMLU)
• Multiple open-source model families

Cléa:

https://scholar.google.com/citations?user=NdToDmMAAAA

👍 Like & subscribe for more deep dives into cutting-edge AI research

🔔 New episodes from EACL 2026 coming soon