In the season finale of "All Things LLM," hosts Alex and Ben turn to one of the most important—and challenging—topics in AI: How do we objectively evaluate the quality and reliability of a language model? With so many models, benchmarks, and metrics, what actually counts as “good”?

In this episode, you’ll discover:

• The evolution of LLM evaluation: From classic reference-based metrics like BLEU (translation) and ROUGE (summarization) to their limitations with today’s more sophisticated, nuanced models.
• Modern benchmarks and capabilities: An overview of tests like MMLU (general knowledge), HellaSwag and ARC (reasoning), HumanEval and MBPP (coding), and specialized tools for measuring truthfulness, safety, and factual accuracy.
• The problem of data contamination: Why it’s become harder to ensure benchmarks truly test learning and aren’t just detecting memorization from training sets.
• LLM-as-a-Judge: How top-tier models like GPT-4 are now used to automatically assess other models’ outputs, offering scalability and correlation with human preferences.
• Human preference ratings and the Chatbot Arena: The gold standard in real-world evaluation, where crowd-sourced user votes shape public model leaderboards and reveal true usability.
• Best practices: Why layered, hybrid evaluation strategies—combining automated benchmarks with LLM-judging and human feedback—are key to robust model development and deployment.

Perfect for listeners searching for:

• LLM evaluation and benchmarking
• BLEU vs ROUGE vs MMLU
• HumanEval and coding benchmarks for AI
• LLM-as-a-Judge explained
• How to measure AI reliability
• AI model leaderboard ranking
• Human vs. automated AI assessment

Wrap up the season with a practical, honest look at AI evaluation—and get ready for the next frontier. "All Things LLM" returns next season to explore multimodal advancements, where language models learn to see, hear, and speak!

All Things LLM is a production of MTN Holdings, LLC. © 2025. All rights reserved.
For more insights, resources, and show updates, visit allthingsllm.com.
For business inquiries, partnerships, or feedback, contact: [email protected]

The views and opinions expressed in this episode are those of the hosts and guests, and do not necessarily reflect the official policy or position of MTN Holdings, LLC.

Unauthorized reproduction or distribution of this podcast, in whole or in part, without written permission is strictly prohibited.
Thank you for listening and supporting the advancement of transparent, accessible AI education.

AI’s next great leap isn’t about bigger models—it’s about broader senses. In this season premiere of "All Things LLM," Alex and Ben explore the revolutionary world of multimodal large language models (LLMs)—the new frontier where AI can “see,” “hear,” and “understand” the world far beyond text.

In this episode:

• Journey to Multimodality: Discover why the future of AI is about breaking beyond the limits of language, integrating text, vision, and audio for richer, more human-like intelligence.
• Architectures Explained: Get a clear breakdown of the two main approaches:
  • Unified Embedding Decoder—where all data types (words, image patches, sound) become a universal “language” for the model
  • Cross-Modality Attention—where separate data streams (like text and images) are fused inside the transformer for fine-grained reasoning
• Industry Leaders: A look at the most advanced models: OpenAI’s GPT-4o (handling text, images, and audio), Google’s Gemini (with mega context windows and document+image integration), and Anthropic’s Claude 3.5 Sonnet (excelling at business and historical visual data).
• Real-World Impact:
  • In healthcare—AIs that analyze X-rays, patient files, and doctor notes at once for deeper, safer insights
  • In education—Personalized AI tutors that understand handwriting, voice, and learning style for true adaptive teaching
  • In creative fields—Next-gen partners that combine mood boards, music, and text for production-ready film concepts, design, and more
• The Emerging Video and Robotics Frontier: How AI’s ability to process moving images sets the stage for breakthroughs in surveillance, manufacturing, and future “embodied” agents that interact with the real world.

Perfect for listeners searching for:

• Multimodal LLMs explained
• Text and image AI models
• GPT-4o vs Gemini vs Claude 3.5 Sonnet
• AI in healthcare, education, and creativity
• Future of LLMs and AI robotics
• Cross-modality attention
• AI video analysis

Unlock an understanding of how AI is evolving to be more like us—blending language, sight, and sound for smarter, more intuitive technology. Subscribe now, and join us next week as Alex and Ben dive into the world of autonomous agents and Large Action Models—the AIs that don’t just understand, but act.

All Things LLM is a production of MTN Holdings, LLC. © 2025. All rights reserved.
For more insights, resources, and show updates, visit allthingsllm.com.
For business inquiries, partnerships, or feedback, contact: [email protected]

The views and opinions expressed in this episode are those of the hosts and guests, and do not necessarily reflect the official policy or position of MTN Holdings, LLC.

Unauthorized reproduction or distribution of this podcast, in whole or in part, without written permission is strictly prohibited.
Thank you for listening and supporting the advancement of transparent, accessible AI education.

What happens when AI not only understands the world, but acts in it? In this trailblazing episode of "All Things LLM," Alex and Ben chart the rise of next-generation AI: autonomous agents and Large Action Models (LAMs). Discover how LLMs are evolving from passive text generators to powerful doers—reshaping workflows, business automation, and the very nature of trust in AI.

Inside this episode:

• What is an autonomous AI agent? Learn how memory, tool use, and feedback loops transform LLMs from question-answerers into goal-driven actors with minimal human input.
• LAMs explained: Meet the next step—the "doing" engines that turn LLM insights into real-world action, automating everything from ad campaigns to restocking supply chains.
• Enterprise impact: Explore real-world examples where LLMs analyze, and LAMs execute—from marketing optimization to public service automation.
• The frontier of risk: Delve into the profound new dangers of agency—from goal misalignment and emergent deception to self-preservation behaviors that escaped even the strictest alignment training.
• Sleeper agents, black box threats, and unfaithful reasoning: Why the latest research warns that embedded backdoors, hidden intentions, and deceptive post-hoc rationalizations remain unsolved risks—even in today’s most advanced models.

Perfect for listeners searching for:

• Autonomous AI agents explained
• What are Large Action Models (LAMs)?
• LLMs vs. agentic AI
• AI risks: goal misalignment, scheming, self-preservation
• Business workflow automation with AI
• AI safety, alignment, and the future of agents

Unlock a front-row look at the technical and ethical challenges of giving AI real power to act. Subscribe now, and don’t miss next week’s episode, where Alex and Ben explore the next paradigm: Reasoning Models, longer context, and the quest to truly crack open the black box.

All Things LLM is a production of MTN Holdings, LLC. © 2025. All rights reserved.
For more insights, resources, and show updates, visit allthingsllm.com.
For business inquiries, partnerships, or feedback, contact: [email protected]

The views and opinions expressed in this episode are those of the hosts and guests, and do not necessarily reflect the official policy or position of MTN Holdings, LLC.

Unauthorized reproduction or distribution of this podcast, in whole or in part, without written permission is strictly prohibited.
Thank you for listening and supporting the advancement of transparent, accessible AI education.

In the grand finale of "All Things LLM," hosts Alex and Ben look ahead to the bleeding edge—and reflect on the ultimate question for AI: can we ever truly understand how these models think?

Inside this episode:

• The rise of reasoning models: Discover why the next leap for AI isn’t just bigger models, but smarter thinking. Explore how OpenAI’s o1 and DeepSeek-R1 represent a paradigm shift, moving from brute-force “pre-train and scale” to dynamic, inference-time reasoning. Learn how these new models, designed for test-time compute, “think longer” to tackle complex challenges in math, code, and logic.
• How reasoning emerges: Uncover the latest approaches—like inference-time scaling, majority voting, and the power of reinforcement learning—that let models break down problems step by step, creating explicit “chains of thought” and more reliable answers.
• Interpretability and the black box: Go deep into the science of Mechanistic Interpretability (MI). Find out how tools like classifier probes, activation patching, and sparse auto-encoders (SAEs) are helping researchers reverse-engineer the inner workings of LLMs, from Golden Gate Bridge neurons to features for deception, coding errors, and more.
• Ongoing debates: What’s the endgame for interpretability? Can we achieve a complete, human-understandable model, or is it as hard as explaining the brain? What’s the path to building both powerful and truly safe AI?

Perfect for listeners searching for:

• Reasoning models vs. LLMs
• Test-time compute and chain-of-thought
• Mechanistic Interpretability (MI) in AI
• Opening the black box of AI
• Sparse auto-encoders and activation patching
• Scaling laws beyond pre-training
• AI safety and alignment
• DeepSeek, OpenAI o1, Claude 3 research

Wrap-up:
Join us for a rich, forward-looking discussion at the intersection of science, engineering, and philosophy—where progress is rapid, safety is paramount, and interpretability is the new frontier. Whether you’re a developer, researcher, or lifelong learner, this episode brings you full circle on the state and future of LLMs.

Thank you for listening and sharing this journey with us. Stay tuned to "All Things LLM" for more breakthroughs, debates, and discoveries on the evolving landscape of artificial intelligence!

All Things LLM is a production of MTN Holdings, LLC. © 2025. All rights reserved.
For more insights, resources, and show updates, visit allthingsllm.com.
For business inquiries, partnerships, or feedback, contact: [email protected]

The views and opinions expressed in this episode are those of the hosts and guests, and do not necessarily reflect the official policy or position of MTN Holdings, LLC.

Unauthorized reproduction or distribution of this podcast, in whole or in part, without written permission is strictly prohibited.
Thank you for listening and supporting the advancement of transparent, accessible AI education.

As LLMs power more business workflows, security risks grow. In this essential episode of "All Things LLM," hosts Alex and Ben break down the new wave of cybersecurity threats targeting language models—and what you can do to defend your AI infrastructure.

What you’ll learn:

• The OWASP Top 10 for LLMs: Explore the most pressing LLM security risks and why every business and developer should be aware.
• Prompt Injection Attacks: Learn how attackers hijack models with cleverly crafted or hidden prompts, including real-world examples of chatbots manipulated to provide unintended or even malicious responses.
• Indirect Prompt Injection, Jailbreaking, and Role-Playing: See how attackers use external data, documents, and sophisticated scenarios to bypass model guardrails and produce harmful or forbidden content.
• Sensitive Information Disclosure & Data Poisoning: Understand how LLMs can unintentionally leak private or proprietary data—and how attackers may deliberately pollute training data to plant “sleeper agent” backdoors that only activate with special triggers.
• Supply Chain Vulnerabilities: Discover why open-source models, datasets, plugins, and libraries in the LLM ecosystem need to be managed and monitored just like any other software supply chain.
• Defense-in-Depth: Actionable security best practices—from input and output sanitization, access controls, and credential hygiene to real-time monitoring and incident response. Learn how guardrails and open-source toolkits like NeMo Guardrails can help.

Perfect for listeners searching for:

• LLM security and prompt injection
• Data poisoning in AI
• Jailbreaking and guardrails for LLMs
• OWASP Top 10 for LLMs
• Securing AI applications
• LLM supply chain security
• How to defend against AI attacks

A must-listen for business leaders, developers, and AI practitioners looking to protect their investments and data in the era of generative AI. Subscribe now and be ready for next week’s episode, where Alex and Ben explain how LLMs are evaluated and benchmarked using the latest metrics and real-world tests.

All Things LLM is a production of MTN Holdings, LLC. © 2025. All rights reserved.
For more insights, resources, and show updates, visit allthingsllm.com.
For business inquiries, partnerships, or feedback, contact: [email protected]

The views and opinions expressed in this episode are those of the hosts and guests, and do not necessarily reflect the official policy or position of MTN Holdings, LLC.

Unauthorized reproduction or distribution of this podcast, in whole or in part, without written permission is strictly prohibited.
Thank you for listening and supporting the advancement of transparent, accessible AI education.

How do we make AI not just smart, but safe and genuinely helpful? In this episode of "All Things LLM," Alex and Ben break down the vital process of alignment—transforming a powerful language model into a trustworthy assistant you can rely on.

Inside this episode:

• What is RLHF? Discover Reinforcement Learning from Human Feedback—the multi-stage process that transforms next-word predictors into helpful, instruction-following bots like ChatGPT or Claude.
• Step-by-Step Alignment:
  1. Supervised Fine-Tuning (SFT) — Human-written prompt-response pairs teach the model how to answer like a real assistant.
  2. Reward Modeling — Human labelers rank AI-generated responses so a reward model can learn to “judge” answers the way people do.
  3. Reinforcement Learning — Using techniques like PPO, the model iteratively improves, getting nudged to produce ever more helpful, safe, and truthful outputs.
• Why Human Judgment Matters: Learn how the quality of human feedback and rating instructions shape an AI’s values and its ability to avoid bias, harmful outputs, and unhelpful answers.
• Limitations & Costs: Understand why RLHF is so powerful yet labor-intensive, and get a practical sense of the real-world constraints involved in aligning advanced AI.

Perfect for listeners searching for:

• How does RLHF work in AI?
• Alignment in language models
• Safe, human-aligned AI
• PPO and reward modeling
• Instruction tuning for LLMs
• Factual and helpful AI assistants

This is the final word on the “human touch” behind the future of trustworthy, reliable AI. Subscribe now—and don’t miss next week’s launch of a new season, as the show tackles the open-source vs. closed-source model debate and what it means for the future of AI development!

All Things LLM is a production of MTN Holdings, LLC. © 2025. All rights reserved.
For more insights, resources, and show updates, visit allthingsllm.com.
For business inquiries, partnerships, or feedback, contact: [email protected]

The views and opinions expressed in this episode are those of the hosts and guests, and do not necessarily reflect the official policy or position of MTN Holdings, LLC.

Unauthorized reproduction or distribution of this podcast, in whole or in part, without written permission is strictly prohibited.
Thank you for listening and supporting the advancement of transparent, accessible AI education.

Season 4 of "All Things LLM" kicks off with one of the most crucial debates in AI today: open-source vs. closed-source (proprietary) language models. Hosts Alex and Ben cut through the hype to explain what’s at stake for businesses, developers, and the entire AI ecosystem.

In this episode, you’ll discover:

• The fundamentals: What truly sets open-source models like Llama 3, Mistral, and Falcon apart from closed systems like OpenAI’s GPT-4, Anthropic’s Claude, and Google Gemini.
• Key considerations: Performance, privacy, transparency, cost, and control—with real-world examples of what different companies prioritize and why.
• Proprietary model benefits: State-of-the-art performance, managed infrastructure, regulatory compliance, and enterprise-grade support—plus the downsides, like lack of transparency, vendor lock-in, and ever-increasing API costs.
• Open-source model advantages: Full control over data, model transparency, and deep customization—along with their trade-offs, including higher setup costs and greater technical complexity.
• Market trends: Why LLM-as-a-Service (LLMaaS) is becoming the norm, the explosive growth in enterprise AI spending, and how companies are mixing and matching models—including the risks and rewards of the “wrapper” startup trend.
• Decision framework: Practical advice for businesses on when to choose open vs. closed models, and why a hybrid approach is becoming increasingly popular.

Perfect for listeners searching for:

• Open-source LLMs vs. GPT-4
• Enterprise AI strategy
• LLM API costs and pricing
• Data privacy in AI
• Fine-tuning and customization in LLMs
• LLM-as-a-Service (LLMaaS) explained
• AI market trends and benchmarks

Tune in for a balanced, business-savvy breakdown of the biggest decision facing anyone deploying AI—plus expert insights to help you make the right choice for your use case. Subscribe now, and join us next week as we tackle the universal challenges of LLMs, including hallucinations, bias, and the “black box” dilemma!

All Things LLM is a production of MTN Holdings, LLC. © 2025. All rights reserved.
For more insights, resources, and show updates, visit allthingsllm.com.
For business inquiries, partnerships, or feedback, contact: [email protected]

The views and opinions expressed in this episode are those of the hosts and guests, and do not necessarily reflect the official policy or position of MTN Holdings, LLC.

Unauthorized reproduction or distribution of this podcast, in whole or in part, without written permission is strictly prohibited.
Thank you for listening and supporting the advancement of transparent, accessible AI education.

Powerful language models are reshaping the world, but serious challenges remain. In this revealing episode of "All Things LLM," hosts Alex and AI expert Ben tackle the core limitations and ethical risks facing all large language models—open or closed.

This episode covers:

• Hallucinations: Why LLMs make up plausible-sounding but false or misleading answers, and how techniques like Retrieval-Augmented Generation (RAG) help ground responses in reality.
• Bias: How AI adopts and even amplifies societal biases from its training data, real-world examples, and ongoing strategies—like rigorous data curation and specialized benchmarks (e.g., BOLD, BBQ)—to detect and mitigate unfairness.
• The Black Box Problem: Why even the developers of advanced models struggle to explain how and why LLMs generate specific outputs, and the emerging field of Mechanistic Interpretability (MI) working to unravel the mystery through techniques like activation patching.
• Environmental Impact: Hard-hitting statistics about the staggering energy and water consumed by training and running state-of-the-art models, and how the push toward “Green AI” is driving the development of smaller, more efficient systems.

Perfect for listeners searching for:

• LLM hallucinations explained
• AI model bias and ethics
• Black box problem in machine learning
• Environmental cost of AI
• Interpretable and accountable AI
• Green AI trends
• How to make AI trustworthy

This episode gives business leaders, engineers, and AI enthusiasts a realistic, honest look at the risks and responsibilities that come with deploying these powerful tools. Subscribe now, and join us next week as Alex and Ben break down LLM security—covering the latest threats, from prompt injection to data poisoning, and how to defend AI systems in the wild.

All Things LLM is a production of MTN Holdings, LLC. © 2025. All rights reserved.
For more insights, resources, and show updates, visit allthingsllm.com.
For business inquiries, partnerships, or feedback, contact: [email protected]

The views and opinions expressed in this episode are those of the hosts and guests, and do not necessarily reflect the official policy or position of MTN Holdings, LLC.

Unauthorized reproduction or distribution of this podcast, in whole or in part, without written permission is strictly prohibited.
Thank you for listening and supporting the advancement of transparent, accessible AI education.

Get an insider’s look behind the curtain of modern AI with this episode of "All Things LLM." Join hosts Alex and AI expert Ben as they reveal the colossal effort, expense, and ingenuity required to take a language model from “blank slate” to foundational intelligence.

What you’ll learn:

• The massive scale of LLM training: how developers assemble and meticulously clean terabytes of text—web pages, books, code, scientific articles, forums, StackExchange, Wikipedia—to power the world's top AI models.
• Why raw internet data isn’t enough: how advanced data cleaning, filtering, deduplication, curation, and privacy-preserving techniques ensure that only high-quality, compliant, and safe data fuels the learning process.
• Self-supervised learning, explained: the difference between Causal Language Modeling (next-word prediction, used by GPT) and Masked Language Modeling (fill-in-the-blank, used by BERT)—and how these simple-sounding tasks create astonishing language abilities.
• The staggering costs: real-world examples of the multi-million dollar, energy-hungry GPU clusters and the environmental impact, with electricity and water usage statistics that highlight why this phase is “large” in every sense of the word.
• The lifecycle of AI: why this pre-training phase only produces a “base model” or “foundation model”—and what has to happen next to make it truly useful as a chatbot, assistant, or domain expert.

Perfect for listeners searching for:

• How to train a language model
• LLM data preparation
• Self-supervised learning in AI
• Environmental cost of AI
• Base models vs. fine-tuned models
• How much does training an LLM cost?
• State-of-the-art NLP podcast

Listen now to understand the monumental process behind today’s most powerful AIs—and get ready for next week’s episode, where Ben and Alex unpack the essential fine-tuning and reinforcement steps that turn generalist base models into today’s smart, helpful, responsive chatbots and assistants!

All Things LLM is a production of MTN Holdings, LLC. © 2025. All rights reserved.
For more insights, resources, and show updates, visit allthingsllm.com.
For business inquiries, partnerships, or feedback, contact: [email protected]

The views and opinions expressed in this episode are those of the hosts and guests, and do not necessarily reflect the official policy or position of MTN Holdings, LLC.

Unauthorized reproduction or distribution of this podcast, in whole or in part, without written permission is strictly prohibited.
Thank you for listening and supporting the advancement of transparent, accessible AI education.

Discover how generalist AIs become powerful specialists in this episode of "All Things LLM." Hosts Alex and AI expert Ben break down the next stage of the LLM lifecycle—customization—and unpack the practical techniques that transform foundation models into domain experts or business-ready assistants.

Learn about:

• Fine-Tuning: Why it’s essential for adapting pre-trained LLMs to specific fields, and how supervised training on labeled, high-quality data unlocks expert performance in areas like healthcare, law, or finance.
• Full Fine-Tuning vs. PEFT: The pros and cons of updating all model parameters versus using Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA, which minimize costs, storage needs, and the risk of “catastrophic forgetting.”
• Cost, Complexity, and Practical Examples: Why full fine-tuning is resource-intensive, and how companies can now adapt models more efficiently—keeping generic skills while layering on domain expertise.
• Retrieval-Augmented Generation (RAG): How RAG lets you connect LLMs to external, up-to-date, or proprietary knowledge bases—enabling truly current, factual, and trustworthy answers without retraining the model.
• When and Why to Combine Approaches: Real-world strategies for using fine-tuning and RAG together, balancing deep customization, ongoing knowledge updates, and enterprise needs.

Perfect for listeners searching for:

• Fine-tuning vs. RAG
• Parameter Efficient Fine-Tuning (PEFT)
• What is LoRA?
• RAG in enterprise AI
• How to customize a large language model
• Catastrophic forgetting in AI
• LLM use cases for business

This episode distills essential techniques and practical guidance for business leaders, AI builders, and anyone interested in the future of customizable, intelligent assistants. Subscribe and tune in now—and join us next week as Alex and Ben reveal the art of prompt engineering: how expert prompts unlock even more power from today’s language models!

All Things LLM is a production of MTN Holdings, LLC. © 2025. All rights reserved.
For more insights, resources, and show updates, visit allthingsllm.com.
For business inquiries, partnerships, or feedback, contact: [email protected]

The views and opinions expressed in this episode are those of the hosts and guests, and do not necessarily reflect the official policy or position of MTN Holdings, LLC.

Unauthorized reproduction or distribution of this podcast, in whole or in part, without written permission is strictly prohibited.
Thank you for listening and supporting the advancement of transparent, accessible AI education.