AI leaps forward every week, but how do we cut through the noise and truly measure progress? This isn't just academic; it's fundamental to trusting and advancing AI. Forget marketing claims – this episode gives you the backstage pass to the essential field of LLM Evaluation, the engine driving genuine AI improvement.

As AI weaves into our lives, from automating tasks to creative endeavors, rigorously assessing its performance isn't a luxury—it's the bedrock of reliability. Why? Because you need to trust these systems before relying on them for anything important. We're diving headfirst into how experts put these powerful tools to the test, separating hype from genuine progress, without drowning you in technical jargon.

Think of LLM evaluation as the crucial compass guiding AI development. It reveals where models excel and, critically, where they still need to grow. This isn't just for developers fine-tuning models; it's for researchers proving new ideas, and for you, the end-user, to ensure the AI assistants you rely on are truly dependable.

In this episode, you'll discover:

• (02:42) The Three Pillars of AI Scrutiny: Unpack the core methods – Automatic Evaluation (computers judging computers), Human Evaluation (the 'gold standard' of expert opinion), and the fascinating LLM-as-Judge (AI evaluating AI).

• (03:01) Automatic Evaluation Unveiled: Understand how speed, scale, and predefined metrics (like Perplexity, BLEU, and ROUGE) offer rapid, cost-effective insights, and where they fall short in capturing nuance.

  • (04:37) Decoding Perplexity (PPL): How AI "surprise" measures language understanding.

  • (05:08) BLEU Score Explained: The machine translation metric now vital for text generation.

  • (06:15) ROUGE for Summarization: How we check if AI captures the gist.

• (07:02) Beyond Basic Metrics: Explore advanced automated tools like Meteor and BERTScore that aim for deeper semantic understanding.

• (09:20) The Human Touch: Why human judgment, despite its costs and complexities, remains indispensable for assessing fluency, coherence, and factual accuracy. Learn about direct assessment and pairwise comparisons.

• (11:34) When AI Judges AI: The pros and cons of using powerful LLMs to evaluate their peers – a scalable approach with its own set of biases to navigate.

• (13:58) What Makes a "Good" LLM?: The critical qualities we measure – from accuracy, relevance, and fluency, to crucial aspects like safety, harmlessness, bias, and even efficiency.

• (16:35) The AI Proving Grounds – Benchmark Datasets: Why standardized tests like GLUE, SuperGLUE, MMLU, Hellaswag, and HumanEval are essential for tracking true progress across the industry.

• (19:36) The Cutting Edge of Evaluation: Exploring the frontiers – how we're learning to assess complex reasoning, tool usage, instruction following, and the interpretability of AI decisions.

• (21:56) The Future is Holistic: Why comprehensive frameworks like HELM are emerging to provide a more complete picture of an LLM's capabilities and limitations.

Stop wondering if AI is actually improving and start understanding how we know. This knowledge is your key to leveling up your GenAI expertise, enabling you to build, use, and critique AI with genuine insight. This changes everything about how you see AI progress.

Large Language Models (LLMs) face significant challenges in effectively using a growing number of external tools, such as those defined by the Model Context Protocol (MCP). These challenges include prompt bloat and selection complexity. As the number of available tools increases, providing definitions for every tool in the LLM's context consumes an enormous number of tokens, risking overwhelming and confusing the model, which can lead to errors like selecting suboptimal tools or hallucinating non-existent ones.

To address these issues, the RAG-MCP framework is introduced. This approach leverages Retrieval-Augmented Generation (RAG) principles applied to tool selection. Instead of presenting all available tool descriptions to the LLM at once, RAG-MCP uses semantic retrieval to dynamically identify and select only the most relevant tools from an external index based on the user's query. Only the descriptions of these selected tools (or MCPs) are then passed to the LLM.

This process significantly reduces the prompt size and simplifies the decision-making required from the LLM. The framework's pipeline involves encoding the user's task input, submitting it to a retriever that searches a vector index of MCP schemas, ranking candidates, and optionally validating them, before the LLM executes the task using only the selected MCP's information.

Key benefits demonstrated by RAG-MCP include a drastic reduction in prompt tokens (cutting usage by over 50% compared to including all tools) and a significant boost in tool selection accuracy (tripling the success rate of baseline methods, achieving 43.13% compared to 13.62% for Blank Conditioning). The approach also leads to lower cognitive load for the LLM, resource efficiency by only activating selected MCPs, and multi-turn robustness. RAG-MCP enables scalable and accurate tool integration and remains extensible, as new tools can be added simply by indexing their metadata without needing to retrain the LLM.

In this episode, we dissect DeepSeek Prover V2, an open-source large language model pushing the boundaries of AI in formal theorem proving using Lean 4.

We unpack its innovative "cold start" training procedure, where the general-purpose DeepSeek-V3 is ingeniously used to generate initial training data by recursively decomposing complex problems into manageable subgoals.

Discover how this approach synthesizes informal, human-like mathematical intuition with the rigorous, step-by-step logic required for formal proofs.

We'll explore the architecture of the 671 billion parameter model, its two-stage training process creating distinct 'Chain-of-Thought' (CoT) and 'non-CoT' modes, and its state-of-the-art performance on challenging benchmarks like MiniF2F, PutnamBench, and the newly introduced ProverBench (which includes problems from AIME competitions). Learn about the significance of its recursive proof search, curriculum learning, and reasoning-oriented reinforcement learning, all aimed at bridging the gap between intuitive reasoning and formal mathematical verification.

Join us as we explore why DeepSeek Prover V2 represents a major stride in AI's ability to tackle complex mathematical logic.

Please also checkout our previous episode for DeepSeek V3 in YouTube, Spotify and Apple Podcast.

Quality Engineering (QE) professionals are well-positioned to transition into AI Improvement Engineering roles due to their deep knowledge of testing, quality assurance, and processes. This transition expands their role significantly – from assuring quality in products to improving the entire system that delivers them. The role demands augmenting traditional QE skillsets with knowledge of AI/ML fundamentals, MLOps, data analysis, and modern DevOps toolchains. Key technical skills include proficiency in AI/ML concepts and workflows, data handling, observability, and automation tools. QEs also need to develop strategic and soft skills, such as mastering continuous improvement methodologies like Lean Six Sigma and Toyota Kata, as well as leadership, coaching, and change management.

In the age of AI, the QE role is evolving to encompass new responsibilities like AI governance, ensuring data quality for AI models, and evaluating model performance and reliability. QEs are shifting towards more advisory and strategic positions, overseeing AI agents and verifying their results. This integration of AI into QE is intended to augment human capabilities, allowing engineers to focus on more strategic aspects while AI handles routine tasks. Emerging roles for transitioning QEs include AI Reliability Engineer, MLOps Engineer, Improvement Engineer, and Engineering Excellence Architect.

Ultimately, Improvement Engineering in AI is about creating a virtuous cycle, using data and AI to drive improvements, leading to better systems, all fueled by a culture that values growth and excellence. This requires cultivating a continuous improvement culture where quality is everyone's responsibility, promoting a blameless culture, and embracing a scientific mindset for experimentation. By following a roadmap of skill development and cultural change, QEs can reinvent themselves, ensuring organizations not only build the right things but also build things the right way, and keep getting better at it.

This episode delves into CaMeL, a novel defense mechanism designed to combat prompt injection attacks in Large Language Model (LLM) agents.

Inspired by established software security principles, CaMeL focuses on securing both control flows and data flows within agent operations without requiring changes to the underlying LLM.

We'll explore CaMeL's architecture, which features explicit isolation between two models: a Privileged LLM (P-LLM) responsible for generating pseudo-Python code to express the user's intent and orchestrate tasks, and a Quarantined LLM (Q-LLM) used specifically for parsing unstructured data into structured formats using predefined schemas, without tool access.

The system utilizes a custom Python interpreter that executes the P-LLM's code, tracking data provenance and enforcing explicit security policies based on capabilities assigned to data values. These policies, often expressed as Python functions, define what actions are permissible when calling tools.

We'll also touch upon the practical challenges and the system's iterative approach to error handling, where the P-LLM receives feedback on execution errors and attempts to correct its generated code.

Tune in to understand how this design-based approach leveraging dual LLMs, a custom interpreter, policies, and capabilities aims to build more secure LLM agents.

In this episode, we unearth the untold story of how Google engineered one of the most powerful information retrieval systems in history—and why its early design principles still echo through today’s cutting-edge AI.

From scavenged servers to hyper-optimized global systems, we follow Google's relentless pursuit of scale, speed, and precision, drawing lessons from Jeff Dean’s landmark 2009 WSDM talk. (video, slides)

You’ll hear how seemingly ancient struggles—handling billions of documents, lightning-fast retrieval, caching, real-time updates—directly mirror the modern battles of building Retrieval-Augmented Generation (RAG) systems and AI models today.

This isn't just nostalgia. It’s a playbook for the next generation of intelligent systems. Join us to connect the dots between the bold experiments of Google's early days and the challenges facing AI engineers right now—and tomorrow.

Chapters include:

• Why scaling breaks everything—and how to fix it

• Document vs. word partitioning wars

• Tricks with doc IDs and early stopping

• Birth of caching: the unsung hero of performance

• How a 10,000× speedup rewired web search

• Lessons from moving the entire index into RAM

• From Universal Search to Universal AI Knowledge

• The eternal race: real-time updates vs. a changing world

Big Idea:

The problems that built Google are the problems that will build the future of AI.

[Level Up With Gen AI Series] We believe in harnessing the power of AI to unlock human potential.

We unpack Bill Gurley’s legendary talk “Running Down a Dream”—a no-fluff, high-octane blueprint for building a career that truly lights you up. Through vivid stories of Bobby Knight, Bob Dylan, and Danny Meyer, Gurley reveals five timeless principles that separate the dreamers from the doers:

• Find your real passion (not your LinkedIn-approved one)

• Hone your craft with obsessive depth

• Seek out mentors—and earn their investment

• Embrace your peers as collaborators, not competitors

• Give credit, pay it forward, stay human

This isn’t just career advice. It’s a philosophy for a life well-lived.
Whether you’re mid-pivot, just starting out, or simply stuck—this episode is your spark. The dream won’t run to you. So let’s run it down.

[Level Up With Gen AI Series] We believe in harnessing the power of AI to unlock human potential.

Are you ready to take your creativity to the next level? In this episode, we explore the philosophy of David Ogilvy, one of the greatest advertising minds of all time. Learn about his concept of 'Divine Discontent' and how it can fuel your ambition, inspire courageous action, and unlock your full potential. We’ll also delve into the specific habits and principles that powered Ogilvy & Mather’s success, giving you the tools to elevate your own work and achieve extraordinary results.

The 8 Habits of Highly Creative Communities are explicitly listed in the sources. These habits, along with their corresponding "evil twins" are:

* Courage (counters Fear)

* Idealism (counters Expedience / The Status Quo)

* Curiosity (counters Boring)

* Playfulness (counters Tyranny of Politeness)

* Candour (counters Cold Arithmetic)

* Intuition (counters Bureaucracy)

* Free‑Spiritedness (counters Giving In)

* Persistence (has no listed "evil twin"; it overcomes obstacles)

These eight creative virtues are presented as a way to cultivate David Ogilvy's concept of "divine discontent" within an organization. The practice of these habits aims to prevent "smugness" and encourage a constant striving for excellence, moving beyond being merely "Very Good" to becoming "Very, Very, Very, Very, Very Good".

The book argues that embracing these habits can help Ogilvy & Mather maintain its creative edge and remain "gorgeous" despite its size.

Are you content with merely functional AI? David Ogilvy wouldn't be.

This podcast embodies his spirit of "divine discontent", pushing beyond the ordinary to explore the fundamental truths of Generative AI.

We delve into the essential building blocks, demanding not just understanding, but mastery. If you refuse to settle for "good enough" and aim for truly groundbreaking generative capabilities, then listen closely.

We believe, as Ogilvy did, that only First Class business, and that in a First Class way, will leave its mark.

Join us as we blaze new trails in the realm of artificial intelligence.

Episodes with Corresponding Motivation (Following the 10-Level Learning Path):

• Level 1: Neural Networks & Deep Learning - Don't Just Model Minds, Mimic Brilliance! The Groundwork of Gen AI.

• Level 2: Introduction to Generative Models - Beyond Prediction: Unleashing AI's Creative Soul.

• Level 3: Variational Autoencoders (VAEs) - Making AI Imagine: The Probabilistic Path to Creative Generation.

• Level 4: Generative Adversarial Networks (GANs) - The Creative Duel: Forging Realistic AI Through Intelligent Competition.

• Level 5: Recurrent Neural Networks (RNNs) & LSTM - Giving AI a Memory: Mastering the Art of Sequential Data.

• Level 6: Attention Mechanisms & the Transformer Architecture - Focus and Breakthrough: The Dawn of Intelligent Language Processing.

• Level 7: Advanced Transformer Architectures & Pre-trained Language Models - The Giants of Language: Leveraging Pre-existing Knowledge for Astonishing Results.

• Level 8: Diffusion Models & Score-Based Generative Modeling - From Noise to Creation: The Elegant Logic of Generative Diffusion.

• Level 9: Multimodal Generative AI Models - Beyond Single Senses: The Rich Tapestry of Multimodal AI.

• Level 10: Ethical Considerations & Future Trends in Generative AI - The Responsible Revolution: Navigating the Ethical Frontier of AI Innovation.

Please find the list of episodes here: https://www.linkedin.com/pulse/leveling-up-journey-through-fundamentals-generative-ai-ma9bc/

How do you teach a sophisticated speech AI to understand and discuss images, especially when paired image-speech data is rare?

This episode unpacks MoshiVis, a new model that achieves just that. We explore the challenges of building Vision-Speech Models and how MoshiVis overcomes them with a unique one-stage training pipeline, synthetic dialogues, and efficient "perceptual augmentation" techniques built upon the Moshi speech LLM.

Join us for a deep dive into the tech that lets AI see, speak, and converse fluidly about the visual world.