The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)

CTIBench: Evaluating LLMs in Cyber Threat Intelligence with Nidhi Rastogi - #729


Listen Later

Today, we're joined by Nidhi Rastogi, assistant professor at Rochester Institute of Technology to discuss Cyber Threat Intelligence (CTI), focusing on her recent project CTIBench—a benchmark for evaluating LLMs on real-world CTI tasks. Nidhi explains the evolution of AI in cybersecurity, from rule-based systems to LLMs that accelerate analysis by providing critical context for threat detection and defense. We dig into the advantages and challenges of using LLMs in CTI, how techniques like Retrieval-Augmented Generation (RAG) are essential for keeping LLMs up-to-date with emerging threats, and how CTIBench measures LLMs’ ability to perform a set of real-world tasks of the cybersecurity analyst. We unpack the process of building the benchmark, the tasks it covers, and key findings from benchmarking various LLMs. Finally, Nidhi shares the importance of benchmarks in exposing model limitations and blind spots, the challenges of large-scale benchmarking, and the future directions of her AI4Sec Research Lab, including developing reliable mitigation techniques, monitoring "concept drift" in threat detection models, improving explainability in cybersecurity, and more.

The complete show notes for this episode can be found at https://twimlai.com/go/729.

...more
View all episodesView all episodes
Download on the App Store

The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)By Sam Charrington

  • 4.7
  • 4.7
  • 4.7
  • 4.7
  • 4.7

4.7

412 ratings


More shows like The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)

View all
The AI in Business Podcast by Daniel Faggella

The AI in Business Podcast

161 Listeners

Data Skeptic by Kyle Polich

Data Skeptic

470 Listeners

Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

Super Data Science: ML & AI Podcast with Jon Krohn

295 Listeners

NVIDIA AI Podcast by NVIDIA

NVIDIA AI Podcast

324 Listeners

AI Today Podcast: Artificial Intelligence Insights, Experts, and Opinion by AI & Data Today

AI Today Podcast: Artificial Intelligence Insights, Experts, and Opinion

143 Listeners

Practical AI by Practical AI LLC

Practical AI

190 Listeners

Last Week in AI by Skynet Today

Last Week in AI

280 Listeners

Machine Learning Street Talk (MLST) by Machine Learning Street Talk (MLST)

Machine Learning Street Talk (MLST)

88 Listeners

Me, Myself, and AI by MIT Sloan Management Review and Boston Consulting Group (BCG)

Me, Myself, and AI

101 Listeners

No Priors: Artificial Intelligence | Technology | Startups by Conviction

No Priors: Artificial Intelligence | Technology | Startups

125 Listeners

This Day in AI Podcast by Michael Sharkey, Chris Sharkey

This Day in AI Podcast

190 Listeners

Latent Space: The AI Engineer Podcast by swyx + Alessio

Latent Space: The AI Engineer Podcast

63 Listeners

The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis by Nathaniel Whittemore

The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis

423 Listeners

AI + a16z by a16z

AI + a16z

29 Listeners

Training Data by Sequoia Capital

Training Data

37 Listeners