March 17, 2025

Detection vs. Noise: What MITRE ATT&CK Evaluations Reveal About Your Security Tools | A Conversation with Allie Mellen | Redefining CyberSecurity with Sean Martin

36 minutes

⬥GUEST⬥

Allie Mellen, Principal Analyst, Forrester | On LinkedIn: https://www.linkedin.com/in/hackerxbella/

⬥HOST⬥

Host: Sean Martin, Co-Founder at ITSPmagazine and Host of Redefining CyberSecurity Podcast | On ITSPmagazine: https://www.itspmagazine.com/sean-martin

⬥EPISODE NOTES⬥

In this episode, Allie Mellen, Principal Analyst on the Security and Risk Team at Forrester, joins Sean Martin to discuss the latest results from the MITRE ATT&CK Ingenuity Evaluations and what they reveal about detection and response technologies.

The Role of MITRE ATT&CK Evaluations

MITRE ATT&CK is a widely adopted framework that maps out the tactics, techniques, and procedures (TTPs) used by threat actors. Security vendors use it to improve detection capabilities, and organizations rely on it to assess their security posture. The MITRE Ingenuity Evaluations test how different security tools detect and respond to simulated attacks, helping organizations understand their strengths and gaps.

Mellen emphasizes that MITRE’s evaluations do not assign scores or rank vendors, which allows security leaders to focus on analyzing performance rather than chasing a “winner.” Instead, organizations must assess raw data to determine how well a tool aligns with their needs.

Alert Volume and the Cost of Security Data

One key insight from this year’s evaluation is the significant variation in alert volume among vendors. Some solutions generate thousands of alerts for a single attack scenario, while others consolidate related activity into just a handful of actionable incidents. Mellen notes that excessive alerting contributes to analyst burnout and operational inefficiencies, making alert volume a critical metric to assess.

Forrester’s analysis includes a cost calculator that estimates the financial impact of alert ingestion into a SIEM. The results highlight how certain vendors create a massive data burden, leading to increased costs for organizations trying to balance security effectiveness with budget constraints.

The Shift Toward Detection and Response Engineering

Mellen stresses the importance of detection engineering, where security teams take a structured approach to developing and maintaining high-quality detection rules. Instead of passively consuming vendor-generated alerts, teams must actively refine and tune detections to align with real threats while minimizing noise.

Detection and response should also be tightly integrated. Forrester’s research advocates linking every detection to a corresponding response playbook. By automating these processes through security orchestration, automation, and response (SOAR) solutions, teams can accelerate investigations and reduce manual workloads.

Vendor Claims and the Reality of Security Tools

While many vendors promote their performance in the MITRE ATT&CK Evaluations, Mellen cautions against taking marketing claims at face value. Organizations should review MITRE’s raw evaluation data, including screenshots and alert details, to get an unbiased view of how a tool operates in practice.

For security leaders, these evaluations offer an opportunity to reassess their detection strategy, optimize alert management, and ensure their investments in security tools align with operational needs.

For a deeper dive into these insights, including discussions on AI-driven correlation, alert fatigue, and security team efficiency, listen to the full episode.

⬥SPONSORS⬥

LevelBlue: https://itspm.ag/attcybersecurity-3jdk3

ThreatLocker: https://itspm.ag/threatlocker-r974

⬥RESOURCES⬥

Inspiring Post: https://www.linkedin.com/posts/hackerxbella_go-beyond-the-mitre-attck-evaluation-to-activity-7295460112935075845-N8GW/

Blog | Go Beyond The MITRE ATT&CK Evaluation To The True Cost Of Alert Volumes: https://www.forrester.com/blogs/go-beyond-the-mitre-attck-evaluation-to-the-true-cost-of-alert-volumes/

⬥ADDITIONAL INFORMATION⬥

✨ More Redefining CyberSecurity Podcast:

🎧 https://www.itspmagazine.com/redefining-cybersecurity-podcast

Redefining CyberSecurity Podcast on YouTube:

📺 https://www.youtube.com/playlist?list=PLnYu0psdcllS9aVGdiakVss9u7xgYDKYq

Interested in sponsoring this show with a podcast ad placement? Learn more:

👉 https://itspm.ag/podadplc

Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

...more

View all episodes

By Sean Martin, ITSPmagazine

33 ratings