The AI Accuracy Files

93% accurate isn’t good enough when the number goes to your CFO. Here’s how AtScale closes the gap.

In Episode 1 of The AI Accuracy Files, a semantic layer roughly doubled how often AI answered business questions correctly on the BIRD-Interact benchmark. In Episode 2, host Ryan Siegler sits down again with AtScale CTO and co-founder Dave Mariani and Chief Marketing Officer Mark Palmer to explain how the rest of the gap gets closed: the AI Accuracy Flywheel.

The flywheel has four parts, and each one fixes a different way AI gets numbers wrong:

CoModel builds and maintains the semantic model, with a human approving every change
Ava (Ask, Verify, Answer) spots when a question could mean more than one thing and asks before it answers
ACE, the AI Computation Engine, turns each question into one governed, deterministic query
Canary tests AI answers against known right answers on your own data, so fixes stick and errors don’t come back

Dave and Mark walk through each part, why all four need to work together, where the human stays in the loop, and the new roles teams need to keep AI accurate.

Benchmark results are preliminary and self-reported, and haven’t been verified by the BIRD team yet.

Learn more about the semantic layer: https://www.atscale.com/glossary/sema... 

AI on its own gets about one enterprise question in five right. What changes when you add a semantic layer?

In the first episode of The AI Accuracy Files, host Ryan Siegler sits down with AtScale CTO and co-founder Dave Mariani and Mark Palmer, AtScale’s Chief Marketing Officer, to unpack BIRD-Interact. It’s an independent public benchmark built to look like real analytics work: ambiguous questions, dirty data, metrics that aren’t in the schema, and a simulated user who answers follow-up questions but won’t write the SQL.

They walk through what the 21% leaderboard average really means, why AI models given the full documentation still guess wrong, and what happened when AtScale ran seven models through the benchmark with a semantic layer in the path.

Key results (preliminary):

• Leaderboard average without a semantic layer: 21.2%
  
  
• First-question accuracy with AtScale: up 72% to 137% across all seven models (best of three runs)
  
  
• On the 317 questions with a valid answer key: 90.6% with AtScale vs 45.8% without (3-run means), and 93.4% in the best run