The Velocity Lab

Testing Is Phase Zero for AI Agents

Listen Later

Episode Summary

Dave and Dan dig into service readiness — the industry calls it "harness engineering" — and why testing just moved to the #1 spot of their 13 repo-readiness pillars. Before you let agents ship code in the background, the question isn't 90% coverage; it's whether anyone actually trusts the tests. They walk through the onboarding plugin that gets a repo test-ready and how test decisions get baked into the agent's planning phase instead of left to human best-effort.

Key Topics

  • Why "no one has confidence in the tests" is universal — slow, flaky, and thin coverage everywhere
  • Testing as pillar #1 of 13 for agentic-engineering readiness ("phase zero")
  • The test-onboarding plugin: research versions → design the greenfield ideal → reconcile what to keep vs. throw out → a reviewed task list
  • All four layers in scope: unit, integration, end-to-end, smoke — plus migrations, seeds, and integrations
  • Baking "which tests does this need?" into the agent's planning phase rather than relying on a human to remember
  • Coverage dogma vs. reality: test what matters, but keep 90% in CI as the practical proxy
  • The daily cron that catches coverage gaps and staleness; local-first test execution for agents

    • Notable Quotes

    • "Every organization we've pretty much ever worked for — no one has confidence in the test. The tests are slow, the tests are flaky, and they don't cover enough stuff."
    • "How do you know that you can trust your agent to ship code that's gonna work?"
    • "We said 90% coverage — great philosophy. Three months later we realized maybe the 10% is the one that matters."

      • About The Velocity Lab

      Dave O'Dell and Dan McAulay work inside engineering organizations every day helping them ship faster with AI. No hype, no BS — just what's working in the field.

      Subscribe: RSS

      ...more
      View all episodesView all episodes
      Download on the App Store
      The Velocity LabBy Dave O'Dell and Dan McAulay