Maintainable

Maintainable

By Robby RussellBusinessTechnologyCareers
Download on the App Store

Maintainable episodes

  • Gabriela Moreira: When 800 Tests Aren’t Enough

    A large test suite should make developers feel safer changing software. Gabriela Moreira, CEO of Quint, has recently experienced something quite different: a new codebase grew from roughly 200 tests to 800 with the help of AI, yet she found herself trusting the suite less and manually verifying behavior more. Gabriela and Robby unpack the difference between having tests and understanding what those tests actually protect, including why she has a particular appreciation for regression tests that preserve the history of real problems.

    That question of understanding leads into Gabriela's work on Quint, an open-source executable specification language influenced by TLA+. Gabriela explains how developers can model behaviors and explore edge cases before those scenarios reach production. Rather than treating formal methods as something reserved for banks and safety-critical systems, she makes the case for using specifications more broadly to reason about concurrency, state changes, multiple users, and other situations where software can behave in more ways than a developer can comfortably keep in their head.

    The conversation also digs into the relationship between specifications and existing test suites. Gabriela describes model-based testing, trace validation, and the challenge of keeping a specification synchronized with the code that actually runs in production. She also shares a Kubernetes example involving the OpenHands team, where modeling a race condition with Quint showed that a proposed fix would still fail in another scenario. The team could explore another solution before waiting for the intermittent production problem to return.

    As AI takes over more implementation and debugging work, Gabriela argues that developers need to strengthen their judgment about how software should behave. She pushes back on workflows where humans are expected to fully describe a solution in a giant Markdown document before handing it to an agent. For Gabriela, some of the useful and enjoyable parts of programming come from experimentation: trying something, seeing what happens, changing your understanding, and trying again. If agents increasingly handle the mechanical work, preserving that ability to explore, evaluate, and make decisions may matter more than simply producing more code.

    Topics & Timestamps
    • [00:00:53] What Makes Software Maintainable? Gabriela starts with trustworthy tests and explains why confidence comes from understanding both the tests and the behavior they protect.
    • [00:03:06] When Hundreds of Tests Still Aren't Reassuring: Gabriela describes a new AI-assisted codebase that quickly grew from around 200 tests to 800, while her confidence in the suite declined.
    • [00:06:00] What Is Quint? Gabriela introduces Quint as an executable specification language and explains how it can be used to model and verify software behavior.
    • [00:08:29] Where Formal Specifications Fit in Everyday Development: Robby asks whether these techniques apply to ordinary web applications, leading to examples involving multiple users, state changes, and edge cases.
    • [00:12:23] Why Build Another Specification Language? Gabriela explains Quint's relationship to TLA+ and why her team wanted to make formal specification more approachable for working programmers.
    • [00:16:15] Who Is Quint Actually For? Gabriela makes the case that reasoning about software behavior isn't limited to developers working on distributed systems or critical infrastructure.
    • [00:18:11] AI Lowers the Barrier to Formal Methods: Gabriela explains how coding agents have changed the cost of learning and writing formal specifications.
    • [00:20:52] Keeping Specifications and Code in Sync: Gabriela walks through model-based testing, trace validation, and the maintenance problems that arise when specifications drift away from implementation.
    • [00:25:00] Who Decides What "Correct" Means? Robby and Gabriela explore the difference between obvious correctness properties and product behaviors that require human context and judgment.
    • [00:31:58] Bridging Academia and Everyday Software Development: Gabriela discusses the divide between programming-language research and industry, and why she wants formal methods to become more accessible.
    • [00:35:22] Catching a Kubernetes Fix Before Shipping It: Gabriela shares how the OpenHands team modeled an intermittent Kubernetes problem and discovered that their first proposed fix wouldn't actually solve every scenario.
    • [00:40:00] What Should We Still Delegate to Humans? The conversation turns to AI development workflows and why Gabriela doesn't want programming to become writing exhaustive instructions for agents.
    • [00:45:25] Human Judgment as a Software Engineering Skill: Gabriela argues that developers who can judge how software should behave may become more valuable as AI takes over more implementation work.
    • [00:47:06] A Small Formal Methods Homework Assignment: Gabriela suggests starting with something familiar, finding the state machine hiding inside it, and modeling it with Quint.
    • [00:50:47] Recommended Reading: Gabriela recommends The Name of the Wind by Patrick Rothfuss.
    Thanks to Our Sponsor!

    Your test coverage says 90%, but that might be misleading. Undercover CI looks at your Ruby pull requests and shows you which parts of your changes weren't tested- not just overall coverage, but what changed and what got missed, down to the method level. Visit undercover-ci.com and use code MAINTAINABLE for 15% off your first billing cycle. Free for public repos. Private repos with unlimited users also available.

    Resources & Links
    • Quint 
    • Quint on GitHub
    • Towards a Solution for Cognitive Debt
    • Quint Connect and the Emerald project
    • TLA+
    • OpenHands
    • Gabriela Moreira on LinkedIn
    • Gabriela's interview with Unite.AI
    • The Name of the Wind by Patrick Rothfuss
    • OpenHands Kubernetes Use Case
    • Trying out Quint: 5-min getting started
    • Sign up for Quint Studio

    Subscribe to Maintainable on:

    • Apple Podcasts
    • Spotify

    Or search "Maintainable" wherever you stream your podcasts.

    Keep up to date with the Maintainable Podcast by joining the newsletter.

    53 min
  • Matija Šošić: Abstractions Aren’t Going Anywhere

    What makes software maintainable might have less to do with how clean the code looks and more to do with how well it serves the people responsible for it. Matija Šošić, co-founder and CEO of Wasp, joins Robby to talk about boring and predictable code, resisting unnecessary abstractions, and why understanding the ugly parts of a system matters.

    They trace Wasp's evolution from a custom domain-specific language to TypeScript, why web development keeps cycling between bundling and unbundling, and what framework authors can learn from Rails and Laravel. They also dig into AI-generated code and Matija's argument that cheaper code doesn't eliminate the need for abstractions. If anything, frameworks and their constraints may become even more useful when agents are writing more of the implementation.

    Topics

    [00:00:51] Software Should Serve the People Maintaining It: Matija describes maintainability in terms of whether developers understand their system and feel comfortable working within it.

    [00:02:25] Boring and Predictable Code: Why a little repetition can be healthier than reaching for another abstraction too early.

    [00:05:39] What Is Wasp?: Matija introduces Wasp and the desire for a more cohesive full-stack JavaScript framework.

    [00:07:15] Where the Idea for Wasp Came From: Framework fatigue, constantly changing JavaScript stacks, and the desire to stop reinventing the same decisions from project to project.

    [00:13:11] Building a Custom Language: Matija explains why Wasp started with its own DSL and the goal of separating an application's specification from its underlying technology.

    [00:17:36] The Economics of an Open Source Framework: Robby and Matija discuss Y Combinator, venture funding, distribution, and how a framework can eventually become a business.

    [00:24:05] Becoming "Abstraction Curators": How the Wasp team decides what deserves an abstraction and why they try not to invent solutions simply for the sake of being different.

    [00:31:03] Do We Still Need Abstractions When AI Writes Code?: Why Matija believes agents benefit from frameworks, conventions, and guardrails rather than making them obsolete.

    [00:36:29] Moving From a Custom DSL to TypeScript: Why Wasp's language became an adoption barrier and how the team preserved the underlying idea while moving to familiar syntax.

    [00:39:47] Why Is Web Development Still So Complicated?: Matija distinguishes domain complexity from implementation complexity and explains the recurring cycle of bundling and unbundling.

    [00:43:36] Are We Entering Another Bundling Era?: Robby and Matija explore why technologies mature into foundational layers while attention moves toward the next abstraction.

    [00:47:14] Finding Something Solid Amid the AI Noise: Matija discusses reevaluating assumptions about AI while holding onto principles that aren't tied to a particular tool.

    [00:50:51] Don't Bet Your Career on React: Why Matija thinks developers should invest in software engineering fundamentals rather than tying their expertise too closely to today's frameworks.

    Thanks to Our Sponsor!

    Your test coverage says 90%, but that might be misleading. Undercover CI looks at your Ruby pull requests and shows you which parts of your changes weren't tested- not just overall coverage, but what changed and what got missed, down to the method level. Visit undercover-ci.com and use code MAINTAINABLE for 15% off your first billing cycle. Free for public repos. Private repos with unlimited users also available.

    Links & Resources
    • Wasp
    • Wasp on GitHub
    • A New Language for Web Dev Was a Mistake
    • Matija Šošić on X
    • Matija Šošić on LinkedIn
    • Zen and the Art of Motorcycle Maintenance
    • Ruby on Rails
    • Laravel
    • Prisma

    Subscribe to Maintainable on:

    • Apple Podcasts
    • Spotify

    Or search "Maintainable" wherever you stream your podcasts.

    Keep up to date with the Maintainable Podcast by joining the newsletter.

    55 min
  • José Valim: Designing a Language You Can Maintain for Decades

    What does maintainability look like when the software you’re responsible for is a programming language? José Valim creator of Elixir and founder of Dashbit, joins Robby to talk about software that can “fit in your head,” the importance of clear boundaries, and why maintaining a language requires a different level of restraint than maintaining an application.

    José traces Elixir’s origins back to his time on the Rails Core team and a concurrency bug that sent him looking for different approaches to software. That journey led through functional programming to Erlang and the BEAM, where he found ideas around immutability, concurrency, and distributed systems that reshaped how he thought about software. He explains why Elixir was designed to be extensible, allowing communities to build things like Phoenix and Numerical Elixir without continually expanding the language itself.

    Robby and José also look at what AI is changing for maintainers. José explains how Tidewave gives coding agents access to context that frameworks already provide to developers, why documentation and good error messages have become unexpectedly valuable to agents, and how AI-generated contributions are changing open-source collaboration. They also explore whether coding agents might change the old calculus around dependencies, SDKs, and choosing a programming language.

    Topics and Timestamps
    • [00:00:53] What Makes Software Maintainable: José compares well-maintained software to good conference Wi-Fi and explains why he values software whose boundaries and moving parts can fit in his head.
    • [00:03:20] Maintaining a Programming Language: How responsibility changes when thousands of applications depend on the software you maintain, and why language designers have to minimize churn for users.
    • [00:10:51] The Origins of Elixir: José tells the story of the Rails concurrency bug that changed his career and sent him exploring functional programming, Erlang, and distributed systems.
    • [00:17:00] Removing the Root Cause: Why José distinguishes between layering a solution over a problem and designing software so the underlying problem disappears.
    • [00:20:47] Designing Elixir to Be Extensible: Why José resisted making Elixir a web-specific language and instead focused on giving developers the building blocks to extend it into new domains.
    • [00:25:42] Restraint and the Elixir Ecosystem: José explains why much of his work moved away from changing the language itself and toward helping communities build on top of it.
    • [00:28:21] Giving Coding Agents Better Context with Tidewave: How Tidewave connects Rails and Phoenix applications with coding agents and exposes information developers already rely on.
    • [00:33:20] Designing for Humans in the Age of AI: Why José believes programming languages should continue optimizing for humans, and how features that help humans often help agents too.
    • [00:36:40] Documentation as Infrastructure: How Elixir’s long-standing investment in first-class documentation, HexDocs, Markdown, and searchable package documentation has translated well to AI-assisted development.
    • [00:41:40] AI and Open-Source Maintenance: José describes the increase in AI-assisted contributions and why AI-generated responses can disrupt the human relationship between contributors and maintainers.
    • [00:46:22] Rethinking Dependencies and Language Choice: Robby and José consider whether AI makes it easier to bring ideas across ecosystems instead of rewriting applications or depending on large SDKs.
    Thanks to Our Sponsors!

    Your test coverage says 90%, but that might be misleading. Undercover CI looks at your Ruby pull requests and shows you which parts of your changes weren't tested- not just overall coverage, but what changed and what got missed, down to the method level. Visit undercover-ci.com and use code MAINTAINABLE for 15% off your first billing cycle. Free for public repos. Private repos with unlimited users also available.

    Mailtrap is a modern email delivery platform built for developers. Native SDKs, a secure Email API and SMTP, and a free tier with 4,000 emails a month. When you need help, you'll reach real people on 24/7 support, not an AI chatbot. Try Mailtrap for free!

    Resources and Links
    • Elixir
    • Dashbit
    • Tidewave
    • Phoenix Framework
    • Erlang
    • Hex
    • HexDocs
    • Numerical Elixir
    • Nx
    • Foundation by Isaac Asimov
    • Dune by Frank Herbert

    Subscribe to Maintainable on:

    • Apple Podcasts
    • Spotify

    Or search "Maintainable" wherever you stream your podcasts.

    Keep up to date with the Maintainable Podcast by joining the newsletter.

    54 min
  • John Athayde: Maintaining the User Side of Software

    Software maintainability is often discussed through the codebase, but users experience years of accumulated decisions through the interface. John Athayde of Meticulous joins Robby to examine maintainability from the front-end and product-design side, starting with documentation and the context it preserves for future teams.

    John describes how older applications accumulate multiple UI libraries, generations of CSS, inconsistent components, and different interaction patterns. Improving these systems does not always require a redesign. Sometimes the more valuable work is removing dependencies the browser no longer needs, consolidating patterns, and understanding which unusual workflows users still depend on. They also explore how design systems can become part of a maintenance strategy, provided teams treat them as products that require governance and restraint.

    Later, Robby and John turn to AI. John is using AI tooling to audit applications, find repeated patterns, remove old dependencies, expand tests, and reconstruct requirements from existing code. These tools can make software archaeology and large-scale changes much faster, but speed does not replace judgment. The challenge is understanding what should change, what needs to remain, and whether today's faster implementation is creating software tomorrow's team can still maintain.

    Topics
    • [00:02:00] Documentation as a Sign of Maintainability: Why documentation and evidence of ongoing care matter when entering an existing codebase.
    • [00:07:02] Maintainability From the User's Side: Dated interfaces, usability, and replacing old dependencies with modern browser capabilities.
    • [00:11:22] Making the Case for Cleanup: Connecting technical cleanup to development speed and the ability to ship future features.
    • [00:13:27] Bringing Coherence to Operational Software: How products accumulate different interfaces and patterns through years of development and acquisitions.
    • [00:19:14] Finding the Why: Discovering how software is actually used before deciding what should change or disappear.
    • [00:24:44] Knowing When to Remove Things: Product ownership, sunsetting features, and the organizational politics of taking functionality away.
    • [00:34:05] Why Designers Should Understand Code: How technical fluency helps designers and UX awareness helps developers.
    • [00:39:00] Choosing Tools in Service of the Product: Balancing new frameworks and experimentation against what a team can maintain.
    • [00:47:19] Design Systems as Maintainable Products: Tokens, components, governance, and auditing years of accumulated UI decisions.
    • [00:51:39] Using AI for Front-End Archaeology: Applying AI and static analysis to find patterns, remove dependencies, and investigate old applications.
    • [01:04:00] Rewrite or Refactor in the AI Era?: How tests, captured requirements, and AI-assisted development may change the economics of rewrites.
    • [01:14:20] Where AI Still Needs Human Judgment: Why faster implementation does not eliminate difficult product and engineering decisions.
    Thanks to Our Sponsors!

    Your test coverage says 90%, but that might be misleading. Undercover CI looks at your Ruby pull requests and shows you which parts of your changes weren't tested- not just overall coverage, but what changed and what got missed, down to the method level. Visit undercover-ci.com and use code MAINTAINABLE for 15% off your first billing cycle. Free for public repos. Private repos with unlimited users also available.

    Mailtrap is a modern email delivery platform built for developers. Native SDKs, a secure Email API and SMTP, and a free tier with 4,000 emails a month. When you need help, you'll reach real people on 24/7 support, not an AI chatbot. Try Mailtrap for free!

    Links & Resources
    • John Athayde
    • Meticulous
    • John Athayde on LinkedIn
    • John Athayde on Bluesky
    • John Athayde on X
    • John Athayde on Ruby Social
    • The Timeless Way of Building by Christopher Alexander
    • A Pattern Language by Christopher Alexander, Sara Ishikawa, and Murray Silverstein
    • The Phoenix Architecture by Chad Fowler
    • Ruby on Rails
    • Axe-core
    • Herb, HTML+ERB tooling by Marco Roth

    Subscribe to Maintainable on:

    • Apple Podcasts
    • Spotify

    Or search "Maintainable" wherever you stream your podcasts.

    Keep up to date with the Maintainable Podcast by joining the newsletter.

    1 hr 23 min
  • Kristin Isaac: The Customer Doesn’t Care About Your Technical Debt

    Technical debt rarely arrives in a support ticket labeled “technical debt.” Customers experience something simpler: the product is slow, unreliable, or the same problem keeps coming back. Kristin Isaac, Co-Founder and CEO of Strudel, joins Robby to look at software maintainability from the customer and product side of the organization. Kristin shares how her background in customer success, sales, and support shaped the way she thinks about reliability and why engineering teams often lack the business context surrounding the problems they’re being asked to solve.

    That disconnect becomes especially important when engineers need to advocate for work customers may never directly see. Kristin and Robby explore why talking about technical debt alone often fails to persuade business leaders, and how engineers can instead connect maintenance work to customer experience, renewals, revenue, reliability, and risk. They also discuss the danger of letting the loudest customer determine priorities, the tension between urgent roadmap work and important foundational work, and why engineers shouldn’t quietly absorb the trade-offs that come with shipping faster. Kristin argues that better storytelling, shared metrics, and stronger relationships with customer-facing teams can help engineering make a much stronger case.

    The conversation also turns to AI and the growing gap between generating software and understanding it. Robby and Kristin consider what happens when engineers increasingly ask AI for help instead of their teammates, and what that might mean for collaboration, onboarding, and learning. Rather than keeping that experimentation inside engineering, Kristin sees an opportunity for engineers to become educators across the organization… helping colleagues understand where AI works, where it falls short, and how teams can experiment with it together.

    Episode Highlights

    [00:00:49] Measuring Well-Maintained Software: Kristin shares practical measures around alert volume, detection time, recovery time, and confidence in frequent deployments.

    [00:02:26] What Strudel Does: Kristin explains Strudel’s focus on bringing engineering, support, customer, and business context together around technical customer issues.

    [00:07:35] What Customers Actually Experience: Robby and Kristin connect technical debt to the problems customers notice, including slow, unreliable, and broken software.

    [00:12:39] Prioritizing With Business Context: They explore competing incentives across engineering, product, sales, support, and customer success, including the danger of relying on the “squeaky wheel.”

    [00:21:14] Different Teams, Different Versions of the Problem: Kristin explains why no single department necessarily has the complete picture and why shared context matters.

    [00:23:29] Who Owns Reliability?: Robby and Kristin consider where responsibility for reliability sits across engineering, product, and the broader organization.

    [00:24:32] Making the Case for Technical Debt: Kristin explains why technical debt is easier to ignore when engineers can’t connect it to measurable customer or business impact.

    [00:26:21] Advocating for Invisible Maintenance Work: The conversation turns to foundational improvements, the tension between urgent and important work, and partnering with customer-facing teams to build a stronger case.

    [00:33:25] Engineers as Storytellers: Kristin argues that engineers can become more effective advocates by learning how to communicate technical concerns in terms the rest of the business understands.

    [00:34:54] AI and the Understanding Gap: Robby and Kristin discuss whether software is becoming faster to generate than it is to understand and maintain.

    [00:37:20] What Happens When Engineers Stop Asking Each Other?: They explore AI’s potential effects on collaboration, learning, onboarding, and the social dynamics of engineering teams.

    [00:46:20] Engineers as AI Educators: Kristin makes the case for engineers sharing their AI experiments with product, sales, marketing, and customer-facing teams instead of keeping that knowledge inside engineering.

    Thanks to Our Sponsors!

    Your test coverage says 90%, but that might be misleading. Undercover CI looks at your Ruby pull requests and shows you which parts of your changes weren't tested- not just overall coverage, but what changed and what got missed, down to the method level. Visit undercover-ci.com and use code MAINTAINABLE for 15% off your first billing cycle. Free for public repos. Private repos with unlimited users also available.

    [Mailtrap]Mailtrap is a modern email delivery platform built for developers. Native SDKs, a secure Email API and SMTP, and a free tier with 4,000 emails a month. When you need help, you'll reach real people on 24/7 support, not an AI chatbot. Try Mailtrap for free!

    Resources & Links
    • Strudel
    • Kristin Isaac on LinkedIn
    • Strudel on LinkedIn
    • Empire of AI by Karen Hao
    • Fablehaven series by Brandon Mull
    • Subscribe to Maintainable on:

      • Apple Podcasts
      • Spotify

      Or search "Maintainable" wherever you stream your podcasts.

      Keep up to date with the Maintainable Podcast by joining the newsletter.

      54 min
    • David Hayes: Boring Software, Clear Incentives, and Better Checklists

      Drawing from his experience at PagerDuty, Sentry, and now FusionAuth, he joins Robby to explore what happens when software becomes trusted infrastructure. Their conversation touches on API design, naming things, product management, self-hosted software, and why customers rarely upgrade as quickly as we’d like them to.

      They also discuss why “boring” software is often the most successful, how engineering teams can better communicate technical debt by connecting it to customer outcomes, and why estimates and scope creep are often influenced as much by human psychology as technical complexity. David shares lessons from supporting long-lived authentication systems and explains why stability, compatibility, and predictable upgrades matter more than constantly shipping flashy new features.

      The conversation takes an unexpected turn into naval history, where David draws parallels between successful fleets and successful software teams. Clear incentives, strong communication, and dependable checklists consistently outperform heroics. As AI becomes part of the software development process, those lessons feel even more relevant. Teams that make expectations explicit, align around customer outcomes, and invest in reliable processes will be better positioned for whatever comes next.

      Episode Highlights

      [00:00:50] What Makes Software Truly Maintainable: David explains why fitness for purpose is at the heart of maintainable software.

      [00:04:32] API Design Decisions That Last for Years: David shares lessons from PagerDuty about naming, breaking changes, and the long-term consequences of API decisions.

      [00:10:44] What FusionAuth Is and Why Authentication Is Different: David explains FusionAuth and why authentication is a problem many teams are better off not solving themselves.

      [00:16:57] The Realities of Maintaining Self-Hosted Software: Supporting self-hosted deployments means balancing upgrades, compatibility, security, and customer control.

      [00:24:49] Why Customers Rarely Upgrade the Way Product Teams Expect: David explains why customers often settle into a product’s existing capabilities and may not rush to adopt new features.

      [00:27:37] What “Boring Software” Really Means: David makes the case that successful software should focus on making users successful rather than giving teams opportunities to build what they find exciting.

      [00:29:56] Connecting Technical Debt to Business Outcomes: David discusses why engineering teams need to connect technical debt to customer outcomes and broader business needs.

      [00:39:22] Estimates, Optimism, and Scope Creep: David explores how estimates can become persuasion tools and how optimism can unintentionally expand project scope.

      [00:43:15] Naval Warfare Lessons for Software Teams: David draws unexpected parallels between naval history, incentives, communication, and software development.

      [00:48:29] Alignment, Checklists, and Operational Excellence: David explains why aligned teams with reliable processes consistently outperform teams that rely on individual heroics.

      [00:51:21] AI, Explicit Instructions, and the Future of Software Development: David discusses how AI is changing the skills, expectations, and checklists developers need to work effectively.

      [00:53:29] Where to Learn More About FusionAuth: David shares where listeners can learn more about FusionAuth and its work around authentication and compliance.

      Thanks to Our Sponsors!

      Your test coverage says 90%, but that might be misleading. Undercover CI looks at your Ruby pull requests and shows you which parts of your changes weren't tested- not just overall coverage, but what changed and what got missed, down to the method level. Visit undercover-ci.com and use code MAINTAINABLE for 15% off your first billing cycle. Free for public repos. Private repos with unlimited users also available.

      [Mailtrap]Mailtrap is a modern email delivery platform built for developers. Native SDKs, a secure Email API and SMTP, and a free tier with 4,000 emails a month. When you need help, you'll reach real people on 24/7 support, not an AI chatbot. Try Mailtrap for free!

      Resources & Links
      • FusionAuth
      • FusionAuth Blog
      • PagerDuty
      • Sentry
      • curl
      • Book Recommendations
        • Castles of Steel: Britain, Germany, and the Winning of the Great War at Sea by Robert K. Massie
        • The Last Stand of the Tin Can Sailors: The Extraordinary World War II Story of the U.S. Navy's Finest Hour by James D. Hornfischer
        • Subscribe to Maintainable on:

          • Apple Podcasts
          • Spotify

          Or search "Maintainable" wherever you stream your podcasts.

          Keep up to date with the Maintainable Podcast by joining the newsletter.

          56 min
        • Diana Pfeil: Building Confidence in Probabilistic Systems

          Machine learning systems can degrade even when the underlying code has not changed. Diana Pfeil of Sunbeam Consulting joins Robby Russell on Maintainable to explain how changing data, model behavior, and non-deterministic outputs create a different kind of maintenance challenge.

          Rather than asking whether a system is simply correct or incorrect, teams need reliable ways to measure confidence in its behavior.

          Diana introduces evals as a way to test AI-generated outputs that may be phrased differently each time. She and Robby discuss using LLMs to judge other LLM outputs, reviewing production traces, sampling unusual cases, protecting sensitive data, and keeping humans involved when automated checks cannot provide enough confidence.

          They also explore how prompts should be versioned and tested against representative examples.

          The conversation turns to the operational costs that come with AI features. Model providers can deprecate dependencies quickly, prompts may behave differently after an upgrade, and teams must continue monitoring systems that might once have been considered finished.

          Diana encourages organizations to ask what can now be automated, how accurate the result needs to be, and whether the benefit justifies the additional maintenance work.

          Diana also makes the case for starting with the simplest model or deterministic process that can solve the problem. Teams can add complexity once the baseline proves insufficient, but designing around imagined future requirements often creates the wrong system.

          Her advice for engineers trying to introduce AI internally is equally direct: build a small prototype that solves a real problem, then let the result make the case.

          Episode Highlights

          [00:00:50] Maintaining Probabilistic Software: Diana explains why maintaining AI systems involves the code, changing data, model behavior, and confidence in the output.

          [00:02:35] What Are Evals?: Robby asks how teams test AI-generated results when the correct response may be worded differently each time.

          [00:04:38] Using an LLM as a Judge: Diana describes using one model to evaluate another and why the judge must be calibrated against human decisions.

          [00:06:39] Monitoring AI in Production: Diana introduces human review, traces, observability, and production sampling.

          [00:08:44] Recognizing Input Drift: A meeting-notes example shows how changing inputs can degrade an otherwise unchanged system.

          [00:12:06] Maintaining Models and Prompts: Diana outlines the code, model, prompt, and data-pipeline changes teams may need to make.

          [00:14:24] Versioning and Testing Prompts: Robby asks how prompt experimentation fits into source control and repeatable testing.

          [00:18:32] Building Confidence in a Black Box: Diana explains how evals and production reviews help teams avoid regressions.

          [00:21:16] Diana’s Machine Learning Background: Diana shares her path from recommendation systems at Amazon to startup leadership and consulting.

          [00:22:26] How Sunbeam Consulting Helps Teams: Diana describes advising leaders on AI strategy and helping teams build machine learning products.

          [00:24:28] Finding Useful Automation Opportunities: Diana explains how teams can identify previously unstructured work that may now be practical to automate.

          [00:27:40] The Cost of Automated Decisions: Robby and Diana compare human error with the oversight and infrastructure required by AI systems.

          [00:30:14] AI Is Not Free to Maintain: Diana discusses model deprecations, vendor dependencies, and the ongoing support required after launch.

          [00:36:45] Keeping Up With Rapidly Changing Tools: Diana explains why teams need room to experiment without constantly disrupting established workflows.

          [00:38:49] Why Simpler Models Often Win: Diana makes the case for starting with a baseline before introducing more sophisticated approaches.

          [00:45:18] Selling an AI Idea Without the Buzzwords: Diana recommends building a useful prototype and allowing the result to make the case.

          [00:46:30] The Inner Game of Tennis: Diana recommends W. Timothy Gallwey’s book about learning, judgment, and performance.

          Resources Mentioned
          • Sunbeam Consulting
          • Diana Pfeil on LinkedIn
          • Pydantic
          • Amazon Bedrock Guardrails
          • Claude Code
          • Cursor
          • OpenAI Codex
          • The Inner Game of Tennis by W. Timothy Gallwey
          • Thanks to Our Sponsors!

            Your test coverage says 90%, but that might be misleading. Undercover CI looks at your Ruby pull requests and shows you which parts of your changes weren't tested- not just overall coverage, but what changed and what got missed, down to the method level. Visit undercover-ci.com and use code MAINTAINABLE for 15% off your first billing cycle. Free for public repos. Private repos with unlimited users also available.

            Mailtrap is a modern email delivery platform built for developers. Native SDKs, a secure Email API and SMTP, and a free tier with 4,000 emails a month. When you need help, you'll reach real people on 24/7 support, not an AI chatbot. Try Mailtrap for free!

            Subscribe to Maintainable on:

            • Apple Podcasts
            • Spotify

            Or search "Maintainable" wherever you stream your podcasts.

            Keep up to date with the Maintainable Podcast by joining the newsletter.

            49 min
          • Chris Coyier: The Long Game of Maintaining CodePen

            What does it take to keep a product healthy after more than 15 years of continuous evolution?

            In this episode, Robby Russell talks with Chris Coyier, co-founder of CodePen, about the long game of maintaining software. Chris shares how CodePen has evolved over time, the trade-offs involved in migrating parts of the platform from Rails to Go, and the challenges of balancing maintenance work with the desire to build what's next.

            They also explore the human side of maintainability, the role of technical debt in shaping priorities, and why small teams often have to make very intentional decisions about where to invest their limited time and attention.

            Whether you're maintaining a side project, stewarding a legacy application, or helping a team navigate change, this conversation offers practical insights into building software that lasts.

            Key Topics
            • Defining what "well-maintained software" really means
            • Why maintainability is often more of a people problem than a code problem
            • The origin story of CodePen
            • Supporting a product that has evolved over 15 years
            • Balancing maintenance work with product evolution
            • Gradually migrating from Rails to Go
            • Using GraphQL across multiple implementations
            • Technical debt and its many interpretations
            • Team size, communication overhead, and organizational design
            • Simplifying software by embracing browser capabilities
            • Links & Resources
              • ChrisCoyier.net
              • Chris Coyier on Bluesky
              • CodePen
              • ShopTalk Show
              • CSS-Tricks
              • Book Recommendation

                Understanding Comics: The Invisible Art (Goodreads) by Scott McCloud

                Thanks to Our Sponsors!

                Your test coverage says 90%, but that might be misleading. Undercover CI looks at your Ruby pull requests and shows you which parts of your changes weren't tested- not just overall coverage, but what changed and what got missed, down to the method level. Visit undercover-ci.com and use code MAINTAINABLE for 15% off your first billing cycle. Free for public repos. Private repos with unlimited users also available.

                Turn hours of debugging into just minutes! AppSignal is a performance monitoring and error-tracking tool designed for Ruby, Elixir, Python, Node.js, Javascript, and other frameworks. It offers six powerful features with one simple interface, providing developers with real-time insights into the performance and health of web applications. Keep your coding cool and error-free, one line at a time! Use the code maintainable to get a 10% discount for your first year. Check them out!

                Subscribe to Maintainable on:

                • Apple Podcasts
                • Spotify

                Or search "Maintainable" wherever you stream your podcasts.

                Keep up to date with the Maintainable Podcast by joining the newsletter.

                55 min
              • Sally Lait: Confidence Is the Real Metric

                Sally Lait joins Robby Russell on Maintainable to explore software maintainability through a different lens… not just code quality, but how teams work together over time.

                Sally is a fractional technology leader and advisor with more than two decades in the industry. You can follow her on LinkedIn or Mastodon.

                They start with a familiar question: what makes software well maintained? Structure and standards matter, but Sally shifts the focus to signals around the edges… documentation, onboarding speed, knowledge sharing, and especially how confident people feel making changes.

                That confidence becomes the thread throughout the conversation.

                Teams with high confidence move faster and adapt more easily. Teams with low confidence hesitate, avoid parts of the system, and struggle to make progress… regardless of what the code looks like.

                Robby and Sally also dig into why maintenance work often struggles to get traction. It rarely speaks for itself. Leaders need to connect it to outcomes the business already cares about… risk, hiring, delivery speed, and long-term sustainability.

                Sally references a LeadDev panel she moderated on why maintenance still feels “stuck in 2015”: Why Software Maintenance Is Stuck in 2015.

                They also discuss modernizing legacy systems and moving away from long-standing in-house software… work that is rarely just technical. It requires trust, clear communication, and navigating the emotional attachment teams have to what they’ve built.

                The episode closes with advice for engineers joining older codebases: stay curious, build relationships early, and use onboarding gaps as opportunities to improve things for the next person.

                Episode Highlights

                [00:01:02] What Makes Software Maintainable: Technical quality matters, but cultural signals often tell the deeper story.

                [00:05:45] Why Progress Still Feels Slow: Even with improvements, teams can feel stuck due to perception gaps.

                [00:07:30] Communicating Small Wins: Lack of visibility into incremental progress impacts morale and confidence.

                [00:12:40] Influencing Without Manipulating: Maintenance work needs to be framed in business terms.

                [00:16:00] Technical Debt as a Hiring Problem: Outdated systems affect recruiting and retention.

                [00:20:22] Modernizing a Siloed System: Unlocking legacy data required both technical and organizational change.

                [00:26:55] Building Trust for Change: Surprise proposals fail… alignment takes time.

                [00:32:39] Letting Go of “Our Baby”: Replacing systems involves emotional and cultural dynamics.

                [00:46:25] Joining an Older Codebase: Practical advice for onboarding and building confidence quickly.

                Resources Mentioned
                • Sally Lait
                • Sally Lait on LinkedIn
                • Sally Lait on Mastodon
                • Why Software Maintenance Is Stuck in 2015 (LeadDev Panel)
                • Lara Hogan
                • The Murderbot Diaries by Martha Wells
                • Death of the Author by Nnedi Okorafor
                • Sally’s Reading & Reviews Site
                • Thanks to Our Sponsors!

                  Your test coverage says 90%, but that might be misleading. Undercover CI looks at your Ruby pull requests and shows you which parts of your changes weren't tested- not just overall coverage, but what changed and what got missed, down to the method level. Visit undercover-ci.com and use code MAINTAINABLE for 15% off your first billing cycle. Free for public repos. Private repos with unlimited users also available.

                  Turn hours of debugging into just minutes! AppSignal is a performance monitoring and error-tracking tool designed for Ruby, Elixir, Python, Node.js, Javascript, and other frameworks. It offers six powerful features with one simple interface, providing developers with real-time insights into the performance and health of web applications. Keep your coding cool and error-free, one line at a time! Use the code maintainable to get a 10% discount for your first year. Check them out!

                  Subscribe to Maintainable on:

                  • Apple Podcasts
                  • Spotify

                  Or search "Maintainable" wherever you stream your podcasts.

                  Keep up to date with the Maintainable Podcast by joining the newsletter.

                  56 min
                • Rein Henrichs: The Real Work of Maintenance Happens Before You Touch the Code

                  Software maintenance is often framed as a technical problem. Refactoring code, fixing bugs, or upgrading dependencies. In this conversation, Robby Russell talks with Rein Henrichs about a different lens, one centered on understanding.

                  Rein is a Principal Software Engineer at Procore, where he works within a large, long-lived system used across the construction industry. Rather than focusing on tooling, Rein emphasizes that well-maintained software is software that makes sense to the people maintaining it.

                  To explain this, Rein introduces the idea of the line of representation, drawing on the work of Richard Cook. Engineers do not interact directly with systems. They rely on representations such as logs, dashboards, and code. These are approximations, not reality, echoing ideas from Plato’s Allegory of the Cave.

                  When those representations break down, teams lose shared understanding, what Rein describes as “common ground.” This often shows up as weak signals. Subtle indicators that something is not quite right. They are easy to ignore, but over time they lead to confusion and slower decision-making.

                  Incidents make this especially visible. Rein explains how teams build alignment under pressure, highlighting that the role of an incident commander is coordination, not control. Clear communication matters as much as technical correctness.

                  The conversation also explores how large systems behave in practice. They rarely fail completely. Instead, they degrade in multiple ways at once. While SLOs can help teams respond to customer-facing issues, they do not capture internal clarity or alignment.

                  Rein references W. Edwards Deming to highlight a common trap. Not everything that matters can be measured. High-performing teams often rely on judgment, experience, and shared context.

                  Toward the end, Rein connects these ideas to The Field Guide to Understanding Human Error by Sidney Dekker, challenging the idea that incidents are simply caused by mistakes. Instead, they emerge from the same behaviors that usually lead to success, just under different conditions.

                  For teams working in complex systems, the takeaway is straightforward. Maintaining software depends on maintaining understanding.

                  Links & Resources
                  • Procore
                  • Rein Henrichs on LinkedIn
                  • Concepts & References
                    • How Complex Systems Fail – Richard Cook
                    • The Field Guide to Understanding Human Error – Sidney Dekker
                    • W. Edwards Deming
                    • Gerald Weinberg – Secrets of Consulting
                    • Referenced in this Conversation
                      • Kent Beck: You’re Ignoring Optionality and Paying for It
                      • Charity Majors: Deploys Are Just the Beginning
                      • Heidi Helfand: The Art and Wisdom of Changing Teams
                      • Thanks to Our Sponsor!

                        Turn hours of debugging into just minutes! AppSignal is a performance monitoring and error-tracking tool designed for Ruby, Elixir, Python, Node.js, Javascript, and other frameworks.

                        It offers six powerful features with one simple interface, providing developers with real-time insights into the performance and health of web applications.

                        Keep your coding cool and error-free, one line at a time! 

                        Use the code maintainable to get a 10% discount for your first year. Check them out!

                        Subscribe to Maintainable on:

                        • Apple Podcasts
                        • Spotify

                        Or search "Maintainable" wherever you stream your podcasts.

                        Keep up to date with the Maintainable Podcast by joining the newsletter.

                        56 min

                      About Maintainable

                      From the publisher's feed

                      Do you feel like you're hitting a wall with your existing software projects? Are you curious to hear how other people are navigating this? You're not alone.

                      More shows like Maintainable

                      Soft Skills Engineering by Jamison Dance and Dave Smith

                      Soft Skills Engineering

                      286 Listeners

                      The Daily by The New York Times

                      The Daily

                      111,876 Listeners

                      Syntax - Tasty Web Development Treats by Wes Bos & Scott Tolinski - Full Stack JavaScript Web Developers

                      Syntax - Tasty Web Development Treats

                      983 Listeners

                      Darknet Diaries by Jack Rhysider

                      Darknet Diaries

                      8,063 Listeners

                      CoRecursive: Coding Stories by Adam Gordon Bell - Software Developer

                      CoRecursive: Coding Stories

                      188 Listeners

                      Hard Fork by The New York Times

                      Hard Fork

                      5,548 Listeners