The Pragmatic Embedded Podcast

The Pragmatic Embedded Podcast

By Luca Ingianni, Jeff GableTechnology
Download on the App Store

The Pragmatic Embedded Podcast episodes

  • Testing Embedded Systems in the Cloud and On-Prem with Marie Meyer and Joel Schulz-Andres

    Most embedded teams run unit tests on the host and integration tests on real hardware (possibly manually), with very little in between. Marie Meyer and Joel Schulz-Andres founded OnMCU to fill that gap, and we talk with them about what belongs there. Joel's approach begins with "on-target unit tests" on bare dev boards, which catch clock misconfigurations and peripheral errata before your own board exists, and we look at what changes once it does.

    Fault injection is the kind of testing most teams never get to, because the rig for it is effectively a second embedded product built alongside the first. We talk about how much of that work you can avoid. Luca suggests injecting faults further up the software stack, wherever your architecture has a seam for it. Joel's rule is to move a test up the hierarchy wherever you can afford the latency. We also discuss the point at which cloud boards stop being enough and the test hardware has to sit next to your own boards.

    If you've ever built a test rig yourself, some of what went wrong for them will sound familiar. They bought USB hubs that claim per-port power switching and don't have it, rack drawers that waste half the space, and looked for a cable chain that doesn't exist. And quite apart from technology, there's the quintessentially German pushback OnMCU has to overcome when selling their cloud-based HILs: "If you don't want a dev board on your desk, you shouldn't have studied electrical engineering."

    Key Topics
    • [00:00] Marie and Joel introduce themselves and OnMCU
    • [01:58] An FFT that "needed an FPGA", and why you can't rent a dev board the way you rent a Mac
    • [06:39] On-target unit tests: catching clock and peripheral bugs on bare dev boards
    • [13:21] Moving from cloud boards to your own hardware, and tunnelling I2C/SPI to a remote sensor shield
    • [16:19] Reusing the test rig for end-of-line testing in production
    • [18:38] Why generic fault injection is hard: open-drain vs push-pull, 1.8/3.3/5 V, bus speeds
    • [23:40] Cloud or on-prem: maintenance, cost, firmware confidentiality, network isolation via 5G
    • [27:57] What they got wrong: cloud scepticism, USB hubs that lie about power switching, rack drawers and cable chains
    • [35:55] The on-prem box and the interposer in detail
    • [38:32] A gap between home-built HIL rigs and dSPACE-priced systems
    • [41:05] Munich Embedded on 27 October: technical talks, no pitches
    • [44:47] How they ran the first conference for under 10k with 80 people
    Notable Quotes

    "There's a blog post from 1999 about daily builds being essential - that was 27 years ago, and in embedded, some companies are not doing daily builds today. There's really this continuity of testing that's missing." — Joel Schulz-Andres

    "Whenever we had to build something from scratch, it felt great. Whenever we had to depend on others - we're standing on the shoulders of giants, but a lot of times you have to really work around it to get something working." — Joel Schulz-Andres

    "Testing in the cloud is really a mindset shift, especially in Germany. People told us: if you don't want to have a dev board on your table, you shouldn't study electrical engineering." — Marie Meyer

    Resources Mentioned
    • OnMCU - Cloud-based and on-premises testing platform for embedded systems with error injection capabilities
    • Munich Embedded Conference - Technical conference in Munich on October 27, 2026, featuring talks on embedded engineering and networking
    • The Joel Test - Joel Spolsky's influential 1999 blog post outlining 12 essential software development practices
    • Goertzel Algorithm - Frequency-selective FFT algorithm mentioned for efficient spectral analysis on resource-constrained MCUs
    • Agile Embedded Podcast Slack - Community discussion channel where you can reach the hosts and connect with other listeners

    Jeff runs BaseRef, a Requirements Management and Documentation tool for medical device startups, at https://getbaseref.com.
    You can find Luca at https://luca.engineer.

    Want to join the Pragmatic Embedded Slack? Click here

    Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/
    Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

    46 min
  • TDD: The Hard Parts - Getting Started with Test-Driven Development

    We're back talking about Test-Driven Development, just 10 episodes after our last TDD discussion. This time, Jeff and Luca tackle the practical challenges that trip up developers trying to adopt TDD - especially that awkward moment staring at a blank screen wondering "what do I even type first?"

    Drawing from Luca's experience training embedded developers, we explore why TDD feels so unnatural at first and how to actually get started. We break down the process: from sketching test ideas as bullet points, to writing test outlines in plain English, to finally implementing actual test code. Along the way, we discuss how LLMs change the game - both helping overcome the fear of the empty screen and creating new traps when they generate overwhelming amounts of test code. Whether you're coding by hand or working with AI assistants, the key is maintaining control through deliberate, incremental steps and keeping your focus on module interfaces and test quality over implementation details.

    Key Topics
    • [02:30] The common struggle: Why developers abandon TDD after trying it once
    • [05:15] TDD is not a design methodology - it comes after decomposition
    • [08:45] The mental leap: Calling functions that don't exist yet
    • [12:00] Starting with test ideas as bullet points, not full test implementations
    • [18:30] Two traps: Coding by hand (fear of empty screen) vs. using LLMs (overwhelming output)
    • [25:00] Turning test statements into questions for effective coding spikes
    • [32:15] Review test code more thoroughly than production code when using LLMs
    • [36:00] Connecting TDD to higher-level specs and requirements
    Notable Quotes

    "TDD is really just a Jedi mind trick of thinking about your tasks in a different way that makes it easier for you to come to good answers." — Luca Ingianni

    "TDD is not how you build the map. It's how you have confidence in a quality implementation and protection against future regressions." — Jeff Gable

    "I've gotten into the habit of reviewing my test code more thoroughly than the actual production code when using LLMs. Because the test code really tells me the shape of the production code." — Luca Ingianni

    Resources Mentioned
    • Luca Ingianni's website - Link site with information about Luca's trainings, coaching, and contact details
    • BaseRef - Requirements, test, and documentation management tool for medical device startups founded by Jeff
    • Episode 92 - Previous TDD episode discussing test-driven development in the age of AI
    • Jacob Beningo's work - Resources on writing simulators and making business logic data-driven rather than hardware-driven
    • Agile Embedded Podcast Slack - Community discussion channel where you can reach the hosts and connect with other listeners

    Jeff runs BaseRef, a Requirements Management and Documentation tool for medical device startups, at https://getbaseref.com.
    You can find Luca at https://luca.engineer.

    Want to join the Pragmatic Embedded Slack? Click here

    Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/
    Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

    46 min
  • The Big Rename: From Agile to Pragmatic

    After five years and 101 episodes, we're doing something we never thought we'd do: renaming the podcast. The Agile Embedded Podcast is now the Pragmatic Embedded Podcast: having the title reflect where we've been heading for quite a while.

    Jeff and Luca discuss why the word "agile" has become cringe-worthy, how the podcast has naturally evolved beyond its original scope, and what this means for future episodes. We talk about the reality of AI adoption in embedded development (spoiler: it's not as widespread as you might think), the importance of engineering fundamentals in an AI-assisted world, and why we're excited to explore topics without feeling constrained by our old name. This is still the same podcast you know, just with a name that finally fits what we actually do: exploring pragmatic approaches to building embedded systems.

    Key Topics
    • [00:00] The big reveal: Agile Embedded becomes Pragmatic Embedded
    • [02:30] Why "agile" has become cringe and why the podcast has outgrown its original name
    • [04:15] How past episodes like the QP framework discussion already pointed beyond agile
    • [06:00] The reality check: AI adoption in embedded development isn't as widespread as the hype suggests
    • [09:45] Jeff's perspective: Every developer is becoming a lead developer managing AI-generated code
    • [12:30] What's next: More freedom to explore what's actually useful for embedded developers
    Notable Quotes

    "I have literally been cringing internally for a long time saying Agile, Agile, Agile. And I'm just really looking forward to broadening the focus and not feeling bad about it." — Jeff

    "My customers never buy the curly brackets, do they? There's a lot of good old-fashioned engineering craft that we can talk about and that we'll continue to talk about." — Luca

    "Writing curly braces is going the way of being dead. But software engineering fundamentals, essentially, I see the trend... every developer is going to be essentially what is now a lead developer." — Jeff

    Resources Mentioned
    • Embedded AI Podcast - Luca's other podcast with Ryan Torvik, focused on AI tools in embedded development
    • Luca Ingianni's website - Training and consulting on AI and embedded systems
    • BaseRef - Jeff's requirements and test management tool for medical device startups
    • Agile Embedded Podcast Slack - Community discussion channel where you can reach the hosts and connect with other listeners

    Jeff runs BaseRef, a Requirements Management and Documentation tool for medical device startups, at https://getbaseref.com.
    You can find Luca at https://luca.engineer.

    Want to join the Pragmatic Embedded Slack? Click here

    Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/
    Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

    17 min
  • 100 Episodes! Break Out the Champagne! A Retrospective

    We flip the script for episode 100! Instead of hosting, Luca and Jeff become the guests as Joe Schneider and Jacob Beningo take over the interview chair. We look back at how two consultants from opposite sides of the world accidentally started a podcast, share war stories about the hardest bugs we've ever chased (spoiler: weeks of work for one-line fixes), and discuss why "Agile" might not be the buzzword it once was. The conversation touches on everything from helicopter avionics to the future of AI in embedded systems, plus some honest talk about what keeps us going after five and a half years and 100 episodes.

    Key Topics
    • [02:30] Origin story: How a consulting group for solopreneurs brought together two embedded engineers from California and Munich
    • [08:00] Favorite episodes and memorable guests: James Grenning, Miro Samek, Matthew Eshelman, and the value of reaching out to people you admire
    • [15:00] Why "Agile Embedded"? Finding an unclaimed niche at the intersection of Agile practices and embedded systems
    • [25:00] War stories: The hardest bugs to fix - from five-day hunts for one-line interrupt bugs to months tracking down temperature-dependent DDR RAM failures
    • [35:00] The dynamic between hosts: Jeff keeps things concrete, Luca pulls back to the big picture - and why that balance matters
    • [42:00] Looking ahead: AI as "the new Agile," requirements engineering's renewed importance, and navigating the hype cycle
    • [50:00] The Gartner hype cycle and AI: Are we heading for the trough of disillusionment, or is this an exponential that never stops?
    • Notable Quotes

      "No one gives you permission. You're listening to us because we started talking, and maybe some of you found it interesting. But you can put yourself out there, and you learn by speaking." — Jeff Gable

      "Testing is like vacuuming. It's only fun if it rattles in the hose. It's just no fun to test a system that has no bugs." — Luca Ingianni

      "AI really is the new Agile - the magical fairy dust that you can sprinkle over your team. It'll take a while for people to realize that this is also much harder than it looks." — Luca Ingianni

      Resources Mentioned
      • Embedded Vibe - Joe Schneider's benchmarking site testing LLMs' ability to generate embedded code in one shot
      • Beningo Embedded Group - Jacob Beningo's consulting and training services for embedded systems development
      • Dojo Five - Joe Schneider's embedded systems consulting and development company
      • Embedded Systems Summit - In-person conference organized by Jacob Beningo and Stefan, planned for mid-November in the Bay Area
      • Agile Embedded Podcast Slack - Community discussion channel where you can reach the hosts and connect with other listeners
      • Jeff runs BaseRef, a Requirements Management and Documentation tool for medical device startups, at https://getbaseref.com.
        You can find Luca at https://luca.engineer.

        Want to join the Pragmatic Embedded Slack? Click here

        Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/
        Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

        1 hr
      • From Vision to Reality: Product Roadmaps in Embedded Development

        In the third installment of our requirements engineering series, Luca and Jeff shift focus from requirements themselves to the bigger picture: how do you actually turn ideas into working products? We explore the journey from product vision through roadmaps to backlogs, discussing what makes a good roadmap and—perhaps more importantly—the surprisingly common mistakes that even experienced teams make.

        We dig into practical challenges like the "progress by PowerPoint" trap (putting milestones like "specification complete" on your roadmap instead of actual value delivery), the dangers of over-committing to dates before you understand the problem space, and why most organizations struggle with prioritization. Luca shares insights from his agile requirements engineering training, including a clever trap he sets for participants around roadmap design. We also tackle the iron triangle of project management and why varying resources is often the least effective lever to pull when a project is running late. Whether you're a product manager, engineering lead, or startup founder, this episode offers a grounded perspective on planning that respects both ambition and reality.

        Key Topics
        • [02:15] The hierarchy of planning artifacts: from product vision to roadmaps to backlogs
        • [03:30] What makes a good product vision (Kennedy's moon landing speech as the gold standard)
        • [06:45] Roadmaps as alignment and discovery tools, not static commitments
        • [11:20] The prototyping phase and managing uncertainty in early-stage planning
        • [15:40] Common mistake #1: Putting the wrong things on roadmaps ("specification complete" vs. actual value milestones)
        • [21:30] The danger of over-committing to dates before understanding the problem (especially in VC-funded startups)
        • [26:15] Tailoring roadmaps for different audiences (internal, customer-facing, investor-facing)
        • [30:45] The failure to prioritize and its impact on roadmap execution
        • [35:20] Finding the right level of granularity: the iceberg principle for roadmaps
        • [40:10] The iron triangle of project management and why varying resources is the least effective lever
        • [43:30] Who should own the roadmap (product management's role in synthesis and collaboration)
        • Notable Quotes

          "If somebody has visions they should go see a doctor—but still, you know, it makes sense to have such a thing. A product vision should be nice and compact, like Kennedy's moon landing speech: what we're doing, the timeline, and the value." — Luca Ingianni

          "I don't care about the stupid presentation. Show me a product increment. Show me captured value. Then we're talking. Anything else is just smoke and mirrors." — Luca Ingianni

          "Plans are useless but planning is indispensable. That process of getting all the key players in the same room and trying to hash out this schedule is where you will uncover the biggest uncertainties and risks." — Jeff Gable

          Resources Mentioned
          • Agile Embedded Podcast Episode 31 - Interview with John Odo on product roadmaps (May 2022)
          • Luca's Agile Requirements Engineering Training - Training course covering requirements, roadmaps, and product planning
          • BaseRef - Jeff's new product for requirements and test management for medical device startups
          • The Mythical Man-Month - Classic book on software project management discussing why adding people to late projects makes them later
          • Agile Embedded Podcast Slack - Community discussion channel where you can reach the hosts and connect with other listeners
          • Jeff runs BaseRef, a Requirements Management and Documentation tool for medical device startups, at https://getbaseref.com.
            You can find Luca at https://luca.engineer.

            Want to join the Pragmatic Embedded Slack? Click here

            Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/
            Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

            44 min
          • Factory Firmware Flashing with Pete Staples

            We talk with Pete Staples, founder of Blue Clover Devices, about the often-overlooked challenge of flashing firmware in production. Pete shares insights from running a contract manufacturing operation in Shenzhen and explains why the handoff from engineering to manufacturing is more like "hucking it over a fence" than a smooth relay race.

            We explore the gap between engineers' assumptions about factory capabilities and the dusty reality of production floors. Pete discusses security challenges, the complexity of modern microcontroller programming, and how Blue Clover's Production Line Tool addresses the middle ground between expensive custom automation and ad-hoc bench setups. We also touch on provisioning, calibration workflows, and why the engineer who designs the product must also define how it's tested.

            Key Topics
            • [02:30] The reality of factory firmware flashing - dusty PCs, hot glue, and cables everywhere
            • [06:15] Security challenges: managing sensitive firmware and the "glass room" solution
            • [09:45] The gap between engineer assumptions and factory reality - no, they don't have better equipment than you
            • [14:20] In-circuit testing and bed-of-nails fixtures explained
            • [22:30] The Production Line Tool: standardizing hardware and software across engineering and factory
            • [28:00] Recording what matters: firmware versions, hardware serial numbers, and test results per device
            • [31:45] Provisioning and security: webhooks, cloud databases, and managing secrets in production
            • [38:20] The Test Agent: a companion device for running third-party software and complex programming workflows
            • [43:00] Who should write the test plan? Why engineers must define "good enough" before production
            • Notable Quotes

              "Engineers assume that the factories are a lot more sophisticated than they really are. In reality, it's a lot more like just hucking it over a fence and just hoping there's somebody there waiting." — Pete Staples

              "They show you their pick-and-place machine and 10-zone reflow oven, and you're like, 'wow, these guys are tipped off.' And then rarely do they say, 'oh, and here's where we do firmware flashing.' It's normally another floor of the building, dimly lit, dusty old PCs." — Pete Staples

              "The engineer responsible for the product has to not only engineer the product, but how it's tested. They can't just say, 'here's a bunch of design files, build it and let's see what happens.'" — Pete Staples

              Resources Mentioned
              • Blue Clover Devices - Pete's company specializing in factory firmware flashing solutions
              • Embedded World (Nuremberg) - Annual trade show in March where Blue Clover exhibits
              • Embedded World North America (Anaheim) - North American version of Embedded World, September 22nd
              • Kinetic (San Francisco) - Hardware-focused event put on by Hardware FYI
              • Jeff runs BaseRef, a Requirements Management and Documentation tool for medical device startups, at https://getbaseref.com.
                You can find Luca at https://luca.engineer.

                Want to join the Pragmatic Embedded Slack? Click here

                Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/
                Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

                51 min
              • Requirements Engineering, part 2: A Practical Process for Safety-Critical Development
                Requirements Engineering Part 2: A Practical Process for Safety-Critical Development

                In this second part of our requirements engineering series, Jeff walks us through his preferred process for developing safety-critical products, particularly medical devices. We explore the crucial distinction between prototyping and design-controlled development, discussing when to start formal requirements work and how to keep your first version minimal yet complete.

                Jeff emphasizes the importance of deeply fleshing out requirements before implementation—including error handling, which often comprises 70% of a product. We discuss tracer bullets as a development strategy, the value of writing test cases alongside requirements, and why tracking requirements completion gives you honest project status. Luca and Jeff also debate the finer points of MVPs versus prototypes, and Jeff announces his upcoming requirements management tool for medical device startups.

                Key Topics
                • [00:00] Introduction and listener feedback on Part 1
                • [02:30] The prototyping phase: answering 'can we build it?' and 'should we build it?' before design controls
                • [06:00] Luca vs. Jeff: The great MVP and prototype debate
                • [12:00] Starting the V-model: minimal but deeply fleshed-out requirements for V1
                • [18:00] Error handling is 70% of your product—don't skip it in requirements
                • [22:00] When to write test cases: early, alongside requirements
                • [26:00] War story: the SATCOM system that needed a satellite slot in five years
                • [32:00] Project management: tracking requirements completion for honest status updates
                • [38:00] Tracer bullets: vertical slices through all layers, not horizontal completion
                • [45:00] Jeff's upcoming requirements and test management tool for medical device startups
                • Notable Quotes

                  "Error handling is 70% of your product, if not more. If you only do the 30% of the requirements for the happy path, you're fooling yourself." — Jeff

                  "If you don't get this right at the outset, it will haunt you through the entirety of your product development process." — Luca

                  "Paper is the best place to figure it out. Actually think through the requirements rigorously before you start building." — Jeff

                  Resources Mentioned
                  • Agile Embedded Slack Channel - Community discussion space for embedded development topics
                  • Matt Pocock's YouTube Channel - AI for serious engineers, discusses tracer bullets and AI-assisted development
                  • The Pragmatic Programmer - Classic software development book, source of the tracer bullet concept
                  • The Art of Unix Programming by Eric S. Raymond - Referenced for the chapter 'Don't just do something. Stand there.'
                  • Embedded.fm Episode 440: Condemned to Being Perfect - Crossover episode with Elecia White and Christopher White, where bootloaders and update strategies came up
                  • Jeff's Requirements Management Tool - Upcoming requirements, risk, and test management tool for medical device startups
                  • Jeff runs BaseRef, a Requirements Management and Documentation tool for medical device startups, at https://getbaseref.com.
                    You can find Luca at https://luca.engineer.

                    Want to join the Pragmatic Embedded Slack? Click here

                    Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/
                    Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

                    51 min
                  • Fuzzing and Dynamic Analysis for High-Integrity Software with Paul Butcher
                    Fuzzing and Dynamic Analysis for High-Integrity Software with Paul Butcher

                    We sit down with Paul Butcher, Unit Director of Dynamic Analysis at AdaCore, to explore verification techniques beyond basic compliance in safety-critical software. Paul shares his experience from Eurofighter to automated trains, explaining how dynamic analysis—from unit testing to coverage analysis to fuzzing—helps find bugs that traditional testing misses.

                    The conversation dives deep into fuzzing: how it works, why it's so effective at finding corner-case bugs (even in well-tested systems), and the challenges of applying it to embedded systems with timing constraints. Paul introduces an intriguing approach that combines static analysis with targeted fuzzing to automatically triage false positives and generate reproducers. We also touch on formal verification, the role of LLMs in verification workflows, and why the simplest software is often the safest. Whether you're working in aerospace, medical devices, or any safety-critical domain, this episode offers practical insights into building more robust systems.

                    Key Topics
                    • [02:30] Paul's background in high-integrity embedded systems: Eurofighter, rail, drones, and AdaCore's dynamic analysis tools
                    • [05:00] Dynamic vs. static analysis: executing code to observe real behavior across different environments
                    • [08:15] How fuzzing works: mutation engines, anomaly detection, and finding bugs through negative testing
                    • [14:20] Challenges of fuzzing timing and concurrency bugs in embedded systems
                    • [17:45] Real-world success: fuzzing the NH90 avionics via the MIL bus uncovered numerous bugs
                    • [22:30] Safety standards (DO-178C, SIL levels) and objective-based approaches vs. checkbox compliance
                    • [28:00] Determining 'enough' fuzzing: coverage, input space complexity, and building certification arguments
                    • [32:15] Combining static analysis with targeted fuzzing to automatically triage false positives and generate reproducers
                    • [38:45] Symbolic execution and theorem provers: breaking through complex branch conditions in fuzzing campaigns
                    • [42:00] Shift-left philosophy: building verifiable software from the start with testing and analysis tools
                    • [47:30] Formal verification in practice: London Underground's Victoria line uses SPARK-proven emergency braking
                    • [51:00] LLMs in verification: cautious adoption for report analysis, but determinism remains critical for core tools
                    • [54:30] High Integrity Software Conference (HISC) in Birmingham, October 2026
                    • Notable Quotes

                      "Software testing is typically about, is it functionally correct? Fuzzing is like a negative testing technique. It's the inverse of that. It fires random inputs into your system with the intent of finding anomalies." — Paul Butcher

                      "Every time I speak to someone who's tried fuzzing, even if it's a system that's considered high integrity with a high level of assurance, they always find something. It's really good at eking out those weird corner case scenarios." — Paul Butcher

                      "With testing you would like to prove the absence of bugs, but unfortunately you can't. So you have to settle for a very distant second place of proving the presence of bugs." — Luca Ingianni

                      Resources Mentioned
                      • Paul's paper on fuzzing in safety-critical contexts - Detailed discussion of how to argue 'enough' fuzzing for certification
                      • High Integrity Software Conference (HISC) - Annual conference in Birmingham, UK (October 2026) covering high-integrity software across industries
                      • AdaCore Dynamic Analysis Tools - Coverage, fuzzing, and unit testing solutions for high-integrity software
                      • SPARK formal verification - Formal proof technology used in London Underground's Victoria line emergency braking
                      • AFL++ - Successor to the discontinued AFL (American Fuzzy Lop): Fuzzing technology mentioned as capable of quickly finding the Heartbleed bug
                      • Jeff runs BaseRef, a Requirements Management and Documentation tool for medical device startups, at https://getbaseref.com.
                        You can find Luca at https://luca.engineer.

                        Want to join the Pragmatic Embedded Slack? Click here

                        Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/
                        Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

                        49 min
                      • Linux Profiling with Mohammed Billoo
                        Linux Profiling with Mohammed Billoo

                        We sit down with Mohammed Billoo, founder of Mab Labs and author of the Embedded Linux Essentials Handbook, to explore the world of embedded Linux profiling and optimization. Mohammed shares hard-won lessons from the field, including debugging a scientific instrument that mysteriously crashed after 60-minute runs and optimizing a sophisticated MANET platform that took a 20% throughput hit.

                        The conversation reveals a fundamental truth: in embedded Linux, the CPU is rarely the bottleneck. Mohammed walks us through his systematic approach to performance problems, starting with simple tools like HTOP before diving into specialized instrumentation. We discuss the critical difference between VM size and VM RSS for memory analysis, why dumping console output can kill boot times, and how to leverage kernel configurations for maximum diagnostic bang-for-buck. Mohammed emphasizes the importance of building instrumentation into systems from day one—not for premature optimization, but to give your future self the data needed when problems inevitably surface. The discussion also touches on how LLMs can accelerate the learning curve for complex tools like Valgrind and perf, while stressing that physical reality remains the ultimate arbiter of system performance.

                        Key Topics
                        • [03:15] The surface area problem: why embedded Linux profiling requires a tool chest, not just a toolbox
                        • [06:30] Case study: debugging a scientific instrument that crashed after 60-minute runs
                        • [08:45] VM size vs. VM RSS: understanding the critical difference in memory analysis
                        • [14:20] Why the CPU is rarely the bottleneck: coprocessors, DMA, and crypto engines
                        • [18:50] Essential kernel configurations: function tracer, perf, and config kallsyms
                        • [24:10] File system bottlenecks: moving from CSV files to SQLite for data integrity
                        • [28:40] Boot time optimization: why console output is one of the biggest time sinks
                        • [32:15] Premature optimization vs. smart instrumentation: building in diagnostic capability from day one
                        • [38:25] Leveraging LLMs for visualization and analysis of perf data and Valgrind output
                        • [43:50] The first five commands: starting with HTOP and working down to specialized tools
                        • Notable Quotes

                          "When you first get started, you have generally this arrogance that like, oh, it works fine. I've tested it. It's good to go. But then as you get more experience, as you become a more senior-level engineer, that arrogance, you start to kind of strip away a lot of that arrogance. You get humbled pretty quickly." — Mohammed Billoo

                          "The CPU is very rarely the bottleneck because it's meant to, and the drivers are implemented in Linux in such a way that they're intelligent enough that they can hand off a lot of the things of the CPU to coprocessors so that the CPU is really idle." — Mohammed Billoo

                          "I don't convince myself of a claim that I'm making until I have data to back it up. So I don't say, oh, you know, this is working fine. Like, well, again, what does fine mean? Or, you know, what does well mean? And what is the data to prove that?" — Mohammed Billoo

                          Resources Mentioned
                          • HTOP - Interactive process viewer for Linux - Mohammed's first tool for getting a high-level view of system performance
                          • perf - Linux profiling tool with performance counters - requires kernel configuration to enable
                          • LTTng - Linux Trace Toolkit Next Generation - provides visibility across both user space and kernel space
                          • Valgrind - Memory debugging and profiling tool for detecting memory leaks
                          • iperf - Network throughput measurement tool with server and client components
                          • GStreamer - Multimedia framework with built-in tools for per-frame timestamp analysis
                          • Tracealyzer - Visualization tool for LTTng and other performance data
                          • SQLite - Embedded database recommended for data integrity over CSV files in embedded systems
                          • Embedded Linux Essentials Handbook - Mohammed Billoo's book published by Packt
                          • Mab Labs - Mohammed Billoo's embedded solutions consultancy with blog on embedded Linux topics
                          • Jeff runs BaseRef, a Requirements Management and Documentation tool for medical device startups, at https://getbaseref.com.
                            You can find Luca at https://luca.engineer.

                            Want to join the Pragmatic Embedded Slack? Click here

                            Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/
                            Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

                            47 min
                          • E94 Requirements Engineering, part 1: Fundamentals
                            Requirements Engineering Fundamentals - Part 1

                            We kick off a multi-part series on requirements engineering by exploring what requirements actually are and why they matter - even for Agilists. Jeff shares his medical device expertise while Luca brings his automotive and aerospace background to discuss the different levels of requirements (from high-level user needs to testable system requirements), the importance of traceability, and why proper tooling beats Word and Excel every time.

                            We dig into practical aspects like the EARS format for writing requirements, the crucial distinction between requirements and design choices, and why glossaries aren't as boring as they sound. Along the way, we tackle the tension between regulatory compliance and actual engineering value, emphasizing that documentation should be an artifact of diligent work - not the work itself. Whether you're in safety-critical industries or just want to build better products, understanding requirements engineering helps manage complexity and prevent costly mistakes.

                            Key Topics
                            • [02:30] What is requirements engineering and why it matters beyond safety-critical industries
                            • [06:45] Don't Agilists hate requirements? Debunking the myth and discussing iteration vs. waterfall
                            • [11:20] The hierarchy of requirements: user needs, system requirements, and subsystem requirements
                            • [18:00] Requirements vs. design choices: where to draw the line and why it matters for testing
                            • [24:15] Writing good requirements: EARS format, must vs. shall vs. may, and the value of glossaries
                            • [32:40] Traceability: linking requirements across levels and to test cases
                            • [40:30] Why Word and Excel don't cut it: the case for proper requirements management tools
                            • [48:20] Risk analysis and mitigation in safety-critical development
                            • [52:00] Documentation as artifact of diligent work, not the work itself
                            • Notable Quotes

                              "The whole agile movement was a reaction to the one time through the requirements specification build test loop that took several years. By the time you got to the end, the requirements no longer applied." — Jeff

                              "Do not use an LLM to manage requirements. Do use the LLM to write tools that help you manage requirements." — Luca

                              "I view any medical device that I work on as if it's going to be used on my child. What do I need to do to convince myself that it is safe and effective? Once I have done that, if there are remaining boxes to check to get it through FDA, I will check those boxes." — Jeff

                              Resources Mentioned
                              • EARS (Easy Approach to Requirements Syntax) - A grammar format for writing clear, verifiable requirements that constrains how requirements are written to reduce ambiguity
                              • FDA Guidance on Agile Development - Regulatory guidance describing how to do Agile development in medical device context
                              • ISO 26262 - Automotive safety standard mentioned as having similar traceability requirements to medical devices
                              • DO-178B - Aerospace software safety standard with similar requirements engineering principles
                              • Jeff runs BaseRef, a Requirements Management and Documentation tool for medical device startups, at https://getbaseref.com.
                                You can find Luca at https://luca.engineer.

                                Want to join the Pragmatic Embedded Slack? Click here

                                Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/
                                Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

                                48 min

                              About The Pragmatic Embedded Podcast

                              From the publisher's feed

                              Pragmatic engineering for embedded systems.

                              More shows like The Pragmatic Embedded Podcast

                              The Amp Hour by The Amp Hour (Chris Gammell and David L Jones)

                              The Amp Hour

                              228 Listeners

                              Embedded by Logical Elegance

                              Embedded

                              190 Listeners