The Pragmatic Embedded Podcast

• [05:30] Cross-disciplinary mobbing with firmware and hardware engineers in IoT development
• [12:45] Addressing efficiency objections: Local vs global optimization in embedded teams
• [18:20] DevOps collaboration patterns and reducing friction between software and infrastructure teams
• [25:10] Selling mob programming to management: Building trust through continuous delivery
• [35:40] Dynamic team composition and the law of personal mobility in technical teams
• [42:15] Coordinating mob teams with waterfall processes and hardware gate systems
• [48:30] Automated firmware protocol documentation and testing infrastructure

"When you initially hear about mobbing, you think everybody's inactive, but there's a lot of excess thought capacity around programming that can happen - somebody can be thinking about refactorings while others are typing the solution." — Chris Lucian

"If you're all there, that whole feedback loop can happen in 15 minutes as opposed to 15 days. It's like technical debt - you pay now or pay later, and if you pay later, it's going to be more expensive." — Austin Chadwick

"We tend to have multiple releases a day all the way to production. Most mobs will deliver to production at least twice a day - four hours of work and then deliver to prod." — Chris Lucian

• Mob Mentality Show - Podcast dedicated to mob programming practices and extreme programming techniques
• Kent Beck's Test-Driven Development Book - Foundational resource for understanding true TDD practices in embedded development
• Amy Edmondson's Teaming Book - Framework for understanding high uncertainty and high interdependence scenarios in technical teams
• Liminal Thinking - Mental framework for breaking through belief-based barriers in agile transformation

You can find Jeff at https://jeffgable.com.
You can find Luca at https://luca.engineer.

Want to join the Pragmatic Embedded Slack? Click here

Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/
Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

• [03:45] Hardware-software coupling challenges and decoupling strategies
• [08:20] Documentation overhead in safety-critical medical device development
• [12:15] Breaking down silos between electrical, mechanical, and software engineering teams
• [18:30] Hardware abstraction layers as database abstraction equivalents
• [22:10] Introduction to mob programming: all brilliant minds working together
• [28:45] Flow efficiency and eliminating handoffs in embedded product development
• [35:20] Optimal mob sizes and team organization strategies
• [42:15] Inverse Conway maneuver for architectural alignment
• [48:30] Getting started with mob programming in embedded teams

"If you can run unit tests on your host and run your embedded code in a simulator on your PC, that's a pretty good indicator that you've successfully decoupled it from the hardware." — Jeff Gable

"You need to understand that documentation is actually part of your product. It's not separate from your product made of curly brackets - it's one and the same thing." — Luca Ingianni

"Typing is not the bottleneck. It's the problem solving. This is a method of problem solving together faster." — Chris Lucian

"If you get the whole system in the room and in the same mob, things just start cranking. You're not waiting for anything because everyone you need is there." — Austin Chadwick

Find the Mob Mentality Show at https://mobmentalityshow.podbean.com/

You can find Jeff at https://jeffgable.com.
You can find Luca at https://luca.engineer.

Want to join the Pragmatic Embedded Slack? Click here

Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/
Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

In this episode of the Agile Embedded Podcast, Luca speaks with Violet Su, Business Development Manager at Seed Studio, about the challenges and opportunities in hardware manufacturing. 

The conversation explores the realities of hardware development, from initial prototyping to mass production. Violet emphasizes the importance of getting products into users' hands quickly to gather feedback, even if they're not perfect. She highlights common pitfalls for newcomers to hardware manufacturing, including underestimating costs, certification requirements, and supply chain complexities. The discussion also covers the growing influence of AI in hardware development and how emerging connectivity technologies are shaping the future of embedded systems.

• [01:30] Introduction to Seed Studio and its focus on open source hardware and AI
• [03:15] Changes in hardware development landscape over the past decade
• [06:45] Common challenges for hardware startups and innovators
• [12:30] The importance of early feedback and iterative development
• [16:20] Success story: OpenUC2 microscopy project collaboration
• [23:40] Indicators of a successful hardware project
• [28:15] Insights from Embedded World 2024 conference
• [33:10] Future trends in hardware beyond AI

"You might be able to design one and it really works when you have a sample or 10 samples, but when we talk about products, there are a lot of nuances that we really need to think about." — Violet Su

"You can fail faster so that you can succeed faster. We couldn't make a perfect project and we shouldn't expect to make a perfect product." — Violet Su

"When they give you the complaints early on, you will be very thankful, and they know you listened as well." — Violet Su

"The projects that might take off either really solve the pain of some people, or get people excited because it's something new." — Violet Su

"There's nothing better than getting customer complaints early on, so you don't get them later on." — Luca Ingianni

• Seed Studio - Company providing open source hardware and AI solutions
• OpenUC2 - Open source modular microscopy project mentioned as a success story

• Reach out to Violet Su at [email protected] to discuss potential hardware collaborations with Seed Studio
• Visit seedstudio.com to explore their range of AI hardware and customization services
• Connect with Violet Su on LinkedIn for updates on hardware manufacturing trends

You can find Jeff at https://jeffgable.com.
You can find Luca at https://luca.engineer.

Want to join the Pragmatic Embedded Slack? Click here

Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/
Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

Luca and Jeff dive into how AI tools can supercharge embedded development workflows. Luca shares his extensive hands-on experience, while Jeff brings a fresh perspective as someone just starting to incorporate these tools. They explore how AI can help with coding, testing, and debugging - while emphasizing that good software engineering judgment remains crucial. The conversation is particularly timely since AI tools are evolving rapidly, unlike their usually more "evergreen" podcast topics.

A key insight they discuss is that while AI tools offer amazing productivity boosts (much like IDEs did), they're not replacing experienced developers anytime soon. However, they raise interesting questions about the future job market for junior developers. Their take? Modern developers should absolutely embrace AI tools, but use them thoughtfully - especially when learning.

Key Timestamps and Topics:

00:00:00 - Welcome and episode overview

00:00:43 - Why this topic won't age well (but that's okay!)

00:02:20 - Breaking down different AI coding tools

00:08:00 - Deep dive into aider and workflow integration

00:20:02 - Using AI for testing and test-driven development

00:27:00 - AI-assisted architecture work and its limitations

00:30:40 - How AI helps explore unfamiliar codebases

00:33:30 - Debugging with AI - pros and cons

00:38:50 - What this means for development jobs

00:43:59 - Using AI to learn new frameworks/languages

00:46:15 - Embedded Online Conference preview

You can find Jeff at https://jeffgable.com.
You can find Luca at https://luca.engineer.

Want to join the Pragmatic Embedded Slack? Click here

Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/
Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

## Key Topics

* [00:30] Introduction to the listener's question about repository granularity in embedded development

* [01:15] The listener's approach: separate repositories for different work products in safety-critical industries

* [03:20] Luca's initial reaction and concerns about over-complication

* [05:45] Discussion of monorepo approaches and configuration management

* [08:10] The concept of micro-repositories and parallels to microservices

* [11:30] Using feature flags and CI pipelines instead of repository separation

## Notable Quotes

> "You're splitting something which ought to be joined together into different repositories and hiding whatever is happening within the repositories from the different users, from the different developers." — Luca Ingianni

> "The risk of course is that you will not spot divergence early enough because people just don't merge because it's a chore and because things might break, and of course that is the point - the earlier you notice that something breaks, the easier it will be to fix it." — Luca Ingianni

> "I'm willing to guarantee that you're going to get the architecture wrong at least on the first try. You think you're being really smart and you cut it up into a bunch of microservices or micro-repositories, and you're just going to get the boundaries wrong." — Luca Ingianni

> "I would opt for fewer repositories and rather do configuration management within the repositories as opposed to between repositories. Use feature flags, use tagging, use whatever you want to insulate changes that might be breaking from the rest of the code base." — Luca Ingianni

## Resources Mentioned

* John Taylor's Embedded Project Cookbook - A resource mentioned by the listener that discusses sequential events in embedded projects

* Trunk-Based Development - Development methodology discussed throughout the episode

* Minimum CD Podcast - Previous podcast episode referenced by the listener

You can find Jeff at https://jeffgable.com.
You can find Luca at https://luca.engineer.

Want to join the Pragmatic Embedded Slack? Click here

Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/
Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

The episode discusses the concept of Minimum Viable Continuous Delivery (Minimum CD), which represents a counter-movement to heavyweight frameworks like SAFe. The hosts explore how Minimum CD provides a set of essential practices for successfully building software-based products without unnecessary complexity. The approach focuses on core principles rather than rigid frameworks, making it particularly relevant for embedded systems development.

The discussion covers the fundamental requirements for continuous delivery, including automated testing, pipeline-driven deployments, and trunk-based development. The hosts emphasize that while these practices may seem challenging for embedded systems, they become increasingly important as devices become more sophisticated and connected.

A key theme throughout the episode is the importance of building trust in the development process through automation, consistent practices, and cultural commitment. The hosts stress that while some practices may seem difficult to implement in embedded systems, the more challenging they are, the more valuable they become when successfully implemented.

Timestamps and Topics:
00:00:00 - Introduction and overview of Minimum CD
00:02:00 - Discussion of Minimum CD as counter-movement to complex frameworks
00:03:45 - Continuous Integration fundamentals
00:15:35 - Pipeline as the only way to deploy
00:27:00 - Production-like test environments
00:29:45 - Rollback capabilities for connected devices
00:32:25 - Configuration deployment with artifacts
00:34:50 - Trunk-based development principles
00:39:30 - Automated testing requirements
00:41:10 - Maintaining delivered work integrity
00:45:55 - Wrap-up and closing thoughts

Shownotes:

Link to minimumcd.org: https://minimumcd.org/
Reference to Brian Finster as instigator of Minimum CD
Reference to Raymond Chen's blog about Windows backward compatibility
Reference to previous episode on trunk-based development
Reference to interviews with Philip Johnston from Embedded Artistry
Reference to interview with Francois from Mend Fault
Link to Agile Embedded Slack group

You can find Jeff at https://jeffgable.com.
You can find Luca at https://luca.engineer.

Want to join the Pragmatic Embedded Slack? Click here

Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/
Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

Key Topics

* [03:00] Ryan's background in offensive cybersecurity and defense contracting

* [04:30] The mindset and challenges of vulnerability research and hacking

* [09:15] How security researchers approach attacking embedded devices

* [13:45] Techniques for extracting and analyzing firmware

* [19:30] Security considerations for embedded developers

* [24:00] The importance of designing security from the beginning

* [28:45] Security challenges for small companies without dedicated security staff

* [33:20] Address Space Layout Randomization (ASLR) and other security measures

* [37:00] Emulation technology for testing embedded systems

* [45:30] Tulip Tree's approach to embedded system emulation and security testing

* [50:15] Resources for learning about cybersecurity and hacking

Notable Quotes

> "When you're on the vulnerability research side, you're trying to find a time when the software does something wrong. When it does something unexpected." — Ryan Torvik

> "Don't roll your own cryptography. Use a standard library for cryptography." — Ryan Torvik

> "We're seeing that the maintenance costs are what are getting people now. You're expected to maintain this device, but now you got to be able to actually update the device." — Ryan Torvik

> "It's so much more expensive to put security in after the fact if it's possible in the first place. Why is that even something that needs to be debated?" — Luca Ingianni

Resources Mentioned

[Tulip Tree Technology](tuliptreetech.com) - Ryan's company focused on embedded system security and emulation

* IDA Pro - Interactive disassembler for firmware analysis

* Binary Ninja - Interactive disassembler from Vector35

* Ghidra - NSA's open-source software reverse engineering tool

* Microcorruption - Beginner-friendly CTF challenge for learning embedded system hacking

* National Vulnerability Database - Public database of security vulnerabilities

Things to do

* Join the Agile Embedded Podcast Slack channel to connect with the hosts and other listeners

* Check out Tulip Tree Technology's website for their emulation tools and security services

* Try Microcorruption CTF challenges to learn about embedded system security vulnerabilities

* Consider security implications early in your design process rather than as an afterthought

* Use secure programming languages like Rust that help prevent common security issues

You can find Jeff at https://jeffgable.com.
You can find Luca at https://luca.engineer.

Want to join the Pragmatic Embedded Slack? Click here

Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/
Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

• [02:30] Definition of BDD as an additional layer of discipline on top of TDD
• [03:15] Common pitfalls of TDD, including testing to implementation and brittle tests
• [08:30] The structure of BDD tests using Given-When-Then format
• [12:00] Applying BDD at different levels, from unit tests to system tests
• [15:45] Using test doubles and spies for hardware interactions in embedded systems
• [22:30] Testing state machines with BDD
• [27:00] Off-target testing and hardware abstraction layers
• [33:00] Why BDD isn't more widely used in embedded systems
• [36:30] Using code coverage as a signal rather than a goal metric
• [39:00] Overcoming the learning curve and maintaining discipline in BDD

"BDD is an additional layer of discipline on top of TDD. Dan North's goal was to get straight to the good stuff of TDD without getting into the pitfalls." — Steve

"The key thing that BDD does by saying we're going to focus on behavior is you look at the API that you've written and you say, what can I do through the public API to affect this, to check the results and so forth?" — Steve

"By having abstraction layers, you create your thin layer that's substitutable with either the real code on target, or with a test double off target." — Steve

"Code coverage as a goal metric is not a good thing. Rather than using code coverage as just this almost dimensionless metric, use it as a signal to guide you." — Steve

"By adhering very strictly to the simple rules of how to do BDD, by forcing yourself to the discipline of that strict adherence, it keeps you on track." — Steve

• James Grenning's book
• Steve's blog post
• Embedded Online Talk
• Ian Cooper's video

You can find Jeff at https://jeffgable.com.
You can find Luca at https://luca.engineer.

Want to join the Pragmatic Embedded Slack? Click here

Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/
Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

John Taylor returns to the podcast to discuss his new book "The Embedded Project Cookbook", co-authored with Wayne Taylor. The book serves as a practical guide for embedded systems development, providing recipes and techniques that John has refined over his 30+ year career. Unlike his previous book "Patterns in the Machine" which focused on software engineering best practices, this new book takes a more hands-on approach to project mechanics - from requirements gathering through release.

The book provides opinionated guidance on setting up project infrastructure, managing requirements, software architecture, and release processes. While organized in a waterfall-like structure for clarity, it emphasizes the need for agility in embedded development. A key theme is establishing good practices early to make releases boring and predictable rather than chaotic.

The content is valuable for different roles - from junior developers seeking context about the full development lifecycle, to technical leads implementing processes, to managers needing justification for development infrastructure investments. While the book presents John's specific approaches, he emphasizes that teams can adapt the recipes to their needs as long as they deliberately address the underlying challenges.

Key Topics Discussed (Timestamps):
00:00:00 - Introduction and background on John Taylor
00:02:00 - Comparison to previous book "Patterns in the Machine"
00:06:00 - Target audience and use cases
00:15:00 - Software Development Plans and their value
00:20:00 - Foundation setup before coding begins
00:27:00 - Managing project variants and configurations
00:30:00 - Communication and collaboration practices
00:35:00 - Release management and ongoing development
00:40:00 - Iterative development cycles
00:43:00 - Book availability and contact information

Links:
Embedded Systems Cookbook: https://link.springer.com/book/10.1007/979-8-8688-0327-7
Patterns in the machine (the previous book): https://link.springer.com/book/10.1007/978-1-4842-6440-9
John's LinkedIn: https://www.linkedin.com/in/john-tanner-taylor/

You can find Jeff at https://jeffgable.com.
You can find Luca at https://luca.engineer.

Want to join the Pragmatic Embedded Slack? Click here

Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/
Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/

In this episode, Jeff and Luca discuss trunk-based development, a software development practice where developers merge their work into the main branch (trunk) frequently - at least daily. They explain how this approach differs from traditional branching models like GitFlow, and address common objections and concerns. The hosts emphasize that while trunk-based development may seem risky, it actually reduces risk by exposing integration problems early and forcing teams to implement good engineering practices like automated testing and feature flags.

The discussion highlights how trunk-based development acts as a "forcing function" that encourages better development practices, smaller changes, and more frequent collaboration between team members. They explain that while this approach originated in web development, it's equally applicable to embedded systems. The hosts cite research from the book "Accelerate" showing that trunk-based development is a predictor of high-performing software teams.

The episode concludes by emphasizing that most objections to trunk-based development actually point to underlying process issues that need to be addressed, and that the benefits of early integration and feedback outweigh the perceived downsides.

Timestamps:
00:00:00 - Introduction and topic overview
00:03:00 - Basic version control concepts and branching
00:08:00 - Definition and principles of trunk-based development
00:13:00 - Feature flags explanation and implementation
00:20:00 - Common objections to trunk-based development
00:27:00 - Application to embedded systems
00:34:00 - Benefits of trunk-based development
00:40:00 - Impact on team dynamics and collaboration
00:47:00 - Research backing and evidence from "Accelerate"

Shownotes / Links:
MinimumCD: https://minimumcd.org/
Accelerate: https://www.goodreads.com/book/show/35747076-accelerate
The nvie branching model: https://nvie.com/posts/a-successful-git-branching-model/

You can find Jeff at https://jeffgable.com.
You can find Luca at https://luca.engineer.

Want to join the Pragmatic Embedded Slack? Click here

Are you looking for embedded-focused trainings? Head to https://agileembedded.academy/
Ryan Torvik and Luca have started the Embedded AI podcast, check it out at https://embeddedaipodcast.com/