We hit a new (and disturbing!) failure mode recently when a production rack that had been up for several months saw every (!) compute sled's service processor become simultaneously unresponsive. Bryan and Adam were joined by the members of the Oxide team who debugged the vexing issue -- and reached its surprising root cause.

In addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide colleagues, Cliff Biffle, Matt Keeter, and Will Chandler.

Previously, on Oxide and Friends:

• OxF s05e03 – Holistic Engineering with Robert Mustacchi
• OxF s04e14 – Rebooting a datacenter: A decade later
• OxF s01e26 – The Pragmatism of Hubris
• OxF s05e20 – Debugger-Driven Development (omdb)
• OxF s05e07 – Transparency in Hardware/Software Interfaces
• OxF s05e31 – Futurelock
• OxF s05e33 – A Grown-up ZFS Data Corruption Bug

Some of the topics we hit on, in the order that we hit them:

• hubris #2304: STM32H7 Ethernet driver stops yielding CPU after many packets
• gist — Summarizing the Hubris side of investigations
• Matt's blog: Hunting a spooky ethernet driver bug

If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Hey hey! We recently tripped over a ZFS data corruption bug–introduced over 18 years ago! Bryan and Adam discuss with members of the Oxide team as well as Matt Ahrens, the co-inventor of ZFS.

In addition to Bryan Cantrill and Adam Leventhal, speakers included Alan Hanson, Matt Keeter, Andy Fiddaman, James MacMahon, and special guest, Matt Ahrens.

Previously, on Oxide and Friends:

• OxF s4e6 - Crucible: the Oxide Storage Service
• OxF s5e28 - Systems Software in the Large

Some of the topics we hit on, in the order that we hit them:

• ZFS fsync can trigger ZIL transaction reordering and data corruption
• RFD 177: Implementation of Data Storage
• the "fix" that introduced data corruption
• PRs needed!

If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Oxide founders, Bryan and Steve, as well as Oxide investor, Seth Winterroth, were joined by Liz Zalman and Jerry Neumann, authors of the book Founder vs. Investor, discussing the collaboration and conflict in company formation. Adam was also present.

In addition to Bryan Cantrill and Adam Leventhal, our guests included Liz Zalman, Jerry Neumann, Seth Winterroth (Oxide investor), and Steve Tuck (Oxide founder / CEO).

Some of the topics we hit on, in the order that we hit them:

• Founder vs. Investor
• Topic
• [@M:SS](link into recording) Leventhal's Conundrum
• PRs needed!

If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

We're big users of async Rust at Oxide, and recently we found (another) very odd and hard to debug pathology related to async Rust that we dubbed "Futurelock". Oxide engineers who diagnosed the problem join Bryan and Adam to describe Futurelock and discuss methods to identify and avoid it.

In addition to Bryan Cantrill and Adam Leventhal, speakers included our Oxide colleagues Dave Pacheco, John Gallagher, Rain Paharia. Sean Klein, and Eliza Weisman.

Previously, on Oxide and Friends:

• OxF s05e22 - When Async Attacks!
• OxF s05e26 - Technological Revolutions with Jerry Neumann

Some of the topics we hit on, in the order that we hit them:

• Oxide RFD 609: Futurelock
• PRs needed!

If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Revisiting an annual tradition--Books in the Box! Bryan, Adam, and the Oxide Friends share book recommendations (and--sometimes--anti-recommendations). Take a listen if you're looking for your next read.

In addition to Bryan Cantrill and Adam Leventhal, we were joined by some guests noted below:

Previously, on Oxide and Friends:

• OxF s05e06 - A Half‑Century of Silicon Valley with Randy Shoup
• OxF s05e26 - Technological Revolutions with Jerry Neumann
• OxF s04e03 - Fork in the Road for Terraform
• OxF s01e16 - The Books in the Box
• OxF s02e18 - Books in the Box Redux
• OxF s03e22 - Books in the Box III
• OxF s04e28 - Books in the Box IV

Other Notes:

• Princeton Review: Happiest Students
• UMass Dining Named Best Campus Food by The Princeton Review
• CHM Oral Histories
• Night Rider (and K.I.T.T.)

From Bryan and Adam (and others)

• The Mouse Driver Chronicles
• Fumbling the Future
• Slingshot
• Chip War
• Technological Revolutions and Financial Capital @bcantrill (economics book recommendation)
• Snow Crash (another Neal Stephenson book)
• The Big Short
• Reinventing The Wheel
• Eccentric Orbits (recommended by listener)
• Language Machines Cultural AI and the End of Remainder Humanism (recommended by listener)
• Molly White's **review ** of Read Write Own
• Careless People
• NOT A RECOMMENDATION If Anyone Builds It, Everyone Dies (If you are Molly White, please destroy this for us!)
• Surreal Numbers by Knuth (recommended by listeners)

From Oliver Herman

• Open Circuits
• Systems Performance
• Why We're Getting Poorer
• Termination Shock

From Tom Lyon

• From Airline Reservations to Sonic the Hedgehog
  • See also Systems We Love: Life of an Airline Flight
• The War of Don Emmanuel's Nether Parts
• The NVIDIA Way

From Dan McDonald

• Inventing the Renaissance
• Charles Sumner: Conscience of a Nation

If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

LLMs have had a dramatic impact on education. There are obvious reasons for concern, but what about the less obvious opportunities afforded by LLMs? Bryan and Adam were joined by Michael Littman, professor at Brown University and Associate Provost for AI, to talk about his role advising the university on productive, innovative, creative uses for AI in higher education.

In addition to Bryan Cantrill and Adam Leventhal, our special guest was Michael Littman.

Previous, on Oxide and Friends:

• OxF s01e18 - Dijkstra’s Tweetstorm
• OxF s04e02 - Open Source LLMs with Simon Willison
• OxF s05e18 - AI, Materials, and Fraud with Ben Shindel

Some of the topics we hit on, in the order that we hit them:

• Michael's home page
• Leslie Kaelbling
• Computing Up: Rich Sutton
• PRs needed!

If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Dave Pacheco is leading Oxide's multi-year effort around full-system update. He recently gave a talk about his experience leading that project, the complexities of designing the system and organizing the team. Dave, Bryan, and Adam discuss the project, the many sources of leadership, and the often underestimated peril of "organizational procrastination".

In addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide colleague, Dave Pacheco.

Previously on OxF:

• OxF s05e21 - Rebooting a Datacenter: A Decade Later
• OxF s01e09 - Agile + 20
• OxF s04e11 - A Baseball Startup with Paul Freedman and Bryan Carmel

Some of the topics we hit on, in the order that we hit them:

• Dave's talk: Path to self-service update (slides)
• Fire trucks dousing the champion Ballers
• Bryan's talk: Debugging Under Fire
• Roger Faulkner: "I'm not here to make it perfect; I'm here to make it better"
• Mid-recording earthquake

If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Bryan and Adam were joined by members of the Oxide manufacturing team to talk about all that goes into ramping up production, from people and processes to expanding the team and refining inefficiencies. It's a great problem to have!

In addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide colleagues, CJ Mendes, Kirstin Neira, Erik Anderson, Aaron Hartwig, and Doug Wibben.

Previously on Oxide and Friends...

• OxF s03e20 - Tales from Manufacturing: Shipping Rack 1

Some of the topics we hit on, in the order that we hit them:

• Topic
• [@M:SS](link into recording) Leventhal's Conundrum
• PRs needed!

If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Jerry Neumann joined Bryan and Adam to discuss his blog post from 2015, examining the work of Carlota Perez on technological revolutions. These waves have similarities, in particular: frenzy, bust, and deployment. Is AI a new wave or the culmination of the IT wave of the last 50 years?

In addition to Bryan Cantrill and Adam Leventhal, we were joined by special guest, Jerry Neumann.

Some of the topics we hit on, in the order that we hit them:

• Jerry's 2015 blog post: The Deployment Age
• PRs needed!

Previous episodes mentioned:

• OxF s05e24 - Oxide’s $100M Series B
• OxF s05e04 - AI Disruption: DeepSeek and Cerebras

If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Scott Hanselman gave a terrific talk about the promises of tech: connection, convenience and creativity. Did it deliver? Scott joins Bryan and Adam to discuss... and also wander around as one expects from an Oxide and Friends episode.

In addition to Bryan Cantrill and Adam Leventhal, our special guest was Scott Hanselman.

Past episodes mentioned:

• OxF s01e12 - A Brief History of Talking Computers
• OxF s02e18 - Books in the Box Redux
• OxF s05e10 - Lip‑Bu Tan’s Intel

Some of the topics we hit on, in the order that we hit them:

• Scott's talk: Tech Promised Everything. Did it deliver?

If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!