Fork Around And Find Out

Come find Justin in-person and get FAFOFM stickers!

• Devopsdays - Portland, OR - September 8-10. Get 20% off tickets with FAFOFM
• Edgecase - Gooiland Hilversum, NL - September 24
• Taloscon - Amsterdam, NL - October 15-16
• KCD UK - Edinburgh - October 19
• Kubecon NA - Salt Lake City, UT - November 9-12

Have an opening? HIRE AUTUMN!!! She's looking for open source, forward deployed engineer, and developer relation roles. They need to be remote or flexible in-office schedules in Seattle.

Disaster recovery is more than automation and infrastructure. There's a lot that goes into your services and some of those things can't be defined as code or automated. When was the last time you restored your database from a backup? How do you use least privileged access when your region changes and how do you even know you're having a disaster. Seth has a lot of experience and a ton of good insights in this episode.

Links

• Seth’s Linktree https://linktr.ee/setheliot
• arpio https://arpio.io/services/
• EKS Auto example repo https://community.aws/content/2sV2SNSoVeq23OvlyHN2eS6lJfa/amazon-eks-auto-mode-enabled-build-your-super-powered-cluster
• Disaster recovery white paper https://docs.aws.amazon.com/whitepapers/latest/disaster-recovery-workloads-on-aws/disaster-recovery-workloads-on-aws.html
• Disaster recovery blog series https://aws.amazon.com/blogs/architecture/tag/disaster-recovery-series/
• Ship it! with Pete Naylor https://changelog.com/shipit/127
• Engineering Resilient Systems on AWS by Jennifer Moran https://www.amazon.com/Engineering-Resilient-Systems-AWS-Resilience/dp/1098162420
• Four Things Everyone Should Know About Resilience https://community.aws/content/2duX45O6vKOE7cmmXQ9Nj3tO3bL/should-know-resilience

Railway wanted to build a better cloud so they started on Google Cloud and ended up building datacenters. Through the burden of success, they figured out there was a lot of things they had to learn and build themselves if they wanted to offer the best cloud experience they could. Jake tells us how they accomplished the move in 9 months, why the built their own orchestrator, and what they’re working on next.

Check out Railway at https://railway.com/ and their blog at https://blog.railway.com. Specifically read the post discussed in this episode. https://blog.railway.com/p/data-center-build-part-one

You can't grow in technology without learning new things. But sometimes those new things are actually old things. We talk with Scott about a wide range of interests about software, video games, 3D printing, and food. If you want to know why junior engineers are important for your teams you need to listen.

Highlights

(0:00) What makes good engineers?
(12:00) Vibe coding
(19:00) Doom scrolling with intention
(24:00) Making vs buying
(26:00) Praising hard work
(30:00) Loss of empathy

Links

• Scott's website: https://www.hanselman.com/
• Hanselminutes Podcast: https://hanselminutes.com/
• Scott and Mark Learn to... Podcast: https://www.youtube.com/playlist?list=PL0M0zPgJ3HSf4XZvYgZPUXgSrfzBN26pf

What exactly is an LLM doing and why do you need to learn so many new terms? Steve Pousty is here to explain that most of those new terms are things you already know. It’s not new technology, it’s new words to describe technologies applied in a new field. We have a wild, ADHD roller coaster looping through embeddings, vectors, RAG, and LLMs. Make sure to keep your hands and arms inside the pod for this one.

Chapters
(0:00) Intro
(9:00) Embeddings
(19:00) Graph DB vs Vector DB
(21:00) Vector Algebra
(36:00) Open Source
(41:00) Vector databases
(51:00) What is RAG?
(58:00) What is an LLM doing?
(1:08:00) Dating advice

Links
• 🦋Steve on Bluesky https://bsky.app/profile/thesteve0.bsky.social
• ▶️ Steve on YouTube https://www.youtube.com/@thesteve0
• 📍Voxel 51 https://voxel51.com/
• 🎮 Vector algebra game https://neal.fun/infinite-craft/
• 📘The Alignment Problem https://www.amazon.com/Alignment-Problem-Machine-Learning-Values/dp/0393635821
• 🎥 Mitchells vs The Machines https://www.netflix.com/title/81399614
• 📀 MNIST dataset https://en.wikipedia.org/wiki/MNIST_database
• 📝 What ChatGPT is not https://blog.techravenconsulting.com/what-chatgpt-is-not/
• 📝 Why I am excited about ChatGPT https://blog.techravenconsulting.com/why-i-am-excited-about-chatgpt/

How Rachel Ray’s crawler lead to Ada developing a new performance testing framework, hyperscale. This leads to a great conversation about the benefits of rust, modern python package managers, and why MySpace went out of business. The importance of connecting what you’re building to business value and understanding every line of code has a cost.

Show Highlights

(0:00) Intro
(8:00) Rachel Ray crawler
(15:00) Performance testing
(20:00) Moving in to tech
(31:00) Hyperscale and uv 
(39:00) Does memory safety matter?
(43:00) Datavant
(54:00) Connecting performance to business
(1:02:00) Spicy takes

Links Referenced

• Performance testing tools:
  • https://locust.io/
  • https://k6.io/
  • https://github.com/hyper-light/hyperscale
• Modern Python packaging: https://github.com/astral-sh/uv
• FAFO starter pack: https://go.bsky.app/LqoyyNq

Sponsor FAFO at https://fafo.fm/sponsor

Is running Kafka on-prem different than running it in the cloud? You’ll find out from Elad Eldor’s years of experience running, tuning, and troubleshooting Kafka in production environments. Elad didn’t set out to learn Kafka, but he kept asking questions and was given the opportunity to dive deep into system performance. He not only knows what all the columns of iostat mean, he knows what his customers want.  Make sure to subscribe to this topic on all your consumers.

Show Highlights

(0:00) Intro
(9:30) Why do people use Kafka 
(15:00) Learning cloud vs on-prem
(18:30) Kafka vs Linux troubleshooting
(27:00) scaling clusters
(38:00) How to get started

Links Referenced

• Elad’s book: Kafka Troubleshooting in Production https://www.amazon.com/Kafka-Troubleshooting-Production-Stabilizing-premises-ebook/dp/B0CJ4FSGMD
• Systems Performance book by Brendan Gregg https://www.brendangregg.com/blog/2020-07-15/systems-performance-2nd-edition.html
• Kafka: The Definitive Guide book by Neha Narkhede https://www.amazon.com/Kafka-Definitive-Real-Time-Stream-Processing/dp/1491936169

Sponsor

https://www.softwaredefinedtalk.com

Sponsor FAFO at https://fafo.fm/sponsor

Sam is back with us for a live episode where we discuss important questions such as “is coffee is good?”, “are people on the Internet good writers?”, and “is content creation consistency actually important?” We also share links about turning your hand writing into a font and hacking McDonald’s delivery app. Finally, we wrap up with a new game FAANG GANG.

Chapters

(0:00) Intro
(9:00) Codifying your handwriting
(10:30) I’m loving it
(27:00) Consistency in content
(39:00) FAANG gang game

Links Shared

• Codifying your Handwriting https://www.amygoodchild.com/blog/cursive-handwriting-in-javascript
• I’m lovin’ it https://eaton-works.com/2024/12/19/mcdelivery-india-hack/

Sponsor

VocalCat: https://fafo.fm/vocalcat

Sponsor the FAFO Podcast!

https://fafo.fm/sponsor 

This episode is stacked with information. You could even say “full stacked.” Sam has built and run some large scale systems as a SRE at Google, now building backend services at Budibase, and he spends his free time teaching others how systems work at understandable scale. We dive into what makes Google SRE different from other companies, what it’s like to be a parent, and how Sam got started with building animations for his blog. Don’t forget to visit and check out the easter eggs he’s hidden throughout.

Show Highlights

0:00 - Intro
2:00 - Sam’s background
6:00 - How Google did SRE
15:00 - Importance of docs
19:00 - The problems with Java
26:00 - Budibase
32:00 - Borg vs Kubernetes 
39:00 - Building animations
46:00 - Being a better teacher
56:00 - Art in the age of AI
1:00:00 - What’s next

Links Referenced

• SRE book https://sre.google/books/
• Sam’s Blog https://samwho.dev/
• Budibase - https://budibase.com/
• Bartosz Ciechanowski’s website - https://ciechanow.ski/
• Life Animated book - https://www.amazon.com/Life-Animated-Sidekicks-Heroes-Autism/dp/1484741234
• Andy Matuschak’s website - https://andymatuschak.org/
• Southern California Linux Expo (SCaLE) - https://www.socallinuxexpo.org/scale/22x

Sponsor

The sponsor for this week is YOU!

Please share and rate this episode!

https://fafo.fm/sponsor

Bluesky has been on a roller coaster of growth for over a year. From the early days of figuring out a new distributed social protocol—AT protocol—to actually building it and inviting 30 million of their closest friends. Not only has the site gone through tremendous growth, the team has been optimizing, re-architecting, and adding features the entire time.

Jaz is a software engineer focused on the infrastructure at Bluesky, and they share how they achieved exponential growth without exponential costs. We cover some of the key components of the protocol and how that affects the architecture.

There’s some amazing advice from the trenches we know you’ll enjoy.

Show Highlights
(0:00) Intro
(5:00) Jaz’s background
(12:30) Bluesky Infrastructure
(17:00) Predicting the future
(20:00) What is a PDS?
(22:30) Relay and firehose
(26:00) Work queues
(30:00) Scaling physical servers
(37:00) How do you handle incidents?
(41:00) Where’s Kubernetes?
(43:30) How video changes
(45:00) Data locality
(46:30) Hardware decisions
(53:00) What bad decisions?
(57:00) Launching video
(1:00:00) What’s next?

About Jaz

Jaz is a software engineer who learned from on-the-job experience. They have a background with hardware which makes them better with software. If they’re not drinking Monster they’re building a single purpose database, or maybe they’re doing both. Jaz went from building with AT protocol to building AT protocol in a matter of months. They also have an impressive collection of plushies and power tools.

Sponsor the FAFO Podcast!

http://fafo.fm/sponsor

Today we find out how building a product at Big Tech can be very different than a startup. Lauren Long has done both. Building parts of Firebase and eventually taking the things she learned to build Ampersand. We discuss what Ampersand is and go into detail about what the back end looks like. We even drop some hot takes about serverless and Kubernetes. We think you’ll love it!

Show Highlights

(0:00) Intro

(1:00) What is Ampersand?

(3:00) What is the backend?

(4:00) What is Lauren’s background

(6:00) How are people using it?

(10:00) How is Temporal used?

(14:00) How to keep APIs in check

(21:00) What did you learn?

(24:00) What has broken?

(26:00) Why use Kubernetes?

(32:00) What have customers done?

(38:00) What’s next?

About Lauren Long

Lauren Long is the CTO and co-founder of Ampersand, an API integration and workflow engine for enterprises to integrate their data with hundreds of applications. Lauren co-founded Ampersand after working at Google on their serverless products and saw a need for a different kind of integration for customers. She’s a developer with a great intuition on how to build reliable and scalable systems.

Sponsor the FAFO Podcast!

http://fafo.fm/sponsor