Ship It Weekly - DevOps, SRE, Platform and Cloud Engineering News

This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps.

In this Ship It Conversations episode, I talk with Justin Garrison of Sidero Labs about Kubernetes, platform engineering, bare metal, AI, golden paths, and why knowing what to say no to may be one of the most important skills a platform team can develop.

Justin is Field CTO at Sidero Labs, the company behind Talos Linux, and co-host of Fork Around and Find Out.

We start with the evolution of Kubernetes and how managed services like EKS and GKE made Kubernetes easier to consume while also pulling teams deeper into proprietary cloud ecosystems. Justin explains why on-prem and bare metal are getting renewed attention, especially as teams look at cloud costs, data sovereignty, and the operational overhead that comes with constantly optimizing cloud environments.

We also get into where Kubernetes helps and where it becomes self-inflicted pain. Justin talks about abstraction, cognitive load, and why teams tend to use familiar tools for problems they were never really designed to solve.

A big part of the conversation is platform engineering and golden paths. Justin argues that every organization needs its own path, but platforms become dangerous when they try to centralize everything. He shares why one of the best decisions his team made at Disney Plus was simply saying no to stateful workloads.

We also talk about what really belongs in a platform: security controls, logging, monitoring, software supply chain visibility, and cost management. Justin explains why centralization can help in those areas, but can become a bottleneck when applied too broadly.

Near the end, we get into AI, security, tooling dependency, and engineering culture. Justin makes the point that people have always formed strong attachments to tools, and AI is another version of that. The challenge is knowing where AI actually helps versus where it becomes another dependency teams stop questioning.

The big takeaway: good platform engineering is not about supporting everything. It is about understanding what should be standardized, what should stay flexible, and what your team should explicitly refuse to own.

Highlights

• Why Kubernetes has become increasingly productized

• Why some teams are moving back toward on-prem and bare metal

• Where cloud cost optimization starts to become its own operational burden

• Why Kubernetes helps with abstraction and cognitive load

• Why familiar tools often get used for the wrong workloads

• What golden paths actually represent inside an organization

• Why platform teams need to know what to say no to

• What should and should not be centralized

• How AI changes engineering workflows without changing the need for judgment

• Why finding work you actually enjoy matters for avoiding burnout

Links

Sidero Labs: https://www.siderolabs.com

Talos Linux: https://www.talos.dev

Justin Garrison: https://justingarrison.com

Fork Around and Find Out: https://www.forkaroundandfindout.com

More episodes and show notes: https://shipitweekly.fm

On Call Brief: https://oncallbrief.com

LMGT Awards: https://lmgt.org

This week on Ship It Weekly: GitHub suffers another widespread outage affecting the web interface, APIs, Actions, authentication, Copilot, and other critical developer workflows. Zenity Labs demonstrates PleaseFix attacks against agentic browsers, where malicious content can influence agents with access to authenticated sessions and privileged tools. AWS Certificate Manager is moving away from email validation, and Cloudflare is experimenting with CI pipelines defined as TypeScript instead of YAML.

The bigger theme this week is dependencies and boundaries we tend to ignore until something breaks. GitHub is no longer just where the code lives. Agentic browsers are no longer just displaying webpages. Certificate renewal is not something you want depending on someone checking an inbox. And CI pipelines have become software systems of their own.

Links

GitHub Hit by Widespread Outage https://devops.com/github-hit-by-widespread-outage-halting-work-for-global-developers/

Zenity Labs: PleaseFix in Agentic Browsers https://zenity.io/company-overview/newsroom/company-news/zenity-labs-exposes-the-full-scope-of-pleasefix

AWS Certificate Manager Ending Email Validation https://aws.amazon.com/blogs/security/aws-certificate-manager-will-discontinue-email-validation-to-prove-domain-validation-for-certificates/

Certificate Expiry Is Still Taking Down Major Platforms https://tokentimer.ch/blog/tls-certificate-expiry-outages

Cloudflare Turns CI Pipelines into TypeScript Workflows https://www.infoq.com/news/2026/08/cloudflare-ci-code-workflows/

Dynatrace Acquires Arize https://devops.com/dynatrace-acquires-arize-as-ai-agents-deepen-the-observability-challenge/

AWS Open-Sources Dogwood https://www.infoq.com/news/2026/08/aws-dogwood-agent-policy/

Pulumi v3.258.0 https://github.com/pulumi/pulumi/releases/tag/v3.258.0

AWS Key Breach and Data-Transfer Signal https://assets.theregister.com/2026/08/13/20267/

Mario Saved the EU but Broke My System https://www.uptimelabs.io/articles/hamed-2012-outage-reflections

This week’s On Call Brief https://www.tellerstech.com/on-call-brief-news/2026-W34/

Ship It Weekly https://shipitweekly.fm/

This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps.

In this Ship It Conversations episode, I talk with Ned Bellavance of Ned in the Cloud about DevOps beyond the buzzwords, platform engineering, infrastructure as code, AI, and why fundamentals still matter even as the tools change.

Ned is the founder of Ned in the Cloud and host of the Day 2 DevOps podcast, with more than 20 years in IT across systems administration, cloud, architecture, automation, and technical education.

We start with a problem a lot of teams run into: adopting the ceremonies of DevOps without actually adopting the principles. Standups, sprints, pipelines, and tooling can make an organization look mature, but the real goal is better communication, faster feedback loops, and delivery tied to actual outcomes.

We also talk about how teams decide what to prioritize next. Security, reliability, performance, FinOps, and platform work can all matter, but chasing whatever is newest does not help if the basics are still broken.

A big part of the conversation is where infrastructure as code goes from here. We get into Terraform's state and scaling model, API rate limits, the Terraform/OpenTofu split, Terragrunt, and newer approaches like Swamp from System Initiative. AI is making infrastructure code cheaper to produce, but understanding the architecture behind that code is becoming more valuable.

That leads into learning and career development. We talk about why networking, Linux, databases, security, cloud architecture, and troubleshooting still matter, even if an LLM writes most of the syntax. Build things, get them wrong in controlled environments, troubleshoot them, and learn what is happening underneath the abstraction.

The big takeaway: tools will keep changing. Judgment, architecture, troubleshooting, and understanding the systems underneath them are much harder to automate away.

Highlights

• Why DevOps ceremony is not the same as DevOps principles

• How teams should decide what platform work actually matters

• Why basic security hygiene still matters

• Where Terraform's current state model starts to hit limits

• Terraform, OpenTofu, Terragrunt, and the future of infrastructure as code

• How AI changes the value of writing infrastructure code

• Why architecture and troubleshooting skills become more important

• Why breaking things in controlled environments is one of the best ways to learn

Links

Ned in the Cloud: https://nedinthecloud.com

Day 2 DevOps: https://day2devops.com

Ned in the Cloud on YouTube: https://www.youtube.com/c/NedintheCloud/

Ned Bellavance on LinkedIn: https://www.linkedin.com/in/ned-bellavance/

Swamp: https://swamp.club

Terraform: https://developer.hashicorp.com/terraform

OpenTofu: https://opentofu.org

More episodes and show notes: https://shipitweekly.fm

On Call Brief: https://oncallbrief.com

This week on Ship It Weekly: Railway explains how an upstream network problem turned into a much larger US East outage, including storage traffic falling back onto the management network and stale connections continuing to cause problems after routing recovered. Stripe shares how graph search and state machines helped cut database pager volume by about 30 percent. Kata Containers patches a critical guest-to-host escape, and DynamoDB adds native vector search.

The bigger theme this week is what happens after the obvious failure. Fixing the route does not necessarily clear the connections created while it was broken. Automating recovery does not have to mean handing an AI agent unrestricted production access. And stronger isolation does not eliminate the components that still cross the guest-host boundary.

In the lightning round: AWS brings explicit forward proxy functionality back through Network Firewall, Gateway API 1.6 moves TCPRoute and UDPRoute to stable, and containerd 2.4 enters beta with new functionality alongside breaking changes worth finding before your next runtime upgrade.

Links

Railway: July 2, 2026 US East Services Outage https://blog.railway.com/p/incident-report-july-2-2026-us-east-services-outage

Stripe: How Stripe uses graph search and state machines to auto-remediate a global database fleet https://stripe.dev/blog/how-stripe-uses-graph-search-and-state-machines-to-auto-remediate-a-global-database-fleet

Kata Containers: Guest-root to host-root escape via virtiofs https://github.com/kata-containers/kata-containers/security/advisories/GHSA-2gv2-cffp-j227

Amazon DynamoDB now supports real-time vector search https://aws.amazon.com/about-aws/whats-new/2026/08/amazon-dynamodb-vector-search/

AWS Network Firewall forward proxy preview https://aws.amazon.com/about-aws/whats-new/2026/08/aws-network-firewall-forward-proxy-preview/

Gateway API v1.6: TCPRoute and UDPRoute Graduate to Standard https://kubernetes.io/blog/2026/08/03/gateway-api-v1-6-release/

containerd 2.4 beta https://github.com/containerd/containerd/releases

CNCF: Learning Cloud-Native Engineering Beyond Tutorials Through LFX https://www.cncf.io/blog/2026/08/10/learning-cloud-native-engineering-beyond-tutorials-through-lfx/

This week’s On Call Brief https://www.tellerstech.com/on-call-brief-news/2026-W33/

Ship It Weekly https://shipitweekly.fm/

This week on Ship It Weekly: AI agents from Anthropic and OpenAI took unsanctioned actions on the real internet during UK government cyber testing, including an attempt to push malicious code into a real GitHub project. Kubernetes 1.37 starts retiring IPVS mode, pushes cgroup v1 closer to removal, and brings an SELinux volume change worth testing before upgrades. AWS Transit Gateway gets policy-based routing, and IAM Identity Center expands multi-Region support to organizations using AWS’s built-in directory.

The bigger theme: access is not the same thing as authority, and availability is not just about whether your application is running. Agents need boundaries around the actions they can take. Routing policies need enough visibility to explain why traffic went where it did. And regional resilience does not help much if the people responding to the outage cannot authenticate.

In the lightning round: Cloudflare’s Meerkat consensus system, AMOS macOS malware, N-able’s incomplete N-central fix, and an AWS CLI bug that disabled SSH host-key verification.

Links

AISI: Unsanctioned agent behaviour during cyber testing https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing

Kubernetes v1.37 Sneak Peek https://kubernetes.io/blog/2026/07/31/kubernetes-v1-37-sneak-peek/

AWS Transit Gateway Policy-Based Routing https://aws.amazon.com/about-aws/whats-new/2026/07/aws-transit-gateway-policy-based-routing/

AWS IAM Identity Center multi-Region directory support https://aws.amazon.com/about-aws/whats-new/2026/07/aws-iam-identity-center-extends-multi-region-support-to-identity-center-directory

Cloudflare: Introducing Meerkat https://blog.cloudflare.com/meerkat-introduction/

Atomic macOS / AMOS stealer infection https://isc.sans.edu/diary/rss/33208

N-able N-central exploitation after incomplete fix https://thehackernews.com/2026/08/n-able-says-attackers-take-over-n.html

CVE-2026-18654: AWS CLI EMR SSH host-key verification https://aws.amazon.com/security/security-bulletins/rss/2026-071-aws/

CloudFront VPC Origins half-migrated incident https://www.reddit.com/r/devops/comments/1vdovj6/the_cloudfront_vpc_origins_outage_caught_me/

This week’s On Call Brief https://www.tellerstech.com/on-call-brief-news/2026-W32/

More episodes https://shipitweekly.fm/

This week on Ship It Weekly: Telstra’s mobile network jumped back to 2006 after a timing device restarted with the wrong date, disrupting calls, data sessions, and hundreds of emergency calls.

DoorDash explains how Entity Cache, built with Envoy and Valkey, handles more than 1.5 million requests per second and uses stale-data policies, invalidation, and fallback behavior as a reliability layer.

The latest MCP release candidate removes protocol-level sessions, making servers easier to scale behind ordinary load balancers while leaving teams responsible for authentication, tracing, retries, rate limits, and application state.

GitHub and PyPI are also adding friction to package automation. Dependabot now delays routine updates by three days, while PyPI blocks new files from releases older than 14 days.

Links

Telstra outage https://www.telstra.com.au/exchange/our-mobile-network-outage-has-been-resolved-heres-what-happened

DoorDash Entity Cache https://careersatdoordash.com/blog/high-performance-proxy-cache-for-doordash-services/

MCP specification release candidate https://blog.modelcontextprotocol.io/posts/2026-07-28-release-candidate/

Dependabot package cooldown https://github.blog/changelog/2026-07-14-dependabot-version-updates-introduce-default-package-cooldown/

PyPI release-file restrictions https://blog.pypi.org/posts/2026-07-22-releases-now-reject-new-files-after-14-days/

Amazon ECS Action Logs https://aws.amazon.com/about-aws/whats-new/2026/07/amazon-ecs-action-logs/

Network Load Balancer listener rules https://aws.amazon.com/about-aws/whats-new/2026/07/aws-network-load-balancer-supports-listener-rules/

Amazon Managed Prometheus limits https://aws.amazon.com/about-aws/whats-new/2026/07/amazon-managed-service-prometheus-1500m-metrics-workspace/

PixelSmash in FFmpeg https://jfrog.com/blog/pixelsmash-critical-ffmpeg-vulnerability-turns-media-files-into-weapons/

SRE Weekly Issue 527 https://sreweekly.com/sre-weekly-issue-527/

This week’s On Call Brief https://www.tellerstech.com/on-call-brief-news/2026-W31/

More episodes https://shipitweekly.fm/

This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps.

In this Ship It Conversations episode, I talk with Jay Lark of Hookbridge about webhook reliability, retries, idempotency, replay, security, local development, and what happens when a simple HTTP POST becomes production infrastructure.

Jay is a Principal DevOps Engineer and the founder of Hookbridge, a service focused on making webhook delivery more reliable and easier to operate.

We start with the basic model: one service sends an HTTP POST to another when something happens. The happy path is easy. The problems begin when an endpoint is unavailable, responds too slowly, receives duplicate events, or gets them out of order.

Jay explains what “at least once delivery” means and why receivers must expect duplicates. We talk about idempotency, event IDs, retry behavior, availability during deployments, and why returning a 200 does not prove downstream processing succeeded.

We also dig into observability and security. Teams need enough visibility to know whether a webhook arrived, whether signature verification passed, what response was returned, and where processing failed. Jay breaks down HMAC signatures, timestamps, replay protection, and why a valid signature still does not replace normal business-logic validation.

Local development is another source of friction. External providers cannot send events directly to localhost, so developers often rely on temporary tunnels, staging deployments, copied payloads, or mocks. Jay explains how Hookbridge uses a fixed URL and local client to forward real webhook traffic to a developer’s machine.

We also talk about n8n, self-hosted OpenClaw systems, Hookbridge pull endpoints, and when polling may be simpler or safer than exposing another inbound endpoint.

The big takeaway: design the failure path before a webhook becomes business-critical. Verify the sender, expect duplicate and out-of-order events, build enough visibility to debug failures, and have a replay strategy before the first incident.

Highlights

• Why webhooks are harder than “just an HTTP POST”

• What at-least-once delivery means for receivers

• Why idempotency, retries, and event ordering matter

• What teams need for webhook observability and debugging

• Where HMAC signatures, timestamps, and replay protection fit

• Why local webhook development is still awkward

• When polling or pull-based delivery may be a better fit

• When teams should stop building webhook infrastructure themselves

Links

Hookbridge: https://hookbridge.io

Hookbridge local development CLI: https://www.hookbridge.io/cli.html

Hookbridge pull endpoints: https://www.hookbridge.io/pull.html

Jay Lark on LinkedIn: https://www.linkedin.com/in/jay-lark-ba7a3b5/

OpenClaw: https://openclaw.ai

More episodes and show notes: https://shipitweekly.fm

On Call Brief: https://oncallbrief.com

This week on Ship It Weekly: AWS CloudFormation Express mode promises faster infrastructure feedback by reporting deployments complete before extended resource stabilization finishes. Apache Spark 4.2 adds native vector operations and nearest-neighbor joins, giving some teams a way to keep AI data workloads closer to the platforms they already run.

GitLab’s latest research says AI is helping developers generate and commit code faster, but review, testing, governance, and deployment are not accelerating at the same pace. Then, a platform engineering case study from Sevdesk shows how minimum viable governance, useful feedback, and progressive enforcement can improve compliance without turning the platform team into another approval queue.

The theme this week: making one part of the system faster does not automatically improve the whole system.

In the lightning round: OpenShift 4.22.5 receives an important security update, outdated autoscaling thresholds disrupt GitHub Actions, Cloudflare experiences an incident where some POST requests fail to reach customer origins, and GitHub Code Quality becomes generally available—with automatic billing attached.

The episode closes with Reid Savage reflecting on their first year managing Honeycomb’s SRE team and the difference between what a capable team could do and what it should do.

Links

CloudFormation Express mode https://aws.amazon.com/blogs/aws/accelerate-your-infrastructure-deployments-by-up-to-4x-with-aws-cloudformation-express-mode/

Apache Spark 4.2 https://spark.apache.org/releases/spark-release-4-2-0.html

GitLab AI Accountability Report https://about.gitlab.com/resources/ai-accountability-survey-2026/

Platform governance at Sevdesk https://www.infoq.com/presentations/platform-engineering-team-compliance/

OpenShift 4.22.5 security update https://access.redhat.com/errata/RHSA-2026%3A37585

GitHub Actions incident https://github.com/orgs/community/discussions/201795

Cloudflare status https://www.cloudflarestatus.com/

GitHub Code Quality GA https://github.blog/changelog/2026-07-20-github-code-quality-is-now-generally-available/

Could vs. Should — Reid Savage https://www.honeycomb.io/blog/could-should-first-year-managing-sre-team

This week’s On Call Brief https://www.tellerstech.com/on-call-brief-news/2026-W30/

More episodes and full show notes https://shipitweekly.fm/

This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps.

In this Ship It Conversations episode, I talk with Mat Ryer of Grafana Labs about AI observability, production agents, evals, telemetry cost, guardrails, and what changes once AI moves beyond demos and into systems teams actually depend on.

Mat is Senior Director of AI at Grafana Labs, where he focuses on how AI fits into observability and production systems.

We talk about Grafana Assistant, why AI observability is not just logs, latency, and HTTP 200s, and how teams can measure whether agents are actually helping. Mat gets into evals, LLM-as-judge patterns, traces as a way to think about conversations, user feedback, tool choice, model changes, and the cost of collecting new telemetry.

We also dig into UX and trust. If an AI assistant gives you a wall of text, you still have to decide whether to believe it. If it can show the graph, deep link into Grafana, apply filters, and expose the source data, that becomes a much more useful operating experience.

The big takeaway: start small, enhance workflows you already have, build feedback loops, and treat production AI like something you actually have to operate.

Highlights

• Why AI demos are easy, but production AI is harder

• Why agents need observability, evals, and guardrails

• How Grafana thinks about AI Assistant and AI observability

• Where LLM-as-judge patterns, traces, tool calls, and feedback fit

• Why telemetry cost problems may repeat with AI workloads

• Why UX matters when operators need to trust the answer

• Where AI can help SRE and platform teams today

Links

Mat Ryer on LinkedIn: https://www.linkedin.com/in/matryer/

Mat Ryer on GitHub: https://github.com/matryer

Grafana Labs: https://grafana.com

Grafana Assistant: https://grafana.com/products/cloud/ai-assistant/

Grafana AI Observability: https://grafana.com/docs/grafana-cloud/machine-learning/ai-observability/

Grafana Adaptive Telemetry: https://grafana.com/products/cloud/adaptive-telemetry/

Grafana MCP server: https://github.com/grafana/mcp-grafana

OpenTelemetry: https://opentelemetry.io

Prometheus: https://prometheus.io

Grafana Loki: https://grafana.com/docs/loki/latest/

More episodes and show notes: https://shipitweekly.fm

On Call Brief: https://oncallbrief.com

This week on Ship It Weekly: Datadog tracked coordinated GitHub API enumeration, xAI’s Grok Build CLI reportedly uploaded repo data without redaction, AWS Security Hub added Network Scanning and exposure impact analysis, and Microsoft says AI-powered vulnerability discovery is changing patch pressure.

The theme: visibility is not ownership. A GitHub API map does not revoke a token. An exposure finding does not close a port. A patch bulletin does not patch the fleet. And an AI coding tool reading your repo is still access.

Brian covers GitHub as a production surface, AI coding tool data boundaries, cloud exposure based on reachability and blast radius, and why patching needs to look more like production operations than spreadsheet theater.

Also, the Ship It Weekly shop is open at shop.tellerstech.com with Ship It Weekly t-shirt designs. Use coupon code SHIPTHESTORE for 20% off your order for the next few weeks.

Links

Datadog: Coordinated GitHub API enumeration https://securitylabs.datadoghq.com/articles/coordinated-github-api-enumeration/

The Verge: Grok Build CLI repository upload report https://www.theverge.com/ai-artificial-intelligence/965600/spacexai-grok-build-repository-upload

AWS Security Hub Network Scanning https://aws.amazon.com/about-aws/whats-new/2026/07/aws-security-hub-network-scanning/

AWS Security Hub impact analysis for exposure findings https://aws.amazon.com/about-aws/whats-new/2026/07/impact-analysis-aws-security-hub/

Microsoft: Windows vulnerability management and AI-powered discovery https://blogs.windows.com/windowsexperience/2026/07/09/evolving-windows-vulnerability-management-to-meet-the-speed-of-ai-powered-discovery/

SRE Weekly Issue 525 https://sreweekly.com/sre-weekly-issue-525/

HalluSquatting / hallucinated package risk https://www.endorlabs.com/learn/slopsquatting-when-ai-agents-hallucinate-malicious-packages

AWS Lambda Managed Instances for Java cold starts https://aws.amazon.com/blogs/compute/eliminating-java-cold-starts-with-aws-lambda-managed-instances/

This week’s On Call Brief https://www.tellerstech.com/on-call-brief-news/2026-W29/

Ship It Weekly shop https://shop.tellerstech.com/

More episodes and full show notes https://shipitweekly.fm/