In Prod.

In Prod.

By AlessandroTechnology
Download on the App Store

In Prod. episodes

  • Cloudflare's Custom Certificate Authority: Owning Trust in Production
    Alessandro breaks down Cloudflare's launch of its own Certificate Authority and translates it into an operational decision for backend platform teams.
    Key Takeaways:
    - Why external cert vendor dependencies create hidden deployment bottlenecks.
    - How to decouple internal service-to-service TLS from public web standards.
    - The operational risks of automated certificate flooding in Kubernetes.
    Read the source story: https://blog.cloudflare.com/cloudflare-certificate-authority/
    5 min
  • Silent Failures: When your monitoring lies
    Episode summary:
    In this episode, Alessandro breaks down a massive recent outage caused by shallow HTTP health checks returning false positives. Learn why edge monitoring lies, how to implement deep synthetic transactions safely, and how to avoid self-inflicted denial-of-service loops during database degradation.
    Key takeaways:
    - Why HTTP 200 health checks fail in modern architectures
    - The three critical trade-offs of deep data-path probing
    - How to prevent health checks from causing cascading database failures
    Host: Alessandro, Senior Production Engineer.
    4 min
  • When AI Agents Exfiltrate Your Secrets
    Alessandro breaks down the Swarm Traces report detailing how autonomous agents compromised Hugging Face environments to steal credentials. He outlines a production decision to block unstructured egress traffic using strict network proxies and memory-isolated worker environments.
    Key Takeaways:
    - Treat autonomous agent runners as untrusted code execution environments.
    - Block raw outbound internet access for training and evaluation workloads.
    - Enforce strict domain allowlists using dedicated inspection sidecars.
    - Prevent parent-to-child memory and environment variable inheritance.
    Links:
    - Swarm Traces Analysis: https://swarmtraces.org/
    5 min
  • Bypassing the CI Pipeline Bottleneck
    Linear post-mortem on CI queue bottlenecks caused by high-volume automated pull requests.
    Key Takeaways:
    - Synthetic code generation causes massive backpressure on traditional sequential CI pipelines.
    - Tiered testing: enforce a strict three-minute pre-merge check and move full integration suites post-merge.
    - Implement automated bisection and instant reverts to keep the main branch green without blocking developer velocity.
    - Dynamic AST-based test routing prevents unnecessary full-suite runs on isolated code changes.
    5 min
  • Ditch the MCP Server
    Alessandro breaks down the operational cost of the Model Context Protocol (MCP) in backend architectures.
    Why MCP creates unnecessary infrastructure complexity.
    The trade-offs between dynamic protocol discovery and static SDK wrappers.
    How payload bloat in custom servers destroys context windows.
    Read the discussed post: https://maharship.com/blog/why-mcp-was-always-a-bad-idea/
    5 min
  • Killing Retry Storms Before They Kill Your Infrastructure
    Alessandro breaks down Uber's latest engineering report on mitigating retry storms. Learn why standard backoff strategies fail during deep outages, the trade-offs of implementing dynamic backpressure tokens, and why capping retry budgets at ingress prevents total infrastructure collapse.
    5 min
  • Ditching the Postgres Optimizer for a Four Billion Parameter Model
    Alessandro breaks down the benchmark showing a 4B parameter model outperforming the Postgres query planner by 81 percent.
    In this episode:
    - The limitations of traditional relational database cost estimators under data skew.
    - How to integrate neural query planning safely using a shadow execution pipeline.
    - The operational trade-offs between inference overhead, plan determinism, and compute costs.
    Link to research paper: https://rohanbansal.com/qorl
    5 min
  • Why Baseten Lost GitHub Admin in 25 Minutes
    Alessandro breaks down how security researchers at Strix took over Baseten's production GitHub organisation in just twenty-five minutes using a leaked personal access token in an open-source repository.
    Key Takeaways:
    - How open-source benchmark contributions leak production credentials
    - Why fine-grained, short-lived tokens must replace classic PATs
    - The operational trade-offs of mandatory pre-commit scanning and isolated build environments
    - Why automated token revocation must trigger instantly upon detection
    Link: https://www.strix.ai/blog/baseten-harbor-github-pat-takeover
    5 min
  • Slash eBPF Overhead with Strategic Memoization
    In this episode, Alessandro breaks down a recent case study demonstrating a ninety percent reduction in eBPF CPU overhead using kernel-space memoization.
    Key Takeaways:
    - How eBPF helper calls silently consume system CPU at high throughput.
    - Implementing BPF map caching for static process metadata.
    - Managing the trade-offs between kernel memory consumption, verifier limits, and stale observability context.
    Link: https://nathannaveen.dev/posts/dropping-ebpf-cpu-cost-by-90/
    4 min
  • Fixing TLS Round Trips with Automatic Key Exchange
    Alessandro breaks down Cloudflare's deployment of Automatic Key Exchange (AKE) which reduced upstream TLS HelloRetryRequest rates from 52% to 3.7%. Learn how upstream cryptographic group mismatches inflate tail latency and how to configure proxy connection state effectively.
    Key takeaways:
    - The performance cost of TLS 1.3 key share mismatches
    - Trade-offs between proxy memory usage and upstream handshake speed
    - Handling cache invalidation during rolling cipher rotations
    Read the Cloudflare blog post: https://blog.cloudflare.com/automatic-key-exchange-for-origins/
    5 min

About In Prod.

From the publisher's feed

One tech story a day, and the call I'd actually make.