tcp.fm

311: The Crawlers are Running the Asylum

Listen Later

Welcome to episode 311 of Two Old Men Yelling at Cloud – aka The Cloud Pod, featuring Matt and Ryan who absolutely, definitely did NOT record an aftershow. 

This week, they’re talking about Cloudflare’s new Pay Per Crawler, a new open-source Terraform provider from mkdev, and lots of fabric news that Ryan doesn’t understand – plus so much more. Let’s get into it!  

Titles we almost went with this week:

(Show Editor note: There are more show titles than emojis. I give up.) 

  • FSx and the City: When File Systems Meet Object Storage
  • The Great Data Lake Escape: No Movement Required
  • OpenZFS Gets an S3 Degree Without Leaving Home
  • Kernel Sanders: Microsoft’s Recipe for Avoiding Another Fried System
  • Windows Gets a Restraining Order Against Overly Attached Security Software
  • Microsoft Builds a Fence Between Windows and Its Rowdy Security Neighbors
  • Windows Gets a Kernel of Truth After CrowdStrike Meltdown
  • Microsoft Kicks Security Vendors Out of the Kernel Clubhouse
  • The Great Kernel Divorce: When Windows Said “It’s Not You, It’s Your Access Level”
  • Google’s Environmental Report Card: A+ for Effort, C- for Supply Chain
  • The Cloud Pod Goes Green: Google’s 10th Annual Carbon Confession
  • Watts Up Doc? Google’s Energy Efficiency Bugs Bunny Would Approve
  • Terminal Velocity: Google’s AI Gets a Command Performance
  • Ctrl+Alt+Gemini: Google’s New CLI Companion
  • The Prompt and the Furious: Tokyo Terminal
  • AI See What You Did There: Google’s New Compliance Framework
  • Control Yourself: Google Cloud Gets Serious About AI Auditing
  • The Audit-omatic: Teaching Old Compliance New AI Tricks
  • Veo 3: Now Playing in a Cloud Near You
  • Google’s Video Dreams Come True (Audio Included)
  • Lights, Camera, API Action: Veo 3 Takes the Stage
  • Prometheus Unbound: Azure Finally Sees What It’s Been Missing
  • VS Code Gets Fabric-ated: Now With 100% More Workspace Management
  • Ctrl+S Your Sanity: Fabric Items Now Created Where You Code
  • The Extension Cord That Connects Your IDE to the Data Cloud
  • Logic Apps Gets Its Template of Doom (But in a Good Way)
  • Copy-Paste Engineering Just Got an Azure Upgrade
  • Microsoft Introduces the IKEA Model for Workflow Assembly
  • WAF’s Up Doc? Security Copilot Now Speaks Firewall
  • The Firewall Whisperer: When AI Meets Web Application Security
  • WAF and Peace: Microsoft’s Treaty Between Security Tools
  • Azure Goes Wild(card) with Certificate Management
  • Front Door Finally Gets Its Wild Side
  • Microsoft Deals Everyone a Wildcard
  • IP Freely: Azure Takes the Guesswork Out of Address Management
  • No More IP Envy: Azure Catches Up to AWS’s Address Game
  • Azure’s New Feature Has All the Right Addresses
  • Terraform and Chill: When Infrastructure Meets AI
  • DynamoDB Goes Global: Now with 100% Less Eventually
  • The Consistency Chronicles: Return of the Strong Read
  • Breaking: DynamoDB Achieves Peak Table Manners Across All Regions
    • Follow Up

    00:47 Microsoft changes Windows in attempt to prevent next CrowdStrike-style catastrophe – Ars Technica

    • Microsoft is creating a new Windows endpoint security platform that allows antivirus vendors to operate outside the kernel, preventing catastrophic system-wide failures like the CrowdStrike incident that grounded flights and disrupted global services in 2024.
    • The CrowdStrike outage highlighted a fundamental Windows architecture problem where security software with kernel access can crash entire systems during boot, forcing IT teams to manually fix millions of machines one by one.
    • This architectural change represents Microsoft’s attempt to balance security vendor needs with system stability, potentially ending decades of kernel-level access that has been both a security necessity and reliability nightmare.
    • Cloud and enterprise IT professionals should care because this could dramatically reduce the blast radius of security software failures, preventing single bad updates from taking down entire fleets of servers and workstations.
    • The move signals a broader industry shift toward isolation and resilience in system design, where critical security functions can operate effectively without having the power to bring down the entire operating system.

      • 02:14 Matt – “I feel like this is also just a fundamental change in the way that we run infrastructure nowadays. Back in the day, you had these mainframes that were massive and you didn’t really care, because you protected them and you were very careful about them and what was on them. But now it’s thousands of small systems that you care because when Ryan has to go log into 1000 systems, he gets very angry at life and starts muttering things under his breath.”

      AI Is Going Great – Or How ML Makes Money 

      04:09 Introducing pay per crawl: enabling content owners to charge AI crawlers for access

      • Cloudflare introduces pay-per-crawl, a private beta feature that implements HTTP 402 Payment Required to enable content owners to charge AI crawlers for access. 
      • The system uses Web Bot Auth with Ed25519 key pairs and HTTP Message Signatures to verify crawler identity and prevent spoofing.
      • Content owners can set flat per-request pricing across their domain and configure three access levels for each crawler: Allow (free access), Charge (require payment at configured price), or Block (deny access with no payment option). 
      • Cloudflare acts as the Merchant of Record, handling billing aggregation and payment distribution.
      • Crawlers can discover pricing reactively by receiving 402 responses with crawler-price headers, or proactively by including crawler-max-price headers in initial requests. 
      • Successful paid requests return HTTP 200 with crawler-charged headers confirming the transaction amount.
      • The implementation integrates with existing web infrastructure after WAF and bot management policies are applied, requiring minimal changes to current security configurations. 
      • Publishers retain the flexibility to bypass charges for specific crawlers to accommodate existing content partnerships.
      • This approach enables future programmatic negotiations between AI agents and content providers, potentially supporting dynamic pricing based on content type, usage patterns, or application scale. 
      • The framework could extend beyond simple per-request pricing to include granular licensing for training, inference, or search applications.

        • 07:13 Matt – “I think this is interesting and seeing also how the bots kind of negotiate pricing. I’m picturing like a spot market in the future.’

        Cloud Tools 

        08:48 Introducing Open Source OpenAI Terraform Provider | mkdev

        • mkdev released an open-source Terraform provider for OpenAI that enables Infrastructure as Code management of OpenAI resources, eliminating the need for manual ClickOps configuration and ensuring consistent security and productivity across projects.
        • The provider supports both OpenAI Administration APIs for managing projects, service accounts, and user permissions, as well as Platform APIs that allow developers to integrate generative AI capabilities directly into their infrastructure deployments.
        • A unique capability demonstrated is “vibe coding,” where developers can use Terraform to generate application code via GPT-4, create images with DALL-E, and automatically deploy the results to AWS Lambda – essentially building and deploying AI-generated applications in a single Terraform run.
        • The provider requires two separate API keys (admin and standard) and handles OpenAI’s API limitations cleverly, such as tracking and restoring rate limits to default states since there’s no API endpoint for deletion.
        • This tool enables platform engineering teams to create self-service modules where non-developers can go from idea to deployed application using prompts, all while maintaining compliance and security through existing Terraform infrastructure.

          • 11:19 Ryan- “…the funny thing is, when I try to imagine the run through of this, like the whole end-to-end resources, like you’re right. This is enterprise – it’s definitely to keep in line with other compliance and procedure steps. But it’s also funny to me, because anyone who’s doing vibe coding, I just don’t think they’re going to go through this endpoint, this whole process to get the resources deployed.”

          AWS

          14:26 Amazon FSx for OpenZFS now supports Amazon S3 access without any data movement | AWS News Blog

          • Amazon FSx for OpenZFS now allows direct S3 API access to file data through S3 Access Points without moving or copying data, enabling use with AWS AI/ML services like Bedrock and SageMaker that expect S3 as their data source.
          • Organizations can attach hundreds of S3 Access Points to a single FSx file system with granular IAM permissions per access point, while maintaining existing NFS access and file system capabilities.
          • The feature delivers first-byte latency in tens of milliseconds (which you need when training models) with performance scaling based on FSx provisioned throughput (because you want to burn money) though customers pay both FSx charges plus S3 request and data transfer costs.
          • Real-world applications include building https://aws.amazon.com/what-is/retrieval-augmented-generation/ with Bedrock Knowledge Bases, training ML models with SageMaker, and running analytics with Athena and Glue directly against FSx-stored enterprise file data.
          • Currently available in 9 AWS regions, including US East, US West, Europe, and Asia Pacific, addressing the common challenge of enterprises wanting to leverage their migrated file data with cloud-native services.

            • 17:17 Ryan- “They’re definitely touting up the compliance features of this. I noticed how heavy this was on access points and the IM restrictions, which I mean, in practice is really difficult to support. But it’s good, you know, I like the idea that you grant API access with a certain level of permissions, but then you can tailor that down via individual permissions per access point, especially with AI and ML workloads.”

            21:08 New Amazon EC2 C8gn instances powered by AWS Graviton4 offering up to 600Gbps network bandwidth | AWS News Blog

            • AWS launches C8gn instances powered by Graviton4 processors, delivering up to 600Gbps network bandwidth – the highest among EC2 network optimized instances. 
            • These instances offer 30% better compute performance than previous C7gn instances with up to 192 vCPUs and 384 GiB memory.
            • The new 6th generation AWS Nitro Card enables the 600Gbps bandwidth, making C8gn ideal for network-intensive workloads like virtual firewalls, load balancers, DDoS appliances, and tightly-coupled cluster computing. This positions AWS ahead of competitors in network performance for specialized workloads.
            • C8gn maintains similar vCPU and memory ratios to C7gn instances, simplifying migration for existing customers. Available initially in US East and US West regions with standard purchasing options including On-Demand, Savings Plans, and Spot instances.
            • The timing aligns with growing demand for high-bandwidth applications in security, analytics, and distributed computing. Organizations running network appliances or data-intensive workloads can consolidate infrastructure with fewer, more powerful instances.
            • Cost considerations remain important – while AWS hasn’t disclosed pricing, the 3x bandwidth increase over C7gn suggests premium pricing. 
            • Customers should evaluate whether their workloads can fully utilize the 600Gbps capability to justify potential cost increases.

              • 23:22 Matt – “They’re getting the bandwidth higher that is directly exposed to the end consumer. If you are running this bandwidth, one, I would love to understand what you’re doing besides inference and training models. But two, I’m just jealous. I feel like Azure doesn’t have good Graviton yet. And even when they do, if you’re running a Windows-based workload, you can’t even leverage them yet.”

              26:37 Build the highest resilience apps with multi-region strong consistency in Amazon DynamoDB global tables | AWS News Blog

              • DynamoDB global tables now support Multi-Region strong consistency (MRSC), enabling zero Recovery Point Objective (RPO) for critical applications like payment processing and financial services that need guaranteed access to the latest data across regions.
              • MRSC requires three AWS Regions configured as either three full replicas or two replicas plus a witness node that stores only change data, reducing costs while maintaining resilience – available in 9 regions including US East, US West, Asia Pacific, and Europe.
              • Applications can enable strong consistency by setting ConsistentRead=True in their API calls, allowing developers to choose between eventual consistency for performance or strong consistency for critical operations on a per-request basis.
              • Pricing follows existing global tables structure which AWS recently reduced by up to 67%, making this enterprise-grade resilience more accessible for organizations building multi-region applications.
              • The feature addresses a gap between DynamoDB‘s multi-AZ architecture and the needs of financial services and payment processors that require immediate consistency across regions during rare regional failures.

                • 28:50 Matt – “I look at it on the other side where, yes, this is definitely a useful feature, definitely something that I can see many use cases for – healthcare data, financial services, that high criticality of consistency, but also like S3 only was strongly consistent a couple years ago.”

                GCP

                31:35 Read Google’s 2025 Environmental Report

                • Google achieved a 12% reduction in data center energy emissions despite a 27% increase in electricity demand, demonstrating successful decoupling of operational growth from carbon emissions through 25 clean energy projects that added 2.5 gigawatts to their grid capacity.
                • The company’s data centers now operate at 84% less overhead energy than the industry average, while their seventh-generation Ironwood TPU uses nearly 30 times less energy than their first Cloud TPU from 2018, positioning GCP as a leader in energy-efficient AI infrastructure.
                • Google’s AI-powered products, including Nest thermostats, Solar API, and fuel-efficient routing in Maps,2 helped customers reduce an estimated 26 million metric tons of CO2 equivalent in 2024, equivalent to removing energy use from 3.5 million U.S. homes for a year.
                • The company is investing in next-generation energy solutions, including advanced nuclear partnerships with Kairos Power and enhanced geothermal projects with Fervo to address the growing energy demands of AI workloads and ensure reliable, clean power for future data center expansion.
                • While data center emissions decreased, total supply chain emissions increased 11% year-over-year, highlighting challenges in regions like Asia Pacifi,c where clean energy infrastructure remains limited and the need for broader ecosystem transformation beyond Google’s direct operations.

                  • 36:04 Google announces Gemini CLI: your open-source AI agent

                  • Google launches Gemini CLI as an open-source AI agent that brings Gemini 2.0 Flash directly to the terminal with 60 requests per minute and 1,000 daily requests free for developers using a personal Google account.
                  • The tool integrates with Gemini Code Assist across free, Standard, and Enterprise plans, providing AI-powered coding assistance in both VS Code and the command line with a 1 million token context window.
                  • Built-in capabilities include Google Search grounding for real-time context, Model Context Protocol support for extensibility, and automation features for script integration, positioning it as a versatile utility beyond just coding tasks.
                  • The Apache 2.0 open-source license allows developers to inspect, modify, and contribute to the codebase while supporting custom prompts and team configurations through GEMINI.md system prompts.
                  • Professional developers requiring multiple simultaneous agents or specific models can use Google AI Studio or Vertex AI keys for usage-based billing, offering flexibility between free personal use and enterprise deployment options.

                    • 38:22 Ryan – “These aren’t quite in the terminal, which is what always bothers me, right? Neither Claude Code or Gemini CLI. I’ve played around both now. These are to take over a terminal, and then you’re sort of interacting with it a lot like a desktop app or the browser from that point. And so it’s kind of good, but it’s not quite what I want. I found that the IDE integration for both of those tools is way more powerful than the actual CLI tool.”

                    40:58 Audit smarter: Introducing our Recommended AI Controls framework | Google Cloud Blog

                    • Google Cloud launches the Recommended AI Controls framework in Audit Manager, providing automated compliance assessments for generative AI workloads based on NIST AI Risk Management Framework and Cyber Risk Institute standards. 
                    • This addresses the growing challenge of proving AI systems comply with internal policies and regulations as organizations deploy more AI agents and automation.
                    • The framework automates evidence collection across Vertex AI and supporting services like Cloud Storage, IAM, and VPC Networks, replacing manual audit checklists with continuous monitoring capabilities. 
                    • Organizations can schedule regular assessments and generate one-click compliance reports with direct links to collected evidence.
                    • Key controls include disabling root access on Vertex AI Workbench instances, enforcing Customer Managed Encryption Keys (CMEK) for data protection, implementing vulnerability scanning through Artifact Analysis, and restricting resource service usage based on environment sensitivity. 
                    • The framework clearly delineates control responsibilities between the customer and the platform under Google’s shared fate model.
                    • This positions Google Cloud competitively against AWS and Azure by offering AI-specific compliance automation, while their solutions remain more generic. The integration with Security Command Center provides a unified view of AI security posture alongside traditional cloud workloads.
                    • Available now through the Google Cloud Console Compliance tab, the service targets enterprises in regulated industries like healthcare and finance that need to demonstrate AI governance. No specific pricing was mentioned, suggesting it may be included with existing Security Command Center licensing.

                      • 44:09 Ryan – “It’s all just open-ended questions and really just a whole lot of movement to try to look good, and not have egg on your face because you don’t really know what the AI workloads are across your business. And so I do like that this is rolled into the compliance manager and security pan center because that means it’s centralized. It means it’s hooked up at the org layer, which means I can turn it on and I can get the glaring red reports – or magically it’s all green somehow.”

                      Azure

                      47:30 [In preview] Public Preview: Azure Monitor ingestion issues with Azure Monitor Workspace

                      • Azure Monitor Workspace now provides visibility into Prometheus metrics ingestion errors, helping customers identify and troubleshoot issues when Azure Managed Prometheus sends metrics to their workspace.
                      • This feature addresses a common operational blind spot where metrics fail to ingest but customers lack visibility into why, similar to AWS CloudWatch Metrics Insights but specifically for Prometheus workloads.
                      • The platform metrics integration means ingestion errors appear alongside other Azure Monitor metrics, enabling unified monitoring and alerting without additional tooling or configuration.
                      • Target customers include organizations running Kubernetes workloads with Prometheus monitoring who need enterprise-grade observability and troubleshooting capabilities for their metrics pipeline.
                      • This preview feature comes at no additional cost beyond standard Azure Monitor Workspace charges, making it accessible for teams already invested in Azure’s Prometheus ecosystem.

                        • 51:32 Microsoft Fabric Extension in VS Code

                        • Microsoft Fabric Extension for VS Code now allows developers to create, delete, and rename any Fabric item directly within their IDE, eliminating context switching between VS Code and the Fabric portal for basic workspace management tasks.
                        • The new tenant switching capability enables users to manage workspaces and items across multiple Microsoft tenants from a single VS Code instance, addressing a common pain point for consultants and developers working with multiple organizations.
                        • This positions Microsoft Fabric as a more developer-friendly analytics platform compared to AWS and GCP offerings, which typically require separate web consoles or CLI tools for similar workspace management operations.
                        • The integration targets data engineers and analysts who prefer working in VS Code for their development workflow, particularly those managing multiple Fabric workspaces for different clients or projects.
                        • While the feature itself is free as part of the VS Code extension, users should note that Fabric items created through VS Code still incur standard Fabric capacity costs based on the compute and storage resources consumed.

                          • 53:43 Matt – “This to me is a consultant feature, where you need that feature…the average consumer that works for a single company – odds are you’re not going to use this.” 

                          54:39 Announcing Public Preview: Organizational Templates in Azure Logic Apps

                          • Azure Logic Apps now lets organizations create and share private workflow templates within their tenant, addressing the gap where teams previously had to either use public Microsoft templates or build everything from scratch. 
                          • This brings Logic Apps closer to AWS Step Functions’ reusable workflow patterns while maintaining enterprise control through Azure RBAC integration.
                          • The new UI eliminates manual packaging by automatically extracting connections, parameters, and documentation from existing workflows, making template creation accessible to non-developers – a notable improvement over competitors, where creating reusable automation patterns often requires significant technical expertise.
                          • Templates support both test and production publishing modes with full lifecycle management, allowing enterprises to safely experiment with automation patterns before wide deployment, particularly useful for organizations standardizing on specific integration patterns or enforcing architectural guidelines across teams.
                          • As first-class Azure resources, these templates integrate with existing subscription and role-based access controls, ensuring teams only see templates they’re authorized to use – this addresses a common enterprise concern about sharing internal APIs and business logic without exposing them publicly.
                          • The feature targets enterprises looking to scale their automation efforts by packaging common patterns like API integrations, data processing workflows, or approval chains into reusable components – reducing development time from hours to minutes for repetitive integration scenarios.

                            • 56:18 Matt – “I love this. I mean, building step functions in the past, I’ve used logic apps only a few times in my day job, but building step functions, being able to share them across the organization and having people do a simple function app to Teams integration (because it’s not simple, because it’s Microsoft Teams) or anything along those lines, like these reusable patterns, connections to Jira, connections to other internal systems, your SRE notification system – and just being able to say, grab this, run it, and be done with it, is so much better than even saying, hey, try to grab this Terraform module, and then having people maintain it and update it because you all know that no one’s going to actually do that.”

                            58:54 [Launched] Generally Available: Azure WAF integration in Microsoft Security Copilot 

                            • Azure WAF integration with Microsoft Security Copilot is now generally available, supporting both Azure Front Door WAF and Azure Application Gateway WAF configurations. 
                            • This allows security teams to investigate and respond to web application threats using natural language queries within the Security Copilot interface.
                            • The integration enables security analysts to query WAF logs, analyze attack patterns, and generate incident reports without switching between multiple tools or writing complex KQL queries. (Trust us, you don’t want to do that.) 
                            • This reduces the time needed to investigate web application security incidents from hours to minutes.
                            • Microsoft continues to expand Security Copilot’s reach across its security portfolio, positioning it as a central hub for security operations. AWS offers similar WAF capabilities but lacks the AI-powered natural language interface, while GCP’s Cloud Armor requires more manual log analysis.
                            • Target customers include enterprises with complex web applications that need to streamline security operations and reduce alert fatigue. The integration is particularly valuable for organizations already invested in the Microsoft security ecosystem.
                            • Pricing follows the Security Copilot consumption model at $4 per Security Compute Unit (SCU), with no additional charges for the WAF integration itself. Organizations should consider the SCU consumption when enabling automated investigations and report generation.

                              • 1:00:57 Ryan – “…anything that allows me to query things with natural language and not some specific DSL to figure out, I do appreciate. It’s been useful in so many other tools. WAF seems like the best use case, really, because there’s so much noise trying to get VPC flow logs, like raw networking related.”

                              1:03:48 [Launched] Generally Available: Azure Front Door now supports managed certificate for wildcard domains

                              • Azure Front Door now automatically provisions and manages SSL certificates for wildcard domains (*.example.com), eliminating the need to manually upload and maintain your own certificates for securing multiple subdomains under a single domain.
                              • This feature brings Azure Front Door to parity with AWS CloudFront and Google Cloud CDN, both of which have offered managed wildcard certificates for years, making multi-subdomain deployments simpler for enterprises.
                              • The managed certificate service is available for both Standard and Premium tiers at no additional cost beyond the standard Azure Front Door pricing, reducing operational overhead for DevOps teams managing multiple staging, regional, or customer-specific subdomains.
                              • Key use cases include SaaS providers offering customer-specific subdomains (customer1.app.com, customer2.app.com) and enterprises with multiple regional or environment-based subdomains that need consistent SSL coverage without certificate management complexity.
                              • The feature integrates with Azure’s existing certificate lifecycle management, automatically handling renewal before expiration and supporting up to 100 subdomains per wildcard certificate.

                                • 1:06:58 [Launched] Azure Virtual Network Manager IP address management

                                • Azure Virtual Network Manager‘s IP address management feature brings centralized IP planning and allocation to complex network environments, addressing a common pain point for enterprises managing multiple VNets and subnets across regions.
                                • The feature provides automated IP address allocation, conflict detection, and visual network topology mapping, similar to AWS VPC IP Address Manager but integrated directly into Azure’s Virtual Network Manager service.
                                • This targets large enterprises and managed service providers who struggle with IP address sprawl across hybrid and multi-region deployments, reducing manual tracking errors and IP conflicts.
                                • Unlike AWS IPAM, which requires separate configuration, Azure’s implementation is built into Virtual Network Manager, potentially simplifying adoption for existing Azure customers already using VNM for network governance.
                                • Pricing follows Virtual Network Manager’s model at $0.02 per managed resource per hour, making it cost-effective for organizations already invested in Azure’s network management ecosystem.

                                  • 1:09:56 Matt – “It has to be a system that’s maintained – otherwise it’s garbage in, garbage out.” 

                                  Closing

                                  And that is the week in the cloud! Visit our website, the home of the Cloud Pod, where you can join our newsletter, Slack team, send feedback, or ask questions at theCloudPod.net or tweet at us with hashtag #theCloudPod

                                  ...more
                                  View all episodesView all episodes
                                  Download on the App Store
                                  tcp.fmBy Justin Brodley, Jonathan Baker, Ryan Lucas and Matt Kohn