In this episode of Search Off the Record, Martin and Gary from the Google Search Relations team take a deep dive into how Googlebot and web crawling work—past, present, and future. Through their humorous and thoughtful conversation, they explore how crawling evolved from the early days of the internet, when scripts could index a chunk of the web from a single homepage, to the more complex and considerate systems used today. They discuss the basics of what a crawler is, how tools like cURL or Wget relate, and how policies like robots.txt ensure crawlers play nice with web infrastructure.   The conversation also covers Google's internal shift to unified infrastructure for all crawling needs, highlighting how different teams moved from separate crawlers to a shared system that enforces consistent policies. They explain why some fetches bypass robots.txt (like user-initiated actions) and the rising impact of automated traffic from new products and AI agents. With a nod to initiatives like Common Crawl, the episode ends with a look at the road ahead, acknowledging growing internet congestion but remaining optimistic about the web’s capacity to adapt. Resources: Listen to more Search Off the Record → https://goo.gle/sotr-yt Subscribe to Google Search Channel → https://goo.gle/SearchCentral   Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.   #SOTRpodcast #SEO #SearchOfTheRecord   Speakers: Martin Splitt, Gary Illyes Products Mentioned: Googlebotl, Gemma, Google AI

In this episode of Search Off the Record, Gary Illyes and Martin Splitt from the Google Search team dive deep into the foundations of how the web works—specifically HTTP, TCP, UDP, and newer technologies like QUIC and HTTP/3. The two reflect on how even experienced web professionals often overlook or forget the mechanics behind these core protocols, sharing insights through technical discussion, playful banter, and analogies ranging from messenger pigeons to teapots. The conversation spans key concepts like packet transmission, connection handshakes, and the importance of status codes such as 404, 204, and even 418 (“I’m a teapot”). Throughout the conversation, they connect these protocols back to real-world implications for site owners, developers, and SEOs—like why Search Console might report network errors, and how browser or server behavior is influenced by low-level transport decisions. With a mix of humor and expertise, Gary and Martin aim to demystify a crucial part of the internet's infrastructure and remind listeners of the layered complexity that makes modern web experiences possible. Resources: Episode transcript →  Listen to more Search Off the Record → https://goo.gle/sotr-yt Subscribe to Google Search Channel → https://goo.gle/SearchCentral Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team. #SOTRpodcast #SEO #Http Speakers: Lizzi Sassman, John Mueller, Martin Splitt, Gary Illyes Products Mentioned: Search Console - General

In this episode of Search Off the Record, Google's Martin Splitt, Cherry, and Gary Illyes discuss the exciting expansion of Search Central Live with the introduction of Search Central Live Deep Dive. They share the history of Search Central events, the inspiration behind creating a more advanced, multi-day experience, and how community feedback helped shape the new format. Cherry explains how the Deep Dive events will offer in-depth sessions, practical workshops, and greater opportunities for networking within the Search community across the APAC region.   The team also offers a behind-the-scenes look at the complexities of planning international events, from budgeting and travel logistics to ensuring high-quality attendee experiences. They highlight plans for new locations including Budapest and South Africa, and outline how interested participants can apply. If you want to hear about the future of Google's Search community events and how to get involved, this episode is a must-watch.   Resources: Episode transcript → https://goo.gle/sotr090-transcript   Listen to more Search Off the Record → https://goo.gle/sotr-yt Subscribe to Google Search Channel → https://goo.gle/SearchCentral

Ever wondered how web standards are made? Martin and Gary from Google Search take you behind the scenes of the internet's governing bodies. From the IETF to the W3C, learn about the consensus-driven processes that shape the web. Find out why these standards are crucial for ensuring a consistent and reliable online experience. Resources: Episode transcript →  https://goo.gle/sotr089-transcript    Listen to more Search Off the Record → https://goo.gle/sotr-yt Subscribe to Google Search Channel → https://goo.gle/SearchCentral   Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.   #SOTRpodcast #SEO   Speakers: Lizzi Sassman, John Mueller, Martin Splitt, Gary Illyes
Products Mentioned: Search Console - General

As 2024 comes to a close, we're reflecting on what SEO things happened this year. Listen along as we chat about what we learned from going to events, the importance of understanding how AI works, and whether everything will be replaced with JavaScript.   Resources: Episode transcript → https://goo.gle/sotr088-transcript Encoding + Googlebot → https://goo.gle/4iZEKXO   Listen to more Search Off the Record → https://goo.gle/sotr-yt Subscribe to Google Search Channel → https://goo.gle/SearchCentral   Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.

In this episode of Search Off the Record, John and Martin talk to Allan Scott from the Google Search team that handles duplicate content. They explore how Google Search deals with duplicated content, clustering, and canonicalization.   Resources: Episode transcript → https://goo.gle/sotr087-transcript   Listen to more Search Off the Record → https://goo.gle/sotr-yt Subscribe to Google Search Channel → https://goo.gle/SearchCentral   Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.   #SOTRpodcast  #SEO   Speakers: Allan Scott, John Mueller, Martin Splitt Products Mentioned: Search Console - General

In this episode of Search Off the Record, Lizzi and John are joined by Rio Ichikawa, a product manager working on UX, CRO, and SEO, to discuss the intersection of SEO and UX. They explore how user experience impacts SEO, emphasizing the shared goals of attracting and converting users. Tune in for a discussion on common UX mistakes - like carousels - and how heatmaps can be a valuable tool for understanding user behavior and optimizing web pages for better engagement and conversion rates.    Resources: Episode transcript → https://goo.gle/sotr086-transcript Rio on X/Twitter  → https://goo.gle/4eAOjJz Rio on LinkedIn → https://goo.gle/3OnaHuY    Listen to more Search Off the Record → https://goo.gle/sotr-yt Subscribe to Google Search Channel → https://goo.gle/SearchCentral   Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.

Sometimes talking to your users is the best way to figure out why that pesky bounce rate is high. But how do you actually do that, especially if you're on a budget?   In this episode of Search Off the Record, we're learning all about user experience research (UXR) and how getting a better understanding of how your users interact with your website can help when you're looking for ways to improve it. Tune in as Iva Barisic Hafner, a user experience researcher on the Google Shopping team, explains how to use UXR methods to get to the bottom of the "why" behind user behavior.   Resources: Episode transcript → https://goo.gle/sotr085-transcript Nielsen Norman Group → https://goo.gle/3YI3xrq List of UXR methods  → https://goo.gle/40sI6Ma Iva on LinkedIn → https://goo.gle/3A7Te6E   Listen to more Search Off the Record → https://goo.gle/sotr-yt Subscribe to Google Search Channel → https://goo.gle/SearchCentral   Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.

What does recursion, Weather Frog, and Momo the Magic Cat have in common?   In this episode of SOTR, Lizzi and John go behind the scenes of a Google Doodle with Jessica Yu, the lead for Doodles and delight on Google Search, about designing experiences that resonate with your audience. Listen along as we learn more about what's involved in getting an easter egg into Google Search, from hatching the idea to designing (and deploying) the experience and measuring success.   Resources: Episode transcript → https://goo.gle/sotr084-transcript Jessica Yu on LinkedIn  → https://goo.gle/3BKgSq1 Momo's first appearance → https://goo.gle/3UbGgeC Submitting an idea for the Google Doodle → https://goo.gle/3A04stC AVIF is now supported by Google → https://goo.gle/3NrwhhK   Listen to more Search Off the Record → https://goo.gle/sotr-yt Subscribe to Google Search Channel → https://goo.gle/SearchCentral   Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.

In this episode of Search Off the Record, Ben Walton and David Yule from the Google Search Site Reliability Team (SRE) walk us through their day to day work, which surprisingly is not just about putting out fires! Learn what Search SREs really do, how they handle incidents that occasionally occur, and what it takes to be an SRE for Google Search.    Resources: Episode transcript → https://goo.gle/sotr083 SRE handbook → https://goo.gle/3XSbVUF David on LinkedIn → https://goo.gle/3XQKavR Ben on LinkedIn → https://goo.gle/4ewAkEX   Listen to more Search Off the Record → https://goo.gle/sotr-yt Subscribe to Google Search Channel → https://goo.gle/SearchCentral   Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.