Search Off the Record

What is a web crawler, really?


Listen Later

In this episode of Search Off the Record, Gary Illyes and Lizzi Sassman take a deep dive into crawling the web: what is a web crawler, and how does it really work? Listen along as the Search team is joined by an expert web developer in the SEO community, Dave Smart, for an in-depth and technical discussion of all things crawling, and maybe dispel some myths along the way.

Resources:

Episode transcript → https://goo.gle/sotr070-transcript

Managing your crawl budget → https://goo.gle/3IzRZxl

Dave Smart on LinkedIn → https://goo.gle/3wPSuRA

Tame the Bots → https://goo.gle/4cfCQ1P

Search Central Help Forum → https://goo.gle/sc-forum

Indexing API docs → https://goo.gle/3v8yVU0

Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.

#SOTRpodcast

Speaker: Gary Illyes, Lizzi Sassman

Products Mentioned: Search Off The Record

...more
View all episodesView all episodes
Download on the App Store

Search Off the RecordBy Google

  • 4.2
  • 4.2
  • 4.2
  • 4.2
  • 4.2

4.2

137 ratings


More shows like Search Off the Record

View all
Planet Money by NPR

Planet Money

30,642 Listeners

Pivot by New York Magazine

Pivot

9,502 Listeners

Social Media Marketing Podcast by Michael Stelzner, Social Media Examiner

Social Media Marketing Podcast

1,437 Listeners

The Digital Marketing Podcast by Daniel Rowles and Ciaran Rogers

The Digital Marketing Podcast

114 Listeners

Marketing School - Digital Marketing and Online Marketing Tips by Eric Siu and Neil Patel

Marketing School - Digital Marketing and Online Marketing Tips

1,254 Listeners

Syntax - Tasty Web Development Treats by Wes Bos & Scott Tolinski - Full Stack JavaScript Web Developers

Syntax - Tasty Web Development Treats

985 Listeners

The Diary Of A CEO with Steven Bartlett by DOAC

The Diary Of A CEO with Steven Bartlett

8,716 Listeners

My First Million by Hubspot Media

My First Million

2,645 Listeners

Morning Brew Daily by Morning Brew

Morning Brew Daily

3,005 Listeners

All-In with Chamath, Jason, Sacks & Friedberg by All-In Podcast, LLC

All-In with Chamath, Jason, Sacks & Friedberg

9,900 Listeners

No Stupid Questions by Freakonomics Radio + Stitcher

No Stupid Questions

3,653 Listeners

A Bit of Optimism by Simon Sinek

A Bit of Optimism

2,128 Listeners

Prof G Markets by Vox Media Podcast Network

Prof G Markets

1,340 Listeners

AI Explored by Michael Stelzner, Social Media Examiner—AI marketing

AI Explored

85 Listeners

OpenAI Podcast by OpenAI

OpenAI Podcast

52 Listeners