Linear Digressions

Optimized Web Crawling


Listen Later

Got a fun optimization problem for you this week! It’s a two-for-one: how do you optimize the web crawling logic of an operation like Google search so that the results are, on average, as up-to-date as possible, and how do you optimize your solution of choice so that it’s maintainable by software engineers in a huge distributed system? We’re following an excellent post from the Unofficial Google Data Science blog going through this problem.
Relevant links: http://www.unofficialgoogledatascience.com/2018/07/by-bill-richoux-critical-decisions-are.html
...more
View all episodesView all episodes
Download on the App Store

Linear DigressionsBy Ben Jaffe and Katie Malone

  • 4.8
  • 4.8
  • 4.8
  • 4.8
  • 4.8

4.8

352 ratings


More shows like Linear Digressions

View all
Global News Podcast by BBC World Service

Global News Podcast

7,654 Listeners

Data Skeptic by Kyle Polich

Data Skeptic

476 Listeners

The Daily by The New York Times

The Daily

110,759 Listeners

Up First from NPR by NPR

Up First from NPR

55,948 Listeners

What's That Rash? by ABC listen

What's That Rash?

248 Listeners

The Ezra Klein Show by New York Times Opinion

The Ezra Klein Show

15,481 Listeners

Prof G Markets by Vox Media Podcast Network

Prof G Markets

1,166 Listeners