Last week Twitter gave their millions of users a lesson on the impacts of rate limiting a service. In this episode, I will discuss why that was an incredibly ham-fisted way of tackling web scraping.
"To address extreme levels of data scraping & system manipulation, we’ve applied the following temporary limits:
Verified accounts are limited to reading 6000 posts/dayUnverified accounts to 600 posts/dayNew unverified accounts to 300/day"Ref: https://twitter.com/elonmusk/status/1675187969420828672
After that it became clear this was a global issue, and it was deliberate policy instead of a bug!If we take that at face value, and ignore the rumours of cloud bills not being paid on time, this is still an incredibly ham-fisted way of tackling web scraping.If you have a public website, it will be scraped, usually for legitimate reasons like the Google bot scraping your pages to add them to it's search index.Beyond that, if a scraper is behaving in an aggressive manner, it can be blocked via Web Application Firewalls (WAFs) or other types of application gateways.Typically, that block will be via IP address or user agent string, or a combination of both.Following that approach, there is zero reason to rate limit legitimate users, especially for verified users who are paying customers!For legitimate large-scale scraping use cases, Twitter provides a rate-limited API with various free and paid tiers. Ref: https://developer.twitter.com/en/docs/twitter-apiPutting rate limits in place there makes perfect sense, and all scrapers should be encouraged to go to the API rather than scraping the site. It is a good revenue generator for Twitter.All companies with public APIs do this, it's common practice in the industry.At no point however should regular users be impacted, and the fallout of this embarrassing instance has once again left a lot of users worrying about the stability of Twitter not only as a platform, but also as a firm.At one point, #RIPTwitter and #Bluesky were trending on Twitter, while as of writing, the tweet above from Elon has received 550m views.This is serious reputational harm not only for Twitter, but for Elon Musk if this service ultimately fails for technical or commercial reasons.I still can't see the end-game here.What I am working on this week:Tech Leadership podcast series: episode 23 will be on "Turning up at meetings", and being present in general.greppr.org : now at 1.8m web pages indexed.Media I am enjoying this week:Doom 2016 replayThe Sound of Waves by Yukio MishimaLongitude by Dava SobelNotes and subscription links are here: https://techleader.pro/a/598-Tech-Leader-Pro-podcast-2023-week-27,-the-Twitter-rate-limit-fiasco
View all episodes
