Tech Leader Pro

Tech Leader Pro podcast 2023 week 38, analysis of a bot attack on greppr

Listen Later

This week my web search engine greppr came under a bot attack, in this episode I will discuss what happened and how I fixed it.

Notes:

  • I am running late this week due to exhaustion.
  • Last week, my web search engine greppr.org was attacked by a bot.
  • Initially I thought it was real traffic: I seen search queries coming in from multiple IPs, that on first impressions looked like real traffic.
  • However the volume was huge! At least, huge for my fledgling service that basically nobody is using.
  • On September 18th, I seen 30K additional search queries, and a further 25K on the 19th. I got excited!
  • On closer examination however, I could see patterns in the search queries: lots of weird searches for maid and cleaning services, from random IPs and user agent strings, all seconds apart from apparently different users.
  • Why did they do this? Perhaps they think they can inject this content into my site?
  • The fix: dynamic query string parameters that change daily, using an unpredictable pattern. Now, the bot gets 404 responses (it is still running).
  • Good bots should identify themselves via the user-agent header in HTTP requests: this is what the web crawler Greppr uses does, along with the bots from big search engines like Google and Bing.
  • Web security remains an arms race.
  • What I am working on this week:
    • Blog: "Choose your mentors wisely" - https://techleader.pro/a/614-Choose-your-mentors-wisely
    • Media I am enjoying this week:
      • Diaspora by Greg Egan.

        • Notes and subscription links are here: https://techleader.pro/a/615-Tech-Leader-Pro-podcast-2023-week-38,-analysis-of-a-bot-attack-on-greppr

        ...more
        View all episodesView all episodes
        Download on the App Store
        Tech Leader ProBy John Collins