Beers and SEO with Barry and Steve

The Truth About AI Crawlers and Publisher Content


Listen Later

On this month's pod we cover the impact of AI on news and publishers and focus on SEO strategies, AI crawling, and protecting content.We'll also share insights on optimising for AI, handling AI bots, and industry collaboration. With the various algorithmic updates and AI policy changes there's a lot to catch up on, so let's dive in!

Key topics:

- AI impact on news and publishers
- SEO strategies for AI search
- AI crawling and bot behavior
- Content protection and regulation
- Industry collaboration and regulation

Chapters

00:00 Introduction to AI in News and Publishing
03:04 The Impact of Google's Core Updates
06:01 Back Button Hijacking and User Experience
10:05 AI Overview Accuracy and User Trust
17:12 AI Crawlers and Bots: Challenges for Publishers
30:09 Navigating AI Optimisation Strategies
31:53 The Value of AI Training Data
33:13 Challenges in AI Visibility for Brands
34:31 The Limitations of Prompt Tracking
35:47 The Importance of Collective Action in Publishing
37:41 Understanding the Low Value Exchange for Publishers
39:49 Optimising for AI Visibility in Direct Sales
41:32 The Dilemma of AI-Generated Content for Publishers
43:14 Technical SEO Strategies for AI Visibility
44:35 The Role of Semantic HTML and Structured Data
46:41 Chunking and Writing for AI vs. Humans
50:30 Crawlability and Internal Linking Strategies
53:20 Meta Tags and Open Graph Optimisation
54:12 The Future of LLMs and Markdown
57:11 Final Thoughts and Future Directions

Resources
AIO accuracies
https://www.nytimes.com/2026/04/07/technology/google-ai-overviews-accuracy.html
https://algorythmic.co/opinions/my-response-to-googles-response-to-the-nyt-ai-overview-story/
Denmark's strategy to front run Europe on Big Tech deals: https://podcasts.apple.com/au/podcast/denmarks-strategy-to-front-run-europe-on-big-tech-deals/id1800563534?i=1000745426241
3rd party scrapers selling to AI companies: https://www.linkedin.com/feed/update/urn:li:activity:7447249264881205249/
--------------------

Connect with Barry & Steve
SEO Summit 2026 - https://newsseo.io
Barry's Consultancy service: https://www.polemicdigital.com/
Steve's Consultancy service: https://www.stevenwilsonbeales.com/

...more
View all episodesView all episodes
Download on the App Store

Beers and SEO with Barry and SteveBy Steven Wilson-Beales, Barry Adams