When should you use a browser for web scraping? John and Ayan join Zyte’s Paweł to explore CDP, Playwright, and the trade-offs between running browsers locally and using managed infrastructure. From fingerprinting and session management to pricing and AI agents, discover what it takes to run browser-based scraping at scale.

In this episode, Ayan and John explore the rapidly evolving world of AI agents, coding harnesses and custom developer tools. They discuss emerging AI models, modular agent ecosystems, web scraping, prompt-to-data workflows, AI evaluation and the security risks of putting agents into production.

They also share new Scrapy projects, reflect on the power of open-source software and Linux, and look at the rise of proactive desktop agents. Plus, there’s time for model pricing, developer workflows and a well-earned appreciation of croissants.

Episode 10 of the Zyte podcast is out, and it is one of the widest-ranging conversations we have recorded this year. John Rooney sits down with Neha Setia Nagpal and Ayan Pahwa for an unfiltered, all-in-style chat covering everything the team has been building, reading, and arguing about lately.

This episode Ayan and John talk to Ian Johnson, staff engineering and author of the Harness Engineering book. We talk through the subject, what it is and what it means to us as builders.

In this episode, John, Neha, and Ayan trade origin stories (requests and Beautiful Soup, a runaway crawl at IBM back in 2016, a wall of JavaScript that broke everything) and get into why a lean, extensible core is exactly what modern data extraction needs. They cover the anti-bot arms race, reverse-engineering back-end APIs, the recent httpx download handler and the move toward an async reactor, and the four plugins John reaches for on every production build.

Ayan teaches us about AI harnesses, what the system prompt is and what it does, how it affects what you actually get from your model back, and we talk about what models we actually use for our day-to-day work. 

The team meet up to chat deploying spiders, both managed and self hosted, proxies infrastructure and monitoring, and how sourdough cultures can tell us a lot about code.

In this episode we talk about the projects we have been working on, from learning hands on javascript challenges, to automating scraping with N8n and MCP servers we find interesting.

In this episode I'm joined by two of Zyte's top antiban team members to talk about some of the new techniques used by vendors to detect bots.