Scarcity is required. Cloudflare’s pay-per-crawl model aims to restore balance by creating enforceable scarcity. “If you don’t have scarcity, there’s no market. Changing the defaults was the only way for a market to develop.”

AI engines need publishers. Models rely on fresh, high-quality publisher content to stay accurate. “If you got asked a question and didn’t know what happened over the last two weeks, your answer is going to be stupider.”

Pay-per-crawl is a first step, not the finish line. Cloudflare’s system lets publishers decide whether to block, allow, or charge AI crawlers. It’s a way to test how a content economy might form. As Cohen put it, “It’s the simplest thing you can do because you can count a crawl.”

The goal is the Spotify model. Cohen pointed to Spotify’s emergence after Napster as a precedent for compromise. “There’s lots of money going to creators in that model… it took a while for that market to develop.”

The problem is Google. Publishers face a lose-lose choice: block Google entirely or allow AI Overviews to rewrite their work. “If they want search, they’re also in AI overviews… They can’t both crawl and pay nothing.”