The Economist has gone public with an experiment: it has created a shadow website featuring an AI-friendly version of its front-of-paywall content. The idea is to improve the odds of this content surfacing in AI answers and responses to AI queries. It’s based on a new standard, llms.txt, which has been described as the robot.txt of AI. What does this mean for communicators? Neville and Shel break it down in this short midweek episode.

Links from this episode:

The next monthly, long-form episode of FIR will drop on Monday, June 22.

We host a Communicators Zoom Chat most Thursdays at 1 p.m. ET. To obtain the credentials needed to participate, contact Shel or Neville directly, request them in our Facebook group, or email [email protected].

Special thanks to Jay Moonah for the opening and closing music.

You can find the stories from which Shel’s FIR content is selected at Shel’s Link Blog. You can catch up with both co-hosts on Neville’s blog and Shel’s blog.

Disclaimer: The opinions expressed in this podcast are Shel’s and Neville’s and do not reflect the views of their employers and/or clients.

Raw Transcript

Shel Holtz: Hi, everybody, and welcome to For Immediate Release. This is episode number 516. I’m Shel Holtz.

Neville Hobson: And I’m Neville Hobson.

Something quiet is happening to the web, and The Economist is one of the first major publishers to talk openly about how it’s responding. A piece published by Digiday last week describes how The Economist is building what its VP of generative AI, Josh Munker, calls two versions of the web.

One version is the one we’re all familiar with: richly designed pages, feature photography, navigation, everything optimized for a human reader browsing with intent. The other version is quite different: stripped back, structured around questions and answers, designed not for you, but for an AI agent acting on your behalf.

Now, if that framing sounds familiar, it should. In episode 515 last week, we spent some time on what Google announced at its developer conference in May: that searching the web will increasingly be done by AI agents rather than by humans, and that people will focus on acting on the information those agents surface rather than clicking links themselves. I made the point then that the question for communicators was shifting from, “How do we get found?” to, “How do we become part of the information environment that AI systems draw from?”

What The Economist is doing is a direct practical answer to exactly that question. And here’s what makes this particularly interesting. The Economist itself published a piece last December describing this shift in precise terms: a move from a pull internet, where people initiate actions, to a push model, where agents act unprompted, setting up meetings, flagging research, handling tasks, often without a human ever typing a query. They wrote about it then as an emerging phenomenon. Now, six months later, their own team is operationally responding to it. They’re not just observers of this trend; they’re participants in it.

The logic behind their approach is straightforward. A growing share of people, particularly in B2B contexts, no longer start their discovery process with a search engine or a home page. They start with ChatGPT or Gemini or Claude. They ask a question, get a synthesized answer, and may never visit the original source at all.

For a publisher like The Economist, that creates an obvious problem. If your content isn’t structured in a way that an AI agent can parse and surface clearly, you effectively become invisible. Not because your content is poor, but because the intermediary can’t read it properly.

So The Economist is experimenting. Right now, the focus is on content that already sits outside the paywall: marketing copy, B2B sales material, the kinds of pages where you want a potential subscriber or corporate client to find you. They’re building parallel versions: the polished human-facing page alongside a clean, agent-readable equivalent. The aim is to show up accurately and usefully in AI-generated answers.

Now, why does this matter to communicators beyond the publishing world? Because what The Economist is describing isn’t a publishing problem. It’s a communication problem.

And it connects to something that one researcher quoted in The Economist’s December piece put plainly: Marketers and communicators may need to pitch not to people, but to agent attention. The audience increasingly will be algorithms, and the humans will act on what these algorithms surface.

Think about your own organization’s public-facing content: press releases, executive bios, policy statements, corporate FAQs, product and service descriptions. All of that content is increasingly being read and summarized by AI agents before it ever reaches a human. If that content isn’t structured to be understood accurately by an agent, you lose control of how your organization is represented in AI-generated answers. And unlike a Google snippet, you may not even know it’s happening.

Alessandro DeSantis, a media consultant quoted in the Digiday piece, puts it bluntly. He calls agent optimization a defensive baseline, not a competitive advantage, but the minimum requirement to remain visible at all.

There’s a deeper question sitting underneath all of this, which we’ll get into: Who do you trust in the AI-intermediated world? What does it mean for the communicator’s job when the first reader of your content isn’t a person at all?

Shel, you and I discussed the Google side of this in FIR 515. Here’s a publisher responding in real time. What’s your take?

Shel Holtz: I have lots of takes on this.

This is, I think, a big issue. The first thing I want to point out is that, as I read the commentary of people who are talking about this, there’s an expectation that in the not-too-distant future, the AI version is all that we’re going to need to publish because we’re going to be publishing for AI as people rely on AI to get their information. I find this a troubling idea.

I think people are ignoring the fact that right now, 25 to 60 percent, depending on the nature of the site, of visits to a website are direct. They are not coming from a search engine. It’s somebody who already knows the URL. As I mentioned in a post I published to LinkedIn last week, nobody going to Amazon starts at Google and says “online retail site” and waits for the URL to come up. They just type Amazon.com.

There are a lot of people who know the URLs. There are URLs published in magazine articles, in advertising, in TV commercials, for example. And then there is the dark web: I send you a link by email or in our Slack channel, and you click it. There’s no search involved at all, so there is no opportunity to see that AI overview.

So I think we have to keep in mind that there are still a lot of people who are coming to our websites, not through Google or some other search mechanism, or starting with Claude or ChatGPT or Gemini or what have you. They’re coming directly to your website, either because they know the URL or it has been shared with them by somebody else. So I think we do need to keep that in mind.

The other reason I think we need to maintain our own websites is because we own them, and we don’t own that intermediary. You publish that Markdown version of a web page and you provide the proper router to it. Was it called LLM text, I think? They’re calling this the robots.txt of the AI era. And it’s going to share with the person who’s making the query what it shares. It may not be exactly what is on your page. So now you’re down to using a third party.

Neville Hobson: Something like that.

Shel Holtz: So, yeah, it’s good to have at least as a statement of record what your original content was. I have some other thoughts about this, but I’ll let you react to that first.

Neville Hobson: Yeah, no, I get it totally. Yet the trend seems to be quite clear. This is the way it’s moving. And I would say that, from what I’ve been reading, not just this, but The Economist is actually a probably good signal for what other media properties may or may not be doing or might want to do, depending on the outcome of The Economist’s experimentation. Being discoverable in this new way is critical; otherwise, you are invisible.

And I think the behaviors of people are gradually changing, not to the extent that it’s a groundswell. I don’t see it like that myself. Maybe it is generational, potentially, I suppose. But I think applying that kind of “it’s because of age differences in the different generations” is a bit of a tired argument nowadays, I think, as a kind of answer to everything.

We’re seeing a shift in how people not only find information, but how they think about going about finding information. So the idea of thinking, “Okay, let me just type into this search box: How do I do so-and-so? Where do I go to?” or whatever words you might use. I kind of talk in literally conversational language when I’m searching for something, and I’ve had more good experiences, way more than not good, with Google Overviews as a result.

And again, I sometimes stop and think, “What did I just do then?” I took Google Overview and I didn’t go any further than that because it gave me exactly what I wanted. And not only that, I’ve got a list of all the sources it looked at as well. So why am I going to spend any more time on this? I got what I need. So this is evolving fast, even though it’s still not totally clear exactly what’s happening.

And I think the idea — and this is my take on this, Shel — the idea of, as a matter of course, sharing URLs in an email or in a WhatsApp message with colleagues in the workplace, that ain’t going to be the way it’s going to be as a matter of course. There are going to be other, better ways of doing it than that, generally speaking, of course.

So this isn’t suddenly going to happen next week. This is going to take some time to evolve. But it seems from what I’ve read, Digiday being one good example, and a handful of others I’ve also looked at, this is the clear trend that’s evolving. And The Economist is clearly betting on their experiment that AI-agent content prepared for AI agents is an essential move for them to take as a publisher so that they do not become invisible. And I get that.

Shel Holtz: I agree that it’s an essential move. I’m not suggesting that organizations not do this. I’m just suggesting that they not abandon their websites. For now, we’ve got to keep that 25 to 60 percent that are coming directly to your website in mind.

You also have to keep in mind that your website may have things on it that can’t be duplicated in an AI Overview. In fact, the advice that’s circulating out there is that if you want to drive traffic still to your website, you need to have that kind of content. It may be calculators, for example, interactive elements that just can’t be summarized. You have to get to the website and interact with that tool.

I know that the ability to purchase from an AI Overview is coming. It’s not here yet. I haven’t heard a timeline for when it’s going to be here. It’s coming, but for now, if you want to make a purchase, you need to go to the website to do that. So I’m suggesting that there are still reasons to provide a website. You need to do both.

Now, when I say you need to do both, I think it’s important to understand that you don’t need to have a one-for-one of all of the content on your website. You need to make a judgment about what on your website you want to have summarized by AI and what is unnecessary or superfluous, or what you otherwise just don’t need to spend the time on. It doesn’t matter if the AI gets this particular content summarized in Markdown so that it can give you the AI summary.

So I think you have to do some strategizing. But undoubtedly, what’s coming from the web development platforms that are out there is the automation of this. I think it’s inevitable that WordPress and Wix and Squarespace and Webflow and all of the other services out there for creating websites are going to add the ability to automatically convert this page to Markdown and put it in the right place. I think that’s probably going to ease this. I haven’t heard any conversations from these website creation companies about this, but I do think it’s inevitable.

Neville Hobson: It is, I agree. One hundred percent, it’s inevitable. And like I said earlier, this is not sudden. So your point about keeping your website with links and all that stuff — for some years, that’s still going to be the case for many people, but it’s going to decline.

I think it’s interesting what you said about some stuff not lending itself well. I totally agree. And in fact, one of the tips for communicators that I came up with from all the stuff I’ve been reading is to go through your key external pages in your organization — the About page, FAQ, leadership bios, product or service descriptions — and ask, “If an AI agent were summarizing this, what would it say?”

And you talked about the examples you made: content buried in carousels, PDFs, JavaScript-heavy pages, or dense narrative prose is harder for agents to parse accurately. So restructuring even a handful of high-priority pages into clear, direct, quick Q&A formats is a practical first step. It could matter a great deal.

And the other thing, which I picked up from Digiday, but also from some others, is worth noting for everyone, I think. The technical landscape is still forming. Nobody has the full picture yet. But as The Economist example tells us, they’re not waiting, and neither should we as communicators. You start with what you can control: your own content, your direct audience relationships, your messaging clarity. And as Digiday concludes, the organizations that experiment now, even tentatively, will be better positioned than those waiting for a definitive playbook that may never arrive.

And I think if I were doing this — and again, it’s easy to say, “Well, if I were doing this, I’d do X.” It sounds easy, doesn’t it? — but the idea of literally kind of duplicating your website for this purpose is quite a bit of work. But it could stand you in good stead where your competitors haven’t done that and they’ve still got these beautifully designed, laid out, immaculate photos and stuff that’s animated, designed for human readers, without the version designed for the algorithms.

And I think that, to me, would be a key thing to truly investigate thoroughly in your organization: What am I committing to if I say, “Okay, our gorgeous website wins awards for its design, and here are all the metrics showing all the leads it delivers and all that kind of stuff. What do we need to do to get on the radar of the AI agents version?”

So you build a version. What’s entailed with that? One thing I didn’t see in Digiday’s piece, and I haven’t found it anywhere else yet, is what exactly is involved in creating a duplicate of your website designed around this? Where, for instance, you’re treating everything as structured data, not just copywritten text, which doesn’t translate well to an agent-intermediated world, to quote Digiday.

So it talks about, you know, starting to think about your key messages as discrete, unambiguous statements that can be lifted and represented accurately without context. That doesn’t mean dumbing things down. And this bit I did like: It means being deliberate about what an agent will extract if it only gets 30 seconds with your content.

Now, there’s a good pointer that the way of thinking about how you present yourself online now isn’t about beautifully designed websites and the nuanced meaning of words and phrases that are emotive about your products or service. It’s about data. And you need to appeal to — and this is a human emotive statement, which isn’t really applicable — but you’ve got to appeal to the AI agent. What will appeal to an AI agent? It doesn’t care, using the word “appeal.” It’s what will make it index you and include you when someone types, speaks, or whatever it is that produces search results, where the engine or tool you’re using goes out and pulls in all the information. You want to be sure you’re included in that. So it’s a bit of work to do.

Shel Holtz: Yeah. It’s not a one-to-one copy and paste from your existing website. In fact, we know that AI likes Q&A, for example. If your website’s not already in a Q&A format and the text can lend itself to that, you’re going to want to repurpose it that way.

We know that it likes what’s in the top 30 percent of the content, so you’re going to want to front-load the most important information. And frankly, if you’re not doing this, then your company is not going to show up in those AI summaries, or at the very least, it’s not going to show up represented the way that you want it to.

I do have two other issues to point out. One of these is the increased use of agents. You mentioned an agent, but it’s an agent that’s working on behalf of the model. Now I’m talking about the agent that’s working on behalf of me. In which case, that LLMS.txt file is going to become the routing layer to your official content.

And this is a big deal for regulated industries, because people have to find the canonical information. They have to find the approved information, not necessarily the one that was created for AI. So if you’re doing this, you do need to make sure that what you are creating for AI is official. It is going to present the information that the lawyers have approved, that is going to pass muster with the regulators.

The other thing is that if I can rewrite the text of a web page for the AI version of this that I’m creating, I can put something completely different on that page. And this becomes an ethical issue that we need to watch out for. People might try to game this. I mean, remember the early days of SEO and all of the gaming of SEO that we saw with tags that had nothing to do with what was on the page. They just knew that this was a popular search term, and it would drive traffic there.

We can see the same kind of gaming with this LLMS.txt routing to the AI-compliant pages, with information on it that’s there to get you into the search results, there to get you into the AI summary, not necessarily to give you the same information that you would have found on a web page. So this can be problematic, and I think it’s something that the industry’s going to be watching for.

Neville Hobson: I agree. So you compile your list of “these are the things we need to pay attention to,” but again, in the back of your mind, you’ve got this reality: The technical landscape is still forming. No one has the full picture yet. That would include what are the requirements of this, that, or the other. So assume it’s going to be like we currently have, although I would argue it’s not going to be like we currently have. It’s going to evolve, but that’s going to take time. So you need to do that precisely.

I would say pay attention to experiments like this. And again, this is a media company doing this, so they are publishers of content that might mention your brand. That’s why it’s important to you that you show up in these kinds of search terms as well. So a lot to pay attention to here, Shel, I think, really.

Shel Holtz: Yeah, and remember, The Economist isn’t doing this with their articles yet because that’s still behind the paywall. This is all that stuff in front of the paywall. I mean, eventually it’ll get there to some degree, although I don’t know how you do this behind the paywall when it’s just showing up on, you know, Google Search or…

Neville Hobson: No, again, the technical stuff is not worked out yet. They’ll find ways here.

The other thing to mention — I’m glad you mentioned The Economist. I meant to mention this, but the final point in Digiday’s piece, I thought, was really interesting. It’s a reminder, I suppose, that for all the experiments, The Economist sees AI as infrastructure rather than authorship: something to speed up research, workflows, and product delivery, not a shortcut to churning out more copy.

So AI is not going to be writing their content, they say. It will do all this stuff at the back end, if you will, that would enable its journalists, its human journalists, to do the stuff that they do well. That’s admirable. And I think The Economist has a track record of that kind of behavior in its history. So it’s great.

I think it’ll be interesting to see what other major properties do, including the big ones in the U.S., the big ones here, and elsewhere in the world. And I bet you they are working on stuff like this, too. They have to be.

Shel Holtz: No doubt. And that will be a 30 for this episode of For Immediate Release.

The post FIR #516: Your New Shadow Website appeared first on FIR Podcast Network.