In this episode, Sarah O’Keefe and Alan Pringle explore how AI transforms content delivery from static documents into dynamic, consumer-driven experiences. However, the need for human-led governance is critical, and Sarah and Alan explore issues of accuracy, accountability, governance, and more. They challenge organizations to define AI success by its ability to deliver accurate, high-impact outcomes for the end user.

Sarah O’Keefe: The metrics that are being used to measure the success of AI are all wrong. We should be measuring the success of various AI efforts based on, “Are people getting what they need? Are they having a successful outcome with whatever it is that they’re trying to do?” The metric we actually seem to be using is, “What percentage of your workflow is using AI? How many people can we get rid of because we’re automating everything with AI?” It’s the wrong metric. The question is, how good are the outcomes?

Related links:

LinkedIn:

Transcript:

This is a machine-generated transcript with edits.

Introduction with ambient background music

Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations.

Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it.

Sarah O’Keefe: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change.

Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off.

End of introduction

Alan Pringle: Hey everybody, I’m Alan Pringle, and today I’m here with Sarah O’Keefe, and we want to do something I’ve kind of dreaded to be honest, to do a check-in on AI in the content space. I’m very ambivalent about this topic. There’s still even two, three years in, there’s still a lot of hype, but there’s also been some good things that have emerged.

We need to talk about it fairly realistically. So, Sarah, get ready. Let’s see if I can not curse during this. We’ll try. I’ll try my best not to be like that in this. Legitimately, there are some things that we need to talk about, and also about the challenges because I don’t think the content world is completely ready for a lot of what’s going on right now.

Sarah O’Keefe: You know that we have AI that can remove cursing from podcasts, so I feel like we’re good here.

AP: Well, also, it’s a challenge to me to behave in a PG-13 more family-friendly kind of way. So I’ll do my best. 

SO: I have no idea what you’re talking about.

AP: Yeah. So let’s start with the good and where things are right now with the positives. What is AI doing well right now? And let’s kind of get beyond the summarization. I think we can say objectively right now, in general, AI does a very good job of summarizing existing content. But I think it’s doing a lot more beyond that, and we should touch on those things instead.

SO: The first thing that I would say is that summarization, but specifically the use case of a chatbot or a large learning model, an LLM, so now we’re talking about Claude, Gemini, ChatGPT and all the rest of them, which has the ability to provide an end user with a way of accessing information, an information access point that is different than what we had previously.

In the olden days, you had a book, and you had to sort of flip it open and look at a table of contents or maybe an index and navigate to a page. Fine. Then along comes online content, and you can do full text search, or you can then go into an internet search, right? You type into the search bar, you get a bunch of results, you click, and you sort of, no, that’s not quite it. You modify your search string, you search again, and you sort of navigate your way to where you’re trying to go. With the interactivity of the, you know, ChatGPT class of tools. What happens is that I ask it a question and it gives me an answer. And then I say, that’s not quite what I wanted. And I can sort of zero in on exactly what I’m looking for and tell it, but actually make this easier. Or I don’t understand the words you’re using. Use simpler language. Give me more. Give me less. Give me a summary. Use this as a source. Do not use that as a source.

It’s a new way to access information. People love it. There is something psychologically helpful about a conversational search. Now, there’s obviously huge issues with this, particularly around people, you know, using chatbots as their therapists, which introduces all sorts of horrifying, horrifying ethical issues. 

AP: Personifying them as a person on the other end. Right.

SO: But in the big picture, used well, it allows you to get to the information you’re looking for and get at it in the way that you want. 

AP: There’s a control issue here. I don’t think the content consumer has ever had this level of control.

SO: Yeah, and as a content consumer, that speaks to me. That is helpful. We’re seeing increasing use of, I would say, guardrails. So, not just slam out the AI with a bunch of stuff, but rather we’ve put some guardrails around it, and there’s various kinds of technologies that you can employ there. And that has been very helpful. And then the third thing I would point to is when we talk about generative AI and generating content, there’s a lot you can do in that sort low fidelity bucket. And what I mean here is I need an image for a presentation, but the background is the wrong color, so I can just swap it out. Now, I can do that with Photoshop. Well, some people can do that with Photoshop.

AP: Well, I was about to say, don’t think you or I should be saying we can do the Photoshop because we kind of can’t.

SO: Right. Well, and that’s exactly it. So it’s lowered the bar, right? Because I can tell the AI to swap out the background, and it will. And it applies a mid-level Photoshop capability to this image. And now I have the image that I need with a dark background so that the white text shows up in my presentation, that kind of thing. 

AP: Right. Yeah.

SO: We can do low-stakes synthetic audio if this podcast, which for the record we are recording with actual human beings, but let’s say that Alan curses extensively and we need to swap it out well, we could pretty easily generate some synthetic audio that sounds like him and that PG of eyes the original wording into something that is You know cleaner it would be way funnier to just bleep it. So I don’t know why we would do this but…

AP: Correct. Well, and it may come to that. The bottom line is what you’re talking about here is things that have very low risk. This is more fun stuff, the thought of doing some of what we’re talking about and stuff that describes how to use a medical device, for example. Not sure I want to go there with that. But for something low stakes like some one-off presentation that you’re giving, maybe some humor is involved, I totally think that’s an acceptable use because there’s no risk there.

SO: That’s really the key point because let’s say you’re writing content for a new medical device. Now you probably have a version one of said medical device, and you’re doing a version two. So, okay, fine. We take the version one content and we sort of, you know, say add color because that’s what we added, you know, in version two, and update all this stuff automatically.

But it then becomes very important to actually read that, look at that information, look at all the images, make sure that everything is correct. And by the time you do that super carefully, you may have given back all the time that you saved on the back end when you basically made a copy and said generate the new version. There’s some, you have to be really careful with that, especially depending on what your stakes are in terms of regulating regulatory or compliance stuff. 

You can, of course, get away with using AI, as you said, for low-stakes stuff. Now, the big risk you run there, and we’re seeing this in my favorite example of low-stakes content, which is video games, the video game industry has seen huge amounts of pushback against AI-generated game content, because it’s not fun. It’s not creative. It feels flat. It’s not art, and it’s not fun to play. And so it just becomes a slog. Again, same thing. Did you use it for maybe some backgrounds here and there? Okay. Did you use it to drive the story that you’re trying to establish or set up? You know, the enemies that you’re hypothetically fighting, and then they all have a certain sameness, or they all, you know, you’re sort of stealthing your way around the map. And it turns out that the AI-generated things are really dumb in that once they turn their back, you can do literally anything and they won’t notice because it was poorly designed.

AP: Right, yeah. And that’s true even in the film entertainment industry. There’s been a tremendous amount of pushback for the very reason I read a review recently talking about a series of clips about history on, I believe it’s on YouTube, by a fairly well-known director I will not name.

SO: Mm-hmm.

AP: And some of the AI is frankly not done well. And one reviewer basically said that a lot of the people, when you look at the back of these AI-generated, like an AI-generated King George, the back of his head looks like a melted candle. This is not what we want here. If you’re so focused on that sort of thing, you’re not paying attention to the message. But again, this is low-stakes content.

We have started getting into kind of more the content creator point of view. We’ve talked about the consumer and how AI gives them much more control, flexibility in how they receive information. But let’s talk about what that means more for the people that have to create the information because it’s a huge shift on multiple levels and this idea of creating, especially in the product content world, these lovely design page-based PDFs and whatever else, and even webpages, hate to say it, those days are gone, or should be at this point.

SO: Yeah, again, you know, we step back to books, and you write the content, it goes through like a manuscript process of some sort, and then it gets poured into a book. It gets printed on paper, which is about the least flexible thing you can imagine, right? Because I, as the book publisher, get to decide what font is on the page and what size.

And if you don’t like that font, well, maybe you can get your hands on a large print edition. Maybe you can get your hands on a braille edition. Maybe. But the form factor of the content was determined by the publisher of the content, or technically, the printer. But, you know, that physical book production process. PDF, not that different in the sense that the content is bound into the PDF and it’s fixed. Now.

You get a little bit more control because you can zoom in. There’s some things you can do in PDF, but ultimately it’s more or less still a page factor determined by the author/publisher/gatekeeper.

So now we talk about the web and HTML. This is all pre-AI, right? HTML goes out there, and there’s actually a decent bit you can do in your browser. You can override the default font. You can override the default font size. You can say, I’m using dark mode or light mode or those kinds of things. 

AP: Light mode, exactly.

SO: If you have an e-book reader, you can override the default font or font size.

AP: I need that font size jacked up, please. Thank you. 

SO: We weren’t going to use that example. Right. Yeah. So you get a little bit more control, right? You have a little bit more control over the presentation. Now, let’s talk about what AI does to this, and particularly the large language models. Now, I, as the author, create a whole bunch of content, and I put it somewhere. And the content consumer says…

AP: I’ll use it.

SO: Tell me about this concept or tell me about this thing or give me information about whatever. And they get a response to that prompt, which is a paragraph or two of, you know, here’s what you need to know. And then they say, make it easier, make it simpler, write this at a fourth grade level, write this at an eighth grade level. I’m a PhD in microbiology. Give me more detail. Right. You can change the writing level. You can say make the font bigger, make the font smaller, give it to me in a PDF, show it to me in a spreadsheet.

AP: I’ve even seen someone create a podcast of this document and have two people talking about it, which was freaky, but you can do that.

SO: Right. So as the author and the content creator and the backend people, right, the content people, we’re accustomed to taking our content and packaging it in certain ways. Like, here’s a topic for you, or here’s a PDF, or here’s a book, or here’s a deliverable, right, a package of content. And although with structured authoring, when that came in, we let go of this idea that we, as the author, got to control the page presentation. That got automated into the system. So the person controlling the page presentation was the person who designed the publishing pipelines. But the publishing pipelines were designed on the backend by the authoring people. Now all of a sudden, we have no control over that end product. Just because I thought it should be a PDF or an HTML page, you can turn around and say, like you said, give it to me in a podcast, make me a video, show it to me in French, and the LLMs will do it.

AP: The publishing pipeline got moved over the fence basically to more of the content consumer side and they get to do what they want more or less. That’s where things are headed.

SO: So pre-AI, we talked about content as a service, right? We load up all the content in a database somewhere, and then you, as the end user of that content or another machine, can reach over and say, give me some content out of there. But it was still a pretty discreet, like, show me that topic or show me that string. And what is fundamentally different about AI and large language models processing that content is the degree to which you can mix and match and rework, reformat, translate, and transform that content to be presented to you, the end user, in the manner of your choice.

So as an author, I kind of hate this, right? Hey, you took my stuff and you mangled it and you presented it in Comic Sans, and how dare you? And that’s where we are. That authors get to create information, but they don’t get to control the manner and means of distribution or presentation or formatting or language of that information.

AP: On the flip side of that, and here I am going to look on the sunnier side of things, which never happens. This may be a pod person version of me. If you, as a content creator, are no longer on the hook for thinking about the publishing pipelines and all of that sort of thing, theoretically, that should free you up to create better content on the back end because you don’t have to think about all those things. Allegedly. I don’t know if it’s happening, but…

SO: It’s very hard as an author to let go of that end product, the target that you’re headed for. But fundamentally, there’s a bigger problem, which is that even if I write the world’s greatest explanation of how to do something, that world’s greatest explanation of how to do something is not being presented to the end user as the thing I wrote. It’s being presented after being run through the transformer, the LLM, the processing that the AI can do when they ask for it. So I could literally write how to do X. And the end user says, hey, tell me how to do X. They are not going to get that chunk of information that I wrote. They’re going to get something reprocessed.

Of course, now we ask the fundamental question, which is, is the reprocessed version going to be better or worse than what I wrote? And the answer is, it kind of depends on whether I am an above average writer with an above average understanding of what that end user wants, or whether I’m a below average writer with a below average understanding of what that end user wants.

AP: To me, it’s almost irrelevant as a content creator. My version is better because if the person receiving the information via the chatbot or whatever thinks that what it’s getting or what they are getting is what they want, that’s all that really matters. That the person on the receiving end of that information gets what they want and fine-tunes it to what they want. If they’re happy with it, then the content creator’s opinion about that is, I hate to say it, immaterial at this point.

SO: Yeah, I kind of hate this timeline because, you know, where does my voice, you know, where does my voice go? And the answer is it’s gone. But you’re right, of course, the purpose of again, what is the purpose of technical and product information that we work on? The purpose is to enable people to use a product successfully. So if shoving it through an AI results in an outcome where that person uses the product successfully, then we’re good.

AP: I don’t disagree.

SO: That’s the purpose of the kind of thing that we produce. I think, though, that looking at this, and this is where I see some of the big challenges going forward. First of all, we have to acknowledge that an enormous percentage of the technical content that’s out there is really bad. Like, terrible. Really, really bad, and might be improved by a little trip through a chatbot that’s gonna render it into actually grammatically correct English. That’s a thing.

AP: Harsh but fair.

SO: Yeah, I think you’re not the only one that’s going to have some bleeping issues in this podcast. But the problem that I see right now is that the metrics that are being used to measure the success of AI are all wrong. We should be measuring the success of various AI layers and chatbots and things based on are people getting what they need?

AP: Yeah. Yeah.

SO: Are they having a successful outcome to whatever it is that they’re trying to do? Is the search or is the process of that conversational, whatever they’re doing, does it get them to the endpoint of, okay, I understand what I need to do and I’m good and I walk away? The metric we actually seem to be using is what percentage of your workflow is using AI? How many people can we get rid of? Because “we’re automating everything with AI” is the wrong metric. The question is, how good are the outcomes?

AP: To me the idea of how much AI versus human effort, there’s a lack of, shall we say, human intelligence being applied here because merely applying AI to something is fundamentally not going to make something that is incorrect, bad, whatever. It’s not going to magically fix it. That’s a huge disconnect for me when you’re talking about measuring outcomes.

Whatever you dump into your large language model, if it is fundamentally bad, as in outdated and incorrect, right now, I am pretty sure merely applying AI to it is not going to fix those two pretty gaping holes. And there’s, I don’t know what it is, people hear AI and they think there’s some magic involved. No, the underpinnings have to be good for that magic to be useful, basically.

SO: And I think all of us have examples of asking the chatbot a question and getting answers that are just flat wrong. Or worse, they look plausible, like they’re in the form of a plausible answer, but then you read it and you read it carefully and you’re like, this doesn’t actually say anything. It’s just word salad. Which, since a chatbot effectively is the average of the database underlying it of content pretty much means that the underlying database of content doesn’t say anything useful on this topic. So I think the place that I kind of go with this is to the question of accountability. 

AP: Yes.

SO: Who is legally responsible for the outcomes? Now, pretty clearly, if I or an organization produces a user guide that covers a specific product and there is wrong information in that user guide, the organization is responsible. I mean, it’s your document, you’re responsible. Okay, if I, as an end user, query a public-facing LLM and get the wrong answer for something, and then I proceed to use that in my life, whose fault is that?

Who is at fault when, or, you we saw this with, when the first came out, people were following the map, right, the GPS map, and it would send them off a cliff or it would send them into a construction area and they would drive off the side. Okay, whose fault is that? And the answer was always, well, it’s your fault because look up from the map and don’t drive past the sign that says, not enter construction zone cliff ahead.

AP: Or one-way street. Right, yeah.

SO: But AI doesn’t come with, I mean, it comes with warning labels, right? But we don’t see them. We don’t process them. What we see is a conversation where we say, tell me more about that. And it tells you more about that. And it feels as though you’re talking to a human. And therefore, when you push on something and say, are you sure? And it says yes, because what’s the typical answer when somebody says, are you sure? It’s yes.

Is it actually sure? No, it’s not sentient. So if I query a public-facing LLM, it reprocesses a bunch of content and tells me how to do a thing that is in direct contradiction to what the official user documentation says, whose fault is that? I think it’s mine because I use the public-facing LLM. Now, what if the organization that makes the product puts up a chatbot and I query the organization’s chatbot? How do I do X? And especially if that chatbot is your frontline tech support, like you cannot get to a human. You have to go through the chatbot. I asked the chatbot a question, and it says, do it this way. And it happens to be wrong. Is the organization liable? I don’t know the answer, I think yes, but I’m not sure. And so fundamentally, yeah.

AP: The bottom line here, yeah, we’re talking about governance here. The bottom line is governance and there is, there has to be some human AI interaction here. There has to be these guardrails that you mentioned earlier and that’s where humans have to be involved.

SO: And the better the AI gets, if it’s accurate half the time, then my hackles are up. I know it’s gonna be wrong. It’s wrong all the time. If it’s accurate 80% of the time, I sort of trend it like psychologically, I just assume it’s accurate all the time. So the better they get, the worse the errors are because we don’t expect them.

AP: That’s also dangerous. Yeah, right. Yeah.

SO: I see occasionally, very, very occasionally, I had directions to go somewhere. And the directions were literally, put this address into Google Maps, but don’t do A, B, and C because it’s wrong. Like, the directions to get to this location are incorrect. Do not follow them. Because these days, our assumption is that the mapping apps just work.

AP: And that’s it’s wrong most of the time, but I think part of this governance angle is we have to realize that AI is going to be wrong. 

SO: Pretty much just do.

AP: And there are lots of reasons we won’t get into all the reasons that can be wrong. So what are you going to do when it is wrong? How are you going to make sure it’s not wrong? Again, there’s this whole process, this whole governance process that has to be in place. And again, I think this is where human intervention is going to be necessary because I don’t think AI at this point has any business correcting itself in these matters. That seems sort of suboptimal to me.

SO: Hmm. Yeah, I mean, hypothetically, you can tell it to check itself. And certainly there’s some people doing that type of work. I think for me, fundamentally, the takeaways are that, like any other tool, there’s some really useful productivity enhancements that we can and should be taking advantage of. To your point, there’s some really important governance work that needs to be done to ensure that your QA is appropriately scaled to the level of risk of your product. Medical device, very high. Silly gaming app, pretty low. Don’t really care. And we need to think about guardrails and what it means to inject the right kind of content and the various kinds of enablement tools that you can use to do that. 

And finally, this issue of AI as a content customer, I think is really, really tricky because it’s a new, from our point as content creators, it is a delivery mechanism, right? Just like a PDF or a piece of HTML or anything else like that. and it’s a delivery mechanism that allows the end user to control how they access the content, which means we have to do way more work around the guardrails of what that means when they query the content and shape it to their own requirements.

AP: Yeah, so things have progressed in the past two years, most definitely, especially in the content space. We’ve seen a lot of improvements. But there are still some big picture things we have to work out. And I think it’s gonna be interesting in the next year or two to see what happens. You briefly mentioned there are some companies who are setting up systems that can do a decent job of checking up on itself. That’s not where everything is right now, but I think the better these systems get, the better the guardrails that get in place, they can start to find out, this is wrong, I need to fix it, or I need to update this with the latest information, let me go get it. So that is starting to happen more and more. I think it will become more part of the LLM to chatbot process, but I don’t think we’re quite there yet. And I’m interested to see what happens next with that sort of scenario.

SO: It’s definitely gonna be interesting. That much I’m sure about.

AP: Yeah, I agree. So we managed to get through this without cursing. So that’s good. I think it turned out to be a more realistic conversation, and we kind of tuned out the hype because that’s what just makes me grit my teeth and sometimes yell at LinkedIn when I see certain promoted posts on LinkedIn that I think are full of you-know-what. So anyways, I think we’ll wrap it up there. Sarah, do you have any final points you would like to sign off with?

SO: I think at the end of the day, when you try and contextualize, like, what is this AI thing and what does it mean for us fundamentally, we can look at some of the other sort of big picture shifts that we’ve made. I’ve been known to pretty dismissively compare it to a spell checker, you know? You can use it and it’ll fix some stuff, but you better check because it doesn’t know the difference between affect and effect, although some of the grammar checkers now maybe they do.

So there’s that, but I think at the end of the day, if you are looking at content strategy, content operations and enterprise level, you really do have to say, okay, where does AI fit into my strategy and how can we employ it productively to do what we need to do inside this organization to produce, manage, deliver the content that we’re working on.

AP: And I think we’re going to wrap up on that very good point. Thank you very much.

SO: Thank you.

Conclusion with ambient background music

CC: Thank you for listening to Content Operations by Scriptorium. For more information, visit Scriptorium.com or check the show notes for relevant links.

The post Check in on AI: The true measure of success for AI initiatives appeared first on Scriptorium.

As AI adoption accelerates, accountability and transparency issues are accumulating quickly. What should organizations be looking for, and what tools keep AI transparent? In this episode, Sarah O’Keefe sits down with Nathan Gilmour, the Chief Technical Officer of Writemore AI, to discuss a new approach to AI and accountability.

Sarah O’Keefe: Okay. I’m not going to ask you why this is the only AI tool I’ve heard about that has this type of audit trail, because it seems like a fairly important thing to do.

Nathan Gilmour: It is very important because there are information security policies. AI is this brand-new, shiny, incredibly powerful tool. But in the grand scheme of things, these large language models, the OpenAIs, the Claudes, the Geminis, they’re largely black boxes. We want to bring clarity to these black boxes and make them transparent, because organizations do want to implement AI tools to offer efficiencies or optimizations within their organizations. However, information security policies may not allow it.

Related links:

LinkedIn:

Transcript:

Introduction with ambient background music

Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations.

Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it.

Sarah O’Keefe: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change.

Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off.

End of introduction

Sarah O’Keefe: Hey everyone. I’m Sarah O’Keefe. Welcome to another episode. I am here today with Nathan Gilmour, who’s the Chief Technical Officer of Writemore AI. Nathan, welcome.

Nathan Gilmour: Thanks, Sarah. Happy to be here.

SO: Welcome aboard. So tell us a little bit about what you’re doing over there. You’ve got a new company and a new product that’s, what, a year old?

NG: Give or take, yep.

SO: Yep. So what are you up to over there? Is it AI-related?

NG: It is actually AI-related, but not AI-related in the traditional sense. Right now, we’ve built a product or tool that helps technical authoring teams convert from traditional Word or PDF formats, which would make up the bulk of much of the technical documentation ecosystem and help convert it to structured authoring. Meaning that they can get all of the benefits of reuse, easier publishing, high compatibility with various content management systems. And can do it in minutes where traditional conversions could take hours. So it really helps authoring teams get their content out to the world at large in a much more efficient and regulated fashion.

SO: So I pick up a corpus of 10 or 20 or 50,000 pages of stuff, and you’re going to take that, and you’re going to shove it into a magic black box, and out comes, you said, structured content, DITA?

NG: Correct.

SO: Out comes DITA. Okay. What does this actually … Give us the … That’s the 30,000-foot view. So what’s the parachute level view?

NG: Perfect. Underneath the hood, it’s actually a very deterministic pipeline. Deterministic pipeline means that there is a lot more code supporting it. It’s not an AI inferring what it should do. There’s actual code that guides a conversion process first. So going from, let’s say, Word to DITA, there are tools within the DITA Open Toolkit that allow and facilitate that much more mechanically, rather than trusting an AI to do it. We know that AI does struggle with structure, especially as context windows expand. It becomes more and more inaccurate. So if we feed these models with far more mechanically created content, they become much more accurate. You’re not trusting them to do much more, more of the nuanced parts of the process. So there’s a big difference between determinism and probabilism. Where determinism is the mechanical conversion of something, probabilism is allowing the AI to infer a process. So that’s where we differ is our process is much more deterministic versus allowing the AI to do everything on its own.

SO: So is it fair to say that you combined the … And for deterministic, I’m going to say scripting. But is it fair to say that you combined the DITA OT scripting processing with additional AI around that to improve the results?

NG: Correct. It also expedites the results so that instead of having a human do much of the semantic understanding of the document, we allow the AI to do it in a far more focused task. Machines can read faster.

SO: Okay. And so for most of us, when we start talking about AI, most people think large language model and specifically ChatGPT, but that’s not what this is. This is not like a front-end go play with it as a consumer. This is a tool for authors.

NG: Correct. And even further to that, it’s a partner tool for authors. It allows them to continue authoring in a format that they’re familiar with. Well, let’s take Microsoft Word, for example. Sometimes the shift from Word to structured authoring could be considered an enormous upheaval. Allowing authors to continue authoring in a format that they’re good at and they’re familiar with, and then have a partner tool that allows them to expedite the conversion process to structured authoring so that they can maintain a single source of truth, makes things a little bit better, more manageable, and more reliable in the long run. So instead of having to effectively cause a riot with the authoring teams, we can empower them to continue doing what they’re good at.

SO: Okay. So we drop the Word file in and magically DITA comes out. What if it’s not quite right? What if our AI doesn’t get it exactly right? I mean, how do I know that it’s not producing something that looks good, but is actually wrong?

NG: Great question. And that’s where, prior to doing anything further, there is a review period for the human authors. So in the event that the AI does make a mistake, it is not only completely transparent, so the output, the payload, as we describe it, comes with a full audit report. So every determination that the AI makes is traced and tracked and explained. And then further to that, the humans are even able to take that payload out anyway, open it up in an XML editor. So at this point in time, the content is converted, it is ready to go into the CCMS.

Prior to doing that, it can go into a subject matter expert who is familiar with structured authoring to do a final validation of the content to make sure that it is accurate. The biggest differentiator, though, is the tool never creates content. The humans need to create content because they are the subject matter experts within their field. They create the first draft. The tool takes it, converts it, but doesn’t change anything. It only works with the material as it stands. And then once that is complete, it goes back into another human-centered review so that there are audit trails, it is traceable. And there is a final touchpoint by a human prior to the final migration into their content management system.

SO: So you’re saying that basically you can diff this. I mean, you can look at the before and the after and see where all the changes are coming in.

NG: Correct.

SO: Okay. I’m not going to ask you why this is the only AI tool I’ve heard about that has this type of audit trail, because it seems like a fairly important thing to do.

NG: It is very important because there are information security policies. AI is this brand-new, shiny, incredibly powerful tool. But in the grand scheme of things, these large language models, the OpenAIs, the Claudes, the Geminis, they’re largely black boxes. Where we want to come in is to bring clarity to these black boxes. Make them transparent, I guess you can say. Because organizations do want to implement AI tools to offer efficiencies or optimizations within their organizations. However, information security policies may not allow it.

One of the added benefits that we have baked into the tool from a backend perspective is its ability to be completely internet-unaware. Meaning if an organization has the capital and the infrastructure to host a model, this can be plugged directly into their existing AI infrastructure and use its brain. Which, realistically, is what the language model is. It’s just a brain. So if companies have invested the time, invested the capital in order to build out this infrastructure, the Writemore tool can plug right into it and follow those preexisting information security policies. Without having to worry about something going out to the worldwide web.

SO: So the implication is that I can put this inside my very large organization with very strict information security policies and not be suddenly feeding my entire intellectual property corpus to a public-facing AI.

NG: That is entirely correct.

SO: We are not doing that. Okay. So I want to step back a tiny bit and think about what it means, because it seems like the thing that we’re circling around is accountability, right? What does it mean to use AI and still have accountability? And so, based on your experience of what you’ve been working on and building, what are some of the things that you’ve uncovered in terms of what we should be looking for generally as we’re building out AI-based things? What should we be looking for in terms of accountability of AI?

NG: The major accountability of AI is what could it look like if a business model changes? Let’s kind of focus on the large players in the market right now. There will always be risk with using these large language models that are publicly facing right now. A terms of service change could mean that all of the information that organizations use in order to leverage these tools could become part of a training data set later on down the road. It’s hard to determine what will happen in the future.

So the ability to use online and offline models encourages the development of very transparent tools. So even if the Writemore tool is using a cloud model, I still hold the model almost accountable to report its determinations. It’s not just making things up, so to speak. So there’s a lot that goes into it. There’s a lot that we don’t know about these tools, to be totally honest. We’re still trying to determine what it looks like in a broader picture, in a broader use case. Because the industry is evolving so quickly that, quite simply, we don’t know what’s coming up.

SO: Sounds to me as though you’re trying to put some guardrails around this so that if I’m operating this tool, then I can look at the before and after and say, “Don’t think so.” I mean, presumably it learns from that, and then my results get better down the road. Where do you think this is going? I mean, where do you see the biggest potential and where do you see the biggest risks or opportunities or … I’ll leave it to you as to whether this is a positive or a negative tilted question.

NG: There’s a lot of potential in order to incorporate into organizations that can’t use these tools. Like we had mentioned earlier, organizations are looking into this. Municipalities are looking into AI. But with the state of the more open models right now, it’s very hard to say. So I know I keep circling back around the ability to use smaller language models. They are not only much more efficient to operate, they’re also cheaper, quite simply, to operate. We know that the large language models require enormous computing power. But if provided focused tasks in order to either assist in the classification of topics or fulfill requests in order to pull files, in that regard, you can get away with using smaller levels of compute. And in today’s day and age of computing, the price relativistically is coming down in terms of density is going up. So it’s cheaper to run a model at higher capacities than it ever has been. And it’s only going to improve over time.

So empowering organizations to be able to incorporate these tools in order to streamline their own workflows is going to be very important to them. And on top of that, being able to abide or follow their information security policies only makes the ideas much more compelling. And on top of that, being able to encourage organizations to take full control of their documentation and not necessarily need it to go out of house allows organizations to keep internal costs down while still maintaining the security policies of making sure their content doesn’t leave their organization. There’s always going to be room for partner organizations to come in and help with their content strategy. But the conversion itself can be done in-house using their tools, using their content, using their teams. Which really helps keep costs down, they drive the priority lists, they can do everything that they need to do in order to maintain that control.

SO: Now, we’ve touched largely … Or we’ve talked largely about migration and format conversion. But there’s a second layer in here, right? Which we haven’t mentioned yet.

NG: There is. There’s the ability also during the conversion phase, it’s to have an AI model do light edits. So being able to feed it a style guide to abide by means the churn that we see with these technical teams isn’t as nearly impactful. You can have technical authors still write their content. But if a new person joins the team, they can still author the material just as normally. But then the tool can take over in order to ensure that it’s meeting the corporate style guide, the corporate language, so on, in order to expedite that process. So onboarding time for new team members shrinks as well. So like I said, it’s a very much it’s a partner tool in order to expedite the processing of content, authoring, conversion, migration, getting into a new CCMS and the real empowerment behind it.

SO: And the style guide conformance. So I think we’re assuming that the organization has a corporate style guide?

NG: Assuming, yes.

SO: Okay. Just checking.

NG: But then again, that’s…

SO: Asking for a friend.

NG: Of course.

SO: So if they don’t have one, where’s the corporate style guide come from?

NG: And that could be something that an organization can either generate internally or, as mentioned, work with an external vendor who specializes in these kinds of things in order to build a style guide so that all of their documentation follows the same voice and tone. The better the documentation, the better the trust of the content overall.

SO: So, can we use the AI to generate the corporate style guide?

NG: Probably. Yes. Short answer, yes. Longer answer, not without very close attention to it.

SO: And doesn’t that also assume that we have a corpus of correctly styled content so that we can extract the style guide rules?

NG: There’s a lot more. Yeah.

SO: So I mean, what I’m working my way around to is if we have content chaos, if you have an organization that doesn’t have a style guide, doesn’t have consistency, doesn’t have all these things, can you separate out what is the work that the humans have to do? And what is the work that the machine can do to get to structured, consistent, correct, voice and tone and all the rest of it? How do you get from the primordial soup of content goo to structured content in a CCMS?

NG: Great question. Typically, that starts with education. We work with the teams in order to identify these gaps first. We don’t just throw in a tool and say, “Good luck, hope for the best.” Because we see it time and time again, even in manual conversion processes where that simply doesn’t work. But in taking the time to work with teams, providing them with the skills and the knowledge in order to be successful serves a much longer term positive outcome than ever before. If we educate these teams on what any tool realistically needs, it means the accuracy of the tool goes up in the longer run. So you’re seeing multiple benefits on multiple sides.

So to your point about primordial soup, well, working with teams in order to identify these gaps, these issues, working to identify the standards that should go into the content prior to anything sets not only them up for success in the long run, but also for any tools that they want to implement down the road. It all starts with strong content going in because, as the adage goes, garbage in, garbage out. So if we can clean up the mess prior or work with the teams prior in order to establish these standards, then the quality of output only goes up.

SO: Yeah. And I mean, I think to me, that’s the big takeaway, right? We have these tools, we can do interesting things with them, but at the end of the day, we have to also augment them with the hard-won knowledge of the people. You mentioned subject matter experts, the domain experts, the people inside the organization that understand the regulatory framework or the corporate style guide, or all of those guardrails that make up what is it to create content in this organization that reflects this organization’s priorities and culture and all the rest of it.

NG: And taking the time to educate users is a far less invasive process than exporting bulk material, converting it manually, and handing it back. Because realistically, if we take that avenue or that road, we’re not educating the users, we’re not empowering them to be successful in the long run. All we’ll end up doing is all the hard work, but then in one, two, five years, we run into the same issue where we’re back to primordial soup of content, and it’s another mess. So if we start with the education and the empowerment and then work towards the implementation of tools, the longer-term success will be realized.

SO: Well, I think, I mean, that seems like a good place to leave it. So Nathan, thank you so much. This was interesting and I look forward to seeing where this goes and how it evolves over the next … Well, we’re operating in dog years now, so over the next month, I guess.

NG: So true. And thanks, Sarah, for having me on.

SO: Thanks, and we’ll see you soon.

Conclusion with ambient background music

CC: Thank you for listening to Content Operations by Scriptorium. For more information, visit Scriptorium.com or check the show notes for relevant links.

The post From black box to business tool: Making AI transparent and accountable appeared first on Scriptorium.

What happens when AI accelerates faster than your content can keep up? In this podcast, host Sarah O’Keefe and guest Michael Iantosca break down the current state of AI in content operations and what it means for documentation teams and executives. Together, they offer a forward-thinking look at how professionals can respond, adapt, and lead in a rapidly shifting landscape.

Sarah O’Keefe: How do you talk to executives about this? How do you find that balance between the promise of what these new tool sets can do for us, what automation looks like, and the risk that is introduced by the limitations of the technology? What’s the roadmap for somebody that’s trying to navigate this with people that are all-in on just getting the AI to do it?

Michael Iantosca: We need to remind them that the current state of AI still carries with it a probabilistic nature. And no matter what we do, unless we add more deterministic structural methods to guardrail it, things are going to be wrong even when all the input is right.

Related links:

LinkedIn:

Transcript:

Introduction with ambient background music

Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations.

Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it.

SO: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change.

Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off.

End of introduction

Sarah O’Keefe: Hey everyone, I’m Sarah O’Keefe. In this episode, I’m delighted to welcome Michael Iantosca to the show. Michael is the Senior Director of Content Platforms and Content Engineering at Avalara and one of the leading voices both in content ops and understanding the importance of AI and technical content. He’s had a longish career in this space. And so today we wanted to talk about AI and content. The context for this is that a few weeks ago, Michael published an article entitled The coming collapse of corporate knowledge: How AI is eating its own brain. So perhaps that gives us the theme for the show today. Michael, welcome.

Michael Iantosca: Thank you. I’m very honored to be here. Thank you for the opportunity.

SO: Well, I appreciate you being here. I would not describe you as anti-technology, and you’ve built out a lot of complex systems, and you’re doing a lot of interesting stuff with AI components. But you have this article out here that’s basically kind of apocalyptic. So what are your concerns with AI? What’s keeping you up at night here? 

MI: That’s a loaded question, but we’ll do the best we can to address it. I’m a consummate information developer as we used to call ourselves. I just started my 45th year in the profession. I’ve been fortunate that not only have I been mentored by some of the best people in the industry over the decades, but I was very fortunate to begin with AI in the early 90s when it was called expert systems. And then through the evolution of Watson and when generative AI really hit the mainstream, those of us that had been involved for a long time were… there was no surprise, we were already pretty well-versed. What we didn’t expect was the acceleration of it at this speed. So what I’d like to say sometimes is the thing that is changing fastest is the rate at which the rate of change is changing. And that couldn’t be more true than today. But content and knowledge is not a snapshot in time. It is a living, moving organism, ever evolving. And if you think about it, the large language models, they spent a fortune on chips and systems to train the big large language models on everything that they can possibly get their hands and fingers into. And they did that originally several years ago. And the assumption is that, especially for critical knowledge, is that that knowledge is static. Now they do rescan the sources on the web, but that’s no guarantee that those sources have been updated. Or, you know, the new content conflicts or confuses with the old content. How do they tell the difference between a version of IBM database 2 of its 13 different versions, and how you do different tasks across 13 versions? And can you imagine, especially when it comes to software where most of us, a lot of us work, the thousands and thousands of changes that are made to those programs in the user interfaces and the functionality?

MI: And unless that content is kept up-to-date and not only the large language models, reconsume it, but the local vector databases on which a lot of chatbots and agenda workflows are being based. You’re basically dealing with out-of-date and incorrect content, especially in many doc shops. The resources are just not there to keep up with that volume and frequency of change. So we have a pending crisis, in my opinion. And the last thing we need to do is reduce the people that are the knowledge workers to update, not only create new content, but deal with the technical debt, so that we don’t collapse on this, I think, is a house of cards.

SO: Yeah, it’s interesting. And as you’re saying that, I’m thinking we’ve talked a lot about content debt and issues of automation. But for the first time, it occurs to me to think about this more in terms of pollution. It’s an ongoing battle to scrub the air, to take out all the gunk that is being introduced that has to, on an ongoing basis, be taken out. Plus, you have this issue that information decays, right? In the sense that when, I published it a month ago, it was up to date. And then a year later, it’s wrong. Like it evolved, entropy happened, the product changed. And now there’s this delta or this gap between the way it was documented versus the way it is. And it seems like that’s what you’re talking about is that gap of not keeping up with the rate of change.

MI: Mm-hmm. Yeah. I think it’s even more immediate than that. I think you’re right. But now we need to remember that development cycles have greatly accelerated. Now, when you bring AI for product development into the equation, we’re now looking at 30 and 60-day product cycles. When I started, a product cycle was five years. Now it’s a month or two. And if we start using AI to draft new content, for example, just brand new content, forget about the old content or update the old content. And we’re using AI to do that in the prototyping phase. We’re moving that more left upfront. We know that between then and CodeFreeze that there’s going to be a numerous number of changes to the product, to the function, to the code, to the UI. It’s always been difficult to keep up with it in the first place, but now we’re compressed even more. So we now need to start looking at AI to how does it help us even do that piece of it, let alone what might be a corpus that is years and years old, that’s not ever had enough technical writers to keep up with all the changes. So now we have a dual problem, including new content with this compressed development cycle.

SO: So the, I mean, the AI hype says we essentially, we don’t need people anymore and the AI will do everything from coding the thing to documenting the thing to, I guess, buying the thing via some sort of an agentic workflow. But what, I mean, you’re deeper into this than nearly anybody else. What is the promise of the AI hype, and what’s the reality of what it can actually do?

MI: That’s just the question of the day. Because those of us that are working in shops that have engineering resources, I have direct engineers that work for me and an extended engineering team. So does the likes of Amazon, other serious, not serious, but sizable shops with resources. We have a lot of shops that are smaller. They don’t have access to either their own dedicated content systems engineers or even their IT team to help them. First, I want to recognize that we’ve got a continuum out there, and the commercial providers are not providing anything to help us at this point. So it’s either you build it yourself today, and that’s happening. People are developing individual tools using AI where the more advanced shops are looking at developing entire agentic workflows. 

And what we’re doing is looking at ways to accelerate that compressed timeframe for the content creators. And I want to use content creators a little more loosely because as we move the process left, and we involve our engineers, our programmers in the early, earlier in the phase, like they used to be, by the way, they used to write big specifications in my day. Boy, I want to go into a Gregorian chant. “Oh, in my day!” you know, but, but they don’t do that anymore. And basically the, the role of the content professional today is that of an investigative journalist. And you know what we do, right? We, we scrape and we claw. We test, we use, we interview, we use all of the capabilities of learning, of association, assimilation, synthesis, and of course, communication. And turns out that writing’s only 15% roughly of what the typical writer does in an information developer or technical documentation professional role, which is why we have a lot of different roles, by the way, that if we’re gonna replace or accelerate with people with AI, have to handle all those capabilities of all those roles. So, so where we are today is some of the more leading-edge shops are going ahead, and we’re looking at ways to ingest knowledge, new knowledge, and use that new knowledge with AI to draft new or updated content. But there are limitations to that. So, I want to be very clear. I am super bullish on AI. I think I use it every single day. I’m using it to help me write my novel. I’m using it to learn about astrophotography. I use it for so much. When the tasks are critical, when they’re regulatory, when they’re legal-related, when there’s liability involved, that’s the kind of content that we cannot afford to be wrong. We have to be right. We have to be 100% in many cases.

Whereas with other kinds of applications, we can very well be wrong. I always say AI and large language models are great on general knowledge that’s been around for years and evolves very slowly. But things that move quickly and change very quickly, in my business, it’s tax rates. There’s thousands and thousands of jurisdictions. Every tax rate is different and they change them. So you have to be 100% accurate or you’re going to pay a heck of a penalty financially if you’re wrong. So we are moving left. We are pulling knowledge from updated sources, things like videos that we could record and extract and capture, Figma designs, code even, to a limited degree that there’s assets in there that can be caught, and other collateral, and we’re able to build out and initial drafts. It’s pretty simple. Several companies are doing this right now, including my own team. And then the question comes, how good could it be initially? What can we do to improve that, make it as good as it can be? And then what is the downstream process for ensuring validity and quality of that content? What are the rubrics that we’re going to use to govern that? And therein is where most of the leading edge or bleeding edge or even hemorrhaging edge is right now.

SO: Yeah, and I mean, this is not really a new problem, and it’s not a problem specific to AI either, but we’ve had numerous projects where the delta between what, let’s say, the product design docs and the engineering content and the code, the as-designed documentation and the actual reality of the product walking out the door. So the as-built product, there was the resources, all that source material that you’re talking about, right, that we claw and scrape at. And I would like to also give a shout-out to the role of the anonymous source for the investigative journalists, because I feel like there’s some important stuff in there. But you go in there, you get all this as-designed stuff, right? Here’s the spec, here’s the code, here are the code comments, whatever. Or here’s the CAD for this hardware piece that we’re walking out the door. But the thing that actually comes down the factory assembly line or through the software compiler is different than what was documented in the designs because reality sets in and changes get made. And in many, many, many cases, the role of the technical writer was to ensure that the content that they were producing represented reality and not the artifacts that they started from. So there’s a gap. And there jobs to close that gap so that that document goes out and it’s accurate, right? And when we talk about these AI or automated or any sort of workflow, any sort of automation, any automation that does not take into account the gap between design and reality is going to run into problems. The level of problem depends on the accuracy of your source materials. Now, I wrote an article the other day and referred to the 100% accurate product specifications. I don’t know about you, I have seen one of those never in my life. 

MI: Hahaha that’s absolutely true. That’s really true. 

SO: The promise we have here is, AI is going to speed things up and it’s going to automate things and it’s going to make us more productive. And I think you and I both believe that that is true at a certain level. How do you talk to executives about this? How do you find that balance between the promise of what these new tool sets can do for us and what automation looks like and the risk that is introduced by the limitations of you know, of the technology itself? What does that conversation look like? What are the points that you try to make? What’s the roadmap for somebody that’s trying to, as you said, know, maybe in a smaller organization, navigate this with people that are, you know, all-in on “just get the AI to do it.”

MI: That’s a great question too, because we need to remind them that the current state of AI still carries with it a probabilistic nature. And no matter what we do, unless we add more deterministic structural methods to guardrail it, things are going to be wrong even when all the input is right. AI can still take a collection of collateral and get the order of the steps wrong. It can still include things or do too much. We’ve been trained to write as professional writers in a minimalistic capability. And we can control some of that through prompting. Some of that can be done with guardrails. But when you think about writing tech docs, some people might think, we document. we’re documenting APIs or documenting tasks and we, you know, we’ve always been heavily task-oriented, but you can extract all the correct steps and all the correct steps in the right order, but what doesn’t come along with it all too frequently and almost universally is the context behind it, the why part of it. I always say we can extract great things from code for APIs like endpoints or puts and, you know, gets and puts and things like that. That’s a great for creating reference documentation for programmers. 

But if you want to know, it doesn’t tell you the why, it doesn’t tell you the steps, the exact steps, the code doesn’t tell you that. Now maybe your Figma does. And if your Figma has been done really well, your design docs have been done really well and comprehensively. That can mitigate it tremendously. But what have we done in this business? We’ve actually let go more UX people than probably even writers, you know, which is, which is counterproductive. And then you’ve got things like the happy path and the alternate paths that could exist, for example, through the use of a product or the edge cases, right? The what-ifs that occur. You might be able to, and we should, we are able to do better with the happy path, but the happy path is not the only path. These are multifunction beasts that we built. When we built iPhone apps, we often didn’t need documentation because they did one thing and they did one thing really well. You take a piece of middleware, and it can be implemented a thousand different ways. And you’re going to you’re going to document it by example and maybe give some variance, you’re not going to pull that from Figma design. You’re not going to pull that from code. There’s too much of it there that the human fact-baking capability can look at it and say, this is important, this is less important, this is essential, this is non-essential, to actually deliver useful information to the end user. And we need to be able to show what we can produce, continue to iterate and try to make it better and better, because someday we may actually get pretty darn close with support articles and completed support case payloads, we were able to develop an AI workflow that very often was 70% to 100% accurate and ready for publish. 

But when you talk about user guides and complex applications, it’s another story because somebody builds a feature for a product and that feature boils down into not a single article, but into an entire collection of articles that are typed into the kind of breakdown that we do for disclosure, such as concepts, tasks, references, Q&A. So AI has got to be able to do something much more complex, which is to look at content and classify it and apply structure to separate those concerns. Because we know that when we deliver content in the electronic world, we’re no longer delivering PDF. Well, of us are hopefully not delivering PDF books made up of long chapters that intersperse all of these different content types because of the type of consumption, certainly not for AI and AI bots. Then when we, so we need to document, maybe the bottom line here is we need to show what we can do. We need to show where the risks are. We need to document the risks, and then we need the owners, the business decision makers, to see those risks, understand those risks, and sign off on those risks. And if they sign off on the risks, then me, as a technology developer and an information developer, I can sleep at night because I was clear on what it can do today. And that is not a statement that says it’s not going to be able to do that tomorrow. It’s only a today statement so that we can set expectations. And that’s the bottom line. How do we set expectations when there’s an easy button that Staples put in our face, and that’s the mentality of what AI is. It’s press a button and it’s automatic.

SO: Yeah, and I did want to briefly touch on, you know, the knowledge base articles are really, really interesting problem because in many cases you have knowledge base articles that are essentially bug fixes or edge cases when I, you know, hold my finger just so and push the button over here, you know, it blue screens.

MI: Mm-hmm.

SO: And that article can be very context-specific in the sense of you’re only going to see it if you have these five things installed on your system. And/or it can be temporal or time-limited in the sense that, while we fixed the bug, it’s no longer an issue. Okay. Well, so you have this knowledge-based article and you feed it into your LLM as an information source going forward, but we fixed the bug. So how do we pull it back out again?

MI: I love that question. 

SO: I don’t!

MI: I love it. No, I’ve been, actually working for a couple of years on this very particular problem. The first problem we have, Sarah, is we’ve been so resource constrained that when doc shops built an operations model, the last thing they invested in is the operations and the operations automation. So when I’m in a conference and I have a big room of 300 professional technical doc folks. I love asking the simple question, how do you track your content? And inevitably, I get, yeah, well, we do it on Excel spreadsheets. To actually have a digital system of record, I get a few hands. And then I ask the question, well, does that digital system of record that you have for every piece of documentation you’ve ever published, does that span just the product doc or does that actually span more than product doc like your developer, your partner, your learning, your support, all these different things. Cause the customer doesn’t look at us as those different functions. They look at us as one company, one product. And inevitably, I’m lucky if I get one hand in the audience that says, yeah, we actually are doing that. So the first thing they don’t have is they don’t have a contemporary system of record that is digital that we can say, we know and can automate notifications as to when a piece of documentation should either be re-reviewed and revalidated or retired and taken out.

The other problem we have is that all of these AI implementations and companies, almost universally, not completely, but most of them, were based on building these vector databases. And what they did, was often to the completely ignoring the doc team, was just go out to the different sources they had available, Confluence, SharePoint. If you had a CCMS, they’d ask you for access to your CCMS or your content delivery platform, and they suck it in. They may date-stamp it, which is okay, but pretty rudimentary. And they may even have methods for rereading those sources every once in a while, but they’re not, unless they’re rebuilding the entire vector database, and then what did they do when they ingested the content? They shredded it up into a million different pieces, right? Because the context windows for large language models have limitations for token numbers and things like that. Maybe they’re bigger today, but they’re still limited. So how would they even replace a fragment of what used to be whole topics and whole collection of topics? And this is why we wrote the paper and did the implementation and share with the world what we call the document object model knowledge graph because we needed a way outside of the vector database to say go look over here and you can retrieve the original entire topic or collection of topics or related topics in their entirety to deliver to the user. And again, we’re still unless we update that content and it’s don’t treat it like a frozen snapshot in time, we’ll still have those content debt problems. But it’s becoming a bigger, bigger, a much bigger problem now. It wasn’t as big a problem when we put out chatbots. And the chatbots we’ve been building, what, for three, you know, two, three, four years now. And, you know, everybody celebrated, they popped the corks, you know, we can deflect X amount percentage of support cases. They can self-service. And I always talk about the precision paradox that once you reach a certain ceiling, it gets really hard to increment and get above that 70%, 80%, 85%, 90% window. And as you get closer and better, the tolerance for being wrong goes down like a rock. And you now have a real big problem.

So how do we do these guardrails to be more deterministic, to mitigate the probabilistic risk that we have and reality that we have? The problem is that people are still looking for fast and quick, not right. When I say right, I mean the building out of things like ontologies and leveraging our taxonomies that we labored over with all of that metadata that never even gets into the vector database because they strip it all away in addition to shredding it up. So if we don’t start building those things like knowledge graphs and retaining all of that knowledge, it’s even… now we’re compounding the problem. Now we have debt, and we have no way to fix the debt. And now when we get into the new world of agentic workflows, which is the true bleeding edge right now, when you have sequences of both agentic and agentive, and the difference between those two, by the way, is agentic is autonomous. There’s no human doing that task. It’s just doing it. And then agentive, which is a human in the loop, which is helping there. When you’ve got a mix of agentive and agentic processes in a business workflow, now you’ve got to worry about what happens if I get something wrong early in the chain of sequence in that workflow. And this doesn’t apply to just documentation, by the way. We’ll be seeing companies taking very complex workflows in finance and in marketing and in business planning and reporting and mapping out this is the workflow our humans do. And there’s hundreds, if not more steps and many roles involved in those workflows. And as we map those out and say, where can we inject AI, not as just individual tools, like just separately using a large language model or separately using a single agent, but stringing them together to automate a complex business workflow with dependencies upstream and downstream. How are we going to survive and make this work? And I think that’s why you saw the MIT study had come out where they said, you know, roughly only 5% or so of AI projects are succeeding. And I think that’s because we did the easy stuff first. We did the chatbots and they could be lossy in terms of accuracy. But when you now, when you get to these agenda workflows that we’re building, literally coding as we speak, now you’re facing a whole different experience and ballgame where precision and currency really matters.

SO: Yeah, and I think I mean, we’ve really only scraped the surface of this. Both of the articles that you’ve mentioned, the one that I started with and the one that you mentioned in this context, we’ll make sure we get those into the show notes. I believe they are on your is it Medium? On your website. So we’ll get those links in there. Any final parting words in the last? I don’t know. Fifteen seconds or so.

MI: No, that’s good. I want to give, I want to tell you the good news and the bad news for tech doc professionals. What I’m seeing in the industry hurts me. I think there’s a lot of excuse right now, not just in the tech doc space, but in all jobs where we’re seeing AI being used as an excuse to make business decisions, to scale back. It may take some time until the impact of some poor business decisions that are being made will reflect themselves, but there’s going to be reality that hits. And the question is, is how do we navigate the interim? I’m confident that we will. I’m confident that those of us that are building the AI, I feel like I’m evil and a savior at the same time. I’m evil because I’m building automation that can speed up and make people much more productive, meaning you need less people potentially. At the same time, I feel like we’re in a position when we do it, rather than an engineer that doesn’t even know the documentation space, we’re getting to redefine our space ourselves and not leave it to the whims of people that don’t understand the incredible intricacy and dependencies of creating what we know as high-quality content. So we’re in this tumult right now, I think we’re going to come out of it. I can’t tell you what that window looks like. There will be challenges through doing that, but I would rather see this community define their own, redefine their own future in this transformation that is unavoidable. It’s not going away. It’s going to accelerate and get more serious. But if we don’t define ourselves, others will. And I think that’s the message I want our community to take away. So when we go to conferences and we show what we’re doing and we’re open and we’re sharing all the stuff that we’re doing, that’s not, hi, look at us. That’s you come back to the next conference and the next webinar and show us what you took from us and made better and helped shape and mold that transformative industry that we know as knowledge and content. And I’m excited because I want to celebrate every single advance that I see as we share. And I think it’s incumbent upon us to share and be vocal. And I think when I write my articles, they’re aimed at not only our own community, they’re aimed at the executives and technologists themselves to educate them, so that if we don’t do it, who will? And it does fall on all of us to do that.

SO: I think I’m going to leave it there with a call for the executives to pay attention to what you are saying, and some of the rest of this community, many of the rest of this community are saying. So, Michael, thank you very much for taking the time. I look forward to seeing you at the next conference and seeing what more you’ve come up with. And we will see you soon.

MI: Thank you very much.

SO: Thank you.

Conclusion with ambient background music

CC: Thank you for listening to Content Operations by Scriptorium. For more information, visit Scriptorium.com or check the show notes for relevant links.

The post Futureproof your content ops for the coming knowledge collapse appeared first on Scriptorium.

Your organization’s content debt costs more than you think. In this podcast, host Sarah O’Keefe and guest Dipo Ajose-Coker unpack the five stages of content debt from denial to action. Sarah and Dipo share how to navigate each stage to position your content—and your AI—for accuracy, scalability, and global growth.

The blame stage: “It’s the tools. It’s the process. It’s the people.” Technical writers hear, “We’re going to put you into this department, and we’ll get this person to manage you with this new agile process,” or, “We’ll make you do things this way.” The finger-pointing begins. Tech teams blame the authors. Authors blame the CMS. Leadership questions the ROI of the entire content operations team. This is often where organizations say, “We’ve got to start making a change.” They’re either going to double down and continue building content debt, or they start looking for a scalable solution.

— Dipo Ajose-Coker

Related links:

LinkedIn:

Transcript:

Introduction with ambient background music

Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations.

Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it.

SO: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change.

Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off.

End of introduction

Sarah O’Keefe: Hey, everyone. I’m Sarah O’Keefe and I’m here today with Dipo Ajose-Coker. He is a Solutions Architect and Strategy at RWS and based in France. His strategy work is focused on content technology. Hey, Dipo.

Dipo Ajose-Coker: Hey there, Sarah. Thanks for having me on.

SO: Yeah, how are you doing?

DA-C: Hanging in there. It’s a sunny, cold day, but the wind’s blowing.

SO: So in this episode, we wanted to talk about moving forward with your content and how you can make improvements to it and address some of the gaps that you have in terms of development and delivery and all the rest of it. And Dipo’s come up with a way of looking at this that is a framework that I think is actually extremely helpful. So Dipo, tell us about how you look at content debt.

DA-C: Okay, thanks. First of all, I think before I go into my little thing that I put up, what is content debt? I think it’d be great to talk about that. It’s kind of like technical debt. It refers to that future work that you keep storing up because you’ve been taking shortcuts to try and deliver on time. You’ve let quality slip. You’ve had consultants come in and out every three months, and they’ve just been putting… I mean writing consultants.

SO: These consultants.

DA-C: And they’ve been basically doing stuff in a rush to try and get your product out on time. And over time, those sort of little errors, those sort of shortcuts will build up and you end up with missing metadata or inconsistent styles. The content is okay for now, but as you go forward, you find you’re building up a big debt of all these little fixes. And these little fixes will eventually add up and then end up as a big debt to pay.

SO: And I saw an interesting post just a couple of days ago where somebody said that tech debt or content debt, you could think of it as having principle and interest and the interest accumulates over time. So the less work you do to pay down your content debt, the bigger and bigger and bigger it gets, right? It just keeps snowballing and eventually you find yourself with an enormous problem. So as you were looking at this idea of content debt, you came up with a framework for looking at this that is at once shiny and new and also very familiar. So what was it?

DA-C: Yeah, really familiar. I think everyone’s heard of the five stages of grief, and I thought, “Well, how about applying that to content debt?” And so I came up with the five stages of content debt. So let’s go into it.

I’m not going to keep referring to the grief part of it. You can all look it up, but the first stage is denial. “Our content is fine. We just need a better search engine. We can actually put it into this shiny new content delivery platform and it’s got this type of search,” and so on and so forth. Basically what you’re doing is you’re ignoring the growing mess. You’re duplicating content. You’ve got outdated docs. You’re building silos, and then you’re ignoring that these silos are actually getting even further and further apart. No one wants to admit that the CMS or whatever system, bespoke system that you’ve put into place, is just a patchwork of workarounds.

This quietly builds your content debt until, actually the longer denial lasts, the more expensive that cleanup is. As we said in that first bit, you want to pay off the capital of your debt as quickly as possible. Anyone with a mortgage knows that. You come into a little bit of money, pay off as much capital as you can so that you stop accruing that debt, the interest on the debt.

SO: And that is where when we talk about AI-based workflows, I feel like that is firmly situated in denial. Basically, “Yeah, we’ve got some issues, but the AI will fix it. The AI will make it all better.” Now, we painfully know that that’s probably not true, so we move ourselves out of denial. And then what?

DA-C: There we go into anger.

SO: Of course.

DA-C: “Why can’t we find anything? Why does every update take two weeks?” And that was a question we used to get regularly where I used to work at a global medical device manufacturer. We had to change one short sentence because a spec change and it took weeks to do that. Authors are wasting time looking for reusable content if they don’t have an efficient CCMS. Your review cycles drag through because all you’re doing is giving the entire 600-page PDF to the reviewer without highlighting what’s in there. Your translation costs balloon and your project managers or leadership gets angry because, “Well, we only changed one word. Can’t you just use Google Translate? It should only cost like five cents.” Compliance teams then start raising flags. And if you’re in a regulated industry, you don’t want the compliance teams on your back, and especially you don’t want to start having defects out in the field. So eventually, productivity drops, your teams feel like they’re stuck. And the cracks are now starting to show across other departments and you’re putting a bad name on your doc team.

SO: Yeah. And a lot of this, what you’ve got here, is the anger that’s focused inward to a certain extent. It’s the authors that are angry at everybody. I’ve also seen this play out as management saying, “Where are our docs? We have this team, we’re spending all this money, and updates take six months.” Or people submit update requests, tickets, something, the content doesn’t get into the docs, the docs don’t get updated. There’s a six-month lag. Now the SOP, the standard operating procedure, is out of sync with what people are actually doing on the factory floor, which it turns out, again, if you’re in medical devices, is extremely bad and will lead to your factory getting shut down, which is not what you want generally.

DA-C: Yeah, it’s not a good position to be in.

SO: And then there’s anger.

DA-C: Yeah.

SO: “Why aren’t they doing their job?” And yet you’ve got this group that’s doing the best that they can within their constraints, which are, as you said, in a lot of cases, very inefficient workflows, the wrong tool sets, not a lot of support, etc. Okay, so everybody’s mad. And then what?

DA-C: Everyone’s mad, and eventually, actually this is a closed little loop because all you then do is say, “Okay, well, we’re going to take a shortcut,” and you’ve just added to your content debt. So this stage is actually one of the most dangerous of the parts of it because all you end up trying to do without actually solving the problem is just add to the debt. “Let’s take a shortcut here, let’s do this.”

The next stage is now the blame stage. “It’s the tools. It’s the process. It’s the people.” These here and then you get calls of technical writers or, “Well, we’re going to put you into this department and we’ll get this person to rule you with this new agile process,” or, “We’ll get you to be doing it in this way.” The finger-pointing begins. Tech teams will blame the authors. Authors will blame the CMS. Leadership questions the ROI of the entire content operations team. This is often where organizations see that we’ve got to start making a change. They’re either going to double down and continue building that content debt or they start looking for a scalable solution.

SO: Right. And this is the point at which people look at it and say, “Why can’t we just use AI to fix all of this?”

DA-C: Yep, and we all know what happens when you point AI at garbage in. We’ve got the saying, and this saying has been true from the beginning of computing, garbage in, garbage out, GIGO.

SO: Time.

DA-C: Yeah. I changed that to computing.

SO: Yeah. It’s really interesting though because the blame that goes around, I’ve talked to a lot of executives who, and we’re right back to anger too, it is sort of like, “We’ve never had to invest in this before. Why are you telling us that this organization, this group, this tech writers, content ops,” whatever you want to call it, “that they are going to need enterprise tools just like everybody else?” And they are just halfway astounded and halfway offended that these worker bees that were running around doing their thing…

DA-C: Glorified secretaries.

SO: Yeah, that whole thing, like, “How dare they?” And it can be helpful, sometimes it is and sometimes it isn’t, to say, “Well, you’ve invested in tools for your developers. You wouldn’t dream of writing software without source control, I assume,” although let’s not go down the rabbit hole of vibe coding.

DA-C: Let’s not go down that one.

SO: And the fact that there are already people with the job title of vibe coding remediation specialist.

DA-C: Nice.

SO: Yeah. So that’s going to be a growth industry.

DA-C: That’s what, if you can get it.

SO: But this blame thing is we are saying, “This is an asset. You need to invest in it. You need to manage it. You need to depreciate it just like anything else. And if you don’t invest properly, you’re going to have some big problems.” And to your-

DA-C: A lot of that-

SO: Yeah, they don’t want to do it. They’re horrified.

DA-C: Yeah. A lot of that comes to looking at docs departments as cost centers. They’re costing us money. We’re paying all these people to produce this stuff that people don’t read. The users don’t want to. But if you look at it properly, deeply, the documentation department can be classed as a revenue generator. What are your sales teams pointing prospects at? They’re pointing at docs. Where are they getting the information about how things work? They’re pointing at the docs. What are you using? Especially if you’re having people looking through trying to find a solution?

I know I do this. I go and look at the user manuals. And first thing that I want to see in there that is properly written, if I see something that does not describe the gadget or whatever I’m trying to buy properly, then I’m like, “Well, if you’ve taken shortcuts there, you’ve probably done the same with the actual thing that I’m going to buy.” So I’m going to walk away.

Reducing costs for online centers. If your customers can find the information very quickly that describes the exact problem that they’re trying to solve, then you’ve got fewer calls to your online help center. And then while escalating onto the next person, because the level, I don’t know how this goes, level three, two, one, let’s say the level three is the lowest level, if that person can not find the information that is true, clear, one source of truth, then they’re going to escalate it onto that person who you’re paying a lot more, is at that level two, that person can’t find it, moved on. So it’s basically costing you a lot of money not to have good documentation. It’s a revenue generator.

SO: So my experience has been that the blame phase is perhaps the longest of all the phases.

DA-C: Yeah.

SO: And some organizations just get stuck there forever and they blame different people every year. I’ve also, I’m sure you’ve seen this as well, we were talking about reorganizing. “Well, okay, the tech writers are all in one group. Let’s burst them out and put them all on the product team.”

DA-C: Yes.

SO: “So you go on product team A and you go on product team B and you go on product team C.” And I talk to people about this and they say, “This is terrible and I don’t want to do it.” I’m like, “It’s fine, just wait two years.”

DA-C: Yeah.

SO: Because it won’t work, and then they’ll put them all back together. Ultimately, I’m not sure it matters whether they’re together or apart because we fall into this sort of weird intermediate thing. What matters is that somebody somewhere understands the value, to your point, and isn’t making the investment. I don’t care if you do that in a single group or in a cross-functional matrix, blah, blah, but here we are. All right. So eventually, hopefully, we exit blame.

DA-C: And then we move into acceptance.

SO: Do we?

DA-C: “Okay, we need a better way to manage that.” And this is like when people start contacting you, Sarah, it’s like, “I’ve heard there’s a better way to manage this. Somebody’s talked to me about there’s something called the component content management system or the structured content,” and all of this.

So teams start to acknowledge, one, that they’ve got debt and that debt is growing. Then they start auditing that content and then really seeing that, “Oh, well, yes, things are really going bad. We’ve got 15 versions of this same document living in different spaces in different countries. The translations always cost us a bomb.” So leadership then starts budgeting for a transformation.

This is where they then start doing their research to find structured content, competent reuse, they enter the conversation. If they look at their software departments, software departments reuse stuff. You’ve got libraries of objects. Variables is the simplest form of that reuse. And they’ve been using this for years. And so, “Well, why aren’t we doing this? Oh, there’s DITA, there’s metadata. We can govern our content better. We can collaborate using this tool.” So there is a better way to do this. And then we know what to do.

SO: I feel like a lot of times the people that reach out to us are in stage four, they’ve reached acceptance, but their management is still back in anger and bargaining and denial and all the rest of that.

DA-C: They’re still blaming and trying to find a reason.

SO: Yeah, blaming and all of it, just, “How dare you?” All right, so we acknowledge that we have a problem, which I think is actually the first step in a different step process, but okay.

DA-C: Yeah.

SO: And then what?

DA-C: And then there’s action. Let’s start fixing this before it gets totally out of control, before it gets worse. Then they start investing in structured content authoring platforms like Tridion Docs, I work for RWS, I’ve got to mention it. They start speaking with experts, doing that research, listening to their documentation team leaders, speaking with content strategists to define what the content model is, first of all, and then where can we optimize efficiency by having a reuse strategy? A reuse without a strategy is just asking for trouble. You’re basically going to end up duplicating content.

And then you’ve got to govern how that is used. What rules have you got in place and what ways have you got to implement those rules? The old job of having an editor used to work in the good old days where you’d print something off and somebody would sign it off and so on and so forth. Now, we’re having to deliver content really quickly and we’re using a lot of technology to do that. And so, well, you need to use technology to govern how that content is being created.

Then your content becomes an asset. It’s no longer a liability. This is where that transformation happens, and then you start paying down your content debt. You’re able to scale the content that you’re creating a lot faster without raising the number of the headcount, without having to hire more people. And if you want to then really expand, let’s say, because you’ve got this really great operation now and you’re able to create that content that takes hours and not weeks, then you’re able to expand your market. You’re able to say, “Okay, well, now we’re going to tackle the Brazilian market. Now, we can move into China because they’ve got different regulations.”

Again, I speak a lot on the regulatory side of things. That’s where I passed most of my time as a technical writer. Having different content for different regulatory regimes and so on is just such a headache where you don’t have something that is helping you with that structure, applying structure to that content, applying rules to that content, making sure that your workflows are carried out in the way that you set it out six months ago and people have changed and so on and they’re not doing their own thing again. If your organization is stuck at stages one to three, as I just mentioned it, it’s basically time to move.

SO: Yeah, I think it’s interesting thinking about this in the larger context of when we talk about writers, the act of writing, right?

DA-C: Yes.

SO: Culturally, that word or that process is really loaded with this idea of a single human in an attic somewhere writing the great American or French or British novel, writing a great piece of literature or creating a piece of art on their own, by themselves, in solitude. And of course, we know that technical writing-

DA-C: Starting at A and going all the way to Z.

SO: And we know that technical writing is not that at all, but it does really feel as though when we describe what it means to be a writer or a content creator in a structured content environment, it is just the 180 degree opposite of what it means to be a writer. It’s not the same thing. You are a creator of these little components. They all get put together. We need consistent voice and tone. You have to kind of subordinate your own voice and your own style to the corporate style and to the regulatory and to all the rest of it. And so it’s just this sort of… I think we maybe sometimes underestimate the level of cultural push and pull that there is between what it is to be a writer and what it is to be a technical writer.

DA-C: Yes.

SO: Or a technical communicator or content creator, whatever you want to call that role. Okay, so we’ve talked about a lot of this and then we’ve not talked a lot about AI, but a big chunk of this is that when you move into an environment where you are using AI for end users to access your content, so they go through a chatbot to get to the content or they’re consulting ChatGPT or something like that, and asking, “Tell me about X.” All of the things that you’re describing in terms of content debt play into the AI not performing, the content not getting in there, not being delivered. So what does it look like? What are some of the specifics of good source content, of paying down the debt and moving into this environment where the content is getting better? What does that mean? What do I actually have to do? We’ve talked about tools.

DA-C: Yeah. So first, you’ve got to understand how AI accesses content and how large language models get trained. AI interprets patterns as meaning. If your content deviates from pattern predictability, then you’re going to get what we call hallucinations. And so asking the ChatGPT without having it plugged as an enterprise AI thing where you’ve really trained it on your own content, you get all sorts of hallucinations. Basically, they’ve taken two PDFs that have similar information, but two different conclusions. And so you’re looking for a conclusion in document A, but ChatGPT has given you the one in B. And it’s mixed and matched those because it does not know how one bit of information relates to the other.

So good source content needs to be accurate. Your facts are correct. They reflect the current state of the product or subject. It needs to be kept up to date. You need to have single copies of it, that’s what we talk about, a single source of truth. You can not have two sources of truth. It’s either black or it’s white. There are no gray zones with AI, it will hallucinate. You’ve got to have that consistency in style and tone.

How do you get that? Well, you’ve got the brand and the way we speak. In French, you would say, “Do you vouvoie or do you tutoie?” Do use the formal voice, formal tone, or do you speak like you’re speaking with your friends? How do you enforce some of that? Well, you can use controlled terminology. These are special terms that you’ve defined, a special voice. But the gold part of it is having that structured formatting and presentation. There’s always a logical structure and sequence to the way that you present that information. Your heading, subheading, steps, lists, are always displayed in the same way. You’ve defined an information architecture to then give that pattern. And the way AI then understands or creates relationships with those patterns is from the metadata that you’re adding onto it.

And so good source content is accurate, up to date, consistent in style and tone, uses control terminology, has structure in formatting. Forget the presentation because that you put on the end of things, in that what it looks like, how pretty it is. But the presentation in terms of I always start with a short description and then I follow up with the required tools. And then I describe any prerequisites, and that is the way every one of my writers are contributing towards this central repository of knowledge, this single repository of knowledge.

And you can do that as well if you’ve got a great CCMS by using templates, building templates into that CCMS so that it guides the author. And the author no longer has to think about, “Oh, how is this going to look? Should I be coloring my tables green, red, blue? Should they be this wide?” They’re basically filling in a template form. And some of the standards that we’ve developed like DITA allow you to do this, allow you to have a particular pattern for creating that information and the ability to put it into a template which is managed by your CCMS.

SO: Yeah, and that’s the roadmap, right? We talk about how as a human, if I’m looking at content and I notice that it’s formatted differently, like, “Oh, they bolded this word here but not there,” and I start thinking, “Well, was that meaningful?”

DA-C: Yeah.

SO: And at some point, I decide, “No, it was just sloppy and somebody screwed up and didn’t bold the thing.” But AI will infer meaning from pattern deviations.

DA-C: Yeah.

SO: And so the more consistent the information is in all the levels that you’ve described, the more likely it is that it will process it correctly and give you the right outcome. Okay, so that seems like maybe the place that we need to wrap this up and say, folks, you have content debt. Dipo is giving you a handy roadmap for how to understand your content debt and understand the process of coming to terms with your content debt, and then figuring out how and where to move forward. So any closing thoughts on that before we say good luck to everybody?

DA-C: Basically before, or, I mean, most enterprises today have already jumped on the AI bandwagon. They’re already trying to put it in, but at the same time, start taking a look at your content to ensure that it is structured and has semantic meaning to it. Because the day that you then start training your large language model on that, if you’ve not built those relationships into it, it’s like teaching a kid bad habits. They’re going to just continue doing it. It’s basically train your AI right the first time by having content that is structured and semantic, and you’ll find your AI outcomes are a lot more successful.

SO: So I’m hearing that AI is basically a toddler? Okay. Well, I think we’ll leave it there. Dipo, thanks, it’s great to see you as always.

DA-C: Thanks for having me.

SO: Everybody, thank you for joining us, and we’ll see you on the next one.

Conclusion with ambient background music

CC: Thank you for listening to Content Operations by Scriptorium. For more information, visit Scriptorium.com or check the show notes for relevant links.

The post The five stages of content debt appeared first on Scriptorium.

How can global brands use AI in localization without losing accuracy, cultural nuance, and brand integrity? In this podcast, host Bill Swallow and guest Steve Maule explore the opportunities, risks, and evolving roles that AI brings to the localization process.

The most common workflow shift in translation is to start with AI output, then have a human being review some or all of that output. It’s rare that enterprise-level companies want a fully human translation. However, one of the concerns that a lot of enterprises have about using AI is security and confidentiality. We have some customers where it’s written in our contract that we must not use AI as part of the translation process. Now, that could be for specific content types only, but they don’t want to risk personal data being leaked. In general, though, the default service now for what I’d call regular common translation is post editing or human review of AI content. The biggest change is that’s really become the norm.

—Steve Maule, VP of Global Sales at Acclaro

Related links:

LinkedIn:

Transcript:

Introduction with ambient background music

Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations.

Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it.

SO: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change.

Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off.

End of introduction

Bill Swallow: Hi, I’m Bill Swallow, and today I have with me Steve Maule from Acclaro. In this episode, we’ll talk about the benefits and pitfalls of AI in localization. Welcome, Steve.

Steve Maule: Thanks, Bill. Pleasure to be here. Thanks for inviting me.

BS: Absolutely. Can you tell us a little bit about yourself and your work with Acclaro?

SM: Yeah, sure, sure. So I’m Steve Maule, currently the VP of Global Sales at Acclaro, and Acclaro is a fast-growing language services provider. So I’m based in Manchester in the UK, in the northwest of England, and I’ve been now in this industry, and I say this industry, the language industry, the localization industry for about 16 years, always in various sales, business development, or leadership roles.

So like I say, we’re a language services provider. And I suppose the way we try and talk about ourselves is we try and be that trusted partner to some of the world’s biggest brands and the world’s fastest growing global companies. And we see it Bill as our mission to harness that powerful combination of human expertise with cutting edge technology, whether it be AI or other technology. And the mission is to put brands in the heads, hearts, and hands of people everywhere.

BS: Actually, that’s a good lead in because my first question to you is going to be where do you see AI and localization, especially with a focus of being kind of the trusted partner for human-to-human communication?

SM: My first answer to that would be it’s no longer the future. AI is the now. And I think whatever role people play in our industry, whether you’re like Acclaro, you’re a language services provider, offering services to those global brands, whether you are a technology provider, whether you run localization, localized content in an enterprise, or even if you’re what I’d call an individual contributor, maybe you’re a linguist or a language professional. I think AI is already changed what you do and how you go about your business. And I think that’s only going to continue and to develop. So I actually think we’re going to stop talking at some stage relatively soon about AI. It’s just going to be all pervasive and all invasive.

BS: It’ll be the norm. Yeah.

SM: Absolutely. We don’t talk any more about the internet in many, many industries, and we won’t talk about AI. It’ll just become the norm. And localization, I don’t think is unique in that respect. But I do think that if you think about the genesis of large language models and where they came from, I think localization is probably one of the primary and one of the first use cases for generative AI and for LLMs.

BS: Right. The industry started out decades ago with machine translation, which was really born out of pattern matching, and it’s just grown over time.

SM: Absolutely. And I remember when I joined the industry, what did I say? So 2009, it would’ve been when I joined the industry. And I had friends asking me, what do you mean people pay you for translation and pay for language services? I’ve just got this new thing on my phone, it’s called Google Translate. Why are we paying any companies for translation? So you’re absolutely right, and I think obviously machine translation had been around for decades before I joined the industry. So yeah, I think that question has come into focus a lot more with every sort of, I was going to say, every year that passes, quite honestly, it’s every three months.

BS: If that.

SM: Exactly, yeah. Why do companies like Acclaro still exist? And I think there are probably a lot of people in the industry who actually, if you think about the boom in Gen I over the last two, two and a half years, there’s a lot of people who see it as a very real existential threat. But more and more what I’m seeing amongst our client base and our competitors and other actors in the industry, the tech companies, is that there’s a lot more people who are seeing it as an opportunity actually for the language industry and for the localization industry.

BS: So about those opportunities, what are you seeing there?

SM: I think one of the biggest things, it doesn’t matter what role you play, whether you’re an individual linguist or whether you’re a company like ours, I think there’s a shift in roles and the traditional, I suppose most of what I dealt with 16 years ago was a human being doing translation, another human being doing some editing. There were obviously computers and tools involved, but it was a very human-led process. I think we’re seeing now a lot of those roles changing. Translators are becoming language strategists; they’re becoming quality guardians. Project managers are becoming sort of almost like solutions architects or data owners. So I think that there’s a real change.

And personally, I don’t think, and I guess this is what this podcast is all about. I don’t see the roles of a few things going away, but I do see those roles changing and developing. And in some cases, I think it’s going to be for the better. And I think what we’re seeing is a lot of, because there’s all this kind of doubt and uncertainty and sort of threat, people are wanting to be shown the way, and people are wanting companies like our company and other companies like it to sort of lead the way in terms of how people who manage localized content can kind of implement AI.

BS: Yeah. We’re seeing something similar in the content space as well. I know there was a big fear, certainly a couple of years ago, or even last year, that, oh, AI is going to take all the writing jobs because everyone saw what ChatGPT could do until they really started peeling back the layers and go, well, this is great. It spit out a bunch of words, it sounds great, but it really doesn’t say anything. It just kind of glosses over a lot of information and kind of presents you with the summary. But what we’re seeing now is that a lot of people, at least on the writing side, yeah, they’re using AI as a tool to automate away a lot of the mechanical bits of the work so that the writers can focus on quality.

SM: We’re seeing exactly the same thing. I had a customer say to me she wants AI to do the dishes while she concentrates on writing the poetry. So it is the mundane stuff, the stuff that has to be done, but it’s not that exciting. It’s mundane, it’s repetitive. Those have always been the tasks that have been first in line to be automated, first in line to be removed, first in line, to be improved. And I think that’s what we’re seeing with AI. 

BS: So on the plus side, you have AI potentially doing the dishes for you, while you’re writing poetry or learning to play the piano, what are some of the pitfalls that you’re seeing with regard to AI and translation?

SM: I think there’s a few, and I think it depends on whereabouts AI is used, Bill, in the workflow. I think the very active translation itself is a very, very common use now of AI. But I think there’s some kind of a, I’m going to call them translation adjacent tasks as well, like we’ve mentioned with the entire workflow. So I think the answer would depend on that. But I think one of the biggest pitfalls of AI, and it was the same again, 2009 when I joined the industry and friends of mine had this new thing in their pocket called Google Translate. One of the pitfalls was, well, it’s not always right. It’s not always accurate.

And even though the technology has come on leaps and bounds since then, and you had neural NT before large language models, it still isn’t always accurate. And I think you mentioned it before, it does almost always sound smooth and fluid and almost like it sounds like it’s very polished, and it sounds like it should be, right? I’m thinking, “I’m in sales myself. So it could be a metaphor for a salesperson, couldn’t it? Not always, right? But always sounds confident. But I think there’s a danger where in any type of translation, sometimes accuracy doesn’t actually matter. I mean, if the type of content we’re talking about is, I don’t know, some frequently asked questions on how I can get my speaker to work as a customer, you’re going to be very patient if it’s not perfect English or if you speaking to the language, if it’s not perfect, as long as it gets you to get your speaker to work, you’re not really going to mind. But there’s other content where accuracy is absolutely crucial. In some industries could even be life or death.

But I go back to my first year or two in the industry, and we had a customer that made really good digital cameras, and they had a huge problem because their camera was water resistant, and one of their previous translators had translated it as waterproof. And of course, the customer takes it scuba diving or whatever they were doing with the digital camera, and the camera stops working because it wasn’t waterproof, it was just water resistant.

So sometimes what would be a very kind of seemingly innocuous choice of term, it wasn’t life or death, but obviously it was the difference between a thousand-dollar camera working or not. So I think accuracy is really critical. And even though it sounds confident, it’s not always accurate. And I think that’s one of the biggest pitfalls. Language is subjective, and some things are sort of black and white or wrong, but other things are a lot more nuanced. And what we see is, especially because a lot of the large language models are trained in English and with English data, they don’t necessarily always get the cultural or the sort of linguistic specific nuances of different markets.

We’ve seen some examples, it could be any markets, but specifically Arabic requires careful handling because of the way certain language comes across. Japanese, the politeness Japanese and what do they say, 50 words for snow. Some things aren’t sort of black or white in terms of whether they’re right or wrong. So it’s very, very gray areas in language. And again, however confident the output sounds, sometimes it’s not always culturally balanced or culturally sensitive.

BS: You don’t want it to imply anything or have anyone kind of just take away the wrong message because it was unclear or whatnot.

SM: Absolutely, absolutely. And especially when you’re thinking of branded content. I mean, some of the companies we work with and some of the companies, I’m sure that people listen to the podcast, they’d spend millions on protecting building, first of all, but also protecting their brand in different markets and the wrong choice of language, the wrong translation can put that at risk.

BS: Yeah. With branding, I assume that there’s a tone shift that you need to watch for. There’s certainly what you can and can’t say in certain contexts regarding the brand.

SM: Well, I think with AI, when you are using GenAI to translate, the other thing is it’s because I think you mentioned before, the technology it is a pattern-based technology. The content could be quite inherently repetitive. And again, whilst they’ll be confident, whilst they’ll be polished, it doesn’t always take into account the creativity or the emotion. And it’s less and less now we’re seeing AI sort of properly trained on a specific brand’s content. The models are more, they’re too big really to be trained just on a brand-specific content. So sometimes the messaging can appear quite generic or not really in step with the identity that a brand wants to portray. I think most of our clients would be in agreement when it comes to brand. It can’t be left to the machines alone.

BS: And I would think that any use of AI or even machine translation in something with regard to branding, where you want to own that messaging and really tailor that messaging, you really don’t want to have other influences coming in from the wild. So I would imagine that with an AI model that’s trained to work in that environment, you really don’t want it to know that there’s an outside internet, there’s an outside world that it can harvest information from because you might be getting language from your competitors or what have you.

SM: Yeah, absolutely. Absolutely. Yeah, you’re sort of getting it from too many sources where it kind of needs to be beyond brand really. I think there’s other things as well that we see. I mean, there’s still quite common cases of bias and stereotyping because like you say, it, taking content if you like, or data from all sorts of sources. And if there’s bias in there, there’s misgendered language, especially with some target languages. I mean, you’ve got, in English, it’s kind of fine, really, but in Spanish and French and German, you’ve got to choose a gender for every noun, every adjective, in order to be accurate.

BS: Otherwise, it’s wrong.

SM: Yeah, absolutely. Yeah, absolutely. And it compounds because the models are built on such scale, it compounds over time. So again, without that sort of active monitoring and without that human oversight, what might be a problem today will compound, and it’d be even worse tomorrow in the months ahead.

BS: How about the way in which the translation process works? Have you seen AI really shifting a lot of those workflows?

SM: So the short answer is yes. So by far, the most common workflow, if you’re looking at translation by far, the most common workflow with our customers now is to start with AI output. And to have a human being review some or all of that output. It is very, very rare. Now, when we are working with the enterprise-level companies, it’s very, very rare that they’d want, well, actually I might hold that thought, but it’s very rare that they’d want, for most content, they would want a fully human translation. Except one of the pitfalls that we have seen is, or one of the concerns if you like, that a lot of enterprises have about using AI is security and confidentiality.

And in fact, we have some customers where it’s written in our contract that we must not use AI as part of the translation process. Now, that could be for some specific content types only, and a lot of the time it’s a factor of, if you like, the attitude to risk or the attitude to confidentiality that that particular customer might have. But a lot of people are still very, very paranoid about that. They don’t want to be risking personal data being effectively leaked or being used to train and being cross pollinated, like your previous example. But in general, the sort of default service now for what I’d call regular common translation is post editing or human review of AI content. So that, that’s probably the biggest change is that’s now really become the norm.

BS: Okay. We talked a lot about the pitfalls here, so let’s talk about some benefits that you get at of using AI and localization.

SM: Well, I think the first thing is scale. I think it just allows you to do so much more because it almost, well, it doesn’t remove, but it significantly reduces those budget and time constraints that the traditional translation process used to have. Yeah, you can translate content really, really fast, very, very affordably, and it’s huge volumes that you just couldn’t consider if that technology wasn’t there.

So you could argue you’ve always been able to do that since machine translation was available. But I think large language models, they do bring more fluency. They do bring more sort of contextual understanding than those sort of pattern-based machine translation models. They can, even though we’ve talked about how some of the challenges around nuance and tone, they can improve style and tone. So we’ve seen a lot of benefits and a good opportunity really in sort of pairing the two technologies, neural machine translation, large language models, and again, you can’t get away when they’re guided by the human expertise.

They can offer a really good balance of scale, but also quality that you weren’t able to achieve before. And this is what I would say to people who are sort of worried about the existential threat of, oh my gosh, I’m a translator, so AI is taking my job. Absolutely, it’s probably changing your job. But we see AI translation not replace human translation, but replacing no translation. So that mountain of content, the majority of content actually that was never translated before because of time and budget constraints can now be translated to a certain level of quality. And so we see the overall volume of content localize, exploding, and ideally a similar level of human involvement or even more, in some cases, human involvement than before, but as a proportion of the overall, it’s a lot less, if that makes sense.

BS: Yeah. So what about multimedia? So audio and video, I know those have been traditionally a more difficult format to handle in localization, particularly when you may need to change the visuals along the way.

SM: If you ask any project manager in our company, the most expensive, the most time-consuming type projects traditionally to deliver, and you’re absolutely right, you make a mistake with terminology and you’re doing a professional voiceover and the studio’s booked and the actor’s booked and you want to change three or four words or three or four terms. Okay, that’s fine. Rebook the studio, rebook the actor. Yeah. I mean, it was traditionally, and I say traditionally, we’re talking only three or four years ago, one of the most expensive forms of content to translate.

So I think what we see is it’s been revolutionized by AI, video localization, audio localization, and this is a great example of actually where it’s replacing no translation. I mean, we had customers who just wouldn’t, we don’t want to dub that video. We don’t want to localize their audio, we just can’t afford it. We haven’t got the time. And now with synthesized voice synthesized videos, the quality is sort of very natural, very expressive, and you can produce training videos and product demos and all those kind of marketing assets in various markets that used to cost you lots and lots of money for 10 times less the cost, and probably more than 10 times less the speed.

BS: Nice. Yeah. I know that one of the things that we saw, particularly with using machine translation is that there was a pretty good check for accuracy built into a lot of those systems, but they weren’t quite a hundred percent. How does AI compare with that because it does understand language a bit more. So with regard to QA, how is that being leveraged?

SM: Well, they can understand. It’s not just about accuracy and grammatical correctness and spelling errors and that sort of thing has always been around, like you say, with machine translation. But the LLMs now, they can evaluate that sort of fluency terminology, use adherence to brand guidelines, style guidelines, and they can do that. So what we see is that whereas before LLMs came around and you had neural machine translation, pretty much most of the machine, unless it was very low value output, and unless it was very invisible or less visible content, let’s say if it was something that the clients cared about, they would want a human review of every single segment or every single sentence effectively. Whereas now, LLMs can help you sort of hone in and identify that percentage of the content that might need looking at by a human. And actually, I mean, there’s no real pattern, but if an LLM as a first pass can look at a large volume of content and say, actually 70% of that is absolutely fine, it matches the instructions that we’ve given it.

Not only is it accurate, but also it adheres to fluency and terminology and so on. Why don’t you human beings focus on this 30%? I mean, that’s a huge benefit to a lot of companies, saves a lot of time, saves a lot of costs, and just again, allows them to localize a lot more of that content than they were ever able to do before. So it’s great as a first pass before an extra layer if you like, a technology-dead layer before any human involvement and focusing the humans on the work that matters and the work that’s going to have the most impact.

BS: Nice. So if someone is looking to adopt AI within their localization efforts, what are the first steps for building AI into a strategy that you would recommend?

SM: Just call me. No, I’m kidding. I think it is any new process bill or any new technology, I think, and it sounds kind of common sense, but I think when deciding on any new strategy, it’s kind of be clear about why you’re doing it. You asked earlier on how AI is changing the localization industry. I think one huge thing I see, I speak to enterprise buyers of localization services every day. That’s my job. That’s what me and my team do. And one of the things that they tell me is that all of a sudden the C-suite know who they are.

All of a sudden, the guys with the money, the people with the money, they know they exist. And oh, we’ve got a localization department because as we said, GenAI, one of the earliest adopters, one of the earliest use cases for this was localization and was translation. So now there’s a lot of pressure from people who previously didn’t even know you existed or sort of maybe just saw you as a cost of doing business. Now they’re putting pressure on you to use AI. How are you using GenAI in your workflow? What can we as a business learn from it? Where can we save costs? Where can we increase volume? How can we use it as a revenue driver? Those sort of things. So that being said, that’s a big opportunity, but where we see it not go right or where we see it go more wrong more often than not is where people are doing it just because of that pressure and they think, oh, I have to do it because I’m getting asked to do it. I’m getting asked to experiment.

Again, it sounds really obvious, but they don’t really know what they’re looking for. Are they looking for time to be saved? Are they looking for costs to be removed? Are they looking to increase efficiencies with in their overall workflow? So I think it’s like anything, isn’t it? Unless you know how you’re going to measure success, you probably won’t be successful. So I think that’s the first tip I’d give people. Be clear about what it is you’re looking for AI in localization to achieve. And again, one of the pitfalls is we see lots of people wanting to experiment and it’s good, and you want to encourage that. I suppose as a chief exec or even with our clients, we’d love to see experimentation, but when you see lots of people doing lots of different things just because it looks cool and they just want to experiment, unless it’s joined up and unless it’s with a purpose, it doesn’t always work well.

So I think what we see when people do it well is they have that purpose. They have it documented actually, they have that sort of agreed, if you like, with they have that executive buy-in, this is why we’re doing it, and this is what we’re hoping to see, not just because it’s cool because it might save us X dollars or it might save us X amount of time. And I think what we see well is when people do that and then they kind of embrace those small iterative tests. One of our solutions architects was on a call with me with a customer, just advise them not to boil the ocean. And again, I know this isn’t specific to AI, but just let’s not do everything all at once. Lots of localization workflows. They have legacy technology, they have legacy connectors to other content repositories, and you can’t just rip it out without a lot of pain and start again.

So you’ve got to decide where you’re going to have that impact. Start small, very small tests, iterate frequently, get the feedback. That’s one of the key things. And then it just becomes any other implementation of technology or implementation of a workflow. One of the things we did at Acclaro is actually publish a checklist to help companies answer that exact same question, but when you read it, there’s not going to be much there about specific AI technologies and this type of LLM is better for this, and that type of LLM is better for that. It’s not prescriptive. It’s just designed as a guide to actually say, okay, well don’t get ahead of yourselves. Just follow a really sensible process, prove that it works, and then choose the next experiment.

BS: Yeah, get people thinking about it.

SM: Absolutely,

BS: We hear a lot from people that, oh, it came down from the C-suite that we have to incorporate AI into our workflows in 2025, in 2026. And yeah, I mean that’s all the directive is usually. Usually there’s no foresight coming down from above saying, this is what we’re envisioning you doing with AI. So it really does come down to the people who are managing these processes to take a step back and say, okay, here’s where things are working, here’s where we could make improvements. Here are some potential footholds that we can start building with AI and see where it goes. But yeah, I think for a lot of people, the answer of how do I use AI? I think it’s going to be different for every company out there. I mean, it might be similar, but I think it might be very different and very unique from company to company as to what they’re actually doing.

SM: That’s what we see. Yeah, that’s what we see. And again, some of those pitfalls we’ve talked about, some companies have a different approach to information security and confidentiality. Some companies are just risk averse. Some company’s content is, they should be more sensitive about it than other company’s content. Some company’s content, think finance, life sciences, medical devices, there’s real-world problems. Let’s say if it’s not accurate, whereas other company’s contents, yeah, okay, it might take you an extra 30 seconds to get that speaker to work or it might not. But I think, yeah, that’s no surprise. One of our customers said to me, AI is like tea. You need to infuse it. You can’t just dump it. You need to infuse it. You need to let it breathe. You need to let it kind of circulate. You got to decide the strength. You’ve got to decide where you get it from. You’ve got to decide what the human being making it has to do to make a great cup. And it’s just going to be different for every single person.

BS: True.

SM: We have five in our house and we have five different types of tea, whoever’s making that tea has to know what everyone’s preferences are. And I think it’s the same with AI. And it’s the same with a lot of technologies, isn’t it?

BS: It is. So when let’s say someone running a localization department, their CEO says, “We need to incorporate AI. Here’s your mandate. Go run, figure it out, implement it.” Do you have any advice around how to report, I guess the results, the findings, the progress back up?

SM: Yeah. My first advice would be, if I was in that situation, to say to that person, listen, we’ve been doing this for 10 years. We just never used to call it AI. We used to call it neural machine translation or machine translation. But my second bit of advice is you’ve actually got to do that because whilst the opportunity is there for localization managers to really drive and shape how AI is implemented, if they don’t do that, or if you pretend it’s something different than it is obvious, if you pretend it’s going away or if you pretend it’s a fad that people are going to forget about, what’ll happen is that somebody else will be asked to implement AI and you won’t be. And it’s quite interesting. We’re seeing a lot now of the persona, if you like, of the people that we’re working with in those enterprise localization teams is getting wider, it’s getting more multidisciplinary.

It’s very, very rare that you’d have any decent sized company, a localization manager making decisions about partners, vendors, technology by themselves. It would always be now with a keen eye from the technology team, the IT team, because everyone’s laser-focused on getting this right. So that’d be my second piece of advice. But I think if you define the results that you’re looking for and you document those and you’re able to capture those, again, it is not rocket science. It’s really just basic project management then. And then try and report on those regularly and quickly in a way that you’re able to iterate. An AI pilot shouldn’t be a six-month project with results at the end of six months. I mean, you should be able to know if you’ve chosen the right size of pilots, you should be able to know within days or weeks whether it’s likely to bring the benefits you thought it would do.

BS: Very true. So you see the return on using it or the lack of return on using it much quicker?

SM: Yeah, well absolutely. Yeah. Again, I think from my own personal experience, we’ve done a lot of helping and guiding clients with pilots, with experiments. It’s not all great results. And again, we haven’t manufactured anything to make it not great results so we stay in a job and people still use the human service. But we have seen really good results. I’m thinking of one, it’s quite a specific use case to do with translation memories, but the client was using GenAI to improve the fuzzy match, if you’re familiar with that term, build a translation memory match, the fuzzy match enhancer, and they found that it improved about 80% of the segments in I think five languages.

So again, if I look at that one, they didn’t pick every single language that they had. They only picked five, probably picked five where they could get some quick feedbacks of five more commonly spoken languages. And they were able to measure in their tool, the post editing time and the accuracy. And yeah, they found it improved 80%. I mean, 20% didn’t improve, so not 100% success, but they were able to provide real data to the powers that be to decide whether to extend it to their other language sets or their other content types.

BS: Nice. Well, I think we’re at a point where we can wrap up here. Any closing thoughts on AI and localization? Good, bad, ugly, just do it.

SM: I think the biggest thing for me is that AI is today. It’s not the future. It’s here. I’m in the UK, like I say, and multi-billion dollar announcement in investments, all specifically to do with AI from companies like NVIDIA, from Microsoft. And AI is the now. So I think you don’t have a choice whether to adopt it, whether to adapt to it being here. It’s just about how you choose to do it really. That’s become our role as a language service provider. As a sort of trusted partner of brands, our role has become to help guide and give our opinions. It’ll continue to change and we’ll have new use cases. And you ask me those same questions, I think Bill, in six months or 12 months, I might give you some different answers because we’ll have found new experiments and new use cases.

BS: And that’s fair. Well, Steve, thank you very much.

SM: Thank you, Bill. I enjoyed the conversation.

Conclusion with ambient background music

CC: Thank you for listening to Content Operations by Scriptorium. For more information, visit Scriptorium.com or check the show notes for relevant links.

Want to learn more about AI, localization, and the future of content? Download our book, Content Transformation.

The post Balancing automation, accuracy, and authenticity: AI in localization appeared first on Scriptorium.

AI, self-paced courses, and shifting demand for instructor-led classes—what’s next for the future of training content? In this podcast, Sarah O’Keefe and Kevin Siegel unpack the challenges, opportunities, and what it takes to adapt.

There’s probably a training company out there that’d be happy to teach me how to use WordPress. I didn’t have the time, I didn’t have the resources, nothing. So I just did it on my own. That’s one example of how you can use AI to replace some training. And when I don’t know how to do something these days, I go right to YouTube and look for a video to teach me how to do it. But given that, there are some industries where you can’t get away with that. Healthcare is an example—you’re not going to learn how to do brain surgery that someone could rely on with AI or through a YouTube video.

— Kevin Siegel

Related links:

LinkedIn:

Transcript:

Introduction with ambient background music

Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations.

Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it.

SO: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change.

Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off.

End of introduction

SO: Hi, everyone, I’m Sarah O’Keefe. I’m here today with Kevin Siegel. Hey, Kevin.

KS: Hey, Sarah. Great to be here. Thanks for having me.

SO: Yeah, it’s great to see you. Kevin and I, for those of you that don’t know, go way back and have some epic stories about a conference in India that we went to together where we had some adventures in shopping and haggling and bartering in the middle of downtown Bangalore, as I recall.

KS: I can only tell you that if you want to go shopping in Bangalore, take Sarah. She’s far better at negotiating than I am. I’m absolutely horrible at it.

SO: And my advice is to take Alyssa Fox, who was the one that was really doing all the bartering.

KS: Really good. Yes, yes.

SO: So anyway, we are here today to talk about challenges in instructor-led training, and this came out of a LinkedIn post that Kevin put up a little while ago, which will include in the show notes. So Kevin, tell us a little bit about yourself and IconLogic, your company and what you do over there.

KS: So IconLogic, we’ve always considered ourselves to be a three-headed dragon, three-headed beast, where we do computer training, software training, so vendor-specific. We do e-learning development, and I write books for a living as well. So if you go to Amazon, you’ll find me well-represented there. Actually, one of the original micro-publishers on this new platform called Amazon with my very first book posted there called, “All This PageMaker, the Essentials.” Yeah, did I date myself for that reference? Which led to a book on QuarkXPress, which led to Microsoft Office books. But my bread and butter books on Amazon even today are books on Adobe Captivate, Articulate Storyline, and TechSmith Camtasia. I still keep those books updated. So publishing, training, and development. And the post you’re talking about, which got a lot of feedback, I really loved it, was about training and specifically what I see as the demise of our training portion of our business. And it’s pretty terrifying. I thought it was just us, but I spoke with other organizations similar to mine in training, and we’re not talking about a small fall-off of training. 15, 20% could be manageable. You’re talking 90% training fall off, which led me to think originally, “Is it me?” Because I hadn’t talked to the other training companies. “Is it us? I mean, we’re dinosaurs at this point. Is it the consumer? Is it the industry?”

But then I talked to a bunch of companies that are similar to mine and they’re all showing the same thing, 90% down. And just as an example of how horrifying that is, some of our classes, we’d expect a decent-sized class, 10, a large class, 15 to 18. Those were the glory days. Now we’re twos and threes, if anyone signs up at all. And what I saw as the demise of training for both training companies and trainers, if you’re a training company and you’re hiring a trainer, one or two people in the room isn’t going to pay the bills. Got to keep the lights on with your overhead running 50%, 60%, you know this as a business person, but you’ve got to have five or six minimum to pay those bills and pay your trainer any kind of a rate.

SO: So we’re talking specifically about live instructor-led, in-person or online?

KS: Both, but we went more virtual long before the pandemic. So we’ve been teaching more virtual than on-site for 30 years. Well, not virtual 30 years, virtual wasn’t really viable until about 20 years ago. So we’ve been teaching virtual for 20 years. The pandemic made it all the more important. But you would think that training would improve with the pandemic, it actually got even worse and it never recovered. So the pandemic was the genesis of that spiral down. AI has hastened the demise. But this is instructor-led training in both forms, virtual and on-site. I think even worse for on-site.

SO: So let’s start with pandemic. You’re already doing virtual classes, along comes COVID and lockdowns and everything goes virtual. And you would think you’d be well-positioned for that, in that you’re good to go. What happened with training during the pandemic era when that first hit?

KS: When that pandemic first hit, people panicked and went home and just hugged their families. They weren’t getting trained on anything. So it wasn’t a question of, were we well-positioned to offer training? Nobody wanted training, period. And this was, I think if you pull all training companies, well, there are certain markets where you need training no matter what. Healthcare as an example, they need training. Security, needed training. But for the day-to-day operations of a business, people went home and they didn’t work for a long time. They were just like, “The world is ending.” And then, oh, the world didn’t end. So now they’ve got to go back to work, but they didn’t go back to work for a long time. Eventually people got back to work. Now, are you on-site back to work or are you at home? That’s a whole nother thing to think about.

But just from a training perspective, when panic sets in, when the economy goes bad, training is one of the first things, you get rid of it. Go teach yourself. And the teaching yourself part is what has led to the further demise of training, because you realize I can teach myself on YouTube. At least I think I can. And I think when you start teaching yourself on your own and you think you can, it becomes, the training was good enough. So if you said, “Let’s focus on the pandemic.” That’s what started it, the downward spiral. But we even saw the downward spiral before the pandemic, and it was the vendors that started to offer the training that we were offering themselves.

SO: So instead of a third-party, certainly a third-party, mostly independent organization offering training on a specific software application, the vendors said, “We’re going to offer official training.”

KS: Correct. And it started with some of these vendors rolling out their training at conferences. And I attended these conferences as a speaker. I won’t name the software, I won’t name the vendor, but I would just tell you I would go there and I would say, “Well, what’s this certificate thing you’re running there?” It’s a certificate of participation. But as I saw people walking around, they would say, “I’m now certified.” And I go, “You’re not certified after a three-hour program. You now have some knowledge.” They thought they were certified and experts, but they wouldn’t know they weren’t qualified until told to do a job. And then they would find out, “I’m not qualified to do this job.” But that certificate course, which was just a couple of hours by this particular vendor, morphed into a full day certificate. They were charging now a lot of money for it, which morphed into a multi-day thing, which now has destroyed any opportunity for training that we have. And that’s when I started noticing a downward spiral. Tracking finances, it would be your investments going down, down, down, down this thing. It’s like a plane, head and nose down.

SO: And we’ve seen something similar. I mean, back in the day, and I do actually… So for those of you listening at home that are not in this generation, PageMaker was the sort of grandparent of InDesign. I am also familiar with PageMaker and I think my first work in computer stuff was in that space. So now we’ve all dated ourselves. But back in the day we did a decent amount of in-person training. We had a training classroom in one of our offices at one point.

Now, we were never as focused on it as you are and were, but we did a decent business of public-facing, scheduled two-day, three-day, “Come to our office and we’ll train you on the things.” And then over time, that kind of dropped off and we got away from doing training because it was so difficult. And this is longer ago than you’re talking about. So the pattern that you’re describing where instructor-led in-person training, a classroom training with everybody in the same room kind of got disrupted a while back. We made a decent living doing that for a long time and there was-

KS: Made a great living doing that. Oh, my God. That was the thing.

SO: But we got away from it, because it got harder and harder to put the right people in the right classes and get people to travel and come to us. So then there’s online training, which we kind of got rid of training. You sort of pivoted to online/ virtual. And then ultimately, the pandemic has made it such, from my point of view, that the vast majority of what we do in this space is custom. We’re doing a big implementation project. We do some custom training that might be in-person, on-site, but much more often it is online, live online instructor-led, but custom. Because all of the companies that we’re dealing with, even if people did return to office, very much they’re fragmented, right? It’s two people here and five people there, and four people there and one in every state. And so, bringing them all together into a classroom is not just bring the instructor in, but bring everybody in and it costs a fortune. And that’s before we get into the question of, can they get across the borders and can they travel?

There’s visa issues, there’s admin issues, people have caregiving responsibilities, they can’t travel. There’s a whole bunch of stuff that goes into actually relocating from point A to point B to do a class at point B. So fine. Okay. So along comes the pandemic that really pushes on the virtualization, right? The virtual stuff. And then you’re saying the vendors get into it and they are clawing back some of this revenue for themselves. They’re basically saying, “We’re going to do official vendor-approved stuff, which then makes it very difficult as a third-party, because you have to walk that line, and I’ve been there, you have to walk that line between, we are delivering training on this product which belongs to somebody else, and we can be maybe a little more forthright about the issues in the product because it’s not our product. So we’re just going to say, “Hey, there’s an issue over here. It doesn’t really work. Do it this other way.” Not toeing the official party line. Okay, so we have all of that going on and all of those challenges already. And now along comes AI. So what does AI do to this environment that you’re describing?

KS: It further destroys it. I’ll give you an example. My blog, Typepad, we received an email September 1st, 2025, and we’re recording this September 4th, 2025, okay? So three days ago I got an email saying, “Hello, we’re shutting down. Sorry.” And I’m like, “What? Yeah, you’ve got 30 days to get your stuff out of here.” Basically being kicked out of your apartment or your house. So I’m like, “All right, well, go to AI and I asked AI, what is the top blog software?” They said, “WordPress.” Love it or hate it, okay. So I went to WordPress. I had no idea how to use WordPress. I had no staff available to help me. So I had to get my stuff out of Typepad and on and on it went. I went to AI, ChatGPT specifically, and I said, “Teach me how to use WordPress,” and specifically how to get my crap out of TypePad. I say crap, my stuff out of TypePad. In a matter of what? Two days I had everything transferred over.

So, didn’t need training, otherwise I would’ve had to go to training to learn how to do that and I didn’t have to. So that’s an example of there’s probably a training company out there that’d be happy to teach me how to use WordPress. I didn’t have the time, I didn’t have the resources, nothing. So I just did it on my own. That’s one example of how you can use AI to replace the training. There’s other examples of training that is not just good enough, it’s fine. It’s good. It’s good. It’s not lacking. When I don’t know how to do something these days, I go right to YouTube and look for a video to teach me how to do it. So given that, some industries where you can’t get away with that. Healthcare as an example, you’re not going to learn how to do brain surgery that you could rely on with AI or video through YouTube.

SO: We hope.

KS: We hope. “Hey, relax. I know this is your first time, Sarah, I’m your surgeon. I watched a video yesterday, I feel pretty good about it as I grab that saw.” I don’t believe you’re going to be comfortable with that. So listen, it’s bad enough. And you mentioned the vendor that is now offering training. So vendor pullback, they want that for a revenue source. This particular vendor is using it as a revenue tool, but there’s also vendors out there that are actively stopping you from offering training classes, and on it goes.

SO: Yeah, I do want to talk about that one a little bit. I know nothing about the specifics of your situation, but this is a losing battle. Because you were just talking about YouTube, I was doing some research for a very, very, very large company that makes farm equipment and I went looking for their content. And they had content on their website, it was like type in your product name or product number and it would give you the official user manual, which was of course ugly and terrible. But I discovered that if you typed in something like, “How do I fix the breaks on my X, Y, Z product?” It would take you to YouTube. And it would take you to this YouTube channel that had a lot of subscribers and was in fact not at all the official company YouTube channel.

KS: It was a dude who was working on it?

SO: It was a dude in Warsaw, North Carolina, which is not the same as Warsaw, Poland. It is a tiny, tiny, tiny little place, mostly known for me as being halfway between where I am and the beach. It’s where we stop to get gas and summer peaches and corn from the farm stand and fried chicken on our way to the beach, because that’s the thing we do. That’s where Warsaw is. It has a population of, I don’t know, 3,000 maybe.

KS: Okay, yeah.

SO: I have no idea. But there’s some guy who works for the dealership there who’s making these videos explaining how to do maintenance on these, in this case tractors, and he has got the audience. Not the official website, which by the way does not have a YouTube channel that I’m aware of, or at least that I could find now. This was five, 10 years ago. It has been a while. But so, there’s all this third-party content out there and there’s this ecosystem of content because it’s digital. You can’t really control that unless, we were talking about this earlier, unless you’re doing something like nuclear weapons, intelligence work, or maybe brain surgery. You can probably control those things. That’s about it. Clearly things are changing and not for the better. If your revenue is built on instructor-led, whether in-person or online, it sounds as though things are changing and not for the better in that space specifically, unless we’re training on brain surgery, which most of us are not. So what’s the path forward?

KS: I’m thinking about it, actually.

SO: I am not signing up for you to do my brain surgery.

KS: I need someone to practice on. Sarah, let me know if you’re available.

SO: Oh, I’m so sorry, you’re breaking up. I can’t hear you. Okay, so what does the path forward look like? I mean, what does it mean to be inside this disruption and where do you go from here?

KS: Okay, so every training company that I have contacts in, they’re all down significantly. The ones that are surviving have government contracts.

SO: Mm-hmm.

KS: And that is to develop training in all of its guises, that primarily they’re seeing a call for virtual reality training. That’s really, really hot right now. But not the virtual reality training that you can create with the Captivates and the Storylines of the world. That’s too lowbrow. They’re talking about immersive, almost gamification, where you build a world. So if that’s your expertise, you can create training in that. That’s what people want. It looks like augmented reality and virtual reality.

I can’t see it. Maybe I’m of a certain age that I’m like, “I’m not putting goggles on to take my training.” But that is pretty popular with other generations. So you can’t ignore it, I think, embrace it. So government contracts, if you can get that, you’ll be okay in the training business. Several of my colleagues have actually done that. So that’s a leg up. The other is to embrace asynchronous training and put your materials out there that live now forever. So I ignored for years these providers of asynchronous training where you put your content there and they sell it for you. I’ve got five classes on Udemy now, and each of them sells pretty well.

Matter of fact, my Captivate Udemy is one of their bestsellers. That does not translate into offsetting the revenue lost from your training gigs when you were bringing in six, seven, $800 a person for a training class. Our prices were between $695 and $895 per person to take a public class, but it certainly does bring in some revenue. So if you have the ability to create the asynchronous training, the video training, and make it really, really good training, really impactful, then that’s going to help you stay in the game as long as you can. I also think embracing AI versus getting under the covers and just, “I don’t want to see it,” is not the way to go.

I now use AI as a tool. I don’t think it replaces me, I think that I have more to offer in guiding the course than AI, but it gives me a nice, “Get me started here.” Maybe you’ve got a little writer’s block, maybe just getting started. It’s a beautiful day out, I can’t get started. Have AI start, you’ve started up. But if you’re going to go that route and you have AI make suggestions, you better fact check it. And just as an example, I was just curious, I asked ChatGPT to create an exam for Articulate Storyline. That is a tool I know really well, I’ve written exams for Storyline and Captivate and Camtasia. I said, “Write an exam. I want to see what you come up with.” And some of the questions were actually worded better than what I had done. They were very similar questions. And I go, “I kind of like the way you, AI, did that.” Which was kind of a bummer. But I would say a good 30% of what I read, while it was well-written, was completely wrong.

SO: Yes, confidently wrong.

KS: Yes, it was confidently wrong. Asking questions, “When you do this on storyline, what is the correct thing? What do you do?” And Storyline doesn’t do that thing. They were talking about Rise as an example. I’m like, “You’ve gone and combined Rise with Storyline.” So if you’re going to use AI, it’s the way you ask the question, your prompts. So get some training on engineering your prompts and fact-checking what you get from those prompts. But I use AI every day in my writing to make sure I don’t have grammar issues. So I’ll tell AI, “Check this for clarity and grammar.” So it’s my words, but it now is saying, “Well, there’s a couple typos, I fix that. And a couple of dangling modifiers, I fix that.” So it makes me feel like I’m writing better. But do keep in mind, if you put your stuff into ChatGPT, it’s now part of this mass of stuff that other people are going to get access to.

So you can’t copyright anything that you put in AI. I wrote a book about copyright and training materials and things to think about, because we have a lot of people finding an image of a nice puppy on Google and using it in their training, and that puppy was copyrighted. So anything you do on AI, any photos that get created, any artwork, anything, any writing can’t be copyrighted because only a human can get a copyright. So that’s something to think about. If you have something really, really good, you really didn’t create that, so you can’t copyright it.

You’re going to have to adapt. You’re going to have to adapt or you’re going to fail in the training industry, again, unless it’s very specific niche markets, or as you mentioned, custom training. If you don’t adapt, you’re going to fail. And that adaptation is going to be, embrace AI asynchronous training to put your training out there, available 24 hours a day, seven days a week when you can’t do it. And that’ll offset getting these onesies and twosies in your class.

SO: And it removes the time-bound, I have to set aside these two hours or these four hours of this day to be in the classroom, whether virtual or not if it’s live. I do think that this idea that we’re going to see a split between things that go higher and higher end that people are willing to pay nearly anything for versus the low-end where the price is going… There’s going to be downward pressure on the price for all the low-end stuff, because the barrier to entry to producing asynchronous training is pretty minimal and it gets lower every single day because there’s so many people out there that can potentially do that.

KS: Anybody can hang out a shingle and say that they’re an expert. So I mean, it’s the credentials of the trainer too, I think. Who is the person that’s teaching this? Is it what we call it, Chuck with a truck? Is it Chuck with a truck? Or is it someone who has actually done this? I wouldn’t want to get trained on handling my content by someone who hadn’t done it. I’d want you to handle that, right? So a content strategy. “I mean, who came up with that strategy? Oh, Bob. Has Bob ever done it? No, but he feels good about it. No, I want to get a Sarah who’s done it for years and years and years.”

SO: Yeah, I mean that’s an interesting point though, because at the end of the day, if you commoditize/ productized training, you’re going to have a product as the asynchronous training that’s a package, and you get what you get. When it’s live with an instructor, you’re going to get that instructor on that day in that context. They’re feeling good, they’re feeling bad. The classroom dynamics are good or bad or weird. Every experience is going to be different. Whereas with async, it’s always going to be the same. I mean, barring internet connectivity or something, as the learner, you’re going to get a consistent experience. Now, it’s not going to be the best possible experience, right? Because the best possible experience is you’re in a group with some other people in a room with an amazing instructor.

KS: That is the best.

SO: That is the best.

KS: There’s good too-

SO: It costs the earth.

KS: Yeah, there’s good too, the asynchronous training, because it’s always the same, it’s going to be consistent. How many times have you read a live class and the attendees, one of the attendees just spoiled the sauce? And you’re reminding me now, a colleague of mine, they were doing their certification as a certified technical trainer, CTT, and back in those days, you actually had to record yourself teaching.

SO: Oh, yes, there was a VHS tape of me and kids. That is video, pre-digital video.

KS: That is correct. VHS tape. And I had to do the same thing, but I remember for this one colleague of mine, and the students in this classroom, fake classroom, were other trainers that were also getting the recording done. And I remember she was being recorded and it was over her shoulder looking at the students, because she had to show the students. And one of these students, she made a comment that she knew was correct, and the student shook her head, “Nope, nope. That’s not right. Nope.” And the trainer is now, “What are you doing? Why are you shaking your head no and contradicting one of us? How about just nod?” And so, at some point God had turned around where the students started shaking their head, but realize, “Oh my God, you’re defeating all of us in this room.”

So yes, that was to your point, that the training can vary wildly in a live class, whether it’s virtual or on-site, based on the attendees. Because listen, I’ve been teaching Captivate since it was called RoboDemo, so years and years and years and years, and no class has ever been the same. No two classes are the same and it’s all based on the dynamics of the students in my live class. And you get one person in there who is stuck, can’t move forward, file open is a mystery. Go to the file menu, choose open. How do you do that? Okay, mouse skills. All of that can either derail or can help your class. Funny moments, whatever they may be. But asynchronous training, if you do it right, is always consistently good. The problem is there’s no live interaction. So you can’t ask that instructor, “Well, what do you think about this? What do you think about that?”

So yeah, you made me laugh when you mentioned that, that the dynamics of your live class, you better be fast on your feet to be a live trainer. So I am not saying, if you’re going to teach virtually, you shouldn’t know how to do it. Because listen, I think you’ll agree, there is a vast difference between teaching a class live on-site versus live online, or God forbid, live online and live on-site, where you’re doing both at the same time. Or if you’re going to do blended learning, you’ve got to mix all three, you better know what you’re doing as a facilitator and a trainer to do that or you’ll fall flat on your feet.

You’ll hear all kinds of complaints that people who teach these live classes on-site that now incorporate virtual, and they ignore the virtual audience completely. So the virtual audience is not included in the training, they feel like they’re watching a recording. So you’ve got to know how to engage this audience. I’m actually really stunned, Sarah, that conferences still survive on-site. We mentioned a couple of times before we turned on this recording, why are those conferences live on-site? People are going there to network face-to-face. I guess that’s the big one, but not the content that you’re learning. That content could have been taught virtually.

SO: Yeah, I’ve had the position for a long time that the most important part of a conference is the hallway track, right? The conversations at lunch, in the hallway, and in the exhibit hall and everywhere else. There’s a couple that are doing online in addition to in-person, and typically the-

KS: ATD does that. Yeah, does a good job at that. Yeah.

SO: Yeah, LavaCon is doing that, they’re coming up. But yeah, they have an online track with a chat, a pretty lively chat, and then they also have the in-person version if you can get there in-person.

KS: Which is successful only if the facilitator addresses the online chat, if the facilitator addresses someone who’s virtual. Yeah.

SO: And fun fact, Phylise Banner has been running that for years and years and years and has done a fantastic job of exactly that, of making sure that the online people get into the conversation, even when there’s 200 people in the room and another couple hundred on the chat, and she’s making sure that they get their questions into the discussion. Okay, so that was cheerful, and that made me feel better, because the first half hour of this was super not encouraging. So I think I’m going to close us out there because I’m pretty sure we could go on forever, but let’s leave it there. Kevin, thank you for coming and for giving us the inside information on what’s happening in training land. And hopefully I’ll see you again somewhere in-person at a conference.

KS: Or virtual, with the camera is fine. So yeah, great working with you, Sarah. Thanks for having me.

SO: Great to see you. Bye. 

Conclusion with ambient background music

CC: Thank you for listening to Content Operations by Scriptorium. For more information, visit Scriptorium.com or check the show notes for relevant links.

"*" indicates required fields

The post From classrooms to clicks: the future of training content appeared first on Scriptorium.

What if you could escape copy-and-paste and build dynamic learning experiences at scale? In this podcast, host Sarah O’Keefe and guest Mike Buoy explore the benefits of structured learning content. They share how organizations can break down silos between techcomm and learning content, deliver content across channels, and support personalized learning experiences at scale.

The good thing about structured authoring is that you have a structure. If this is the concept that we need to talk about and discuss, here’s all the background information that goes with it. With that structure comes consistency, and with that consistency, you have more of your information and knowledge documented so that it can then be distributed and repackaged in different ways. If all you have is a PowerPoint, you can’t give somebody a PowerPoint in the middle of an oil change and say, “Here’s the bare minimum you need,” when I need to know, “Okay, what do I do if I’ve cross-threaded my oil drain bolt?” That’s probably not in the PowerPoint. That could be an instructor story that’s going to be told if you have a good instructor who’s been down that really rocky road, but again, a consistent structure is going to set you up so that you have robust base content.

— Mike Buoy

Related links:

LinkedIn:

Transcript:

Introduction with ambient background music

Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations.

Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it.

Sarah O’Keefe: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change.

Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off.

End of introduction

Sarah O’Keefe: Hi everyone, I’m Sarah O’Keefe. I’m here today with Mike Buoy. Hey, Mike.

Mike Buoy: Good morning, Sarah. How are you?

SO: I’m doing well, welcome. For those of you who don’t know, Mike Buoy is the Senior Solutions Consultant for AEM Guides at Adobe since the beginning of this year of 2025. And before that had a, we’ll say, long career in learning.

MB: Long is accurate, long is accurate. There may have been some gray hair grown along the way, in the about 20-plus years.

SO: There might have been. No video for us, no reason in particular. Mike, what else do we need to know about you before we get into today’s topic, which is the intersection of techcomm and learning?

MB: Oh gosh, so if I think just quickly about my career, my background’s in instructional design, consulting, instructor, all the things related to what you would consider a corporate L&D, moving into the software side of things into the learning content management space. And so what we call now component content management, we, when I say we, those are all the different organizations I’ve worked for throughout my career, have been focused in on how do you take content that is usually file-based and sitting in a SharePoint drive somewhere, and how do you bring it in, get it organized so it’s actually an asset as opposed to a bunch of files? And how do you take care of that? How do you maintain it? How do you get it out to the right people at the right time and the right combination, all the rights, all the right nows, that’s really the background of where I come from.

And that’s not just in learning content; at the end of the day, learning content is often the technical communication-type content with an experience wrapped around it. So it’s really a very fun retrospective when you look back on where both industries have been running in parallel and where they’re really starting to intersect now.

SO: Yeah, and I think that’s really the key here. When we start talking about learning content, structured authoring, techcomm, why is it that these things are running in parallel and sitting in different silos? What’s your take on that? Why haven’t they intersected more until maybe now we’re seeing some rumblings of maybe we should consider this, but until now it’s been straight up, we’re learning and your techcomm, or vice versa, and never the twain shall meet, so why?

MB: Yeah, and it’s interesting, when you look at most organizations, the two major silos that you’re seeing, one is going to be product. So whether it’s a software product, a hardware product, an insurance or financial product, whatever that product is, technical communication, what is it? How do you do it? What are all the standard operating procedures surrounding it? That all tends to fall under that product umbrella. And then you get to the other side of the other silo, and that’s the hey, we have customers, whether those customers are our customers or the internal customers, our own employees that we need to trade and bring up the speed on products and how to use them, or perhaps even partners that sit there. And so, typically, techcomm is living under the product umbrella, and L&D is either living under HR or customer success or customer service of some sort, depending on where they’re coming from.

Now in the learning space you, over the last probably decade or so, seeing where there’s a consolidation between internal and external L&D teams and having them get smarter about, what are we building, how are we building it, who are we delivering it to, and what are all those delivery channels? And then when I think about why are they running in parallel, well, they have different goals in mind, right? techcomm has to ship with the product and service and training ideally is doing that, but is often, there’s a little bit of a lag behind, “Okay, we ship the thing, how long is it before we start having all the educational frameworks around it to support the thing that was shipped?”

And so I think leadership-wise, very different philosophies, very different principles on that. techcomm, very much focused on the knowledge side of things. What is it? How do you do it? What are all the SOPs? And L&D leans more towards creating a learning experience around, “Okay, well here’s the knowledge, here’s the information, how do we create that arc going from I’m a complete novice to whatever the next level is?” Or even, I may be an expert and I need to learn how to apply this to get whatever new changes there are in my world and help me get knowledgeable and then skilled in that regard.

So I think those are kind the competing mindsets and philosophies as well as, I won’t say competing, but parallel business organization of why we don’t usually see those two. And if we think about from a workflow perspective, you have engineering or whoever’s building the product, handing over documentation of what they’re building to techcomm and techcomm is taking all of that and then building out their documentation, and then that documentation then gets handed to L&D for them to then say, “Well, how do we contextualize this and build all the best practices around it and recommendations and learning experiences?” So there is a little bit of a waterfall effect for how a product moves through the organization. I think those are the things that really contribute to it being siloed and running in parallel.

SO: Yeah. And I mean many, many organizations, the presence of engineering documentation or product design documentation is also a big question mark, but we’ll set that aside. And I think the key point here is that learning content, and you’ve said this twice already, learning content in general and delivery of learning content is about experience. What is the learning experience? How does the learner interact with this information and how do we bring them from, they don’t understand anything to they can capably do their job? The techcomm side of things is more of a point of need. You’re capable enough but you need some reference documentation or you need to know how to log into the system or various other things. But techcomm to your point, tends to be focused much less on experience and much more on efficiency. How do we get this out the door as fast as possible to ship it with the product? Because the product’s shipping and if you hold up the product because your documentation isn’t ready, very, very bad things will happen to you.

MB: Bad, bad, very bad.

SO: Not a good choice.

MB: It’s not a good look. It’s not a good look.

SO: Now, what’s interesting to me is, and this sort of ties into some of the conversations we have around pre-sales versus post-sales marketing versus techcomm kinds of things, as technical content has moved into a web experience, online environment, and all the rest of it, it has shifted more into pre-sales. People read technical documentation, they read that content to decide whether or not to buy, which means the experience matters more.

And conversely, the learning content has fractured into classroom learning and online instructor led and e-learning a bunch of things I’m not even going to get into, and so they have fractured into multi-channel. So they evolved from classroom into lots of different channels for learning where techcomm evolved from print into lots of different channels, but online and so the two are kind of converging where techcomm needs to be more interested in experience and learning content needs to be more interested in efficiency, which brings us then to, can we meet in the middle and what does it look like to apply some of the structured authoring principles to learning content? We’ve talked a lot about making techcomm better and improving the experience. So now let’s flip it around and talk about how do we bring learning content into structured authoring? Is that a sensible thing to do? I guess that’s the first question: is that a sensible thing to do?

MB: Yeah, and here’s the thing that I like to keep in mind when talking about structured authoring, the context for why in the world would we even consider it? And when I think of traditional L&D training courses, whether it’s butts in seats at an instructor-led training event, whether I’m actually in a physical classroom or I’m sitting virtually in a Zoom class for example, or it’s self-paced e-learning, so much great content is built and encapsulated in that experience and is not able to be extracted out.

My favorite example of talking about this is I’ve got a big truck sitting in my driveway, I need to change the oil on it, it’s time. If it’s the first time I’ve ever changed oil, absolutely, I want all the learning. I want the scaffolding. I want the best practices, how I’m going to set up my work environment, the types of tools. How I’m going to need to deal with all the fluids, what I need to purchase. I’m going to dive into all that. In the real world, university of YouTube, I’m going to go watch videos on this and there’s going to be some bad content, there’s going to be some gems, and I’m going to pay attention to the ones that are good.

Now as I go from a novice, I’m going to build that knowledge of how to do it, I’m going to apply that knowledge. I’m actually going to go do it, now I’m probably going to make a mess and make mistakes my first time through, but that’s also building experience. So I’m moving from novice to knowledgeable to building skills to as I do it more and more, I move into that realm of being experienced.

Now as you move further up that chain, you need less and less support to the point where I’m like, “Crap, which oil do I need to buy? What are the torque specs on my drain plug?” I really only need three or four data points to do the job now. So that’s where as I move from a novice to an expert, I need to be able to skim and find exactly what I need in the moment of need, the just enough information. And so I’ll take the oil changing experience and let’s take that to any product or service training your customers, the people who are consuming your content are going through the same thing.

So learning-wise, why structured? Once I get to the expert level of things, I am not going to log into the LMS and I’m not going to launch that e-learning course, and I’m not going to click next 5 to 10 to 20 times to get to the answer that has the specification tables of, here’s what I need and what I need to do in order to accomplish the task at hand. Everybody’s nodding their head. Every time I ask, “When was the last time you logged into the LMS to get an answer to a question?” The only time I’ve ever had somebody go, “Oh, me,” it was actually an LMS administrator.

So learning is great at creating that initial experience, but their content’s trapped. It is stuck inside that initial learning experience. So getting back to the question, why structured authoring? Well, if you move to a structured authoring where you’re taking your content and building it in chunks, yes, you can create that initial learning experience where you’ve assembled that very crafted, we’re taking you from novice, getting you the knowledge, giving you the opportunities to practice the skill in a safe environment and fail well and learn from that and get you to a place where you move from novice to skilled. And then over time, this is where a lot of the L&D in general, because their content’s trapped in that initial learning experience, they can’t easily extract that information out and provide the things people need to move from skilled to experienced and experienced to mastery.

So that’s where when I think about, “Well, what does techcomm do really well? Techcomm supports that, I’ve got enough skills to do the job and I need to reference the very specific information, or the SOP, I’m on step four, I forget what are the things I need to enter in to get through step four, I can hop over the documentation and find that. So techcomm has figured out the structured authoring part. You mentioned creating new varied experiences for getting to the technical communication. Multi-channel delivery, I want to hop on and hit my search or hit my AI chatbot and pull up the information and just get me just enough to get through the tasks that I’m doing.

Learning’s still often stuck, if we equate it to the tech communication side, they’re still stuck in the, “I’m hand building a Microsoft Word based 500 page user guide that to get anything out of that, it’s a lot of work to build it, it’s a lot of work to maintain it, and it’s not easy to extract that information out to use it for other things.”

So why structured authoring, feature proof your content, make it more flexible. You’ve invested so much time and energy creating great content, great experiences, why not make it so it’s modular so you can pull things out and create new and different ways of consuming that content and delivering it in different bite size bits and pieces along the way?

SO: And I guess we have to tackle the elephant in the room, which is PowerPoint. So much learning and training, in particular, especially classroom training, is identified with an instructor standing at the front, running through a bunch of slides. And we like to say that PowerPoint is the black hole of content, that’s where content goes to die, and once it goes in, you never get it back out. So what do we say to the people that come in and they’re like, “You will pry PowerPoint from my cold, dead hands.”

MB: Such a great question. I’ll jokingly refer to PowerPoint as “My precious.” Here’s the reality: PowerPoint is not the knowledge chunk. That knowledge is actually sitting in the head of the instructor, the PowerPoint is providing the framework for them to deliver and impart that knowledge and impart those best practices. It’s there to provide guardrails so that it’s done in a consistent fashion, and there’s a bare minimum amount of structure that… There’s a bullet point there, they’re going to talk about it. The degree to the quality of how they’re going to talk about it and present it is going to vary based on the person delivering the content. So if you’ve got a bunch of PowerPoint slides, you don’t necessarily have all of your training material well documented. Now, if you’ve got parallel instructor guides and student guides that talk about the details of what should be said behind those bullet points, you’re a lot closer to having that information.

So why structured authoring? Well, it’s kind of, again, the good thing about structured authoring is you have a structure. You have a, if this is the concept that we need to talk about and discuss, here’s all the background information that goes with it. So with that structure comes consistency, and with that consistency, that means that you have more of your information and knowledge documented so that it can then be distributed and repackaged in different ways. Because if all you have is a PowerPoint, you can’t give somebody a PowerPoint when they’re in the middle of an oil change and say, “Here’s the bare minimum you need.” When I need to know, “Okay, what do I do if I’ve cross-threaded my oil drain bolt?” That’s probably not in there. That may be an instructor story that’s going to be told if you have a good instructor who’s been down that really rocky road. But again, structure and being consistent about it is going to set you up so that you have robust base content. 

We’ve got Legos in the house, I got two boys. Gosh, I’ve stepped on so many Legos in my life, it’s ridiculous. But the Lego metaphor works because you have a more robust batch of Legos that you can create new creations from, rather than a limited set if you’re only doing PowerPoint.

SO: And because you’re nice, and I’m not, I’ll say this, we can produce PowerPoint out of structured content, that is a thing we can do. I’m not saying it’s going to be award-winning, every page is a special snowflake PowerPoint, but we can generate PowerPoint out of structured content. And if you’re using it as a little bit of an instructor support in the context of a classroom or live training, that’s fine.

A lot of the PowerPoint that we see that people say, “This is what I want, and if you don’t allow me to do this,” and there’s this rainbow unicorn traipsing across the side of the page kind of thing, and no, we can’t do one-off slides, we can’t do crazy every slide is different stuff, but the vast majority of the content that I see that is PowerPoint based and kind of all over the place is not actually effective. So it’s like, this is not good. We have the same issue with InDesign. We see these InDesign pages that are highly, highly laid out, and it’s like, “We need this.” Well, why? It’s terrible. I mean, it’s awful. What are you doing here? No, we can give you a framework.

MB: Now, you’re telling somebody that their baby’s ugly when you say that, that’s somebody’s baby.

SO: I would never tell somebody that their baby is ugly, but I have seen a lot of really bad PowerPoint. Babies are wonderful.

MB: Yes.

SO: It’s so bad. So why does the PowerPoint exist, and how do we work around that? And also, are you delivering in multiple languages? Because if so, we need a way to localize this efficiently, and we’re right back to the structured content piece.

MB: And as soon as you’re talking about with PowerPoint, it is the poster child of pixel-perfect placement. As soon as I take a perfectly placed pixel product and have to translate it from English to let’s just say French, just the growth of the text alone, now I’ve got what was a perfectly placed pixel layout, my beautiful slide is now a jumbled mess. So just because you can doesn’t mean you should. And the thing is, PowerPoint and Microsoft Excel are the duct tape that runs business. Everybody has it. Everybody uses it. That’s the reality.

Now, the thing is, does everything have to be structured? I don’t believe it has to be. They are absolutely the one-off snowflake instances where, you know what? PowerPoint is the exact right tool for the job. Maybe it’s the one-off presentation that really is not going to see any reuse, it’s expendable, it’s disposable. We need to get the information communicated quickly. I’m going to fire it PowerPoint. I’m going to use it as my, I’m going to do air quotes, “My throwaway content” because it’s something that is short, sweet, and needs to be communicated, absolutely. I’m not, and I don’t think you are either, saying that PowerPoint has to go away, it’s the when is it appropriate and when is it not?

SO: I mean, I am the queen of the one-off can never be reused content being developed in, now I refuse to use PowerPoint, but in slideware for a short presentation, so the next one of you that’s listening to this and walks up to me at a conference and says, “Oh, is your presentation structured content?” No, it is not. Thank you for asking. Why isn’t it structured? Because I don’t reuse it at scale. Because in fact, every presentation at every conference is a special snowflake and has been lovingly handcrafted by me to deliver the message that I need, the context that I need, potentially the language, but to your point, even if I’m not localizing the presentation itself, the cultural context matters. So if my audience is largely English-speaking or primarily English, or… I mean, we’re going to Atlanta for LavaCon, that is going to be mostly a US-based audience, and maybe we get some Canadians, eh. And other than that… But mostly US and a US context. Will I be using excessive amounts of images from the Georgia Aquarium? Yes, I will.

Now, when I go to conferences elsewhere, so let’s take tcworld in Germany in November, that audience is, we’re delivering content in English, and the audience ranges from perfect English speakers to sort of barely hanging on. And so my practice at a conference like that is to include more text on my slides because if I include some additional text, it gives the people that are not quite as comfortable in English, a little bit more scaffolding to hang onto as they’re trying to follow my ridiculous analogies and insane references to cultural things. I also do try to pay attention to the kinds of words that I’m using and the kinds of idioms that I’m using so that they’re just not completely lost in space or things are not coming from left field or whatever. So the context matters, and no, my presentations are not structured.

But pulling this back, let’s talk about the potential. So when we look at learning content and you think about saying, okay, we’re going to structure our learning content or we’re going to structure some of our learning content, what does that mean in terms of what gets enabled? What are the possibilities? What are the things that you can do with structured learning content that you cannot do in unstructured, by which I mean PowerPoint, but unstructured, locked-in content? If we break this stuff into components and we deliver on structured learning content, what are the ideas there? What are the possibilities?

MB: Well, as you’re explaining the PowerPoint point of view, a word that came up a few times was scale. I’m not having to do it at scale. Effectively, it is a one-off. Yes, I’m going to personalize it for the audience, and the degree of personalization and customization that you’re doing per conference, per audience, per default language that they’re speaking, you’re able to scale that to the degree that you need to. There’s no need for you to put your content in data and localize it and do all the things that you need to do. So it’s really that word at scale, that, I think, is the key word.

It’s when you hit that tipping point where the desktop tools that you’re using today, and we can say this with tech communications as well, I was using Word and Excel and copy and pasting and keeping things in sync, it works until you get to a tipping point where the scale no longer is sustainable. That same exact problem exists in training. So when you’re looking at things like, I have my training content that when I deliver it in California, I have to put my Prop 65 note in everything because Lord forbid, as soon as I step across the state line into California, everything that’s around me is going to give me cancer. Prop 65 is the default thing that you see plastered everywhere.

So do I need to customize my content for delivering in California? Perhaps. Maybe different states have different regional laws or policies that apply to only that audience. That’s where that mass customization and mass personalization are really hard to scale because now you don’t have just one course, you have potentially 50 courses, if I’m just talking about the US, 50 states, 50 courses, and I have to have 50 different variations, which means that not if something changes, but when something changes, now I have to open up and change 50 different courses, and it’s not, did I miss anything? It’s, “What did I miss?” That’s the thing that you wake up in the morning in a cold sweat of, “Oh my God, what did I miss?”

So why structured for learning? Largely when you get to that tipping point where you’re copy/pasting, and I call it the copy/paste published treadmill, when you are on that hamster wheel of copy/paste/publish, copy/paste/publish, and that is the majority of what you’re doing, and you’re looking at a pie chart of how much time is spent maintaining your courses or taking a base course and creating all the variations, that precious PowerPoint that is the handcrafted bespoke one-off, you can’t do that anymore. That’s the equivalent of, you look at a Lamborghini, how many do they make a year? They can afford to make a very small number per year because they’re really expensive to make. When you look at a Ford Mustang, which probably gives you 80% of the performance at a fraction of the cost and exponentially scales well beyond, it’s because they’ve taken that structured approach of, every frame’s the same, every hood’s the same, very few handcrafted things, and the things that are going to be handcrafted, that’s when I go order the special edition Shelby Cobra that has some handcrafted components put onto the basic structure. That’s that same metaphor applied.

So why structured content? Because I want to have modular content that can be reassembled really quickly, that I may have chunks that are reused so that when I need to slip in my Prop 65 disclaimers, I can do that at scale and have 50 variations of a course, but when it comes time to update it, I’m literally updating one or two things and it’s automatically updating all 50 courses and of course all the efficiencies of publishing things out in a structured format.

So that pixel-perfect placement, I’m going to give that up to stay sane so I can get home and have dinner with my family, because the amount of time that I’ve spent in my life doing pixel-perfect placement and updating things, God, I wish I could hit the way back machine and reclaim all that time in my life. How many… Guilty as charged. Show of hands of anybody who’s listening, how many times have you sat there and fiddled with the slide or a text box in InDesign then design to get it just right, that two days later, something changes and you’re back there spending 10, 15 minutes doing it to fiddle it in just right. So, as I affectionately like to say, I’m a recovered FrameMaker, InDesign, PowerPoint, and Word user because I want to author it in a structured format so that I am giving up the responsibility of layout and look and feel.

SO: I like to tell people, “I’m not lazy, I’m efficient.” The fact that I don’t want to do it is just a bonus; I can get out of doing all this work.

MB: That’s right, that’s right.

SO: Because we are not allowed to leave any podcast without covering this topic, what does it look like to have AI in this context?

MB: There are two sides of the AI coin from a content perspective, I think, and it’s the, “How can AI help me do my job better to create content?” Some things that when we’re looking at duplication of content, things that AI can do really well that, working smart, not hard, help me find things that already exist in my repository of structured content that look like this, that are really close. The human in the loop, so helping me deduplicate or help me not create new unnecessary variations of content. I think that’s one area of AI-based assistance for content creation that people may not be necessarily thinking about. Because right now, the easy one is like, “Hey, ChatGPT, help me write an introduction or an overview for the following,” it spits that out. That’s great, but that overview and that content may have already been written by somebody else, and so what ends up happening is you start generating content drift where it’s almost exactly the same but just slightly different. And in reality, yes, I could have used the one that was already there.

So I think that’s one of the areas where AI from a content authoring perspective is one that I’m really excited about. Because at the end of the day, and this leads us into the second part of AI, AI is only as good as what you feed it, and if you feed it junk food, you’re going to get junk results. So it’s that whole thing of do you eat healthy food or are you going to eat Cheetos? If you’re pointing your AI at a SharePoint repository and saying, “Hey, read all of this,” and all the content shifts and variations and content drift and out-of-date and perhaps out-of-context content that exists inside of that repository, your results are not going to be as accurate as they need to be. So, how do you ensure that AI is providing good results? Well, you feed good content.

And so within an organization, I think the two silos that we started our conversation with, technical communications and L&D, tend to have some of the most highly vetted, highly accurate, up-to-date content in an organization. And so this is my encouragement to everybody who’s in this space, you are the owners of what is good, highly nutritious food that you can feed your AI. So taking it back to the structured content perspective, if I’m authoring in the structured content, publishing it out in a format that is AI ready, all of your tags, all of your enrichments, all of your, here’s the California version of the content versus the Georgia or Florida’s version of the content, all of that context and enrichment and tagging that’s gone on, you’re now feeding AI all of that context so that AI can provide the proper answer. So that’s my short, it’s sweet for the AI side. We could talk for probably days on all sorts of other variations, but right now, that’s where I’m seeing the biggest impact that it’s going to have on techcomm and L&D.

SO: I think that’s a great place to wrap it up. And I want to say thank you for being here and for a great conversation around all of these issues, and we will reconvene at a future conference somewhere to cause some more trouble and talk some more about all of these things. So Mike, thank you.

MB: You are welcome. And yeah, I think the next conference we’re going to see each other is going to be LavaCon, so I’ll be talking in and around the convergence of L&D and techcomm and what life can look like with that. So certainly a deeper dive and continuation of what we started here, and super excited to sit on your session as well.

SO: Yep, super. I will see you there. I’m pretty sure I’m doing one on the same topic, but it will be more complaining and less positive, so that seems to be my role. Okay, with that, thank you everybody, and we’ll see you on the next one. 

Conclusion with ambient background music

CC: Thank you for listening to Content Operations by Scriptorium. For more information, visit Scriptorium.com or check the show notes for relevant links.

"*" indicates required fields

The post From PowerPoint to possibilities: Scaling with structured learning content appeared first on Scriptorium.

Every time someone views your product content, it’s a purposeful engagement with direct business value. Are you making the most of that interaction? In this episode of the Content Operations podcast, special guest Patrick Bosek, co-founder and CEO of Heretto, and Sarah O’Keefe, founder and CEO of Scriptorium, explore how your techcomm traffic reduces support costs, improves customer retention, and creates a cohesive user experience.

Patrick Bosek: Nobody reads a page in your documentation site for no reason. Everybody that is there has a purpose, and that purpose always has an economic impact on your business. People who are on the documentation site are not using your support, which means they’re saving you a ton of money. It means that they’re learning about your product, either because they’ve just purchased it and they want to utilize it, so they’re onboarding, and we all know that utilization turns into retention and retention is good because people who retain pay us more money, or they’re trying to figure out how to use other aspects of the system and get more value out of it. There’s nobody who goes to a doc site who’s like, “I’m bored. I’m just going to go and see what’s on the doc site today.” Every person, every session on your documentation site is there with a purpose, and it’s a purpose that matters to your business.

Related links:

LinkedIn:

Transcript:

Introduction with ambient background music

Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations.

Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it.

Sarah O’Keefe: Change is perceived as being risky, you have to convince me that making the change is less risky than not making the change.

Alan Pringle: And at some point, you are going to have tools, technology, and process that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off.

End of introduction

Sarah O’Keefe: Hi, everyone, I’m Sarah O’Keefe and I’m here today with our guest, Patrick Bosek, who is one of the founders and the CEO of Heretto. Welcome.

Patrick Bosek: Thanks, Sarah. It’s lovely to be here. I think this is may be my third or fourth time getting to chat with you on the Scriptorium podcast.

SO:  Well, we talk all the time. This is talking and then we’re going to publi- no, let’s not go down that road. Of all the things that happen when we’re not being recorded. Okay. Well we’re glad to have you again and looking forward to productive discussion here. The theme that we had for today was actually traffic and I think web traffic and why you want traffic and where this is going to go with your business case for technical documentation. So, Patrick, for those of you that have not heard from you before, give us a little bit of background on who you are and what Heretto is and then just jump right in and tell us about web traffic.

PB: No small requests from you, Sarah.

SO: Nope.

PB: So I’m Patrick Bosek. I am the CEO and one of the co-founders of Heretto. Heretto is a CCMS based on DITA. It’s a full stack that goes from the management and authoring layer all the way up to actually producing help sites. So as you’re moving around the internet and working with technology companies, primarily help_your_product.com or help_your_company.com, it might be powered by Heretto. That’s what we set out to do. We set out to do it as efficiently as possible, and that gives me some insight into traffic, which is what we’re talking about today, and how that can become a really important and powerful point when teams are looking to make a case for better content operations, showing up more, producing more for their customers, and being able to get the funding that allows them to do all those great things that they set out to do every day.

SO: So here we are as content ops, CCMS people, and we’re basically saying you should put your content on the internet, which is a fairly unsurprising kind of priority to have. But why specifically are you saying that web traffic and putting that content out there and getting people to use the content helps you with your sort of overall business and your overall business case for tech docs?

PB: Yeah. So I want to answer that in a fairly roundabout way because I think it’s more fun to get there by beating around the bush. But I want to start with something that seems really obvious, but for some reason it isn’t in tech pubs. So first of all, if you went to an executive and you said, I can double the traffic to your website, and then you put a number in front of them, probably say a hundred thousand dollars, almost like any executive at any major organization is like a hundred thousand dollars, of course, I’ll double my web traffic. That’s a no-brainer. Right? And when they’re thinking of website, they’re thinking of the marketing site and how important traffic is to it. So intrinsically, everybody pays quite a bit of money and by transference puts a lot of value on the traffic that goes to the website and, as they should. It’s the primary way we interact with organizations asynchronously today.

Digital experience is really important. But if you went to an executive and you said, I can double your traffic to your doc site, they would probably be like, wait a second. But that makes no sense because nobody reads the docs for no reason. I want to repeat that because I think that’s a really important thing for us, as technical content creators to not only understand, I think we understand it, but to internalize it and start to represent it more in the marketplace and to our businesses and to the other stakeholders. People might show up at your marketing site, because they misclick an advertisement. They might show up in your marketing site because they Googled something and your market and a blog like caught them and they looked at it. So there’s probably a lot of traffic where people are just curious. They’re just window shopping. Maybe they’re there by mistake. But nobody shows up at your documentation site.

Nobody reads a page in your documentation site for no reason. Everybody that is there has a purpose and that purpose always has an economic impact on your business. People who are on the documentation site are either not utilizing your support, which means that they’re saving you a ton of money. It means that they’re learning about your product, either because they’ve just purchased it and they want to utilize it, so they’re onboarding, and we all know that utilization turns into retention and retention is good because people who retain pay us more money, or they’re trying to figure out how to use other aspects of the system and get more value out of it. There’s nobody who goes to a doc site who’s like, I’m bored. I’m just going to go and see what’s on the doc site today. So every person, every session on your documentation site is there with a purpose and it’s a purpose that matters to your business. So that’s why I want to start. That’s why it matters. That’s why I think traffic is important, but you look like you want to contribute here, so.

SO: We talk about enabling content. Right? Tech docs are enabling content. They enable people to do a thing, and this is what you’re saying. People don’t read tech docs for fun. I know of, actually, I do know one person. One person I have met in my life who thought it was fun to read tech docs. One.

PB: Okay. So to be fair, I also know somebody who loves reading release notes.

SO: Okay. So two in the world.

PB: But hang on, hang on. But this person, part of the thing is this person is an absolute, can I say fanboy, is that, they’re a huge fan of this product and they talk about this product in the context of the release notes. So even though this person loves the release notes, the release notes are a way that they go and generate word-of-mouth and they’re promoting your product because of the thing they saw in the release notes. The release notes are a marketing piece that goes through this person. All the people who are your biggest fans are going to tell people about that little thing they found in your release notes. Sorry. Anyways.

SO: So again, they’re trying to learn. Okay. But, so two people in the universe that we know of read docs for fun. Cool. Everybody else is reading them, as you said, for a purpose. They’re reading them because they are blocked on something or they need information, usually it’s they need information. And then you slid in that when they do this, this is producing, providing value to the organization or saving the organization money. So what’s that all about?

PB: Well, I mean there’s a number of ways to look at this. You want to start with the hard numbers, the accounting stuff, the stuff you can take the CFO. That stuff is actually, it’s pretty easy to do. You can do it in just a couple of lines. So every support ticket costs a certain amount of money. Somebody in your organization knows that number, if your organization is sufficiently large and sufficiently large is like 20 people probably. Maybe that’s not that small, but if you’re a couple hundred people, everybody knows what that number is. So it’s very easy to figure out how much it costs when somebody actually goes to the support.

SO: Somewhere between $20 and $50 is kind of the industry average per call. You may have better numbers internally in your organization, but if you don’t or you don’t know where to start there. Every call is $25.

PB: Yeah. $20, $25. A little more, if you’re in a complex industry. The reality is that when you start comparing it to how much you spend answering a question with content, it’s kind of like, oh, is it a thousand times cheaper or is it 2,000 times cheaper? So it’s not really that big of a difference. The cost of answering a question with content is also pretty straightforward. So all you really need to know is how much are you spending on your content, which is typically speaking just the combination of the people and tools, so people in content operations stack that you’re using to get that content out in front of people. And then the page views. I mean, fundamentally if you exclude search, so take search out of your page views, take home page out of your page views, if you can filter section pages, so just look at actual content pages and then you have to pick a resolution rate.

Obviously, if you want to say 100%, if you don’t have any better metrics, that’s probably too high. Maybe it’s unreasonable, but it’s very simple. It makes the equation easier. If you want to say that 50% percent of people who read what you’ve considered to be like a content page, resolve their issue, that’s probably too low. So pick a number between those two things and you run the multiplication on that and you’re going to find out that it’s going to cost you, in most situations, less than a penny to answer a question, typically way less than a penny to answer a question with content as opposed to the $25. That’s the pure economic math of it. There’s more though.

SO: Okay. So yeah, we did some math and we’re basically saying, looking at this in a tech support centric way, usually we talk about call deflection. Right? So the idea is that every time somebody does not call tech support, you save $25 and spend a penny, a fraction of a penny instead, which seems good. Now interestingly to me, I think, you can look at this as the first time somebody hits that site and hits a content page, costs really a lot of money. Right? Because the people and the tools and the setup and the publishing, but then the next one is zero.

So you’re replacing sort of an upfront planned cost with a recurring cost because every time somebody calls, it’s another 25 or 50 or whatever dollars. So there’s a huge scalability argument here, and I can make a decent case for if you are a startup, a day one startup, you have no content, you have nothing, you have no infrastructure, cool. Hire a tech support person. Let them do their thing for maybe a year, and then look at the top 10 queries that they had and write some docs and deflect off those top 10 queries and handle it that way. But most of our customers, speaking for both of us, are medium to large to incredibly large organizations that have content. We’re not talking about the you have nothing start from scratch scenario.

PB: A hundred percent. When you’re really thinking about where you get the value, both on the accounting side, like saving money, so bottom line stuff, and then also the customer experience, which I think is worth getting into in a minute, that’s really going to take place when you start scaling up. I agree with you that a startup style organization should write content. Even small organizations benefit from it. I think they, small organizations actually benefit in a slightly different way than the deflection, which is the word you’re using. And I’m going to come back to that because I have a pet peeve with that word, but I’ll use it right now for the purposes because we’ve been using it. I think that what, the value that a smaller organization gets is not in the deflection, but it’s actually in the presence. So if you’re trying to show up and you’re trying to compete with larger organizations and you’re doing something, which is technical or considered to be highly important, so you’re in a high technology industry, your buyer is going to go look at your documentation. They’re going to look at your competitor’s documentation as well.

And if your documentation appears to be not that great, it’s very thin, there’s not a lot there, that’s going to be a factor in a buying decision. And I know everybody kind of like, yeah, but it really is, and I can tell you because we’re not a huge organization, that we’ve won deals because our docs were better. We invest in it, as we should. We’re a documentation tool. You know? So it does matter at the smaller end, even if you can’t build a really scalable content operation stack that you probably don’t need.

SO: Now, personally, I’m okay with deflection, and I’ll also say that the key thing here is that if you’re doing additional research on this as a listener, call deflection is sort of the industry term that will help you in your Google/AI searches. But tell us about why call deflection is bad and evil.

PB: Okay. So that is true. If you are talking to executives, you should probably say deflection, but maybe forward-thinking executives would appreciate why I think deflection is a bad term. I think we should use, you’re shaking your head at me. Fine. I think we should be talking about call avoidance. And the reason that I think this is because when most people think about deflection, they’re thinking about it as being very reactive, and it’s that box that pops up when you’re trying to put a support ticket in that’s like, well, have you already looked at this? And by the time someone has arrived at your support site and they have decided that they want to interact with a human, they are annoyed. They don’t want to be there. Nobody visits the support site because they want to. They have made the emotional commitment that they’re going to go and deal with one of your human beings to solve their problem, which is not something they planned on doing today. Nobody wanted to do this when they got up in the morning. So you’ve already failed. And at that point in time, the best thing you can do is get them to a human efficiently without sticking things in front of them and trying to deflect them. So that’s why I don’t like deflection. Avoidance is that that never happened. They Googled it because Google is tier zero support for everybody, even if yours is bad. They got an answer or they ChatGPT-ed it, different topic, there’s problems there. But probably they Googled it. They got an answer very quickly. They solved their issue. You never heard about it. It cost you a fraction of a penny. They had a great experience. It’s how they prefer to get their information, and you avoided the support rather than trying to deflect them to save yourself a couple of bucks when they were annoyed and broke their customer experience.

SO: Yeah. I’m on board with that. It’s just that terminology-wise, we’ve got to work with what we’ve got. But I would agree that avoiding the call in the first place, and I talk about how when people call tech support, they’re mad. If you think about the emotional state of your customer, the tech support person is angry. There’s also the issue that they asked ChatGPT and it said something wrong, and then they call up tech support and yell at you because ChatGPT was wrong, which is, that’s a whole other podcast. So let’s just set that aside for a moment, but okay.

PB: Maybe you’ll invite me back. We can talk about that.

SO: Yeah. So the, it’d be a long podcast and we’ll have to lift our no profanity rule for that one just to get through the topic.

PB: Oh. Special edition.

SO: Special edition. Okay. So you were talking about the value though of a documentation site and we’ve sort of paired it with tech support and with this avoidance, deflection, get them the answers that they need before they get angry at the product. Right?

PB: Yeah. For sure.

SO: How does the customer experience tie into that? And what is the value of the customer experience?

PB: So the value of the customer experience is subjective, but every organization already has an opinion on it. Some organizations place a lot of value in customer experience, have done a lot of work to tie customer experience to the metrics and analytics and things like that they use to track financial performance. Other organizations less. So the first thing I would say is go and see where your organization is relative to their thinking on customer experience. But, as you’re talking about customer experience, other than the support, which I think we’ve covered that quite a bit, for someone who’s showing up your documentation site, really what it’s touching on is a couple of things, what they’re trying to get to. So there’s the discovery aspect of it. And this can be very, very simple or it can be very, very complex. The simple one I like is like let’s say you sell gym equipment and that gym equipment goes out to people who own gyms, as it would make sense, and they’re going to go and they’re thinking of buying a new treadmill or something from you.

They’re going to want to know, is this going to fit in my gym? Can the power I have set up work with it? What are the other details of this product? And then how much information is there to service it? So somebody, once they get past the whole like, okay, I kind of like this brand, maybe this is a good thing, it’s kind of cool, they’re going to go into the documentation because they’re making a purchase that matters to them. And having confidence and trust in the product based on the depth of information that they get prior to purchasing it is a major factor. And this only increases as the economic value and the end implementation, like how critical it is, how system critical it is, increases. So there’s a discovery, evaluation, and confidence, those are the three things I think of, aspect to your documentation or your help site that is there, even if you’re not thinking about it, even if it’s not coming up directly in sales conversations. I promise you, because I have the data that people are doing this during the process of deciding if they want to work with your organization. And that’s the kind of pre-customer experience that’s really, really critical that most organizations are just not thinking about and they’re probably leaving a lot on the table relative to their competitors that either could be advantage or they’re behind.

SO: There was a study. It was a while back, maybe five or 10 years ago that came out from, I always have trouble finding it. It was either PwC or IBM. The gist of it was that 80% of people that were buying consumer products were doing pre-buying research, the technical research. So they were looking at specs and they were looking at how do I install this thing and various other things that we consider to be not marketing information. They were looking at what is traditionally labeled post-sales documentation.

PB: Yeah. Because people care. And the other thing too is like as we move into an economic environment where people are more careful about what they’re spending on, they’re only going to do more research to make sure the things they’re buying are things that are going to last and be supported. I bought a pair of headphones a year ago, and I have an issue with one of them. They’re like the ones that go in your ears, one of them’s not working. I ended up going to the documentation to try to figure it out, and the documentation was so bad I could not make heads or tails of it. And I just gave up and I was like, okay, if I had spent hundreds of dollars on these, I’d go through the process, but they were like 30 bucks or whatever. But I’m never going to do business with that company again. Ever.

If I see another one of the products, I will never buy it. So they don’t know about that experience. But if you have, not even just bad content operations, because frankly their site was, it was kind of nice, it wasn’t bad. I think it could have been better, but you know, funny that I would have that opinion, but it was really the information architecture, so it was kind of the stuff that Scriptorium, Sarah, you guys would help them with. It wasn’t so much they had bad tools. They had terrible organization, and the content was, I’m not allowed to swear, which I wouldn’t anyways…

SO: Sorry.

PB: … but the content was, think of your word, it was bad. It was completely unhelpful. So you can have the best content operations in the world, but without the right information architecture, who cares?

SO: Yeah. This is the infamous if a tree falls in the forest. You know, and to your point, A, the company doesn’t know that your headphones are broken and you’re unhappy, but you just told me, and the next time I’m in the market to buy a pair of headphones, I’m going to remember this story and I’m going to call you, I won’t call you. I’ll send a text and say, hey, what brand was that? Right? And you’re going to tell me and then I’m going to not buy them. So the impact of this failure of documenting, well, actually it’s a product fail, right, but also of support. Because if they had come back with, oh, we’re so sorry, send them in or we’ll send you a new pair, whatever, they could have rescued this encounter, but they didn’t. So the next thing that’s going to happen is that you and every single one of your friends that hears the story will never buy that brand.

PB: Right.

SO: So as we talk about this, the really critical point here though is, I think, there are a bunch of really critical points, but the one that I really want to zoom in on is that the content has to be there. Right? You have to have helpful content that solves the problem that a person is on the website for. And, in your case, it might have been, oh, sometimes this happens and you have to repair them, or you have to this or you have to that, you know, press all these weird buttons in this weird sequence and sacrifice the chicken and stand on your head. Cool.

PB: Right. Which I would’ve done.

SO: Which you would’ve done. But the bigger problem is that you went to their website and we don’t actually know whether or not this problem is fixable because you didn’t find it. Right? You didn’t find the answer. And that means that it’s sort of like a last mile problem. I can write all these really good procedures, they can be super accurate, they can be amazing, blah, blah, blah, blah, blah. You come onto my website, you can’t find the answer your question, it exists, but you can’t find it, right, you, the customer, and so it fails, and now you either, A, tell all your friends that company XYZ is terrible, or, B, you call tech support and you’re mad. Right?

PB: Yeah.

SO: That’s actually the best outcome.

PB: It is. Yeah.

SO: Yeah. And interestingly, we’ve got some, I’ll be very non-specific, but we have a project right now where one of the top tech support topics, you know how you look at what are the top 10 things that people call and ask about, and it’s like, my headphones aren’t working, or how do I return this or whatever. One of the most common reasons that people call their tech support is to ask, where is the documentation? I can’t find it.

PB: Do you have any idea how common that is? I mean, you probably do, but it’s so common.

SO: Yeah.

PB: And we’ve started doing this thing in the process of helping people think through this where we have a very simple tool that we use. It’s a sheet, happy to share it with anybody, and a process where you effectively go through and you just do a very simple 15 to 30 minute interview with X number of support people. You know, we recommend three to five. Some people do more. And you just go through the last 10 support cases, the ones that they worked on, and there’s a few things you mark off, but the idea is to do lightning round, very, very quick. And could this be solved by documentation? And the amount of it that is just looking for documentation, I can’t find it, is so funny. And you’re like, I think that’s a problem. And people are like, wow. But you can’t blame them because people don’t think about these things and it doesn’t make sense that you would because it’s non-obvious, and I think that’s one of the really critical things I want to leave people with.

And I have one other thing that I, you know, we’ve been talking for a while that I want to let people go soon, but this one, I want to zoom in on this for a second. People shouldn’t feel bad that they haven’t thought about this. They shouldn’t feel bad that they haven’t thought about the value of the traffic, the impact of the traffic, the customer experience side of it, the cost ratio of the traffic relative to support people. It really isn’t that obvious. And there’s so much momentum around the way that we’ve done business in having people solve problems for other people in direct communications that even if that isn’t ideal, that’s just the way it’s done and that’s what feels obvious. So don’t feel bad about not having thought about this if you haven’t. Your colleagues shouldn’t either. But it is the way the world is moving, and I think it’s critical to start thinking about it now.

SO: Yeah. And you started this by talking about customer experience needing to be asynchronous. People can get the stuff, self-service when they want it and digital as opposed to call somebody on the phone. So let’s sort of wrap this up and say, what’s your advice to people that know that they’re struggling with this? They know that they have huge tech support volumes and nobody’s happy. And I mean, we know we have a problem. So where should they start? What’s the first step that they can take to begin attacking this thing in a way that will lead to forward progress within a large organization that has as their informal motto, oh, they can just call tech support.

PB: Yeah. So I would say buy-in is always step one, and that means that there’s going to be some selling that has to happen at the organization. You have to get people to recognize the value, the potential, and also the ability to achieve it. So it’s those things when they come together, there can be a ground swell where people are going to actually support these projects and fund them and get involved, and then you’ll have really successful projects. One of the big challenges with getting that buy-in historically has been that there’s no precedent. So when you’re looking for a better website, you already have a website. What if you increase traffic by 10%? You know, people can start to draw some lines between that and sales or the bottom line or value, those types of things. And oftentimes, even organizations that I would say are somewhat up to maturity curve in terms of tech pubs, they don’t have any metrics about their site, like how many people come to it? I don’t know.

They just don’t track it. So there’s not this historical precedent of metrics that can be back to results, and that can create some issues. So the advice that I give organizations that are in that situation is if you are in a technology field and you have a relatively complex product, so something where it breaks, it’s not always obvious how to use it, there’s a reason that people would need to learn about your product for some reason, what our data shows from having done this many times with organizations that fit that profile is that a well-implemented documentation help site, whatever you want to call it, gets about as much traffic as the dot com, the primary marketing site. It tends to be plus or minus 15%. We’ve actually seen as high as 65% of the total traffic between the two sites being on documentation.

That’s a bit of an outlier, but so is 30%. You know, we’ve seen that too. So if you’re want to be conservative, say you’ll get 40% of the total traffic. So four sessions for every six on a marketing site. If you want to be, what we tend to see on an average, just say it’s one for one. If it’s one for one and you don’t have metrics, that’s a target. And you have to ask the internal question, what’s the value of that? If we get a hundred thousand sessions per month or per year or whatever on the marketing site, what if we had a hundred thousand sessions on the help content? Well, those people are there for a reason. Remember? They’re there because they’re not calling support. They’re there because they’re onboarding and using our system better, or they’re there because they’re trying to figure out if our stuff’s going to work for them.

So like how valuable would that be? And once you get the organization to a place where they’re like, oh, that would actually be quite valuable, could we get that, I think 80% of the work is done and well, 80% of the work of getting started is done. And then you probably call somebody like Scriptorium or Scriptorium specifically, if you’re not familiar with this, and you start the process of actually thinking through of how to do it. But I do think the organizational buy-in and giving people in the right head space to think about the value of this is step one, and that’s the process I use for it.

SO: Yeah. I think I would agree with all of that, especially the part where they should call us.

PB: Go figure.

SO: But the key thing in here is, and you said this a different way, but changing the momentum, right, getting organizational buy-in, getting people on board with this concept. The other thing I’ll say is that ultimately one of the biggest problems we face in content ops is that so much of it is invisible in the sense that we’re going to refactor this and we’re going to do it better, and we’re going to produce it faster and we’re going to automate, okay, great, but you’re still producing the same thing. One of the most powerful things we can do early in the process is say to people, look at this portal that we can deliver. Look at this experience that we can deliver. It’s not the first thing or the only thing or even necessarily the most important thing we need to do, because the portal has to have content. Right?

PB: Yeah.

SO: I mean, it’s kind of a chicken and egg thing, but showing people the vision of what can be works typically much, much better than saying we should do structured content because it will help automate things and speed up time to market. That’s all behind the scenes, and it’s not visual and nobody cares. I mean, people care, but it’s hard to visualize. So, okay, I think we’ve promised people a whole bunch of resources. We will put those in the show notes. I’m quite certain that we could go on for a very long time about this topic, but I am going to wrap it up there ’cause I feel like we hit a good starting point for people.

PB: Yeah.

SO: So if there are other questions, I would say reach out to me or to Patrick, because I know we’ve only scratched the surface on this thing. Patrick, thank you for being here.

PB: Of course. Always a blast.

SO: Always good to see you. And we will wrap this thing up, and thanks for being here. Feel free to reach out if you have any other questions.

The post Every click counts: Uncovering the business value of your product content appeared first on Scriptorium.

In this episode of the Content Operations podcast, Sarah O’Keefe and Bill Swallow unpack the promise, pitfalls, and disruptive impact of AI on multilingual content. From pivot languages to content hygiene, they explore what’s next for language service providers and global enterprises alike.

Bill Swallow: I think it goes without saying that there’s going to be disruption again. Every single change, whether it’s in the localization industry or not, has resulted in some type of disruption. Something has changed. I’ll be blunt about it. In some cases, jobs were lost, jobs were replaced, new jobs were created. For LSPs, I think AI is going to, again, be another shift, the same that happened when machine translation came out. LSPs had to shift and pivot how they approach their bottom line with people. GenAI is going to take a lot of the heavy lifting off of the translators, for better or for worse, and it’s going to force a copy edit workflow. I think it’s really going to be a model where people are going to be training and cleaning up after AI.

Related links:

LinkedIn:

Transcript:

Introduction with ambient background music

Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations.

Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it.

Sarah O’Keefe: Change is perceived as being risky, you have to convince me that making the change is less risky than not making the change.

Alan Pringle: And at some point, you are going to have tools, technology, and process that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off.

End of introduction

Sarah O’Keefe: Hey, everyone. I’m Sarah O’Keefe, and I’m here today with Bill Swallow.

Bill Swallow: Hey there.

SO: They have let us out of the basement. Mistakes were made. And we have been asked to talk to you on this podcast about AI in translation and localization. I have subtitled this podcast, What Could Possibly Go Wrong? As always, what could possibly go wrong, both in this topic and also with this particular group of people who have been given microphones. So Bill.

BS: They’ll take them away eventually.

SO: They will eventually. Bill, what’s your generalized take right now on AI in translation and localization? And I apologize in advance. We will almost certainly use those two terms interchangeably, even though we fully understand that they are not. What’s your thesis?

BS: Let’s see. It’s still early. It is promising. It will likely go wrong for a little while, at least. Any new model that translation has taken has first gone wrong before it corrected and went right, but it might be good enough. I think that pretty much sums up where I’m at.

SO: Okay. So when we look at this … Let’s start at the end. So generative AI, instead of machine translation. Let’s walk a little bit through the traditional translation process and compare that to what it looks like to employ GenAI or AI in translation.

BS: All right. So regardless of how you’re going about traditional translation, there is usually a source language that is authored. It gets passed over to someone who, if they’re doing their job correctly, has tools available to parse that information, essentially stick it in a database, perhaps do some matching against what’s been translated before, fill in the gaps with the translation, and then output the translated product. On the GenAI side, it really does look like you have a bit of information that you’ve written. And it just goes out, and GenAI does its little thing and bingo, you got a translation. And I guess the real key is what’s in that magic little thing that it does.

SO: Right. And so when we look at best practices for translation management up until this point, it’s been, as you said, accumulate assets, accumulate language segment pairs, right? This English has been previously translated into German, French, Italian, Spanish, Japanese, Korean, Chinese. I have those pairs, so I can match it up. And keeping track of those assets, which are your intellectual property, you as the company put all this time and money into getting those translations, where are those assets in your GenAI workflow?

BS: They’re not there, and that’s the odd part about it.

SO: Awesome. So we just throw them away? What?

BS: I mean, they might be used to seed the AI at first, just to get an idea of how you’ve talked about things in the past. But generally, AI is going to consume its knowledge, it’s going to store that knowledge, and then it’s going to adapt it over time. When it’s asked for something, it’s going to produce it with the best way it knows how, based on what it was given. And it’s going to learn things along the way that will help it improve or not improve over time. And that part right there, the improve or not improve, is the real catch in why I say it might be good enough but it might go wrong as well, because GenAI tends to … I don’t want to say hallucinate because it’s not really doing that at this stage. It’s taking all the information it has, it’s learning things about that information, and it’s applying it going forward. And if it makes an assumption based on new information that it’s fed, it could go in the wrong direction.

SO: Yeah. I think two things here. One is that what we’re describing applies whether you have an AI-driven workflow inside your organization where you’re only allowing the AI to access your, for example, prior translation. So a very limited corpus of knowledge, or if you’re sending it out like all of us are doing, where you’re just shoving it into a public-facing translation engine of some sort and just saying, “Hey, give me a translation.” In the second case, you have no control over the IP, no control over what’s put in there and how it’s used going forward, and no control over what anyone else has put in there, which could cause it to evolve in a direction that you do or do not want it to. So the public-facing engines are very, very powerful because they have so much volume, and at the same time, you’re giving up that control. Whereas if you have an internal system that you’ve set up … And when I say internal, I mean private. It doesn’t have to be internal to your organization, but it might be that your localization vendor has set up something for you. But anyway, gated from the generalized internet and all the other people out there.

BS: We hope.

SO: Or the other content. You hope. Right. Also, if you don’t know exactly how these large learning models are being employed by your vendors, you should ask some questions, some very pointed questions. Okay, we’ll come back to that, but first I want to talk a little bit about pivot languages. So again, looking at traditional localization, you run into this thing of … Basically many, many, many organizations have a single-language authoring workflow and a multi-language translation workflow. So you write everything in English and then you translate. So all of the translations are target languages, they are downstream, they are derived from the English, et cetera. Now let’s talk a little bit about… First of all, what is a multilingual workflow? Let’s start there. What is that?

BS: Okay. So yeah, the traditional model usually is author one language, which maybe 90% of the time is English, whether it’s being authored in an English-speaking country or not, and then it’s being pushed out to multiple different languages. In a multilingual environment, you have people authoring in their own native language, and it should be coming in and being translated out as it needs to be to all the other target languages. Traditionally, that has been done using pivot languages because infrastructures were built. It is just the way it is. It was built on English. English has been used as a pivot language more than any other language out there. There are some outliers that use a different pivot language for a very specific reason, but for the sake of this conversation, English is the predominant pivot language out there.

SO: So I have a team of engineers in South Korea. They are writing in Korean. And in order to get from Korean to, let’s say, Italian, we translate from Korean to English and then from English to Italian, and English becomes the pivot language. And the generalized rationale for this is that there are more people collectively that speak Korean and English and then English and Italian than there are people that speak Korean and Italian.

BS: With nothing in between, yeah.

SO: With nothing in between. Right. Directly. So bilingual in those two languages is a pretty small set of people. And so instead of hiring the four people in the world that know how to do that, you pivot through English. And in a human-driven workflow, that makes an awful lot of sense because you’re looking at the question of where do I find English … Sorry, not English, but rather Italian and Korean speakers that can do translation work for my biotech firm. So I need a PhD in biochemistry that speaks these two languages. I think I’ve just identified a specific human in the universe. So that’s the old way. What is a multilingual workflow then?

BS: So yeah, as we were discussing, the multilingual workflow is something where you have two, three, four different language sources that you’re authoring in. So you’re authoring in English, you have people authoring in German, you have people authoring in Korean and, let’s say, Italian. And they’re all working strictly in their native language, and those would go out for translation into any other target language. It’s tricky because the current model still uses a pivot language, but I think when we talk about generative AI, it’s going to avoid that completely. It’s going to skip that pivot and just say, “Okay, I know the subject matter that you’re talking about and I know the language that you’ve presented it in. Let’s take this subject and meaning and just represent it in a different language and not even worry about trying to figure out what does this mean in English. It doesn’t matter at this point.”

SO: Right. And so I think the one caveat here as we’re looking at this issue is to remember that GenAI in general is going to do better when it has larger volumes of content. And a lot of the generative AI tools are tuned for English. That’s kind of where they started. But it’s also useful to remember that GenAI is math. GenAI doesn’t really have a concept of knowledge or learning or any of these other things. It’s just math. So math is a language of its own, and we should be able to express mathematical concepts in a human language of choice. So there’s some really interesting stuff happening there. Okay. So stepping back a little bit from this, let’s talk about where this is coming from and the history of machine translation in translation localization. Where did we start? And isn’t it true that localization really was one of the leaders in adopting AI early on?

BS: It really was. So way, way, way back, you had essentially transcription in a different language. So people were given a block of text and asked to reproduce it in a different language, and they went line by line and just rewrote it in a different language. Then you start getting into the old-school machine translation or statistical machine translation. What this did was it kept, essentially, a corpus of the translations that you’ve done in the past, and it also broke down the information that you were feeding it into small segments. And it would do a statistical query, taking one segment from what your source said and throwing it out into its memory and say, “Okay, is there anything out here? Was this translated before? And give me a ranking of these results of what was done before.” And essentially, the highest result floated to the top, and it used that. Translators could modify those results over time based on actual accuracy versus systematic or statistical accuracy. But that is forever old. Over the past 10, 15 years, we’ve seen neural machine translation come out, which is getting a lot closer to AI-based translation. So it takes away the text matching and replaces it with more pattern matching. So it’s better at gisting. It will find, let’s say, a 95% match and can fill in those gaps for the most part, or at least say, “Hey, this gets us 95% of the way there. I’m going to put this out over here, and then the translator will essentially verify that translation going forward.” It’s a bit more accurate, but it still relies on this corpus of translation memory that you build over time. And now we’ve got generative AI machine translation, which completely takes everything that was done before, and it doesn’t necessarily throw it away, but it says, “Thank you for all the hard work you did. I will absorb that information and move forward.”

SO: Does it actually say thank you?

BS: It could. It depends on the prompt you use. But I mean, really, you’re looking at a situation where the generative AI model, it uses a transfer learning model to do the translation work. So it takes everything that it knows, applies it to what you feed it for translation, produces an output, learns a lot of things along the way in getting that translation to a point where you say, “Okay, great, thank you. This was good,” and then applies what it learned to the next time you ask. And it keeps doing that and doing that and doing that. On the plus side is that, yes, you can train your generative AI to get really, really, really good if you train it the right way. If someone … And I am not saying it’s malicious or anything, but if you train your GenAI translation model to start augmenting how it translates, then you’ll start getting these mixed results over time because it’s going to learn a different way to apply your request to provide an output.

SO: So the question that I actually have, which I’m not going to ask you to answer because that would be mean, is whether AI is actually storing content in a language, like in English, or is language, in the case of GenAI, just an expression of the math that underlies the engines? You don’t want to tackle that, no. Moving on.

BS: Well, it’s worth poking at, at least, because … Does GenAI actually do anything with the language that we give it now, just for answers? If we’re asking it to write a stupid limerick about a news event, or are we asking, “Summarize this document,” does it care that it’s written in any language? I honestly don’t know.

SO: As meta as it is to ask the question, what is the math that underlies it, the other thing that’s helpful to me, and again, we’re grossly oversimplifying what’s going on, but what is very helpful to me is to think of AI as autocorrect, or autocomplete, actually, on steroids. It’s more than that, but not a lot more. It has just learned that every time I type certain words in my text app, certain other words are likely to follow and it helpfully suggests them. And sometimes it’s right and sometimes it’s wrong, but it’s just doing math, right? Autocorrect learns that there are certain words that, when misspelled or that I do not wish to have corrected, or perhaps it introduces the concept that that word needs to be corrected to the word that I use more commonly, which can be extremely embarrassing. We had some questions about this. We’ve done some prior localization AI conversation, and I wanted to bring in a question that came from one of our audience members. Their question was, “Will we get to the point where we can effectively ask an AI help system a question in a foreign language, the AI system will parse the source language content, and then return the answer in the user’s language? Will translating documentation eventually be no longer necessary?” And what’s your take on that?

BS: Well, I think the answer is yes, and my take is that we are nearly there already. We already have… even apps that you can run on your phone. We have apps that can translate on the fly from verbal language. And I have used them when I travel abroad and I don’t know the language very well, to be able to speak it into my phone and it essentially translates the text for the person I’m trying to communicate with. There are other apps that take a step further and use a synthetic AI voice to read it so that they don’t have to look at my screen. They can just hear what the phone has to say because obviously I’m unable to say it myself.

SO: There’s also a version that does that through the camera. So you point the camera at a sign or a menu, more importantly, and it magically translates the menu into your language while you’re looking at it through your phone, or through your camera.

BS: That has been so helpful.

SO: Yes. Now that is actually a really good example, though, of a place where this kind of translation is hard because there’s very little context, and there’s a tendency in food culture to have very specific terms for things that maybe are not part of the AI’s daily routine. We were talking not too long ago about … What was it? We came up with half a dozen different words in German for dumpling. And we got into a big argument about which one was what and which one is correct for this type of dumpling and all the rest of it. So yeah. The thing I would point out here is that the question was, if someone comes in and asks the AI help system a question in, let’s say, French, but the underlying system is in, let’s say, English, but it would then return French. It’s a very English-centric perspective, to say, “Well, the French people … Our AI is going to be in English, essentially. Our AI database.” And that is a really interesting question to me. Is the AI database actually going to be in English? And maybe not.

BS: Probably not.

SO: I tried this about a year ago with ChatGPT. And you might experiment with this if you speak another language, or combine it with machine translation, which should work as well. I asked ChatGPT a specific question, and I got an answer. Cool. And then I asked the same question again and added, “Respond in, in this case, German.” The answer that I got in German was, obviously, it was in German, step one, which I wasn’t actually sure it could do. But step two, the reply that I got in German, the content was different. It wasn’t just a translated version of the English content. It was functionally a different answer. So it’s like in English, I said to ChatGPT, “What color is the sky?” And it said, “The sky is blue.” And then I said the same thing, “What color is the sky? Respond in German,” and it came back with, “The sky is green.” Now, it was actually did a DITA-related question, which kind of explains what happened here. But what happened was that ChatGPT, even though the prompt was in English, it pretty clearly used German language sources to assemble the answer. And those of you who know that DITA is more popular in the US than it is in Germany would not be too surprised that the answer I got regarding something DITA-specific in German was very much culturally bound to what German language content about DITA looks like. So it was processing the German content to give me my answer, not the English content. Now, if you ask an AI help system, the next question is what’s sitting in that corpus? Because if you ask it a question in French and it has no French in the corpus, then it’s probably going to generate an answer in English and machine translate. But if it has four topics in French and you ask it something in French, it is probably going to try and assemble an answer out of that French content, which could be…

BS: Before it falls back, yeah.

SO: Fascinating, which brings me to my next meta question that we’re not going to answer, which is can we capture meaning and separate it from language? And a knowledge graph is an attempt to capture relationships and meaning. And that can be rendered into a language, but it is not itself specifically English. It’s a database entry of person, which has a relationship with address, and you can say, “Person X lives at address Y,” but that sentence is just an expression of the mathematical or the database relationship that’s sitting inside the knowledge graph. I want to talk about the outlook for LSPs, for localization services providers. What does it look like to be an LSP, to be a translation service provider, in this AI world? What do you think is going to happen?

BS: I think it goes without saying that there’s going to be disruption again. Every single change, whether it’s in the localization industry or not, has resulted in some type of disruption. Something has changed. I’ll be blunt about it. In some cases, jobs were lost, jobs were replaced, new jobs were created. And I think that for LSPs, I think AI is going to, again, be another shift, the same that happened when machine translation came out, when neural machine translation came out, all of this. They’ve had to shift and pivot of how they approach their bottom line with people. GenAI is going to take a lot of the heavy lifting off of the translators, for better or for worse, and it’s going to force a more copy edit workflow. And perhaps, I guess, a corpus editing role or basically an information keeper who basically will go in and make sure that the information that the AI model is being trained on is correct and accurate for very specific purposes, and start teaching it that when you talk about this particular subject matter, this is the type of corpus we want you to consume and respond with, versus someone who actually does the translation work and pushes all the buttons and writes all of the translations. It’s really going to be a model where I think people are going to be training AI and cleaning up after it, essentially. And I don’t know any further than that. I mean, it’s still pretty young. I think also you will see LSPs turning more into consultative agencies with companies, rather than just a language service provider. So they will help companies establish that corpus and train their AI and work with their corporate staff to make sure that they are writing better queries, that they are providing better information out of the gate, and so forth. So I think it’s going to be a complete shift in how these companies function, at least between now and what’s to come.

SO: Yeah. The cost of a really bad translation getting into your database when it was human-driven… this AI thing is going to scale. There’s going to be more and more of it, everything’s going to go faster and faster. And we already have these conversations about AI slop and the internet degrading into just garbage because there’s all this AI-created stuff. And so if you apply that vision to a multilingual world, it’s quite troubling, right? So I think you’re right. I mean, this idea of content hygiene. How do we keep our content databases good, such that they can do all this interesting math processing instead of becoming more and more and more and more error-riddled is really interesting. We started by saying clearly this is a disruptive innovation. Disruptive innovations start out bad, clearly of lower quality than the thing they’re disrupting, but they’re cheaper and/or faster and/or have some aspect that they can do that the original thing cannot. So mobile phones are a great example. They were worse than landlines in every possible way, but they were mobile, right? They were not tethered to a cord in the wall. And then over time, a mobile phone turned into something that really is a computer that is context and location-aware and can do all sorts of nifty things. It doesn’t look at what resemblance it bears to POTS, to plain old telephone service. And we hear people. Oh, I don’t use my phone to make phone calls. Why would I do that? That’s terrible, because we have all these other options.

So from a localization point of view, any organization that is using person-driven, manually-driven, inefficient, fragmented processes is going to be in trouble. And that stuff’s all going to get squeezed out. And I think it’s actually helpful to look at the structured authoring concept and how it eliminated desktop publishing, right? It just got squeezed right out because it all got automated. We do the same thing with localization. I think AI is going to have a similar impact, whether it’s on content creation in any language, that it’s going to remove that manual labor over time. And I think that maybe we’re going to reach a point where content creation is just content creation. It’s not creating content in English so that I can translate it into the target languages. I think that that distinction between source and target is really going to evaporate. It’ll just be somebody created content, and then we have ways of making that available in other languages, and that’s where this is going to go. I’ve talked to a lot of localization service providers recently, and certainly this is one of the things that they are thinking about and looking at, is the question of what it means, to your point, to be a localization service provider in a universe where language translation specifically is automatable, maybe. Okay. Bill, any closing thoughts before we let you go here?

BS: I think this is a good place to end this one.

SO: We’ll wrap it up, and they will come and take away our microphones and put us back in the corner. Good to see you, as always.

BS: Good to see you.

CC: Thank you for listening to Content Operations by Scriptorium. For more information, visit scriptorium.com or check the show notes for relevant links.

The post AI in localization: What could possibly go wrong? (podcast) appeared first on Scriptorium.

Is AI really ready to generate your training materials? In this episode, Sarah O’Keefe and Alan Pringle tackle the trends around AI in learning content. They explore where generative AI adds value—like creating assessments and streamlining translation—and where it falls short. If you’re exploring how AI can fit into your learning content strategy, this episode is for you.

Sarah O’Keefe: But what’s actually being said is AI will generate your presentation for you. If your presentation is so not new, if the information in it is so basic that generative AI can successfully generate your presentation for you, that implies to me that you don’t have anything interesting to say. So then, we get to this question of how do we use AI in learning content to make good choices, to make better learning content? How do we advance the cause?

Related links:

LinkedIn:

Transcript:

Introduction with ambient background music

Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations.

Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it.

Sarah O’Keefe: Change is perceived as being risky, you have to convince me that making the change is less risky than not making the change.

Alan Pringle: And at some point, you are going to have tools, technology, and process that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off.

End of introduction

Alan Pringle: Hey everybody, I am Alan Pringle, and today I’m talking to Sarah O’Keefe.

Sarah O’Keefe: Hey everybody, how’s it going?

AP: And today, Sarah and I want to discuss artificial intelligence and learning content. How can you apply artificial intelligence to learning content? We’ve talked a whole lot, Sarah, about AI and technical communication and product content, let’s talk more about learning and development and how AI can help or maybe not help putting together learning content. So how is it being used right now? Let’s start with that. Do you know of cases? I know of one or two, and I’m sure you do too.

SO: Yeah. So the big news, the big push, is AI in presentations. So how can I use AI to generate my presentation? How can it help me put together my slides? Now, the problem with that from our point of view, for those of you that have been listening to what we’re saying about AI, this will be no surprise whatsoever, I think this is all wrong. It’s the wrong strategy, it’s the wrong approach. If you want to take AI and generate an outline of your presentation and then fill in that outline with your knowledge, that’s great, I think that’s a great idea. Also, if you have existing really good content and you want to take that content and generate slides from it, I don’t have a problem with that. But what’s actually being said is AI will generate your presentation for you. If your presentation is so not new, if the information in it is so basic that generative AI can successfully generate your presentation for you, that implies to me that you don’t have anything interesting to say.

AP: And you’re going to say it with very pretty generated images and a level of authority that makes it sound like there’s something that’s actually there when it’s not.

SO: Oh, yeah. It’ll look very plausible and authoritative and it will be wrong, because that’s how this generative stuff-

AP: Or not even wrong, surface-skimmy, just nothing of any real value there.

SO: Yeah. So then, we go into this question of, how do we use AI in learning content to make good choices, to make better learning content, how do we advance the cause?

AP: Well, there’s that one case where we have done it, because we have our own learning site, LearningDITA.com, and we were trying to think about ways to apply AI to our efforts to create courses, to tell people how to use the DITA standard for content. And I think you and I both agree, one of the strengths of artificial intelligence is its ability to summarize and synthesize things, I don’t think that’s controversial. So if you think about writing assessments from existing content in a way that’s summarizing, so one of us suggested to our team, why don’t y’all try that and see what these AI engines can do to generate questions from our existing lesson content. And then, of course, we suggested that they—the people who were creating the courses—review them. So our folks reviewed them, and I think some of the questions were actually quite usable, decent.

SO: And some of them were not.

AP: True, this is true.

SO: But the net of it was they saved a bunch of time, because they said, “Generate a bunch of assessment questions,” they went through them, they fixed the ones that were wrong, they improved the ones that were maybe not the greatest, they got a couple that were actually pretty usable. And so, it took less time to write the assessments than it would’ve taken to do that process by hand, to slowly go through the entire corpus to say, “Okay, what are the key objectives and how do I map that to the assessments?” So that’s a pretty good example, I think, of using generative AI, as you said, to summarize down, to synthesize existing content. On the LMS side, so when we start looking at learning management systems and how the learning content goes into the LMS and then is given or delivered to the learner, there are some big opportunities there, because if you think about what it means for me as a learner, as a person taking the course, to work my way through course material, maybe the assumptions that the course developer made about my expertise were too optimistic. I’m really struggling with this content, it’s trying to teach me how to use Photoshop and I am just not good at Photoshop. There’s this idea of adaptive learning, this is not an AI concept, the idea behind adaptive learning is that if you’re doing really well, it goes faster. If you’re struggling, it goes deeper, or maybe you do better with videos than you do with text, or vice versa. It’s that adapt to the learner and to the learner’s needs in order to make the learning more effective. Now, if you think about that, that is a matter of uncovering patterns in how the learner learns and then delivering a better fit for those patterns. Well, that’s AI. AI and machine learning do a great job of saying, “Oh, you seem to be preferring video, so I’m going to feed you more video.” Now, we can do this by hand or we can build it in with personalization logic, but you can also do this at scale with AI and machine learning. So there are definitely some opportunities to improve adaptive learning with an AI backbone.

AP: I think it’s worth noting at this point, when you’re talking about gathering the data to make, I hate to, I’m going to personalize AI, so it can make these decisions or do the synthesis, there’s got to be intelligence that’s built into your content, and that goes all the way back to the content creation, going back from the presentation layer, back to how you’re creating your content. And again, this loops back, in my mind, to the idea of building in that intelligence with structured content, that is your baseline.

SO: Yeah. I know we’re just relentless on this drum of you need structured content for learning content, but it’s because of all these use cases, because as you try to scale this stuff, this is what you’re going to run into. I also see a huge opportunity for translation workflows specifically for learning content. So if you look at translation and multilingual delivery, there’s a lot of AI and machine learning going on in machine translation. So now, we think a little bit about what that means for learning content, and of course, all of the benefits that you get just in general from machine translation still apply, but the one that I’m looking at that I think would be really, really interesting to apply to learning is learning has a lot of audio in it, audio and video, but specifically audio, and audio typically is going to be bound to a language. You’re going to have a voiceover, you’re going to have a person saying, “Here’s what you need to know, and I’m going to show you this screenshot,” or, “I’m going to show you how to operate this machine.” And so, you’ve got audio and potentially captions that are giving you the text or the audio that goes with that video. Okay, well, we can translate the captions, that’s relatively easy, but what about the voiceover? And the answer could be that you do synthetic voiceovers. So you take your original, let’s say, English audio and you turn it into French, Italian, German, Spanish or whatever else you need, but you synthesize the voice instead of re-recording. Now, is it going to be as good as a human, an actual human person who has expression and emotion in their delivery? No. Is it better than the alternative where you don’t provide it in the target language at all? Probably, yes. And when we start talking about machines, “Here is how to safely operate this machine,” the pretty good synthetic voice in target language is probably better than, “Here it is in English, deal with it,” or, “Here it is in English with a translated caption in German, but no audio.” I think that’s what we’re looking at is, is the synthetic audio good enough that it will improve the learner experience, and I think the answer is yes.

AP: I’m turning this over in my mind, and there’s part of me that’s very resistant to the idea of these synthesized voices. For example, and this is bias on my part, when I am downloading audiobooks from the library, they now, in the app that I use that’s connected to the local library, a lot of the narration, it will say, “This is an AI-generated voice.” I tend to avoid those, I do, because sometimes the inflection’s a little odd, there’s no personality there. However, I can buy that having that slightly robotic-esque voice in another language is better than not having it at all, I can buy that.

SO: Right. And I think the audiobooks that we listen to for fun are different than I need to figure out how to use this machine without hurting myself, those are different, and I don’t need a… It wouldn’t hurt. I don’t need a personable obviously human voice to voiceover the video that helps me figure out how to use this thing on the factory floor. I wouldn’t object, but I would prefer to get something in my language. That’s really the key, because when we start asking the question, the question is less, would you prefer a really good artistic performance voiceover versus a robotic voice… That’s what you’re getting from the library, you’re saying, I am not going to consume entertainment content that is like this, and I think a lot of people are onboard with that. But what about technical product and learning content that you need? You’re not making a choice that this is something I want to do in my downtime, but rather, if I can’t figure out how to do this, bad things will happen.

AP: Yeah. There is a legitimate use case there, and they’re two different things, and I do think, based on some of the synthetic voices I’ve heard, they are getting better, quite better, and sounding a little more realistic as well.

SO: Right. We’ve already experimented with this. We have a podcast where we actually generated, it was a synthetic voice, but it was based on a person’s voice print. So it wasn’t fake AI, it was fake AI voice, but it was fake AI voice generated off of a specific person. The audio is quite good. Every once in a while between paragraphs, it shifts weirdly as you’re listening, as a new thought is introduced, and it shifts in ways that a human would not, but all in all, I thought it was pretty acceptable. So I think that what I’m trying to say in an extremely long-winded way is that when you have scalability issues in your content production, learning content or otherwise, AI has the potential to help you with productivity across multichannel workflows with repurposing content from it’s the learning content versus it’s the assessments, it’s in language A versus language B, it’s audio, it’s video. There are things that we can do there to use the AI tools for productivity to support these workflows and scale them, and to your point, and therefore, we need underlying structured content. We can’t do this with slapped-together one-off formatted mess.

AP: Yeah. The intelligence has to be built in at the very foundation, and that is when you are creating the content. That intelligence really can’t be a layer that’s put on when you transform things or you connect to an LMS, it’s not a presentation layer thing. The presentation layer needs to pull that intelligence from your source content. Again, this is why you need structured content, the metadata built in, to help drive the way you transform and distribute your learning content.

SO: Yeah. I’m, again, very skeptical of GenAI in the process of generating net-new content, new information, nobody’s ever written it before, it’s a new product, it needs to be explained, taught, whatever. Maybe an outline, this is what a typical intro course looks like, now go fill in the details, okay, maybe even a first draft, especially if product A is based on product B, or I guess the other way around. But our world is structured content, obviously, but also our world is content where it matters that the content is accurate, because when the content is wrong, bad, bad things happen, people get hurt, people die, companies get shut down for compliance reasons, that type of thing. So the content has to be accurate, and at the end of the day, it’s actually quite difficult to get GenAI to gen accurate content. That’s not what it does; that’s not its function. So I’m very interested in applying AI to various product and content roadmaps to enable productivity, to enable new deliverables, to enable new synthesis summaries, et cetera, but I’m very, very worried about what happens if you apply it on top of bad content or you apply it to the wrong use case in an effort to just get your stuff for free, essentially.

AP: So what I’m hearing in summary is that content creation for learning, AI is probably not a good fit now. To support you and help you possibly develop on the edges of that content or give you outlines and ideas, and also to augment and support delivery channels, it could be helpful. So it’s a support mechanism for the development of the content, distribution of the content, but not necessarily for the direct creation of that content.

SO: Yeah, I think that’s fair, and I think that’s where we land. I’d be quite curious to hear from our listeners, what they’re doing with this and where they’re going with it.

AP: And I’m sure people are having the same struggles right now over the best way to apply it. But I think right now, as of the moment that we’re recording this, AI is in no way ready for prime time to basically take the place of a learning content person. It should be there to support them, not to replace them.

SO: Yeah, for meaningful content.

AP: Exactly.

SO: And if it’s not meaningful, what are you even doing?

AP: Right.

SO: Well, that’s cheery, okay.

AP: And on that very cheerful note, we’re going to wrap up. So thank you, Sarah. And folks, do get in contact with us to let us know how you’re using AI, because that is of great interest to us. So thank you. Thanks, Sarah.

SO: Thank you. And maybe let us know how you’re being made to use AI.

AP: That too. Thanks, everyone.

Conclusion with ambient background music

CC: Thank you for listening to Content Operations by Scriptorium. For more information, visit Scriptorium.com or check the show notes for relevant links.

"*" indicates required fields

The post Help or hype? AI in learning content appeared first on Scriptorium.