Content + AI

Rebecca Evanhoe

Rebecca Evanhoe practices, teaches, and writes about conversation design, a key UX practice that is taking on fresh importance in the age of chat-based AI applications.

Since the publication of her book Conversations with Things (co-authored with Diana Deibel) three years ago, the tech and media worlds have fundamentally transformed, but the conversation-design principles that she teaches remain as relevant as ever.

We talked about:

the conversation design and UX writing courses she teaches

reflections on the book she co-wrote several years ago, "Conversations with Things" and the changes in the conversation-design world since

how the focus on principles in

a framewwork set out in their book that helps designers decide on whether or not and how to ascribe personality to a chat agent

her identification as a UX designer

how she's incorporating LLMs into her course curricula

her take on the misappropriation of the term "prompt" in new practices called "prompting" and "prompt engineering" and their divergence from traditional use in the conversation design field

the differences in the conversation designer role in the LLM world compared with NLP

the linguistic concept of "conversation repair" and how it manifests in "bot land"

how to adjust confidence level in conversation design

how intent classification in NLU works

her preference for humans and human conversation

the importance of including people with a humanities background in conversation design

the ongoing importance of humans in the content and conversation design process for our ability to think strategically about how to maximize the success of conversational technology

Rebecca's bio

Rebecca Evanhoe is an author, teacher, and conversation designer. With degrees in chemistry and fiction writing, she's passionate about how interdisciplinary thinking can combine arts, humanities, sciences, and tech. She teaches conversational UX design as a visiting assistant professor at Pratt Institute, and co-authored Conversation with Things: UX Design for Chat and Voice (Rosenfeld Media, 2021).

Connect with Rebecca online

LinkedIn

Video

Here’s the video version of our conversation:

https://youtu.be/xJkB03uH8ek

Podcast intro transcript

This is the Content and AI podcast, episode number 19. We're all talking to computers a lot more these days - telling Alexa to set a timer, asking Midjourney to create an image for a party invitation, or prompting ChatGPT to draft an outline for a slide deck. Rebecca Evanhoe is an expert on the interaction design practices that guide these conversations. Three years ago, her book "Conversations with Things" set out a principles-based approach to conversation design that remains super-relevant in the age of large language models.

Interview transcript

Larry:

Hi everyone. Welcome to episode number 19 of the Content and AI podcast. I am really happy today to welcome to the show Rebecca Evanhoe. Rebecca is really well known in the conversation design world. She's a conversation designer. She's the co-author of the really excellent book Conversations with Things that came out a few years ago, and she teaches conversation design and other kinds of design work at Pratt University in New York. So welcome to the show, Rebecca, tell the folks a little bit more about what you're up to these days.

Rebecca:

Yeah, hi Larry, it's nice to be back. Yeah, these days I am teaching, I think you said conversation design, and specifically this semester I'm teaching a class in UX writing, which I love because it doesn't matter what kind of writing I'm teaching, it's like a chance to think about language and celebrate how cool language is with my students. And yeah, I've been teaching, I am doing some work at a cool place that I won't get into here. But yeah, it's been a really interesting couple of years.

Larry:

Yeah, because we last talked right before your book came out, I think it was maybe a few months before the book came out. And since then, I mean conversation had been a thing. I had talked to Phillip Hunter and several other content designers before I had you and Diana on the show, but it seems like I'm going to guess that more has happened in the last four years than in the four years before you wrote the book. Is that accurate?

Rebecca:

I think that's definitely accurate. Yeah, our book came out in April of 2021, and I think that ChatGPT became publicly available in November of 2022. So our book has been amazingly well received, tons of enthusiasm. It really seems to be sticking around and people are finding it useful. But if you control F and search our book, there is not one mention of the term large language model. And I think there's only one mention of natural language generation.

Rebecca:

It's been interesting to look at our book through the lens of the fact that technology keeps changing. And I think, and other readers think as well that it's based enough in principles that it really applies to any conversational technology, or at least that's the hope. And when I think back about the things that have happened in the last few years, when the book came out, I remember people kind of wanting us to put a couple things in the book that we didn't.

Rebecca:

People really wanted more information about how you build an Alexa Skill or a Google Action. Those were very visible at the time. People also wanted us to put a list of prototyping tools for conversations into the book. And we didn't, and I think things like that future-proofed it a little bit, because Alexa Skills and Google Actions... Like Google Actions aren't around anymore, Alexa Skills are very much de-emphasized. And a lot of the prototyping tools that we had a few years ago were acquired or were kind of sunset. So yeah, I think we made some lucky decisions to future-proof it. But certainly it doesn't have a mention of LLMs, which is-

Larry:

Well that's really... I got to say, this is super interesting because I remember from, we talked on the Content Strategy Insights podcast about this, that you and Diana both emphasized principles. And I don't know that you specifically stated that, but in retrospect it's like, yeah, that's way better than focusing on any specific technology or practice. Can you talk... I remember you covered those really well in the book. But is it possible to do a quick overview of some of the guiding principles? And maybe more to the point, how are they helping you through the arrival of LLMs and generative pre-trained transformers and all that stuff?

Rebecca:

Absolutely. I think that one of the concepts from the book that has become even more important today is the idea... And in our book we call it level of personification, and it's in the personality chapter. So I think a lot of people are thinking more about personality design, but also specifically how much of a character, how much of a mind the AI is sort of presenting itself as.

Rebecca:

So is it presenting itself as a fully realized character that's your friend and it refers to itself as I, or is it behaving more like a machine? So the example that I always give is if you have a remote control where your voice is the input, it doesn't need to be named Sandy, and it loves, Thanksgiving is its favorite holiday, and... It doesn't need to be a character in mind. So thinking through that spectrum really for any AI experience you're creating, I think is really important. How much of a person should it present itself as? I think that becomes a lot more visible.

Rebecca:

And an example that I would give for the LLM world, it's like, if you talk to chat GPT or Claude, those bots use I. And you can ask them a little bit about themselves and they'll tell you, they'll generally clarify like, oh, I'm an AI so I don't have feelings, but I can describe feelings or talk to you about feelings, stuff like that. But then there are other LLMs that they don't have any personification at all.

Rebecca:

So for example, Perplexity AI is a platform that is an LLM and you can talk to it and it does all the LLM-ey stuff, meaning you could ask it to summarize, you can ask it to give you bulleted lists, you can ask it to imitate a turn-taking conversation with you, but it doesn't really present itself as a character at all. And I think those kinds of decisions are still very much ones that conversation designers should be involved in, because that level of personification really impact user expectations, how they're going to behave toward it, and then how successful those interactions are going to be.

Larry:

Yeah, that's really interesting. How do you make that decision? Because I can picture making the wrong decision for good reasons. Oh, we like our customers, we want to be close to them, so we're going to act like their friend, where it would probably be more appropriate in a business setting to not be that way. Are there sort of guidelines around that, how you decide the kind of personality?

Rebecca:

Yeah, I mean in our book there's sort of a framework that walks through a lot of the facets of it. But generally I would say I think people over-personify these interactions. They think that having a character must be more interesting and fun, and they forget that people want their thing fixed, they want their task completed, they want their problem solved. And people also forget that lots of people are very happily solving these problems already through an app, through a website. People do like to solve their own problems, and conversations are not necessarily easier and more efficient unless they're designed to be so.

Rebecca:

So yeah, I think one of the things that we think through in the framework is first defining interaction goals that are independent of the conversation. So an example that I always use is, if you're making a voice bot that takes orders for a drive-through,...

Andy Crestodina

Andy Crestodina has been developing high-quality content for his business customers at Orbit Media for more than 20 years.

As they have incorporated AI into their workflows at the agency, Andy has discovered that the best use of these new tools is to improve the quality of their content and service offerings rather than simply doing more.

We talked about:

his work as co-founder and CMO at Orbit Media

how he uses AI to do audience research, develop personas, and address their needs through gap analysis

his playbook for querying and validating information that AI generates for him: prompt, response, edit

how they manage prompts at Orbit Media

how their business operations are evolving to incorporate AI practices into their operations

how he uses AI in his marketing analytics

how the comprehensiveness of coverage that AI brings to his content helps with conversion

the essential skills that persuasion copywriters need to develop to work effectively with AI

his concern that some LLMs may be getting worse, not better

the importance of setting aside a focus on how to be faster and instead focus on how to be better - to focus on quality over quantity of content

Andy's bio

Andy Crestodina is the co-founder and Chief Marketing Officer of Orbit Media, an award-winning 50-person digital agency in Chicago. Over the past 23 years, Andy has provided digital marketing advice to 1000+ businesses.

Andy has written 500+ articles on content strategy, search engine optimization, visitor psychology, analytics and most recently, AI. These articles reach more than three million readers each year. He’s also the author of Content Chemistry: The Illustrated Handbook for Content Marketing.

Andy gives up to 100 webinars and presentations per year and is a frequent repeat speaker at many of the top national marketing conferences.

Connect with Andy online

LinkedIn

Orbit Media

YouTube

Video

Here’s the video version of our conversation:

https://youtu.be/yNZiusTV5kY

Podcast intro transcript

This is the Content and AI podcast, episode number 18. Creating content that gets found by Google and then persuades potential customers to act is a core competency for modern marketers. Andy Crestodina and his colleagues at Orbit Media, the agency he co-founded 20 years ago, have built websites for hundreds of businesses and created content for them that helps turn their prospects into customers. Andy uses AI extensively in his work now. His top finding? Focus on how AI can improve the quality of your work, not just your productivity.

Interview transcript

Larry:

Hey everyone. Welcome to episode number 18 of the Content and AI podcast. I am really happy today to welcome to the show Andy Crestodina. Andy is the co-founder and CMO at Orbit Media. It's a Chicago based agency that does website development and a lot of other marketing stuff. So welcome, Andy. Tell the folks a little bit more about what you're doing these days.

Andy:

Sure, Larry. Well, thanks for having me. 20 however many years ago, 2001, co-founded an agency. An agency totally focused on the website itself. So we build sites and we improve them forever after doing optimization work, both search and conversion optimization. It's a 55-person firm that we grew strictly from organic and content marketing. So I'm someone that you might see if you go to an event like Content Marketing World, Social Media Marketing World, MozCon. I speak at a lot of search events and analytics events, but I'm an old-school content marketer who's built an agency focused on websites.

Larry:

Nice. Yeah. We must have run into each other somewhere along the line because I ran in that world a lot back around that same time, the early 2000s. But one of the things that's so interesting ... So we've seen this evolution together and seen a lot of the ... There's always been a lot of drudgery associated with our work and a lot of intellectual work involved and you're really excited about, and have experimented as much with, the new AI tools. Let's start with the customer and the audience analysis stuff that you do. That was really interesting to me when I read about that. Can you talk a little bit about that?

Andy:

Sure. If you write a prompt that says draft a blog post of 2,000 words that talks about supply chain ... You're going to get something pretty boring. It's going to be inherently undifferentiated. I joke that AI stands for average information. AI ate the internet. Literally the Common Crawl is 85% of the internet. We know that ChatGPT was trained on the Common Crawl. And it comes back and just gives you vanilla. It tastes like water. Of course it's boring. It's not for anybody. It's generic. All you did was say, write me a blog post. So all of my most successful adventures in AI, and these are daily, begin by teaching it or training it on your target audience. Now, if you've got battle tested, ideal client profiles or marketing personas, you can upload them and it will read them.

Andy:

You can also write prompts that will do it. Create the persona of a job title within an industry, at a company size, in a geography with an objective and a challenge and then tell me their hopes and dreams, tell me their pain points, their frustrations, their fears, and their decision criteria for selecting a company like mine. It's going to write you a persona. It will be incorrect. Of course, AI is not accurate. Don't ever expect the AI to be accurate. Go fix it and prove it, validate. And now once you've improved that persona and you believe in it and it looks good, now ask it to write an article for that persona or to draft an outline or to write a headline or to write a social post or to research a keyword or to suggest an influencer, or to do whatever it is you want to do. It's going to be far, far better results if you began that conversation with AI by teaching it who you're talking to. It's absurd for marketers to believe that they're going to get any good response without focusing on the audience. It's weird, right? I think it's weird.

Larry:

It is. And it's like that thing that anybody ... I'm sure you've done a lot of presenting over the years. You always analyze your audience and figure out what they want first. What you're saying reminds me that so much of the fuss the last year plus since ChatGPT-3 was introduced, has been about the generative capabilities of AI. We were talking a little bit before we went on the air and you were like, "Nah, it's not about efficiency and time saving. It's more about better stuff." And like you just said, it sounds like you're getting ... How do you feel about the personas that you're developing now compared to what you're doing before you had ChatGPT to query?

Andy:

Well, if I'm on a conversation with a client or a prospect or a friend and they say, "Hey, check out this thing. Is this good?" I've done digital strategy forever and I've been part of the planning process for more than a thousand projects and I'm an SEO and a conversion person and persuasion copywriting nerd. So people are frequently asking me to evaluate something they made. But it's really hard for me to quickly understand their audience. So instead, if I just begin with a persona prompt and together with the person I'm talking to, we get to where we believe, yeah, that is my buyer. Yeah, that looks like them. That feels right. Okay, good. Now I'm going to copy and paste in that thing that you wanted me to review and I'm going to have the AI tell me what it's missing.

Andy:

AI powered, persona driven gap analysis on any page on your website. It will immediately tell you you failed to meet your audience's information needs, or you did not address an important objection, or there's a critical unanswered question with your persona after reading this copy. Those are things it's very hard for a human brain to do. To look at something and say, what's not there. Human brains are simply not good at doing that. AI is amazing at doing that, but only if you train it on the audience first. At that point, now everyone has a sense for it. And do we agree? Do we just automatically assume it's correct? No. Here I joke, AI stands for another input. Earlier I jokingly said, AI stands for average information, which is what it does. If you just ask for general prompts, you're lazy prompting, it's going to give you back average information.

Andy:

But now I'm actually using it as a mini-consultant or research assistant or once I trained it on the persona, giving it a piece of copywriting and it's giving me another perspective. Do I have to take it? No. Is it useful? Maybe. But it was a fast exercise that put me in the mindset of my audience and it's got me looking at a key piece of content, a sales piece, a service page or a product page or a sales page. That exercise, sure, it is fast. I don't love it because it's fast. I love it because it's going to help me generate more leads. But that exercise is 15 minutes, less, and the improvement that you might make from that ... Dammit. I forgot to mention this important thing. Will be a durable improvement. Go fix your homepage and it'll be a better homepage for the next 10,000 visitors you'll have over the lifespan of that page. To me, this has been the most successful use of AI. Starts with the persona, give it a piece of copy, have it do gap analysis, and then just take it or leave it. But you have an opportunity now to do better, you could say, just conversion copywriting.

Larry:

Yeah. You're reminding me now that the common and ubiquitous modern affliction is attention when none of us have it anymore. We're incapable of paying attention for long periods of time. But these machines are just like, once they know what they're looking for, they're just on the job. They're not picking up their phone and looking at it. So their attention to detail, I get that....

Markus Edgar Hormess

Markus Edgar Hormess offers this advice: "Never prompt alone."

Markus was working with AI long before the current wave of excitement. He experimented with early versions of ChatGPT and quickly identified new opportunities to collaborate with both his human colleagues and his new AI coworkers.

He's currently building a community - Teaming with AI - to study and share these new practices and to explore the future of teamwork in the age of AI.

We talked about:

his background in strategic prototyping and how he's applying it in his Teaming with AI initiative

his first exploration of AI, in 1986

one his first applications of current AI tech, a use of ChatGPT-2 to accelerate service design prototyping activities

his work and experimentation on ways to engage AI tools as collaborators on design teams

how to consume research on AI, but also the importance of getting out in the field since research develops more slowly than professional craft

his insight that you should "never prompt alone" so that you and your collaborators can eliminate bias and get better answers

some of the opportunities that AI creates for real-time research and accelerated implementation of research insights

how important it is "to put people in the center of this"

the benefits for design practitioners of diving in and experimenting with AI tools, always with collaborators

Markus's bio

Markus Edgar Hormeß is a well-known consultant, practitioner and educator in the field of service design and design thinking. In his daily work, Markus helps organizations tackle complex business problems and make team cultures more agile and human-centered. The focal point of his work is strategic prototyping, where he constantly pushes the boundaries of what a dedicated team can achieve with limited resources.

Markus is a strong believer that we should break down the perceived boundaries between technology, design and business – and that cheap experiments and prototypes are efficient tools to move your company, your strategy, your team, or your project forward. Based on this mindset, he has shaped multi-year programmes to help multinationals shift towards a more hands-on, pragmatic and effective approach to customer experience and innovation.

Markus has a passion for good design, human technology, practical experiments, authentic services, and playfulness in all things. He is co-Founder of WorkPlayExperience, a service innovation consultancy which helps organizations worldwide change how their staff, partners, and customers work together – and – how they can strategically discover and create new products and services. His practice builds on his experience of service design and business consulting, and on his background in theoretical physics.

In 2010, Markus co-initiated the world’s biggest service innovation event: the award-winning Global Service Jam. This was soon followed by the Global Sustainability Jam and the Global GovJam, and Markus has been a leading figure in establishing the culture of experimentation and prototyping which Jammers worldwide call “DoingNotTalking”.

Markus co-wrote “This is Service Design Doing” and “This is Service Design Methods”, top-selling books which have become the standard reference books for many practitioners and academics. He teaches service design, innovation, and sustainability at various universities globally, and is adjunct professor for service design thinking at IE Business School in Madrid.

In 2023 he co-initiated the Teaming with AI conference and community. His growing interest centers on how AI influences our approach to teamwork and collaboration, as well as the broader impacts on innovation and the development of strategies that are resilient in the face of future challenges..

Connect with Markus online

LinkedIn

Teaming with AI website

Video

Here’s the video version of our conversation:

https://youtu.be/HlHhpsr2lW4

Podcast intro transcript

This is the Content and AI podcast, episode number 17. As AI tools arrive in our workplaces, we're discovering that this isn't just another technology adoption cycle. The generative nature of tools like ChatGPT permits rapid iteration on ideas and quicker learning about their impact. For a prototyping strategist like Markus Edgar Hormess adding these AI agents to his service-design teams has been a boon, letting him and his colleagues collaborate and experiment in ways they couldn't have imagined just a few years ago.

Interview transcript

Larry:

Hi, everyone. Welcome to episode number 17 of the Content + AI podcast. I am really happy today to welcome to the show Markus Edgar Hormess. I first met Markus a year ago at a service design workshop in Amsterdam, and we've been talking ever since about getting him on the show. So it's great to finally have you here, Markus.

Larry:

Markus, he's one of the co-authors of the book This is Service Design Doing. He's real active in the service design community and in that world he's really focused on strategic approach to prototyping, which is what we first wanted to talk about. And then AI came along. So we're on the Content + AI podcast. So anyhow, welcome, Markus. Tell the folks a little bit more about what you're up to these days.

Markus:

Hey, Larry. Thank you for having me. Yeah. So you mentioned it, so I'm super interested in strategic prototyping and prototyping in all kind of aspects. And when this whole wave of AI came about, we thought, "There is no books, there is no papers that tell you how to do this, so we need to prototype our way into this new world," and that's why we set up an initiative, which is called Teaming with AI, where we focus on the impact AI tools have on the way we collaborate in teams. So a small group of people that have a common goal, that trust each other, hopefully, and try to make something happen in the world. Might be nonprofit, for-profit, wherever you are.

Markus:

And so we set up a couple of events, a little Unconference early last year and one in the middle of the year. Then we started writing a white paper about this. This is about to be published soon, so hopefully we get some conversation about this. But all of this is really about giving a space, a play space for people that are interested to explore what is happening there. Only a few people actually focus on that team aspect. That's why we have a strong focus on that, because you know it, service design, what we always say, it's what is the key skill that you have to have in service design? That's facilitation. That's working with a group, whether you're part of that group or if you're facilitating a different group. And now one part of that group is AI and how does it change things? It changes it, and it doesn't change it in other parts, but certainly a lot of shift going about.

Larry:

Yeah. There's two things in there that are really interesting to me. One is that we're all still humans and we're going to be throughout this, whatever this AI thing turns out to be, but also the fact that, I feel like, you're living in the future a little bit, because when I met you a year ago, you were already deep into this and really exploring it. And now you're way into this collaborative paper and you've given it a lot of thought and you're going to be providing these materials that you just said didn't exist yet. So thank you for that. But tell me a little bit about when and how did you first get interested in AI? And how does it fit in specifically with your... Because you first came to my attention or you really stood out in that workshop as the prototyping guy. And so talk a little bit more about that. Yeah.

Markus:

Yeah, sure. I gave this a bit of thought, and then I remembered something, that back in 1986, I think I was in seventh or eighth year in school, I did a big presentation about the state of AI at the time. That was during one of these first waves, big promises in AI, "We're going to fix this by the end of the decade," and it never happened. But that was still when there was this kind of, "Oh, we can maybe do this." So this was time of programming languages like LISP and stuff. I think that was where I got curious. Then I forgot about it for a long time. And then just after I finished university, I started to work at the Bavarian Research Center for Knowledge-Based Systems, which basically was a spinoff of the chair of AI at the LMU University. But that was, again, during a time where we were in niche use cases. The machines weren't fast enough to do the big stuff that we can do today. But that's where I learned that, yeah, niche use cases can be useful and they still are to this day.

Markus:

And then fast-forward, me getting into service design and innovation. And three or four years ago, no, three years ago now, when GPT-2 came out, it was accompanied by a wave of tools that would allow you to come up with better marketing texts. And that's where we pick them up and use them in prototyping. Because in service design, if you design a new customer experience or service, how do you make this tangible, right? And one super simple way is to create a little advertisement for a new idea that doesn't exist yet. It's easy to test because people know the format. So it's a really good way to test the waters if people like that or value that way you're trying to sell.

Markus:

And using these tools, there's this little, "Oh, give me 10 variations of a Facebook advertisement or a Google ad." And then the teams would just use these tools within our workshops. We get these 10 and then curate the ones where it's, "Oh, yeah, that fits what we thought." And they could go faster, which is, I'm not obsessed by faster. There is a caveat there, but within the design process, being able to get something faster means you can iterate more, and that means you can learn more. So you can reflect on, "Oh, what does this do?...

Dan Porder

A few years ago, Dan Porder was teaching poetry to university students. Now he's at IKEA training large language models to generate useful, usable content for user experiences.

He's picked up new skills along the way, like Python programming, but much of his work still relies on well-established content and design crafts like content strategy and inclusive design.

We talked about:

his role as a senior content designer at IKEA, where he focuses on AI

some of his early experiments in composing and evaluating poetry

his longstanding interest in AI and the development of his tech skills

how content designers can leverage their skills to work in AI

his perception that there is currently more opportunity than threat to content professionals in the AI world

the make-up of the cross-functional teams he works with: data scientists, engineers, developers, content people, designers, subject matter experts

how to brief and guide generative AI to get the outputs your users need

how writing abilities prepare content designers to do prompt engineering

the stack of data and technology that underlies AI and the orchestration mechanisms that connect them

some of the tools he uses in his AI design practice

the role of data in content design for generative AI

the importance of staying aware of bias in training data and always wearing your inclusive design hat

the role of explainability in AI ethics

the importance of knowing how to ask data scientists and engineers questions that reveal as much as possible the inner workings of the "black box" in which AI content is generated

his take on democratization opportunities that arise with the arrival of AI tech

Dan's bio

Dan Porder is a Senior Content Designer and Content Engineer at IKEA. His recent work focuses on the intersection of AI, structured knowledge, and experience design. Outside of work, he runs an international writing community.

Connect with Dan online

LinkedIn

Video

Here’s the video version of our conversation:

https://youtu.be/VFXLG4h6ylE

Podcast intro transcript

This is the Content and AI podcast, episode number 16. AI is quickly changing the way content designers work. New content duties are emerging that require fresh skills, but at the same time traditional skills like content strategy are becoming more important. In his work as a content designer at IKEA, Dan Porder has developed new skills, like Python programming, and has applied the writing skills he perfected as a poetry teacher as well as the inclusive design practices he developed earlier in his content design career.

Interview transcript

Larry:

Hey everyone. Welcome to episode number 16 of the Content and AI podcast. I'm really happy today to welcome to the show Dan Porder. Dan is a senior content designer at IKEA, where he's currently focusing on AI stuff, and his title is content designer, but he is really more of a content architect. So welcome to the show, Dan. Tell me a little bit more about your AI and content adventures.

Dan:

Hey Larry. Thanks for having me on. Yeah, maybe I could just start by giving a little bit of background. I think at heart, despite what I'm doing now, I think of myself as a writer, and that's been my life's focus since I was young. Writing poetry, writing fiction. I did my bachelor's in English literature and later did a masters, masters of fine arts, actually, in poetry. Some of it was more on a conceptual side, thinking of language as data. So there was some unusual experiments in the tech world even then for me. Using Google data to create poems. So imagining Google queries as a representation of the collective zeitgeist, and how can we leverage that data to create meaning in poetry? Or using NLP to find meaningful relationships in texts where you didn't know they were there. But all of that then led me into copywriting, so like brand copywriting, product copywriting, ads, copywriting as creative direction.

Dan:

And then eventually back to the Google data, so SEO copywriting and SEO strategy. And I focused for a while on optimization, research, data analysis for SEO, some technical SEO. And then, yeah, my recent journey has been more in the design world. Content design, content strategy, user experience design. And I'd always been interested in AI and the question was always, how do you do that as a job? Particularly from the position I was coming from as a former student and teacher of poetry and writing. Of course, when ChatGPT came out, like for many people, the connection became clear to me and I started incorporating it immediately into all my work.

Dan:

I realized that I also needed to brush up on my coding skills, and particularly get more invested in Python. And I took some courses specifically on generative AI and machine learning for that purpose, just to make sure I was prepared. But now I think I'm leaning more into the world of knowledge, thinking about the data that we need for AI. The data structures that create meaning for these systems to ingest or to retrieve or to do with what they need. And in the case of generative AI, this is content. This is a task that requires a content designer, content strategist. It's going to be primarily images, text, audio. So that's what I've been up to lately. And yeah, I'm excited to talk to you about it.

Larry:

Well, that's great. I got to say, it's hard to imagine anyone better prepared for this stuff, because to go from playing with Google and poetry stuff, the notion of vectorized word embeddings was just like, "Oh, cool, that's another way to do that." I can almost picture this evolution going pretty smoothly for you. But a lot of content people are not as technically curious as you are, or haven't had the same technical opportunities. And you have a lot of colleagues who are more like conventional content design kind of folks. Have you thought about how people who are less natively technically inclined can jump more into AI stuff?

Dan:

Yeah. I think it's about leaning on their expertise, especially abstracting that expertise. So for a content designer who maybe imagines themselves more as a UX writer or comes from a copywriting background, it's an understanding of information, of messaging, of what content works best for people in what scenarios. And that kind of knowledge, that's less of the craft side and more of the wisdom of content, is incredibly valuable to data scientists and to engineers working on AI.

Dan:

This is some of the expertise that's needed, is subject matter expertise, including on content. So generative AI consumes data and puts out data. That data is content. You need a content person to figure out what it will be, what the use case is, and what content you want these models to produce on the other end, either for a system or for an end user. So you're giving up a bit of control on the craft side, but on the strategic side you're actually, if you're willing to have those conversations with the technical people, you are asserting control in a way.

Larry:

Right. That's so interesting because, as you're saying that, I'm picturing... it's sort of like the way, a lot of these models, there's attempts to capture subject matter expertise and incorporating that in there. But you also need that subject matter expertise to train the models. To write the prompts, do all the other stuff as well. Can you talk a little bit about that relationship between... and this gets at people's concerns of AI replacing them, because if we capture all that subject matter expertise, then all of a sudden it's like, "Oh, we don't need content designers." I personally don't think that's coming, but what do you think about that idea?

Dan:

Yeah. People have talked a lot about this. I think some of the concerns are overblown. Of course there's a grain of truth in this. Theoretically, if we were to all give all of our best data, most of which is just in our minds as experts, so it doesn't exist in the right data format, but if we were and we were to train models that somehow are still usable and not unwieldy as a result of that, you would start to replace people.

Dan:

That's not what's going on right now. That's not the technical capabilities. Anyone who's using these tools or working with them can see that. And also just the actual process of properly curating the data and testing and iterating on methods of fine-tuning and reward functions, and getting the right feedback from the right experts. That's a lot of work. That's a lot of resources, even for small use cases. So I don't think that's the worry. I think it's more like an opportunity. This is an exciting opportunity to make your work more scalable faster. I think, especially from the content design perspective, also to be able to assert governance over content creation through the consistency of machines that doesn't necessarily exist in people always.

Larry:

Right. And what you just said, I realized that my question was sort of like I'm projecting the alarm that I feel in a lot of circles. But I think more often the answers are like what you just said. It's much more hopeful and optimistic in that, at every juncture, there's going to be more need for our expertise that will, probably not for the next couple of decades, anyway, be codified in machines. So that kind of leads me back to one of the things that wanted to talk about a little earlier, actually. It's just, done both conventional content design for regular, old digital products. And then now you're working more on the AI side. Can you talk a little bit about the evolution of the practice as you go from one realm to another?

Dan:

Yeah. Well, I think, as we were just talking about, one thing to notice is the importance of cross-functional teams. So having not just the tech people in there, the data scientists and engineers and developers, but also the content people,...

Rebecca Nguyen

In her work as a content designer at Indeed.com, Rebecca Nguyen is finding new opportunities to assume a leadership role on teams working with generative AI.

Rebecca feels fortunate to work with teams that recognize the value of writing and design skills. She's also finding that generative AI is the perfect place for content design to take the lead.

We talked about:

her work as a senior UX content designer at Indeed and her recent shift to focus on product teams using generative AI

how well-suited content designers are to AI products

the unique challenges of working with non-deterministic large language models

their process for designing prompts and how they evaluate them

her learning curve around the loss of some language control that you get in conventional content design

the main differences between prompt engineering (the how) and content design (the what)

her ability as a content designer to lead more in the AI space than in prior design roles

how they balance the use of outsourced LLM solutions like OpenAI versus developing their own models

the lack of genuine intelligence in LLMs

how her fear and concern about AI is eased the more she works in the LLM world

how the evaluation component of designing content for AI creates more work for content folks

one of the main benefits of LLMs - their ability to take on tedious rote content work

the child-like nature of LLMs

the surprising liberating effects of simply not worrying about whether or not you have a seat at the proverbial table

Rebecca's bio

Rebecca Nguyen (she/her/hers) is a Senior UX Content Designer at Indeed. She’s been part of marketing, UX, and product design teams at Bankrate, Northwestern Mutual, and LPL Financial, where she established the content strategy practice. A Confab speaker and workshop instructor, Rebecca is also an award-winning memoirist.

Connect with Rebecca online

LinkedIn

RebeccaAnneNguyen.com

Video

Here’s the video version of our conversation:

https://youtu.be/8WnxlXXKxeY

Podcast intro transcript

This is the Content and AI podcast, episode number 15. Just as content design was emerging as its own craft and profession, along came generative AI. At first it looked like ChatGPT and large language models might displace content designers (unfortunately, it appears from recent layoffs that some executives may still think this is the case), but at Indeed.com, Rebecca Nguyen has found that working with LLMs has given her more work, not less, and that her content design efforts are now more interesting, rewarding, and impactful.

Interview transcript

Larry:

Hi everyone. Welcome to episode number 15 of the Content + AI podcast. I'm really happy today to welcome to the show Rebecca Nguygen. Rebecca is a senior UX content designer at Indeed. Welcome, Rebecca. Tell the folks a little bit more about what you do at Indeed.

Rebecca:

Hey, thank you so much, Larry. Great to be here. Yeah, I'm a senior UX content designer at Indeed. I've been there for a couple of years now, going on two years, and I work on product teams to make sure their content is useful and useful and accessible and inclusive and all those goodies that we're used to. And in the past six months or so, my role has really shifted and I've been almost exclusively focused on working with product teams who are using generative AI in their products.

Larry:

And that's why I wanted to have you on the show is we talked about this a while back. And that's one way to think... One way I think about that is all of a sudden we have new collaborators in two senses. One, we have these new, we're talking to machines in our work because they're generating some of the language we work with, but there's also a lot of other new collaborators. Tell me a little bit about how the people around you have changed over the last six months.

Rebecca:

Yeah, that's such a great point. So we're probably, if we're working in product content, we're used to working with product managers, we're used to working with UX designers, engineers. And that has shifted in that the team that I am partnering with now is made up of engineers and product managers, but we're also working really, really closely with data scientists and we do not have a UX designer or UX researcher on the team right now. So UX content design is really the entire voice of UX in this group, which is really cool.

Larry:

That's really interesting because often we're the last one in. How does that feel going in there as a sole UX person?

Rebecca:

It's exciting. It's been a little bit intimidating, but I haven't found myself feeling completely lost or anything. I think it's been great. As we were chatting earlier and you said we're really... We're creating a content product when we're working with these language models. The output is text and language, and so who better could be suited to drive and design the language when working with one of these models? It's been a really natural fit. And then the activities and tasks and approach has been different from anything I've done before, but it's well-suited to a content designer skills, I would say.

Larry:

Well, that's it. So what has that transition been like? You said the activities and the tasks differ. It sounds like it kind of rhymes with your old conventional product work, but how is it different now?

Rebecca:

Like that. Yeah, I sort of talk about it as if we think of a sandwich and in that content creation moment, that's the meat. That's sort of like when we're going through a design thinking process, we're doing some discovery or research or we're deciding on the problem that we want to solve, and then we get to that moment where we make the thing, we design the thing and we might be writing words. And after that we are iterating and getting feedback and seeing how it performs and measuring and iterating more, et cetera.

Rebecca:

The difference for me with generative AI has been spreading my focus out and becoming more of the bread. So instead of the meat, that creation moment, when you're working with a language model, the model takes on that task. They're the ones creating the content. And your focus as a human is all of that stuff on the periphery of that, so the prepping, which we would be sort of the prompt engineering and design where we're telling the model what we want it to do, and then the evaluation piece where we're looking at what the model did and saying, "Okay, was it successful? Did it follow directions? Could we do it better?"

Rebecca:

So it's almost like you become a teacher of content design instead of a content designer where you're actually making the thing yourself.

Larry:

Interesting. I have not heard it articulated that way, but that makes perfect sense because... Well, they're called learning models and you're the teacher. That's great. And you mentioned both prompts and one of the things you just said made me think that people always talk about prompt engineering, and you talked about engineering and designing prompts. Do you go into prompt creation with a designer hat on because you're working with engineers? Do you think more as a designer in that world?

Rebecca:

Yeah, I definitely think so. Particularly as a content designer, thinking about how does the language inside the prompt impact the output and to make sure that content design considerations are represented in the prompt as much as possible to make sure that we're getting the output where we want it to be. We're sort of preemptively correcting mistakes or anticipating mistakes that could happen.

Rebecca:

For example, when you get familiar with a model like ChatGPT, you can see, and we all can see as content designers sort of that out-of-the-box tone that the model assumes, the model that's been trained on the internet. So it's a very casual tone. It is, in my opinion, it's overly friendly in a way that can be kind of annoying. There's lots of exclamation points, there's a lot of celebration for small things that may not require such celebration. It helped you with a task and it's like, "You're so welcome. Awesome." And you're like, "Calm down."

Rebecca:

That tone and that voice isn't always appropriate for a product. And so when you're getting in there and designing prompts, you have this opportunity to modify as best you can. And the cool thing about prompt engineering is that you can do a lot of playing around and you can see how different instructions impact the outputs and then tweak and adjust from there. But that was surprising to me because I think that on my team and at my organization, at first we were sort of thinking about this on the other end, once we see the output, then let's evaluate it and give feedback. But the problem is that once the output has happened, it's too late. It's not like working with a human where you can revise it and create this static thing. It's always going to be different every time. It's that non-deterministic nature of a large language model. And so really we want to get in at the prompt stage to try and drive and direct before the output happens.

Larry:

That's so interesting. But you're still getting some feedback from it, too. You mentioned earlier how one of the pieces of bread is about iteration in your sandwich. And then, as you're talking there, I'm also reminded back when you said that you don't have UX researchers on the team. Are there more automated ways of getting feedback? Like for you, because you're still looking at it after the fact to see compliance with... Not compliance, but sort of alignment with voice and tone and that kind of thing. I guess tell me a little bit about that loop.

Rebecca:

Yes. So we have content design at the beginning, which would be the... Actually, even before we do prompt design, a content designer can create a mock in Figma or whatever you're designing as this north star....

May Habib

May Habib is the CEO at Writer.com, a generative-AI platform that has been helping enterprises use AI since 2020.

Her company builds its own award-winning large language models and is pioneering approaches like "headless AI" to help employees across an enterprise use AI to be more creative and productive.

We talked about:

her work as CEO at Writer.com, a "full-stack generative-AI platform," for the past four years

her decade-long work in the AI and NLP space, beginning with translation solutions

her take on the "over-chat-ification" of AI products, the reliance on chat interfaces as opposed to other ways to access AI capabilities

her prediction that 2024 will the "get real" year for AI

the use of fine-tuning and/or RAG to connect learning models

the inadequacies of vector databases for knowledge retrieval and their exploration of knowledge graphs to fill the gap

a new role, the "AI ontologist"

another new role, the "AI program director" which includes a mix of left- and right-brain thinking and technical skills

some of the use cases for "headless" AI

their approach to securing and protecting the various kinds of data used in their LLM

how she sees the role of data scientists in AI

their tactical approach to building knowledge graphs for specific business use cases

their work at Writer on no-code and low-code tooling to help their customers build solutions and tooling on the platform

new content job roles that are emerging as AI takes hold in enterprises

May's bio

May Habib is CEO and co-founder of Writer, the only fully-integrated generative AI platform built for enterprises. Leading companies, including Vanguard, Intuit, L’Oreal, Accenture, Spotify, Uber, and more, choose Writer to help them deploy generative AI across their businesses, allowing them to automate and augment key operational activities and increase employee creativity and productivity. Writer’s family of large language models (LLMs) are state-of-the-art, topping leaderboards for natural language understanding and generation. The company’s security-first approach means that Writer’s large language models and generative AI platform are deployed inside an enterprise’s own computing infrastructure.

Launched in 2020, Writer has seen immense success with customer adoption, has grown revenues by 10x in the last two years, and has over 150% net revenue retention. May and the Writer team have successfully raised over $126M in funding from notable investors, including ICONIQ Growth, Balderton Capital, and Insight Partners.

May began her entrepreneurial journey as a teenager, and founded her first language startup, Qordoba, a localization software company, 10 years ago. May is an expert in AI-driven language generation, AI-related organizational change, and the evolving ways we use language online. She has been recognized for many different awards, including the recent 2023 Forbes AI 50 and Inc.'s 2023 Female Founder Award. She is a MELI Fellow with the Aspen Institute. She graduated from Harvard University and spends her time between San Francisco, where Writer is based, and London, where her two children live.

Connect with May online

LinkedIn

email may at writer dot com

Video

Here’s the video version of our conversation:

https://youtu.be/lFTfA4X8CkA

Podcast intro transcript

This is the Content and AI podcast, episode number 14. Over the past year and a half, innovative artificial intelligence startups have taken the tech and content worlds by storm. In her position as the CEO of the generative AI platfom Writer.com, May Habib has been right in the middle of the excitement, and out in front of it. Writer and their clients were deploying LLM-driven generative AI programs inside of large enterprises long before OpenAI's ChatGPT 3 captured the headlines and launched the current wave of AI disruption.

Interview transcript

Larry:

Hey everyone. Welcome to episode number 14 of the Content and AI podcast. I am really delighted today to welcome to the show Me Habib. May is the CEO and co-founder at Writer, an app many of you're familiar with. They're just having a great year and I was excited to get her on the show towards the end of 2023 to talk about topping the MMLU leaderboard with the Palmyra, their LLM, closed the nice funding round. Sounds like things are going well at Writer, May.

May:

Oh, thanks Larry. It's so nice to come back and chat with you. Yeah, we've had a great year, thank goodness. I've got our last all-hands of the year after this conversation, and so it was definitely nice to look back. We do these weekly updates to the whole company. I write them, and I went back and looked at week one and compared it to week 52, and then one's like, "Oh, let's go back a little further." I went 2022, week 52, and then 2021, week 52, and yeah, it's awesome to see things build and all the progress.

Larry:

Yeah. Well and one thing, and you've been part of that progress. ChatGPT, which is where the current kerfuffle is all about, that's barely a year old, but Writer's older, and Cordova was even older than that, right?

May:

Yeah, well we've been in the NLP space for a decade, me and Waseem, and starting in machine translation. I think we were able to come to the world of transformers with maybe two distinct advantages, I think, over the folks that are in the space now, OpenAI and others as well included in that.

May:

One is, we were very much less a technology and search for a problem. Because we saw so many content challenges in the enterprise that could be solved with AI, having come from translation. So, it allowed us to really take a solution-based, outcome-based approach to thinking about how to productize this cool technology versus not.

May:

Now, obviously a general-purpose AI-based chat has captured the imagination, and has been an incredible thing that the OpenAI team has introduced that we obviously didn't think of, but in a lot of ways what it's done is open up what people thought could be possible with AI, and it's made room for solutions like ours to really explode, because we really serve that enterprise need, that very solution-specific application that is enterprise-ready, is secure. So anyway, it has been a fun road and a really fun four years with Writer.

Larry:

One of the implications, and you know as much about this stuff as anybody, in fact you're right up there with OpenAI in terms of your accomplishments and the power, the service you offer. I'm curious, what are you like... And I think there's a couple of things in this question and that I'm hoping to get out of this conversation. One is just the general state of the AI market. A lot of what you just said, I think it's going to help people ground themselves and feel it. But I think one of my questions is, for example, is this just another SaaS app that the software in the background is an LLM, or will there be fundamentally different things you think that content folks have to consider as they go into both working with these tools and working on these tools?

May:

Yeah, I think maybe taking that question a couple of ways. One, the user-experience cut of the market, and then the where-are-dollars-being-spent cut of the market. I think it'll allow you to see the gap that we see and that we feel, actually, looking at it in these two ways. I think from an end-user perspective, that cut of the market, there is this over-chatification of what AI can do, and everything is a fricking dialogue to get stuff out of AI, and it's just so early, and the interfaces obviously shouldn't all be chat UIs, but that's kind of the case right now. Whether somebody gave you a Copilot license or you're personally paying for ChatGPT Enterprise, I think most people aren't getting the value they thought they would, given all of the headlines.

May:

That adoption gap isn't because the capabilities aren't there because we are building the capabilities. They are fricking crazy magical, and I think when we chatted last, I said probably something along the lines, if this was 18 months ago, I probably said, "Larry generative AI is like giving everybody an assistant and a chief of staff." I mean, that's not what it's like anymore. It's giving you the best version of yourself, 20-years expert into the future. There is so much, even in 18 months, so much that the models can do.

May:

Anyway, all to say that the end-user experience cut of the market is super under-optimized and today, despite all of the hubbub, I can't go into my sales force and say, "I'm in London in January, who should I see of our deals that are closing in Q1?" So even the AI that's supposed to get built into all of our systems of record, isn't really doing the things that we want it to do. Folks who are trying to connect Copilot to their Microsoft data aren't seeing the kind of answers they would like. And I think power users who have figured out how to get a lot of value from ChatGPT are, but your median user really isn't. So, that's that cut of the market.

May:

In terms of where there are real dollars being spent here, I think the enterprise is probably over-investing in the infrastructure and the utility model layer, and are trying to rebuild from scratch every use case, and there are a lot of things that are breaking about that experience. And the total cost of ownership, I think, isn't making sense for a lot of companies. The accuracy and business impact of some of these pilots and POCs isn't materializing.

May:

So, next year is going to be get-real year, which is exciting. I think we'll see a lot more exciting end-user interfaces and experience that build on the toy making and piloting of tools this year. And then I think enterprises are going to be really looking for just more comprehensive solutions to filling generative AI needs internally.

Larry:

Yeah, a couple of follow-ups. First thing,...

Laura Costantino

Laura Costantino is watching the emergence of AI in content professions from two interesting and valuable perspectives: as a content designer working on LLMs at Google and as an active participant in the social-media communities where content professionals gathers.

In their work at Google, they have returned to their roots as a content strategist to manage the challenges that come with designing content at a massive scale.

Through their interactions in the community, they have had the chance to hear the concerns of content designers who are navigating the new world of AI - and to inspire them with advice and success stories.

We talked about:

their work at Google as a senior content designer training LLMs

how their content strategy background is helping in their current work

the difference in working with content at a huge scale, as is required in their work with large language models

how their work is operationalized in the ever-changing workflows at Google

the community of knowledge sharing that has arisen organically among a variety of content crafts at Google

their advice on how to cope with the rapid pace of change in the world of AI

how they works with data scientists, machine-learning engineers, and other AI collaborators

their cautiously optimistic view of future of the content-design profession

their advice to content designers for taking a proactive and curious approach to new AI technologies and practices

Laura's bio

Laura Costantino (they/them) is a senior content designer and strategist working on AI and large language models (LLMs) at Google. For the past ten years, they have worked at the intersection of UX, content, and marketing for some of the world's largest tech companies. Laura developed a passion for storytelling early on and received a MA in Cinema Studies in San Francisco, where they worked as a curator for a range of film festivals and cultural institutions around the Bay Area. Outside of work, Laura is committed to mentoring people transitioning into UX and tech, advocating for content, and sharing advice on LinkedIn. They currently live in NYC, were born in Southern Italy, and speak both English and Italian fluently.

Connect with Laura online

LinkedIn

Video

Here’s the video version of our conversation:

https://youtu.be/EdgyXGC3xlI

Podcast intro transcript

This is the Content and AI podcast, episode number 12. The arrival of large language models and chatbots like OpenAI's ChatGPT, Anthropic's Claude, and Google's Bard is creating both existential concerns and new opportunities for content professionals. In their work as a content designer at Google and through their extensive professional networking, Laura Costantino has the chance to witness the full range of work experiences and personal emotions that come with the rapid adoption of new artificial intelligence practices.

Interview transcript

Larry:

Hi everyone. Welcome to episode number 12 of the Content and AI podcast. I'm really delighted today to welcome to the show Laura Costantino. Laura is a Senior Content Designer at Google, doing really interesting work around AI and content stuff. So welcome Laura. Tell the folks a little bit about your role there at Google?

Laura:

Hi, Larry. Thanks for having me. It's so nice to be here. Yeah, so I've been at Google for about a year and a half, but somewhat recently, maybe three and a half, four months ago, I moved from my previous team to my current team, and I am at the moment working as a senior content designer, training large language models. So that's my new job.

Larry:

Well, training large language models at one of the biggest tech companies in the world, that's pretty interesting, especially for folks in the content world. There's so much to ask about that. I guess the first thing I'd ask is what's the biggest change? What's the biggest difference in training a language model versus the content design work you were doing a year ago?

Laura:

Yeah, that's a great question. I came up to content design through content strategy and to an extent marketing as well. And so I think for me, it really helped to have that content strategy background, meaning really being familiar with content at scale, content governance. And I think that's been the biggest difference for me, that in my current role, I had to go back to my past and brush up on some of those skills that I think I learned more in the past, versus in my most recent roles as a content designer. I think my day-to-day was still a little bit more writing strings and felt a little bit more like bespoke and... I don't want to say in the moment because of course, ideally it wouldn't be in the moment, but unfortunately sometimes it is in the moment when someone asks you to write a string or edit a string, versus right now I do think my role, it's a lot more focused on the strategy at scale, and I do think it's a function of the role more than say, me growing in my career or something.

Larry:

That's so interesting because when you think about it, because most content design roles, like you just said, you're embedded in a specific product working on just strings and error messages, but also the narrative of the whole product and all that stuff, but then you move up a notch to this kind of thing and all of a sudden like, "Boy, I'm glad I have this content strategy background because I need it again."

Larry:

Tell me a little bit about how that manifests in training a large language model? It seems clear, I think I get why you need to be strategic about it, but can you talk a little bit about why you've had to go back in your toolbox for your content strategy stuff?

Laura:

Yeah, of course. So training data for a large language model, of course, we're talking about volume of data that are really hard to wrap our heads around, and two techniques and one in particular that we've been using that are used as to train large language model or fine-tuning and reinforced learning. And there is all sorts of methodologies that are used and most methodologies require to look at content at scale, like ingest. And some of the technicalities, I admit, I don't fully understand myself, but I create metaphors in my mind or images of how I think certain things work. And I always imagine these large quantities of data, which in this case is really content, sentences and words and so on and so forth being ingested into these box and that then creates more content out of it.

Laura:

And so for me, I think in that sense, working on content at scale because some of the content is content that is created by UX, but we also work with a lot of other people. So it's not so much like me as a content designer, I have a full handle on all the strings that are going to go into a flaw, that's just not going to happen. And so it's more creating the guidelines. And some of that of course, is the work of a content designer, but I think here it becomes even a little bit more not just the guidelines in terms of style and voice and tone, but also operationally, how do we make sure that creating content at scale can work for the team to a scale that is big enough that it helps training the model?

Larry:

Yeah, as you talk about that, I'm wondering, first, there's two things in there that really interest me. One is you're using content as data, and then data as a design material. So are you looking for patterns in the data and the content or... Because at scale, you can't just look at every data point and go like, "Oh, we'll treat this one this way." Tell me a little bit about that?

Laura:

Yeah, that's exactly it. And I think that's, again, going back to what I was saying, my days as a content strategist and doing some sort of, for example, taxonomy work or thinking about in the past, how to label certain kinds of data. And this isn't necessarily what I do now, but when I did that in the past, when I work on categorizing content, a lot of what I had to do was looking at patterns and trying to figure out... And I have been in my head a little bit and loosely wanting to use the Pareto principle. If I remember correctly, 20% reflects the rest of the 80%, you only need 20%. Maybe this isn't a really good explanation, but that's what I think, sampling through the data and trying to find patterns, just like you said, and seeing how the model is responding. And from that, figuring out how do we constantly improve it with new training data.

Larry:

And you talked about that because you're actually in there training the model. And you mentioned two terms there, fine-tuning and reinforcement learning. Here's my little tiny brain's interpretation, is that fine-tuning seems like go on one level deeper than prompt engineering and doing higher level fine-tuning of the model itself. And then reinforcement learning, as I understand, is a neural network thing that's like a Skinner box kind of reinforcement, giving little food pellets to the model when it gets something right. Is that how it works?

Laura:

From my understanding, yes, the reinforce is a little bit more like saying, "This is good, this is bad," in a very simplified way, and that's one side of it. And then the other side, the fine-tuning, right now for me has been more working with a really large scale of content, like a really large amount of model responses.

Larry:

Yeah. Hey, and when you first started talking about this, you mentioned that the ultimate goal out of all this work with the modeling and your work in general is to operationalize it, to get it ensconced in your day-to-day work, I guess. How does that differ? Because I've seen them done a lot of that kind of work in the content design world, but in the AI world, not so much. How does operationalization look in your world?

Laura:

Yeah, that's a good question because I do think we're still figuring it out....

Chris Cameron

At Booking.com, they've been helping travelers with their trip planning for many years.

The arrival of generative AI has given them new ways to help travelers with this business-critical task.

Over the past year, Chris Cameron has applied his UX writing and content strategy skills in ways both familiar and new to help build a new AI-powered Trip Planner tool that integrates with Booking.com's travel-booking app.

We talked about:

his work as a principal UX writer at Booking.com on their "writing system," which is sort of like their version of a design system for UX writers

his recruitment to a "tiger team" at Booking to develop a new travel-planning AI chatbot for their travel-booking app

the key differences between his prior product work and his work on this AI product

the new kinds of collaboration that have arisen in his work on a generative AI product, in particular his work with machine-learning engineers

the transition from the prototype of the app to its current position as an established product

the product-feedback mechanisms that are built into the Booking "Trip Planner"

how to jump start your learning if you're new to working on generative-AI tools

how they were able to leverage components in their current design system to build the new Trip Planner app

the prompt engineering skills he developed by creating an AI "story robot" for his three-year-old son

his optimism about the employment prospects for UX writers

how traditional content strategy practices like establishing voice and tone and consistent terminology manifest in AI product design

how new AI practices are just as likely to show up as enterprise productivity improvements as in customer-facing products and features

Chris's bio

Chris Cameron has over 13 years of professional writing experience across journalism, marketing, and UX. As a Principal UX Writer at Booking.com, Chris oversees UX Writing Systems, managing the tools and workflows that enable over 80 UX writers to efficiently create high-quality content localised into over 45 languages and dialects. Born in Boston and raised in Phoenix, Chris now lives in Amsterdam with his wife and son.

Connect with Chris online

LinkedIn

Video

Here’s the video version of our conversation:

https://youtu.be/bptOvimY4uU

Podcast intro transcript

This is the Content and AI podcast, episode number 11. As generative-AI tools are introduced into consumer products and enterprise workflows, the core work of content designers and UX writers still feels familiar, but the context for the work and many of its details are evolving. Over the past year, at Booking.com, where he has been working on an AI-powered travel-planning app, Chris Cameron has seen first-hand how the traditional concerns of content strategy and UX writing manifest in the world of generative AI.

Interview transcript

Larry:

Hi, everyone. Welcome to Episode #11 of the Content + AI Podcast. I'm really happy today to welcome to the show Chris Cameron. Chris is a principal UX writer at Booking.com, the big travel booking agency based in Amsterdam. Welcome to the show, Chris. Tell the folks a little bit more about what you do there at Booking.

Chris:

Well, thanks, Larry, for having me. Yeah, I'll give a bit of my background as well. Like yourself, I started in journalism and then got into copywriting. And after moving to Amsterdam from the US at a very young age, 25, I guess, I eventually joined Booking in 2016, a little over seven years ago. And back then, the role was actually called copywriting. There was about 25 of us. And over the years we sort of discovered that we were actually UX writers, and we've become now this community of over 80 UX writers.

And now, I am a principal UX writer, and the area I look after we call writing systems. And what that is is sort of like the writing version of design systems, but it's not so much a system, it's more like the tools and the workflows that we use to get our jobs done. So my role is to work on those tools and work on those workflows and make sure it's easy for our writers to get their jobs done in an efficient and easy way so they can create high quality content. And more recently, one of the areas I've been interested in looking into is GenAI and how we might use that to improve our workflows.

Larry:

Yeah, that's why I wanted to have you on the show. You told me about this product you developed, the Trip Planner, that's based on AI. Can you tell us a little bit about how that project arose and how you got involved with it?

Chris:

Yeah, definitely. So my involvement with AI and GenAI in general started when ChatGPT came out. I think a lot of people took notice back then. That was late last year, 2022. And I started playing around with it. I'm always a bit of a nerd and early adopter of technology, so I started using it for different things. I have a toddler at home, so I was actually using it to create bedtime stories for him. I would say, "Let's ask the story robot what kind of story you want to read tonight", and he would just generate a story idea, and ChatGPT would help us with the rest. It was a lot of fun.

Chris:

But professionally, I started thinking, "Okay, how could this be useful for our writers at Booking or how Booking as a company could use it?" And early this year, 2023, the company was seriously looking at GenAI and thinking, "Okay, what are we going to do with this?" And because I was already exploring it, I got pulled into some early discussions, and I thought, "Okay, we're going to have some brainstorms, some chats about how GenAI did," but actually the company was already like, "Let's go build a GenAI chatbot and put it in the app, and this is going to be the only thing you focus on for the next couple of months." And I'm like, "Okay, let's do it. Let's roll."

Chris:

And so basically, a task force was formed within the company, sometimes called a tiger team, we called it sometime, and it was representatives, multiple people from writing, design, research, product, and then also machine learning, our iOS and Android engineers, of course, data science, and marketing and legal. It was a big team. In the end, it was almost like having a little startup within the company, it was about 70 people. And the UX work stream was sort of one half of it, and the other half was all the machine learning and the engineering that was going on.

Chris:

And this sort of kicked off in mid-April when we started this, and two months later, we were able to launch the AI Trip Planner in June. Just so people understand what it is we built, we built basically an AI chatbot into the Booking app, and people can chat with it and ask their travel questions, and it can help them get inspiration for where to go or what hotel to stay at or build an itinerary, these sorts of things. And it integrates some of the traditional booking experience, like with carousels and images and property ratings and things like that, right into the chat so it feels a bit more natural. And then if they tap on a property, they can go straight into the booking process and make a reservation.

Chris:

And so a lot of it uses some of our existing machine learning knowledge we've built up at the company over the years and then relies a bit on OpenAI ChatGPT to do that generative AI piece and really create a nice conversation. So if people want to try it out, if they're in the US or they can VPN to the US, they can sign into the Booking app on iOS and Android and make sure their language is set to English, and they should see the AI Trip Planner right on the home screen.

Larry:

That sort of gets at some of the complexity around this, because I know you localize into 50 languages and cultures.

Chris:

Yeah, I think 45. Yeah.

Larry:

And so right now it's just English only and in the US, so that's interesting. And really, as you talk about that, I'm wondering from a user perspective, it's almost like just a UI thing. For an end user, you could almost perceive it that way. "Oh, another way to interact with this thing and do my trip planning." But on the backend, like you said, it's 70 people on this tiger team that put that together. How similar was it to other products, because as a principal you've worked on a lot of different projects probably at this scale, how much of it was familiar and similar and how much of it was new? Tell me a little bit about that.

Chris:

Yeah, definitely. There was a lot that was familiar to just a normal building a product, but there were some key differences. For example, for working with a GenAI product specifically, it's such a new thing that there's not a lot of existing research. So if you're going to go to your researcher and say, "Okay, what do we know about GenAI?" It's like, well, they're still learning too. So a lot of that was involved, looking at what is out there in the market, what competitors are doing, but then we were also able to combine that with the existing understanding of user needs, because essentially this is a search experience that we've been dealing with for a long time at Booking. So we know a lot about what the user's looking for in that moment when they come to the app. So those needs didn't change, but the way they were expressing those needs is the whole new thing.

Chris:

And in the early stages, when we were trying to test something, it's not that easy to build a GenAI prototype. If you're building a prototype in Figma, you can't really insert the AI part in there very easily. Maybe soon that will be a thing. So we had to wait until we actually had a working build of the tool where we could play with it internally, and that's when we started actually doing a lot of the understanding of, "Okay, what's working, what's not?" that sort of thing. So there was that challenge.

Chris:

But from a content and writing perspective,...

Lance Cummings

Education often lags behind tech trends. Not in the case of AI. And not when Lance Cummings is involved.

Lance conducts academic research on AI content operations and has worked both with technical communicators and with content entrepreneurs in the creator economy.

Along the way he has discovered concepts and practices around structured content that apply across prompt engineering, tech writing, and influencer content creation.

We talked about:

his work as a rhetoric and writing professor and research on the creator economy

his view of content operations and workflow, especially new practices around AI

how the introduction of the idea of "AI content operations" clarifies the writing process for content creators of all kinds, including the technical writers that he teaches

how a structured-content approach can help writers of all kinds cultivate a garden of ideas

how the real value around your content lies in interactions with your community, not necessarily the content itself

his approach to collaborative prompting, knowledge management, and development of AI tools

how standards and practices like DITA and object-oriented knowledge management

how structured content can actually make us more creative

why creative writers generally excel in the tech writing field

Lance's bio

Lance Cummings is an associate professor of English in the Professional Writing program at the University of North Carolina Wilmington. Dr. Cummings explores content and information development in technologically and culturally diverse contexts both in his research and teaching. His most recent work looks at how to leverage structured content with rhetorical strategies to improve the performance of generative AI technologies and shares his explorations in his newsletter, Cyborgs Writing.

Connect with Lance online

Cyborgs Writing

LinkedIn

Video

Here’s the video version of our conversation:

https://youtu.be/lneGOV6tNbY

Podcast intro transcript

This is the Content and AI podcast, episode number 10. We are quickly discovering that AI can help content professionals across the span of their work. Lance Cummings is a consultant and college professor who is exploring intersections that most content folks haven't had time to ponder. For example, he has found that his approach to AI content operations can clarify and improve the writing process for both technical documentation authors at big enterprises as well as fiercely independent members of the creator economy.

Interview transcript

Larry:

Hi everyone. Welcome to episode number 10 of the Content + AI podcast. I'm really delighted today to welcome to the show Lance Cummings. Lance is a professor of English in the professional writing program at the University of North Carolina in Wilmington. He does a lot of interesting research, and we'll talk about that as we get going. But one of the interesting things in the intersections of his research and academic interests is this notion of applying structured content, looking at structured content, rhetorical strategies, and AI technologies and workflows around that. I'm really excited to talk about all this stuff with you, Lance, but welcome to the show. Tell the folks a little bit more about what you're up to.

Lance:

Yeah, so I'm a professor in rhetoric and writings, which generally just means that we study how people write, make meaning, get things done with text. And more recently I've been researching the creator economy, actually before AI. That's how I stumbled across AI in 2021. And if you don't know what the creator economy, it's what this podcast is. It's people creating content directly to audiences using the various digital platforms out there, and oftentimes either making some money or a lot of money or making a living even off of doing this. I would say different from influencers, I would say creators are creating useful content for their audiences that they can use and very specific audiences. And since COVID that has risen 50% every year, but when you get deep into the creator economy, they really think about content in terms of workflow and how do you create a process to develop content consistently, and how do you be creative?

Lance:

Because as a content creator, you have to consistently build content for your community. I stumbled upon AI and I thought, "Well, we're all going to be using this in two years," and here we are. And so I've been exploring then how AI works into the writing process, both in my own content development and also among creators. And then thinking about that in terms of technical writing, content and content management. One of the things that I think content specialists or tech writers have to offer us is a more structured conception of content and how that works into this idea of workflow. I was at a conference, I go to a conference every year in Krakow called SOAP, and last year the topic was content operations, which it seems like that's a term going around a lot. And we just simply defined it as the people, processes, and things that are around content creation.

Lance:

So thinking about writing as a networked activity of relationships between us and people and the technology, and then thinking about how that shifts and changes. And that's what we would call a workflow, rather than thinking about, "Okay, here's step one of the writing process, step two, three, four, five," thinking of it, "Okay, here's the network of things that are happening to make this content work."

Larry:

One of the things we talked before we went on the air a little bit about, that workflow, I think anybody who has done any content operations has a conception of workflow in their head, but there's a specific meaning to workflow in the academic world, which I wasn't aware of. Can you talk a little bit about, well, first that academic meaning of workflow, like when you're teaching professional writers or professional writing skills, how you think about workflow and then how that is applied in your current work in AI.

Lance:

So workflow is really, we're trying to make a shift from thinking of writing just as a process. So the traditional way of teaching writing is actually based on the canons of rhetoric, but you brainstorm or come up with ideas, you organize those ideas, then you draft, get peer review, and then publish, right? Well, I guess thinking in terms of workflow is thinking about content much more from the software development side where content is actually constantly changing, shifting, and people who create content or write are constantly thinking about how to change or tweak their workflow. So it's not like a number of steps set in stone, but rather this network of actions or interactions that we have with people and things that are constantly changing on their own, but then we can also find points where we can tweak those to make it more efficient, more creative, more interesting.

Lance:

And I think that tech writers or content specialists have been doing this for a while. The best content writers that I know are constantly tweaking their workflows and adapting, and I think that's why you see a lot of content creators, tech writers, adopting AI fairly early because it's an easy next step to think about AI as where does it fit into this workflow, rather than how does it take over this part of writing? So integrating AI in a way that extends what we're doing and enhances our workflow, but doesn't necessarily take over.

Larry:

Yeah. That's a great way to think of it. I think some executives just think of it, "Oh, we'll just replace people," but it's like, "No, we can extend and enhance the work that we're already doing." As you talk about that, some of the specific ways that AI can fit into this, because conventionally, I just think that AI is, I think Sam Altman said AI is really good at tasks, but not very good at jobs. And so are you looking at it that way, as what are specific tasks in these workflows that AI can help you with? Is that where you start?

Lance:

I think task is an important way to think about AI because if you don't give it a clear task, it doesn't necessarily know what to do, but then you have to think about writing your workflow as a set of tasks. So when you start to think about what you're going to create for this blog, what is the task that you're technically doing in this workflow? I think a lot of times, especially good writers, a lot of what we do is intuitive, implicit, but we don't necessarily always explicitly can say what we're doing as writers. And actually, I think this is one of the best things that AI can do for writing education, is to force us to think about, "All right, what exactly am I doing when I'm creating this piece of content? What are the tasks, and can AI do this task? Should AI do this task? Will AI do this task?" And I think that's part of what I would call AI content operations, is actually deciding what AI can do or should do and when, and I think task is probably one of the better ways of thinking about it.

Larry:

Yeah, I'm really curious now how you tease that out, because as I think about that, so much of writing used to be perceived as magic, just like you just think and magical things happen. But like you've mentioned, both the creators and the creator economy and tech writers are really good at analyzing what they're doing. Are they exemplars or just lucky in the kind of work that they do requires them to be reflective and curious about where did that intuition come from? How can I tease that out and get better at that? Does that make sense?

Lance:

Yeah, I think tech writers are forced to do that in their ... Obviously it's going to depend on the context, but if you're working with people, you have to make your workflow explicit to them if it's going to be successful. If you're working with AI, actually, you have to make your workflow explicit....

Dave Birss (AI-generated)

Dave Birss has had a busy 2023.

Since developing his first AI course for LinkedIn Learning early in the year, he has produced five more courses and has become the learning platform's most popular AI instructor.

We talked about:

his experimental approach to teaching AI

how he helps companies understand the true benefits of AI

the importance of using AI to augment people's skills rather than just to try and save money

the elements of his AI manifesto

use AI responsibly

be ethical

support your employees

assign leaders

keep learning

always add a human layer to AI output

the importance of critically consuming advice from anyone who proclaims to be an AI expert

the importance of companies learning for themselves because there are few reliable consultants available now

how unlocking the true benefits of AI can change companies' perspectives and help them see new opportunities

the crucial task of understanding people and addressing their needs as AI is adopted

his observation that it "cannot be AI or human, which is the way that a lot of companies are seeing it, it's got to be AI plus human"

how the adoption of AI supports his point of view that generalists have an equally important role in the modern workforce as specialists

Dave's bio

Dave Birss combines the analytical mind of an AI geek with the butterfly mind of a former advertising creative director. This helps him make the ever-changing world of AI approachable, relevant, and occasionally entertaining.

At the start of 2023, he launched his first LinkedIn Learning course on Generative AI. Since then, he’s released another five courses, all of which have gained fantastic ratings and reviews. In July LinkedIn announced that he’s now the most popoular AI instructor on the platform.

But Dave isn’t just about online courses. He’s also a globe-trotting educator and public speaker, helping companies and individuals get more value out of Generative AI.

He’s also a best-selling author with several books on creativity and innovation. And a former broadcaster and film-maker.

As a sought-after keynote speaker, Dave speaks about AI, innovation, and creative thinking with a blend of science and dad-jokes.

He’s a Scotsman who lives in London with his Haitian-American wife and two delightfully confused children.

Connect with Dave online

LinkedIn

DaveBirss.com

Video

Here’s the video version of our conversation:

https://youtu.be/2QL01qN6uzY

Podcast intro transcript

This is the Content and AI podcast, episode number 9. Over the past year, we've all been getting up to speed on AI. Over that time span, Dave Birss has become the most popular AI instructor on LinkedIn Learning. Dave would be the first to tell you that he's not an expert on artificial intelligence. But he's a very experienced technology professional who has witnessed several major earlier tech revolutions, and he's an experienced teacher and consultant, so he brings a very pragmatic approach to incorporating AI in your work life.

Interview transcript

Larry:

Hi, everyone. Welcome to episode number nine of the Content and AI podcast. I am really delighted today to welcome to the show Dave Birss. Dave is an educator, author, and consultant currently focusing on AI and AI education. He's the most popular AI instructor at LinkedIn Learning. Welcome, Dave. It's great to have you here. Tell the folks a little bit more about what's going on these days.

Dave:

Thanks, Larry. Yeah, I've been creating courses on AI this year, really. And I can't really call myself an AI expert. I guess I'm an enthusiast and I am an experimenter. I guess I do research to find out what works best, and then I share that knowledge with people.

Dave:

If you told me a year ago that I was going to be doing AI as my main thing, I wouldn't have believed you because OpenAI only released ChatGPT on, I think it was the 30th of November last year, so it's still less than a year old. And when they launched it, I just threw myself in, absorbed as much as I could, created some frameworks, easy ways of being able to teach people, and then I just released these as courses.

Dave:

I've got now six courses on the platform. Just released another one last week. And I'm about to release some courses on my own website as well. Yes, that's my life these days, doing courses and then helping companies get onto their AI journey in the best possible way because I think a lot of them have got the wrong attitude. They're not looking at AI in the right way.

Larry:

Yeah, interesting. Tell me more about that, because I think we all have opinions about AI. What are you discovering?

Dave:

Well, of course, companies, as you know, they will tend to have, "Here's our quarterly target, here's our quarterly goal. Can we make more money and spend less money in this quarter?" That's what they do. It feels as if that's the responsibility of a company is to do that.

Dave:

Now, if that's your attitude towards AI and that you're only interested in using it to save money, really, it's all about productivity, then you're really missing out on 90% of the benefits of AI. Because the real benefit of AI is not just to help you do less work and do work faster, it's to help you do better work. And when you do better work, that gives you an advantage in the marketplace. If you think that you've got this line across here, that this is the profit and loss of a company, you can only nibble away at that by starting to replace humans and tasks by AI. And what you do in return is you do a deal with the devil, which is you embrace the fact that AI is fantastic at adequacy, which means that you're going to get stuff that's all right, maybe just about average that you're going to get from AI. And if you're replacing humans by AI, you will save money, but you get adequacy in return.

Dave:

When you use AI, on the other hand, to make humans more capable of doing phenomenal work, of stretching further than they were able to stretch before, then really, the sky's the limit. At that point, you're gaining profit rather than trying to save cost. And the problem is that most companies are so focused on saving costs and this incremental growth, they're missing out on what is the real potential of embedding AI into your company, into your system, which is to do better work, to reach higher, achieve more.

Larry:

I love the way you're contextualizing that because going from that... I love that they're fantastic at adequacy. And it's like-

Dave:

They excel at it.

Larry:

They excel at adequacy. But the real potential here is in unleashing way more human potential on this work. And this speaks to the need for... Because that quarterly focus of enterprises, it's just notorious in any number of circumstances. Have you had any success or do you see ways that companies might get past that and start to think more strategically about how to embed the benefits of AI in their orgs?

Dave:

Well, when I talk to companies about it, they get it. But there are so many companies that their form of motivation for people in senior leadership is keep cutting costs. We're only focused on this quarter. And that kind of short-termism, I think, will really come round and bite you in the butt when it comes to business.

Dave:

One of the things that I've been doing from conversations with businesses over the last, well, this year since I've really been doing this AI thing, is that I've developed a manifesto that I'll shortly be releasing. Let me see if I can find my cursor on here and I can maybe bring up my manifesto. There we go. This manifesto is all about helping companies understand what they need to do to embrace AI properly.

Dave:

Zoom is doing funny things for me here. If I go back here, I can then share what I've got. I've created this manifesto, and I'll quickly take you through some of the points in the manifesto. I think that this first thing is what I was talking about, is that it's important that we use AI to augment people's skills rather than just to try and save money. I think that that's the main focus for companies that want to get real success and value out of AI.

Dave:

Obviously using data responsibly is an important thing to do. And that's something you have to communicate to your staff, what we mean by using data responsibly. You've got to be ethical because you've got to be guided by your head or your heart. Companies are used to being guided by laws, but we don't have those laws here yet. There will be lots of court cases that is going to generate some laws over the next few years, but you do not want to become a legal precedent. Because of that, you should make sure that you're thinking properly and guided by your heart and ethical responsibilities when you're making your decisions.

Dave:

You need to support your employees. That means give them training, give them guidance, let them know. If there are employees that are worried about this, you need to give them emotional support as well to help them on this journey. I think it's important of leaders. You need a butt to kick and you need a back to slap. And it's important to keep learning because this stuff's changing all the time.

Dave:

Just in the last few days, OpenAI has pretty much exploded as a company internally when Sam Altman was fired. And it looked as if he might be joining again, but now he's joining Microsoft. And everything's changing so fast. And then two weeks ago at OpenAI's Dev Day, they introduced so many things that really, really changed the whole world of AI. And then you've got some-

Larry:

I've got to interject real quickly. We're recording this on November 20th. And by the time this airs in a couple of weeks, it'll be completely out of date. But I think the elements of your manifesto are timeless. Yeah, sorry.

Dave:

Yeah. Yeah,...