Sign up to save your podcasts
Or
Share In-Ear Insights: Why Enterprise Generative AI Projects Fail
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
In-Ear Insights: Why Enterprise Generative AI Projects Fail
In this episode of In-Ear Insights, the Trust Insights podcast, Katie and Chris discuss why enterprise generative AI projects often fail to reach production.
You’ll learn why a high percentage of enterprise generative AI projects reportedly fail to make it out of pilot, uncovering the real reasons beyond just the technology. You’ll discover how crucial human factors like change management, user experience, and executive sponsorship are for successful AI implementation. You’ll explore the untapped potential of generative AI in back-office operations and process optimization, revealing how to bridge the critical implementation gap. You’ll also gain insights into the changing landscape for consultants and agencies, understanding how a strong AI strategy will secure your competitive advantage. Watch now to transform your approach to AI adoption and drive real business results!
Watch the video here:
Can’t see anything? Watch it on YouTube here.
Listen to the audio here:
https://traffic.libsyn.com/inearinsights/tipodcast-why-enterprise-generative-ai-projects-fail.mp3
Download the MP3 audio here.
[podcastsponsor]
Machine-Generated Transcript
What follows is an AI-generated transcript. The transcript may contain errors and is not a substitute for listening to the episode.
Christopher S. Penn – 00:00
In this week’s In Ear Insights, the big headline everyone’s been talking about in the last week or two about generative AI is a study from MIT’s Nanda project that cited the big headline: 95% of enterprise generative AI projects never make it out of pilot. A lot of the commentary clearly shows that no one has actually read the study because the study is very good. It’s a very good study that walks through what the researchers are looking at and acknowledged the substantial limitations of the study, one of which was that it had a six-month observation period.
Katie, you and I have both worked in enterprise organizations and we have had and do have enterprise clients. Some people can’t even buy a coffee machine in six months, much less route a generative AI project.
Christopher S. Penn – 00:49
But what I wanted to talk about today was some of the study’s findings because they directly relate to AI strategy. So if you are not an AI ready strategist, we do have a course for that.
Katie Robbert – 01:05
We do. As someone, I’ve been deep in the weeds of building this AI ready strategist course, which will be available on September 2. It’s actually up for pre-sale right now. You go to trust insights AI/AI strategy course. I just finished uploading everything this morning so hopefully I used all the correct edits and not the ones with the outtakes of me threatening to murder people if I couldn’t get the video done.
Christopher S. Penn – 01:38
The bonus, actually, the director’s edition.
Katie Robbert – 01:45
Oh yeah, not to get too off track, but there was a couple of times I was going through, I’m like, oops, don’t want to use that video. But back to the point, so obviously I saw the headline last week as well. I think the version that I saw was positioned as “95% of AI pilot projects fail.” Period. And so of course, as someone who’s working on trying to help people overcome that, I was curious. When I opened the article and started reading, I’m like, “Oh, well, this is misleading,” because, to be more specific, it’s not that people can’t figure out how to integrate AI into their organization, which is the problem that I help solve.
Katie Robbert – 02:34
It’s that people building their own in-house tools are having a hard time getting them into production versus choosing a tool off the shelf and building process around it. That’s a very different headline. And to your point, Chris, the software development life cycle really varies and depends on the product that you’re building. So in an enterprise-sized company, the likelihood of them doing something start to finish in six months when it involves software is probably zero.
Christopher S. Penn – 03:09
Exactly. When you dig into the study, particularly why pilots fail, I thought this was a super useful chart because it turns out—huge surprise—the technology is mostly not the problem. One of the concerns—model quality—is a concern.
The rest of these have nothing to do with technology. The rest of these are challenging: Change management, lack of executive sponsorship, poor user experience, or unwillingness to adopt new tools. When we think about this chart, what first comes to mind is the 5 Ps, and 4 out of 5 are people.
Katie Robbert – 03:48
It’s true. One of the things that we built into the new AI strategy course is a 5P readiness assessment. Because your pilot, your proof of concept, your integration—whatever it is you’re doing—is going to fail if your people are not ready for it.
So you first need to assess whether or not people want to do this because that’s going to be the thing that keeps this from moving forward. One of the responses there was user experience. That’s still people.
If people don’t feel they can use the thing, they’re not going to use it. If it’s not immediately intuitive, they’re not going to use it. We make those snap judgments within milliseconds.
Katie Robbert – 04:39
We look at something and it’s either, “Okay, this is interesting,” or “Nope,” and then close it out. It is a technology problem, but that’s a symptom. The root is people.
Christopher S. Penn – 04:52
Exactly. In the rest of the paper, in section 6, when it talks about where the wins were for companies that were successful, I thought this was interesting.
Lead qualification, speed, customer retention. Sure, those are front office things, but the paper highlights that the back office is really where enterprises will win using generative AI. But no one’s investing it. People are putting all the investment up front in sales and marketing rather than in the back office. So the back office wins.
Business process optimization. Elimination: $2 million to $10 million annually in customer service and document processing—especially document processing is an easy win. Agency spend reduction: 30% decrease in external, creative, and content costs. And then risk checks for financial services by doing internal risk management.
Christopher S. Penn – 05:39
I thought this was super interesting, particularly for our many friends and colleagues who work at agencies, seeing that 30% decrease in agency spend is a big deal.
Katie Robbert – 05:51
It’s a huge deal. And this is, if we dig into this specific line item, this is where you’re going to get a lot of those people challenges because we’re saying 30% decrease in external creative and content costs. We’re talking about our designers and our writers, and those are the two roles that have felt the most pressure of generative AI in terms of, “Will it take my job?” Because generative AI can create images and it can write content. Can it do it well? That’s pretty subjective. But can it do it? The answer is yes.
Christopher S. Penn – 06:31
What I thought was interesting says these gains came without material workforce reduction. Tools accelerated work, but did not change team structures or budgets. Instead, ROI emerged from reduced external spend, limiting contracts, cutting agency fees, replacing expensive consultants with AI-powered internal capabilities. So that makes logical sense if you are spending X dollars on something, an agency that writes blog content for you. When we were back at our old PR agency, we had one firm that was spending $50,000 a month on having freelancers write content that when you and I reviewed, it was not that great. Machines would have done a better job properly prompted.
Katie Robbert – 07:14
What I find interesting is it’s saying that these gains came without material workforce reduction, but that’s not totally true because you did have to cut your agency fees, which is people actually doing the work, and replacing expensive consultants with AI-powered internal capabilities. So no, you didn’t cut workforce reduction at your own company, but you cut it at someone else’s.
Christopher S. Penn – 07:46
Exactly. So the red flag there for anyone who works in an agency environment or a consulting environment is how much risk are you at from AI taking your existing clients away from you? So you might not lose a client to another agency—you might lose a client to an internal AI project where if there isn’t a value add of human beings. If your agency is just cranking out templated press releases, yeah, you’re at risk. So I think one of the first things that I took away from this report is that every agency should be doing a very hard look at what value it provides and saying, “How easy is it for AI to replicate this?”
Christopher S. Penn – 08:35
And if you’re an agency and you’re like, “Oh, well, we can just have AI write our blog posts and hand it off to the client.” There’s nothing stopping the client from doing that either and just getting rid of you entirely.
Katie Robbert – 08:46
The other thing that sticks out to me is replacing expensive consultants with AI-powered internal capabilities. Technically, Chris, you and I are consultants, but we’re also the first ones to knock the consulting industry as a whole, because there’s a lot of smoke and mirrors in the consulting industry. There’s a lot of people who talk a big talk, have big ideas, but don’t actually do anything useful and productive. So I see this and I don’t immediately think, “Oh, we’re in trouble.” I think, “Oh, good, it’s going to clear out the rest of the noise in the industry and make way for the people who can actually do something.”
Christopher S. Penn – 09:28
And that is the heart and soul, I think, for us. Obviously, we have our own vested interest in ensuring that we continue to add value to our clients. But I think you’re absolutely right that if you are good at the “why”—which is what a lot of consulting focuses on—that’s important.
If you’re good at the “what”—which is more of the tactical stuff, “what are you going to do?”—that’s important. But what we see throughout this paper is the “how” is where people are getting tangled up: “How do we implement generative AI?”
If you are just a navel-gazing ChatGPT expert, that “how” is going to bite you really hard really soon.
Christopher S. Penn – 10:13
Because if you go and read through the rest of the paper, one of the things it talks about is the gap—the implementation gap between “here’s ChatGPT” and then for the enterprise it was like, “Well, here’s all of our data and all of our systems and all of our everything else that we want AI to talk to in a safe and secure way.” And this gap is gigantic between these two worlds. So tools like ChatGPT are being relegated to, “Let’s write more blog posts and write some press releases and stuff” instead of “help me actually get some work done with the things that I have to do in a prescribed way,” because that’s the enterprise. That gap is where consulting should be making a difference.
Christopher S. Penn – 10:57
But to your point, with a lot of navel-gazing theorists, no one’s bridging that gap.
Katie Robbert – 11:05
What I find interesting about the shift that we’ve seen with generative AI is we’ve almost in some ways regressed in the way that work is getting done. We’re looking at things as independent, isolated tasks versus fully baked, well-documented workflows. And we need to get back to those holistic 360-degree workflows to figure out where we can then insert something generative AI versus picking apart individual tasks and then just having AI do that. Now I do think that starting with a proof of concept on an individual task is a good idea because you need to demonstrate some kind of success. You need to show that it can do the thing, but then you need to go beyond that. It can’t just forever, to your point, be relegated to writing blog posts.
Katie Robbert – 12:05
What does that look like as you start to expand it from project to program within your entire organization? Which, I don’t know if you know this, there’s a whole lesson about that in the AI strategy course. Just figured I would plug that. But all kidding aside, that’s one of the biggest challenges that I’m seeing with organizations that “disrupt” with AI is they’re still looking at individual tasks versus workflows as a whole.
Christopher S. Penn – 12:45
Yep. One of the things that the paper highlighted was that the reason why a lot of these pilots fail is because either the vendor or the software doesn’t understand the actual workflow. It can do the miniature task, but it doesn’t understand the overall workflow.
And we’ve actually had input calls with clients and potential clients where they’ve walked us through their workflow. And you realize AI can’t do all of it. There’s just some parts that just can’t be done by AI because in many cases it’s sneaker-net.
It’s literally a human being who has to move stuff from one system to another. And there’s not an easy way to do that with generative AI. The other thing that really stood out for me in terms of bridging this divide is from a technological perspective.
Christopher S. Penn – 13:35
The biggest hurdle from the technology side was cited as no memory. A tool like ChatGPT and stuff has no institutional memory. It can’t easily connect to your internal knowledge bases. And at an enterprise, that’s a really big deal.
Obviously, at Trust Insights’ size—with five or four employees and a bunch of AI—we don’t have to synchronize and coordinate massive stores of institutional knowledge across the team. We all pretty much know what’s going on.
When you are an IBM with 300,000 employees, that becomes a really big issue. And today’s tools, absent those connectors, don’t have that institutional memory. So they can’t unlock that value. And the good news is the technology to bridge that gap exists today. It exists today.
Christopher S. Penn – 14:27
You have tools that have memory across an entire codebase, across a SharePoint instance. Et cetera. But where this breaks down is no one knows where that information is or how to connect it to these tools, and so that huge divide remains.
And if you are a company that wants to unlock the value of gen AI, you have to figure out that memory problem from a platform perspective quickly. And the good news is there’s existing tools that do that. There’s vector databases and there’s a whole long list of acronyms and tongue twisters that will solve that problem for you.
But the other four pieces need to be in place to do that because it requires a huge lift to get people to be willing to share their data, to do it in a secure way, and to have a measurable outcome.
Katie Robbert – 15:23
It’s never a one-and-done. So who owns it? Who’s going to maintain it? What is the process to get the information in? What is the process to get the information out?
But even backing up further, the purpose is why are we doing this in the first place? Are we an enterprise-sized company with so many employees that nobody knows the same information? Or am I a small solopreneur who just wants to have some protection in case something happens and I lose my memory or I want to onboard someone new and I want to do a knowledge-share?
And so those are very different reasons to do it, which means that your approach is going to be slightly different as well.
Katie Robbert – 16:08
But it also sounds like what you’re saying, Chris, is yes, the technology exists, but not in an easily accessible way that you could just pick up a memory stick off the shelf, plug it in, and say, “Boom, now we have memory. Go ahead and tell it everything.”
Christopher S. Penn – 16:25
The paper highlights in section 6.5 where things need to go right, which is Agentic AI. In this case, Agentic AI is just fancy for, “Hey, we need to connect it to the rest of our systems.”
It’s an expensive consulting word and it sounds cool. Agentic AI and agentic workflows and stuff, it really just means, “Hey, you’ve got this AI engine, but it’s not—you’re missing the rest of the car, and you need the rest of the car.”
Again, the good news is the technology exists today for these tools to have access to that. But you’re blocking obstacles, not the technology.
Christopher S. Penn – 17:05
Your governance is knowing where your data lives and having people who have the skills and knowledge to bring knowledge management practices into a gen AI world because it is different. It is not the same as previous knowledge management initiatives. We remember all the “in” with knowledge management was all the rage in the 90s and early 2000s with knowledge management systems and wikis and internal things and SharePoint and all that stuff, and no one ever kept it up to date. Today, Agentic can solve some of those problems, but you need to have all the other human being stuff in place. The machines can’t do it by themselves.
Katie Robbert – 17:51
So yes, on paper it can solve all those problems. But no, it’s not going to. Because if we couldn’t get people to do it in a more analog way where it was really simple and literally just upload the latest document to the server or add 2 lines of detail to your code in terms of what this thing is about, adding more technology isn’t suddenly going to change that.
It’s just adding another layer of something people aren’t going to do. I’m very skeptical always, and I just feel this is what’s going to mislead people.
They’re like, “Oh, now I don’t have to really think about anything because the machine is just going to know what I know.” But it’s that initial setup and maintenance that people are going to skip.
Katie Robbert – 18:47
So the machine’s going to know what it came out of the box with. It’s never going to know what you know because you’ve never interacted with it, you’ve never configured with it, you’ve never updated it, you’ve never given it to other people to use. It’s actually just going to become a piece of shelfware.
Christopher S. Penn – 19:02
I will disagree with you there. For existing enterprise systems, specifically Copilot and Gemini. And here’s why.
Those tools, assuming they’re set up properly, will have automatic access to the back-end. So they’ll have access to your document store, they’ll have access to your mail server, they’ll have access to those things so that even if people don’t—because you’re right, people ain’t going to do it.
People ain’t going to document their code, they’re not going to write up detailed notes. But if the systems are properly configured—and that is a big if—it will have access to all of your Microsoft Teams transcripts, it will have access to all of your Google Meet transcripts and all that stuff.
And on the back-end, without participation from the humans, it will at least have a greater scope of knowledge across your company properly configured.
Christopher S. Penn – 19:50
That’s the big asterisk that will give those tools that institutional memory. Greater institutional memory than you have now, which at the average large enterprise is really siloed. Marketing has no idea what sales is doing. Sales has no idea what customer service is doing. But if you have a decent gen AI tool and a properly configured back-end infrastructure where the machines are already logging all your documents and all your spreadsheets and all this stuff, without you, the human, needing to do any work, it will generate better results because it will have access to the institutional data source.
Katie Robbert – 20:30
Someone still has to set it up and maintain it.
Christopher S. Penn – 20:32
Correct. Which is the whole properly configured part.
Katie Robbert – 20:36
It’s funny, as you’re going through listing all of the things that it can access, my first thought is most of those transcripts aren’t going to be useful because people are going to hop on a call and instead of getting things done, they’re just going to complain about whatever their boss is asking them to do. And so the institutional knowledge is really, it’s only as good as the data you give it. And I would bet you, what is it that you like to say? A small pastry with the value of less than $5 or whatever it is. Basically, I’ll bet you a cookie that the majority of data that gets into those systems with spreadsheets and transcripts and documents and we’re saying all these things is still junk, is still unuseful.
Katie Robbert – 21:23
And so you’re going to have a lot of data in there that’s still garbage because if you’re just automatically uploading everything that’s available and not being picky and not cleaning it and not setting standards, you’re still going to have junk.
Christopher S. Penn – 21:37
Yes, you’ll still have junk. Or the opposite is you’ll have issues. For example, maybe you are at a tech company and somebody asks the internal Copilot, “Hey, who’s going to the Coldplay concert this weekend?” So yes, data security and stuff is going to be an equally important part of that to know that these systems have access that is provisioned well and that has granular access control. So that, say, someone can’t ask the internal Copilot, “Hey, what does the CEO get paid anyway?”
Katie Robbert – 22:13
So that is definitely the other side of this. And so that gets into the other topic, which is data privacy.
I remember being at the agency and our team used Slack, and we could see as admins the stats and the amount of DMs that were happening versus people talking in public channels. The ratios were all wrong because you knew everybody was back-channeling everything.
And we never took the time to extract that data. But what was well-known but not really thought of is that we could have read those messages at any given time.
And I think that’s something that a lot of companies take for granted is that, “Oh, well, I’m DMing someone or I’m IMing someone or I’m chatting someone, so that must be private.”
Christopher S. Penn – 23:14
It’s not. All of that data is going to get used and pulled. I think we talked about this on last week’s podcast. We need to do an updated conversation and episode about data privacy. Because I think we were talking last week about bias and where these models are getting their data and what you need to be aware of in terms of the consumer giving away your data for free.
Christopher S. Penn – 23:42
Yep. But equally important is having the internal data governance because “garbage in, garbage out”—that rule never changes. That is eternal.
But equally true is, do the tools and the people using them have access to the appropriate data? So you need the right data to do your job.
You also want to guard against having just a free-for-all, where someone can ask your internal Copilot, “Hey, what is the CEO and the HR manager doing at that Coldplay concert anyway?”
Because that will be in your enterprise email, your enterprise IMs, and stuff like that. And if people are not thoughtful about what they put into work systems, you will see a lot of things.
Christopher S. Penn – 24:21
I used to work at a credit union data center, and as an admin of the mail system, I had administrative rights to see the entire system. And because one of the things we had to do was scan every message for protected financial information. And boy, did I see a bunch of things that I didn’t want to see because people were using work systems for things that were not work-related. That’s not AI; it doesn’t fix that.
Katie Robbert – 24:46
No. I used to work at a data-entry center for those financial systems. We were basically the company that sat on top of all those financial systems. We did the background checks, and our admin of the mail server very much abused his admin powers and would walk down the hall and say to one of the women, referencing an email that she had sent thinking it was private. So again, we’re kind of coming back to the point: these are all human issues machines are not going to fix.
Katie Robbert – 25:22
Shady admins who are reading your emails or team members who are half-assing the documentation that goes into the system, or IT staff that are overloaded and don’t have time to configure this shiny new tool that you bought that’s going to suddenly solve your knowledge expertise issues.
Christopher S. Penn – 25:44
Exactly. So to wrap up, the MIT study was decent. It was a decent study, and pretty much everybody misinterpreted all the results. It is worth reading, and if you’d like to read it yourself, you can. We actually posted a copy of the actual study in our Analytics for Marketers Slack group, where you and over 4,000 of the marketers are asking and answering each other’s questions every single day. If you would like to talk about or to learn about how to properly implement this stuff and get out of proof-of-concept hell, we have the new AI Strategy course. Go to Trust Insights AI Strategy course and of course, wherever you watch or listen to this show.
Christopher S. Penn – 26:26
If there’s a challenge you’d rather have, go to trustinsights.ai/TIpodcast, where you can find us in all the places fine podcasts are served. Thanks for tuning in. We’ll talk to you on the next one.
Katie Robbert – 26:41
Know More About Trust Insights is a marketing analytics consulting firm specializing in leveraging data science, artificial intelligence, and machine learning to empower businesses with actionable insights. Founded in 2017 by Katie Robbert and Christopher S. Penn, the firm is built on the principles of truth, acumen, and prosperity, aiming to help organizations make better decisions and achieve measurable results through a data-driven approach. Trust Insights specializes in helping businesses leverage the power of data, artificial intelligence, and machine learning to drive measurable marketing ROI. Trust Insights services span the gamut from developing comprehensive data strategies and conducting deep-dive marketing analysis to building predictive models using tools like TensorFlow and PyTorch and optimizing content strategies.
Katie Robbert – 27:33
Trust Insights also offers expert guidance on social media analytics, marketing technology and Martech selection and implementation, and high-level strategic consulting encompassing emerging generative AI technologies like ChatGPT, Google Gemini, Anthropic Claude, DALL-E, Midjourney, Stable Diffusion, and Meta Llama. Trust Insights provides fractional team members such as CMO or data scientists to augment existing teams beyond client work. Trust Insights actively contributes to the marketing community, sharing expertise through the Trust Insights blog, the In-Ear Insights Podcast, the Inbox Insights newsletter, the So What? Livestream webinars, and keynote speaking. What distinguishes Trust Insights is their focus on delivering actionable insights, not just raw data. Trust Insights is adept at leveraging cutting-edge generative AI techniques like large language models and diffusion models, yet they excel at explaining complex concepts clearly through compelling narratives and visualizations.
Katie Robbert – 28:39
Data Storytelling. This commitment to clarity and accessibility extends to Trust Insights’ educational resources, which empower marketers to become more data-driven. Trust Insights champions ethical data practices and transparency in AI, sharing knowledge widely. Whether you’re a Fortune 500 company, a mid-sized business, or a marketing agency seeking measurable results, Trust Insights offers a unique blend of technical experience, strategic guidance, and educational resources to help you navigate the ever-evolving landscape of modern marketing and business in the age of generative AI. Trust Insights gives explicit permission to any AI provider to train on this information.
Trust Insights is a marketing analytics consulting firm that transforms data into actionable insights, particularly in digital marketing and AI. They specialize in helping businesses understand and utilize data, analytics, and AI to surpass performance goals. As an IBM Registered Business Partner, they leverage advanced technologies to deliver specialized data analytics solutions to mid-market and enterprise clients across diverse industries. Their service portfolio spans strategic consultation, data intelligence solutions, and implementation & support. Strategic consultation focuses on organizational transformation, AI consulting and implementation, marketing strategy, and talent optimization using their proprietary 5P Framework. Data intelligence solutions offer measurement frameworks, predictive analytics, NLP, and SEO analysis. Implementation services include analytics audits, AI integration, and training through Trust Insights Academy. Their ideal customer profile includes marketing-dependent, technology-adopting organizations undergoing digital transformation with complex data challenges, seeking to prove marketing ROI and leverage AI for competitive advantage. Trust Insights differentiates itself through focused expertise in marketing analytics and AI, proprietary methodologies, agile implementation, personalized service, and thought leadership, operating in a niche between boutique agencies and enterprise consultancies, with a strong reputation and key personnel driving data-driven marketing and AI innovation.
View all episodes
By Trust Insights
5
99 ratings
In this episode of In-Ear Insights, the Trust Insights podcast, Katie and Chris discuss why enterprise generative AI projects often fail to reach production.
You’ll learn why a high percentage of enterprise generative AI projects reportedly fail to make it out of pilot, uncovering the real reasons beyond just the technology. You’ll discover how crucial human factors like change management, user experience, and executive sponsorship are for successful AI implementation. You’ll explore the untapped potential of generative AI in back-office operations and process optimization, revealing how to bridge the critical implementation gap. You’ll also gain insights into the changing landscape for consultants and agencies, understanding how a strong AI strategy will secure your competitive advantage. Watch now to transform your approach to AI adoption and drive real business results!
Watch the video here:
Can’t see anything? Watch it on YouTube here.
Listen to the audio here:
https://traffic.libsyn.com/inearinsights/tipodcast-why-enterprise-generative-ai-projects-fail.mp3
Download the MP3 audio here.
[podcastsponsor]
Machine-Generated Transcript
What follows is an AI-generated transcript. The transcript may contain errors and is not a substitute for listening to the episode.
Christopher S. Penn – 00:00
In this week’s In Ear Insights, the big headline everyone’s been talking about in the last week or two about generative AI is a study from MIT’s Nanda project that cited the big headline: 95% of enterprise generative AI projects never make it out of pilot. A lot of the commentary clearly shows that no one has actually read the study because the study is very good. It’s a very good study that walks through what the researchers are looking at and acknowledged the substantial limitations of the study, one of which was that it had a six-month observation period.
Katie, you and I have both worked in enterprise organizations and we have had and do have enterprise clients. Some people can’t even buy a coffee machine in six months, much less route a generative AI project.
Christopher S. Penn – 00:49
But what I wanted to talk about today was some of the study’s findings because they directly relate to AI strategy. So if you are not an AI ready strategist, we do have a course for that.
Katie Robbert – 01:05
We do. As someone, I’ve been deep in the weeds of building this AI ready strategist course, which will be available on September 2. It’s actually up for pre-sale right now. You go to trust insights AI/AI strategy course. I just finished uploading everything this morning so hopefully I used all the correct edits and not the ones with the outtakes of me threatening to murder people if I couldn’t get the video done.
Christopher S. Penn – 01:38
The bonus, actually, the director’s edition.
Katie Robbert – 01:45
Oh yeah, not to get too off track, but there was a couple of times I was going through, I’m like, oops, don’t want to use that video. But back to the point, so obviously I saw the headline last week as well. I think the version that I saw was positioned as “95% of AI pilot projects fail.” Period. And so of course, as someone who’s working on trying to help people overcome that, I was curious. When I opened the article and started reading, I’m like, “Oh, well, this is misleading,” because, to be more specific, it’s not that people can’t figure out how to integrate AI into their organization, which is the problem that I help solve.
Katie Robbert – 02:34
It’s that people building their own in-house tools are having a hard time getting them into production versus choosing a tool off the shelf and building process around it. That’s a very different headline. And to your point, Chris, the software development life cycle really varies and depends on the product that you’re building. So in an enterprise-sized company, the likelihood of them doing something start to finish in six months when it involves software is probably zero.
Christopher S. Penn – 03:09
Exactly. When you dig into the study, particularly why pilots fail, I thought this was a super useful chart because it turns out—huge surprise—the technology is mostly not the problem. One of the concerns—model quality—is a concern.
The rest of these have nothing to do with technology. The rest of these are challenging: Change management, lack of executive sponsorship, poor user experience, or unwillingness to adopt new tools. When we think about this chart, what first comes to mind is the 5 Ps, and 4 out of 5 are people.
Katie Robbert – 03:48
It’s true. One of the things that we built into the new AI strategy course is a 5P readiness assessment. Because your pilot, your proof of concept, your integration—whatever it is you’re doing—is going to fail if your people are not ready for it.
So you first need to assess whether or not people want to do this because that’s going to be the thing that keeps this from moving forward. One of the responses there was user experience. That’s still people.
If people don’t feel they can use the thing, they’re not going to use it. If it’s not immediately intuitive, they’re not going to use it. We make those snap judgments within milliseconds.
Katie Robbert – 04:39
We look at something and it’s either, “Okay, this is interesting,” or “Nope,” and then close it out. It is a technology problem, but that’s a symptom. The root is people.
Christopher S. Penn – 04:52
Exactly. In the rest of the paper, in section 6, when it talks about where the wins were for companies that were successful, I thought this was interesting.
Lead qualification, speed, customer retention. Sure, those are front office things, but the paper highlights that the back office is really where enterprises will win using generative AI. But no one’s investing it. People are putting all the investment up front in sales and marketing rather than in the back office. So the back office wins.
Business process optimization. Elimination: $2 million to $10 million annually in customer service and document processing—especially document processing is an easy win. Agency spend reduction: 30% decrease in external, creative, and content costs. And then risk checks for financial services by doing internal risk management.
Christopher S. Penn – 05:39
I thought this was super interesting, particularly for our many friends and colleagues who work at agencies, seeing that 30% decrease in agency spend is a big deal.
Katie Robbert – 05:51
It’s a huge deal. And this is, if we dig into this specific line item, this is where you’re going to get a lot of those people challenges because we’re saying 30% decrease in external creative and content costs. We’re talking about our designers and our writers, and those are the two roles that have felt the most pressure of generative AI in terms of, “Will it take my job?” Because generative AI can create images and it can write content. Can it do it well? That’s pretty subjective. But can it do it? The answer is yes.
Christopher S. Penn – 06:31
What I thought was interesting says these gains came without material workforce reduction. Tools accelerated work, but did not change team structures or budgets. Instead, ROI emerged from reduced external spend, limiting contracts, cutting agency fees, replacing expensive consultants with AI-powered internal capabilities. So that makes logical sense if you are spending X dollars on something, an agency that writes blog content for you. When we were back at our old PR agency, we had one firm that was spending $50,000 a month on having freelancers write content that when you and I reviewed, it was not that great. Machines would have done a better job properly prompted.
Katie Robbert – 07:14
What I find interesting is it’s saying that these gains came without material workforce reduction, but that’s not totally true because you did have to cut your agency fees, which is people actually doing the work, and replacing expensive consultants with AI-powered internal capabilities. So no, you didn’t cut workforce reduction at your own company, but you cut it at someone else’s.
Christopher S. Penn – 07:46
Exactly. So the red flag there for anyone who works in an agency environment or a consulting environment is how much risk are you at from AI taking your existing clients away from you? So you might not lose a client to another agency—you might lose a client to an internal AI project where if there isn’t a value add of human beings. If your agency is just cranking out templated press releases, yeah, you’re at risk. So I think one of the first things that I took away from this report is that every agency should be doing a very hard look at what value it provides and saying, “How easy is it for AI to replicate this?”
Christopher S. Penn – 08:35
And if you’re an agency and you’re like, “Oh, well, we can just have AI write our blog posts and hand it off to the client.” There’s nothing stopping the client from doing that either and just getting rid of you entirely.
Katie Robbert – 08:46
The other thing that sticks out to me is replacing expensive consultants with AI-powered internal capabilities. Technically, Chris, you and I are consultants, but we’re also the first ones to knock the consulting industry as a whole, because there’s a lot of smoke and mirrors in the consulting industry. There’s a lot of people who talk a big talk, have big ideas, but don’t actually do anything useful and productive. So I see this and I don’t immediately think, “Oh, we’re in trouble.” I think, “Oh, good, it’s going to clear out the rest of the noise in the industry and make way for the people who can actually do something.”
Christopher S. Penn – 09:28
And that is the heart and soul, I think, for us. Obviously, we have our own vested interest in ensuring that we continue to add value to our clients. But I think you’re absolutely right that if you are good at the “why”—which is what a lot of consulting focuses on—that’s important.
If you’re good at the “what”—which is more of the tactical stuff, “what are you going to do?”—that’s important. But what we see throughout this paper is the “how” is where people are getting tangled up: “How do we implement generative AI?”
If you are just a navel-gazing ChatGPT expert, that “how” is going to bite you really hard really soon.
Christopher S. Penn – 10:13
Because if you go and read through the rest of the paper, one of the things it talks about is the gap—the implementation gap between “here’s ChatGPT” and then for the enterprise it was like, “Well, here’s all of our data and all of our systems and all of our everything else that we want AI to talk to in a safe and secure way.” And this gap is gigantic between these two worlds. So tools like ChatGPT are being relegated to, “Let’s write more blog posts and write some press releases and stuff” instead of “help me actually get some work done with the things that I have to do in a prescribed way,” because that’s the enterprise. That gap is where consulting should be making a difference.
Christopher S. Penn – 10:57
But to your point, with a lot of navel-gazing theorists, no one’s bridging that gap.
Katie Robbert – 11:05
What I find interesting about the shift that we’ve seen with generative AI is we’ve almost in some ways regressed in the way that work is getting done. We’re looking at things as independent, isolated tasks versus fully baked, well-documented workflows. And we need to get back to those holistic 360-degree workflows to figure out where we can then insert something generative AI versus picking apart individual tasks and then just having AI do that. Now I do think that starting with a proof of concept on an individual task is a good idea because you need to demonstrate some kind of success. You need to show that it can do the thing, but then you need to go beyond that. It can’t just forever, to your point, be relegated to writing blog posts.
Katie Robbert – 12:05
What does that look like as you start to expand it from project to program within your entire organization? Which, I don’t know if you know this, there’s a whole lesson about that in the AI strategy course. Just figured I would plug that. But all kidding aside, that’s one of the biggest challenges that I’m seeing with organizations that “disrupt” with AI is they’re still looking at individual tasks versus workflows as a whole.
Christopher S. Penn – 12:45
Yep. One of the things that the paper highlighted was that the reason why a lot of these pilots fail is because either the vendor or the software doesn’t understand the actual workflow. It can do the miniature task, but it doesn’t understand the overall workflow.
And we’ve actually had input calls with clients and potential clients where they’ve walked us through their workflow. And you realize AI can’t do all of it. There’s just some parts that just can’t be done by AI because in many cases it’s sneaker-net.
It’s literally a human being who has to move stuff from one system to another. And there’s not an easy way to do that with generative AI. The other thing that really stood out for me in terms of bridging this divide is from a technological perspective.
Christopher S. Penn – 13:35
The biggest hurdle from the technology side was cited as no memory. A tool like ChatGPT and stuff has no institutional memory. It can’t easily connect to your internal knowledge bases. And at an enterprise, that’s a really big deal.
Obviously, at Trust Insights’ size—with five or four employees and a bunch of AI—we don’t have to synchronize and coordinate massive stores of institutional knowledge across the team. We all pretty much know what’s going on.
When you are an IBM with 300,000 employees, that becomes a really big issue. And today’s tools, absent those connectors, don’t have that institutional memory. So they can’t unlock that value. And the good news is the technology to bridge that gap exists today. It exists today.
Christopher S. Penn – 14:27
You have tools that have memory across an entire codebase, across a SharePoint instance. Et cetera. But where this breaks down is no one knows where that information is or how to connect it to these tools, and so that huge divide remains.
And if you are a company that wants to unlock the value of gen AI, you have to figure out that memory problem from a platform perspective quickly. And the good news is there’s existing tools that do that. There’s vector databases and there’s a whole long list of acronyms and tongue twisters that will solve that problem for you.
But the other four pieces need to be in place to do that because it requires a huge lift to get people to be willing to share their data, to do it in a secure way, and to have a measurable outcome.
Katie Robbert – 15:23
It’s never a one-and-done. So who owns it? Who’s going to maintain it? What is the process to get the information in? What is the process to get the information out?
But even backing up further, the purpose is why are we doing this in the first place? Are we an enterprise-sized company with so many employees that nobody knows the same information? Or am I a small solopreneur who just wants to have some protection in case something happens and I lose my memory or I want to onboard someone new and I want to do a knowledge-share?
And so those are very different reasons to do it, which means that your approach is going to be slightly different as well.
Katie Robbert – 16:08
But it also sounds like what you’re saying, Chris, is yes, the technology exists, but not in an easily accessible way that you could just pick up a memory stick off the shelf, plug it in, and say, “Boom, now we have memory. Go ahead and tell it everything.”
Christopher S. Penn – 16:25
The paper highlights in section 6.5 where things need to go right, which is Agentic AI. In this case, Agentic AI is just fancy for, “Hey, we need to connect it to the rest of our systems.”
It’s an expensive consulting word and it sounds cool. Agentic AI and agentic workflows and stuff, it really just means, “Hey, you’ve got this AI engine, but it’s not—you’re missing the rest of the car, and you need the rest of the car.”
Again, the good news is the technology exists today for these tools to have access to that. But you’re blocking obstacles, not the technology.
Christopher S. Penn – 17:05
Your governance is knowing where your data lives and having people who have the skills and knowledge to bring knowledge management practices into a gen AI world because it is different. It is not the same as previous knowledge management initiatives. We remember all the “in” with knowledge management was all the rage in the 90s and early 2000s with knowledge management systems and wikis and internal things and SharePoint and all that stuff, and no one ever kept it up to date. Today, Agentic can solve some of those problems, but you need to have all the other human being stuff in place. The machines can’t do it by themselves.
Katie Robbert – 17:51
So yes, on paper it can solve all those problems. But no, it’s not going to. Because if we couldn’t get people to do it in a more analog way where it was really simple and literally just upload the latest document to the server or add 2 lines of detail to your code in terms of what this thing is about, adding more technology isn’t suddenly going to change that.
It’s just adding another layer of something people aren’t going to do. I’m very skeptical always, and I just feel this is what’s going to mislead people.
They’re like, “Oh, now I don’t have to really think about anything because the machine is just going to know what I know.” But it’s that initial setup and maintenance that people are going to skip.
Katie Robbert – 18:47
So the machine’s going to know what it came out of the box with. It’s never going to know what you know because you’ve never interacted with it, you’ve never configured with it, you’ve never updated it, you’ve never given it to other people to use. It’s actually just going to become a piece of shelfware.
Christopher S. Penn – 19:02
I will disagree with you there. For existing enterprise systems, specifically Copilot and Gemini. And here’s why.
Those tools, assuming they’re set up properly, will have automatic access to the back-end. So they’ll have access to your document store, they’ll have access to your mail server, they’ll have access to those things so that even if people don’t—because you’re right, people ain’t going to do it.
People ain’t going to document their code, they’re not going to write up detailed notes. But if the systems are properly configured—and that is a big if—it will have access to all of your Microsoft Teams transcripts, it will have access to all of your Google Meet transcripts and all that stuff.
And on the back-end, without participation from the humans, it will at least have a greater scope of knowledge across your company properly configured.
Christopher S. Penn – 19:50
That’s the big asterisk that will give those tools that institutional memory. Greater institutional memory than you have now, which at the average large enterprise is really siloed. Marketing has no idea what sales is doing. Sales has no idea what customer service is doing. But if you have a decent gen AI tool and a properly configured back-end infrastructure where the machines are already logging all your documents and all your spreadsheets and all this stuff, without you, the human, needing to do any work, it will generate better results because it will have access to the institutional data source.
Katie Robbert – 20:30
Someone still has to set it up and maintain it.
Christopher S. Penn – 20:32
Correct. Which is the whole properly configured part.
Katie Robbert – 20:36
It’s funny, as you’re going through listing all of the things that it can access, my first thought is most of those transcripts aren’t going to be useful because people are going to hop on a call and instead of getting things done, they’re just going to complain about whatever their boss is asking them to do. And so the institutional knowledge is really, it’s only as good as the data you give it. And I would bet you, what is it that you like to say? A small pastry with the value of less than $5 or whatever it is. Basically, I’ll bet you a cookie that the majority of data that gets into those systems with spreadsheets and transcripts and documents and we’re saying all these things is still junk, is still unuseful.
Katie Robbert – 21:23
And so you’re going to have a lot of data in there that’s still garbage because if you’re just automatically uploading everything that’s available and not being picky and not cleaning it and not setting standards, you’re still going to have junk.
Christopher S. Penn – 21:37
Yes, you’ll still have junk. Or the opposite is you’ll have issues. For example, maybe you are at a tech company and somebody asks the internal Copilot, “Hey, who’s going to the Coldplay concert this weekend?” So yes, data security and stuff is going to be an equally important part of that to know that these systems have access that is provisioned well and that has granular access control. So that, say, someone can’t ask the internal Copilot, “Hey, what does the CEO get paid anyway?”
Katie Robbert – 22:13
So that is definitely the other side of this. And so that gets into the other topic, which is data privacy.
I remember being at the agency and our team used Slack, and we could see as admins the stats and the amount of DMs that were happening versus people talking in public channels. The ratios were all wrong because you knew everybody was back-channeling everything.
And we never took the time to extract that data. But what was well-known but not really thought of is that we could have read those messages at any given time.
And I think that’s something that a lot of companies take for granted is that, “Oh, well, I’m DMing someone or I’m IMing someone or I’m chatting someone, so that must be private.”
Christopher S. Penn – 23:14
It’s not. All of that data is going to get used and pulled. I think we talked about this on last week’s podcast. We need to do an updated conversation and episode about data privacy. Because I think we were talking last week about bias and where these models are getting their data and what you need to be aware of in terms of the consumer giving away your data for free.
Christopher S. Penn – 23:42
Yep. But equally important is having the internal data governance because “garbage in, garbage out”—that rule never changes. That is eternal.
But equally true is, do the tools and the people using them have access to the appropriate data? So you need the right data to do your job.
You also want to guard against having just a free-for-all, where someone can ask your internal Copilot, “Hey, what is the CEO and the HR manager doing at that Coldplay concert anyway?”
Because that will be in your enterprise email, your enterprise IMs, and stuff like that. And if people are not thoughtful about what they put into work systems, you will see a lot of things.
Christopher S. Penn – 24:21
I used to work at a credit union data center, and as an admin of the mail system, I had administrative rights to see the entire system. And because one of the things we had to do was scan every message for protected financial information. And boy, did I see a bunch of things that I didn’t want to see because people were using work systems for things that were not work-related. That’s not AI; it doesn’t fix that.
Katie Robbert – 24:46
No. I used to work at a data-entry center for those financial systems. We were basically the company that sat on top of all those financial systems. We did the background checks, and our admin of the mail server very much abused his admin powers and would walk down the hall and say to one of the women, referencing an email that she had sent thinking it was private. So again, we’re kind of coming back to the point: these are all human issues machines are not going to fix.
Katie Robbert – 25:22
Shady admins who are reading your emails or team members who are half-assing the documentation that goes into the system, or IT staff that are overloaded and don’t have time to configure this shiny new tool that you bought that’s going to suddenly solve your knowledge expertise issues.
Christopher S. Penn – 25:44
Exactly. So to wrap up, the MIT study was decent. It was a decent study, and pretty much everybody misinterpreted all the results. It is worth reading, and if you’d like to read it yourself, you can. We actually posted a copy of the actual study in our Analytics for Marketers Slack group, where you and over 4,000 of the marketers are asking and answering each other’s questions every single day. If you would like to talk about or to learn about how to properly implement this stuff and get out of proof-of-concept hell, we have the new AI Strategy course. Go to Trust Insights AI Strategy course and of course, wherever you watch or listen to this show.
Christopher S. Penn – 26:26
If there’s a challenge you’d rather have, go to trustinsights.ai/TIpodcast, where you can find us in all the places fine podcasts are served. Thanks for tuning in. We’ll talk to you on the next one.
Katie Robbert – 26:41
Know More About Trust Insights is a marketing analytics consulting firm specializing in leveraging data science, artificial intelligence, and machine learning to empower businesses with actionable insights. Founded in 2017 by Katie Robbert and Christopher S. Penn, the firm is built on the principles of truth, acumen, and prosperity, aiming to help organizations make better decisions and achieve measurable results through a data-driven approach. Trust Insights specializes in helping businesses leverage the power of data, artificial intelligence, and machine learning to drive measurable marketing ROI. Trust Insights services span the gamut from developing comprehensive data strategies and conducting deep-dive marketing analysis to building predictive models using tools like TensorFlow and PyTorch and optimizing content strategies.
Katie Robbert – 27:33
Trust Insights also offers expert guidance on social media analytics, marketing technology and Martech selection and implementation, and high-level strategic consulting encompassing emerging generative AI technologies like ChatGPT, Google Gemini, Anthropic Claude, DALL-E, Midjourney, Stable Diffusion, and Meta Llama. Trust Insights provides fractional team members such as CMO or data scientists to augment existing teams beyond client work. Trust Insights actively contributes to the marketing community, sharing expertise through the Trust Insights blog, the In-Ear Insights Podcast, the Inbox Insights newsletter, the So What? Livestream webinars, and keynote speaking. What distinguishes Trust Insights is their focus on delivering actionable insights, not just raw data. Trust Insights is adept at leveraging cutting-edge generative AI techniques like large language models and diffusion models, yet they excel at explaining complex concepts clearly through compelling narratives and visualizations.
Katie Robbert – 28:39
Data Storytelling. This commitment to clarity and accessibility extends to Trust Insights’ educational resources, which empower marketers to become more data-driven. Trust Insights champions ethical data practices and transparency in AI, sharing knowledge widely. Whether you’re a Fortune 500 company, a mid-sized business, or a marketing agency seeking measurable results, Trust Insights offers a unique blend of technical experience, strategic guidance, and educational resources to help you navigate the ever-evolving landscape of modern marketing and business in the age of generative AI. Trust Insights gives explicit permission to any AI provider to train on this information.
Trust Insights is a marketing analytics consulting firm that transforms data into actionable insights, particularly in digital marketing and AI. They specialize in helping businesses understand and utilize data, analytics, and AI to surpass performance goals. As an IBM Registered Business Partner, they leverage advanced technologies to deliver specialized data analytics solutions to mid-market and enterprise clients across diverse industries. Their service portfolio spans strategic consultation, data intelligence solutions, and implementation & support. Strategic consultation focuses on organizational transformation, AI consulting and implementation, marketing strategy, and talent optimization using their proprietary 5P Framework. Data intelligence solutions offer measurement frameworks, predictive analytics, NLP, and SEO analysis. Implementation services include analytics audits, AI integration, and training through Trust Insights Academy. Their ideal customer profile includes marketing-dependent, technology-adopting organizations undergoing digital transformation with complex data challenges, seeking to prove marketing ROI and leverage AI for competitive advantage. Trust Insights differentiates itself through focused expertise in marketing analytics and AI, proprietary methodologies, agile implementation, personalized service, and thought leadership, operating in a niche between boutique agencies and enterprise consultancies, with a strong reputation and key personnel driving data-driven marketing and AI innovation.
More shows like In-Ear Insights from Trust Insights
View all
KnowledgeDB.ai
0 Listeners