Leading Learning

Decades of research say the traditional learner survey—the smile sheet many learning businesses still rely on—is virtually uncorrelated with whether people actually learned anything. So what would it take to do better?

In this episode of the Leading Learning Podcast, co-host Celisa Steele talks with Dr. Will Thalheimer, author of Performance-Focused Learner Surveys and someone wise in the ways of science-backed learning evaluation. Will shares takeaways from his global survey of how workplace learning professionals evaluate their programs, the three fundamental reasons to evaluate learning, and how his LEADS framework (Learning Evaluation As Decision Support) can help a learning business decide which programs to keep, revise, retire, or reprice.

To tune in, listen below. To make sure you catch all future episodes, be sure to subscribe on Apple Podcasts, Spotify, or wherever you listen to podcasts.

Download a PDF transcript of this episode’s audio.

Celisa Steele: [00:00:03] If you want to grow the reach, revenue, and impact of your learning business, you’re in the right place. I’m Celisa Steele.

Jeff Cobb: [00:00:10] I’m Jeff Cobb, and this is the Leading Learning Podcast.

Celisa Steele: [00:00:17] As our guest for this episode notes, hardly anyone gets fired for doing bad learning evaluation. But there are good reasons to try not to do it badly.

Jeff Cobb: [00:00:26] That guest is Dr. Will Thalheimer, a consultant, speaker, researcher, and founder of Work-Learning Research. Will has spent years studying learning evaluation and translating it into practical guidance and even developing the Learning-Transfer Evaluation Model (LTEM).

Celisa Steele: [00:00:42] He’s written the books Performance-Focused Learner Surveys and The CEO’s Guide to Training, eLearning & Work. And, earlier this year, Will completed a comprehensive study on how workplace learning professionals evaluate learning, and we talk about some of the takeaways from that survey.

Jeff Cobb: [00:00:59] You and Will also talk about three major reasons to evaluate learning and the value of using evaluation to support the decisions a learning business has to make about its portfolio.

Celisa Steele: [00:01:11] Will also makes the case that good evaluation is a potential competitive advantage for learning businesses.

Jeff Cobb: [00:01:16] If you want to make sure your evaluation practices are helping you improve and not just checking a box, we think you’ll get a lot from this conversation with Will Thalheimer.

Celisa Steele: [00:01:33] Earlier this year, you released results from your global research on workplace learning evaluation, which is a survey of hundreds of L&D professionals around the world about how they evaluate learning. Most of your respondents work in a corporate setting, but about a quarter serve learners outside their own organization, and about another fifth serve both learners inside and outside their organization. I would love to hear from you what surprised you most in those results, and, in particular, which findings do you think might be most salient for learning businesses—those listeners that we have for the Leading Learning Podcast?

Dr. Will Thalheimer: [00:02:13] A couple of things jumped out at me. Number one, learning professionals are still frustrated with their ability to do the learning evaluation they want to do—that’s across the globe, across industries, across the kind of industry, the kind of organization you’re with, et cetera. The other thing that jumps out is we are primarily relying on learner surveys. Even when we ask people, “How are you measuring impact?,” the number one thing people say is that they are using learner surveys. And that’s not good because a survey is not the best way to find out if people are actually having an impact. Another one of the findings: When we look at those learner surveys, one of the things I did was I gave people 20 different learner survey tactics, ways you could do them. Whether you are using Likert scales, numeric scales. What kind of questions are you using? Are you using sliders, hearts, stars, or lucky clovers? What are you doing? And a lot of those are not best practices, but we’re doing a lot of those. There’s a lot of room for improvement in general, and that’s an opportunity for all of us in the field. Because, if the baseline is not very good, and we do better than the baseline, then we’re creating advantages for ourselves, our learners, our organizations.

Dr. Will Thalheimer: [00:03:45] I wouldn’t say this is a surprise because this is my hypothesis, but one of the interesting questions we asked was, “What are senior leaders telling us they want in learning evaluation?” And my hypothesis, from being in the field for so many years, is that a lot of them aren’t paying attention. A lot of these senior leaders aren’t really paying attention to what we’re doing in learning evaluation, and we—L&D folks—don’t always have a one-way connection with them. I ask people, first of all, “Hey, you’re L&D folks. Have you talked to your senior leadership about learning evaluation?” Only 31 percent said yes. A lot of people said no. And some people said, “I’m pretty sure that our leadership doesn’t care or doesn’t have an opinion about learning evaluation.” So then I took the subset of people that said, “Yes, we talked to our leadership about learning evaluation.” “Great. What do they want?” Somewhere around 30 percent said, “We want to measure our work performance and work results.” But, if you multiply the 31 percent by the 30 percent, that’s a pretty low level. It’s like 11 percent or something like that—not that many. And the second highest grouping of those who talked to their learning leaders said that the learning leaders have no opinion, or they said, “I’ve talked to them, but I’m not really sure what they’re thinking about it.” What’s interesting about this is that there have been a lot of evaluation gurus over the years telling us we must measure impact because that’s the most important thing to senior leadership. Also, we in L&D are going to get fired if you don’t do this. And that’s one of the things we found—that’s one of the lowest problems if you don’t do learning evaluation. Hardly anybody gets fired for doing bad learning evaluation. A bunch of different stuff. This is a long report. It’s over 100 pages, so there’s a lot in there. But those are some of the highlights.

Celisa Steele: [00:06:02] When I think about learning businesses, many of them track satisfaction scores, so they might be using something like the net promoter score. But you and I know that those measures say not a lot about whether people actually learned. I would love to hear from you on how a learning business should be thinking about balancing measures that can predict whether those customers will come back and enroll in another course/pay for another course with measures that tell them whether that learning experience, that course, say, actually worked.

Dr. Will Thalheimer: [00:06:38] Those of us who sell learning obviously have to talk to our customers and see what resonates with them. Sometimes, if they tell us, “Hey, I want the net promoter score,” well, we probably need to use that. But most of us are in the field because we believe in what we’re teaching, and we want to do that well. I like to talk about three major reasons that you might want to measure learning. One is to demonstrate the value—prove that you’re doing well. Another is to support our learners in learning—we can do some measurement stuff that directly helps our learners. And the third reason is to improve our learning—keep what’s good. It’s more complicated than that, but, if you think about those three things, if you are improving your learning, creating those cycles of improvement, you’re going to do better to support your learners. You’re also going to create more value, and then it’s a lot easier to demonstrate that value. So, to me, the fundamental is improving.

Dr. Will Thalheimer: [00:07:44] Let’s just focus on learner surveys right now. We ask questions about the reputation of the course, like the net promoter score or general satisfaction—we’re learning a little bit. But traditional smile sheets have always done that, and there’s a whole bunch of science on this. There are over 150 scientific studies and two meta-analyses that cover these things that show that those kinds of questions are uncorrelated with learning results. So you get high marks on your net promoter score and have a good course but, equally likely, a bad course. You get low marks; you could have a poorly designed course but, equally likely, a well-designed course. With some of our traditional smile sheet questions, we’re going to leave some blind spots for ourselves. My argument is can we do better? Yes, we can. We can ask questions that are more aligned with what we know about learning. If we ask a question, “Hey, are you going to be able to put this into practice in your work?,” it’s focusing on their performance.

Dr. Will Thalheimer: [00:08:51] We ask them, “Are you going to be motivated to apply what you’ve learned?” That’s one of the keys. There’s this research on transfer, whether people will use the information in their work, and there are several key things in that. One is motivation to apply. Another is did they understand the concepts in the first place? If you don’t have a clear idea, then you’re not going to be able to apply them very well. Another aspect is remembering. Are you likely to remember what you’ve learned? Because, if you don’t remember it, you’re not going to use it in your work. Now, you can’t ask people directly, “Did you remember this?” People’s minds are set up to say yes to that question because everything’s top of mind when you ask them. But you can ask them, “Could you get the kind of interactions, did you have the kind of experience that would set you up for remembering? Did you get a lot of rigorous practice? Did you have realistic discussions? Did you figure out different contingencies or ways to do things?”

Dr. Will Thalheimer: [00:09:54] We ask that, then we’re more likely to know about remembering. I call these the four pillars of training effectiveness. The final one of these four is are you getting after-training support or after-learning support? Do you have a coach to mentor you? Do you have job aids to use? Are you going to be reminded of what you learned? We know that after-training support is valuable for people to be able to apply what they’ve learned, so let’s use questions that are focused on these things that are related to learning effectiveness. Also, let’s ask better questions. Likert scales/numeric scales are not good enough. They’re too fuzzy. People have a hard time answering them. In the old days—you’ve seen the paper-based smile sheets—people would circle the same number down the whole smile sheet. That means they’re not paying attention. That means they’re not really answering your questions. So, if they don’t answer your questions, if they don’t think about them, you’re getting bad data. And, if you’re going to make decisions based on your bad data, well, that’s no good.

Celisa Steele: [00:10:59] We’ve already been talking about impact some, and I know that you have a paper called “Ten Reasons NOT to Measure Impact.” You’re arguing that impact really shouldn’t be a top priority in learning evaluation. But we also know many learning businesses feel pressure to show impact or at least a return-on-investment-type value. Even though it can be a real challenge to potentially assess that impact, especially once learners are back on the job and out of the course and maybe away from the learning business that has provided this experience, what do you recommend? What do you think? What’s realistic? What can a learning business do when they’re pressed to demonstrate impact? What would that look like?

Dr. Will Thalheimer: [00:11:43] Let me start with that article because I wrote a provocative title to get eyeballs. I’m sure your listeners can relate to that. My point was not to not measure impact—my point was don’t just measure impact. Because, if you just measure impact, then you don’t know all the learning factors that are going to enable you to improve your course. If you just measure your ROI or your KPIs, then do you know whether your learners are likely to remember? Do you know if they understood the content in the first place? Do you know if they’re motivated to apply? Do you know if they’re getting after-training support? Do you know if they can make better decisions? Do you know if they can take better actions? If you don’t know all those things…those are critical for us to create those continuous cycles of improvement. So it’s not that we shouldn’t measure impact, but we should not just do that by itself.

Celisa Steele: [00:12:43] There’s also, I think, a bit of an argument too around how that can be tricky if you’re going to measure it well. You were talking about, even with impact, a lot of your survey respondents were saying they were still relying on learner surveys to determine impact.

Dr. Will Thalheimer: [00:13:00] There’s more you can do. A lot of us use knowledge checks. Now, a lot of our knowledge checks aren’t that well designed. They tend to ask questions about trivial knowledge versus foundational knowledge—in other words, knowledge that helps people make decisions. But that’s probably a little bit better. We can also use my favorite thing, one of the best investments, high value, is using some kind of scenario. I’ve taught courses on how to write scenario questions. I’m a big believer in this. I used to work for a company that built simulations based on scenario questions. But they’re one of the most powerful things we can do—give people a realistic scenario, and then write these questions; get them vetted by experts. There are ways to validate your questions so that they differentiate between those who know—who have some expertise—versus those who don’t know. There’s a whole process behind that. But that kind of focus on decision-making competence is something that’s valuable. Let me back up and say those of us who are in the learning business, or the business of selling learning, have competitors. And, if everybody is using a traditional smile sheet, if we’re using a traditional smile sheet, then we’re doing what our competitors are doing. We don’t have an advantage over our competitors. We’re doing the least we need to do. How can we gain a competitive advantage? We need to do more than that. One way is to create learning that’s more effective.

Dr. Will Thalheimer: [00:14:45] And, if we have better evaluation, we’re going to create those cycles of learning design improvement. But we can also go to the marketplace and say, “Hey, we used to just measure smile sheets. We used to just ask the learners. But now what we’re doing is this—we’re using scenario-based questions. We’ve done a pre-test to post-test.” I always recommend to organizations, if they can do it—it’s not always easy—to have trusted partnerships with some of their clients. Bring them in and say, “Hey, we want to do some evaluation with you, and we’re not going to charge you for this, or we’re not going to charge you very much. We just want to see what happens and do some comparison testing. And the benefit to you will be you’ll get really good information about the program.” We’ll be improving our programs, and then we can create some content marketing around this. Instead of just going out and saying, “Oh, our course is great; we have good content; yada, yada,” we can say, “Hey, we were doing some research on this, and this is what we found. And we, as an organization, are dedicated to improving what we’re doing and making sure it’s world-class.” That’s a big opportunity here to think, number one, can we create a competitive advantage? Number two, can we create content marketing that enables us to differentiate ourselves from our competitors?

Celisa Steele: [00:16:16] I really like that idea of thinking about evaluation as a potential factor on which to differentiate from competitors. That’s a really interesting takeaway for listeners to think about that lever, if they’re not already thinking about that, as something they could be playing with to improve their offering.

Celisa Steele: [00:16:34] You’ve written a lot about learner evaluation, but you’ve also written a book called The CEO’s Guide to Training, eLearning & Work. I want to talk about a framework that you have in that book called LEADS, which stands for Learning Evaluation as Decision Support. What does it mean to treat evaluation as a support for decisions rather than proof of value? And how should learning business leaders make decisions with evaluation data? I’m thinking about things like which programs to keep, retire, or revise, but maybe there are other decisions that you have in mind as well.

Dr. Will Thalheimer: [00:17:14] Five, ten years ago, I was thinking, “Why do we do evaluation in the first place?” And I thought one of the key things is we want to help ourselves make better decisions. We want to get data that helps us make better decisions so we can take better actions. Well, that’s interesting. If our end goal is better decision-making, why don’t we start with what decisions we have to make and then build our learning evaluation system to help us make those decisions? So then I started writing about this, and I wrote a couple chapters in that book about this idea. You can do this in a super complex way, and I give people some ammunition in the book to do that. Or you could do it in a simple way—every time you’re thinking about what you’re doing for evaluation, ask yourself, “What decision is this going to help me make?” Even if you’re just going to use a survey, for every question that you are contemplating asking, ask yourself, “What decision will this question enable me to make?” And, if your answer is, “I don’t know,” then don’t ask that question.

Dr. Will Thalheimer: [00:18:27] In the book, I offer 14 categories of decisions that people in the learning field have to make. Those are things like how do we support remembering? How do we support understanding? How do we support motivation? How do we make ourselves credible to our business leaders? You can decide what you’re focused on and then work backwards. You mentioned things that I haven’t thought a lot about—what course to keep and what new course to deliver. The one caution I have is sometimes we ask people questions, and they may not be able to tell us whether it’s really going to work for them. Back in business school, they told us, “Don’t just survey people. If you want to know whether they’re going to buy a product, you create an MVP; you create something.” In the workshop world, we could do something like, “In six months, we’re going to offer this course, and, if you sign up now with our early-bird discount three months in advance….” And then, if nobody signs up for your thing, then you know maybe this is not something we should build out.

Celisa Steele: [00:19:52] I feel like we should talk about AI. It’s the kind of thing you almost have to do these days.

Dr. Will Thalheimer: [00:19:58] Yes, that’s right.

Celisa Steele: [00:20:00] You asked a little bit about AI in that survey that we were talking about, and you found that AI is near the top of the list of what people are hoping to try in evaluation, but you were cautioning that it could be herd behavior. We’re all talking about AI, we’re all hearing about AI, and so maybe it’s more herd behavior than being driven by a strategic choice. Talk about where and how you think AI can contribute positively to how a learning business handles evaluation, both in terms of potentially creating evaluations and then analyzing the results of those evaluations, the data that comes from them.

Dr. Will Thalheimer: [00:20:43] So this GROWLE [Global Research on Workplace Learning Evaluation] research was conducted in January and February and a little bit of March this year. It’s very recent, but AI is moving fast. We’re now in the September-October area, so things could have changed. But what we found earlier this year was that about half of workplace learning professionals’ organizations were already thinking about using AI for learning evaluation, but half weren’t. Also, hardly any of them had vendors who were doing learning evaluations. So, if anybody’s out there, that might be an opportunity for somebody to do that. But what was interesting to me is we asked people, “What are you doing now in using AI, and what might you want to do next year?,” and the kind of things people were doing were fairly straightforward, solving some big problems in evaluation. One problem on a survey is open-ended responses, comment questions. In the past, we’d have to sift through those with our eyeballs, and we might—if we were sophisticated—do category sorts and things like that. Now AI can do that very quickly for us, and now our open-ended questions can have a lot more benefit for us. That was one of the use cases that everybody’s doing.

Dr. Will Thalheimer: [00:22:10] People were also using it to analyze data. We asked one question: “What learning-evaluation innovation are you hoping for in the next couple of years?” AI was the number one thing, by far. Then we asked them, “What do you think you want to do with AI and evaluation?” And the responses were very generic—most of them. Certainly the things I’ve already mentioned—the open-ended responding, processing that, data analysis—were all talked about. But the one that jumps out at me as a real opportunity is to create AI-based simulations, behavioral or decision-making simulations, and those can be your medium of evaluation. Where I’ve been recommending scenario-based questions, that’s great, but an AI can create scenario-based questions, a string of them, and analyze that together and dig down deeper than just plain scenario questions on its own. I haven’t seen anybody nail this yet, but I think that’s a real opportunity for us going forward.

Celisa Steele: [00:23:24] In a smaller learning business—I’m thinking just a few people are working in the learning business, pretty modest budget—if that group wanted to make an improvement in how it evaluates its programs this year, what would you recommend?

Dr. Will Thalheimer: [00:23:39] This is an easy one. Most of us are using poorly designed learner surveys. We’re using smile sheets. We’re using Likert scales. We’re using numeric scales. Use a performance-focused learner survey with distinctive answer choices. Those are ones that are more concrete, that learners can get their heads around. They’re more motivated to apply them, and the data you can get can be much more valuable. Most data we get on learner surveys, we get an average between 3.8 and 4.5. We sort of become paralyzed—what’s good, what’s bad—and then we don’t do anything, and that’s no good. But these concrete answer choices are much more valuable because you can see, “Oh, for this question, some people said they’re ready to apply this learning now. Some people said they need more experience. Some people said they’re still confused about the concepts.” Now I have more granular intelligence about how my learning is doing. The number one thing you can do is ask better questions on your learner surveys.

Celisa Steele: [00:24:46] I’d love to hear some of your specific habits, practices, or sources that help you continue to learn and grow—and, in particular, if there’s anything that’s a newer habit, practice, or source that you’re using.

Dr. Will Thalheimer: [00:24:59] I’m a research guy, so I still go to the scientific research. Not everybody’s going to be able to do that. You have to have some skills in that. But that interests me. And so one of the things I’ll say is, “What do you have curiosity about? What can you sustain for yourself? What gets you jazzed?” You’ve got to figure out what works for you. What I do might not work for you. The other thing I do is I’m a big audiobook reader. I listen to books on audio when I go to the gym, when I take a walk, when I’m cleaning the dishes, when I’m cooking dinner, and that’s such a great…. I can’t concentrate on it like I would when I’m reading a book. I can rewind, but I don’t usually. But it gets me thinking about different things.

Dr. Will Thalheimer: [00:25:55] Let me digress a bit. I’m going to oversimplify, but there are two kinds of learning. One is somebody tells you something, it goes into your brain, and now you understand it too. But the other kind of learning is what I call insight learning, where somebody triggers something that makes you think something new. You have a new insight, maybe not the insight that they wanted you to have, but you get these ideas.

Dr. Will Thalheimer: [00:26:17] If you’re constantly feeding your mind with new concepts, new areas of thought, then you’re more likely to have creative ideas, creative insights, and that, to me, is learning. That’s an exciting way to learn when you’re having new insights. Now, I’m four decades into my career here. I’m getting pretty old, so I might not be like your normal listeners, but I’ve begun to focus on other things now as well. I’ve gotten very interested in democracy—and the failures of democracy. I’m reading much less in the learning space and more on reading about the constitution and people’s ideas for how to fix democracy. I’ve become fascinated with that. I still listen to other things as well. Podcasts. Podcasts are a great way to get ideas. I listen to a whole bunch of different podcasts in the social sciences. I’m a big believer in the social sciences because we’re human, and a lot of what we do is based on our humanity. And so those who are studying social sciences and how humans think and learn and get along together and all that really can give us insights about our work and our family life and everything.

Jeff Cobb: [00:27:46] That wraps up the conversation with Dr. Will Thalheimer, founder of Work-Learning Research and author of Performance-Focused Learner Surveys and The CEO’s Guide to Training, eLearning & Work. Stick around for our recap.

Celisa Steele: [00:27:59] On his site, you can find his research and information about his workshops, like the LTEM Boot Camp, and his speaking.

Jeff Cobb: [00:28:14] If you found this episode valuable, please share it to help more people find the show and benefit from the Leading Learning Podcast.

Celisa Steele: [00:28:21] Will made the point that, if everyone is using the same smile sheets, nobody has an advantage. But, if you invest in better evaluation, you create better learning, and that can help you stand out in your market.

Jeff Cobb: [00:28:35] I liked how Will boiled why we do learning evaluation down to three big-bucket reasons: to demonstrate value, to support learners, and to improve your offerings. And he was clear that the last one, improvement, is really the foundation—get that right, and the value becomes easier to demonstrate, and your learners are better supported.

Celisa Steele: [00:28:56] And his LEADS framework gives us a simple test: Before you ask any evaluation question, ask yourself what decision it’s going to help you make. If you don’t have an answer, maybe don’t ask the question.

Jeff Cobb: [00:29:08] Thanks again for listening—and see you next time on the Leading Learning Podcast.

To make sure you catch all future episodes, please subscribe on Apple Podcasts, Spotify, or wherever you listen to podcasts. Subscribing also gives us some data on the impact of the podcast.

We’d be grateful if you’d rate us on Apple Podcasts or wherever you listen. Reviews and ratings help the podcast show up when people search for content on leading a learning business.

Finally, follow us and share the word about Leading Learning. You can find us on LinkedIn.

The post Learning Evaluation as a Competitive Advantage with Will Thalheimer appeared first on Leading Learning.

The new year will be here before we know it, so now is a natural time to make plans. While it can be tempting to copy and paste this year’s plan, make a few tweaks, and move ahead, now is a great time to reflect more deeply on what your learning business does and why.

This episode of the Leading Learning Podcast pulls on past interviews to offer six perspectives that might help you rethink and reimagine the year ahead. Tessa Forshaw, Eric Ries, Luke Dowden, Artesha Moore, and Stephanie Mercado each speak to the importance of questioning a default, and Nancy Harhut is our reminder that that questioning isn’t the same as following through.

To tune in, listen below. To make sure you catch all future episodes, be sure to subscribe on Apple Podcasts, Spotify, or wherever you listen to podcasts.

Download a PDF transcript of this episode’s audio.

Celisa Steele: [00:00:03] If you want to grow the reach, revenue, and impact of your learning business, you’re in the right place. I’m Celisa Steele.

Jeff Cobb: [00:00:10] I’m Jeff Cobb, and this is the Leading Learning Podcast.

Celisa Steele: [00:00:16] As one year winds down and you look ahead to the next, it can be tempting to copy this year’s plan, adjust a few dates, make a few tweaks, and carry on. Consider this an invitation to do something different.

Jeff Cobb: [00:00:29] In this episode, number 489, we’re trying a compilation format—six clips pulled from past conversations and curated because each, in its own way, pushes past tweaking toward genuinely questioning a default.

Celisa Steele: [00:00:44] Cognitive scientist Tessa Forshaw pushes back on the common assumption that learning is about content, and Eric Ries, author of The Lean Startup, talks about why testing beats forecasting, especially in times of change.

Jeff Cobb: [00:00:56] Artesha Moore of Association Forum, Luke Dowden of the Alamo Colleges District, and Stephanie Mercado of the National Association for Healthcare Quality give an example of what it looks like to question fundamental assumptions in their organizations.

Celisa Steele: [00:01:12] And Nancy Harhut, author of Using Behavioral Science in Marketing, adds a reality check. Present focus bias doesn’t just stop learners from signing up—it’s also what will stop us from following through unless we build in nudges for ourselves.

Jeff Cobb: [00:01:27] Five different people speaking to the importance of questioning the status quo, and one caution that asking questions isn’t the same as follow-through. Let’s jump in.

Celisa Steele: [00:01:42] First up is Dr. Tessa Forshaw, a cognitive scientist and faculty member at the Harvard Graduate School of Education. Tessa questions a default many learning businesses don’t even think to question: that learning is fundamentally about content. She argues learning is a neurobiological process, which has profound implications for what a learning business offers and how it offers it.

Celisa Steele: [00:02:05] I would love to hear what you see as some of the biggest mismatches between that science of learning and what we know and then the way that adult workers are commonly taught or the way we design learning for those adult workers.

Dr. Tessa Forshaw: [00:02:20] I love that that’s what you’ve gleaned from my work. I definitely think that. When I talk to a lot of learning leaders, learning is often discussed as content delivery. They will talk about learning as the content that is being created or the event that is being held. But it’s really important that we remember that learning isn’t content. Learning is a neurobiological process, and that process is influenced by chemicals in our brain and hormones in our body that are shaped by the environment and the context that we sit within. When we understand learning in that lens, it both opens up where learning can happen and what it looks like, but it also puts the responsibility of establishing the conditions that create and foster those positive neurochemicals and hormones to incite learning and alleviate the ones that don’t. And it puts that responsibility on L&D teams or on organizations providing learning experiences in a way that it hasn’t typically been considered before.

Celisa Steele: [00:03:35] You were saying, before we got started, that one of the things that you talk about in the certificate are five principles for the future of workplace learning. Could you outline and briefly explain those five principles?

Dr. Tessa Forshaw: [00:03:50] Yes, I absolutely can. The five principles that we talk about start with this idea: Learning is biological. We understand that behavior change comes from changes in the mind, which comes from changes in the brain, which comes from changes in the cell. I’m sure you’re familiar with those games that were really popular about ten years ago, where they were all like, “Neuroplasticity!” There was a lot of fake going on there, but there is a seed of truth or a grain of truth, and that is that neuroplasticity is the foundation of learning. The ability for us to create a new neural pathway between two neurons is the foundation of all learning. When we understand that the cell influences the brain, which influences the mind, which influences behavior, that helps us think about how we can create learning in organizations, especially because so many workplaces are full of things that make that biological process harder. They’re full of cortisol. They’re full of cognitive overload. They’re full of constant context switching. They’re full of being triggered and having your amygdala hijacked when stress takes over, and your brain doesn’t operate at its peak performance anymore. When we understand learning is biological, we can also start to understand that learning at work is very rarely about effort or attention, and it’s much more about the conditions in our bodies, in our environments that are blocking this cellular change from happening. So that’s one component.

Dr. Tessa Forshaw: [00:05:24] The next principle is that learning is social. We know from decades of research on expertise development and the nature of learning that people become experts by being around people who are further along in that area of expertise. Having people more experienced share mental models, expose their reasoning to you, and foster that traditional romantic idea of apprenticeship that we have from hundreds of years ago. So learning is very social, and it happens in relationships with others. The third principle is that learning is situated. This is focused on the idea that pulling people out of work to go and learn is a tough design choice because the brain encodes skills along with the context in which they are learned in. A lot of people don’t realize that our environment plays a huge role in the retrieval of learning because we associate different knowledge and skills, practices to the physical environment in which we learn them. When we’re in those environments, they can help us cue it forward. If we learn something in an abstracted classroom in a beautiful L&D center that’s off-campus, and then we’re supposed to go back and apply it on a factory floor, that’s going to be very hard because the environment isn’t cueing that knowledge that we’ve learned to be brought forward, and those retrieval pathways aren’t yet strong because it’s new knowledge.

Dr. Tessa Forshaw: [00:07:03] The fourth principle is that learning takes effort. We talked a little bit about learning being a biological process and there being the creation of a pathway between two neurons. If you could imagine a bank of sand, and you have a stick at either end, and you want to create a pathway between the two of them that stays, you have to really dig into the sand and put some effort in there. That’s the same in your mind—you actually need effort to forge that pathway. Learning takes effort. Being easy and super fun isn’t necessarily what we’re going for. We’re going for experiences that provide opportunities for effort. Finally, that learning has to transfer so that the point of learning is that it can be applied in future contexts. Especially today, where those future contexts are often ambiguous, and we don’t know how they’re going to change, that means that when we think about learning, we have to think about designing opportunities to expansively frame all of the different ways things could be applied, and, again, that’s part of helping that encoding in a way that’s much easier for us to retrieve it at that moment of need. So those are five principles.

Jeff Cobb: [00:08:22] Next is Artesha Moore, CEO of Association Forum. Artesha questions the default board conversations that center membership growth and retention by running an exercise that reimagines her organization from a blank slate. She closes with a rule she holds herself to every year: Do something you’re willing to fail at.

Artesha Moore: [00:08:42] I did an exercise and led a facilitation with our board a couple of years ago, and it was all around Marvel’s cartoon show What If…? What if we were in an alternate universe? I used ChatGPT to come up with…. I said, “Here’s the plot. We are this. We’re a $3 million organization. We focus on this. We do this. What if…? Open it up.” It gave, “You could be this kind of business. You could be this.” And then—I love Shark Tank—I gave the prompts to the board and put them on teams: “Now take this and do it. Y’all pitch it to us like Shark Tank.” Here’s what it did: It stopped the conversation about membership growth, acquisition, and bringing back the community, and so what could we do? We went to market these ideas. Skills became really big for us. We did a skill study. We did this. We did this what-if—playing games. They pitched it. There was some of it that was fun, but it unleashed their brain of what could be possible. Right now, whoever’s listening, open up an AI—whatever your favorite one is—and start to say, “If I had nothing, what could I be?” Don’t give it a whole bunch of “We’re this and that and this” because that just defeats the mission. Use a wide view. You don’t have to build that today. Stop being so literal. Have an imagination. Then go for a walk. You know what it’ll do? It’ll get you thinking about something different. I love all of my alphabets: ASAE [American Society of Association Executives], Association Forum, and PCMA [Professional Convention Management Association].

Artesha Moore: [00:10:32] We are doing hard work as associations because we compete with LinkedIn. We’re a leadership development, and so LinkedIn is always like, “How did I get out here?” Reimagine. Imagination is key. But it’s got to be with the board. It can’t just be with your staff because then you create this us-against-them. We do see things differently as staff, but, if you start to imagine what you could be, that would be the thing I would say: Just imagine. Don’t put a lot of constraints on it. And then the other part is do something that you’re going to fail at every year. Every year, do something that you are going to fail at—and that doesn’t have to be big at work. It could be at home. I’m a painter now. I’m terrible. But you know what, Bob Ross, thank you for inspiring me because I’m almost 50, and I started painting maybe in the last three years. It has unlocked what I never imagined, which was creativity in my technical work. Use that AI to challenge what you think. I probably have, at least every three weeks, a session where I just argue with the AI. But try some new stuff. And then, Jeff, you’re doing great work in this space. You have people on that have interesting ways of looking at things. It doesn’t matter what your industry is. Especially for those at the top, top, innovation is what we’re hired to do, but navigation is also needed. You need a direction that you’re going that inspires—and then I think we’ll make it on the other side of this thing.

Celisa Steele: [00:12:23] Eric Ries, author of The Lean Startup and Incorruptible, questions the default businesses have run on for a century: that planning and forecasting are how you manage uncertainty. He makes the case that when the ground keeps shifting, testing beats forecasting every time.

Jeff Cobb: [00:12:39] I’d love your perspective—15 years out from publishing the book, how has that concept held up from your viewpoint, and how has it evolved over that time period?

Eric Ries: [00:12:51] It’s held up pretty well. People get it confused all the time because people think, “Oh, if we’re just building a minimum product, that’s opposed to quality or long-term planning or something like that.” But the point of minimum viable product is simply, if we’re in a situation where there’s going to be a long number of steps that have to unfold in order for whatever the thing is to be successful—I can tell you stories about where the thing itself is like a whizbang gadget; a cool platform; a deep-sea, oil-well-drilling platform; an internal process change; or a new curriculum (I really have seen every kind of thing you can imagine this thing can apply to)—there are two dimensions (I said this 15 years ago or more) of the world that are going to make this more and more difficult. One is the changes in the technological landscape are going to democratize the tools of production but also drop cycle times. So things are going to get faster. It’s going to become easier for more people to build things, try things, communicate things. That prediction’s held up pretty well.

Eric Ries: [00:13:51] But, at the same time, we’re going to see a great increase in the level of underlying uncertainty about what’s going to work. Partly, number one, because consumers, employees, and whoever we’re trying to reach, students, they all have more options than ever before, and they’re facing a bewildering array of experts, advice, best practices, advertising, and the fragmentation of social media. When we have that combination of fast cycle time capability and extreme uncertainty in the environment, we need a process, a structured way of thinking about what to do with the infinity of things we might do. Traditional management, general management that has been practiced all over the course of the 20th century, was based in two principles: the principle of planning and the principle of forecasting. If you go back and read the foundational works of 20th-century management thinking—Fred Taylor. I’m thinking especially of Alfred Sloan’s My Years with General Motors because he really lays out how difficult it was to run a business without these concepts. It was very difficult. The key idea is that, in order to make a plan, in order to hold people accountable to that plan, we need to have an accurate forecast of what’s supposed to happen. If you want to claim credit for having done a good job as a manager, you can’t just say, “Well, I sold more cars this year than last year.” Maybe the economy’s up, and everybody sold more cars this year.

Eric Ries: [00:15:10] Similarly, you can’t say you’re fired because you sold fewer cars than last year. Maybe we’re in a recession, and, if you did slightly less bad than everybody else, that’s a win. The key is to be able to forecast and beat the forecast. Most of us have been taught that in our careers, not as an idea but just as the way work is done. The hidden assumptions are the killers, always. Most people that I talk to cannot imagine that this idea ever had to be invented. To them, it’s as natural as gravity. But it’s not. It was an idea. It was invented at a certain time. It has certain strengths and also certain liabilities—the biggest one, where do accurate forecasts come from? Accurate forecasts require a long and stable operating history from which to extrapolate. And who feels like the world is getting more stable every day? Uh-oh. First of all, whenever we’re doing something new, we’re doing entrepreneurship, no matter what it says on our business card. A lot of people become what I call “involuntary entrepreneurs” when the world changes around them, and, all of a sudden, they’re having to scramble to reinvent something they’ve done consistently, maybe for many, many years. Parallels to AI are pretty obvious. That is the essence of minimum viable product: to figure out what is the structured, scientific, least-effort way we can discover whether the core business hypothesis that we are operating against is true or false.

Jeff Cobb: [00:16:35] Luke Dowden, vice chancellor for Digital Learning and Transformation at the Alamo Colleges District, questions the default academic calendar. He describes pushing for a different structure and sticking with it, against initial skepticism, until it became the norm.

Luke Dowden: [00:16:51] You ask about the challenges faced. Understanding the market, that’s something that I’ve watched different organizations struggle with. Do they really understand the market that they’re serving? And then will you adapt the delivery in a fast enough method that you can serve more of the market? Meaning that, “Well, should we just always offer them in sixteen weeks, or could it be eight weeks?” One of the institutions I was at, I said, “We want to do what we do in the summer, twice in the fall and twice in the spring,” and people looked at me funny because we would have an eight-week session in the summer and two four-week sessions. I said, “We just want to do what we do in the summer, twice in the fall and twice in the spring.” And I said that enough that, when I had left that organization, that’s what they were doing. You can create a self-fulfilling prophecy. What I mean by that is create the conditions that you are doing something, but you’re maybe doing it over here, and you need to think about your calendar or how you’ll approach the rhythm or the different options and the different configurations of learning.

Luke Dowden: [00:17:58] In Oklahoma one time, there were 52 Tuesdays. So, 52 Tuesdays, if you come, you can earn your degree. And so I still think that there are those opportunities to improve. A couple of the other challenges to your original question of adapting and understanding the market is you do have people that stop in and out. When you work with folks that are learners, that learning is not going to be the first priority because of other priorities in their lives. Then you have to create a way for them to come back in easily. A lot of the work that I did at the last institution in Louisiana was around designing for people to stop out, expecting that, and creating pathways for them to come back in. And the hardest part is that there’s a lot of competition for people’s attention. We’re competing for attention. You’re not just competing against an online competitor or another brick-and-mortar competitor, but you’re creating an experience. Think about the things that we like to go and spend money and time on. I would say that people are very mindful of the experience they want, the expectations they’re coming in with. I think there’s also a new set of expectations that learners are bringing with them, based on what they’re experiencing with other services.

Celisa Steele: [00:19:26] Next up is Nancy Harhut, author of Using Behavioral Science in Marketing. Nancy is this episode’s reality check. Even if you’ve decided to question a default, present focus bias can keep you from following through on making changes. She talks about what it takes to get people moving.

Celisa Steele: [00:19:44] A lot of our listeners offer certifications, education, all sorts of professional development opportunities, and these are things that the people they serve will say that they value, but they still won’t necessarily do them. They’ll still postpone them or not sign up for that course or that conference. Talk about what behavioral science might help us understand about why adults might delay acting, even on things that they say they themselves value and matter to them.

Nancy Harhut: [00:20:19] That’s a great question. What it comes down to is very often people will say one thing and do another. You’re like, “Why did they lie to me?” But they didn’t really lie. They believed what they were saying at the time, but then something else happened. A couple of things happened. One is this idea of present focus bias. Behavioral scientists have found that humans love instant gratification. We want what we want, and we want it now. And, if the payoff is a little bit further down the road, if it’s more distant, we have a tendency to devalue it. Imagine you’ve got $100 in your hands. You’d probably like to go out to dinner and a show versus save that money for a rainy day, put it in your retirement account. It’s the same thing with education. It’s like, “Yes, I think this is a great idea. We should have professional development. We should have these courses. We should have these Webinars or seminars or conferences. And, yes, I want to go, but not today. I’m going to do it later.” Because the payoff is later. “I have to go. I have to learn this, and, eventually, it’s going to help me in my career.” And because it’s a more distant payoff, we’re less likely to act on it right away. We want the immediate gratification.

Nancy Harhut: [00:21:31] Another thing that behavioral scientists talk about is something called the intention-action gap. And there, again, it’s like, “Yes, I intend to take this. Yes, I want to. Yes, I think this is a great idea.” But we just don’t quite get around to doing it. We think we’re going to. We assume that we will. It makes sense at the time. And yet other things come up. Things distract us. Things get in the way. The important takeaway here is it’s not that people are lying when they say they value this, that it’s important. They’re not lying. They’re not really good at predicting their future behavior. There are some behavioral science studies that show a complete stranger can better predict what you will do in the future than you yourself can. So it’s tough. That’s what we’re working against. What marketers need to do is to overcome that idea of present focus bias. We need to build a bridge between who you are right now—saying, “Yes, I work in this field, and I like this field, and I want to continue to succeed in this field, and so I want to keep learning and growing.”—and the person who is actually going to benefit from it. It’s like, “You’re saying this right now, and, if you really believe that you want to do well in this field, and staying in this field requires continuing education, then you need to take this step so that one month, one year, two years down the road, you’ll have what you need.”

Nancy Harhut: [00:22:59] There are ways to create that bridge. One is by getting you to think of yourself today but in the future—so using words like “imagine.” “Imagine having this new certification.” “Imagine having taken this class and learning these three points.” Because, if you ask someone to imagine something, they can create that mental picture. And, if they can create the mental picture, they’re much more likely to act on it. Because, when you think about it, when people make a decision, it happens twice. It happens once in their brains, once in their heads; they decide to do it. And then they physically do it. Build that bridge by saying, “Imagine yourself…” or “Can you see yourself…?” or “Picture yourself….” That can be very, very helpful. The other thing we can do, again understanding that people prefer instant gratification, we can pull some little element out where we do get instant gratification. If we’re asking someone to sign up for a course, and the course is going to take ten weeks, or we’re asking them to sign up for a conference and the conference isn’t for three months, that’s not instant gratification. But maybe you can sign up in a minute—that’s instant gratification. Or maybe “Sign up, and we’ll immediately send you the cheat sheet or the curriculum.” Anything that we can pull forward that has instant gratification, even if it’s just a small element of the larger course, conference, what have you, that can work in our favor in terms of overcoming present focus bias.

Jeff Cobb: [00:24:34] We’ll close with Stephanie Mercado, CEO of the National Association for Healthcare Quality. Stephanie asks a question many leaders might be afraid to ask: Should this organization even exist?

Celisa Steele: [00:24:48] Based on your experience, what advice—or maybe it could be a question to consider—what would you offer to other association CEOs or learning business leaders as they’re trying to think about, “What do I need to do to be successful?”

Stephanie Mercado: [00:25:04] I’m a big fan of the notion that Simon Sinek brought forward, which was start with why and that people don’t buy what you do, they buy why you do it. I think that the secret to make you success or the secret to my success has really been searching for those answers. Why are we here? What can we do that nobody else can do as good as us? And how can we add value to the market and solve those challenges? I remember, when I first joined NAHQ as the executive director, I heard in the interview process that the board was looking for what I would categorize as more, better, faster. But when I asked them, “Specifically, more, better, faster what?” no one could really articulate what that was. I went in search of finding out what it might be that we could do that would really add value to the market and be our differentiating value proposition. The question I asked every board member and five past presidents and everybody who happened to be on our advisory board at that time, so about 25 people, I interviewed, and I said, “If NAHQ was wiped off the face of the earth tomorrow, would anyone start this organization again and why?” And that was a question that stopped everybody in their tracks, and they’re like, “Oh my gosh! I don’t know. I’ve never thought about it.”

Stephanie Mercado: [00:26:32] I think that when you take a breath and say, “Okay, what are we really trying to do here, and let’s get consensus on what that is,” it doesn’t start with tactics. It really starts with the problem that you’re trying to solve for. Luckily for us, what became very clear to me as I asked that question, “If NAHQ was wiped off of the face of the earth tomorrow, would anybody start it up again and why?” the answer was because we’re the only organization focused on workforce competencies for quality and safety. All right, then let’s do more of that. Let’s do that really, really well. We don’t have a competency framework. We should build one. We don’t have a competency-based assessment. We should build one. We don’t have a way to work with healthcare organizations. We should find a way. That, I think, has been really, really important. I think that the secret to NAHQ’s success and the secret to my success is to be intentionally curious and persistent and not give up because it was hard. It took a long time for us to answer that question. I had some of the people I admire and respect said, “Stephanie, I don’t know why you’re trying so hard. Maybe there isn’t an answer.” I said, “I know that there’s an answer, and we’re going to figure it out.” And we did.

Jeff Cobb: [00:27:55] We just heard six perspectives that might help you rethink and reimagine what your learning business does and how it does it going forward. Instead of treating next year as this year plus a few tweaks, take time to question the default.

Celisa Steele: [00:28:10] Links to the full episode these clips came from follow.

Jeff Cobb: [00:28:20] I see Tessa Forshaw’s reminder that learning is neurobiological and Eric Ries’s case for testing a hypothesis instead of trusting a forecast as two sides of the same coin: Question the default before you build on it.

Celisa Steele: [00:28:35] Really, that idea of questioning the default is at the heart of all we heard. It’s there in Luke Dowden’s insistence on changing the standard academic calendar, Artesha Moore’s board exercise, and Stephanie Mercado’s willingness to question whether anyone would even start her organization today.

Jeff Cobb: [00:28:52] And Nancy Harhut is our reminder that questioning something isn’t the same as doing something. Present focus bias doesn’t just stop learners from signing up—it’s what will stop us from following through too, unless we build in nudges for ourselves.

Celisa Steele: [00:29:07] Thanks again for listening—and see you next time on the Leading Learning Podcast.

To make sure you catch all future episodes, please subscribe on Apple Podcasts, Spotify, or wherever you listen to podcasts. Subscribing also gives us some data on the impact of the podcast.

We’d be grateful if you’d rate us on Apple Podcasts or wherever you listen. Reviews and ratings help the podcast show up when people search for content on leading a learning business.

Finally, follow us and share the word about Leading Learning. You can find us on LinkedIn.

The post Stop Tweaking, Start Questioning appeared first on Leading Learning.

Many learning portfolios accumulate over time—a Webinar added here, a course there—until nobody’s quite sure which offerings are still earning their keep and whether, collectively, they reflect what the organization is trying to accomplish.

In this episode of the Leading Learning Podcast, co-host Celisa Steele talks with Lauren Brechner, founder of Ember Forge. Lauren shares how a portfolio audit can double as a pressure test of a learning business’s strategy, why she recommends focusing on a handful of flagship offerings rather than tackling an entire catalog at once, and how her background in marketing and digital experience design shapes the way she diagnoses what’s really behind an underperforming offering.

To tune in, listen below. To make sure you catch all future episodes, be sure to subscribe on Apple Podcasts, Spotify, or wherever you listen to podcasts.

Download a PDF transcript of this episode’s audio.

Celisa Steele: [00:00:03] If you want to grow the reach, revenue, and impact of your learning business, you’re in the right place. I’m Celisa Steele.

Jeff Cobb: [00:00:10] I’m Jeff Cobb, and this is the Leading Learning Podcast.

Celisa Steele: [00:00:17] If your learning portfolio has grown more by accumulation than by design—a Webinar here, a course there, and nobody quite sure which ones are still earning their keep—this episode is for you.

Jeff Cobb: [00:00:29] Our guest in this episode, number 488, is Lauren Brechner, founder and principal consultant at Ember Forge, which partners with organizations to realize their business goals through custom e-learning strategy and development. Lauren comes to this work from a background in marketing and digital experience design, and her roots in audience segmentation, user journeys, and iterative testing shape how she approaches learning design.

Celisa Steele: [00:00:55] Lauren makes the case that a portfolio audit isn’t just a housekeeping exercise—it can double as a pressure test of your strategy. Do your learning offerings reflect what you say your mission and vision are?

Jeff Cobb: [00:01:07] You and Lauren walk through what a portfolio audit looks like in practice and why she typically recommends organizations focus on a handful of flagship offerings rather than trying to fix everything at once.

Celisa Steele: [00:01:19] If you’re sitting on a portfolio that’s grown less by design and more by accumulation, we recommend you give a listen to this conversation with Lauren Brechner.

Celisa Steele: [00:01:34] Give us a thumbnail sketch of what Ember Forge does, who you serve, how you serve them.

Lauren Brechner: [00:01:39] Ember Forge is a learning strategy and development partner for organizations and learning teams, especially in the nonprofit, mission-driven, and association space. We step in as an end-to-end partner, from determining how learning fits into organizations’ business goals to developing learning experiences to helping deploy and measure the impact of those experiences for both the organizations’ business goals and their learners.

Celisa Steele: [00:02:04] Having talked with you before and read some about your background, you come from marketing, and you have digital experience design as well. What concepts from that world show up most in how you approach things now? I’m thinking of things like audience segmentation, user journeys, iterative testing. Maybe it’s other things. But I would love to hear how that marketing and digital experience background influences what you do now.

Lauren Brechner: [00:02:34] Absolutely. All of those things that you mentioned. That experience has been foundational to how I approach learning and e-learning especially. I think about it as a pyramid where in one lower corner is the marketing side. How do we segment audiences? How do we identify who the right learners for the content are? How do we market that to folks who are going to purchase it—whether it’s the learners themselves or organizations buying it on behalf of their employees—making sure that content resonates with them and they’re the right receivers of that? And then how do we title a course to make sure it’s actually representative of what the content is? Those simple things. The other corner would be that digital experience. I came from the association world, and my last role there was manager of digital experiences. Web sites, microsites, conference apps, anything digital where our members were interacting with us solely remotely—I managed that. And I learned a lot from how to get people from point A to point B, how to do it meaningfully, and how to make sure they gain understanding along the way.

Lauren Brechner: [00:03:47] What are people calling about, and how do we answer those questions upfront through the journey that we deliver them through in their membership journey? Those two things squish together and push up into how I approach learning. You get to the top of the pyramid, and that’s where learning science, learning evaluation come into play, and the practices that make learning effective and contribute to long-term retention. It’s not just about a good and nice learning experience—those are wonderful and nice. But, if you’re trying to make long-term impact and do capability building, that learning science is essential. But the science can’t get through to the learner, those impacts can’t get through to the learner without making sure that’s the right learner (marketing) and making sure they can traverse the course and experience it, which is that user experience design. So all of those things mesh together and have brought me to this holistic philosophy of how we approach learning.

Celisa Steele: [00:04:44] I love all three of those and this idea that you have the two squishing up into learning, as you said—that was very nice. But the marketing has to be there and has to be right, not just in the sense of promotion. You were talking about other things, other aspects of marketing there. And then that course, that user experience is so important because if it’s too hard, then you have learners spending their mental bandwidth on something other than the peak, the top corner of your triangle, where they’re actually engaging and, as you said, hopefully building true capability, not just completing a course.

Lauren Brechner: [00:05:21] Exactly. If they’re trying to spend their energy figuring out how to get from one lesson to the next, you’re failing the learner. They need to be spending that energy purely taking in the content. You need to curate that experience for them.

Celisa Steele: [00:05:33] Let’s talk a little bit about what has to be in place before a learning business can build a product offering or, once they have that product offering, meaningfully assess that product that they have or that whole portfolio of products that they have—things like goals or target audiences, the business rationale that needs to be in place. Talk about what has to be there before a product offering can exist or before it can be assessed and evaluated effectively.

Lauren Brechner: [00:06:10] Goals, target audience, business rationale—these are all things we can agree about on paper. But, in practice, that’s where folks diverge. Here’s my approach: A learning product can have multiple benefits, but it should solve for one primary problem. What is the one thing it’s solving for? If there’s a capabilities gap in an industry for an association, for example, and they want to help their members close that gap, that is the goal. That’s the problem that they’re solving for. The driver is equipping people with skills that help them succeed. So you’d start with prospective learner surveys to find out what they feel they need help with most. What’s that gap? And then the content learning experiences need to address that gap as well. But, if it also drives revenue, great. If it also puts your name out there as an authority in this field, that’s great. But the focus has to be the one problem you’re solving for. That’s not to say it can’t do many things. On the other side of things, if it’s a revenue driver first, those priorities shift. You can still create real impact with learning. You might choose a less expensive delivery method and work backward from a pricing model, and you might budget more aggressively for marketing and consider all of that upfront.

Lauren Brechner: [00:07:23] That doesn’t mean revenue-driving offerings should be low-quality, that they should be a product you cash in on. It just means you’re working within budget constraints from the start and then working backwards to maximize what you have in that budget. Every offering should also have goals. Launching a course is really testing a hypothesis, in my opinion. You may have research and expertise that give you the confidence in making those decisions, but the real test is putting it in front of real learners, especially if they’re paying for the product. They’re going to give you feedback. And before you put it in front of those learners, define your success metric upfront. If you are trying to bridge that capabilities gap, for example, maybe your goal is that 75 percent of those registrants actually complete that course, or half of them complete 90 percent of the course because that 10 percent isn’t relevant for them. That’s something you’ll want to define early on. Or, if it’s a revenue driver, maybe it nets $20K in the next year or two after breaking even. Whatever it is, you need to define it and measure against it and then adjust delivery and content as you go.

Celisa Steele: [00:08:30] I really appreciate your comment. It’s easy to agree, in theory, that goals are important, but you do have to get clear. Yes, it can do many things, but you have to have that agreement around what is the primary goal for this particular offering because that’s going to drive so many of the decisions, not only into how the product is made and rolled out, but, to your last point, around how you assess and see is it successful? I really appreciate that need for clarity and that need for agreement around a primary goal.

Lauren Brechner: [00:09:08] Exactly.

Celisa Steele: [00:09:09] I want to talk a little bit about a portfolio audit. That is something you help organizations with. From our own experience, often a learning business less than 100-percent intentionally lands on their portfolio of offerings—it’s what amasses and aggregates over time rather than being a 100-percent thoughtful approach. What does a portfolio audit look like in practice, and what do you measure existing courses against as you’re making that decision about is this something to keep, to retire, or to modify and rebuild?

Lauren Brechner: [00:09:50] First, a portfolio audit relies on a clear mission, strategy, and vision from the organization. That’s often the problem. As you said, a learning portfolio can grow over time because sometimes you’re addressing an immediate need, and then the relevance isn’t there. A misalignment between the learning portfolio and current state and the organization’s real trajectory is a real problem. In an audit, you compare every offering against its ability to fulfill the mission and drive the vision. Are we doing what we need to do now, and are we going where we need to go? Practically, that starts with listing every offering in a spreadsheet—the year it was created, whether the topic is still relevant to the industry, how many registrations there have been over time, how many completions, are there assessments at the end, and how well are people doing in those? If you have learner feedback, what did they think of the course? What is their feedback on it? If courses are falling well below your threshold metrics (which you should define overall for each of those), you need to understand why and figure out a path forward. Another thing that is included in an audit is delivery method. A lot of organizations do Webinars, and they’re very valuable, and then they have the recording and stick it in their library.

Lauren Brechner: [00:11:08] Sometimes those are hidden gems, and those might be incredibly relevant to the industry, but the recorded Webinar format just isn’t delivering that learning impact like it really could. That’s an opportunity for working with that same subject matter expert who delivered that Webinar and creating an online self-paced course. Those are the opportunities you want to look for. Once you get everything into a spreadsheet, you can decide what to do with that information. If several courses have high registration but low completion, and completion matters to you, that’s a priority. It also helps to categorize everything by type—the self-paced courses, Webinars, conference recordings too. If there’s a lot of registrations for those, that’s an opportunity to develop something a little more robust. There are different lifts as well for each of these, so the effort is worth noting. If you are converting a Webinar to a self-paced course, you are taking on a completely different product type, and there’s an effort with that. But if you’re updating an existing self-paced course, that effort’s much smaller. You can tweak that. Those are things worth noting too. What is the effort level on a scale of 1 to 10? And, to be clear, an audit doesn’t mean acting on every single offering that’s coming up in there. My usual recommendation is to identify a handful of flagship courses—three, five—that best align with the current strategy and that are driving the mission and vision forward and focus there.

Lauren Brechner: [00:12:35] Some of these libraries and portfolios have hundreds of offerings, and that is daunting. That is no small task. I’m not going to pretend like it is. But narrowing it down to, if we only had five offerings to represent what we’re trying to do, what would they be? And just focus on them and what you can do with them. If it’s a topic that’s really important, but you’re seeing it’s underenrolled, that’s not even a content problem necessarily—that’s a marketing problem. That is something you do way in the front. And so you can get people in that course first by redirecting your marketing efforts and then evaluate their response so you actually understand. What we’re looking for is all of those and also if there’s a consistent feedback mechanism across all of your offerings. If you’re comparing things on the same footing, like the baseline Kirkpatrick Model (reaction, learning, behavior, results) plus topic-specific questions: Did this help me apply x more effectively in my job? Those are really important to have a baseline that you’re able to compare them all against each other. My piece of advice, because this is a very overwhelming task, is to bring someone without a relationship to these offerings—folks who didn’t create them, the subject matter experts who weren’t featured in them, from outside your department or outside your organization—to pull that raw data and organize it objectively for a first pass.

Celisa Steele: [00:13:57] As you said, this could be a tremendous amount of work if you have a large portfolio, so talk a little bit about your thoughts on cadence. How often should a learning business be doing this sort of portfolio audit? There’s probably a heavier lift upfront the first time you’re putting such a spreadsheet together, but then you have it. And then going forward, it’s more a matter of updating and adding the newer offerings to it and confirming that some of the past assumptions are still valid. But, yes, talk about cadence.

Lauren Brechner: [00:14:33] The first pass is the biggest lift. Once you’ve got that spreadsheet together or whatever database you’re using to track this, number one, keep it updated. You have a new offering; it needs to go in that spreadsheet so you don’t have to pull everything all over again. Update metrics as frequently as quarterly—if that’s within your capabilities as an organization, just to see how those go, if there’s an automated way you can do that—at least once a year. It depends on what makes the most sense. Don’t drink the whole glass of water at once. Take a few sips. If you feel like you’ve addressed your top three courses, and you’re seeing the impact…that period of time that you’ve piloted those changes is important to sit in. Making too many changes at once across your whole portfolio can be detrimental if your hypothesis is wrong. And so letting time pass to get that first hypothesis out there, get the feedback, and figure out, “Okay, this is a thing we should apply to all of our offerings,” that’s a piecemeal thing you can do. If it is “Let’s add a survey to every offering we have—forget about the content; let’s just have a survey there and make sure that’s across the board,” that’s a great way of doing that. And then maybe each year, pick three more courses you want to evaluate. It’s up to the organization to say, “What is our appetite for doing this? And what is our real capability?” Because you don’t want to bite off more than you can chew because that’s where you start to feel underwater, and that whole project can fall to the wayside.

Celisa Steele: [00:16:06] You’ve mentioned having a hypothesis at least a couple of times. Let’s go a little bit deeper there, and, if possible, I would love to hear some examples of hypotheses or how you think about what goes into making a good hypothesis that then you’re going to be trying to test.

Lauren Brechner: [00:16:23] Yes, I really like the scientific model there. I use the word “hypothesis” because sometimes we launch a course and say, “This is the product. Here it is. Enjoy it or don’t.” And then we just leave it out there to do its thing. If we’re being good stewards of learning practices, we do need to listen to feedback, and we do need to have our assumptions tested. I’m a big advocate for pilots. Let’s say we hope that folks who are in a leadership position in a specific industry that’s at particularly high stress, we want them to be able to update their strategy every six months. And they, in current state, aren’t able to because of all of the fires they’re always putting out. The hypothesis is “We’re going to put this training together that will enable them to do that every six months.” You can have a survey at the end that says, “Do you feel this has equipped you to do that?” And they might say, “Yes, I feel like it did.”

Lauren Brechner: [00:17:23] Those end-of-offering and -course surveys tend to be much more optimistic than the reality. The way you test the hypothesis is follow up with them in six months. Send a follow-up survey saying, “Hey, did you actually do that?” And that’s where the real results come in. I talk about the forgetting curve all the time. We feel really ambitious once we’ve crammed on a certain topic or course, and we feel we can conquer the world sometimes, and then even weeks later it’s like, “Well, I don’t really know how to put this into practice.” So the measure is months from now: “Did we give them the tools and the skills to put that into practice?” And if they didn’t, they can say what the gap was. That’s the thing you need to build next. That is the part that needs to be adjusted for your next iteration.

Celisa Steele: [00:18:11] Let’s say you do something like what you just described. You put out a product that you think is going to help these leaders be able to adjust their strategy every six months. You follow up six months after to find out if they really were. Let’s say the results were amazing. They were able to do that. You’ve proven the hypothesis at that point. Except, then, the world changes. In your example there, would it be every six months on an ongoing basis you would continue to check in so that you are finding out is it still doing what it did based on our initial findings?

Lauren Brechner: [00:18:43] To do this, you have to have the courage to hear that what you put out there didn’t serve the purpose of what you were trying to do and be okay to hear that feedback—that’s an important place to come from. You shouldn’t be reaching out with the expectation that folks are going to continue to sing the praises of the product. You are essentially opening a can of worms that you probably wouldn’t open otherwise unless you do this touch point. Each time you do that, it will create a new level of work. Even as infrequently as every two years is a good starting point—if that feels like a lot for you—because a lot can change in two years, but it feels like an approachable cadence for things. So learners who took something a year before, see if you can reach out to them, and see if you can have a conversation about what they put into practice, what tools they use from the course, what they remember about the course—whether it was a specific speaker or an exercise. Anything that sticks with them can speak to how the retention is happening and things that are enabling these learners to do what they need to do. Every six months would be wonderful, but that’s a full-time gig right there just having someone get this feedback. Two years is pretty good and is a good start for that.

Celisa Steele: [00:20:04] Let’s go back to talking about a portfolio audit, about tying to the mission and the vision of the organization that is behind this portfolio. What happens if the underlying strategy isn’t there, if maybe the mission isn’t clear, the vision isn’t clear, or there’s not a coherent strategy around how they’re achieving that mission and going to reach that vision? If that’s not in place, what then happens to a portfolio audit? How radically does that end up limiting what you can do when you sit down to pull your spreadsheet or database together with all of your offerings and the various metrics?

Lauren Brechner: [00:20:48] We need to admit sometimes that tying anything to a strategy can be hard because it can feel so abstract, especially if the strategy isn’t well defined or frequently talked about, and what it means in practice doesn’t always translate clearly. I like to anchor to mission and vision instead because those feel more actionable and tangible. Mission is what you’re doing, what you want to accomplish in the world. Vision is where you’re heading. Those are helpful framings to start to make it a little bit more piecemeal. Strategy is more the nuts and bolts of how things happen across the organization, that overarching thing, but sometimes it feels too big. If we can even distill to mission and vision, that’s really helpful. One mistake organizations make is…learning is part of your strategy. It needs to be embedded with that. Learning needs to be part of the discussion of strategy. There is some component of learning in any industry, any organization, whether it’s internal training, external training, capabilities building, trying to inform people about certain things. That is part of every organization, and it needs to be embedded as part of the strategy. Does a weak strategy limit the audit? Yes and no.

Lauren Brechner: [00:22:03] Mostly, a weak strategy makes conversations longer. When you sit down to talk about the relevance, reach, and impact of your offerings, the understanding of that alignment may be at odds with someone else’s, and that conversation is really important. You need to understand where someone else is coming from and how they’re defining how learning drives towards your strategy, and then what you’re thinking does that because those definitions may not mean the same things for you. A portfolio audit is a great way to pressure-test strategy and ask the question, “Are we doing what we think we’re doing, and does our learning reflect that?” I think of it as holding up a mirror. Here’s what you say you’re doing, and here’s what you’re actually putting out there. Does that line up? And, if not, which side of that needs to shift? Does the learning portfolio need to shift to meet current strategy, or does the learning portfolio better represent what your mission and vision are? And the strategy might need to better encapsulate that because it better captures reality. Learning people should be in the room when we’re talking strategic planning conversations for exactly that reason because, if there’s a gap there, then you’re telling two different stories, and you’re reaching for two different goals.

Celisa Steele: [00:23:18] I really like that idea of the portfolio audit being this chance to pressure-test strategy in the way that you were just describing. But then also, as you alluded to earlier, sometimes what can happen as part of that portfolio audit is also a difference of opinions—“Which are our top products?” or “Why are they our top products?” If there is internal disagreement or lack of alignment around “These really are our flagship offerings,” that probably does point to some level of strategy that’s either unclear in general or certainly unevenly communicated within the organization.

Celisa Steele: [00:23:53] Now, we talked about learning experience/user experience a bit earlier, and it can be common for organizations to sometimes feel like they need new tools before they’re going to be able to improve the learning experience. But I would like to hear how you think about the tech stack and what’s possible at an organization with what they already have, without necessarily needing new systems to do what they want to do.

Lauren Brechner: [00:24:24] I came from the association world (I mentioned before), and you have to be a little scrappy. I think that was a really interesting perspective to bring into my career. Also, even in our personal lives, we’ve all blamed tools for our inability to accomplish something, but that usually misses the real diagnosis. What’s prompting the desire to improve a learning experience? Are people not signing up? Are they signing up but not completing? Are they giving feedback that the course didn’t deliver the training results they’d hoped for? Those problems are almost always about content relevance, learning transfer, or marketing. It’s not the tool set. Nearly any existing platform can address them. And that’s why the portfolio audit matters so much—if you can define what that goal is for each of those offerings and adjust, in most cases you’re going to uncover that it’s content and marketing, and it is about applying the learning science to make sure that it has that impact over time. An LMS update is great. An LMS upgrade is great. There are wonderful tools out there that can do those things. But you want to pressure-test your content first, your marketing strategy first. And, after doing those things, once you feel like that’s a solid foundation, you feel like you need a tool that can build upon that, that’s the time to evaluate tools. We jump to the new shiny thing too frequently because there are a lot of promises out there about what it can do, but, if your content isn’t up to par, then it’s not really going to make a difference.

Celisa Steele: [00:26:05] I like that idea of thinking through “Why are you even considering new tech?” Because, again, that would put the emphasis on a hypothesis about what is or isn’t working, and then it gives you a chance to test that. As you’re saying, so many of the causes are not necessarily specific to one tool that you’re using, and they tend to often point to bigger issues around marketing or content alignment with audience needs and all of that.

Celisa Steele: [00:26:31] One thing that we always like to ask guests is about their own approach to lifelong learning. Lauren, what are some of your specific habits, practices, or sources as you continue to grow professionally and personally?

Lauren Brechner: [00:26:47] My approach to lifelong learning is that learning happens every day, everywhere. You just have to be open to receiving and acting on it. I love a YouTube video, and I like to listen to audiobooks. But I also learn a lot through conversations with people who love what they do and do it well, hearing about their failures and their triumphs. Getting that perspective is a method of learning. You can read something on paper, and it will have a certain impact, but to hear a lived experience is really valuable. Something that we sometimes forget is that the experiences of others can be a method of learning. Also, I just love the podcast format. I started listening nearly two decades ago when the iPods were coming out, and Harvard Business School was putting out some of the first ones. I listen to them all the time while driving, cleaning, whatever, and it keeps me sharp and gives me time to reflect. And I also keep a digital notebook with me all the time. I’m thinking in all directions at all times. But if something hits me that I’m like, “I want to research that later,” or “Wow, that was a good point; that’s a great perspective,” I jot it down so I can at least capture it and either reflect or act on it as I need to. If I don’t write it down, it’s not actionable for me, so I try and be consistent about that.

Celisa Steele: [00:28:09] We’ve covered a fair amount of ground in our conversation today. If you were to be able to point listeners to one, two, or three points that you really hope they take away from our conversation, what would you highlight?

Lauren Brechner: [00:28:26] We talked a lot about the logistics of a portfolio audit and its benefits, but I think the most important takeaway about that is it’s a huge undertaking, and I don’t want to undersell that. If you’re feeling overwhelmed about where to start, zoom out. Ask, “Which course or offering best represents who we are as an organization and what we’re trying to accomplish?” Find that one course, find maybe three courses—even if you want to go that far—and narrow your focus there. Forget the rest of the portfolio for now. You don’t even need to put them in a spreadsheet. You can identify that course on what you know about your industry, members, clients, customers. Form that hypothesis, put it out for low-stakes testing, make some updates, and get some feedback, and then you’re on your way to making improvements that are actually backed by data. The other point we just talked about—learning offerings can take many forms. Learning happens everywhere all the time, but microlearning is even a way of doing this. You don’t have to have your huge on-your-LMS portfolio.

Lauren Brechner: [00:29:33] Taking into practice some of the things I learned from marketing, a drip e-mail campaign that hits folks every two weeks with a little bit of information about a topic they’re interested in is a genuinely effective way to fight the forgetting curve when people are cramming so much information, and then they forget it. That is a wonderful, inexpensive, low-touch way of helping extend learning over time. So, if someone took a course online and you want to extend that learning, maybe they get dropped into a drip campaign that continues to reiterate those points over a year. We can also have a PDF with a video. A page with a PDF and a video is another very accessible way to do something. I believe in interactive learning, and it has a lot of impact on retention. But, if that’s feeling daunting, don’t over-engineer it. Get your best content out there in a way that feels deliverable, but make sure you’re measuring the results. If you don’t have an LMS, and you just have a Web site, you can have a Web page with some of these courses, with PDFs and accompanying videos. Just start somewhere. Start getting feedback and measuring it. I believe every organization has the capability to share out their unique perspective through learning and provide value.

Jeff Cobb: [00:30:55] That wraps up the conversation with Lauren Brechner, founder and principal consultant at Ember Forge. Stick around for our recap.

Celisa Steele: [00:31:02] If you found this episode valuable, we’d be grateful if you’d share it so that more people can find the show and benefit from the Leading Learning Podcast.

Jeff Cobb: [00:31:21] One thing I’d like to underline from the conversation is Lauren’s perspective that a product can deliver multiple benefits but has to solve one primary problem. And getting clear on what that primary problem is will allow a learning business to honestly assess a product’s performance.

Celisa Steele: [00:31:38] I also appreciate that she admitted that doing a portfolio audit can feel like a daunting task, but don’t let the size of the task stop you. Zoom out, pick the few courses that best represent who you are and what you’re trying to accomplish, and start with those rather than trying to inventory everything at once.

Jeff Cobb: [00:31:56] And Lauren’s idea that a portfolio audit can double as a pressure test of strategy is worth repeating. Ask whether your offerings actually reflect what you say your mission and vision are, and, if not, decide whether to shift your portfolio or your mission and vision.

Celisa Steele: [00:32:13] Thanks again for listening—and see you next time on the Leading Learning Podcast.

To make sure you catch all future episodes, please subscribe on Apple Podcasts, Spotify, or wherever you listen to podcasts. Subscribing also gives us some data on the impact of the podcast.

We’d be grateful if you’d rate us on Apple Podcasts or wherever you listen. Reviews and ratings help the podcast show up when people search for content on leading a learning business.

Finally, follow us and share the word about Leading Learning. You can find us on LinkedIn.

The post Auditing Portfolios and Pressure-Testing Strategy with Lauren Brechner appeared first on Leading Learning.

Courses and conferences are the backbone of many learning businesses’ portfolios, but for frontline and deskless workers—who make up roughly 80 percent of the global workforce—that model often doesn’t fit how the jobs actually get done.

In this episode of the Leading Learning Podcast, co-host Celisa Steele talks with JD Dillon, founder of Every Shift Counts and author of The Frontline Enablement Playbook. JD explains why frontline and deskless workers need a different approach to learning—one built on reliable access to information and connection to peers rather than courses and events—and makes the case that what works for frontline learners tends to transfer to desk-based ones too, so there are lessons here even for learning businesses that don’t serve frontline workers directly today.

To tune in, listen below. To make sure you catch all future episodes, be sure to subscribe on Apple Podcasts, Spotify, or wherever you listen to podcasts.

Download a PDF transcript of this episode’s audio.

Celisa Steele: [00:00:03] If you want to grow the reach, revenue, and impact of your learning business, you’re in the right place. I’m Celisa Steele.

Jeff Cobb: [00:00:10] I’m Jeff Cobb, and this is the Leading Learning Podcast.

Celisa Steele: [00:00:17] Help center agents. Nurses. The people fixing your car and ringing you up at the grocery store. They’re all part of the frontline workforce, which makes up roughly 80 percent of the global workforce—and yet when we talk about “learning,” we’re usually picturing someone with a desk and the ability to decide to carve out dedicated time for focused learning.

Jeff Cobb: [00:00:39] Our guest in this episode, number 487, is JD Dillon, founder of Every Shift Counts and author of The Frontline Enablement Playbook. JD spent the first half of his career in frontline operations, a decade at Disney supporting hospitality and contact center teams, and the last decade building technology used by four million frontline workers around the world.

Celisa Steele: [00:01:02] JD and I talk about why courses and conferences—the traditional backbone of a lot of learning businesses’ portfolios—don’t fit how frontline work happens and what is more effective: reliable access to information at the point of need, connecting learners with peers who can show them how it’s done, and short bursts of learning that can fit into a few minutes.

Jeff Cobb: [00:01:22] And what works for frontline learners also tends to work for people who sit at a desk—even though the reverse is often not true. So this conversation isn’t only for learning businesses that serve frontline and deskless workers—there’s something here even if your learners have more control over their time.

Celisa Steele: [00:01:40] JD and I also talk about how a learning business that sells into an organization needs to understand that the buyer, the manager, and the employee each care about something different.

Jeff Cobb: [00:01:51] So, without further ado, here’s the conversation with JD Dillon.

Celisa Steele: [00:02:01] Your most recent book is about frontline workers. It’s The Frontline Enablement Playbook. You write in the book that you really scrutinized each word in the title other than “the”—that got a pass. But let’s talk about the “frontline” and how you define frontline workers.

JD Dillon: [00:02:19] That’s an interesting part of the conversation. It’s the first thing you always get asked because most people, including people who support frontline teams, don’t identify as frontline anything. They identify as retail, grocery, construction, and hospitality—the worlds in which they play. The way that I look at this definition is the frontline workforce are the teams whose primary role is to work directly with your customers, products, and services every day. They’re the people that we, as a society, rely on to keep the machine moving—retail grocery, healthcare, all of these different types of environments. In a lot of cases, they’re customer-facing individuals who have to work on site, don’t sit at a desk all day, are constantly moving, have a different relationship with information and technology and all the tools and resources that make up work. And, in some cases, they’re back-of-house, whether they’re chefs in a restaurant or in a manufacturing facility working on the line or in the agricultural space—people that we don’t regularly see as part of our day-to-day. It’s different types of work in different environments in different industries with different goals. But the way the job is done, how people spend their time, and both the organizations as well as our reliance on them are consistent in a way that is not true (in a lot of cases) when it comes to corporate team members.

Celisa Steele: [00:03:42] “Desk” came up when you were talking about these folks, and “deskless” is another word—you used it in the subtitle. You maybe considered it as part of the main title. What do you see as the difference between deskless and frontline, to the extent that you see one at all?

JD Dillon: [00:03:59] Consider contact center operations. I spent about six years supporting contact center operations in a higher education environment, as well as physical campuses where people were directly working with students. If you’re a contact center agent, you do sit at a desk all day, in an almost painful way because you can’t leave the desk. Rule number one of a contact center is don’t take people off the phone because that’s where the job is done, and that’s where sales are made, and that’s where customers are supported. There are different elements of that experience that support improved enablement and performance. I can reach you—you might have an e-mail address or a Microsoft Teams account. You’re staring at a screen—you can pull up information when you need to. A lot of that is missing when it comes to people who are in a deskless environment, like people who work in a movie theater or in a theme park.

JD Dillon: [00:04:48] However, there are still commonalities in terms that, even if I work at a desk as a frontline employee (likely an hourly employee), I work a structured shift. I don’t have a ton of autonomy. I can’t leave the desk. I can’t decide, “I’m going to stop taking phone calls for the next 30 minutes or so in order to do some additional training.” That’s just not how it works. So the elements of how the job is done—lack of autonomy, task-focused, customer-focused—are all there. There’s just a difference in terms of access to information and the tools that are used on the job. Even if you’re a contact center operations agent or the front desk person in a healthcare facility. There are a bunch of different people who fit under the frontline umbrella who are desked, but they still have common attributes with those who meet the deskless side of the workplace.

Celisa Steele: [00:05:36] Thanks for that clarification and highlighting the similarities of whether there is a desk there or not in terms of these frontline workers. As you know, a lot of frontline workers are employed by organizations that don’t have a human resources or an L&D function at all—some car washes, mom-and-pop-type shops, those kinds of things. I’m going to ask you a leading question here for the Leading Learning Podcast. Is that kind of need something that associations and other types of learning businesses can help fill for frontline workers whose employers don’t necessarily have those HR and L&D functions?

JD Dillon: [00:06:15] It’s a “yes, and…” type of an answer because I do not think it requires a dedicated L&D and/or HR function. Part of the challenge with enablement of large, distributed workforces is that we silo by function, tool, accountability, and metrics in a way that is detrimental to the person doing the job. Because you have a corporate communications team pushing information. You have the product owners pushing information. You have compliance and legal, HR, learning and development, operations. Where all of those conversations meet is the management team and the frontline employees who have to execute on all of their different priorities and instructions. The more we silo, the more difficult that gets.

JD Dillon: [00:07:01] When you look at smaller businesses or mid-market businesses, you start to see a natural consolidation of function because they just don’t have that many people in a support center. A regional grocery chain might have four people in their corporate training team, and they also own corporate communications and a bunch of other activities because they’ve naturally consolidated based on resourcing, but it also simplifies the experience for employees who are just looking for help to do the job. It’s less about needing these discrete corporate functions and more that there has to be someone who cares about helping people get the support they need to do their job effectively and have as positive an experience as they can in this job to make sure that they’re getting value from it and then supporting the value to the organization. It’s looking more at what people need to be successful and to drive outcomes and less about distinctly bucketing that by function.

Celisa Steele: [00:07:53] That idea of having an enablement person who’s focused on that and the outcome focus, like you’re saying, and what those frontline employees really need to be able to do their job well—it sounds like there is the potential for that enablement person to be drawing on many different sources, including potentially offerings that an association that’s relevant to that field or profession might be providing and offering.

JD Dillon: [00:08:16] They can’t do everything themselves. If you’re a multi-billion-dollar company, you have the resourcing—from a corporate perspective—to be able to build resources and tools internally. But then again it gets complicated because you have so many different competing siloed functions. It’s essential for someone who’s playing that kind of enablement role in businesses that don’t have so many people and so many tools to be able to reach out and get those resources. And the prototype is sales enablement. It’s right there. If you look at the difference between the traditional corporate training team and the sales enablement team—whoever’s driving sales enablement is goal-oriented. They’re driving pipeline and deals in terms of what they do.

JD Dillon: [00:08:57] Learning and development is one of the tools they have at their disposal, but they’re directly connected into the CRO or VP of sales, and they’re driving outcomes by enabling the team. It’s that same model. They’re not building all of their own sales methodologies. They’re reaching out to the models that are proven to work in this type of environment, pulling in advisors, consultants, content, and shaping the ecosystem in a way that helps drive the outcome but maximizes their resources. Because, if you have one person, that one person, even with AI and related tools at their disposal, can’t do everything on their own. And they shouldn’t be recreating the wheel, given that other organizations—especially organizations that are in a similar space, similar business model, or similar industry—are likely struggling in the same types of ways.

Celisa Steele: [00:09:50] If we think about association learning portfolios for a minute, at least historically, they’re often largely built around courses and conferences and certifications. I want your take on how that kind of portfolio works or potentially doesn’t work for reaching those frontline workers.

JD Dillon: [00:10:10] There’s a fundamental gap in terms of traditional tactics and the reality of how the job is done. As an example, there are a lot of small shops in the auto mechanics space. Well, let’s just talk about the facility. How many auto shops have a back room and/or a classroom setup where people can separate themselves from the work, sit down, not hear everything that’s going on, and focus on a course? How many auto shops schedule the number of people where they can pull people out of the garage, away from customers, and dedicate time to development? Now if we want to scale a support mechanism across this space, how do we do it in a way that’s respectful of how the job is done and fits and drives an outcome? Part of that conversation is can you make it easier for these organizations to make information available to their teams because they’re working with complex products? Does everyone need their own knowledge base, or is there a shared knowledge base opportunity that you can make available? Is it less selling courses and more selling reliable access to information in a way that’s easily consumed and applied? There’s still a place for courses in all of this story. I just don’t think it can be the primary mechanism or business model. Same for events.

JD Dillon: [00:11:24] There’s huge value in bringing people together, having conversations, and sharing practices, but it’s not something that you can regularly do because of how the work is done and how labor budgets are created. Can you make it easier for people to access information? Can you make it easier for people to practice in certain ways, working through certain challenges, or working with a simulated model of a piece of equipment that they might not regularly see but need to be ready to handle when it happens? Or a certain type of vehicle that, when it rolls into the shop, they need to be ready to go, knowing certain things—how to repair this safely while referencing certain information? Regardless of the industry and the type of work, there’s a certain consistency in terms of what people need, and it’s reliable access to information in a way that they can apply it. Increasingly in the frontline, it’s also about accessing information in the language that I prefer and using tools (maybe I’m not the greatest reader) that can help me understand and apply information. Is there a different way we can think about tooling people with resources, scaling those types of resources across an industry, and being that type of team or that type of provider?

JD Dillon: [00:12:34] I’ve worked with providers that support associations in providing those practice activities and reinforcement activities. It’s not just about courses. If you work in a shop, and you have three minutes today, we could provide an activity that’ll help refresh what you need to know or introduce a new topic to you for three minutes. It’s similar to courses, but it’s restructured to match the access points and the time availability people have on the job. When I talk about three minutes of shift, that doesn’t sound like a lot. What can you even learn in three minutes? Three minutes every day of focused activity adds up pretty quickly versus once a quarter we put everyone in a room and tell them stuff because that’s not how people can effectively consume and retain information. Three minutes a day actually matches how we work as people in terms of our memory, especially if we’re able to apply that information quickly. So it’s restructuring the model around how the job is done and what time is available because those are the biggest obstacles. Even in corporate environments people struggle with those exact same factors.

Celisa Steele: [00:13:42] How much is your critique of the course-and-event-heavy approach to learning specific to frontline workers? Or is it what we know about how humans tend to work and learn best? It’s in those shorter and more consistent bursts. It’s in the flow of work, all of that.

JD Dillon: [00:14:04] Based on my experience, I find enabling frontline workers to be infinitely more interesting than talking about corporate employees. Because it represents 80 percent of the world—about 2.7 billion people. This is where the numbers are. Every environment has these unique attributes to them, or it’s a puzzle of figuring out how to match how we can best support people based on how people work and the unique environment and challenges of that particular industry, company, workspace, job. It’s a really interesting puzzle. But, at the same time, if we figure this out for frontline workers, or we have practices that work in these types of environments, they transfer because we’re talking about people, like you said. The limitations of a course mirror the limitations of a course from the frontline to the corporate environment. The challenge is it doesn’t go the other direction very often. Tools, resources, and technologies that work in a corporate setting have a hard time in frontline environments because, for some reason, we decided, as an industry, that learning and development is a digital concept. I don’t know why. I don’t think e-learning is that good and has been that helpful for the last 25 years to anybody, but it provided scale and consistency and speed. It’s cheaper and faster to put people through that environment versus figuring out the logistics of putting people in a room, especially if those rooms are not all in the same place.

JD Dillon: [00:15:33] Then again you look at the frontline workforce—the vast majority of structured learning activity takes place shoulder to shoulder with a peer or manager, and it’s hands-on job training, a conversation we just don’t have a whole bunch anymore. A lot of the things, if you make it work in the frontline where time is restricted, people in corporate roles also don’t have a lot of time. Overloaded. All types of stuff. But most corporate employees can maybe decline that meeting. A lot of people are salaried, so they can decide, “I’m going to stay late today” or “Over the weekend, I’m going to….” Most frontline workers—at least in certain parts of the world—can’t make those decisions. In fact, we often cut off their access to information as soon as they’re clocked out for the day. Rules. The cool thing is that, if we have more conversations that are not necessarily focused on frontline and deskless workers but inclusive of frontline and deskless workers, it works for everybody in a lot of cases. We just rotate in the direction of people who sit at a desk and do their job the way I do my job and forget that, oh, yes, 80 percent of the world doesn’t do that.

Celisa Steele: [00:16:44] You brought up the asynchronous side—the emphasis on these e-learning modules that can be consumed on your own at whatever point. Even if they’re shorter than what may be the typical module, even if it’s that three-minute thing, it’s still on your own, doing it yourself. But you see the value of social learning, the peer learning, as you were saying there. Talk a little bit about how that could happen. Because, like you’re saying, those place-based events where you bring people together for the sharing—very valuable also very hard, depending on the individuals you’re trying to bring together, to get them there, to have that time out of the office, to be able to afford bringing those people together. What do you see that’s working to connect some of these frontline workers with other peers socially so that some of that transfer you’re talking about can happen?

JD Dillon: [00:17:37] A couple of different things I’ve seen. One is thinking about cohorts differently because a lot of people are moving through the same motion or trying to develop the same set of skills or move forward in a similar way, but they don’t see one another, and they’re distributed in particular ways. Onboarding is an easy example. There are a lot of people being onboarded right now in the same company in different locations and never seeing one another. Is there a way that we can leverage technology to put people into a virtual cohort where they feel comfortable talking to people who are more like them and asking them questions? This solves a problem that I see on the Internet because, for some reason, there are a lot of frontline workers asking questions on Reddit. They’re in the subreddit of their company asking very tactical job questions. I sit there and wonder, “Where’s the trainer and/or manager?” Because these are easy. You shouldn’t have to turn to the Internet for this. Are you trusting Reddit? You shouldn’t. But, in a lot of cases, people are giving good information because it’s a peer group that understands. Reddit is not necessarily reflective of the entire employee experience, but it’s a lot of people who are very loud and have opinions, and you start to see trends when you pay attention to the things they talk about. There are examples in my book from those types of conversations.

JD Dillon: [00:18:50] There are ways to look at cohorts differently, even from a virtual perspective, even a chat group in an employee communication application that might put people into a group to say, “Hey, you all are onboarded at the same time. If you’re running into situations and can’t get to your manager, or your trainer doesn’t know what they’re doing, you can talk about and share your experience.” Maybe it prompts people to say, “How was the first day? What are you still not thinking about?” So people know that they’re not in this alone. They’re not the only person onboarding in this company right now. They might be the only person onboarding in this location or on this shift. A company I worked with that works in the airport lounge space, one of their priorities was making their big company feel small. They always looked for ways to connect people through social tools so that they could learn from one another but also experience different activities with one another. It was less about social media-type activity because we over-rotate into thinking people will act at work like they act on Instagram when they’re nowhere near as interested in what they’re doing every day, nor willing to be like, “Let’s take photos of everything.” But, if you prompt people in certain directions, and you recognize people for certain types of sharing, people are more likely to engage—not everybody, but some people will engage. And that’s an element of the story.

JD Dillon: [00:20:06] An example from another organization I worked with in the retail space—this person was trying to figure out how to improve product knowledge across a network of retail stores. They don’t come from a training background, but they do come from a media background. Instead of building e-learning or any traditional solution, they just went to a store with their phone and recorded a conversation they had with an employee about a particular product. You wouldn’t necessarily look at that and think it’s a social activity, but it featured people who look like me, who do the job that I do, talking about how they have conversations about certain products or solve certain problems in the same environment. We make a mistake in assuming that people who do this job every day are going to react to either content that doesn’t reflect their reality or content that’s coming from corporate or the executive team, when, in a lot of cases, people want to see people like them.

JD Dillon: [00:21:10] It may be someone from my company, may be someone who does my job. If you’re a mechanic, and I’m a mechanic, we just reflect differently. Another company I was talking to the other day sells content for people who do roofing, and their videos are low-production videos of people on a roof. It’s a guy who clearly does this job, talking about how he does particular things. Reflecting the world in which these people work and leveraging people who have credibility in their willingness to share what they’re doing. And then we can have a whole conversation about the effective use of AI avatars in this story. Todd’s example from product videos is in my book. It’s also the reason why this is the only book about frontline performance that will ever include a deer urine story because he works in a retail store that sells deer urine. He shot a video about deer urine, and then wacky things happened in that video. But guess what? After the video got released, they sold more deer urine. That’s the point.

Celisa Steele: [00:22:16] You brought up AI avatars, so I’m going to ask you about the use of AI among frontline workers. You brought up Reddit and people turning there. To what extent do you see AI as a threat? A tool? How do you view it? Because there can be workers just turning directly to whatever they might have access to on their own phone or if they do have access to a computer.

JD Dillon: [00:22:50] Frontline workers have been looking up stuff on the Internet on their own devices forever, so none of this is new. Whether it’s approved or not. My research as well as my experience around the use of AI in frontline teams has to be informed by the fact that, again, we leave frontline workers out of conversations around the future of work constantly. The remote and hybrid work conversation is a great example, where the vast majority of workers have no option, and at no point did we include…because the hybrid/remote work conversation is ultimately a conversation about workplace flexibility. You can have a conversation about workplace flexibility with frontline workers too, but it’s more about scheduling practices, making it easy to trade shifts, and that type of stuff where there is a role for technology to play to make all of that more seamless. Today you should not require a manager to be less busy to figure out if you can adjust your schedule. We should have figured this one out by now. But there’s a whole future-of-work conversation that’s missing from the frontline workforce, and that includes the broader conversation around AI, where I’m excited about what the tech can do now for frontline workers. And it doesn’t have anything to do with the word “agentic.”

JD Dillon: [00:24:01] When you talk about AI in the frontline workforce, it’s a combination of how tools are used and then how people are impacted in a very specific way. Frontline workers are way less concerned about this because there’s a level of stability in frontline work. A lot of people who do these jobs understand the value that they provide, and the automation conversation’s been impacting them for decades at this point because companies are finding ways to reduce labor hours, and it’s a constant struggle to be able to get my hours, to get the paycheck I need if you work in these jobs, in a lot of cases. That’s not new. That’s the nature of this work. It’s a lot newer on the corporate side of the conversation right now. But, when you look at existing technology capabilities, inclusive of AI, there are certain things there. A simple example: access to information, being able to ask a question and get an answer back. Huge when it comes to these jobs because how often are you relying on the person next to you to be able to answer whatever question a customer comes in with? And customers come in with way more information than they used to. I’m one of them.

JD Dillon: [00:25:04] When I buy a car, I know more about the car than the person selling me the car because I’m doing a ridiculous amount of research and then coming into the conversation informed. And that is true in retail. It’s true in hotel environments in terms of the services available at this hotel, restaurants in terms of the ingredients and the food. People just know things and therefore have a higher expectation of what the employees know and are able to do. Having information at your disposal, where, if you just started yesterday, or you’ve been here for 30 years, we can level out that knowledge base a little bit more. The translation piece of things is huge for frontline workers. The fact that we’re going from a world where, when I was developing training in a corporate environment, we had to pick because translation was a time-consuming and expensive problem. We translated information to three to four languages—the most commonly spoken languages in our environment—not knowing who did we just leave behind because they need information communicated differently. When I was in tech, one of my favorite things to do was to pull language information from the customer base, just to see who was using our tools in what languages, and I loved finding a company where there was one employee in one region of the world who was picking one particular language—because that was the whole point.

JD Dillon: [00:26:22] That one person now has a chance where, in the past, they spent a lot of time potentially understanding information, yet alone learning or applying anything. So translation in all forms, while not perfect by any means…. AI is still largely written in English in terms of its capabilities, but we can do more to the point where, when I present in different languages, I translate all of my own stuff, and I’m usually near to perfect, but I always have someone who has expertise in that language review the information for me when it’s important. Social exchanges, basic communication stuff—we can translate that in a large number of languages now as well as we need to be able to. And then one layer deeper (I already mentioned this) is regardless of what the job description says, you have a lot of people who are doing these jobs who are not great readers for different reasons. We can now use language differently, and people can talk to and engage with information and get the information to come back to them—that’s meaningful. Again, in a way that we tend not to have to think about in a corporate environment. There are obviously accessibility considerations and all of that side of it.

JD Dillon: [00:27:33] I put a mobile device in your hand, and it can translate the information—great. But what if you can’t read or can’t read well? Now you can talk to the information, and the information can talk back. It’s things like that that could knock down or reduce some significant barriers in terms of people’s ability to learn and perform on the job that are driven by AI, that don’t have to be complex workflow automations, and all that kind of stuff. I would add there’s a key thing to consider around how AI is being used in ways that impact people that don’t have anything to do with the tools they use on the job every day. If you’re using AI for hiring practices, for scheduling practices, if in some way the data and AI is being used to inform decisions around promotion and training, we have to keep in mind that people need to know how that works, how tools are being used to influence their earning potential and their lives.

JD Dillon: [00:28:35] Especially because there are a lot of frontline environments where you absolutely have to make that information available because you’re working in a unionized environment, or you’re working in a highly regulated environment, or we’re in Europe at all. There are distinct ways where technology and data influence the job and the meaning and motivation behind the job that we have to make sure is clear. And it can’t be a black box that managers can’t explain why people aren’t getting hours. It has to be explicit. Some people are very much not going to care about that story. Some people are very much going to care and click the little question mark next to whatever information and want to understand. “Why am I getting recommended this training?” or “Why am I being scheduled in a particular way?” or “Why didn’t I get recommended for promotion?” We need to be able to answer those questions based on why people do this work.

Celisa Steele: [00:29:31] You’ve been talking a lot about the importance of reaching workers on the job in their actual workflow. That’s a real goal that has real benefits, being able to do that. Associations and other learning businesses obviously have a hurdle that those internal L&D teams don’t. They can’t readily reach those individual frontline employees as directly, as easily. They might be selling to the employer, and then their access to that worker is limited to or mediated through the employer or potentially managers. This kind of proximity issue exists in other contexts too, like franchises. I would love to hear a little bit about how you might solve for that distance issue or at least think about it as something that could be addressed or how it might be addressed.

JD Dillon: [00:30:25] The motivating factor for the buyer, the manager, and the employee all need to make sense based on what they care about. And the answer to all of those is not learning. Because fundamentally no one on the operations side of a business—from the employee through the entire hierarchy—fundamentally cares about the concept of learning. I didn’t use the word “learning” very much at all for the past 10 years, despite the fact that I’ve been in employee enablement and performance the whole time. However, I talked about things like confidence, capability, safety, and risk—those factors. It’s critical that, whatever the solution is, it has to check the box in terms of something I care about for the person who’s making the decision to bring it into the organization. If you’re selling to the VP of operations or the COO or whoever it might be, what do I care about? I care about growing the business and outcomes associated to that. How is this solution going to help me? And the answer is never learning. It’s something about the capability. I love using the term “readiness” around operations teams because they understand that “We need to be ready,” right? “Yes, we do need to be ready.” What does it take to be ready?

JD Dillon: [00:31:34] Knowledge, skill, and capability are all underneath that, but it’s not the headline. It’s supporting capabilities. Then you drop it down to the management team because the management team owns the day. They’re your most important stakeholder because, if you don’t have them, you have nothing. When it comes to franchised operations, as an example, corporate may love it, the franchisee may love it, but, once you hit the restaurant, the shift manager has something to do, and they have a limited number of people to do it, and they’re facing off against the customers and the food inspectors and all of that type of stuff. How is what you’re providing going to help me (manager) hit what I need to hit? Because usually managers make decisions based on two criteria. One, what’s going to get me a phone call? I do whatever it takes not to get the phone call. And, two, how I’m bonused. So the metrics that impact my compensation and the things I’m held accountable for by my boss, how does this help me with those things? That’s going to earn the time and access that you need to be able to reach. Ultimately the manager should be included in the solution, as well as the employee.

Celisa Steele: [00:32:36] If you were to point to something that you hope listeners take away from our conversation today, what would you most like them to remember or think about after hearing this?

JD Dillon: [00:32:48] Managers are the most important people in the workplace. If you invest in anything when it comes to enabling frontline teams, look straight at the managers because of how much they influence the experience. And the more that you understand how the job gets done and the workflow, the more likely you are to be able to shape a solution that people will actually use, actually get value from, and you’ll actually be able to retain that organization, that customer base. Because so often things sound like a great idea—the buyer leans in because it’s trendy; it’s AI. Then you slam into reality, and people aren’t motivated to take advantage of the solution, don’t have time to take advantage of it, and then you have a really short relationship with that stakeholder or buyer in that organization. Being a person who came from the technology space for the last 10 years, what made me effective was that I did the homework. I spent a lot of time in grocery stores. I not only understood how a grocery store works, but I understand what the owner of a grocery store very much cares about.

JD Dillon: [00:33:51] One particular gentleman said something to me that is in the book, and it echoes in the back of my head: The fact that I come from frontline operations means I don’t just think about ideas and principles and practices. I’m thinking about people when I make decisions or when I make recommendations because it’s the experience I had with these individuals and what I learned from them that I’m pulling together to shape a method, a framework, and practice. This one individual, who runs a set of grocery stores, said to me, “JD, I’m not changing for you. I want spandex, not concrete.” He needed a solution that wrapped around how he runs his business because he’s not going to change his business for what I can do. I’m an expert in human performance. He’s an expert in the grocery business, so I have to meet him where he is. The more you can do that—especially if you specialize in an industry, if you specialize in automotive, you need to know automotive, not just sell to automotive. Because people who run these businesses and own these businesses, this is what they do. It’s serious. I’ve had so many conversations with grocery store owners about loss prevention, waste, and cheese. For some reason, whenever I visit a grocery store, people want to show me the cheese case.

Celisa Steele: [00:35:02] I’d want to see the cheese case.

JD Dillon: [00:35:04] It’s a higher-margin product. And then they’ll also walk me through the spoilage process in terms of produce. They try to sell the pepper, and, if the pepper doesn’t sell, then they slice up the pepper and put it in packaging. They sell you pre-sliced peppers. If you don’t buy it by the sell-by date in that case, they slice it up more; they put it in a salad. They try to sell you the fully-made food product. If that doesn’t sell, then we have a waste situation. They’re doing everything they can. They need the employees to understand that waste prevention. Spoilage equals major costs for a low-margin business, so they definitely want you to understand those things. For whatever business that you support or that you’re selling products or services to, do you understand their version of that so that you can go in and have a conversation?

JD Dillon: [00:35:47] When I go to a manufacturing facility, and the VP of safety is sitting there, I am absolutely not saying the word “learning” in this meeting, but I’m going to say, “Is it important to you, as the vice president of safety, to have a capable, confident team that’s ready to make the right decision at the right time, to keep the line moving, to keep people safe?” The person on the other side of the desk is going to say, “That is absolutely important.” And then we’re going to talk about how we’re going to be able to get there. So do the homework—understand, appreciate, and respect the work that is done by these folks every day so that you can provide a solution that makes it clear that you’re contributing to their industry and their business. You’re not just trying to sell them stuff.

Jeff Cobb: [00:36:31] That wraps up the conversation with JD Dillon, author of The Frontline Enablement Playbook. Stay with us for our recap.

Celisa Steele: [00:36:38] earn more about The Frontline Enablement Playbook and about JD’s work in general. And, if you found this episode valuable, we would be grateful if you’d share it so more people can find the show.

Jeff Cobb: [00:36:56] One point JD makes is that courses and events alone aren’t effective for frontline workers—their jobs simply don’t allow for it. He points learning businesses instead toward reliable access to information, connecting workers with peers who can show them how it’s really done, and short, frequent practice that fits into the gaps in a shift. And he argues that what works for frontline learners tends to transfer to people who sit at a desk. So, even if frontline and deskless workers aren’t who your learning business serves, there’s likely something here worth borrowing.

Celisa Steele: [00:37:33] When it comes to reaching the frontline workforce, JD’s advice is to design for the buyer, the manager, and the employee, with a particular focus on the manager since they own the day-to-day experience.

Jeff Cobb: [00:37:45] Thanks for listening—and see you next time on the Leading Learning Podcast.

To make sure you catch all future episodes, please subscribe on Apple Podcasts, Spotify, or wherever you listen to podcasts. Subscribing also gives us some data on the impact of the podcast.

We’d be grateful if you’d rate us on Apple Podcasts or wherever you listen. Reviews and ratings help the podcast show up when people search for content on leading a learning business.

Finally, follow us and share the word about Leading Learning. You can find us on LinkedIn.

The post Frontline Lessons for Every Learning Business with JD Dillon appeared first on Leading Learning.

Many learning businesses are data-rich and insight-poor. It’s easy to track what’s easy to count—e-mail opens, registrations, completions—but harder to know if the products are doing what they were created to do.

In this episode of the Leading Learning Podcast, Jeff Cobb and Celisa Steele discuss what learning businesses should be measuring and why. They identify the most common metric traps—including surrogation and vanity metrics—and introduce a five-lens framework for thinking about which metrics matter at which stage of a product’s lifecycle: Validate, Reach & Convert, Engage, Retain & Reactivate, and Prove Impact. They also make the case that a metric without a target is just a number and that the most valuable feedback often comes not from dashboards but from conversations with learners and employers.

To tune in, listen below. To make sure you catch all future episodes, be sure to subscribe on Apple Podcasts, Spotify, or wherever you listen to podcasts.

Download a PDF transcript of this episode’s audio.

Celisa Steele: [00:00:03] If you want to grow the reach, revenue, and impact of your learning business, you’re in the right place. I’m Celisa Steele.

Jeff Cobb: [00:00:10] I’m Jeff Cobb, and this is the Leading Learning Podcast.

Jeff Cobb: [00:00:18] Talk to enough learning business leaders, as we do, and a common worry crops up: Is this program we offer working and having an impact, or have we just gotten used to running it?

Celisa Steele: [00:00:30] And that worry isn’t contained to a single offering. Most learning businesses, of course, run a whole portfolio—a conference or two, some on‑demand courses, maybe a certification—and the honest answer to “Is this working?” might be different for each product.

Jeff Cobb: [00:00:47] Which raises another question: How do we know if a product is working?

Celisa Steele: [00:00:51] Elizabeth Engel touched on this back in episode 470, when she walked through the build-measure-learn cycle of the Lean Startup methodology. She said that figuring out which metrics matter could be a whole podcast in itself.

Jeff Cobb: [00:01:06] So we’re taking her up on that especially because the importance of metrics is a theme that keeps resurfacing.

Celisa Steele: [00:01:13] Steven Khraiss talked about lagging versus predictive indicators in episode 485. And, in episode 479, Eric Ries (of Lean Startup fame) talked about how metrics can become a substitute for the thing they’re supposed to measure. Metrics are essential to our own focus on reach, revenue, and impact and making sure that learning businesses are achieving all three of those.

Jeff Cobb: [00:01:39] Metrics are something everyone agrees matter and very few people feel settled about. So today we want to dig into what learning businesses should pay attention to, when, and why. We’ll name some of the traps that make metrics less useful than they should be, and we’ll offer a set of lenses for thinking about which metrics matter at which point in an offering’s lifecycle.

Celisa Steele: [00:02:05] We’ll also make the case that a metric needs a partner—a target, an expectation, a reason you think you’ll see what you’re hoping to see. Because a number on its own can be misleading.

Jeff Cobb: [00:02:18] We mentioned that there can be some traps with metrics, things you can fall into that aren’t the behavior that you want in using metrics. So let’s talk about the traps that make metrics less useful than they should be.

Celisa Steele: [00:02:31] A very common problem is that any specific learning business has a ton of data, but there’s no real plan for what to do with that data. Back in episode 436, Alaina Szlachta said, if you don’t have a plan for what you’re going to do with data, then don’t even collect it in the first place. Because once you have it, it can be a little bit of, “What do we do with it now?”

Jeff Cobb: [00:02:55] Another problem can be categorized as surrogation, or metrics becoming a surrogate. Eric Ries uses that term in episode 479 for when a metric becomes a stand-in for the thing it was supposed to measure. His example is customer service, and the concept of average hold time became a big goal instead of the proxy that it was meant to be. Companies found ways to get people off the phone without solving their problem. Maybe you have experienced this—I know I have—and customer service got roughly twice as bad over the course of 50 years.

Celisa Steele: [00:03:32] And then there’s also the trap of vanity metrics. Amy Michalski and Shelly Strickland of AIIM in episode 482 talk about this. They believe that engagement and the quality of engagement matter a lot more than things like open rates, which Shelly characterized as a vanity metric because it’s not really telling you a whole lot about what your learners or your audience values or needs. It’s the attendance, the engagement, the genuine engagement that tells you what your learners are looking for.

Jeff Cobb: [00:04:06] I had something like this happen recently on a survey an organization was running, and the marketing team came back and said, “Very high open rate. A 90-percent open rate on the survey e‑mail.” But nobody had taken the survey. It didn’t really matter that they opened the e‑mail, so vanity metrics are definitely a danger. Another is the danger of metric fixation, or an over‑reliance on a measure rather than the goal you’re after. It can get easy to game the numbers. We see this happen all the time with things like the Balanced Scorecard, for example, which can be so valuable, but, if it becomes all about the numbers on the scorecard, and you forget what the scorecard was there for in the first place, then that value erodes. Make sure that the real goal, the real mission behind the metric, is not lost.

Celisa Steele: [00:04:54] One other trap or danger to watch out for when you’re thinking about metrics is that no single metric is going to tell the whole story. This came up in that episode that I did with Amy and Shelly from AIIM. It might be that a Webinar doesn’t convert to a course sale, and that could be seen as a failure. But maybe it’s bringing in highly engaged prospects who ultimately convert to membership six months later. If you’re just looking at a single metric—did someone come to this free Webinar and sign up for the paid course immediately after?—that’s not going to tell you that whole story of that cultivation that you might be doing as part of that free Webinar to ultimately bring people into the fold, into membership, and into your learning portfolio. Again, it can be hard to rely on any single metric to tell you the whole story.

Jeff Cobb: [00:05:47] Lots of potential traps. There are others that we haven’t even named here. You need to be careful about them, but we’ll stress that none of this is an argument against metrics. We are big fans of metrics. It’s just an argument for being very deliberate about which ones and why and being intentional about how you’re using those metrics.

Celisa Steele: [00:06:07] What we want to talk about next is the idea of pairing your metrics with a target or a hypothesis. I want to go back to something that Alaina Szlachta said in her interview that we did with her. She very elegantly and succinctly described measurement and evaluation as answering this question: Did we do what we said we would do? And that means you need to have said what you were trying to do. This is her idea of an “impact hypothesis”—you need to be clear on what difference you’re trying to make. What are you trying to do? And then you can start measuring whether you’re having that impact, if you’re making that difference that you set out to do. You have to have this hypothesis, this idea of what kind of impact you want to have before you get going because that’s then what you’re going to measure against.

Jeff Cobb: [00:06:56] An example might be “We should see higher open rates because of this segmentation.” And then you test that. Do you get higher open rates because of the segmentation that you’re doing? Note that AIIM sets targets around registration, lead generation, and membership campaigns but stays flexible because audience behavior shifts year to year.

Celisa Steele: [00:07:17] The point there is that you could have this impact hypothesis about what you’re expecting to see, but you don’t have to treat that as being set in stone. You can learn from what data is coming in. It might help you tweak your impact hypothesis, get a little bit more specific, as long as it’s based in your audience’s behavior and what you know about them. That ties into the idea that a missed target isn’t necessarily a failure. You want to have the metric. You want to have that target. Maybe you don’t hit that target, but treat that as a reason to ask why rather than treat it as an outright failure. And, by asking why, you can begin to think about what you might need to do differently to help you achieve that impact that you were hoping to achieve in the first place.

Jeff Cobb: [00:08:03] A practical, concrete tip might be that, before you launch a product or before you pull a report, write the sentence “We expect to see _____________ because _______________.”

Celisa Steele: [00:08:16] Once you know what you expect and why, the next question is “What counts as a reasonable baseline for comparison?” This is why we want to talk about having that right expectation that is specific to your audience and to your learning business.

Jeff Cobb: [00:08:35] Related to the last point, your target—or, if you’ve been collecting numbers for a while, the baseline you’re comparing against—needs to reflect how a specific audience or segment behaves, not a generic assumption. We’ve had a client, for example, that had many of its audience who couldn’t get to its annual event every year because someone had to stay and keep the practice open. This was an industry in which people are running health‑related practices. So, for a lot of attendees, the real cycle is closer to every third year, not every year. And a generic year-over-year repeat attendance target would read that as a retention problem when it really isn’t. It would be a case of a wrong target, not behavior that indicates a problem with the offering.

Celisa Steele: [00:09:26] Yes, and we’ve heard that story from other organizations as well—other types of professionals who either can’t afford in terms of dollars or can’t afford in terms of time out of the office and away from work to attend that annual conference every year. In at least one case—and there are many, many more—part of what the learning business did was add a virtual option for the years when people can’t attend in person. That’s a simple solution of building an offering around the real cadence for a specific audience rather than trying to fight that natural cadence and deal with, in some ways, a non-existent problem, as you were saying, Jeff.

Jeff Cobb: [00:10:09] Right. A practical, concrete tip would be, before treating a number as a problem, ask what the true, reasonable rhythm is for the specific audience that the number applies to and whether your target reflects that.

Celisa Steele: [00:10:26] This is very much in line with a caution that Steven Khraiss raised in episode 485 about benchmarking (episode 485). It may sound obvious, but comparisons only mean something if you’re comparing apples to apples. What might be the cadence or the expectation for one particular audience may not apply to another. Make sure that you’re benchmarking against the appropriate measures and data.

Jeff Cobb: [00:10:55] Now let’s move to looking at different lenses to view metrics through, a lifecycle of lenses for metrics.

Celisa Steele: [00:11:05] We have five that we want to offer, and these five can apply to any offering. Which of the five lenses matters most is going to depend on where that offering is, how mature it is, or where a particular cohort is within it across its lifecycle. As a preview—we’re going to talk about these more—in order across an offering’s lifecycle, the five lenses are (1) Validate, (2) Reach & Convert, (3) Engage, (4) Retain & Reactivate, and (5) Prove Impact.

Jeff Cobb: [00:11:38] If you’re talking about a brand-new offering, there’s only going to be one cohort initially, so the lenses would unfold sequentially in that case—there simply hasn’t been time for later lenses to apply. For a more mature, ongoing offering, the lenses are concurrent because cohorts overlap. Take the example of an in‑person workshop. This year’s new registrants are in Reach & Convert, last year’s attendees are in Retain & Reactivate, and the cohort from three years ago is old enough to assess in Prove Impact.

Celisa Steele: [00:12:14] As you’re saying, Jeff, the lenses can be both sequential and concurrent. Whether this looks sequential or also looks concurrent depends on how much history the offering has. Again, a brand-new offering, one cohort, so you’re going to watch it move through those lenses one at a time. An offering that’s been around for years is going to have those multiple cohorts stacked up at once, so multiple lenses are going to be in play simultaneously.

Jeff Cobb: [00:12:39] Let’s talk about each of these lenses a little bit more. We’ll start with Validate. The purpose of this lens is to test assumptions before committing significant resources. This is the beginning of the cure against “Build it, and they will come.” You want to do some testing first to see if it’s something that your audience is interested in.

Celisa Steele: [00:13:02] This might be where you’re building a minimum viable product and measuring with a real call to action. You’re not just doing some hypothetical “Would you sign up for this?” or “What would you think of an offering like this?” You’re asking learners or prospective purchasers to take an action, and then, based on that call to action, you’re going to then decide, “Okay, yes, we’re going to persevere with it. Nope, we’re going to pivot it. It’s not quite landing right. Or, you know what, it’s really not gotten any traction at all, so we’re going to kill it.”

Jeff Cobb: [00:13:33] And this can be simple. A classic here is asking people to submit their name and e-mail to get more information about something. If they’re willing to take that small action, that starts to show you some interest. But you want that call to action, that concrete action, to be a test. It’s not a product launch yet. You may not be asking them to purchase something or, at least, not purchase the full thing that you want to build. You’re, at this point, willing to walk away from the whole idea depending on what you learn. If nobody’s clicking anything, if nobody’s putting in their e-mail address, if nobody’s willing to make a micropayment, then you’ve got good information about whether you should proceed or not.

Celisa Steele: [00:14:09] This Validate lens is clear Lean Startup territory. It’s what we talk about with Elizabeth Engel and Jamie Notter in episode 470. It’s what we talk about with Eric Ries himself in episode 479. Again, for a brand-new offering, this is going to be the only live lens. That said, it’s still worth setting a soft impact hypothesis even here. You still want to begin thinking about at this point what does this need to be? What does it need to do if it’s going to work? What are we trying to do here? For a mature offering, this Validate lens is going to resurface when elements or components are shifted, and that’s going to happen not necessarily on any set schedule but when you vary an element or component of an offering. For example, let’s say you have an existing annual conference, but you’re going to try out a new conference session format. You’re going to need to validate that and spend a little bit of time in some of these lenses with that new element or component. It might mean adding that virtual option along with the in-person conference attendance option. Even something as simple as swapping out a Webinar title to see what helps move conversion—that’s something that came up with Shelly and Amy from AIIM in episode 482. And then occasionally Validate will resurface at the whole offering level too.

Jeff Cobb: [00:15:37] Yes, that’s right. When you’re getting a weak signal from one of your later lenses—so you might have a real drop in impact or engagement, not just a natural dip in your cadence—that can send you back to ask whether the whole offering still deserves to exist or whether you want to test some sort of key change that might drive better performance from the offering going forward. All of this also points to the topic of portfolio governance and sunsetting, which we covered in depth in episode 476.

Celisa Steele: [00:16:13] Validate is the first lens. The second lens is Reach & Convert. In this lens, you’re looking at making sure that people who would benefit from a specific product know that it exists—that’s about your reach—and that you’re turning some of those people into paid participants or, at least, registered participants (if it’s a free offering)—that’s where you’re converting.

Jeff Cobb: [00:16:37] This is different from the concrete action in Validate. To get to this Reach & Convert lens, you’ve already decided the offering is real because of your testing and that you’re going to proceed in offering it as a full-fledged item in your portfolio. Now, as part of that validation process, you’ve started to build the reach that you’re going to need for this; you’ve got some people interested in it. The action that people take here is going to be part of the ongoing funnel that you’re now developing to drive people into converting to and paying for the product. It’s not just going to be that one-off test of whether to proceed.

Celisa Steele: [00:17:18] In episode 482, one of the takeaways from Amy and Shelly at AIIM is this idea of, as you’re trying to Reach & Convert, you probably want to focus on outcomes, not necessarily the content. For AIIM, this took the format of saying, “Here is what this can help you solve—what this Webinar/conference can help you solve.” It’s not, “Hey, here’s our on-demand library, and you can access it 24/7” with no real connection to what those people care about in terms of the problems they’re grappling with on a day-to-day basis.

Jeff Cobb: [00:17:52] Yes, and it’s a nice connection because that outcome is the reason that you’re creating the product in the first place. It’s where you’re ultimately going to be showing impact, and that comes back into early in the process of convincing and converting people to participate in the product because they feel like they’re going to get that outcome. Now, we will give a caution here that Reach on its own or the metrics that go with Reach are really part of that classic vanity metric trap that we mentioned a little earlier—impressions and opens are easy to measure, easy to inflate, and easy to feel good about. Conversion is the real signal. Your validation, your testing, is already taking you towards that because you’re actually asking people to do it. But, obviously, when you get them to step up to the full product and convert and register or buy, that’s the real measure that you need here.

Celisa Steele: [00:18:47] We’ve talked about two lenses on metrics: Validate and then Reach & Convert. The third lens is Engage. This is where you’re tracking real-time signals of participation, and that’s then going to enable you to act while there’s still time to help someone who’s potentially struggling in an offering. This is about not just looking in the rearview mirror at what’s already happened; it’s about being a bit more predictive and forward‑facing or at least in the moment aware of what’s going on.

Jeff Cobb: [00:19:16] You want those real-time, behavioral signals that happen during participation, not just after the fact. You want those leading indicators, not just the lagging indicators that we often get in our evaluations and assessments and everything like that.

Celisa Steele: [00:19:30] Steven Khraiss mentioned in episode 485 the idea of an engagement score. They’ve seen learning businesses that put together something that rolls up an indication of engagement. It might be a composite that’s built from log-in frequency and how long that learner stay active during that session. How many modules did they make it through? Have they completed the practice exam or the test that’s associated with that content? Whether or not those would be components that would make sense for your learning business, the idea is to come up with some indicators that would help you predict who’s likely to finish and who’s potentially quietly slipping away. Because then you can help applaud those people who are making good progress, and you can also then hopefully reach out and provide some level of remediation for those folks who might be in danger of slipping away.

Jeff Cobb: [00:20:24] When you’re rolling up data into the dashboards that you might have in your learning management system or in any sort of business intelligence that your organization is generating, this is the kind of thing you want a real-time dashboard on, that you want to be looking at day in and day out. Again, contrast this with the traditional lagging indicators like completion, enrollment, and evaluation data—by the time you’ve got those and can look at them, it’s probably going to be too late to help a learner who’s struggling or to, in real time, be able to see that engagement issue happening and be able to address it

Celisa Steele: [00:21:00] The fourth lens is Retain & Reactivate. This is where you’re focused on getting those individuals to come back. You’re worried about staying sticky. You want those learners to come back to you when they have their next need.

Jeff Cobb: [00:21:17] But that needs to be according to the right cadence. As we mentioned earlier, we had the organization where people were attending every three years. We had one where it switched off, and this year might not be somebody’s year, but the next year would be. You have to understand that about your audience. You’ve got to go out and get that data so you know what retention and reactivation needs to look like, and you’re not expending resources on trying to retain and reactivate people who really don’t have an issue. It’s just part of their cadence that they’re not showing up this year, but they might show up again next year or the year after.

Celisa Steele: [00:21:51] I want to highlight one of the things that came up in the conversation I had with Steven Khraiss because it’s such an important piece of retention and reactivation: the sense to which learners feel identified with the provider of that learning. Steven talks about it, and Holmes Corporation talks about it, as “membership-integrated learning.” And the finding that they have seen in their experience working with various clients is that members outperform nonmembers on success rates even before controlling for engagement—membership and learning reinforce each other.

Jeff Cobb: [00:22:28] This harks back to research that ASAE (the American Society of Association Executives) did probably more than a decade ago that we’ve mentioned a number of times, and I still don’t think it gets the level of attention that it should. They did a Decision to Learn study that showed the number one factor determining whether somebody is going to participate in an organization’s learning is their sense of affiliation with the organization. If they are a member, and if they feel that strong member affiliation, they’re likely to participate in the organization’s learning. The flip side of that too is, if they participate in the organization’s learning, that’s the way they’re engaging with the organization and feel more affiliated with the organization. You start to have this virtuous cycle that develops. As with all of this, you want to be measuring that. You want to be looking at that. Is that happening? How are you making that happen? It should be a major focus for organizations.

Celisa Steele: [00:23:19] The fifth lens is Prove Impact. This is where you’re honestly assessing and determining whether the offering is making a real difference for stakeholders. In episode 474, Jeff, you and I talked about a four-perspective framework: Is the offering meaningful for the learner, employer, learning business, and broader field? You want to make sure that you’re thinking about all those different stakeholders. And then you also want to measure that at different points in time. Some impact you’re not going to see right at the end of a workshop or conference when you’re having them fill out that initial survey. You’re going to need to follow up later on to find out what stayed with them, what are they struggling with, what are they using. It might be three months out, six months out, twelve months out—whatever cadence makes sense based on your audience and the content. But you want to hopefully be able to see what is (or isn’t) being applied, what is (or isn’t) working in the workplace for those learners.

Jeff Cobb: [00:24:25] A major theme throughout this—and it’s the theme here again—is you want to plan for this from the beginning. You’re basically building in your impact plan from the initiation of the product. When you’ve got a new offering, impact obviously isn’t measurable yet. But you want to design for it. This goes back to Alaina’s “impact hypothesis” idea: Get clear on what difference you’re trying to make before you start measuring whether you made it. You should be building towards making that difference. That goes into the product design and deployment, and you build in the follow-up mechanism that’s going to prove it. You’ll measure it later, once enough time has passed, but you’ve got to have it built in from the beginning to be able to do that.

Celisa Steele: [00:25:11] We would encourage you to really try to validate the impact. What that means is, if you can help it, don’t rely solely on self-reported learner data. You want to also ask other stakeholders, especially the employer, about what changed and what is helping (or potentially not helping) once that learner is back and working on the job.

Jeff Cobb: [00:25:35] And, of course, get that employer input in the first place around what they want to see changed. Again, you’ve got the plan you’re designing to make that happen, and then you can go and check whether it did happen. The five lenses only work if you’re willing to go out and ask people, not just infer from the data. So let’s talk a little bit about going out and asking them.

Celisa Steele: [00:25:57] You want to get out in the field and talk to people directly. You want to talk to those learners, employers, other major stakeholders, whoever that might be for your audience or for that particular offering. This takes me back to the Elizabeth Engel and Jamie Notter episode, where they talk about design thinking and pairing it with some of the Lean Startup methodology and that build-measure-learn cycle. And one of the things they talk about in that episode, which is based on a white paper, is that empathy is so important. That’s a big part of why it is so valuable to talk to the learners, particularly out in the field, to see what is it that they’re dealing with, what are the barriers that they’re up against, what would help them learn and work in their environment. You can certainly make use of online surveys—they can be a good instrument for reaching many of your learners—but there’s also a whole lot to be said for going out and getting that close proximity in the field to a handful of your learners.

Jeff Cobb: [00:27:04] And this needs to be built in as a cultural thing for more learning businesses than it is. A huge part of the work we do at Tagoras is doing this asking for organizations. We will do surveys. We will do interviews. We will do focus groups. We will show up at events and talk with people close up. And that’s valuable. We enjoy doing that. We provide value to organizations in doing it. But, in most cases, we’re only going to be called in every two, three, four, five years to do something like that. Organizations need to be doing this on a regular basis and building it into how they approach learning. And it’s important when you’re doing this—the whole measure in build-measure-learn is a real call to action, not an opinion poll—to give people something and watch what they do to the extent that you’re able to do that. Don’t just ask about preferences or opinions. Really try to get at the behavior, not just what they say, but what they actually do or what they have done. And the only way you find that out is by engaging with them and asking and watching.

Celisa Steele: [00:28:06] Elizabeth shared that, when she worked with the Children’s Hospital Association, she spent the better part of a year going out and visiting every corporate partner in person to ask what they were trying to achieve and to see the barriers that they were up against and what it was like in their environments. That same kind of approach, that willingness to go out and have those conversations can apply to learning products and developing those as well. We have all sorts of organizations that collect survey data right after a Webinar finishes or someone finishes an online course. The important thing is then to make sure that you’re not just asking, but you are listening and taking action based on what you hear and changing things, stopping things, adding things.

Jeff Cobb: [00:28:54] And AIIM, you found out in the interview with them, they survey after every Webinar and event; more importantly, they act on what they hear—they’ve stopped doing things people said were too much. And, related to this, organizations collect a lot of this type of data, and it’s hard to act on it because there’s just so much data, and somebody’s got to be in charge of it, and you’ve got to figure out what the actions are from it. This is the type of area where AI can help so much now. Feeding that data that you’re getting from your Webinars and your events into AI and saying, “Based on our plan, based on what we’re trying to achieve”—going back to square A, you have to have a plan; you have to know where you’re trying to go—“what should we now do? What should we start? What should we stop? What should we change?” That’s the kind of thing that AI can dramatically expand an organization’s capacity around.

Celisa Steele: [00:29:41] We are arguing that, across these five lenses—for each of your products or at least your product lines—you’re going to want to have metrics that help you see how you’re doing when it comes to validation, when it comes to reaching and converting, when it comes to engagement, when it comes to retaining and reactivating, and then proving impact. We can’t tell you what those specific metrics are going to be for any of your learning products, but the main thing is, if you have that clear idea, as you were just saying, Jeff, of what this product is setting out to do, that’s going to tell you a lot about what metrics you need to be tracking to help you see, “Are we actually doing that?”

Jeff Cobb: [00:30:22] Let’s bring this all together and offer some practical ways that you can apply what we’ve been saying in your learning business. We propose that there are four questions that you might want to ask.

Celisa Steele: [00:30:37] First, does what we’re measuring match where this offering is in its lifecycle and what it was designed to do? Second, what did we expect for this offering, and why did we expect that? Third, do we care about this metric, or is it just the easiest thing to count? And, fourth and finally, have we recently asked the people this is for—learners, employers—directly about value and impact?

Jeff Cobb: [00:31:05] A few relatively straightforward questions. They’re not necessarily easy to answer, but, if you ask them and answer them honestly and objectively about a single offering, they’re going to tell you a lot about that product’s place in your portfolio.

Celisa Steele: [00:31:29] When metrics are paired with a real target, checked against the right baseline, and grounded in what learners and employers tell you, they stop being a scorecard and start being a genuine feedback loop, something meaningful and actionable.

Jeff Cobb: [00:31:45] Making that shift—from scorecard to feedback loop—is what we encourage learning businesses to do. Having those metrics in place and then making choices based on what they tell you is a key part of growing your reach, revenue, and impact.

Celisa Steele: [00:32:00] We’ve offered a handful of lenses—Validate, Reach & Convert, Engage, Retain & Reactivate, and Prove Impact—and we’ve offered some cautions that go with them. Pair metrics with a target, check your baseline against your audience’s real cadence, and don’t let an easy number stand in for the thing that you really care about.

Jeff Cobb: [00:32:22] And go ask people. Certainly learners but also employers. Enrollment numbers and dollars are only going to tell you so much.

Jeff Cobb: [00:32:37] And if you found this episode helpful, please share it with a colleague who might want some help in thinking through what to measure and why.

Celisa Steele: [00:32:44] Thanks again for listening—and see you next time on the Leading Learning Podcast.

To make sure you catch all future episodes, please subscribe on Apple Podcasts, Spotify, or wherever you listen to podcasts. Subscribing also gives us some data on the impact of the podcast.

We’d be grateful if you’d rate us on Apple Podcasts or wherever you listen. Reviews and ratings help the podcast show up when people search for content on leading a learning business.

Finally, follow us and share the word about Leading Learning. You can find us on LinkedIn.

The post Metrics That Matter: A Lifecycle Framework for Learning Businesses appeared first on Leading Learning.

Many learning businesses evaluate their programs with data that only tells them what already happened. By the time a lagging indicator like a completion rate reveals that a learner is struggling, the window to help has often closed. The good news is that shifting to real-time and predictive analytics is readily achievable for most learning businesses.

In this episode of the Leading Learning Podcast, co-host Celisa Steele talks with Steven Khraiss, who heads data strategy and analytics at Holmes Corporation, about the ideas behind the e-book The Association Leader’s Guide: How to Turn Learning Data into Personalized Member Experiences. Steven explains what an engagement score is and what it enables, why there’s no mention of AI in the e-book even though it’s powering a lot under the hood, and why he names culture, not technology, as the harder challenge in becoming data-driven. He also offers a practical first step for any learning business leader who wants to start but doesn’t know how: Start with one question you wish you could answer.

To tune in, listen below. To make sure you catch all future episodes, be sure to subscribe on Apple Podcasts, Spotify, or wherever you listen to podcasts.

Download a PDF transcript of this episode’s audio.

Celisa Steele: [00:00:03] If you want to grow the reach, revenue, and impact of your learning business, you’re in the right place. I’m Celisa Steele.

Jeff Cobb: [00:00:10] I’m Jeff Cobb, and this is the Leading Learning Podcast.

Jeff Cobb: [00:00:17] Too many learning businesses are trying to evaluate their success by looking in the rearview mirror. By the time the data shows a learner is struggling, it’s often too late to help.

Celisa Steele: [00:00:26] In this episode, number 485, I talk with Steven Khraiss, who heads data strategy and analytics at Holmes Corporation. Our conversation is structured around a Holmes e-book, The Association Leader’s Guide: How to Turn Learning Data into Personalized Member Experiences.

Jeff Cobb: [00:00:43] Steven makes the case that the shift from lagging indicators like enrollment counts and completion rates to real-time and predictive analytics is achievable for most associations. And the cost of waiting is rising faster than many leaders realize.

Celisa Steele: [00:00:57] Steven and I get into how to create an engagement score and what it enables, why Holmes deliberately left any mention of AI out of the e-book even though AI is powering so much under the hood, and why Steven names culture, not technology, as the harder challenge in becoming data-driven.

Jeff Cobb: [00:01:15] And Steven offers a practical first step for any learning business leader who sees the value but doesn’t know where to begin: Start with a question you wish you could answer.

Celisa Steele: [00:01:31] Steven, for listeners who may not be familiar with Holmes Corporation (HC), give us a quick sketch of what you all do, who you serve, and how you serve them.

Steven Khraiss: [00:01:42] Yes, absolutely. At Holmes Corporation, we partner with professional associations to design, build, and distribute knowledge and skills assessments—our flagship certification preparation. But we’ve also branched out into microcredentials as well as other learning programs globally. The goal is driving member engagement and revenue at scale. It’s turning those learning programs that actually move careers; that’s the gap we fill. We partner with some of the largest in the world: SHRM, ASCM, ICF, APIC, AFCPE, PayrollOrg. But the differentiator there is we span many different verticals—HR, supply chain, coaching, healthcare, finance—and everything from market analysis, content, marketing, sales, ongoing customer care. We’re not just a platform vendor. We really try to help associations meet their end goals. My role in this data and analytics layer sits underneath all that, and that’s the part that lets our partners actually see what’s working and what’s not. We keep evolving as our association partners’ needs do.

Celisa Steele: [00:03:04] Speaking of data, the impetus behind this conversation is an e-book that Holmes released called The Association Leader’s Guide: How to Turn Learning Data into Personalized Member Experiences. When I read that publication, it mentions three converging trends that are driving the importance of having good learning data now: the rising accountability expectations from stakeholders, the reskilling imperative that’s creating demand for learning programs, and then external and competitive pressure to keep pace. If you had to pick one of those, what do you see as the biggest driver of urgency right now for learning businesses? Or maybe it’s something other than those three at this point? I would love to hear what you think is driving urgency now.

Steven Khraiss: [00:03:55] Absolutely. All three are very real. But, if I have to pick the one that’s creating the most urgency—not only this year but over the past five years—it’s accountability. Association boards or enterprise partners, just association partners themselves, are asking a sharper question now of, “What did we actually get for this investment?” As budgets become tighter and every learning dollar, and in this role, having to balance a budget, it’s harder than it was years ago. But simply saying “Enrollment was up” is not the acceptable answer anymore. And the reskilling wave is real. We quote some stats in the e-book, but the World Economic Forum says 59 or 60 percent of workforce needs reskilling by 2030, and 95 percent of organizations today have ongoing and upskilling efforts underway, per Workday’s 2025 data.

Steven Khraiss: [00:04:59] In my mind, the demand is not theoretical anymore. And regarding competition, we frame it in the e-book well, but the gap between the real-time insights and a static quarterly PDF, or weekly, is the difference from knowing how the training is working and hoping it does. But they’re all pushing in the same direction. I’m living this firsthand when rebuilding our data foundation and infrastructure and having to connect different systems into a unified data lake house, so we could answer, “What did we actually get out of this investment?” and with evidence, not just “Well, we got some cool things.” When I say accountability is the driver, it’s what we built for.

Celisa Steele: [00:05:51] You were talking there about some of the things that, historically, organizations have relied on, things like enrollment counts and completion dates. But the e-book makes the point that those are lagging indicators, and so they give you limited visibility into is this program working and making a difference? What are the risks when learning businesses rely on those kinds of data, those lagging indicators, to tell them, “Is this working?”

Steven Khraiss: [00:06:22] Excellent question. The core problem is timing. Like I mentioned, the numbers tell you what already happened if you’re not working with real time, and so, by the time something looks bad, or you see learners’ progression fall off, you’ve usually already lost them at that point. While “Enrollment was up 10 percent” sounds great in a board deck or displays well on a report, it doesn’t really tell anything of whether anyone’s learning anything within a core set of products. In the e-book, we break it down into four costs. Delayed intervention—you spot a struggling learner that’s maybe too late to help. Limited accountability—it’s hard to prove ROI to leadership with just those surface-level metrics. Three, the resource drain—having to pull (I’ve lived this again) the manual reports and trying to chase learners one to one doesn’t scale. And four is those missed broader patterns of you can see what’s working, what’s not with learners, and what are they actually doing within products. Every one of those four problems is what that real-time or near real-time visibility and predictive scoring are essentially built to solve. And so relying on sports references, it’s like coaching a game off of the final score. It’s accurate, but it may be too late to change anything versus having to go back and review it.

Celisa Steele: [00:08:04] You began to talk there a little bit about those real-time but then, even beyond that, predictive analytics. What does it look like when a learning business shifts from that reactive lens to a predictive lens? And what does that mean? I’m thinking internally for staff at the learning business and also for those learners that learning businesses serve.

Steven Khraiss: [00:08:26] Reactive is reading the autopsy, and predictive is reading the vital signs. For it to work, the model needs history, some of those validated signals (the key performance indicators), and then the outputs that translate into the next step and not just a scary number. For staff, the shift is to viewing reporting in a proactive approach and doesn’t just replace it. For learners, it helps show, for example, if they’re unengaged. It’s going deeper than a “Hey, we miss you” message versus where they are in their journey. And we’ve seen this pay off in our own analysis and data. From a member and learner perspective, we analyze thousands and thousands of learner records across various dimensions, membership status. Typically, our products will have an initial pretest, so we measure engagement there. And then, if it’s a cert prep product, a practice exam—we measure engagement with those. And then overall engagement with the product. We’ve seen engagement in the practice exam behavior as one of the single biggest levers. We can design nurture programs that say, “Hey, we noticed that, Celisa, you plan to take your test in two weeks. It’s very vital that, if you haven’t yet, please take your practice exam. It’ll give you that peace of mind and framework that really makes a difference.”

Celisa Steele: [00:10:07] One of the things that the publication talks about is this idea of an engagement score. For those individual learners, you have this idea of how engaged are they? That’s, I think, based primarily on activity in the platform, logging in, and have they completed certain modules, those sorts of things. Talk a little bit more about the engagement score—how it works but, even more importantly, what having that engagement score then enables for an organization.

Steven Khraiss: [00:10:34] At the simplest level, the engagement score can answer one question extremely fast: In your ecosystem of learners, who’s thriving, and who’s quietly slipping away? It’s built off of those behavioral signals, like you mentioned: platform activity, login frequency, how long per session are they engaged? Is it 10, 20 minutes versus two and a half hours? How are they moving through the various modules and, with that, their assessment activity—whether it’s taking quizzes, or if we have flashcards built into the product, or downloading various items? When you consolidate all that and put it together, that engagement score is reflecting the likelihood of finishing the program successfully, whether that’s ultimately taking a certification exam or earning a microcredential. What it really unlocks at scale is, for the exam, without having to dig into individual learner trends, it helps you see the overall macro picture. But I always have one honest caveat there: The score is only as good as the behaviors you feed it. Keep it tied to real learning, not vanity clicks. The mental model I use is to think of the credit score for engagement, one number, many different behaviors, and it moves and shifts over time.

Celisa Steele: [00:12:02] Talk about what the engagement score is. Is it a scale? Is it a number between 0 and 10? Are you categorizing learners into specific tiers? Talk a little bit about that.

Steven Khraiss: [00:12:16] Absolutely. We’ll look at the wider dataset based on trends. Whether engagement with nurture programs, whether it’s an e-mail campaign, or an SMS-triggered campaign, we’ll assign various scores. And they’re mostly weighted. If we know, for instance, a practice exam is a high driver, then we might give a learner a score of 20 points. You could set your scale to whatever is relevant to the business, but then you create these tiers of low engagement, middle, high, super high, or super user. Time and time again, we’ll run these analyses, and we always say around the office, “If you use the learning system or our products as they should be intended, you will have a higher likelihood of success.”

Celisa Steele: [00:13:11] In essence, that engagement score then gives the provider of that content a shorthand way to see who is most engaged—perhaps offer encouragement to those who aren’t as engaged, maybe offer kudos to those who are very engaged, and so it’s about providing the appropriate feedback and incentives along the way to keep the learners going.

Steven Khraiss: [00:13:35] Yes, you just hit something right on the head there. We’ll have tap-on-the-shoulder nurture campaigns where a learner may be interactive and engaged early on, say, for a certain time period, and then they fall off. Life happens. Or you get down this runway of  “Yes, I’m going to really engage here.” And so we have triggered campaigns that say, “Hey, Celisa, you’ve been absent for a couple of weeks. Let’s get back and focus and do x, y, and z.” Likewise, if a learner is really engaged, we have campaigns that praise them along the way. It’s that full end-to-end on a journey. It’s almost like a coach that’s cheering you on throughout a game or throughout your sports journey.

Celisa Steele: [00:14:25] I see you’re back to sports again there, Steven.

Steven Khraiss: [00:14:26] Yes.

Celisa Steele: [00:14:28] In the publication, there’s no mention of AI, which is a bit surprising, given that this is a piece focused on learning data, on personalization. I want to give you a chance now, Steven, to go on record and talk about the role of AI in making effective use of learning data.

Steven Khraiss: [00:14:49] I’m really glad you’re asking this one. Honestly, it was a strategic decision. We left off the word AI throughout the piece on purpose because the fact is it’s already in there and not just as the buzzword of AI. Because, in my mind, most association leaders don’t really want AI for its own sake. They want the outcomes that it delivers. When you look under the hood of the e-book, it’s all there. When we talk about predictive risk scoring, the engagement score, behavioral automation that fires off the personalized nudges based on how the learner is interacting, that is essentially the machine learning doing the useful work quietly in the background. And through those automation processes and workflows, we’ve seen real results. We’re cutting our partners’ workload by, in one instance, over 45 percent, and we’ve seen more than three times the industry average in terms of open rates and click rates with our e-mail campaigns.

Steven Khraiss: [00:15:53] We chose to lead with the result instead of the buzzword. The flip side is the guardrail. AI is obviously an accelerant but not always the foundation. If the data is fragmented and ungoverned, then AI will scale that mess faster. The common analogy is garbage in, garbage out, but AI at machine warp speed. Even though I’m a huge tech guy, I feel like the technology should disappear behind the result. In my mind, if a member feels seen, it doesn’t matter if it’s AI or a person—that’s generally the whole premise. AI can be the accelerant but not necessarily the foundation. And the associations, at least in our experience, that win won’t have the flashiest AI, but they have the clean data to trust.

Celisa Steele: [00:16:51] That’s a really important point about the data that goes into any model, especially an AI model. It needs to be in good shape if you’re going to expect it to be able to help you generate meaningful results. One of the other things that comes up in the e-book is benchmarking as a way to use some of the learning data. That’s interesting because benchmarking can be really powerful and can be useful. But it also seems like sometimes benchmarking can create a bit of uneven comparison, like an apples-to-oranges problem. What does it take to benchmark meaningfully, and what makes those benchmarking comparisons valuable and useful rather than misleading—being compared to not a similar organization or a similar situation or a similar learning product?

Steven Khraiss: [00:17:46] Benchmarking is one of those things. It’s incredibly powerful when it’s done right. And, on the flip, it can be quite dangerous when it’s done wrong because, like you alluded to, a bad comparison can give you false confidence in a dataset that you’re looking at. Performance data is definitely valuable, but performance data with context is even more powerful. I would say make it real. First is the breadth of context—the comparison has to be grounded in data that spans, whether it’s learner groups or industries. If it’s too narrow, it can mislead. Second, if you’re comparing similar courses or benchmarking for us different organizations or cohorts, that makes the peer comparison done right. The key with all of that is the consistent definitions part. Completion has to mean the same thing on both ends of what you’re comparing, and historical trends can also show you the real improvement over time. Often, even in our leadership meetings, it’s like, “Okay, well, x percent is really great, but what does that really mean in context?” The differentiator for us is understanding how adult learners and customers engage and not just the technology that tracks them. Benchmarking is truly useful when you’re comparing the like to like. Otherwise, you can generate very confident but also wrong conclusions.

Celisa Steele: [00:19:28] That’s an interesting thing that Holmes Corporation—because you work with a variety of different organizations—does have that ability to draw on data from different organizations and do some of that benchmarking of program to program, organization to organization. Do you also find at times that it’s useful to benchmark within a singular association, to look at their own program A versus program B versus looking at association A and association B?

Steven Khraiss: [00:19:56] Yes, that’s exactly it. The nature of our association partners in different verticals allows that high-level view. But, to exactly your question, drilling down within a particular industry, within a particular partner across products, it is interesting. You’ll see different trends where, even if the products are designed similarly or with similar functions based on the content, they interact completely different. One of the biggest values for us is being able to do those comparisons and looking at equal time frames and those other environmental factors as well.

Celisa Steele: [00:20:46] One of the other things that I found very interesting in the e-book is this finding around the connection between membership status and learning. Could you talk a little bit about this value of “membership-integrated learning”? I think that’s how the e-book put it.

Steven Khraiss: [00:21:00] Yes. In our experience, here’s something associations have always believed but couldn’t always prove: When we look across partner programs in our own analysis, time and time again, members outperform non-members on success rates even before you factor in product engagement. We know association partners that we work with have so much knowledge and industry authority that it becomes this virtuous cycle where membership can unlock the learning, the learning drives engagement, and then engagement ultimately drives renewal and credential revenue. When you finally get that one and unified view of the member, the whole journey—instead of having learning sit in one silo, membership in another—that single consolidated view is where the personalization starts. It means you can recommend the next relevant product or credential and not truly blast the whole catalog. Learning is one of the stickiest reasons a member renews. You weave it in. And every course can make someone engaged, more credentialed, and ultimately more likely to stay.

Celisa Steele: [00:22:19] Like you said, it’s often talked about—that relationship between membership and learning and how they are this nice, virtuous cycle if you get them going. As you said, to have some of that data, to be able to look at that and then prove that that’s more than a hunch or the way that it might work, that it actually is the way that it’s working, that those people who are members do seem to engage more or do better in the learning. One of the other things that the e-book talks about is what it takes for an organization to be ready to shift to this data-driven approach to learning, and there are at least two parts to that. There’s at least the technology side of it, and then there’s at least a cultural part to it as well. In your experience, is it the technology issues or the cultural issues that make it harder to shift to a data-driven approach to learning?

Steven Khraiss: [00:23:15] I love this question. As I mentioned, I’m a tech guy, so people that know me would expect me to say “technology,” but I feel that it’s really not. It’s culture every time—the e-book talks explicitly about this. And technology alone isn’t enough. We outlined some of the readiness pillars near and dear to my heart. Data literacy—can your stakeholders read what’s on these dashboards that you’re creating and understand it? Process adaptation—do the workflows change around insights? I talked earlier about actionable insights from the data. Organization-wide or cultural commitment—do you act on what you see? The technology is absolutely necessary, but those three are what separate different organizations that truly win.

Steven Khraiss: [00:24:13] When you have a dashboard that contradicts a senior leader’s gut intuition, the room might just default to that, but I would say that’s the cultural gap, not a technology gap. And so my favorite line is “It takes leadership to model that behavior.” Once teams really trust the data, then there’s this shared willingness to let the numbers challenge all the everyday assumptions. And then it creates this culture that makes it safe to say, “Oh, yes, I was wrong.” You can buy a platform, but you can’t buy willingness to trust in the numbers when it tells you something that you didn’t necessarily want to hear.

Celisa Steele: [00:24:56] For a learning business leader who is going, “Okay, I get it. I understand the value of becoming more data-driven, data-informed,” but maybe she’s not sure where to begin, what would you recommend as a first step to get a little bit closer to being more data-informed/data-driven about decisions?

Steven Khraiss: [00:25:18] I get asked that question a lot in my journey. The most common mistake is thinking you need, out of the gate, a giant platform and this brilliantly designed master plan. You really don’t, and I’ll get to why I think this way. Don’t start with the platform, but start with a question you wish you could answer. The momentum, in my experience, ultimately builds that master plan. You have the framework and a roadmap for sure, so pick something that is concrete and that you wish you could answer. Which of our learners or which segments are most likely not to finish, and why? And then you chase the data to answer that. You don’t have to have everything unified on day one to start out. The e-book provides a little checklist of starter questions. Can you get to real-time data without manual report pulls, or can you predict risk with enough ramp and lead time? Can you automate support so you’re freeing up organizational-wide time? Do you have that peer benchmarking for context? Any one of those is a fine front door. The reason I push more for that one question is momentum, and each small win builds that larger belief and, quite frankly, the budget for a larger investment in the future.

Celisa Steele: [00:26:51] That answer reminds me of AI not being apparent in the e-book. It’s there in the background. Having a question to start with echoes that it’s about the purpose—what are you trying to do—rather than focusing on the tools or the specific methodologies.

Steven Khraiss: [00:27:07] Yes, that’s a great point. Just like AI, it starts with a prompt. You might know where you’re trying to go, but you build momentum with, “Oh, I didn’t think of it in this way.” So it leads it with that excitement and that full end-to-end journey.

Celisa Steele: [00:27:25] Because this is the Leading Learning Podcast, we always like to ask guests who come on about their own approach to learning. Steven, what are some of your specific habits, practices, or sources as you continue to grow and learn professionally and personally?

Steven Khraiss: [00:27:42] One of my favorite learning experiences, most recently, has been coding with my son. He’s big into Roblox. I know it sounds silly, but wrangling that with him honestly sharpens how I think about logic and systems in ways that my day job doesn’t, even though I am a tech geek. But that’s my whole approach. I learn best hands-on, building things. And projects honestly teach me a lot more than maybe a certain course would, just with that hands-on approach. I deliberately cross-pollinate—you’ve heard that I’m a sports guy—so I pull a ton. I used to coach my daughter’s basketball team, so I like to think of data in that context, as with teams. I try to pull from a lot of different perspectives before locking into my first instinct. And I always try to maintain a thirst for additional knowledge. I listen to podcasts, attend Webinars, conferences—probably too many. I’m on a lot of newsletter lists, and I always try to stay up to date on the latest trends and even test drive new products that interest me. If I had to sum it up, my best learning habit is staying a little uncomfortable because, if I’m in a room, and I understand everything, then I’d be in the wrong room.

Celisa Steele: [00:29:18] That wraps up my conversation with Steven Khraiss of Holmes Corporation. Stick around for our recap.

Jeff Cobb: [00:29:24] Check out the e-book from Holmes Corporation and Steven on LinkedIn. And please help us spread the word about the Leading Learning Podcast.

Celisa Steele: [00:29:39] Steven believes learning businesses need to stop evaluating programs based on data that only tells them what already happened. By the time a lagging indicator like a completion rate reveals a problem, the window to help learners has often closed.

Jeff Cobb: [00:29:55] Holmes made a deliberate choice to leave mention of AI out of the e-book entirely, not because AI isn’t relevant but because, as Steven puts it, AI can be the accelerant, but it can’t be the foundation.

Celisa Steele: [00:30:08] His practical first step for any learning business leader who wants to become more data-informed but doesn’t know how to start is to begin with one question you wish you could answer, and build from there. I love that recommendation—because he’s saying, essentially, start with curiosity.

Jeff Cobb: [00:30:25] Thanks for listening—and see you next time on the Leading Learning Podcast.

To make sure you catch all future episodes, please subscribe on Apple Podcasts, Spotify, or wherever you listen to podcasts. Subscribing also gives us some data on the impact of the podcast.

We’d be grateful if you’d rate us on Apple Podcasts or wherever you listen. Reviews and ratings help the podcast show up when people search for content on leading a learning business.

Finally, follow us and share the word about Leading Learning. You can find us on LinkedIn.

The post Turning Learning Data into Personalized Experiences with Steven Khraiss appeared first on Leading Learning.

The association sector is at a genuine inflection point because so much is changing at once. AI is reshaping entire professions. Federal funding is in flux. University pipelines are shrinking. And, in the middle of all of it, associations have an extraordinary opportunity to play a critical role in workforce development—if they’re willing to move fast enough.

In this episode of the Leading Learning Podcast, co-host Jeff Cobb talks with Artesha Moore, CEO of Association Forum, about what that opportunity looks like and what it will take to seize it. Artesha shares why mission-led thinking matters more than member surveys at this moment, how associations can close the visibility gap with employers, and why the leaders most likely to succeed will be the ones willing to imagine broadly and fail productively.

To tune in, listen below. To make sure you catch all future episodes, be sure to subscribe on Apple Podcasts, Spotify, or wherever you listen to podcasts.

Download a PDF transcript of this episode’s audio.

Celisa Steele: [00:00:03] If you want to grow the reach, revenue, and impact of your learning business, you’re in the right place. I’m Celisa Steele.

Jeff Cobb: [00:00:10] I’m Jeff Cobb, and this is the Leading Learning Podcast.

Jeff Cobb: [00:00:17] Associations are among the most powerful forces in workforce development and professional advancement in the U.S. But almost no one outside the association world grasps the scope of that impact—and sometimes not even those in the association world.

Celisa Steele: [00:00:31] Our guest for this episode, number 484, is Artesha Moore, CEO of Association Forum. Artesha has spent nearly three decades in association management and the nonprofit space.

Jeff Cobb: [00:00:43] Artesha makes the case that we’re at an inflection point that’s different from the disruptions associations have faced before because it’s not one force at a time. AI, workforce displacement, shifting federal funding, and declining university pipelines are all hitting simultaneously.

Celisa Steele: [00:01:00] You and Artesha also discussed the gap between what associations do and how they’re perceived—why a university president gets quoted in mainstream media on workforce development but association CEOs rarely do and what it would take to change that.

Jeff Cobb: [00:01:15] And Artesha offers direct, practical advice for association leaders: how to use imagination and a willingness to fail as strategic tools.

Celisa Steele: [00:01:24] The conversation kicks off with Artesha introducing herself.

Artesha Moore: [00:01:33] I’m Artesha Moore. I am the CEO of Association Forum. But, if I could introduce myself in any words, it would be I am a beautiful hip-hop Buddhist Jedi. And what does that mean? I look at the world from a beautiful rhythmic realness. I’ve been in the association management and nonprofit space for about three decades at this point; and, as I age, I get more centered and present in the Buddhist sense. The Jedi part is because I’m a sci‑fi girlie—a technologist—and thinking about how technology, sci-fi, and science fiction have become the reality of what we’re living right now. So, for a person who has helped, in my career, move people forward through learning, I’m excited to be here today and to hash it out with you, Jeff, because, when we met, it was electric.

Jeff Cobb: [00:02:30] Indeed it was. You’re definitely the first hip-hop Buddhist Jedi that we have had on the podcast. So I’m looking forward to that conversation. You and I have spoken before. We’ve been in a very engaging discussion before, which I’ll give props to Mark Jones at LearnUpon for pulling us together on. We were talking about where associations are in general, but also where they are with education since that tends to be such an important part of what associations do. And one of your comments was that we’re really at an inflection point right now, that we’ve seen a fundamental shift, and that this moment is maybe a little bit different from the changes that we’ve experienced before. Associations obviously have experienced changes before, but what’s different now from your standpoint?

Artesha Moore: [00:03:20] There are a lot of things that are coming in that are different. It isn’t just new advanced technology; it’s a new way of looking at industry, I think. When you look back at matriculation rates from 2017 and 2018, the new folks going into university, lower numbers. The new folks coming out of university aren’t prepared for the workforce in the same way, traditionally, that they were about 10 years ago. We look at that and then the changes of funding mechanisms. I’ve spent most of my career in STEM. The number one funder of STEM in the United States is the government. As the government is trying to settle its dust-up, what does that look like for the impact of all of these industries? When I say an inflection point that we haven’t seen before, I think it’s just the acceleration of all of these factors at one time. We’ve seen them all happen, but all of them coming at one time under this umbrella of collective uncertainty. You’re hearing a lot of headlines: “Careers Are Going to Go Away,” “30-Percent Workforce Replaced in 18…” That uncertainty also adds a new element, I think, that makes it really hard to navigate strategy.

Jeff Cobb: [00:04:50] From your viewpoint, so many things are converging right now. Associations do have a role to play in this, obviously. There’s an incredible opportunity for them to play a role in this right now. But, in my experience, associations are very focused on the individual member. What members are asking for, that individual member engagement—that’s a holy grail for everybody. But, when we’re seeing such wide change and shift happen, do associations have a broader responsibility than that? What’s the responsibility to the profession, to the industry, or to society in what we’re seeing right now?

Artesha Moore: [00:05:31] I sit on a little bit of a different side of this. The member value is important, and, as a membership organization, I definitely would lean into that. But I’m a mission‑based person. What is the mission, and what was that intent of the mission to do? Many associations, not just the ones I’ve worked for, their mission does not say “Maintain member value.” Their mission is big. My mission at Association Forum, our mission is to advance the practice of association management, where the practice goes. Also, the opportunity and the need for associations around their mission is future-forward—it’s about the future. You do need to assess the members, but I think about it in that kind of old Henry Ford quote—are we getting a faster horse? They’re going to tell you about their pain today: “I need the Web site to work,” or “I need…,” whatever those issues are. But associations traditionally represent the top tier of a profession. There’s a whole middle tier that isn’t represented there, the profession. So, when you start to have these futuristic or, dare I say, present conversations, it’s important for us to read the tea leaves of the future and push the profession towards advancing, like you said, societal things, being able to be an incubator for non-traditional thought that will change the world. That’s the opportunity that associations have.

Jeff Cobb: [00:07:26] Does that practically connect in your mind? Does that mean more connection to employers? You mentioned workforce and workforce development earlier. What role should associations be playing there from your perspective?

Artesha Moore: [00:07:43] It’s important to connect with employers because, if we’re a professional society advancing the practice, most association dues are paid by the employer or the funder. Again, when I mentioned the federal government, the funder is paying for the dues, the scientific review of the publications, all of those things. I mentioned before about the pain points of members. Some of the pain points of members—let’s be frankly honest—is the employer. However, I recently heard a podcast interview with Scott Galloway, and one of the things that he said that was very interesting was the amount of GDP in the United States that’s spent on reinvestment in skill development. We spend less than half a percent, whereas other Western nations spend one to two percent. When you think about that number and what’s needed around that in this continuous reimagining of work, associations fit right in there. But, if you’re not understanding what the employer is actually looking for.… For instance, we talk about professional development, networking because it’s a people-connection thing. “Come in and get professional development. Come in and network.” To what end?

Artesha Moore: [00:09:13] There was an article in Harvard Business Review in 2018, and the headline was about employees. And it said employees care about three things: community, career, and cause. I stole that and said, well, members care about that too—community. But, for what end? Career and cause. So your mission is the cause. I was with a healthcare group in the 2010s for 13 years, and it was about infection prevention, and our underlying mission was a world without infections—big cause. Get down with that. Yes, I’m a technologist, and I’m doing that, but…big cause. The community that we created, we’re able to do things, like help stamp out the Ebola outbreak that came in 2008. As we look at this career element, how do we translate what we do in advancement of careers? Some parts are there, the leadership. Yes, you can speak and present and those things. But, as we are in this inflection point, our opportunity is to partner with employers to ramp up the skills development so that the next generation of workforce can solve the next generation of problems.

Jeff Cobb: [00:10:42] I’m interested in how well you feel associations are doing what you just described right now, and, just as importantly, how well they’re perceived to be doing or how their role is perceived in this shift that we’re talking about: helping to support professions, helping to support workforce, helping to take us into the future. Part of why I raise this—and I’ve raised this with a number of association CEOs—is you hear a lot about lifelong learning now. You hear a lot about workforce development, the need to be flexible and agile, and what’s going to take us into the future. And when I read The New York Times, The Economist,and who they’re going people out there in the world to talk about this. They’ll quote a university president. They’ll go to somebody in the corporate world. I don’t see association CEOs quoted much in that context. And I’m not really sure that the role that associations can play and, to a large extent, do play in helping, leading us forward is actually recognized. Do you perceive that gap at all?

Artesha Moore: [00:11:46] Absolutely. It’s like ding, ding, ding, ding, ding—you win. That’s it. And you said it beautifully because it’s two different questions. How well are associations doing? Wonderful. We are an industry that’s almost two centuries old. We’re doing it beautifully, even today. We’re leaning in and able to lean into communities that industry and university can’t tap into because of our nonprofit umbrella of support, workforce development, and all those things. There are so many groups out there that are doing groundbreaking levels of engagement: ways to roll up, new ways of connection and learning that are changing skills, and being able to have that impact. Does anybody know about it? Nope. We are the most unknown powerful group of individuals in the workforce right now. We have a collective knowledge of strategic learning, engagement, and advancement in workforce, as well as the thing that we do very well as an industry is advocacy. Insert any industry. Umbrella, all industries. But my mom still doesn’t know. I just told you my whole hip-hop Buddhist Jedi thing. My mom’s like, “So what do you do again?” “I change the world, Mom.” But we could use some of the advertising art to say the value, not just what the association does but how it has and continues to empower the workforce—if they know about us.

Jeff Cobb: [00:13:36] It does seem like part of this is just a baseline communications issue. The word needs to be gotten out more. We were talking about employers in connection to the employment market before. I feel like a lot of times the employers in a profession or industry don’t necessarily see the role that the association is playing or could play in a practical/tactical example of that. I won’t say puts the blame on, but it points to the associations themselves. Organizations will put out all these credentials, certificates, and all sorts of things that they’re awarding to the people who take their education, but the employers maybe don’t even know about it and maybe don’t care about it if they do know about it. There is not that tight connection that needs to be there between “We’re creating this value” and “Here’s how it translates into what you are doing as a business in this field, industry, or sector that you’re in.” What’s your take on that?

Artesha Moore: [00:14:33] It’s a blurred line because, even within our industry, there are fractures. There are the individual professional societies, and then there’s the trades, and the trades are where the corporations are. But, when you go and look, they don’t participate in the same circles as the professional society CEOs—very little overlap. That direct translation is happening at the trade. The corporations understand the value of a trade association, period. Point blank. They understand it. They lean into it. They see value. They get things like Chatham House Rule conversations to get ahead of various trends. They get to use their collective voice to say, “This is the certificate that we’re going to honor at the employer level and push forward with that.” What happens is that the professional society tends to be the purist, the scientist, the research. And I’m not saying scientific. Any. SHRM is more of a corporation. But there’s a healthcare association that’s very focusedly inward. They’re doing the society representation of the people, empowering the people. There’s not always a direct connection when you leave that to say, “Give this to your boss,” to tell them what you learned. I’ve been in rooms where they’ve talked about the need for a credential. It is almost insular. “We think that this is important, so we will do it.” Without bringing in the boots on the ground. Without bringing in the people who are doing it.

Artesha Moore: [00:16:17] I started my career as a secretary. I was not even a secretary; I receptionist, thank you very much, and I was an administrative assistant. As you look at those people, I was the one who knew probably more about what was going on in our nonprofit, but nobody courted me to be in the membership because my title was receptionist. And leadership development means that you have to have this, this, this, and this. We’re missing out on that opportunity. And then my employer didn’t know about it because we were in this other ilk that wasn’t the purist. There’s an opportunity, and we need to get it together right now, fast, because universities are also in an inflection point, employers. When I was with the American Geophysical Union, I sat on a group of workforce development that was talking about this bridge between academia, university, industry. What industry was saying was these folks come out great scientists, but they don’t have the professional skills—how to do a budget, how to manage a team, how to do such and such. Well, hey, that’s where we are. Associations help with that kind of skill development. Universities are like, “Hey, we could do that too. We can have professional development right from the university where you graduated from and cut out the middleman.” Us. So I think that there’s an opportunity here, but it’s been a division for decades. And there are new players. The tech bros. We haven’t even talked about them, how they’re rolling out learning—free, good, usable, recognized by employers because of their name. Not university, not association.

Jeff Cobb: [00:18:13] Things are shifting fast. Associations have done, are doing great things. But we’ve got to evolve. There needs to be some innovation, particularly when you think about how people are learning now, what that looks like. But there is a lot in associations that can slow things down, be a barrier to innovation. You think about how boards and committees and subject matter experts have traditionally worked. We’re still very much embedded in the “Let’s have an annual conference; let’s have breakout sessions” type of model. What’s it going to take to change that for associations to be able to evolve rapidly into this future that we’re talking about?

Artesha Moore: [00:18:55] That’s a lot of big unpacking there.

Jeff Cobb: [00:19:00] And, if you can answer the question, you’ll be able to make a million dollars.

Artesha Moore: [00:19:02] I’m trying to tell you. As I think about that, let me attack it from some different angles. Number one, governance is something that we absolutely need to modernize. Volunteer management, governance at the big, broad level are still at the root of it, not the skills in order to lead and drive change. I put my hand up. “I know everything about the association because I’ve served at all these levels; let me go to the board.” As many associations are getting more sophisticated about leadership competencies to lead. Not just “I know Jeff. Jeff’s my boy. Let me go ahead and get him on the committee so we can do such and such.” Competency development. And people lean too much on the pandemic around some of the pieces, not around innovation, but we needed a desperate energy to survive. Meetings were grounded. Various things. But what we did see was an innovative spirit that came forward. What is hard is the monetization of it. Professional societies are not rich. There are some rich ones. Shout out to y’all. I see y’all. There are some rich ones that have war chests, endowments, investments, property, land.

Artesha Moore: [00:20:29] The majority of us are small—less than $5 million, less than 10 staff. When you start to look at the need to innovate, you have to have a board that will say, “Let’s move, let’s have a direction, let’s be willing to fail, and let’s put a little bit of our money into the future.” That’s almost a luxury when you have to pay the rent, upskill your own staff, and when you’re competing with a startup that could go to market in eight months as a membership that is also competing with you because they don’t have overhead; they don’t have a board. Right now, there’s a lot of opportunity because of, frankly, the dollars. The money is different. So, as the economy goes up and down and up and down, the reality for us that do live events is it costs big bank. Coffee. Whoo! I love coffee. I live on coffee. One of my first jobs was at Starbucks. It costs—and convention centers understand that. Getting your materials there costs. Getting your people there costs.

Artesha Moore: [00:21:52] Are there opportunities to have a different way to do it? Yes. But here’s what I know (Buddhist part): You and I shared an energy together on a stage in real life with people. Felt different. I love this. This is fantastic because we can share our ideas. But I said more yes because I just wanted to keep our energy going. I love technology, and everybody get at me in the comments to talk about the sci-fi stuff because this summer is going to be fantastic. It’s for us. We’re going to the movies. But it doesn’t have the same cachet. I’m still learning in this. But now I get to feel that you’re a real person too and share that energy. It’s a luxury. Not everybody has the opportunity to do that, especially when you’re trying to find something new. So we have to. We have to. But how we go about doing that, what we go about doing with it, what that looks like—it’s going to be a lot of failure first. And we’re not as good as an industry with that because our boards are not comfortable with that.

Jeff Cobb: [00:23:05] I want to come back to academia, universities, corporates. What role do you feel partnerships could potentially play in this? You said this is an inflection point for universities, but they’ve also got a lot of issues too, and the dollar factors in there. They have the same in-person, online scenarios. They’re wrestling with AI. And even the tech bros that are putting stuff out there, they’re wrestling with these issues too. What role could partnerships play in your mind?

Artesha Moore:[00:23:41] Huge role. I don’t see it as othered. They’re just coming at it from a different angle but the same passion around the community, pressure around their product, passion around the future. So don’t other them. Come on in. Let’s get all together. Let’s create an opportunity that really redefines mission. Are we clear about what we’re trying to do? Who can help us do that? In 2016, I was on a panel, and I said, “What is it going to be the first association that allows an autonomously-run company to be a member?” Again, sci-fi. That was in 2016. Here we are. We have, absolutely, an agentic workforce coming forward, but they can’t be in the room because there are still associations that are worried about, and rightly so, intellectual property and how to hold on to that for today instead of saying, “Maybe I will unleash this all.” Project Hail Mary. “Maybe I’ll release all the knowledge that we have to do our mission in a different way and bring partners in, bring inventors in, bring legislators in, everybody in who cares about the mission of whatever that is and then build something for the future in an incubator that may implode.” Great, we learned some things, but we now have connection.

Artesha Moore: [00:25:17] I’ve seen, in my history and experience, associations partnering directly with university to create curriculum. Boom. Works. We could look at the arc of this and have a future where there’s a Harvard and Association Forum—yes, Harvard, come forward—partnership that does spaceships for the future. Who knows? But now we’re in this partnership, and I see it. We partner with the Center for Creative Leadership on an emerging leaders program. They are an outstanding global leadership development. I could have went at it alone. We had a program that was our members doing it. But why wouldn’t I go after a global brand and give my people access to tens of thousands of dollars’ worth of global leadership development training for a quarter of the price as a nonprofit? Can we bring in a university? We had partnerships with universities. So it isn’t just how do we bring them into the profession? How do we do this? It’s how do we solve real problems? How do we solve real things? And how do we learn from each other, not just from corporate? As I look at university systems—I like history—in the United States, they’ve also been places where alternative thought led to global change. I think about the civil rights movement, the students, and the universities that empowered them, that funded them, and changed the world. So every insert we could do that. And I think that there’s an opportunity.

Artesha Moore: [00:27:01] I just read an article in MIT Technology Review, and I was arguing with the article on the plane, like, “What in the world? Why?” It was about mirror sequencing of microbes. “Y’all are about to create The Terminator. I don’t know what y’all are doing.” But what I wrote in the margin that inspired me was the article said that part of the reason why they were going down this doomsday path is because they didn’t have interdisciplinary conversations. Boom. Insert an association. That’s what we need so that we don’t kill ourselves because we’re smart, and we can use technology to do something. I’m sorry, if you don’t get me and Jeff and somebody else in this room to help y’all, don’t do that. But I see that those are the defined new futures. We can work together. We’re all trying to—I hope—do good. How do we do good today and set us up for tomorrow? It isn’t a member survey and committee-driven thing that makes it perfect, and we’re going to take over the world. It’s not that. It just isn’t.

Jeff Cobb: [00:28:23] It seems, to me, this is the big nut to crack or the elephant in the room, however you want to phrase it. All of these different constituents or potential partners—universities, corporate world—have an issue to solve. This traces back through employers, and it goes back to where we started around the role of associations with professions and industries and their role in society. There is about to be—and it’s already starting—a huge disruption to work, to career out there because of what AI is doing and will continue to do. You talked earlier about an autonomous company. That means there are a lot of people not employed as a result of that in a field or industry. It means something is disrupting that field or industry. People are being displaced. People are having to evolve. How do you see the role of associations in that disruption that AI is bringing and will bring? How should they be helping their people get ahead of that? How should they be helping employers navigate that? That’s another big, hairy question, but I’d love to hear your thoughts.

Artesha Moore: [00:29:30] No, I love it. In 2024, we did a skills study for Association Forum, and it was to start to attack what you just brought up. There was this wave, and that was just when OpenAI was rolling into the mainstream in 2023, this kind of conversation. What do our profession folks need in order to do their work today and in the near future? Here are these skills. Upskilling and reskilling are imperative. It’s not now a theoretical “Should we do it?” My parents are a beautiful product of the ’70s vocational system. My dad’s a nurse. My mom was a stenographer but office manager. They got skills. But that kind of work led my dad into the military and overseas in Europe, where I went to high school in Germany. Vocational is still today an arm of where people can displace from work, go into a vocational trade, get reskilled, go into work. That’s where I think associations can play. So I’ve been using AI. I have one model that’s running about nine months or so in the future, 100 years from now, but then I start to think about, “Okay, well, if I need to get a new job today, I’ve been doing this for 30 years; I have some kind of skill set. If there are no associations, where do I work?” That helped me understand what my gaps are in skills. And then, of course, I’m a techie, so I built a course for myself and taught myself some new skills.

Artesha Moore: [00:31:10] I’ve been telling young people in association management what we’re not saying out loud is our industry is constricting. In the ’90s and early 2000s, there was this explosion in healthcare-type associations. I have on purple lipstick today, so I’m going to be the Nurse Society of Purple Lipstick. We only get in purple lipstick, and that’s it. We got an annual meeting. We’re giving out partnerships of this. We did this. Great. Then the economic recession in 2008: Okay, well, we’ll let in blue and red because it makes purple; we’re still going to be going. Insert pandemic. Don’t nobody even wear lipstick no more, girl. We need to close. We’re going to be a part of the American Nursing Association now, and we’ll be a chapter. Great. Boom. As we start to look at this constriction, that funding thing that I mentioned in the beginning with STEM is having an impact right now. There’s a constriction. All of those jobs aren’t going to come back, but that’s a ripe workforce to reimagine what they can do—your skills. That’s that separation for me as I started to study my Eastern religion philosophies—the separation of identity from my title and my person. That’s important because right now, in this inflection point, it isn’t just one industry; it’s all of them.

Artesha Moore: [00:32:43] Hollywood is about to ramp this way up because they just released a movie—I think it was Pete Davidson. All the extras, all the scenes, all of the things, AI. I didn’t understand. I was like, “Oh, that’s great advancement.” I didn’t understand the impact on the bodegas, where they’re shot in New York. I didn’t understand all the grips. I didn’t understand the production people, the catering, and the such and such. What do they do now? The Screen Actors Guild is an association. They have an association to help them get new jobs using skills in different ways. That’s where I think associations can help today. But you have to then say, “Okay, we’re going to have sessions in our annual meeting that say ’Reimagine Your Career and Leave the Profession.’” Nobody wants to talk about that. Don’t talk about leaving the profession. Their employers won’t send it. Instead, we’re talking about saving human intellect. You’re not your job. Your job gave you skills. If I were my job, I would still be looking for receptionist jobs instead of being senior staff at 27. You’re not your title. And, associations, get out of your own way—help your people learn what they need to advance the world.

Jeff Cobb: [00:34:13] That might be a great place to bring it all together and start wrapping up.

Artesha Moore: [00:34:19] I’m staying forever, Jeff.

Jeff Cobb: [00:34:20] I would love to keep talking forever. We’re going to have to figure out some ways to keep continuing this conversation. But, for right now, if you’re an association leader—you’re trying to get out of your own way because you can see there’s plenty of noise, confusion, and competition out there, but there’s also plenty of opportunity—how do you articulate your unique value to the people who are going to most benefit from that value? It goes back a little bit to what I was saying earlier, that associations aren’t necessarily recognized or perceived for this very important role that they do play and can play. How do you seize that in your particular world?

Artesha Moore: [00:35:07] This is where fun play comes in with your board. I did an exercise and led a facilitation with our board a couple of years ago, and it was all around Marvel’s cartoon show What If…? What if we were in an alternate universe? I used ChatGPT to come up with…. I said, “Here’s the plot: We are this. We’re a $3 million organization. We focus on this. We do this. What if…? Open it up.” It gave, “You could be this kind of business. You could be this.” And then—I love Shark Tank—I gave the prompts to the board and put them on teams: “Now take this and do it. Y’all pitch it to us like Shark Tank.” Here’s what it did: It stopped the conversation about membership growth, acquisition, and bringing back the community, and so what could we do? We went to market these ideas. Skills became really big for us. We did a skill study. We did this. We did this what-if—playing games. They pitched it. There was some of it that was fun, but it unleashed their brain of what could be possible. Right now, whoever’s listening, open up an AI—whatever your favorite one is—and start to say, “If I had nothing, what could I be?” Don’t give it a whole bunch of “We’re this and that and this” because that just defeats the mission. Use a wide view. You don’t have to build that today. Stop being so literal. Have an imagination. Then go for a walk. You know what it’ll do? It’ll get you thinking about something different. I love all of my alphabets: ASAE [American Society of Association Executives], Association Forum, and PCMA [Professional Convention Management Association].

Artesha Moore: [00:37:02] We are doing hard work as associations because we compete with LinkedIn. We’re a leadership development, and so LinkedIn is always like, “How did I get out here?” Reimagine. Imagination is key. But it’s got to be with the board. It can’t just be with your staff because then you create this us against them. We do see things differently as staff, but, if you start to imagine what you could be, that would be the thing I would say: Just imagine. Don’t put a lot of constraints on it. And then the other part is do something that you’re going to fail at every year. Every year, do something that you are going to fail at—and that doesn’t have to be big at work. It could be at home. I’m a painter now. I’m terrible. But you know what, Bob Ross, thank you for inspiring me because I’m almost 50, and I started painting maybe in the last three years. It has unlocked what I never imagined, which was creativity in my technical work. Use that AI to challenge what you think. I probably have, at least every three weeks, a session where I just argue with the AI. But try some new stuff. And then, Jeff, you’re doing great work in this space. You have people on that have interesting ways of looking at things. It doesn’t matter what your industry is. Especially for those at the top, top, innovation is what we’re hired to do, but navigation is also needed. You need a direction that you’re going that inspires—and then I think we’ll make it on the other side of this thing.

Jeff Cobb: [00:38:53] That concludes my conversation with Artesha Moore of Association Forum. But stick around for the recap.

Celisa Steele: [00:38:59] If this episode was useful to you, we’d love it if you’d share it—whether on social media, in an e-mail, or in a conversation.

Jeff Cobb: [00:39:18] Artesha argues that, because AI, workforce disruption, federal funding shifts, and shrinking university pipelines are all converging at once, this moment is genuinely different from disruptions associations have faced before—and that changes both what they need to do and how urgently they need to do it.

Celisa Steele: [00:39:37] Artesha is direct in saying that associations are, in her words, “the most unknown powerful group” in workforce development right now. Closing that visibility gap isn’t just a communications problem. It’s a strategic one that requires associations to connect their work more explicitly to employer needs.

Jeff Cobb: [00:39:56] Her practical advice for leaders is to build imagination into their work and to commit to failing at something every year. The capacity to experiment is a muscle associations need for this moment.

Celisa Steele: [00:40:08] Thanks again for listening—and see you next time on the Leading Learning Podcast.

To make sure you catch all future episodes, please subscribe on Apple Podcasts, Spotify, or wherever you listen to podcasts. Subscribing also gives us some data on the impact of the podcast.

We’d be grateful if you’d rate us on Apple Podcasts or wherever you listen. Reviews and ratings help the podcast show up when people search for content on leading a learning business.

Finally, follow us and share the word about Leading Learning. You can find us on LinkedIn.

The post Associations at the Inflection Point with Artesha Moore appeared first on Leading Learning.

The credentialing landscape is changing fast. With 1.8 million credentials now in the U.S. marketplace and AI reshaping entire professions, organizations need to understand not just what credentials to offer but how the meaning of competence is evolving.

In this episode of the Leading Learning Podcast, co-host Celisa Steele talks with Denise Roosendaal, executive director of the Institute for Credentialing Excellence (I.C.E.). Denise shares how assessment is moving from point-in-time tests toward ongoing, real-world evidence of competence, why the most important skills of the future may be the uniquely human ones, and what it takes to build a culture of foresight that helps credentialing bodies and learning businesses get ahead of what’s coming.

To tune in, listen below. To make sure you catch all future episodes, be sure to subscribe on Apple Podcasts, Spotify, or wherever you listen to podcasts.

Download a PDF transcript of this episode’s audio.

Celisa Steele: [00:00:03] If you want to grow the reach, revenue, and impact of your learning business, you’re in the right place. I’m Celisa Steele.

Jeff Cobb: [00:00:10] I’m Jeff Cobb, and this is the Leading Learning Podcast.

Celisa Steele: [00:00:17] Certification, certificate, microcredential, badge—these terms often get lumped together. But they’re not interchangeable. They represent different business models, different purposes, and different promises to learners and employers.

Jeff Cobb: [00:00:32] Our guest in this episode, number 483, is Denise Roosendaal, executive director of the Institute for Credentialing Excellence (I.C.E.). Denise has worked for more than 35 years in the credentialing space and has led I.C.E. for 15 of them.

Celisa Steele: [00:00:46] Denise walks us through the credentialing landscape—which includes a whopping 1.8 million credentials in the U.S. marketplace—and what that ecosystem means for workforce development and the learning businesses operating in it.

Jeff Cobb: [00:00:59] You and she also cover how assessment is evolving from point-in-time tests toward ongoing, real-world evidence of competence and what that may mean for credentialing bodies and learning businesses.

Celisa Steele: [00:01:12] And Denise makes a case for why durable, uniquely human skills—critical thinking, creativity, and problem-solving—are going to matter more, not less, as AI reshapes the workforce.

Jeff Cobb: [00:01:25] If you offer credentials or are thinking about adding them to your portfolio, or if you simply want to understand where credentialing is headed, listen to this conversation with Denise Roosendaal.

Celisa Steele: [00:01:41] For listeners who are maybe not super familiar with the Institute for Credentialing Excellence, give us a short version of what I.C.E. does, the problems that it exists to solve, and the work it exists to support.

Denise Roosendaal: [00:01:55] I.C.E. is a community of credentialing professionals. These professionals want to raise the bar on credentialing best practices. This community has always been incredibly generous with their thought leadership—what’s working, what’s not working, sharing things that they have learned, lessons learned. And now we are pushing the envelope on what the future holds for credentialing, whether that is the standards that we have and accredit to, which are NCCA [National Commission for Certifying Agencies] for certification programs and ICE 1100 for assessment-based certificate programs. We are always looking to figure out how programs build and maintain high-quality, rigorous certification programs. In addition to the standards, we offer education for those individuals who are making their way in the certification credentialing world. We offer research that really tries to hit at those highly relevant and important aspects of this work. How can we talk about it? How do we benchmark it? How do we tell the story of credentialing, both internal to those organizations that understand credentialing as well as to external audiences who may not be as familiar with the terms we use, how we go about it, or the purpose that it plays in the marketplace?

Denise Roosendaal: [00:03:30] We also have a certificate program ourselves for credentialing topics, and we also have the ICE-CCP [I.C.E. Certified Credentialing Professional] certification program. We are always trying to educate the association world and population on credentialing and the role that it plays and trying to recognize those who have demonstrated proficiency in the certification field that we want to aim toward. Some of the topics that we are leading on are how we articulate the value of credentialing: what it means for the individual certificate, for employers, for external stakeholders like regulators, legislators, third-party payers, how they think about credentialing as a sign of competence in the field. We also, as I mentioned, think a lot about the future of credentialing and exercising foresight, understanding what all the signals in the marketplace might be telling us about credentialing, and how the business of credentialing may be evolving.

Denise Roosendaal: [00:04:38] We’re going to talk some about AI. But there are so many aspects of the business of credentialing. Ninety-five percent of our members are nonprofit organizations, 501(c)(3), 501(c)(6) organizations, and they are on a mission to protect and uplift their profession with the work that they do in identifying competence, talking about competence, measuring competence in their field. Those are the topics and the areas that we are trying to address in our work, and we feel that we have a very tight-knit community but always welcoming others who are curious, who just want to understand a little bit more. While we like to see programs get accredited so that they can demonstrate that they have met those standards, not all of our members are accredited. Many organizations come into I.C.E. to better inform themselves about best practices, create a community, a network of other certification professionals who talk their language, and understand the terms that we use. I’m very proud to represent this organization and the community.

Celisa Steele: [00:05:51] One of the things that can be an issue in this space is terminology. We have certification, certificates, microcredentials, digital badges, a lot of other terms. Those are just a few. I would love to hear from you about where you see the biggest points of confusion and what distinctions really matter with some of those terms.

Denise Roosendaal: [00:06:15] Great question. In the 15 years that I’ve been with I.C.E., this continues to be a very important topic because terminology and words matter, and so how we talk about what we do is important. We have a terminology document that we are about to release. It’s an updated version of our 2020 document, and it’s going to get into some of the distinctions that I’ll talk about. But we use credentialing as that umbrella term. It means everything that can be used to recognize skill, competence, or knowledge. Certification is a subset. Certificate, microcredentials, diplomas, apprenticeships, badging—all of that fits underneath that credentialing umbrella. This is important because these are different business models, intended to achieve specific and different outcomes. We’ve seen a lot of misapplication of these terms, especially certification and certificate programs. Let me underscore that a little bit. Certification is intended to recognize the knowledge, skills, and abilities that an individual has acquired in their career through education, practice, or occupation. It typically has eligibility requirements, renewal obligations, and a rigorous assessment. Once an individual achieves success in the application process, then they can carry a designation—letters after their name.

Denise Roosendaal: [00:07:48] A certificate program is an educational program with specific learning objectives that should be demonstrated in the course assessment, assuming there is a course assessment at the end. It’s a point-in-time measurement of understanding of those learning objectives. It typically does not have eligibility or renewal requirements, and, according to our standards—the ICE 1100 standard—it doesn’t carry a designation after a person’s name. That’s where the confusion comes into the marketplace, and we recognize that. But we have stuck with that distinction because we didn’t want to contribute to the confusion in the marketplace. We wanted a clear distinction. One is not better than the other. One is meant to be recognition while the other is meant to be an indication of an individual having gone through an educational course. They are very different in the way they operate and what they’re trying to accomplish for their certificants. There’s another class of emerging models, which are those microcredentials you mentioned, and typically that will come with a badge. In our terminology, we like to think about badging as the recognition of having demonstrated skills attained. But we recognize that in other spaces, especially in the academic field, badging is the program itself.

Denise Roosendaal: [00:09:15] We understand that there are differences in how we describe these things. For us, microcredentialing means a smaller slice of the overall skill set that has been articulated for a profession or occupation. And you can get really creative with that microcredential. You can create stackable models, perhaps a series—maybe they are synchronous/asynchronous; maybe they are meant to build on one another; maybe they are all discrete. But it’s an interesting way to get in the game of skill set identification, skill set training, and recognition. What’s important here is that the field has begun to respond to what learners are telling us. They want those smaller bits of recognition. They want to be able to say, “I’m proficient in this area, and let me put that on my LinkedIn; let me put that in my resume,” so that they can indicate to employers that they have gone through training; they have gone through a series or a program to get recognized. It’s also important that we understand what employers are telling us. Employers are saying (according to a study out of Northeastern University, I think), “This is great. Be innovative. Make sure you’re responding to the marketplace. But what we need as employers is validated skills. So let’s not forget the assessment component of those items so that we know that they have been validated in the process.”

Denise Roosendaal: [00:10:55] What’s important with all of these models is to understand the goal of the organization in creating such a program, understanding the market, doing the market research so that employers and potential participants have a say in what they want or need or what they think is relevant—really doing that rigorous work to identify the key competencies, especially in the case of a certification where you’ve got to identify what will be assessed. And get very clear about what that is. That’s super important. It takes time. It takes money. But that investment is well worth it. I can’t tell you how many programs come to us and say, “Hey, we’re two, three, four years in, and the program’s not growing,” and then we realize they didn’t do any of that homework, and they didn’t do any of the connection between the goal of the organization and what it is they were putting out in the marketplace. In many cases, they had to start from ground zero and start all over. It’s very difficult to go back and retrofit a program if that work hasn’t been done. We have resources on all of this.

Denise Roosendaal: [00:12:02] One of the things I would point to in addition to the terminology document is something that we call the “Defining Features” document, which is literally putting certificate programs next to certification programs and outlining what those differences are. It’s free, downloadable for anyone. We want the community to be educated on this. There’s also a video. Many folks have reported to us that they use the video with their boards so that they can help them understand the differences. And we also have a strategic questions document—free, downloadable to anyone. That is a good checklist for boards to go through and say, “Okay, what is it that we are trying to accomplish? Let’s go through these series of questions to get alignment.” Just a couple of resources we have on this super important topic.

Celisa Steele: [00:12:54] When you look at the credentialing landscape today, what would you point to that feels most important for learning businesses to understand about where credentialing is now and where it may be heading?

Denise Roosendaal: [00:13:07] Let me throw some stats at you because that gives us a foundation to work from. Credential Engine has a report out. They updated it last year. There are 1.8 million credentials in the U.S. marketplace. That includes academic diplomas. It includes badges. It’s a big, wide ecosystem. But 1.8 million credentials is a lot, and it’s serving an important population. This report goes on to say that the field of adult education, served by these credentials, is a $2.3 trillion industry—that’s significant. If you break down the 1.8 million credentials, we’re looking at 14,000 professional licenses, 6,900 certifications, and 3,400 microcredentials. All of those are a subset of that 1.8 million. But the key here is that this ecosystem is growing. Employers and academic environments have recognized the importance of not only traditional models but also innovative models for making sure folks feel they have the skills that they need and then connecting them with employers in their fields. This is a an editorial comment: I do hope this ecosystem, this industry, is growing for the right reason because it’s not just about monetary gain; it’s about the enhancement of the workforce.

Denise Roosendaal: [00:14:43] That’s why I like the fact that I.C.E. is so filled with nonprofit organizations because there is a business component to this, and these nonprofits have to operate under business constraints just like everyone else. But I feel that, with a nonprofit mission orientation, they are thinking hard about what it takes to be proficient in their profession, and that’s really important. There are a lot of for-profit entities in this space as well. I’m not going to knock that. But it’s important that we make sure the competencies that are associated with a profession are well-articulated, and the value of a credential is well-defined—a real hallmark of those that are in this for the right reason. A quick word about foresight. As we think about this landscape and where we are headed, I.C.E. has been doing a lot of work in the future of credentialing. We have an incredibly large toolkit—and we can talk more about that in a second—but it’s important to think about the culture of foresight within an organization. We’re all so busy in our day-to-day worlds, and it’s hard to lift our heads out of that grindstone. But we need to look to the horizon and understand what’s possible, what’s headed our way, what the risks and opportunities are, and build the future scenarios so we can say, “Okay, this is the place we want to go,” and “How do we begin to make those decisions to get there?”

Denise Roosendaal: [00:16:26] Nonprofits, like many organizations, have a finite set of resources, and so deploying those resources in a way that can help the organization get to that preferred future is critically important. I often talk about this as the culture and the competency of foresight, building that into the organizational mindset and practices. As I think about that future landscape, I can’t tell a particular profession, “This is exactly what’s happening or going to happen.” I don’t have a crystal ball. But foresight helps us gain the language to begin the analysis of those potential futures and get comfortable in a very uncomfortable space. In talking about I.C.E’s look at the future of credentialing, we have a paper that we’ve written that looks at the future of credentialing through the lens of technology, through the lens of the business model, and through the lens of government and how they may have a say in that future as well. We have tools that go along with that to help begin to have that uncomfortable conversation with governing bodies, with certificants, and to digest what that future might look like. I suspect we should probably talk about AI, which is an important component. I am a big advocate of making sure we think about what AI can bring.

Denise Roosendaal: [00:18:05] I feel like we’re over the hump in terms of the fear factor of AI. It’s an important aspect. We need to think about the harm or risk that AI might bring. But I feel like many credentialing bodies are diving into one of those opportunities that AI can bring them, always with the caveat: Anytime we’re using AI, we’ve got to have that human intervention, that human element really thinking about what those opportunities might be, making sure we’re gathering data about how we’re using AI, and what those improvements might be. There’s a lot of great opportunity with AI, and we shouldn’t be so afraid of it. Other topics we’re certainly looking at are what are the impacts of changes in higher ed? What are the needs for alternative pathways for eligibility for certification? What might be the impact of that ideological battleground that we might find ourselves in with state or federal stakeholders? A lot of topics we’re looking at. I’ve only scratched the surface here.

Celisa Steele: [00:19:15] I know that foresight is such an important topic for you and an area of focus. You’ve connected foresight to the idea of the future of professions and the workforce and where things are going there. How would you suggest that learning businesses think about their role and how they can shape the future of the fields they serve so they’re not just observing what’s happening, but they’re trying to have that influence on where things might go?

Denise Roosendaal: [00:19:44] Really important question. I highly recommend organizations look out at the field and see what the data might be telling us. One report that I rely on heavily and cite often is the World Economic Forum. They released a 2025 version of the study of the future of the workforce, and they are predicting significant shifts in the types of occupations and the growth of occupations in the next five years. 2030, they’re saying there’s going to be a 7-percent growth. But that’s not a straight-line growth. That’s retiring of many occupations, which may become obsolete even as we grow new professions and occupations that we haven’t seen or understood just yet. Another piece in that report talks about durable skills, which are critically important for us to think about. They have a top ten list of skills that will be needed by 2030. You would think that would be largely occupied by technological skills. Two or three of them do talk about the importance of understanding technology, but the rest are all about durable skills. It’s all about how we are uniquely human and what humans are bringing to their occupation—critical thinking, problem-solving, creativity, teamwork. I can’t say enough about this report and how they are setting up and shaping that conversation around durable skills. Also, we have to think about how we measure those skills, and that’s going to be a big challenge for credentialing bodies moving forward.

Denise Roosendaal: [00:21:23] There’s a lot of research out there on how it’s being done. But one of the things I ran across in a conference I attended was an AI application to teach doctors how to have better bedside manners. And I thought, “AI is teaching doctors how to be better humans!” That was really interesting, but I can’t emphasize enough how important I believe human skills will be in the future of our workforce. ASAE, the American Society for Association Executives, has a ForesightWorks set of drivers. They look at 50 different drivers. One of them that I have relied on heavily in the I.C.E. world is “more human humans.” How are we going to deploy those ideas moving forward? But, in the ways that we’re thinking about foresight, I should also mention I.C.E.’s partnership in the Advancing Workforce Mobility initiative. It’s a grant coming out of the Education Design Lab, and we are partnering up with Credential Engine, funded by the Walmart Foundation. It’s a really significant grant to think about how we might analyze or build a prototype for using micro-internships for skills validation. And CredUp Hub, led by Mickie Rops, is going to help us think through that. We don’t have all the answers yet, but, again, this is an innovative way to think about what the future of credentialing might look like if we get creative. I’m really excited about that work as well.

Celisa Steele: [00:23:12] You were just talking about durable skills, workforce mobility, those sorts of things. Clearly tied in with part of that is the assessment. How do you measure, as you were saying, some of these skills and knowledge? You’ve argued that in the future assessment, systems may increasingly draw more on real-time, on-the-job evidence of proficiency. Talk a little bit about how you see assessment evolving and what that might mean for more traditional credentialing approaches.

Denise Roosendaal: [00:23:47] It’s a really important area and one that’s evolving quickly. I don’t have all the answers, but here’s what I’m learning. I want to do a shout-out to our psychometricians in the space because they are going to want to make sure that whatever we do in the future is psychometrically sound, and they are looking at this as well. Technology is going to have a profound impact on how we gather performance data. Currently, we rely quite a bit on measuring continuing competence, either through assessments or portfolio reviews, where a team of individuals analyze skills. But I believe technology can take us incredibly far in terms of how we measure and collect performance data. One thing I’m hearing quite a bit about is wearable technology, and what does that mean for on-the-job measurement? And then, over time, what does that mean for calculating enhancement of performance, and how do we demonstrate that performance over time? One of the great examples I’ve heard—I use this a lot; I hope they’re still innovating—is the idea of individuals who work on trees, tree cutters, tree trimmers wearing technology in a helmet that might then measure how much they have respected a cone of safety around a project that they’re in and whether they impeded in that zone of safety or respected it and then, again, how they changed their behavior over time. Calculating this real-time data and analyzing it over time can give us a great indication of an individual’s performance on the job. There’s some real promise there that’s important.

Denise Roosendaal: [00:25:43] I also think that AI and other technologies will do a great world of good in terms of how we do initial assessments. I think performance testing will become more and more important and perhaps less and less expensive because they hold an important element for that initial assessment of how we are able to demonstrate, in a practical way, the skills we have identified for competence in a particular area. That’s a very intriguing area as well. Another area that I’m beginning to understand better is how organizations are investing in longitudinal examinations of competencies of their professions, so it’s not just a point in time but, again, over time. This wearable technology idea is probably an example of longitudinal exams. But what’s interesting about it is that individual practitioners can get real-time feedback in terms of what they have learned or areas where they are weak so that they can begin to enhance their professional development to seek better knowledge or skills in particular areas. Some of this is the MOCA Minute, which is a model that many organizations have adapted, where you get a question on your phone every day or once a week, and you answer that question. Over time, it begins to build to say, “You’re really strong in this area, but here’s an area you might want to pay attention to in your professional development.” And then some organizations are building that into their recertification process. That’s really interesting and some very promising innovations.

Celisa Steele: [00:27:33] I especially appreciate the idea of moving beyond the point-in-time level of assessment because it speaks much more to true proficiency and true knowledge and skills. You began to talk a little bit about professional development and how some of the feedback that professionals are getting might influence what professional development they undertake. A lot of the learning businesses we work with are embedded in associations. They provide a lot of education and professional development to serve those professionals. We’ve been talking mostly about credentialing, but there is clearly this other side, which is education and professional development. Talk about that relationship between education and credentialing. Where do you see them naturally supporting one another, and where is there tension that might arise?

Denise Roosendaal: [00:28:25] The tension is a healthy tension. It’s a really important element and concept to think about—that traditional separation of church and state that is in our own NCCA standards. This is all about the firewall approach. And there are good reasons for this separation, so let’s hit those first. You don’t want educational instruction to be so tied to an exam that you’re teaching to the test. That’s a really important aspect. You want a well-rounded audience of subject matter experts developing the credentials so that you’re not just relying on educators to create the content that is used in a credential. You want the creation of the credential to be separated from those who are building content. You want exam development activities—item writing, exam development, job task analysis—to be separate from education. The firewall is also good for security measures and keeping each side of the house independent of each other so that exam questions/items are under good lock and key, even though educators can reference tools that programs will publish, like an exam blueprint or a body of knowledge, to make sure folks understand what the exam will cover.

Denise Roosendaal: [00:29:48] Lastly, there are really good governance reasons why you want this separation so that there’s not undue influence on credentialing decision-making from others in the organization. So keeping those things clean is also important. But it’s also important for credentialing bodies to understand the lifelong-learner nature of their constituency and be able to serve that. There’s a really important element to understanding how the particular field is changing quickly. You don’t want to be so isolated from the constituency or education population that you’re not appreciating how quickly changes are coming about, and you want to make sure your credential is reflecting the important changes that are coming about. It’s important that credentialing bodies respect that separation. But, again, it’s perfectly fine for organizations to be engaged in test prep or continuing education, as long as they have that healthy respect for that firewall internally. There are ways to navigate that. Again, we have a lot of resources on that—articles, things like that. This idea of lifelong learning is really important in society generally, and so we want to make sure that we are respecting those needs in our own communities as well.

Celisa Steele: [00:31:12] Since you’ve brought up lifelong learning, I want to ask a question we love to ask all of our Leading Learning Podcast guests, which is about your own lifelong learning. What are some of your specific habits, practices, or sources that you use to continue to grow?

Denise Roosendaal: [00:31:27] Several years ago, I took the Clifton StrengthsFinder [CliftonStrengths] and discovered that I am a learner; that’s one of my strengths. When I learned that about myself, so many things finally made sense. I’m the one who’s always asking questions like “Why?” or “What if?” I was that kid growing up—always asking your parents, annoying them on the road trip, asking about everything you saw along the way. I am definitely that person. Curiosity, to me, is a core value. I’m constantly reading. I have more books in my house than furniture to house them. I have a Kindle. I have my Audible app. I’m constantly enrolling in online classes in my spare time. I have two graduate degrees, and the second one I got is in creative writing, just for the fun of it. I am constantly trying to feed my brain with good information. One of the things I have found that brings me joy and energy for my very tired brain is to sit in a room with really smart people and ask the big questions. This happened recently in a meeting between Smithbucklin, our management partner, and McKinley Advisors, where we asked just big questions. It was only an hour, but I left that meeting incredibly energized, thinking about how are we are operating in this fast-changing environment. I can’t say enough about surrounding yourself with that kind of environment. Go to conferences; enroll in meetings; listen to podcasts like this. The importance of our network and learning from our network is critically important. Those are the ways that I try to be a lifelong learner, and I don’t think there’s any end in sight.

Jeff Cobb: [00:33:23] That wraps up the conversation with Denise Roosendaal of the Institute for Credentialing Excellence. Stay with us for our recap.

Celisa Steele: [00:33:29] Here are links to some of I.C.E. resources Denise mentioned:

If you found this episode valuable, we would be grateful if you’d share it so more people can find the show and benefit from the conversations we have.

Jeff Cobb: [00:33:52] Denise made the point that the terms we use in credentialing aren’t just semantics—they reflect genuinely different models with different purposes. Knowing the distinctions between certification, certificates, and microcredentials helps learning businesses make better strategic decisions about which fits their goals and market.

Celisa Steele: [00:34:12] On the future of assessment, Denise sees real promise in moving beyond point-in-time tests toward ongoing, real-world evidence of competence—from wearable technology that captures on-the-job performance to longitudinal approaches that give practitioners feedback over time rather than at a single moment.

Jeff Cobb: [00:34:29] And, as AI takes on more technical work, Denise believes the skills that will matter most are the uniquely human ones—critical thinking, creativity, and problem-solving. Credentialing bodies that get ahead of that shift will be better positioned to serve their professions.

Celisa Steele: [00:34:45] Thanks again for listening—and see you next time on the Leading Learning Podcast.

To make sure you catch all future episodes, please subscribe on Apple Podcasts, Spotify, or wherever you listen to podcasts. Subscribing also gives us some data on the impact of the podcast.

We’d be grateful if you’d rate us on Apple Podcasts or wherever you listen. Reviews and ratings help the podcast show up when people search for content on leading a learning business.

Finally, follow us and share the word about Leading Learning. You can find us on LinkedIn.

The post Foresight and the Future of Credentialing with Denise Roosendaal appeared first on Leading Learning.

Many learning businesses market what they offer. The ones that get better results market what their learners need.

In this episode of the Leading Learning Podcast, co-host Celisa Steele talks with Amy Michalski and Shelly Strickland, who handle business development and marketing at the Association for Intelligent Information Management (AIIM). They share what moves the needle in marketing education offerings—from how they segment and nurture prospective learners to what they’ve learned about conversion, measurement, and the difference between reach and genuine engagement.

To tune in, listen below. To make sure you catch all future episodes, be sure to subscribe on Apple Podcasts, Spotify, or wherever you listen to podcasts.

Download a PDF transcript of this episode’s audio.

Celisa Steele: [00:00:03] If you want to grow the reach, revenue, and impact of your learning business, you’re in the right place. I’m Celisa Steele.

Jeff Cobb: [00:00:10] I’m Jeff Cobb, and this is the Leading Learning Podcast.

Jeff Cobb: [00:00:17] Most professionals aren’t searching for training. They’re searching for answers to problems. For learning businesses trying to market their offerings, that distinction makes a real difference.

Celisa Steele: [00:00:27] Our guests in this episode, number 482, are Amy Michalski and Shelly Strickland of the Association for Intelligent Information Management (AIIM). Amy is AIIM’s chief business development officer, and Shelly is marketing manager.

Jeff Cobb: [00:00:40] Amy and Shelly walk through AIIM’s approach to marketing a full portfolio of education—from on‑demand training and certification to events and Webinars—including how they think about the difference between selling to individual learners and selling to organizations.

Celisa Steele: [00:00:55] We also cover segmentation, conversion, and measurement—and why engagement quality matters far more than the vanity metrics some organizations are still chasing.

Jeff Cobb: [00:01:05] And Amy and Shelly make a strong case that associations already have real advantages over commercial learning providers—advantages like trust, community, and direct access to practitioners.

Celisa Steele: [00:01:16] If you want a practical, real-world perspective on what it takes to market education effectively, even with a small team, stick around for this conversation with Amy Michalski and Shelly Strickland.

Celisa Steele: [00:01:33] For listeners who may not be familiar with the Association for Intelligent Information Management, would you give us a short explanation of what AIIM does? Amy, I’m going to ask you to do that.

Amy Michalski: [00:01:46] Sure, happy to. AIIM is a global community. We’re focused on both intelligent information management and helping those organizations become AI-ready. So our name, A-I plus I-M, really fits what we’re doing. Everybody’s racing at this point to implement AI, but the reality is, if your information is everywhere across your organization, in multiple systems, and isn’t labeled accurately, nobody really knows what’s governed and what’s not. AI can quickly become more of a liability than an asset. What we really do and what we’re trying to help our community do is help them get their information house in order. We do this through training. We offer certification. We have multiple events, both virtual and in person. And we have a strong community of practitioners who are trying to solve these same challenges together.

Celisa Steele: [00:02:48] All that AIIM does is done with a small but mighty staff—five staffers to run the full portfolio, including certification, events, on‑demand courses, Webinars. Shelly, as marketing manager, you’re doing, I presume, a lot of the heavy lifting on marketing those education offerings. Paint for us a picture of what marketing at AIIM looks like. Who are you trying to reach? What channels are you using? And what does the toolkit that you’re using look like on a day‑to‑day basis?

Shelly Strickland: [00:03:23] Marketing at AIIM is definitely a wear-a-lot-of-hats environment, but it’s also what makes it very exciting. With the small team, there’s no room for silos. Everyone contributes to the organization and marketing, which means balancing strategy, execution, analytics, communication, and customer experience all at once. My role touches on every part of that learner’s journey. On any given day, I could be building campaigns for certification, promoting Webinars, virtual training, managing conference marketing, writing e‑mail copy, updating the Web site, creating social campaigns, and even troubleshooting, registration workflows as well. It’s very hands‑on, but it gives us the ability to move quickly and stay connected to our audience. Our audience is primary information managers. That includes record managers, information governance leaders, compliance professionals, legal teams, IT leaders, and content managers. These folks are increasingly focused on AI governance, digital transformation. One of the biggest shifts that we’ve seen recently is that AI is expanding that conversation. Now we’re reaching professionals who haven’t traditionally thought of themselves as information managers, but suddenly they realize that the information quality, the governance, and the structure are all foundational to making AI work.

Shelly Strickland: [00:04:53] Channel-wise, we do a lot of e‑mail marketing here at AIIM because our audience still responds very well to e‑mail. We do some social marketing, especially on LinkedIn because that’s where we find a lot of our audience. And it’s become increasingly important for us to do thought leadership and community engagement through those social channels. We also use Webinars strategically, not just as education experiences, but they’ve become top-of-the-funnel for engagement opportunities to introduce people to the AIIM system. Our toolkit is a mix of marketing automation, our CRM, some social scheduling, Web site management, event platforms, and a lot of content development. But one of the most important tools that we have found is just paying close attention and listening to what the people in the industry are talking about. They start to tell us what challenges keep coming up, which topics they’re interested in, where AI is gaining traction, and what they need help with. That helps us shape the messaging and create programs that feel timely and not generic to them.

Shelly Strickland: [00:06:05] With a really small team, it’s important for us to prioritize constantly, and you learn pretty quickly that the programs and the activities that you’re doing, if they don’t drive revenue, they’re just creating a lot of noise, and there’s not a lot of time. So we focus heavily on practical value, audience relevance, building trust over time. In the association space, most people don’t buy a course. They buy credibility. They buy a community. And they want the confidence that the education that we’re giving them is helping them solve real problems. So we focus on solving real problems that they have in their work.

Celisa Steele: [00:06:47] Thank you for outlining that approach. You were talking about hearing from the people you’re targeting as learners about what they need. Are you doing that through listening, say, on LinkedIn, to what you’re hearing members saying, or are you doing it because you’re reaching out through a survey? Talk a little bit about how you’re getting that information about what they need or want.

Shelly Strickland: [00:07:08] We do surveys. After we have every Webinar, after we have every event, we send out a survey to them. And then, not only do we look at the survey results, we try to pivot based on what those results are saying. If they’re saying they have too much of this, then we stop doing that, and we focus somewhere else. But a lot is just keeping up to date with what’s going on on LinkedIn with people that are in our community. As a company, we follow our members on LinkedIn and different platforms, and we see the things that they’re talking about. We see the pain points. We see the frustrations. And a lot of that education that we come up with is based on those things that they’re saying are a pain point for them.

Celisa Steele: [00:07:56] Amy, you’re the chief business development officer. I see that as sitting at the intersection of marketing and sales, and I’m imagining that you’re thinking a lot about how to take awareness and make sure that that leads to revenue. Talk about how you and Shelly divide and coordinate the work. And how do you think about the overall go-to-market approach for AIIM’s learning portfolio?

Amy Michalski: [00:08:21] Good question. Shelley’s more focused on the individuals with membership, training enrollments, event registration, and everything that she just described that’s involved with that, our Web site design, all of it. I focus more on the organization itself, things like team training deals, group certifications like our CIP (Certified Information Professional), and sponsorships, which is a big component of our revenue. But honestly there is a lot of overlap, in a good way. We collaborate often. We’re on phone calls multiple times during the week together, building our content strategy, our messaging, what’s our plan for this week, how can we bring new people into the AIIM community. For example, right now we’re working on rebuilding our new contact sequence, and we’re also working on a re‑engagement campaign because we realized that our old approach is a little outdated, a little too transactional. So we want to do some more segmentation and make it more personalized for each individual. And then, naturally, there’s a handoff process. If I start seeing, for example, multiple people from the same organization downloading content or attending one of our Webinars or showing up to events, then that usually means it’s my time to reach out and have a conversation.

Amy Michalski: [00:09:59] I’ll get on an exploration call with them to try to understand what problems they’re trying to solve, why they’re having team members go to multiple activities from AIIM, and, from there, I’ll put together a customized proposal that makes sense for them. Sometimes it includes items like team training. Other times maybe it’s a bulk purchase of our certification exam. And then many times it’s a sponsorship activity. Sometimes it combines all three. Shelly’s building the awareness and trust at the top of the funnel, and I’m helping turn that interest into, hopefully, a longer‑term partnership and revenue. Thankfully, tools like our CRM system help us stay coordinated, so we’re not stepping on each other’s toes, and we know who’s doing what.

Celisa Steele: [00:10:52] Direct-to-consumer marketing, especially of on‑demand content—a little bit less of what you were just talking about, Amy, and a little bit more of what Shelly was talking about before—is something that a lot of association learning businesses genuinely struggle with. Actually managing to connect with those prospective learners and getting attention is very tough these days. Shelly, I would love to hear a little bit more about some of AIIM’s approaches to marketing its online training library. What does that look like? Have you found things that help you move the needle in terms of getting people to read and then convert?

Shelly Strickland: [00:11:33] One of the things AIIM has learned over time has been that we don’t market our training as on-demand. We do say it’s an on-demand library, but we sell more of the outcomes. Instead of saying, “Here’s our library,” we say, “Here’s what we have that can help you solve this problem.” It’s about solving their immediate problems and how to help them move forward in their careers or their organizations. Because our attention spans are really short, and there are people everywhere we’re training, and competition is everywhere. Most professionals are so overwhelmed that they don’t go out searching for “I want training.” They’re searching for “How can I fix this?” or “How do I do this?” That’s how we focus our marketing to them—what is it that they need help with, and what do we have that matches up to the things that they need help with? A big part of what moves the needle for us is, again, relevance and timing. We focus on topics that have real urgency in our market, like AI Governance or M365. There’s a lot around security and automation right now. But, again, these people aren’t searching for training; they’re searching for “How do I actively solve that problem?” And then we build our marketing around that pain point first and foremost.

Shelly Strickland: [00:13:05] We also try to make it feel less transactional and more connected to your personal growth. Instead of positioning AIIM+ Pro as just a content library, we position it as “Here’s your career development path. Here’s what you can do to grow in your career. Here are the things you need to know to get certified.” They want more outcomes than they do content, and so we focus our marketing on what those outcomes will be for them. Another thing that we’ve done is leverage live experiences like our on-demand Webinars, conferences, workshops, and things like that. We don’t necessarily use them as, “Here’s a Webinar. Now buy our training.” It’s more like, “Here’s our Webinar, and here are some of the other things we do.” By getting their foot in the door and building that credibility with them, then we’re able to say, “If you want more help going towards this path, we have all these things here that can help you.”

Shelly Strickland: [00:14:10] One of the last things I would say is that being consistent with them is what really matters. We really test our messaging. Sometimes switching a Webinar title might be the easiest thing to switch your marketing. It’s the same Webinar, but how you message that to them in a way that helps them overcome their pain point is what we try to do. Being a small team, a really nice thing is we can switch that on a dime. We can put up a Webinar, and, if it doesn’t have the conversions we’re looking for, the next week we can change that Webinar. It’s very easy to do when you’re in a small team, and it’s a little bit harder when you’re a big team, but we focus quickly and nimbly on what messaging reaches that right person.

Celisa Steele: [00:15:02] Shelly, you’ve been talking about the individual learner and sending those e-mails, making sure that you’re trying to align to their pain points. Amy, let’s talk a little bit more about that other side of the equation, the B2B side, in terms of corporate or institutional buyers. What does that look like? You shared some of it. But, when you’re approaching those prospects, how do you think about that B2B sale differently than you think about reaching those individual learners, like Shelly was just explaining?

Amy Michalski: [00:15:34] Shelly has to segment and put individuals in buckets based on the needs that they have, where, for corporate sales and B2B, it’s individualized, and it has to be. It’s reaching out and understanding your audience’s needs, one to one. For example, there’s a huge need right now around AI governance when they’re trying to get their data ready for AI. I’m meeting with a government agency, and they have a real need for an AI data-readiness-type workshop. We offer workshops, but we didn’t have anything specific for them. So I reached out to a few consultant partners that we’re working with to customize a workshop for them. Really, it’s all about trying to figure out how we can match the needs of the industry and quickly create content with partners that matches those specific needs to be able to work with them. If the opportunity makes sense, and we find that there is an audience for this, it makes a lot of sense for us to collaborate and partner with our industry to be able to provide this service for our members. We wouldn’t be able to do that for one individual, where we can do it for a large organization, say, of 50, and then expand from there because we know and see that there’s value beyond.

Celisa Steele: [00:17:14] Shelly, it might be useful to ask you a question about the segmentation of individual learners. Can you talk a little bit about how you think about the segments and what those segments look like for AIIM?

Shelly Strickland: [00:17:29] Some of the segmentation comes from what they click on our Web site—what it was that drove them to fill out a form, what particular thing they’re interested in. We have, in the past, asked them, “What is your interest?” or “Do you want to hear about this, this, or this?” We stopped doing that, but we’re going back to doing that. We’re in a project right now to make those preferences a little more granular in the database. Basically, we know what training they’ve taken, what things they’ve clicked on, and what brought them to us to begin with. If they’ve signed up for Webinars, that might put them in the bucket that they want Webinars. Or, if they’ve taken a course on M365, then that might be my M365 bucket. A lot of it comes in your database as where did the people come from? What are they spending time on? What have they responded to? And what e-mails have they opened? If you sent a general e-mail with ten different things, and they clicked on only one that had something to do with M365, then that buckets them and scores them for M365 or whatever bucket that they would fall in.

Celisa Steele: [00:18:49] I would love to talk about measurement. A lot of learning businesses struggle with knowing what is it right to measure, what are appropriate targets. Shelly, would you talk about what measurement looks like at AIIM? What metrics do you track, are you setting explicit targets, and does what you learn from those numbers change how you work or what you offer?

Shelly Strickland: [00:19:11] Measurement is something we think about all the time at AIIM. Because we have such a small team, we can’t afford to spend a lot of time on activities that don’t produce anything. We track a mix of engagement, conversion, revenue metrics, depending on the campaign or program. On the marketing side, we look at those e-mail engagement and conversion rates, Webinar registrations, event registrations. We look at membership growth, how many people are interested in this certification, social engagement, and things like that. But one thing we’ve learned is that there is no single metric that tells the full story about any particular program. For example, a Webinar might not immediately convert to sales of training courses, but it may bring in more highly engaged prospects who later decide that they want membership or might come to a conference, or they might decide to take this certification.

Shelly Strickland: [00:20:09] We do set targets, especially around registration, lead generation, and membership campaigns. But we try to stay flexible with audience behaviors. They change so fast that what we did and who used to be engaged last year is not the same as who is engaged this year. And just because they did this last year doesn’t mean that they’re going to do that this year. Data tracking has absolutely changed how we work, and it’s made us more audience-focused. We get better at adjusting messaging faster when something isn’t opened, when somebody isn’t clicking on things. Sometimes even a small change in positioning helps those metrics go up. It’s very much about paying close attention to everything you put out there to see how well it works across the board. The biggest lesson for us has been that engagement quality matters a lot more than vanity metrics. Open rates and all of that don’t hold the same value they used to. What holds more value is are people clicking through? Are people attending the Webinars? And how engaged are the people that we do have? It’s much better to have a small audience of highly engaged people than it is to have a huge audience of people out there who never open or interact whatsoever.

Celisa Steele: [00:21:37] Let’s talk more about conversion because that’s a pain point for a lot of learning businesses. You can perhaps get traction on generating awareness or get someone to sign up for and then come to that free Webinar, but then getting them to move from that general awareness or that free offering to a paid one can be a bit of a hurdle. Amy, how are you all thinking about moving people from just aware and interested to actually enrolled?

Amy Michalski: [00:22:07] This is where intentional nurturing makes all the difference. Because not everyone who shows up, like you said, to a free Webinar is like, “Oh, yeah, sign me up,” and that’s okay. We’re happy to have them as part of our community because it helps us fulfill other goals. Even though they’re not ready today, that doesn’t mean they won’t be ready a couple months from now. Needs change. What we’re offering changes. Shelly touched on how we segment, and segmentation is key. Sometimes we have to make sure we’re announcing something like a new event to everybody, but we have the best opportunity when we can personalize and segment to specific audiences. We do this by topic, by engagement, by location. For example, by topic, if they’re interested in something like AI readiness versus more general information management or records management, we can share different messaging with those two audiences. Also by engagement level—this is the big one. For example, if we are trying to get some more people to join membership or take our certification, we’ll take a look at those that maybe were doing a lot of engagement with us in the past, have been a past member, have been looking at CIP opportunities. That will be our lowest-hanging fruit, so we’ll send them a personalized message, maybe give them a discount, and get some of those to convert into a sale.

Amy Michalski: [00:23:45] We just started re-offering regional events, and we’re very specific about the audience. We want to make sure that they’re local to that specific area, and we make it free to attend. A lot of times, what works well is ensuring that we’re sharing the other types of people that are going to be there because the key to joining AIIM is the community, the sense of community—like minds that can share ideas, get their answers heard, and get their business pain solved. One thing that also works is reducing risk. We emphasize the community side. I just mentioned these memberships. The membership for AIIM includes meet-ups, peer connections, cohort groups, book clubs, invitations to regional events. People don’t want to just consume content without engaging; they want to know there’s a group of people, like minds, trying to solve the same problems. I’m more focused on training partnerships. After having that consultative call, if one of our on‑demand training makes sense for them, I’ll offer those key decision-makers a trial to ensure the content quality fits and the content itself fits what they’re looking for before they commit. The key is meeting people where they are. Someone who just discovered us last week needs a very different type of conversation than somebody who has been consuming our content for over a year. So it’s less guessing and hoping it sticks and more using the data and having much better conversion rates.

Celisa Steele: [00:25:34] You mentioned in your response, Amy, the importance of nurturing and making sure to send some personalized message, making sure to follow up with that. You’re a staff of five, though, so talk about how things like those nurturing and personalized messages happen. I assume you’re relying some on technology to help with that process.

Amy Michalski: [00:25:54] Yes, and that’s something we always set up on workflows in our CRM, and we’re in the process of updating these. If, for example, they do this, then they’ll get this type of follow‑up. We’re working to intentionalize this even more so we can ensure that we’re reaching them at the right time with the right message versus just blasting and blasting. For example, if they just took the CIP practice exam, we want to make sure we follow up based on how they did. And, if they haven’t already purchased the exam, that’s a great time to touch base with them to see if they want to get additional training or if they’re ready to purchase the actual exam. Sometimes I’ll offer an incentive to do so.

Celisa Steele: [00:26:43] Shelly, I would love it if you could tell us about a specific marketing initiative, campaign, or experiment that you feel really delivered—something that you could point to and say, “Hey, that actually worked.”

Shelly Strickland: [00:26:58] One of the best campaigns that we’ve done at AIIM was something we put together on the fly after a big mandate that came down from the government on how things are being regulated in a certain industry. When that happened, we could see all the chatter that was happening around it and all the questions around it. So we pulled together a Webinar with the professionals that are having the pain points, and we did that in under a week. We didn’t have some big, polished campaign. We just moved quickly and built the Webinar around the issue. Then we brought in professionals that are (1) affected by it and (2) the ones that are having the problem with it, and then we put them all together in a room. We put on this Webinar within a matter of a couple of weeks, and it was so successful because it brought in a lot of people that we didn’t have in our database before. It took so much less marketing effort than most of our other campaigns. People were so invested in the topic that they came, and then the speakers that were speaking were so invested in it that they shared it with their networks. All these people were sharing it virally without us having to do anything. I think we might have made one social post on it, and all the rest of that advertising came from the people who were involved.

Shelly Strickland: [00:28:26] That was such a successful campaign, and it all came from listening to what was going on in the industry and the loudest voices in that industry. We had 400 people register in the first couple of weeks without us doing anything, and then about 90 percent of them showed up. If you hold Webinars, you know that your show rate is usually around 50 or 60 percent. But there was a 90-percent show rate, and those people are new people who had never heard of AIIM, so we gained about 300 new names from that. We’re always looking to replicate that. We do constantly see what’s going on and try to use those topics because we have weekly or monthly meetings that are just for members, and it’s usually one of those experts in the field who are speaking at those or leading those meetings about a hot topic that’s going on. That’s one of the ways that we do it. And then, when we can, if we have a Webinar that’s coming up, we will try to get those experts in there as much as we can to try to replicate that kind of success.

Celisa Steele: [00:29:42] Amy, I would like to ask you about what you’re seeing that’s changing in terms of how you market and sell the educational content that AIIM makes available. Are there perhaps things that used to work reliably in the past that are no longer working as well? And, on the flip side, are there some newer approaches or tactics that you’ve tried that have produced really good results, maybe better results than you expected?

Amy Michalski: [00:30:09] Before I talk about our tactics and how we had to shift in the way we sell, we also had to shift in what we offer. Because what we found is that nobody really wants it more. When we’re talking about educational content, we’re talking not only about training; we’re talking about events because content is everywhere, and that’s what we’re sharing with our audience. Nobody wants a PowerPoint lecture anymore. They want hands-on, interactive learning where they’re doing something and able to engage. We completely rebuilt our annual event last year around a concept that we call “intentional engagement.” This is more cohort‑based learning, workshops, interactive sessions, and lots of networking-type opportunities. Even our virtual training moved to workshop style with a sandbox environment. Let’s say you’re looking to learn more about Microsoft Purview. You’re not just listening to somebody talk about Purview; you’re clicking around, trying it yourself. People want practical how-to training and sessions, not just theoretical deep dives.

Amy Michalski: [00:31:32] The other shift is topic. We took our opportunities. We’ve been talking a lot about AI data readiness and specific tool training like M365, Purview, governance. Those are resonating with our audience. Once we were able to determine what the community really wants, it shifted our messaging to “Come to our opportunities. We’re not the same type of conference,” or “We’re not offering the same training.” That is our unique differentiator. It’s funny—we just had our annual event a couple of weeks ago, and so many people came up to me, saying, “I loved this event because I come in knowing some people in this audience,” because we have these cohort meetups with the attendees prior to even being on site. They feel connected, and they have some homework to do ahead of time to come up with their business issues that they’re trying to solve. They come there ready to ask these questions, and we give them ample opportunity to engage and get those questions specifically answered. It helped us be able to market because I can use those one-to-one e-mails, or, what Shelly’s doing, she can focus on the engagement piece for those events and workshops to share how we’re different. It’s incredibly important to keep up with what your audience wants and what works. We were struggling before we changed this model with our event, changed some operational things too, and now we’re seeing a lot more success. It’s working, which is exciting.

Celisa Steele: [00:33:19] That is exciting, and it sounds like you’re doubling down on community, the access to peers, and also the hands-on aspect and getting to do something. We know how important the social side is with good learning, as well as being able to apply it and try it out.

Amy Michalski: [00:33:38] Yes, and we’re also relying on our community. We used to try to do everything ourselves. We’re now working with our community to help. We like to say we’re the platform for education, and then our members can provide the actual education through partnerships or asking volunteers to create content for us. It’s a new model that we’ve never done, but, when you’re a team of five, you need to rely on your members. It’s great to have multiple voices sharing their expertise with their community because having many perspectives other than just what we think is what’s so valuable.

Celisa Steele: [00:34:17] Shelly, I want to get your take on advice that you might offer to another association learning business. Maybe this is one that feels a little bit uncertain or underpowered in its marketing and sales approach. Based on your years of experience in different industries and now at AIIM, where would you tell them to focus to try to get some better results?

Shelly Strickland: [00:34:42] I would tell them to stop underestimating the value that they already bring to the table. Associations have many of these commercial learning companies beat because they already have the trust, community, and direct access to the professionals, the practitioners who are living their challenges every single day. A lot of organizations feel underpowered because they think that they need big budgets or big teams. But, honestly, relevance matters a lot more than scale does right now. Associations seeing the best results are the ones that stay close to their audience and can respond quickly to what people are struggling with in the moment. I also would encourage them to stop marketing courses as products and start focusing on outcomes. Most people are not waking up thinking, “I want training.” They’re thinking, “I need help figuring this out,” and “I don’t know how this works,” and “I need somebody to help me figure out how it works.” The more directly you connect your education to their real-world problems, the stronger that engagement will be for them. That credibility is earned, and then your business will grow from that.

Shelly Strickland: [00:36:01] Another thing: You can’t wait for things to be perfect before you get started. Some of the most successful things that we’ve done were fast and practical, and they were built around what the industry needed. People and organizations pay attention when you show up with that timely message. They’re going to seek you out when you deliver answers for them. Finally, like Amy said about leaning into the community, if they lean into their community more intentionally, content becomes easier to find. People will look for those types of connections, shared experiences, and conversations with peers, which positions associations to be ahead of the other learning companies in that respect. They’re looking for people that are doing the same job as them that can help them do it the right way. And that credibility is what they need to focus on.

Celisa Steele: [00:37:01] I want to ask you both about how you approach your own lifelong learning. This is the Leading Learning Podcast, so we always like to talk to our guests about that. I would love to know some of your specific habits, practices, or sources that you use as you continue to grow professionally and personally. Amy, how about I ask that of you first?

Amy Michalski: [00:37:22] I feel ongoing learning is so important in the association world. I’m currently working toward my CAE, the Certified Association Executive, which has been a great way to formalize 25 years of working at an association. But my best learning happens in a community. I’m part of PAR, the Professionals for Association Revenue, and I serve on their leadership advisory board. Also, internally, I learn every day from our CEO, Tori Miller Liu, who’s very instilled in the association world, and other staff, board members, sponsors, training partners, and our own AIIM community. I love hearing different perspectives and learning how different people tell me different types of information, and it helps what we’re offering at AIIM and think of things differently of how we’re selling them.

Amy Michalski: [00:38:22] Some of my best growth comes from being willing to learn in public. For example, I use AI tools daily to manage this large revenue portfolio. I’ve presented at PAR. Even though I don’t know all the answers, and I was a little bit nervous the first time I did it. What I found is that, sometimes, the best conversations happen when you’re a little messy in the middle of learning alongside everyone else and admitting that I don’t know everything and learning, and I want to learn from you as well. Outside of work, my hobby is photography, and that helps keep my creativity sharp. And, at AIIM, I’m always looking for the right angle, the right framing, what to include, and what to cut. Kind of cheesy, but it does play a part. So I think that mindset directly helps with business development.

Celisa Steele: [00:39:16] Thank you, Amy. Shelly, how about you? How do you go about approaching your own lifelong learning?

Shelly Strickland: [00:39:21] I have the mindset of always be learning. I’m always willing to learn and adapt to new things that are out there. I love new technology. I love testing out new things. I think that, if you have a fear of changing, you fall behind so quickly. Personally, I’m a hands-on learner, and I do my best not by reading but getting my hands in there and getting dirty with things. I like to test things, break things, experiment, and see how they work in the real world. A lot of learning that I have comes from being willing to try something before I fully know how it’s going to work out. One of the mindsets that I have for learning—and I tell my sons that I homeschool all the time—the first thing to learning is the willingness to be able to learn. If they’re not willing to learn something, then they’re not going to be able to learn it. They have to set their minds that they’re willing, at least willing, and open to trying, and then all the rest of it comes. The ability comes later, but the willingness has to start. People need to be less afraid of failure. Some of the biggest learning things in my career have come from getting in there, and things didn’t work perfectly. You learn really quick what to do different the next time. You just can’t be afraid to ask questions, you can’t be afraid to fail, and you can’t be afraid to adjust your opinion of what you thought something was. Again, flexibility, change, and getting in there and doing it are the most important things to learning.

Jeff Cobb: [00:41:08] That wraps up the conversation with Amy Michalski and Shelly Strickland of AIIM, the Association for Intelligent Information Management. But stay with us another minute to catch our recap.

Celisa Steele: [00:41:18] If you found this episode valuable, we would be grateful if you’d share it so more people can find the show and benefit from the conversations we have.

Jeff Cobb: [00:41:38] Amy and Shelly returned to a central message throughout the conversation: Your prospective learners aren’t searching for training. They’re searching for help solving specific problems. The more directly you connect your education to real-world needs, the stronger your marketing will be.

Celisa Steele: [00:41:54] On measurement, Shelly made the case that engagement quality matters far more than vanity metrics, like open rates. A smaller audience of highly engaged people is worth far more than a large list that never interacts. And she made a point that I agree with wholeheartedly—no single metric tells the full story of any program.

Jeff Cobb: [00:42:15] Amy highlighted the power of intentional nurturing—meeting people where they are, segmenting by behavior and interest, and recognizing that someone who isn’t ready to buy today may be exactly the right customer a few months from now.

Celisa Steele: [00:42:29] Thanks again for listening—and see you next time on the Leading Learning Podcast!

To make sure you catch all future episodes, please subscribe on Apple Podcasts, Spotify, or wherever you listen to podcasts. Subscribing also gives us some data on the impact of the podcast.

We’d be grateful if you’d rate us on Apple Podcasts or wherever you listen. Reviews and ratings help the podcast show up when people search for content on leading a learning business.

Finally, follow us and share the word about Leading Learning. You can find us on LinkedIn.

The post Leading with Outcomes with Amy Michalski and Shelly Strickland of AIIM appeared first on Leading Learning.

When everyone has access to the same powerful tools and technologies, polished content becomes table stakes, and learning businesses that compete primarily on content will find it increasingly hard to stand out. So what do you compete on instead?

In this episode of the Leading Learning Podcast, co-host Jeff Cobb talks with Panos Siozos, CEO of LearnWorlds. They discuss why engagement remains the core unsolved challenge in digital learning, how we’re moving from a content economy to a trust economy, and what role AI should and shouldn’t play in a learning business. The conversation also covers the shift from courses to learning ecosystems and what business model innovation might look like in the years to come.

To tune in, listen below. To make sure you catch all future episodes, be sure to subscribe on Apple Podcasts, Spotify, or wherever you listen to podcasts.

Download a PDF transcript of this episode’s audio.

Celisa Steele: [00:00:03] If you want to grow the reach, revenue, and impact of your learning business, you’re in the right place. I’m Celisa Steele.

Jeff Cobb: [00:00:10] I’m Jeff Cobb, and this is the Leading Learning Podcast.

Jeff Cobb: [00:00:17] Content has never been easier to create. But, for learning businesses, that’s as much a threat as it is an opportunity because, when everyone can produce polished content quickly, content stops being a differentiator.

Celisa Steele: [00:00:29] Our guest in this episode, number 481, is Panos Siozos, co‑founder and CEO of LearnWorlds, whose platform he describes as a business engine for people to create and sell online courses. Panos has spent more than 25 years in the e‑learning space, beginning with his doctoral work in educational technology.

Jeff Cobb: [00:00:48] In our conversation, Panos traces what has and hasn’t changed in digital learning over the past 25 years and makes the case that we’re shifting from a content economy to a trust economy, where curation, credentialing, and community are the real differentiators.

Celisa Steele: [00:01:03] You and Panos also discuss how the most successful learning businesses are using AI—not to generate content in bulk but to scale engagement and personalization in ways that weren’t possible before AI.

Jeff Cobb: [00:01:16] Yes, and Panos shares his thinking on where learning businesses need to focus in the years ahead: moving from courses to learning ecosystems and rethinking business models to stay relevant in a fast‑changing market.

Celisa Steele: [00:01:30] Stick around for Panos’s view of where digital learning is headed and what it takes for learning businesses to stand out when everyone has access to the same powerful tools.

Jeff Cobb: [00:01:46] You actually started building a learning platform while you were working on your PhD in education or in learning technology. What was the focus of your PhD?

Panos Siozos: [00:01:56] It was educational technology. My particular focus was computer‑assisted assessment. My two co‑founders were working on other aspects of educational technology. Our first LMS was called EduTech2000. It used to be futuristic back then, but now I know that it sounds old. We were trying to create an all‑in‑one LMS to use in the academic space, but we started thinking about the potential—what this thing could become. These were quite early days for educational technology. Educational technology used to be always a revolution two, three, five years in the future but not always being there ready for prime time. So we continued building several e‑learning applications in this space as we were working on our postgraduate studies and in various research projects.

Jeff Cobb: [00:02:47] At the time that you were doing that, if you can remember back this far, were there things that you saw digital platforms just weren’t doing well that you wanted to address in developing a platform?

Panos Siozos: [00:03:00] Engagement is always the main thing. Back then, we didn’t have the tools that we have today. Online video was difficult to accommodate in great quality and high speed, but always engagement was the main thing. People were getting access to content, and, still today, we’re getting access to tons of content. But getting the engagement right, getting the completion rates right, getting people to really stick and engage with each other, with the content, with the instructors—that was always the main thing. And, as things progress, as we have gone through all this democratization of e-learning with access to great platforms, great tools, content now becoming a commodity, this still for me is the core issue—being able to create this engagement and have people really commit to what they’re doing and get not just access to the content but go through this transformational change that learning should be about.

Jeff Cobb: [00:04:03] Engagement is still out there as an issue. You still hear people talk about engagement an awful lot. That’s still a holy grail, so to speak. What would you say has changed since those early days that has improved substantially, that we have really gotten right that maybe we weren’t in the late ‘90s, early 2000s about digital learning?

Panos Siozos: [00:04:27] We have speed. We have all these tools that allow us to create, to launch courses, experiences, learning platforms at scale. Do you remember how it used to be five, definitely ten, years ago when you had to duct tape several tools just to be able to launch a platform or to launch a learning experience? Now the platforms are there. You can launch an online course in a couple of hours if you have the content. The tools are there. Definitely AI helps us with speed of creation, repurposing content, translating content, personalizing content. So this is really there. Also the scale is there now. Distribution allows us to, in some cases, create global always‑on experiences. Ten years ago, people were very happy if they were launching a course to 100, 500, or 1,000 people. Now we are able, at least in LearnWorlds, to power some online course platforms that have more than one million students. These are amazing numbers, things that we couldn’t really imagine when we were starting.

Panos Siozos: [00:05:36] And also the expectations have changed a lot. Now premium, high‑quality content is there. Choice is taken for granted. Everybody’s going for this Netflix-style experience, where you can just turn on your learning channel and have access to anything that you ever wanted to learn. These are the things that have changed. On the other hand, as we said, the core of what is learning is not the consumption of learning or the access to learning. It’s the interaction. It’s the engagement. Just because we have all these tools and all these affordances doesn’t mean that learning is easier. Learning is still difficult. You still need to engage with the content. You still need to sweat. It’s a muscle. It doesn’t happen just because you got access to some impressive platforms or to some great content out there. People still need the motivation. They still need to get the engagement right, to have their incentives at the right place. They still need the community, to strive together. I consider learning to be a team sport, a social endeavor, not just something that you do in isolation. These things remain the same, and I believe they will, even in the era of AI.

Jeff Cobb: [00:06:55] There are a lot of threads there that I want to pick up on. I’ll start with the whole idea that speed and scale, just getting content online has become so much easier than it used to be. I worked for an e‑learning company back in the mid ‘90s, and it was a big deal to put together a course and get it online. Now you can do it in a few minutes. But a lot of organizations are now confusing putting content online with having a thriving digital learning business. What do you see the most successful organizations doing that the less successful ones aren’t? We tend to talk in terms of three terms here at Leading Learning: reach, revenue, and impact. When you think about learning businesses that are achieving reach, reaching their learners, generating revenue, having impact, what are they doing that the less successful ones don’t?

Panos Siozos: [00:07:46] The smarter businesses and the ones that eventually succeed are the ones that are fast. They’re shipping content fast. They’re not perfectionists, sitting on a course for six months until they get it absolutely perfect. What they do is they ship it quickly, even if it’s not perfect, but then they iterate constantly. They learn from the market, from the learners, from their audience. They stay close to customers, to their learners, in order to be able to learn. So it cannot just be a broadcast-type experience. You need to maintain these relationships. And, at the end of the day, they manage to build ecosystems. You call them communities. You’ve talked a lot about building communities around the courses. They manage to build communities around their content because content is just one part. It’s content; it’s a community—the people who are there, staying close and engaged with your customers, offering them support so that you can still learn from them in order to adapt your courses and to create the next thing and the next iteration. It’s not one release once a year. It’s a constant iterative process of launching, tweaking, and being close to your customers. Like every other digital product or software, you have to be there. You have to constantly keep learning, have these feedback loops, and continue to improve your offering.

Jeff Cobb: [00:09:16] I’ve watched some of the big MOOC providers, like Coursera and edX. Back in their original days of the massive open online courses (MOOCs), they were very community-driven. It really was people coming together, and the community almost preceded the content in many ways and often led to the content. The reverse seems to have happened as those companies have gotten big. They got a whole bunch of content, but the community has dwindled and dwindled and dwindled—it’s just not there anymore. Is that inevitable? Or do you see paths for successful learning businesses to maintain engagement, to maintain community even as they scale and reach more and more learners?

Panos Siozos: [00:09:55] A great observation. It is difficult when you grow. But, for me, this is the ultimate use case for something like AI that allows you to massively personalize and massively make/provide very specific feedback that otherwise would have been lost in a massive community. For communities, there is a critical mass and a Goldilocks zone where they work. If they’re too small, they can’t work. If they’re too big, it’s not a community anymore; it’s an agora, something that is huge. That’s where you need to break things—segment, create these communities that feel personal, that feel like safe spaces, that have the right volume, that also bring together people with complementary needs and complementary skills, and where you can offer to them the personalized feedback, create the interactivity, the interactive learning that they need.

Panos Siozos: [00:10:55] Something like this happens in a traditional classroom. It’s something that a traditional teacher can do when they have a classroom of 30 people. You cannot do that with 300 people, and definitely you cannot do that with 300,000 people. But that’s where tools like AI can help us create content and engagement in an efficient way. Engagement is what will scale the results at the end because, if we just use AI to scale content, we’re just creating, adding more slop to the planet, and we’re adding things there that cannot be consumed on their own. But, if we are able to use AI to scale engagement, that’s where we can bring in the results, and that’s where we can maintain the safe space with the right people and engage with the right kind of content, and I can drive forward with my skills and cover the needs of myself or my organization.

Jeff Cobb: [00:11:50] You may not have visibility into this—and feel free to say it if you don’t—but, when you see organizations implement your platform, they’ve obviously got people who are creating courses, deploying the courses, tracking the data, and that sort of thing. Do they tend to have somebody who is focused on, responsible for community, for engagement when they deploy the platform?

Panos Siozos: [00:12:13] Not always, depending on where they come from. If they approach it with a content angle, they feel that, if they just publish the best possible content out there, it will become a flywheel on its own, and it will fly. We try to convince them that they cannot afford to forget this. This is what will create the feedback loop: investing in the community. Which is difficult. We know that it needs effort. You need to reach that critical mass that we were saying. But content is not sufficient, especially now. At some point, AI might take this over and provide this experience. But, for now, it’s not there. You need the people in the middle of that to provide this kind of support, experience, and engagement.

Jeff Cobb: [00:13:06] You’ve got a number of AI tools integrated into your platform at this point, and we’re seeing that with a lot of platforms now. There are freestanding tools that help you with all sorts of different versions of productivity, mainly. Where are you seeing AI have a positive impact, where organizations have figured out how to use it effectively—not just to turn out more content but to do things that do impact their learning businesses? And where are they still struggling, from your perspective, in figuring out how AI can be productive for them?

Panos Siozos: [00:13:44] First of all, LearnWorlds, from the very beginning, was introducing AI as a support mechanism, not just outsource things to AI, prompt something, and then let AI create courses or anything else. We don’t think that it’s ready for that or that it should be used for something like this. We try to prompt and to have AI on the side like an instructional design assistant, somebody who can help you not just create a bunch of content but the right content for the right interaction for the right activity, with this kind of instructional design mindset. What we see from successful organizations is that AI can have a strong impact on content creation, accelerate content creation. This can be very important in our day because things are moving so fast. The half-life of content is shrinking. Sometimes you need to be able to react to market needs very, very fast. You need to bring a course to the market very quickly.

Panos Siozos: [00:14:45] Software is changing so fast. Products are changing so fast. Processes are changing. Sometimes this can also be extremely important—repurposing the content, iterating fast on the content, and getting all this feedback from thousands of users and trying to create the next version. AI can help you digest all this and prioritize and help you design the next version, translating in some cases, or creating these personalized learning flows that would have been impossible in a high-scale environment. Just to give you one example from LearnWorlds’ experience, where you might create a 100-question multiple-choice test with questions of varying levels of difficulty (based on Bloom’s taxonomy), and then you have 1,000 people going through that exam. How can you provide personalized feedback to all these tens of thousands of combinations of answers without having something—AI—that can go in and tell you precisely where you did wrong and where the right answer in the text can be found or something like that? These personalized learning flows can only happen with AI. We see lots of potential there.

Panos Siozos: [00:15:56] Organizations that understand how to use that, they can get a significant advantage. Where we see lots of hype and people thinking that they can outsource learning, that they can get fully autonomous learning just by using a few prompts, that they can replace instructional design by doing some smart prompting—that’s not there. And especially for the audience here that is very much focused on continuing professional education and certifications and all this stuff, AI cannot provide the validation of learning at this moment. The trust that is needed, the man in the middle, validating all the knowledge and making sure that it’s up to standards—that’s something that cannot happen at this point. We still believe that, obviously, humans are needed. So AI can be a powerful assistant, but it cannot be a learning system right now. And not for years to come.

Jeff Cobb: [00:17:07] There’s the old saying—William Gibson said it—“The future is already here—it’s just not evenly distributed.” We’re moving pretty quickly to the point where everybody will have much more comfort with AI tools, just like everybody can use Zoom now when they hardly knew what it was five, ten years ago. So we’ll get to that point where we have smart people with very smart tools—AI—making very smart content, and you can see, in any given profession, industry, whatever, the competition could get pretty fierce around content, high-quality content that’s out there. What do you think? If you get to that point—because a lot of our listeners are probably going to find themselves there, if they’re not there already—how do you stand out in that kind of market as a digital learning provider?

Panos Siozos: [00:17:53] You have to move to higher levels of value. We’ve seen similar things happening in the past. Ten, fifteen years ago, people were saying, “Who is ever going to create online courses when we have the MOOCs and the best professors in the world teaching about marketing or teaching about everything else?” And then we had YouTube. How can you compete with that? People were saying, “You just go to YouTube. You will find a video about everything.” But all these things, what were they missing? They were missing the curation. With all these massive amounts of high‑quality content, how do you know that you know the things that you should know? How do you know what’s the path that you should follow in order to become an accomplished professional in your field? How can you gain trust in your knowledge? How can the people who hire you trust that you have the skills? Who can assess this knowledge?

Panos Siozos: [00:18:50] And also the community part that we’re saying. It’s not just one person on their own accessing a bunch of knowledge. Is there, somewhere, a professional community of like-minded people that go through the same learning experience like me—who can be my coworkers, who can be my competitors, who can be the people who will motivate them? Who is the company who will manage to create that kind of community? They will have a huge advantage. Everybody will have the content, but what about the trust, the curation, the personal brand, the professional brand, the community of people that will be there? And also proving to your audience and to your students that the learning outcomes can be created through your content, your experience, and all the things that are out there.

Panos Siozos: [00:19:41] This is where things will move. Content becoming a commodity, as we said. The half-life of content becoming increasingly smaller. We’re moving from an attention economy and a content economy to a trust economy, where you will try to find who are the experts in this field. In some cases, it might be the universities, corporations, big brands, or the people who are extremely knowledgeable in their field. They’re moving very, very fast. They’re producing top-line content, but also they have created this mode that’s not just content but also the curation, the trust, the presence, the support, the community around them. I understand that this might sound a bit vague, but the validation and the trust are going to be increasingly very, very important in this kind of ever‑present, high‑quality content.

Jeff Cobb: [00:20:42] Yes, multiple parts to it. It used to be, in our world, that, if you’re a certain type of provider, you could just crank out the Webinars, record somebody doing their presentation, keep putting those out. And that’s declined significantly as a way to succeed, at least in the marketplace. Are you seeing your customers or other organizations you’re aware of doing more with microcredentialing, other forms of validation, as you said? You talked about trust—you’ve got to be able to verify—that can certainly help with that.

Panos Siozos: [00:21:14] We see more and more of that. And again, because content is becoming a commodity, so many more things to learn about, a massive certification is not scalable. It might not be relevant. We see microlearning, microresources, microcredentials—all these things where people can mix and match and create their own learning paths and gather the things and the signals that they need for themselves or for their employees, for their organizations, for their associations. These are the things that we are seeing out there. Learning not just continues but becomes ever-present, omnipresent; it has to happen at every single hour of the day. But also learning is increasingly not just the consumption of content but some kind of trust signal coming along with that content. Microcredentials, all these things are becoming the norm. We see that, and that’s something also that we facilitate through LearnWorlds. We see the most successful course providers doing things like that.

Jeff Cobb: [00:22:27] When you look out three to five years—to the extent that anybody can see out that far—as you see learning, learning technology evolving, what sorts of opportunities have you most excited? Where do you think somebody who’s leading a learning business really needs to be focusing their attention to make sure that, three to five years out, they’re still going to be relevant, they’re still going to be thriving as a learning business?

Panos Siozos: [00:22:51] As they say, making predictions is difficult, especially about the future. I’m trying to read lots of science fiction, not just Gibson but everybody else, to try to open my mind about what might be possible out there. Five years might be too much. Let’s say in about the next three years. As you were saying, content is becoming a commodity. Now we are moving from courses to learning ecosystems, which are not just the content but also the community. It’s also the trust that comes along with that and the things that an organization needs to build. It’s thinking more about the learning experience and how this translates into different environments and how realistic this can be because I also believe that realistic learning is also going to be a big part of that. Enabling these learning experiences in situ, where learning happens. All organizations will have to think more about not having learning as an isolated path, an isolated instance, or something that happens once a year, once every few months. Learning has to be always on, 24/7. You will need access to this very realistic microlearning experience, microcertificates, or signals of trust out there that will allow you to accomplish your goals/objectives anytime they’re needed.

Panos Siozos: [00:24:29] The future isn’t going to be about more content—we already have that—or faster content or better courses or more courses. It’s about learning ecosystems that are organic, iterative, fast, and adapting to the needs of the learners. It might sound a bit abstract, but we see that that’s the combination. Already learning businesses should start thinking this way. It’s no longer about having the right learning management system. It’s no longer about having high-quality, polished courses. Everybody will have that. It’s about how you build your own learning ecosystem, how you invent your own business model that will allow you to stay at the forefront of innovation. We see out there a huge amount of innovation in the business models that people employ: how they create content, how they distribute content, how learning becomes not just a medium but, in some cases, a huge catalyst for their businesses, whatever they might be doing. I think these are some lines of thought for our audience here.

Jeff Cobb: [00:25:44] In terms of business model innovation, is it subscriptions? Is it customer training related to other product offerings? What sorts of things do you have in mind?

Panos Siozos: [00:25:55] It’s already beyond that. We’ve seen in the past years, and I’m sure you followed all this, going from the individual courses to the subscription plans to Netflix and the SaaS-ification of online courses, where you have all these different subscription models. But we are seeing even more things. For me, education is the best form of marketing that you can do for all sorts of businesses. We are seeing consulting businesses that start to add learning on the side as a byproduct. We see learning businesses that become consulting businesses. Customer education is increasingly becoming a huge factor, whether you’re selling software or a service or a product or anything else. Training customers also means retaining customers. We see things that people didn’t consider as standalone businesses or being relevant to learning. Now learning is relevant to everything. It’s all around the things that we do. We’re seeing lots of new ideas coming out from out there.

Jeff Cobb: [00:27:04] We always like to ask guests—particularly somebody like yourself who has the academic intellectual background in learning, but you’re also an entrepreneur, and you have to keep up and evolve and build a company day in and day out—how do you approach your own learning? What are some of the things you do to keep you on top of your game?

Panos Siozos: [00:27:24] In LearnWorlds, our motto is “getting better every single day,” and we’re trying to stay very humble—I guess that’s the academic background. We know that we know nothing, as Socrates, the Greek philosopher, used to say. We are eating our own dog food. We’re trying to stay close to academic conferences and business conferences, reading papers, anything that has to do with instructional design, especially now with how AI is transforming instructional design. These are things that we are always trying to keep very close to us. Not all of this academic input can be transposed into a business environment, but it’s very useful for thinking and for understanding how things might evolve out there. We’re trying to get the academic input but also to ground it on the real needs of our customers. Half of the input definitely comes from our customers that are out there. LearnWorlds now has more than 13,000 customers—a huge amount of input in the things that they do, how they evolve, the challenges they’re facing every day. how they’re using AI. Who is the competition that they’re facing? Because they’re not just competing against the other providers that are selling online courses. In some cases, they’re competing directly against the LLMs that their potential customers might be using. So we’re trying to use all this input and make sense of it and understand where the market is going; how, with our own platform, we can offer solutions to these customers; and what are the things that we should be building.

Celisa Steele: [00:29:07] That’s the end of our conversation with Panos Siozos, CEO of LearnWorlds. But stay with us another moment to catch our recap.

Jeff Cobb: [00:29:14] You can find Panos Siozos on LinkedIn and learn more at LearnWorlds, where you can explore the platform and start a free trial.

Celisa Steele: [00:29:28] If you find the Leading Learning podcast valuable, we would be grateful if you’d spread the word to others who may benefit.

Jeff Cobb: [00:29:35] Panos opened by reflecting on what’s changed—and what hasn’t—in digital learning over the last 25 years. The tools are faster and more powerful than ever, but engagement remains the core challenge: getting learners to truly commit, not just access content.

Celisa Steele: [00:29:52] Panos said we’re moving from a content economy to a trust economy, and that certainly jibes with our view and our own focus on Credibility CapitalTM. What sets successful learning businesses apart is curation, credentialing, community, and the trust that comes with them.

Jeff Cobb: [00:30:09] On AI specifically, Panos drew an important distinction. Using AI to scale content is just adding more noise. The real opportunity is using AI to scale engagement—personalized feedback, adaptive learning flows, and experiences that feel human even at massive scale.

Celisa Steele: [00:30:26] And, looking ahead, Panos sees the future belonging not to courses but to learning ecosystems—always-on, iterative, and built around trust signals like microcredentials.

Jeff Cobb: [00:30:37] Thanks again for listening—and see you next time on the Leading Learning Podcast!

To make sure you catch all future episodes, please subscribe on Apple Podcasts, Spotify, or wherever you listen to podcasts. Subscribing also gives us some data on the impact of the podcast.

We’d be grateful if you’d rate us on Apple Podcasts or wherever you listen. Reviews and ratings help the podcast show up when people search for content on leading a learning business.

Finally, follow us and share the word about Leading Learning. You can find us on LinkedIn.

The post From Content Economy to Trust Economy with Panos Siozos appeared first on Leading Learning.