Dollars to Donuts

Today we’ve got a quick program note about new episodes of Dollars to Donuts, an announcement about my new book, and an interview with Steve Portigal.

Steve Portigal: Hi, everyone, it’s Steve Portigal. The podcast has been sitting quietly for a little while now, I guess, but I’m back today with two quick announcements and a bonus.

First of all, new episodes are coming. I don’t know exactly when but the wheels are in motion. I’m looking forward to some great conversations about leading user research, and I’m excited to share those with you.

Second, I have a new book just about to come out. I wrote Interviewing Users in 2013, and it’s become a classic text about user research. Now it’s 2023, 10 years later, and lots of things have changed. So I’ve updated it! The book – Interviewing Users, second edition, will be published on October 17, 2023, and is available NOW for pre-order from Rosenfeld Media, at a 15% discount.

The bonus is my conversation with Lou Rosenfeld, where we talked broadly about user research, as well as the second edition of Interviewing Users. This interview originally appeared on the Rosenfeld Review Podcast. So welcomed by the sound of the trumpeting elephant, we’ll go there now, and I’ll see you back here before too long with more episodes of Dollars to Donuts.

This episode of Dollars to Donuts features my interview with Danielle Smith, the Senior Director of Experience Research & Accessibility at Express Scripts.

Something that I’ve really changed the way I thought about since I’ve been at Express Scripts — we are in the healthcare ecosystem. So the experiences we deliver, if they are not of quality, they do have serious repercussions on people’s lives and people’s time. We are ethically bound to measure the user experience from different perspectives. Before something launches. We have prototypes or concepts or ideas, we do our due diligence in terms of user experience research, to make sure that the thing that we’re putting out on the world doesn’t just happen to people. – Danielle Smith

Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.

Steve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.

Many years after we moved into our house, we finally hung up our art. Sure, we had hung up individual pieces, something that was already in a frame, say. But it was always piecemeal, a nail here, a hook there. And we continued to accumulate meaningful pieces from travels or family events. And we continued to occasionally pull out the box (a moving box, made for a mirror, I believe) and spread out all the various bits and pieces and just generally fantasize about having them up.

My goal, however, was to have a plan, an intentional way of placing the different posters, prints, photos throughout our home. Every time we would try that, I would get overwhelmed and give up. I tried taking small bites: after seeing homes with big frames and medium photographs, we chose a few photographs from our travels, blew them up, ordered specially cut mat boards, and frames. We mapped out where in our living room these would go; we essentially carved off part of the home and planned the photographs that would go there. I hoped that this would simplify the challenge of where to put the remaining posters, but we found ourselves stuck, still.

Eventually, we opened up that box and made some hard decisions about what to hang and what to set aside, and then – before too much time could go by – arranged to get everything framed. We were inching closer, but sitting with our posters and prints, all framed, we still couldn’t figure out what to hang where and so (this shouldn’t be surprising) I got overwhelmed and gave up.

But buried in that frustration and surrender was a recognition of what skills I’m missing – an ability to reorganize visual information spatially in a few different ways, a set of starting principles for grouping, placing, and so on. Surely someone must have this expertise and be offering it as a service?

It turns out that, yes, there are professional picture-hangers. We called one and booked an appointment. A few weeks later, on the scheduled day, at the expected time, the doorbell rang. We answered the door and before we could finish greeting them, two men were in our foyer, one of them having magically unfolded and gently placed a table (please assume it transformed from a flat briefcase shape to a table with a soft whoop noise).

The guy with the table handled production. The other one handled creative (and the clients). We showed him all of our framed pieces and talked a very little about what they meant, and we showed him around our house, pointing out areas we were interested in and the few pieces we had already hung. Meanwhile he was riffing constantly, throwing out ideas, getting energized, and delegating to the production guy, who began attaching extra mounting hooks and hanging wire to all of our store bought frames. After a short time, we backed off, and we watched them work.

The “creative” began moving frames into different rooms, laying them on the floor and trying different combinations. Like a problem-solving algorithm, the solution began to appear, bit by bit. The floors throughout our house began to fill with clusters and arrangements of different prints, both thematic and visual groupings. We were called in for frequent consultations as the plan emerged. Eventually there was a plan for where everything was going to go. This was the piece we could not have done ourselves and in a short time, they had done it.

Then came the rest of the production. They began hanging up everything. That meant figuring out where each item went – exactly, putting in nails in exactly the right spot, using a level, all the details. Especially because so many pictures were clustered, something being slightly off would really show, so perfect execution was key. This was also something we could have not pulled off ourselves.

The final results were astonishing. Rather than hanging things in a grid, with the top edges aligned and a consistent space between each, they put together a number of clusters where the posters emanated from a central point in an almost-spiral flow. And they chose how to place different prints within that cluster in order to create a kinetic sense, such as having a poster with a bird along the left hand side, with the bird facing to the right, so that the content of the images supported the physical placement on the wall. This was not something we could have even imagined, let alone executed.

It is immensely gratifying to be in the presence of a highly-skilled individual. When those skills are being deployed for your benefit, it adds another layer of delight. I believe that delight is further enhanced when we ourselves have tried and failed. This story is a reminder to me to seek out these magically-talented individuals and take advantage of what they have to offer, whenever I can.

And it’s my goal to be a magically talented individual for the people I work with, someone who brings in a level of skill that isn’t achievable without my help. I strive to conduct user research with that kind of finesse, I hope that when I coach and train teams in doing research, I’m helping them see what that magical level of skill looks like and move them forward on their own path towards achieving that.

I would love to hear from you about what you are working on and how my expertise can support you in moving your effort forward. Please keep me in mind.

Now let’s go to my interview with Danielle Smith. She’s the Senior Director of Experience Research & Accessibility at Express Scripts.

Steve: Well, Danielle, welcome to Dollars to Donuts. Thanks so much for being on the podcast.

Danielle Smith: Thanks for having me.

Steve: Why don’t we start with an introduction from you to say a little bit about what you do where you work.

Danielle: My name is Danielle Smith. I am Senior Director of experience research and accessibility at Express script. And Express Scripts is actually Cigna company. But we are a pharmacy benefit provider. So we help with getting your prescription drugs. If you have insurance coverage, we are the companies that help with that. And my team measures the quality of the user experience from a few different perspectives. One includes user experience research. The other is our NPS program for our digitally engaged customers, and our data science team and some digital metric reporting, oh resides with my team.

Steve: So I want to ask about your team. But I want to just back up slightly, who are the kinds of people that are having this digital experience with your company.

Danielle: So our digital experience is primarily geared towards people who have prescription drug coverage through their health insurance provider or directly if you’re on Medicare Part D, but you are interested in getting home delivery of your maintenance medications. Our website does have kind of general information. If you do have a health plan, you want to go in there and look up a medicine see how much it costs which pharmacy But most of the functionality on our digital tools for consumers really look at the home delivery of medicine,

Steve: and then use the phrase to kind of describe what your team does. It’s about measuring quality. Is that right?

Danielle: Yeah,

that’s sort of the way I think of it.

Steve: Can you pick apart that phrase a little bit? It’s an interesting one. And I’m curious how you think about

Danielle: something that I’ve really thought about or changed the way I thought about since I’ve been at Express Scripts, we are in the healthcare ecosystem. So the experiences we deliver if they are not, if they are poor, or if they are not of quality, they do have serious repercussions on people’s lives some people’s time. So when I think about my team, we are almost ethically bound to measure the user experience from different perspectives. Before something launches. We have prototypes or concepts or ideas, we do our due diligence in terms of user experience research, to make sure that the thing that we’re putting out on the world doesn’t just happen to people. We have some ideas It’s usability, its appropriateness. And we’ve really tried to test a test and air quotes, do research on everything before it hits the site. And some cases, of course, we can’t do that. And I’ll talk about that in a second. But we’ve tried to do, I would say far above 80%, of what hits the site goes through some level of user research related to that. And another aspect of what we do to ensure quality is to make sure it’s accessible. And most of our accessibility work has really geared towards visual impairments or different visual abilities, because we are the digital team. So my team serves as subject matter experts to help our developers aren’t marketers, and to deliver experiences that are usable by people that use screen readers or have low vision just right out of the gate. So we don’t have a separate experience for folks who may be having any sort of different visual abilities. And that was a big point of focus for the first few years of my role here was to just really get that going. But it’s part of our user experience, practice and it’s in our The research team because we do specifically do studies, blind and low vision users throughout the year, to make sure even though it’s something’s technically compliant, it is actually usable by folks that use a screen reader specifically. But following good design practice, and rolling in those things before you launch still does not guarantee good experiences. And so the other parts of my team work to measure how what we build interacts with reality, and to see if it does actually generate a good experience and has, you know, high quality like we want. So the NPS program I know NPS itself is a problematic metric and has its taters in the industry. And that’s fine. I’m not here to defend it. But what it does is gives it lets us speak in a voice that executives and understand and it gives us the leeway to have an open channel with our users. So we send out these monthly surveys for one part of the program. But another part of the program allows users to leave feedback directly in our mobile app on our website. Tell us what’s going on. So yeah, that gave us a score, report the score, executive lift score. But the words that they use when they tell us if we’re on the mark or off the mark have been priceless and making sure that things actually work as intended given reality. And then the other piece is behavioral analytics. So we run a B tests, we instrument the site and make sure that things are working the way we think they should work in terms of people flowing through different funnels or parts of the site and monitor the experience that way. So we work really closely with our product teams to help them understand their metrics, make sure they’re gathering metrics and help them use them and interpret them in the right ways. So when you use the word quality, what does that mean for you? So for me, I’m using quality to be synonymous with a good user experience and healthcare so complicated. I’m not going to be so arrogant as to say we’re going to delight people. I just want them to be able to get their task accomplished which is getting their drugs prescriptions delivered. or checking the status of their delivery. I want them to get their tasks accomplished with ease. And that’s what I mean by quality. Are we not getting in their way? Do we make them feel like they’re part of the process? And do they understand what’s going on? Because it’s complicated enough.

Steve: You also use the phrase, I hope I get this right. working as intended given reality.

Danielle: Oh, yeah. Reality when you’re talking about healthcare is something that we cannot ignore. And we can’t we can’t even formulate all of the scenarios a person might be in at least you can’t do that yet. And people’s realities may be like right now like, Okay, well, now my doctor’s office is closed, and I need a refill. How do I do that? It’s a pandemic, what’s going on. So we have to be making sure that we are listening to user feedback to make sure we update the website, and even communications and talk to our friends in the call center to make sure that we are ready to respond to our users. reality, smaller, like everyday situations happen. As you can imagine, if you’re sick, there are some conditions that give you kind of a transient low vision situation. So you use the website yesterday, I, my team wants to make sure you can use website tomorrow. Just because you’re on this medication or you have this condition should not take away your ability to use certain tools. We also have lots of families, family situations are fluid who has access to your account today, you might not want them to have access to your account tomorrow. And this is your health information. So we have to be always ready to listen and react when people’s contexts change, or their actual reality is way more complicated than we even thought of when we were designing the user interface.

Steve: Can you say a little more now about the word measure is kind of the key verb that you use to describe kind of overall what you all are doing.

Danielle: Sure, and I’m glad you picked up on that because I don’t want to give people the wrong impression that Don’t do discovery or you know the call, but I am of the belief that the work that we do is still measurement. We may not have a type metric, we might not have a number around it, but it is still the collecting of data to get at some underlying construct as best as we can. And I was recently reminded that I came from a grad program that was more quiet leaning. And it’s probably why I did grow my team, the way that it grows more, including analytics, as well as user research and having them live side by side. So it’s not just about things we can put a number one it is about understanding scenarios and understanding people and listening to how we can resonate with our users and more. So I don’t want to miss categorize what we do, but I think of measure in the broad sense of the word.

Steve: I mean, it’s a really powerful frame for what research does and there are lots of different frames for you know, if you have to say one word, what is it that we do and measure is One that I have heard all that often, but it’s very compelling, especially when you kind of explain, hey, there’s lots of different kinds of measurements. And even when you’re talking before about NPS, and part of its value as a way to open up a conversation with other people in the organization, when you say measure overall, I imagine that that is similar that by framing this around measurement, you are positioning yourself relative to your stakeholders and colleagues that this is the kind of guidance and information that you can bring.

Danielle: Yeah, and I talked about it that way I talked about what my team does is x data sounds very superhero like that. It’s like it’s data about the user experience. And just because I say data does not mean it’s all quantitative, and it’s taken a while for our partners and even some folks on my team to get on board with this idea. But we need to be able to have some sort of convergence on the user experience. And convergent validity is a goal of most research teams that I’ve been exposed to where you want to have some analytics show problems and user research give you the other side of it, maybe some survey data to give you some aspect of scale of some of those opinions. And by us all being on the same team, it gives us that ability to be fluid in our methods and speak to the business in a way that allows them to hear us. And sometimes that way is to use NPS or a web analytic metric to get our foot in the door and expand how they view the users by layering on the qualitative and sometimes it’s vice versa. We have different stakeholders that are more interested in in qualitative data and then we overlay some quants to help them understand scale and focus and or if it’s even able to get to that level of maturity given where we are in the process.

Steve: Accessibility is part of the group and for me it’s new to hear accessibility specifically called out, alongside other different roles. How did accessibility end up being part of what you’re focused on.

Danielle: Oh man, it’s hilarious he was just like a line in my job description analysis like that. Okay, and I thought what that meant was, this is literally my first week on the job I thought that would be similar to my accessibility role in other parts of my other jobs I’ve had before, which was its usability person I need to be aware accessibility exists, and I enrolled in some tests in my normal course of evaluation that help the accessibility team, check their boxes, about compliance. However, I did not do my due diligence about looking at the expressor sweb site to understand what that might in my job description actually meant. Just say like at previous jobs at Dell and larger companies have specific hardware interface requirements that come from the accessibility team that kind of get handed to the usability team that we just make sure as we go through development, make sure that they get rolled in, but what I did not notice and Express Scripts with that. Back when I joined the company in 2016. They had two different websites, there was a link on the top of the site that said accessible view. And it took you to a text only view of the site, and during my first week on the job My boss is explaining this to me and how, nobody really wanted that to be the strategy and the teacher. I was like, Oh, so we need to learn how to build accessible websites, she’s like yeah, like, oh, okay, well, let me figure out how to do that. So it was a whole different a different perspective on the same problem, but it gave me that opportunity for us to say that we’re going to build a website that works for folks of different abilities, without having a separate site that was personally offensive to me as a Black person to have something separate for people who are different, really annoyed me. So, I wanted to make that go away. And having more inclusive web experience. So that became part of our world, because at the time Express skip splits really open to having a much more modern digital experience, we came up with a plan to help spread that awareness spread that skillset across the organization, one person on my team really picked up the mantle and dove into accessibility, best practices, compliance. She’s not a lawyer but building relationships across the organization to have that compliance and a network, and then diving into doing research with totally different abilities like specifically people they use screen readers because we wanted to make sure that not only did it kind of pass all the checks, but that it, like I said, was usable, and bring that video and bring those usability results into the organization, so that they can understand what it meant to be inclusive in that way. And that just really fit with the way we do user research, because the whole point. A lot of the point of user research is to bring the voice of the user outside of our walls into our walls to help people inside the company, understand the operating environment of people outside of the company. And by having a focus on accessibility within user research allows us to apply those same principles same practices that folks were kind of used to, but with a different audience, and it just really worked well. And so, I can’t remember the date but it was at least two years ago, we sunset it that outside, we have one experience, and we have a group of folks within the organization that are really passionate about this developers help each other out. Under normal situations we would actually run usability testing on site at the American Council of the Blind. To get more engagement from the community, it actually turned into a natural extension of the usability team meeting over it’s showing my age but user experience team, because, as designers, as researchers, we are responsible for inclusivity and accessibility is one part of that, and in health care, especially cannot, an all good consciousness exclude that important of a population. And so we do have focus on that and we have it in the name of the group, we want to remind people that this is who we are, what we do, and now we have this culture in my work that I almost don’t have to, but I probably sit

Steve: right there, designers who are specifically focused on designing the accessible experiences.

Danielle: Now, every designer is responsible for it. So, we had, like I said, this me and my team helped to make sure that add new designers have gone through whatever training modules that are we have available designed for accessibility is rolled into our design language, and it is part of the core competency of every designer, the content team of the development team to just build it into everything we do. Can you talk a little bit about some of the ways that your team I know your team covers a lot of different functions, but how you work with other parts of the organization. So Express Scripts senior healthcare org it’s a little different than a lot of the technology companies I’ve worked with before the design team, lives in it. So there is one big group that handles all of the external and internal facing technologies, and that’s a little different from what I was used to, but it works out really well here we work very closely with our technical product owners and partnerships within the business.

Steve: How do you plan for what it is that people on your team are going to be working on.

Danielle: So we are aligned to what the technology or so, I’d say that once a year, we hire on our big buckets of work so we know we are going to work on the website, on the site of member experience for these of students who work on the website, we’re going to work on the mobile application, and we’re going to work on, maybe one or two other properties or different experiences. And so, once we’ve aligned on that across the product team, the design team, and the front end engineering team, then we each kind of go off to our own teams and look at how much work can we actually do. So, this is an ongoing area of maturity for us as a leader of people. It’s something I’ve had to learn to take different approaches to doing, and I noticed wasn’t really your question but I’m going to talk about anyway. Go for it. Easy research design content, kind of a big deal of design, we’re all in this field because we have this level of empathy for folks to believe we want to design things that make the world better. And what I admired as a leader of people it’s, who has to manage like resources and organizational requests and put together budgets plants and stuff, it’s hard to figure out how much work we actually can do when no one on the team wants to say no to anything. And, as a leader I’ve had to figure out different ways of training manager, it’s been very illuminating because I’m sure I was this person when I was like fresh out of school to, kind of, I can do it, I can do all this stuff, and then later I’ve my work quality suffers turns out I’m working on the weekends and at night. I might start to resent the work I’m doing, and if I just raised my hand and said hey I know I said I could do all these, all these things but turns out I can only do one of them, my life the good life. So, as a leader of people I try been working really hard closely with other leaders on my team, you know they’re people on the team to help understand how much work can be done in a given situation. To help evangelize a message of how much more impact we can do if we just focus on one or two things that we know we can achieve and say no politely, but start to learn to build the muscle of No, that’s been my own personal growth area as a leader, and as a individual I’m guilty of that too, like I said, but trying to plan the work and where to focus becomes easier as you learn as a UX professional to accept things that you can do, and be cognizant and aware of what you can’t do and communicate that to your, your leadership your peers, so that everyone knows what’s possible.

Steve: You reframe for me when you say become scribing No, as I can’t do it not know as I refuse to do it right, there’s a human limitation that we are always taking into account,

Danielle: right, there’s a human limitation to everything. And we know that our takes time. I mean, everyone’s work takes time, but if you’re going to do a good job. I mean, you know, as a consultant, it’s hard to tell a client No, but you’re not really telling them no or cheese you’re telling them. Well, I could do that, but it would suck.

If you want to do it well. It’s gonna take this long, or we can do, fewer things in a shorter amount of time, having that conversation is a skill that needs to be developed and it will make you much more successful in the long run.

Steve: Makes me think that there needs to always be a flat no but it can be an example you just model. You’re coming back with peers to trade off stickers to consequences. So you’re facilitating something.

Danielle: Yes, that’s my kind of, when I say my personal growth that that’s my, because then my growth is not just no good to say, Here’s why. I don’t think it’s a good idea, or I won’t be able to do these things here’s why. And this is what I mean time, right, this is how I think, engage other options to get this stuff, it’s not a plan now, and it’s not the Yeah, I’ll do it and I’m secretly working out.

Steve: When I asked you about accessibility you went all the way back to your first week on the job and what that was like and we’re talking a little more now about where things are at right now. I wonder, maybe just a stepping back question, if you could describe a little more of that arc, like what you came into how your job is evolved over the years that you’ve been there and up to date where we’re at now.

Danielle: When I came in. It was at the beginning of what we added at our technology transformation. And part of that was building a design team. So they were ugly at hand handful of folks, and I was brought in as a director to build a user research team. And that was a lot of executive conversations about the value of user research building an understanding about how we work. That’s a process where lots of quick and quick in your clips, does that mean it’s really quick, in healthcare, but doing some initial studies that demonstrate the utility of what we do, creating lots of new templates and getting feedback on presentation style so that we can communicate value and consistency and that didn’t take as long as I would have expected and it wasn’t as hard as I was expected. And by that I mean, I come from other organizations that had more or less established UX disciplines for some sort of experience if so, when I had been a consultant for a couple of years. So, what I was expecting coming in and like to have to do a lot of r&d, honestly to talk about how this is important and we needed it. I had to do very little that, and that was a real surprise. So lesson me it looks like it’s not able to super easy, it’s just that doing that work did not take up as much of my life as I thought it would. Because it was mostly an awareness problem, people just didn’t know that this existed, and once they heard about him can understand the work that we did, they became avid consumers. So it was a big we got a lot of fans. I did a couple of presentations to senior leadership, to this day, Express Scripts president we’re waiting in hallway because he back in those days it was a lot of conversations about what users are doing. And I also presented to clients a lot. So the way that our business works, like I mentioned the beginning is that we help with your prescriptions, if you have coverage with health insurance. So, our clients are really good health insurance plans are different businesses if they self fund their health insurance. And because this was a new function within the company, we wanted to make sure our clients understood that we were doing this and that, if I sent some of the survey, and they reached out to their health plan administrator, it wasn’t new news, so that was a different thing for me to have all these client conversations, and I still do that to an extent but now our sales team, kind of knows about what we do, so they can speak to it. I see pulled in a client conversations on a less frequent basis, but back then I was probably going to St. Louis, a couple times a month to show clients our disability lab and talk to them about what it is that we did for the first and probably two years after like the first couple of years, usability practice maturity, we had a couple of folks on my team to start to do analytics and data science lead and have taken over the NPS program to clean it up and systematize it, and now we have a winner I would say it’s healthy analytics practice where we can start to put things in place like a B testing and talking about how that is used in the organization and become consultants for different analysis questions. so we’ve gone from kind of not like having cabbie leaves little pieces of UX that there was a couple UX people bringing across the organization a couple of folks doing usability testing, they started building a website before I got hired, honestly, so they knew they wanted to do this they had executive buy in, but really, rolling it into product and having it not be optional check the box thing really started to happen over the last couple of years, and now let’s say almost gotten to be a management of demand like under this umbrella shield to keep my team from getting silex projects as we call it. So, and my job itself. I started off with a small team as a director. Now I’m the senior director some pretty well removed from doing research. Back then I think I managed a vendor on one project I was pretty well involved with a couple others I might have done a survey to myself, and now like I can’t, I joke all the time like I don’t know what you mean. I don’t know what it is the PowerPoint slides. I’ll barely do it. I just, I listen to what my team tells me, and I make a path for them that have impacted the organization. So my job has changed from building the competency, demonstrating that we have this competency and getting buy in to making sure that what we do the work that we do, is impactful, and I do that by creating barriers helping were a candidate, identify stakeholders and to fix some sort of weird problems in my own cover, to help in my team to understand how the business works, and vice versa. Okay, the business understand how different new parts of the business, understand how my team works and how they may or may not work with us.

Steve: What are sideways projects

Danielle: and sideways projects are, you are I some violence aged some big translate into like the lunch and learn or something, and someone in a different part of the business, let’s say, in our traditional IT department I say they they make dashboards for infrastructure monitoring, I make it so they heard of my team I heard it was their VR thing. They love this idea, they contact any person on my team, ask them if they can help them by running a usability test on their infrastructure monitoring thing, and that is a sideways project. I have to make sure that we don’t say yes to those kinds of things we don’t necessarily say no because they are under Resources they can point people to like I said like the sauce. But we don’t get involved with too deeply was just to preserve our family and focus, and it’s one of those tough ones. It’s not like things are unworthy. It’s just the human limitation, the team’s ability to do the work. If you’re doing that, then you’re not doing something else, because everybody is busy. If we do have spare cycles spare bandwidth, then we do consider those kinds of projects, but you shouldn’t be down.

Steve: So you’ve talked about the healthcare ecosystem, but I assume, when you call it an ecosystem that means you’re sort of outside. You’re not a care provider for example, but I’m wondering if you’re impacted by regulatory concerns as part of your role in that ecosystem.

Danielle: Yes, we are. And that was another thing that was surprised during my first week on the job. I have an academic background, doing user research, or research like this, and I’ve also been in industry for a good 1015 years. So, I understand, and expected that we would have MBAs that we need to put in place with our research participants, we would have to get informed consent, and we’d have to have certain ethical practices about letting people participate and back out and, you know, things that I come to expect is second nature. What I did not understand, was that healthcare is healthcare. There are quite a few laws that come into play, how you can talk about sensitive health information, and I had to get very friendly with all of our attorneys. And I will say all of them but quite a few. So my first week maybe, and that I got introduced to one of the attorneys that set in your mind. At the time, and he was like you did but let me invite people to come here and tell us what, and stuff like that they kicked out a series of meetings as me speaking with our attorney, are different, and they’re several different legal departments inside of a healthcare organization. I also learned, but he was feud with a few of them about what user research is and how we can use the data, and who we can and cannot talk to and what’s very important as a user researcher in healthcare, is that you are not soliciting information about somebody’s health condition, but it is the context of which they’re using our service. So we do have to could, and I learned this in this first couple of months, being on the job, and I now have to teach it to new researchers on the team put this person in place, but we do have to do very specific things to protect the data that I’ll share with us, because they are, as you can imagine, if you are in a usability session about using a prototype to refill a prescription, we have to make sure that none of that data is real, it’s a prototype, so you’re not pulling your health records to build this prototype, but as you get feedback he might start talking about what happened last time, and start talking about your specific prescription drugs, and my researchers have to redirect you. The reason why is because that is protected health information, and we have to be sensitive to that. And it’s just something I never really thought about coming from academia coming from industry, this idea that if we run like a common thing we do and I’m sure people still do, is in a gorilla hallway research where we just grabbed somebody, and it’s awesome sponsor of the project, give me unless you still a feedback or do an interview about how they might manage this certain situation at home. Well, you do that in healthcare, and you’re talking to an employee, and they might, as part of the usability evaluation or interview reveal some of their health conditions, and we have observers present. That might be a breach of their privacy. So, again, we have to be careful of who who observe sessions, who has access to recordings, and how we anonymize and research participant grids even when we send them out to observers we have to normalize, all of these things that I just didn’t even think of before I got you, and we I spent a lot of time with legal. Now the seas that broad term with legal for a couple of years to marginal boundary. So what we can and can’t do, so that we do, you know, he can do research more quickly, but in the beginning, it was kind of a curry let’s, let’s make sure we can recruit these people, let’s make sure that we are compensating them with fair market guidelines and not. And we have often documented the appropriate place because there’s also loss of power. It’s just an awareness of the regulatory environment we’re in how it impacts research that was such a surprise. Same thing applies for analytics, think, certain ways you cannot look at the data because it is not relevant to design, directly or because it uses your pH, your private health information.

Steve: Maybe we can switch gears slightly here. Great to hear you talk about, as you grow the team, what kinds of things you look for in user researchers,

Danielle: because I’ve grown the team in different ways. I look for a few different things, some things are the same. We do look for people that have some sort of demonstrated expertise, either in user research or analytics data science, you know, you can talk about your craft in cohesive coherent way and communication, again, is consistent, no matter where then where we do want to know that you can present data. One of those more intangible things that we look for and I’m can’t say that I have a perfect way of doing this is patience. One of the things that I think I’ve alluded to is that healthcare is we have a lot of rules to follow some things we cannot change very quickly, but we can work towards changing, and it may take a long time. And that can be challenging for some folks, it could be come from an environment that was very fast moving, or look at work worked at the placements where I was in charge of changing the UI so I could literally open a source file and change the words in the application, myself, and just publish it, and be done with it, and we have several rounds of review, to make sure that what we’re saying is legal for lack of a better term, summarize. So that’s something that I do look for is we don’t want to get folks in who just become really frustrated by how slow things move on the other side we also don’t want folks who are okay with slow things. So, it’s a delicate balance. We look for people who have the ability to kind of see the long view and make steps towards it, you know, push boundaries where you can but not just burn out. By doing that, other skills we look for. Honestly, it helps if you are methodologically agnostic from a user research perspective that means that you’re comfortable doing interviews, some ethnographic ethnographic methods and coursework, or something like that. Like you can do a few different methods may not be great at all of them that you don’t like I have met people who have specialized and we only do contextual interviews. That’s how we do. And this team would not be the place for someone like that. And the other thing that, again, that’s more difficult to select for guys some level of empowerment, were mentioned earlier, a little bit where you have the bounds of what you can do, and know what we’re trying to achieve and suggest studies or suggest data sets that may help with making decisions to get us there, and stead of waiting to be asked to do a thing, taking medicine, like being an order taker, like of course as collaboration where you sometimes get requests to do a study, but also having some initiative to be able to see something happening or see a problem, and suggest a way of getting more information to address

Steve: the problem, how someone demonstrates patience, but not too much patience, or empowerment.

Danielle: It’s all been in kind of conversations and interviews and just a description of your past work, and I know it’s very subjective and I, as I said I’m not really saying I’m great at it, and I love a better way, but we do try to ask questions about you know typical questions about like what was your involvement in projects. How did it start, how did it end. What happened with the findings and just listen to what people talk about. And that has helped a lot in knowing if you have a demonstrated history of taking initiative, or if you don’t. And if you don’t, it’s not necessarily the kind of disqualification, but we do try to ask questions to get out if you can, if you’re interested in taking initiative, and being empowered to do certain things. I have met people who are not interested. it’s like they do you really like getting requests for a specific thing and executing against it, and we try to get at in a conversation in different interview settings.

Steve: I want to go back, you talked about some of the kinds of places you’ve worked in, you talked about getting a PhD of wondering if you could talk about if there were some decision points. How did you decide to pursue your graduate degree, and when did you learn about user research.

Danielle: I’m sure my age, I wanted to be a doctor, and I was, and this is a bear with me. So when I was a kid I want to be a doctor, so I went to high school for health professions and he said I went down that path. I started down that path at a very early age, and when I was in high school, I met a psychologist, and it was a difficult time actually one of my classmates committed suicide. And so, the school district brought in, psychologists, and at that time I’ve been in hospitals doing rotations. As a teenager, so take that for what it means and what that is, but I saw this interaction with this, psychologists, much more interesting and helpful there when I was exposed to other other practices. So I shifted focus to becoming a psychologist in high school. That sounds like I have it all figured out. But anyway, so I did that in high school, and in undergrad, I worked as a clinical psychologists research assistant, and I didn’t like about clinical psychologists psychology was not very interesting, and even though they had interesting problems to set that experience was not, it was kind of boring I can’t think of a better way to say it, so I was in my kind of a research methods lab class, and the TA in that class asked if I want to be a research assistant for an applied cognitive psychology professor, and I said yes and started down the path of doing research on eyewitness testimony, and that was super interesting and Smit to two and a half years, my last two, two and a half years of my degree, doing that. My last year, to my senior year of college, I finally got to ask the grad students that I was working with how much cognitive psychologists make, and this was in the late 90s, and he told me like becoming like $35,000, as a professor, and as kind of freaked out because my mom did not do all this like work all these jobs, it could be through college and do this for me to come out and make about how much money she did so, like, I think it might have taken me a week to go back to my professor and tell him I was probably going to have to stop doing my research assistantship because I’m switching my major back to pre med and I was just going to become a doctor because I just needed to make more money than that. And he’s like, Whoa, let me set up some meetings with you with other parts of other professors in the department to tell you about other branches of psychology that are more profitable than becoming the professor of applied cog, and I was like, Okay, well you got a couple of weeks until I have to go to my counselor and change formation. And that’s how I found out about human factors and industrial organizational psych, and I also like I sent a man with a couple of professors to tell him about these other branches, I took a couple of courses, and apply for IO programs and human computer interaction and human factors for a fence for grad school, and I ended up going to rice that had combined IO Human Factors department, and really loved it. I took a couple classes in undergrad, like I said, and they didn’t really have a human factors class I just had read about it, so it’s kind of on the fence about IO psych but Bryce had a combined department, so it’s like okay like he figured out grad school, and took my first human factors class and it’s like, oh, this is my jam. This is my thing. Technology applied to humans, helping design technology to be more suitable to humans. Yes, super interesting I feel like I can help people and, and be able to pay my student loans. So, that’s how I ended up in the field or sticking with the field. And when I was in grad school I did a couple of internships, the one that really sealed the deal for me was, I worked in the habitability and Human Factors department at Johnson Space Center in Houston, as a contractor with Lockheed, but work in human factors department at NASA, Mike helped with disability evaluations of physical print instructions, websites, Space Station habitat layouts and all kinds of different stuff and I did a little bit of design too. So that was the thing that made that convinced me that I made the right choice because it was so interesting, and I could nerd out with all these people, like other key factors folks, and it was like meaningful work, so kind of ethic I ended up leaving NASA because I was taking too long graduate, and so I had to kind of finish my dissertation graduate early, and then around that time was like it was 2006 2007, and that’s when you started seeing kind of user experience jobs show up as call that versus Human Factors engineers or usability engineers and I ended up taking my kind of first post graduation job at Dell, actually I think I was a usability engineer for a while there. My job title change later to be a UX person. That’s how I ended up cheaper. So series of, kind of lucky events, and unlucky events but it’s not something I knew about early on in undergrad, but I do get a sense have a degree in physics.

Steve: What was the topic of your dissertation.

Danielle: Oh man, warning label design. So I was a risk perception and safety person, and my dissertation was about how to design warning labels for over the counter directives so how do you warn somebody to do something that they really really want to do you want to read it.

Steve: I mean, so from this period that you’re describing the different programs and the different, different disciplines that you are discovering and maybe these internships, do you feel like any events, maybe not in the literal sense but you feel like any of that is present in the work that you’re doing now.

Danielle: Definitely. I feel like be wrapped with my experience in terms of different methods, I had a lot of exposure early on to very complicated statistics took a lot of stats classes, my dissertation, it was a mix of ethnography interviews, and lots of regressions, so where I am now is a joining, all those things, so I familiar with offline experiences and familiar with online experiences, since the very extreme in our events in very, they need to design interactions that are flexible for different scenarios and different physical abilities, those are some things that I’ve learned at NASA, that come into play now. My experience doing like usability testing different styles of interviews and accelerate construction like all that stuff that I did in undergrad, our stuff we did at NASA at PayPal, like all the different environments I’ve been exposed to all come into play here, where I’m trying to help different types of researchers, understand work in an agile environment, even though it might not feel so natural. Like, understanding those principles and how we can consistently work to get better is all relevant. And one thing that I kind of consistently kick myself for is dropping the academic chair seconds, are the things that are learned in school that are still relevant today that I really dropped and try to distance myself from and early on in my career. Most cars for funerals, because there is this idea that, Oh, you don’t want to be to epidemic, you don’t want for folks who think that research takes too much time or you need to do all this math, so just drop all that, and I can I can take myself for that because I didn’t have to relearn it in this role. We learned a lot of the stats that I dropped and kind of the research design principles that I use a lot earlier, like a NASA and in grad school, because now we have so much quantitative data at our disposal, and a lot of us do come from a background like I do, where you did learn how to do a lot of statistics and think about the problem from a measurement perspective in terms of like chunking up a problem like okay this part should be qualitative. These are parts of the answer quantitatively these parts, we probably can ignore for a while until we get these first two chunks down, and that level of thinking has been invaluable in managing a team like this where we have to chunk up problems we have to try to answer things as much as he can, and it’s a useful skill but I do wish I could have kept can protect it more. Early in my career, now I get to bring effectiveness in really, really valuable.

Steve: I mean maybe just building a little a little bit new if you look at the profession, maybe it’s your own organization or maybe it’s just the collective profession of user research. Are there things that you would hope to see terms of evolution or growth in the practice of user research, maybe over the next few years.

Danielle: Yes, I do feel like there’s a lot of things that I would like to see, and I hadn’t really thought about this before so answers are not in any sort of priority or top of mind. I know that everything is Software as a Service, but there are some of us that world that’s highly regulated. So, for me tools and methods perspective, I really miss Moorea miss being able to run for having my team be able to run usability session to record things local people to have, like, analysis and work using local tools. I know it’s a crazy thing to kind of tap into a desire to see as a profession, we need a wide variety of tools and I feel us gravitating towards the same solutions that don’t really work very well for some of the hairier. The human existence, and I am going to go ahead and say that healthcare. If you think about the things that we have to design, we have their parts of our lives just people trying it, and trying to figure out your healthcare situation or trying to navigate some aspects of the government. It’s just really hard. And so those are opportunities for a user experience professional to be different, and happy. But another area, I’d like to see. And I believe my other other guests mentioned this too is a maturation and methods. We don’t see a lot of time on coming up with different approaches, and I don’t see a lot of meta analysis, and that term is to be loaded to but before you look at a breath, or even qualitative usability as we really like to see us advanced ways that we’ve looked at across time and across multiple iterations to pull out different insights and patterns that we should be looking into another is on the other side. I’ve really like to see a more user focus when it comes to data science, and this is not an express script statement but just in general, you see a lot of automation in place like bots, to help. Whatever you want, like, fill in the blank you think you’re on your I am in with a person and it’s about for you go to IBM and think you’re going to talk to a person. It’s hot, but like lots of applications of data science, and like user facing automation, we put in place to solve a problem. These are habits have been put in place to solve the problem business habits. So, in terms of user research it’s I feel like we can add a tremendous amount of value in that space to help bring that kind of work along in terms of solving real problems and real Tasks tab, and to do that you need to learn some of that. I will say, learn the methods that that world. Learn how it works at a high enough level to speak intelligently to user centered design in that context.

Steve: Those are great. Is there anything that we should have talked about today that we didn’t get to want to make sure we cover.

Danielle: I think we covered it but I’m away for starting to talk about the broad umbrella, that the experience research, the way that my, my team is made up way that I think is part of our ethical responsibilities, do you think that is come across in that conversation compensation but like I said earlier, we really want to make sure that serious. It’s not just happening to people that they are defined with their voices, considered as part of a solution, and that it’s continually refined. And I do feel like in some scenarios user experience professionals can be laser focused on a new for a few strains within the UI, this, that you. What does this mean in context of your users reality, and I know that can be kind of big and overblown, but you can have an approach that chunks better into reasonable solvable problems that you can then work with other folks in the organization to

Steve: I really like how you brought us full circle from where we started at the beginning to where we ended up. Thank you for the conversation today and for being on the podcast It’s been really excellent to get to speak to you.

Danielle: Oh, great! And I just realized that husband decided to go chop some wood while we were talking. But, okay. I think we’re having barbecue this weekend.

Steve: Great, well, thanks again!

Danielle: Thanks, Steve.

This episode of Dollars to Donuts features my interview with Amber Lindholm, the Head of User Research at Duo Security.

That’s the sign of a really good researcher – it can never be just about research for research’s sake, like this is a cool project, this is a neat thing, I really wanna go in-depth and understand perceptions of XYZ with these people, if you don’t have that ability to understand the organizational and business contexts and the types of decisions that are having to be made every day by the rest of the folks in your organization, your research isn’t going to have an impact. – Amber Lindholm

Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.

Steve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.

I read recently about a new genre of TikTok videos that featured people applying makeup, while lip syncing to standup comedy routines by John Mulaney. I believe that TikTok has it roots as a platform for lip-sync performances, and of course makeup tutorials and demonstrations are their own thing on the Internet. But how do we end up with the combination, and not just one but a whole series? How did this come about, why are people doing it, and are there other niche sub-genres or patterns that this relates to? Even if we are tempted to dismiss these behaviors as just a bunch of people being weird, we need to be doing the research to understand how, and why this is happening.

Sure, people at TikTok should know how people are using their service. But insight about this also is valuable to other platforms like YouTube and Twitter and Instagram. How could this information give you a new perspective on user behaviors if you work at Dropbox? Or if you work for Michelin, either their travel department or their tire division? How could Nationwide Insurance make use of this?

Our culture swerves and leaps and when these emergent behaviors poke their head up through into the mainstream, it’s an invitation to take note, and to be curious.

This is what clients hire me to do, whether it’s to be the one that leads the investigation, to unlock the motivations and desires of current and prospective customers, or to be someone who helps their team as they themselves dig into the hidden behaviors and objectives of users. I help companies look at their own organization, their culture, and their processes to help set the stage for this kind of discovery work to flourish and have impact.

If you’re a fan of this podcast, then remember you can support the podcast by supporting me and my practice. Please reach out to me at Portigal dot com and let’s find a way to work together.

Okay it’s time for my interview with Amber Lindholm, she’s the head of Design Research at Duo Security.

Steve: Well Amber, thank you for being on dollars to donuts. It’s really great to get to speak with you.

Amber Lindholm: Hey, Steve, it’s great to be here.

Steve: Hey, I guess I should have said, hey, hey, Amber. So I’m gonna ask you to introduce yourself.

Amber: So I’m Amber Lindholm. I’m currently the head of design research at Duo Security, which is part of Cisco. And my background is in design, both kind of traditional print design, moving into interaction design, and then design research, which I practiced before, you know, moving into more leadership positions.

Steve: What does do a security to

Amber: do security, we provide various security products that help protect our customers basically access to data. So our core product is a multi factor authentication product. So when you’re going to, let’s say log into, you know, a tool that you use at work, you know, after you do your password, you get some sort of maybe a push notification or something that verifies that To you, so we help protect organizations and their data as well as individuals.

Steve: So if I am a consumer and I go to my bank and they said, oh, we’re going to send you a text message to prove that you’re you before we can let you access your account, is that the kind of tools that you’re providing?

Amber: Yeah, that’s, that’s a great example a lot of folks are familiar with, with us through like financial institutions. And we basically have created a product that’s super easy for organizations to roll out and get folks enrolled in. But it does that where it’s it’s verifying through a second method that you are actually who you say you are,

Steve: eventually, let’s get to talk more about the work that you are doing to a security but since you said a little bit about some of what your background was it be great to just hear more about like you mentioned, print design. So what’s kind of the arc or the history for you from when you got into figuring out your profession to you know, how you move to where you are now.

Amber: So I started out as I was mentioning in graphic design, so I was trying I went to University of Illinois we did really kind of what I would call more classical, like Bauhaus, you know, educational style, learning all about formal typography and form and color. And so after school, I followed that path, I found a work with a PR agency at first and then in house at a nonprofit called Rotary International. And this was up in Chicago. And I spent my time you know, designing brochures, you know, reports, invitations, billboards, all kinds of print materials. And during that time, I just remember there was this particular project I was working on where we translated most of our materials into nine languages. So it was going out around the world. And I was creating these, they were kind of these little packets that had a CD that had files for the different rotary clubs to produce their own kind of marketing materials. And I worked with the translators to you know, translate the copy in the on the files and the file names, but after that, I Didn’t know when those things were sent out in the world if they were going to be used properly if they were going to meet their needs or expectations, and at that time, I started searching around and trying to think, you know, I feel like something’s missing. And I came across, you know, the terminology then was Human Centered Design. And it was just a revelation. For me, it seems like, you know, you can create these things, but if you don’t understand how people are going to use them and what they need, you’re really gonna miss the mark. So that was a catalyst for me to continue like my education. So I found a grad program at Institute of Design in Chicago where I went and I learned, you know, how to do research. They taught all about human factors, all kinds of approaches pulled from social sciences, very qualitative in nature, learned about design strategy, I learned more about interaction design, and that was really I really felt like I found my sweet spot. I loved the research side of it. I loved going out and talking to people. And trying to understand you know who they are and how they behave and why. So when I graduated actually, right before I graduated, I did a couple of things. So one of the things was I wanted to do a little bit of living abroad. So I went for a year to New Zealand and I worked inside the government there so did a project looking, it was more like a service design customer experience project, which was really great, because it neat to get that government experience came back and my husband and I moved to Austin, just sight unseen moved down here. And I’ve had a couple of different jobs here in Austin. I started out at frog and I was still mostly focused on design. I was doing interaction design, like mobile, tablet type design, but then I started adding in the research projects, and that again, really flourished there and love doing that work. Got to do some research in China and other places. And this was more contextual work like going in people’s homes, spending a lot of time with them to understand their world and A couple other things I did after that I went and worked in healthcare for a year trying to do similar kind of research understanding. For instance, I did a project about the new parent experience. And then I went back to consulting to a consultancy called Project 202. So spent, you know, quite a few years actually being more of a consultant role working across many industries, which was fun and started leading a team that project to research and Insights Team. So we would work on, you know, tons of projects, looking at really large customer journeys. And then after consulting for so long, I decided that I wanted to really understand the other side, so to go more in house to work at a product company. So I worked at Atlassian and led a design and research team for a product called stripe, which is a video and chat tool. And then I moved to do where I’m at now. And again, I lead the research team there love helping our team grow love the side of research that really has an impact and provides value. So that’s kind of my story

Steve: to talk about today. There’s two sort of vectors that I hear one is in house and consulting as different perspectives. But you’ve also talked about research and design. And it sounds like over the course of your career, your title at least has. Well, I guess you also said some of the work you’re doing might lean more heavily towards one or the other. It’s hard to ask the question in an open ended way as I want to. So I’ll just try the leading version. We think of research and design, like they’re different words. We call them different things. Sometimes they’re different kind of career paths. But one of the threads of your narrative that is compelling to me is that you’ve kind of worked with both those pieces of research and design. I guess I’m wondering when you think about for yourself at least and where you’re contributing where you’re adding value or how you put all these pieces together? What’s the relationship between the kind of the R word research and the D word design in your makeup and your skill set.

Amber: I think they really complement each other. I think it makes me as a designer, it made me a stronger designer to have that you know, research mind and as a researcher, it made me stronger because I can Really make sure that the research I was doing was framed around and presented in a way that provided information relevant to the folks having to make decisions. So I always think about that. It’s like when we’re trying to figure out what to research and what to prioritize. There’s always a whole set of questions. And there are decisions that are having to be made that some are more ambiguous questions where you really don’t have a lot of information. Some are really high risk questions. So you want to make sure you’re understanding those and providing that information because people are having to make big decisions and tactical decisions and thinking about how you frame that research and present it back to folks or include them in that process is really important. So I think understanding how designers work and also you know, now knowing how product companies work, understanding the questions that product managers are trying to answer engineers are trying to answer and helping them actually articulate those questions is something that I use strive to do. So I think they just really complement each other.

Steve: The way you describe how you want to make sure that the research you do connects and supports, I hear an echo of your brochure project where you were, you wanted that context to make sure that you knew that the work you’re doing was gonna have the outcomes and have the results. It sounds very similar, that kind of questioning of, Hey, who’s doing what and how do we make sure that we help them do what they’re going to do that that applies to your quote users as well as your quote, stakeholders or colleagues?

Amber: Yeah, absolutely. We spend a lot of time like when I’m working with the researchers on my team, and we’re looking at a new project, or we’re looking across the projects that we could support over quarter whatever it is, we’re looking exactly at that like where can we have the most impact? Where’s the greatest need? Sometimes we’re looking at is there a new project spinning up Is there a new product manager coming in is there you know, a team that’s just starting out that’s never gone through this process. before. We really like to prioritize those kind of things as well so that we help get them off on the right foot so that they do frame those questions really well, we help show them a good process to follow. And those are really, really fruitful projects.

Steve: What’s the split between proactive you identifying what research you want to do versus something that’s more reactive, where there’s requests coming in? Hey, can you help us do this?

Amber: That’s interesting to try to assess out. I think it’s so it’s a little bit hard, because so at duo we have, we’re part of the product design team. There’s about 30. Folks on that team, research team. I have four researchers on a team and a research coordinator who helps Do you know all of our recruiting and scheduling and all kinds of stuff. And so none of the researchers are dedicated to a particular product because duo does have multiple products. We kind of stay as a little bit of a hub, but we start to embed like I was saying, Okay, let’s kind of embed over here on this team. Let’s embed over this on this team for this particular project or this time span. And so the researchers are communicating with the designers, I’m communicating with the design managers and product managers to learn, you know, what their roadmap looks like or what kind of questions they have coming up. So I end up really driving the prioritization of the work we’re doing based on all those conversations, looking at the priorities of the business and you know, kind of larger trajectory of where the business is going and make the decisions of like, where we can actually prioritize our time and where we’re going to have the most impact. But we do also get requests. So we work with folks outside of the kind of r&d process as well. We’ve been working with our creative team on website design. We’ve been working with technical communications folks in our like knowledge community, so those people will come to us with requests. We hold office hours every Monday, so sometimes, we’ll just provide some support or advice or kind of review things and other times we’ll decide to take on a full project.

Steve: Can you maybe reward And a little bit and talk about how you came to do and what was research like there maybe what was the context that you that was established when you joined and what some of the progress and evolution has been since you’ve been leading the team?

Amber: So do I guess the design team was formed about five years ago. And so Sally Carson’s, our head of design, so she was hired to come in and, you know, build up this team. And really early on, she wanted to make sure that research was part of it. So one of the maybe the second or third hire was a researcher, and His name’s Mark Thompson Kohler. He’s still with the team. And he came in and at that point, because design was so new and research with some new, the focus was, you know, on really rapid, like usability testing, right. So like, every two weeks, we’re going to do testing. We’ve got people, you know, what do you want to test and so the company was a startup, it was a lot smaller. And the idea was just sort of this like rapid, more evaluative type research. And that went on for a while. And then The team, Mark developed a set of personas that got embedded across the entire dual organization. You know, as a team continue to grow, we’re still kind of like a one person research shop, do the way the design team has grown though designers are hired that also have an interest and skillset and research. So designers also do research. When I came in, another researcher had been hired. So there were two researchers, and there wasn’t what you would call like a research team yet. So when I came in, we decided, Okay, let’s sit down and really talk about you know, who we are, what’s our mission? What’s our value? What are the strengths that we have going for us and the gaps and how do we want to move this team forward? And we did you know, some kind of off site exercises and that kind of thing. And we started at basics. So there was a ton of knowledge. There was a ton of things had been done, and we’re like, let’s make sure like we have it right. So that first quarter was a lot of poetry. together in a place where people can find all of the research that’s been done, document some of the tools and templates we have so that other people, you know, on the team can can use this stuff. And then we wanted to make sure that instead of just focusing on the evaluative work, you know, we were allowing the designers to really pick up a lot of that and that we could move into more foundational kind of generative type research projects. And so that’s what we did after that first quarter, getting all those foundations in place doing some additional trainings with the design team, the researchers, then were able to focus on these more larger, more strategic type research projects. And then we added in research Ops, we hired a research coordinator to help us so that the researchers weren’t spending you know, half their time coordinating, recruiting, maintaining our database of participants and all that good stuff. So she came in last summer. Her name’s Annie fan, she’s been just a fantastic value to the whole team. And then I hired two more researchers this spring, and now everyone, you know, sort of has their own area that They’re working on right now. And it’s been great. It’s just been fantastic.

Steve: you’re describing some of that, I think is a pattern and a lot of organizations that give the designers the tools to do some of the evaluative work and you know, a shift in what the research team itself specializes in. What’s been the outcome for the designers or the design work that they’re doing as you’ve empowered them to do that evaluative work?

Amber: I think it’s great. I mean, I think rather than, you know, us being like, No, we can’t help you and know that, you know, those questions aren’t going to get answered, the designers do have the toolset available. And we, you know, depending on somebody’s comfort level, or you know how much they’ve done this in the past will provide more or less support, right. So, like I said, it’s kind of like reviewing the plans, reviewing the script, doing test runs, maybe we’ll moderate the session, and then they pick it up after that, or some folks, you know, they’ve done this many times, and they can run more fully, but they are the experts in sort of this area that they’re working on. And they have the belief With their product manager and their engineers, and they can take all of that, you know, directly back into the work that they’re doing. So I think it’s I think it’s great, you know, I always have some some hesitancy if it’s the subjectivity that might come out of somebody who’s testing, you know, their own work. So we do kind of talk about that, keep an eye out for that. But so far like, it’s it’s been great and allowed our researchers to, like I said, focus on not just like those larger research projects, but a researcher on my team was Donovan right now is helping kind of consult across six or seven teams for this larger project. And so she has really good line of sight around the different questions being asked and helping people, you know, make sure that they’re all kind of sharing the insights they’re learning across those teams. So it’s allowed us to really level up our scope of influence.

Steve: That’s a great explanation and I feel like there’s an implicit value judgments around what kind of research is cool and fun and worth our time and then We can offload some of these tasks to the lower value tasks to other people, then we can work on the cool stuff. Not that anyone has ever said that. But it feels like that’s lurking underneath this move to kind of, you know, empower people to do the value of work. But I think what I’m hearing from you is, in a lot of ways, those people are better positioned to do that work, because they’re very close to the problem. And they can kind of zoom in on how they can get the most actionable results, and do do so maybe more quickly. And that research, I think your point about having influence. So the work that you’re doing is is more around the organization and more strategic. And that’s kind of what you’re positioned to do.

Amber: Yeah. So besides, you know, helping bring those designers together, Liz is helping to align pm as well are kind of like share across the hims. And she’s running a set of like metrics workshops across those teams like a created as a standard sort of workshop to help us to find UX metrics look like For signals and define those metrics, and so she’s bringing those to each of those teams. So now again, like it’s a more kind of strategic view across those, but then we’re still able to weigh in on and influence the the type of research, the type of questions and even the synthesis of the research that’s happening across those teams.

Steve: What are some other things that you’re doing to whether it’s around influence or you know, kind of bringing the customers into the organization?

Amber: When we set out on our kind of mission and the things that we do, you know, obviously sort of like executing and running this kind of product focused or foundational focus research is really important. But sort of the other half of it that we wanted to make clear about was like our role and helping to get folks closer to the customer to bring that context to them, to keep the customer Top of Mind, and I do frankly, it’s pretty easy because the organization across the board is extremely customer centric. We’re not like begging for people to listen to us. The value of research is really understood. It’s, it’s asked for actually, like, people are really interested in hearing more about the context RPMs. You know, they’re fantastic. They out there out there, they do tons of customer calls, but they’re still, as we’ve continued to grow. The risk that we identified a couple of things like early on, I mentioned that there was a set of personas that were created and these get referred to all the time across the board in big meetings, you know, big call hands and all of that. Salespeople reference them, our customer success, people reference them. But as people join the company, as we were kind of in hyper growth mode, the real deep understanding of those personas may not have persisted. Right. So we wanted to take a step back and say, you know, how do we, as a research team, help other folks have the type of context that we have. So Mark, who developed the personas, we have a program set up where, you know, as new hires come in, he presents something called personas one on one to make sure that those are really top of mind and understood and we do a deep dive session on one of the more key personas. He also does something called these fireside chats. So we just chat with different folks. So for instance, different IT administrators or help desk managers or helped us folks or CSOs, we’ll set up you know, an hour long chat to just learn about their day. And that’s open invitation for folks to learn. Listen and on. And then we also with the whole of our research sessions and things, they’re open for folks to come and listen to. So it’s very much accessible. We have a research Condor, everyone can see and they can pop in on different things. And lots of folks take advantage of that. And then I think another thing that we wanted to do was make sure that we had created additional sort of artifacts and frameworks for people to think about when they’re making decisions. So we took all this data that had been collected over across, you know, about four years of tests and interviews, and ran these large workshops, and Mark, you know, synthesize all that information into a set of design principles that are persona specific. So they’re not these like really super high level principles that are hard to make decisions with. They’re very applicable to making, you know, decisions on UI on making decisions on copy you might be creating. And so those principles are referenced by You know, designers, they’re used in critique sessions. They’re referenced by product managers as they’re talking about their roadmaps, folks like technical communications and things like that. So the personas and the principles, they’re not just an artifact that just kind of sits on the table, we bring them to the forefront and a lot of different ways.

Steve: Could you genericized one of the principles just to give a sense of the granularity of it?

Amber: Yeah, so for our IT administrators, for instance, one of the principles is around the ability to take action. And so anytime we’re making a decision about something in our UI or a flow, if there’s a dead end, that that person cannot actually, let’s say they’re seeing some kind of thing that may alert them to a security problem. If they cannot immediately take action on that item, then we’re not meeting that principle. It’s very important for folks to be able to move from information that they’re seeing directly into action. And then like for our end users, one of our principles is reduce friction to the extreme. So we’re not in the kind of product where, you know, you’re trying to create a sticky situation where someone’s in your product, you’re trying to actually minimize their interaction with your product, because what they’re actually trying to do is they’re, you know, get into their work application. So we think a lot about that, like, how do we reduce friction to that extreme?

Steve: So those are specific, as you said, and meant to be actionable. Yeah. And how many principles in a project like this, how many principles should one be trying to come up with?

Amber: What a question Steve? No, I think we narrowed it down to five principles for our two main personas and that I think is a good number like three is too few eights, too many kind of a thing. But five has been like the perfect number. And when we first rolled these out, we we did some fun things to where we created some really neat posters. We put out in all the offices, you know, we socialize them with people. We brought them around to meetings, we personified the principles even, which was really fun. But yeah, I think you got to, you can’t have too many.

Steve: How do you personify a principle?

Amber: So one of the designers on our team name, Andrea actually came up with this idea to, you know, get designers more familiar with all of these. And so each of the principles, one of us was given, and you had to basically just act like that principle. So we had to create a character. And in this sort of mock critique, we had to just care about that particular principle. So it’s actually ended up to be quite fun to do.

Steve: Sounds like a good improv game was very fun. Just going back to one of the points you made was that this is a an organizational culture that believes in research and she said asks for research. Do you have any perspective on for duo how that came to be?

Amber: I imagine it’s a couple of different factors. I mean, I think that the folks, Doug and Johnno, who co founded the company, that was like a big thing. Like they really cared about the end user experience. They cared about creating a company that was different than the other security companies out there at the time. Their whole mission is about democratizing security. So making it easy and effective was there from the beginning. And it wasn’t just, you know, something they were saying was something they really believed in, they saw how complicated you know, so many security products were and they wanted to make sure that when these products are out there, and it’s, you know, an IT administrator, it’s not somebody with, you know, a cybersecurity background or anything that these more generalist folks can protect their organization. So I do think a lot of it came from the founders and then the folks that they hired. So like our head of engineering, Chester, who we product design actually reports up through engineering. He’s completely supportive of design. You know, we hire engineers that have that kind of mindset also have like customer experience. So I think it just kind of permeates through the culture, because as you said, starts at the beginning and starts at the top. I think so. I mean, that’s that’s just my impression. I mean, I wasn’t there at the beginning. But I see that legacy. And I see that through the people that were hired on that continued to be in leadership.

Steve: I have so many interactions in so many different venues where somebody who maybe is at an individual contributor level sees that their culture is not like the one that you’re describing. And they’re being blocked in a lot of different ways. And the question is often, like, well, what can I do to succeed this way when these things are kind of arrayed against me? I don’t know if you’ve encountered that or heard that question. I personally, I find it challenging to give them super actionable advice when I feel like you’re describing you know, hey, it starts at the top that says how this company was founded. You know, they’ve grown and hired and supportive that and if you’re in a situation Well, that hasn’t happened. It’s an interesting challenge for people that aren’t embedded in those kinds of organizations to make those kinds of changes and advocate for that.

Amber: Yeah, I think especially like, to your point, if you’re not at a leadership level where you have a, if your scope of influence is just with your immediate project team, it’s gonna be really hard to sort of like, level that up through the organization.

Steve: You’ve named some of the people on your team and the kinds of contributions they’ve made. And I’m just wondering how you think about, but I think you said, you just you hire two people not that long ago to join the team. Yeah, yeah. Can you describe a little bit about your approach to finding people what that looks like in terms of saying, Hey, this is someone that would fit with our organization?

Amber: We have a probably what’s it fairly standard interview process, but I’ll just describe that and then sort of what kind of questions I’m asking and looking for, but I typically, you know, reach out to folks I have like an initial conversation with them. Just to understand what they’re looking for in an organization, or what their kind of career goals are just to make sure it aligns with, you know, the kind of goal we have open, then we do a longer interview. And we use, again, pretty standard, like behavioral interview questions to ask people to describe real scenario. So behavioral interviewing, it’s kind of like doing user research or like doing design research, where you’re really trying to get somebody to give you very specific real examples like to tell a story, because you want to understand what the situation was, how they thought about it, and how they reacted to it versus like a generalized answer. So we follow that I put together like a set of questions. And then once folks kind of go through that, they go into like a two hour we do a bit of a portfolio review. And then a final round interview is more like a four hour thing. And we have designers in there. We have PMS come in, sometimes engineering managers, so different folks who they might you know, interact with. than their day to day, participate in that more long kind of final interview. But some of the things that I’m always looking out for like, obviously, you know that they can do the work, but the way that they present the work is really important. So, you know, coming from a consulting background, I am design background, I do a lot of emphasis on how people share their work and how clear it is and how compelling it is. How do they tie that story together? How do they tell high level insights, but get into enough details to really support and paint the picture of why that insight matters? So that’s a big part of it. And then the behavioral interview questions I’m always looking at, can someone learn from a situation like can they I don’t want to say like, admit they’re wrong, but if they encountered something where it went wrong, how did they go about learning, growing and kind of improving for the next time? I’m also looking at people that are willing to ask questions, so I really love hearing the questions, I always leave a lot of time for folks to ask me questions. So do they have really good questions? Were the kind of topics they’re thinking about? Are they people who are going to come into an organization and be really curious and willing to ask people questions and not just kind of sit and be scared if they don’t know the answer, trying to think what else? We ask questions, you know, around working in diverse teams that’s really important to us, and just how they kind of collaborate and work with others.

Steve: Yeah, I want to ask about one of the things you mentioned, telling a story about an insight and you know, help it come to life. And you’ve worked in consulting. So you’ve experienced some version of this, I think, I mean, I, in some ways, I think I’ve grappled with this for much of my career, that there are things that happen over the course of a project that are galvanizing that just change everything, but to tell somebody else outside that organizations, that’s a harder story to tell. So I’m wondering, what are the best practices for that? How do I make a story about an experience I had In the past come to life for an audience that didn’t have the assumptions, the biases, the worries, the you know, all the sort of things that make it so impactful. How do I help somebody else understand that part of the insight?

Amber: Well, so what you just said is exactly what I would expect is that, you know, folks that know how to tell a story upfront, they give me enough context about the organization or the problem, or the assumptions that were there in the beginning, and what kind of impact their research had, and setting the stage of that context. It’s not just about the project, right, exactly what you’re saying, like, every single organization is unique. Every single set of people has different assumptions about things, the challenges that someone might be up against within that organization are going to be different. So a really good researcher is going to help you understand that context of why the thing that they’re telling you matters. I think that’s the sign of a really good researcher is that it can never just be about, you know, research for research sake like this is a cool project, this is a neat thing I really want to, you know, go in depth and understand perceptions of XYZ with these people. If you don’t have that ability to understand the organizational and business context and the types of decisions that are having to be made every day by the rest of the folks in your organization, your research isn’t going to have an impact. And so I really listen for that. And interviews are people able to help me understand how they made an impact and how they understand that larger ecosystem that they’re working in?

Steve: So back to the archetype, I was describing the person that approaches me and I think all of us with here’s why I’m unable to have impact. Here’s how my culture says doesn’t support me. If that person comes in to speak to your team and tell stories about the research they’ve done. How do they highlight the impact of their research when there’s forces arrayed against them to have that impact?

Amber: Yeah, I mean, I think that is a really challenging and I have had interviews where folks have revealed some of those more negative challenges, I would say like, it can sometimes feel like a person puts up all the barriers and just those like I couldn’t really do anything versus somebody saying like, this was the situation. This is how I tried to take my work and have influence at the level I could influence right. So you know, maybe it’s getting the one person to shift their perspective and get them to like, listen in on the interview, I don’t know, whatever that small measure of change or influences I think focusing on that is better than just saying, you know, there’s all these constraints and I couldn’t do anything.

Steve: That’s good. I’m gonna use all these because we are we’re always doing storytelling and I think it just to kind of reflect on how we can most effectively communicate overcoming challenges, having success, success at different levels, all that context. Yeah. It’s a really good way of putting it. What are you seeing in terms of where people are coming from to research,

Amber: maybe it’s helpful to describe like the folks on my team just because that’s, you know, who I’m interacting with. So one of them was actually in journalism for most of his career before transitioning into user experience research. One of them was working in academia for a while doing all kinds of different unrelated, you know, research around philosophy and and other things and then transitioned into technology actually, being a technology director for again, publication before transitioning into user research. One was a designer before making a transition. And one of them was actually doing market research for consumer packaged goods companies. So all of them had career changes before coming into the research side.

Steve: She’s interesting. Would you include yourself under that umbrella?

Amber: Yeah, absolutely.

Steve: It’s just a fascinating artifact of what makes up our field. Mm hmm.

Amber: Yeah. And my so I have an intern coming this summer. And she, again was more she’s been in the writing field. So she’s been a professor like an English professor and now working on her master’s to transition into research. So yeah, a lot of I think folks who come from like that, writing journalism, storytelling background, it’s a really neat transition to see. Do you

Steve: have any theories as to what is it about research that pulls people in from these different backgrounds?

Amber: I think in that case, like, there’s a easy connection there where journalism folks, they are trying to uncover information. They’re trying to understand people, they’re trying to tell stories. So that makes sense. I think other folks like the woman who was working more in technology, and then the woman who was working more in design and same with me is that you find yourself drawn to the beginning of the process where you’re really trying to understand the problem or you come into a project where you’re delivering something where you didn’t under And the people or the context, and you’ve gone through that yourself, and you want to, you’re really drawn to like, how do I make sure this doesn’t happen again? Or how do I learn more about this? And then people kind of stumble into it sometimes. You know,

Steve: I’ve been identifying as a researcher for a long time. And I always felt like yeah, I stumbled into this. But that was more it felt like that was more common because the professional wasn’t defined. We didn’t have, let’s say, the program, instead of design that that you went through, like, I don’t think that existed. If it did, no one knew about it. So I feel like oh, I came from this sort of prehistoric era where field was nascent, and so we had to kind of stumble into it. That’s how it was being formed. And it’s just fascinating to be many years later, decades later, and programs like yours and so many more, and yet, you’re characterizing a bunch of people with amazing skills and oriented towards different work and kind of transitioning into this. Yes, still what’s going on. I don’t know if that may be how careers happen. Regardless of what discipline or business you’re in, but I guess I always thought it was gonna change. And it’s just so curious that we still have this happening now in this field,

Amber: at least from what I can see, like from the, you know, folks on my team and other folks I’ve interviewed recently, I think having that other kind of background or perspective, as gives them an upper hand in some ways, because they bring a whole nother set of skills and perspective into the work they’re doing. I’ve just found, yeah, just the way that people can craft those stories or the experiences that they can share, help build like a bigger picture of just different perspectives, kind of bring them to this work and help them have kind of a point of view around certain things that is very interesting,

Steve: if you’re saying this explicitly, but I’m thinking as part of a team when you have a range of other contexts or other skill sets that people bring in even more so even better, because you have some just diversity of frameworks and superpowers.

Amber: Yeah, absolutely. It’s, it’s awesome because the other team can build on you Other, they have these different experiences. And right now I’m actually like the next kind of skill set I’m hoping to bring in is somebody who has pretty solid background and mixed methods. So bringing in more of the quantitative, somebody who’s done a lot more quantitative work, but still really understands the qualitative approach as well, because I think that’s an area I don’t have a lot of expertise in on quantitative research. And so I’m hoping to continue to add to the team in that way.

Steve: So just a slight topic shift here and pulling in some of the things that you’ve talked about when you’re kind of giving the story of different roles that you’d had you described at one point moving into, I think it was a project 202 like moving into leading a team, and then that being, I think, a defining characteristic of the work you’d been doing then and since then, research, leadership is new ish, relatively speaking, it was kind of the impetus for me in this podcast, like, Hey, here’s kind of a new role that we just didn’t have. Again, if you go back a number of years, do you have, you know, a perspective on what research leadership Looks like is it different than other kinds of leadership?

Amber: Yeah, I don’t know that it’s different than other kinds of leadership. I think there’s all the similar aspects, right? You’re trying to sort of set a vision of you know, who you are and what value you provide. You’re working with folks who I love being a manager, I love understanding people’s goals and their career paths and the things that they’re trying to do, and figuring out how do we match that with the needs of the organization? How do we help someone sort of grow and get the opportunities that they want, at the same time that we’re providing, you know, the value that we need to provide to a company. And so that’s one of my favorite things. And then thinking about, I guess, as a leader, you need to really, again, understand the business understand and keep in contact with a lot of different folks know how to grow your team, how to think about, I guess one of the big decisions with research that might be a little bit different, but is similar to things like data To science or QA, it’s like you can have a centralized team or you can have a distributed team. There’s something about how you organize yourself and how you work with parts of the organization that the leader really has to have a vision around that. And the kind of, again, I would call it like principles, but like, what are the values or principles for how your team is going to operate? So like, one of the things we talk about a lot is we don’t want anyone to have any barriers to interact with us. So we don’t require you know, somebody to go through a forum or put in a request or anything like that. We do office hours, you can ping us on slack. You can, you know, there’s all these different ways that you can interact with us. So I think kind of setting those values for how your group operates is really important.

Steve: Given that you’re someone who’s done research many points in your career, does that influence or help define the way that you approach being a manager for research?

Amber: That’s a good question. I mean, I think just trying to About I don’t know if this is a good example. But when we were a little bit over a year ago, like helping figure out what we wanted to do our goals for the next year, this was actually driven by one of the researchers, Liz, she created a survey and went and did you know, one on one interviews with everybody on the team? So we actually kind of researched the situation that was happening in order to determine let’s say, do we need to provide more education around, you know, writing a discussion guide or something like that. So we kind of used our own approach of research to understand our own situation, come up with ideas and prioritize them. I guess that that skill of listening and talking to people and being able to analyze a lot of information and synthesize it is a skill set that I find valuable every day. The ability to kind of like take that synthesis and make a plan, organize it and prioritize things has been very handy as a leader.

Steve: So there’s an example of superpowers that researchers have that you have given your career that influence support your management work. Do you have other superpowers? Or things in your background that maybe that we haven’t talked about that you find yourself drawing from?

Amber: It’s a good question. I don’t know if it’s from my background, or just like, my personality or what exactly I just say, like being able to simplify complexity into really clear communications, whether that’s written or you know, a slide deck or whatever it is, is something that I’ve honed over time, right? It’s it’s something that I especially like, being a consultant, and then you know, research like how do you how do you take all this information and how do you put something in front of somebody that they’re going to be able to remember or respond to, or, you know, kind of take in? So thinking about that. And then I think to just like, researchers have to have to have a plan. Right, like you’re going in to do research. You have to, there’s a lot of like logistical things. There’s a lot of strategic things too. And I think being able to both think high level and also think, you know, tactically are really important. So, okay, we need to get you know, in in six months, we need to understand the answer to this huge question. How are we going to get there? I think researchers, you know, have to figure out how they’re going to get to that end goal. designs the same way, right, you have to kind of break things down into manageable chunks and kind of make this plan. And I think that’s, that’s a really good skill, like, a lot of people, I would say a lot of people but I think some people have, you know, visions of something they want to get to, but don’t have a way to kind of break it down and figure out how to move to that vision. And I think researchers and designers are skilled in that.

Steve: It just makes me wonder, you know, if we sort of go see researchers and they’re off work lives, like what kinds of life skills what do we see or what kinds of applications so some of these researcher strengths would we see in the non work parts people’s lives. Because I think you’re right that we’ve talked about things like storytelling and complexity and planning and just to those things manifest, or do we shut those things off? Or are there other things that kind of come to the fore when we’re quote on the job?

Amber: I mean, I guess the other thing I see from a lot of researchers and I love is just their curiosity in other areas. So what obscure thing, are they off kind of looking at reading about learning about that they then, you know, take something from and bring it back into their work. Like, I always find that fascinating. I think that’s what I loved about school, you know, grad school, undergrad, where you’re learning about all these different subjects and all of a sudden you find that something about acupuncture and the philosophy like applies to a design project that you’re doing. So I think, you know, having those curiosities and finding those connections are really important.

Steve: So there’s anything else that you think we should talk about that ask you about?

Amber: I can’t think of anything to be honest. I think that was really fun to to share all those different things. I’m trying to think if there’s anything else,

Steve: I’ll throw another one at you then. Okay. When you think about whether it’s the field or you and your team at duo, what do you sort of look forward to in the future? Where do you think things are heading? Maybe if we say five years, do you have any vision or anticipation for where we might be at that point?

Amber: Wow, five years is a long time. I don’t know, I find those questions really, really challenging, I guess, thinking about where my team at is, that is one thing or just the field in general. You know, I do see you know, there’s so there’s so many evolutions around different aspects of technology that are happening right now that I don’t think we quite know how they’re going to play out, you know, around like, augmented reality and voice assisted technology and all these things that I don’t think we’ve quite embedded in how we do our work. Like I don’t know if there’s going to be some Different ways of us interacting with people. I mean, just even thinking about the current climate right now, you know, we’re trying to figure out how to gain contextual information when we cannot be in context with people. So is there a way to get that kind of closeness or realness with people in a way that is different than being on site with someone or different than, you know, like a diary study tool or something like that? I don’t know what that might look like. Cleaning. That’s one thing. I think I’m also, you know, there’s a lot of folks who practice like mixed methods, that’s still an area that I feel like we still have quite a separation between qualitative and quantitative specialties and not quite figuring out how to bridge that gap. So that’s something that I want to watch for in the future, like how do we really take advantage of lots of data that we have access to that could help us make decisions and learn to think what else? I don’t know. I don’t know if those are very good answers, but

Steve: they’re great. answers because my question was a terrible one. It said, what’s the future going to be? And you said, Well, here’s what I’m going to be looking for, which is a good researcher response. Yeah. here’s, here’s the signals I’ll be paying attention to.

Amber: It’s interesting. I really, I just, I don’t know. I mean, I think research is so needed, I hope that we evolve where we are not. There’s always there’s kind of like, you know, UX research. And then like my team, we call it design research. There’s research that’s, you know, in the service of sort of like product UI versus like research that’s, you know, much more looking at, you know, the customer journey and the customer lifecycle more holistically. And there’s a lot of areas that have been developed in other countries, I would say like in service design aspects in like the UK and Australia, New Zealand that just aren’t quite here in the US. So that’s something that, you know, I hope that researchers are included more or we we get that influence in other areas like public works. Like government departments that really impact people’s lives. Like I think there’s a whole spectrum of things that researchers could have a big impact on that we’re just not. There’s just those areas just aren’t really built up here yet.

Steve: So it’s it’s a demand issue, not a supply issue. Yeah,

Amber: for sure. Yeah.

Steve: This has been very interesting and lots to think about. I just really appreciate you sharing so much about, you know, your own path and all the great work that you’ve been doing it to security and thanks again for being on the podcast.

Amber: Thank you, Steve. I enjoyed the conversation as always, and guys record to next time we get to catch up.

This episode of Dollars to Donuts features my interview with Julia Nelson, the Director Of Research at MOO.

All researchers say to some degree that they don’t necessarily have a traditional background when they come into the research field. But I think there’s a lot of strength in welcoming people with different perspectives onto your team, so someone who used to be a designer or someone who comes from a more academic background or someone who comes from a completely different application of qualitative research, there’s an element of resilience and perspective that that lends to a team which is the sum is greater than the parts, and that’s something that is crucial to seek out on a research team. – Julia Nelson

Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.

Steve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.

Over the past few weeks many of us have spent a lot of time on Zoom, on FaceTime, or Google Hangout, or whatever, for work, for meetups, for catching up with family and friends, for celebrations and holidays, and for other newly urgent reasons. I’m not referring to relatively passive consumption of all the “new” experiences, from film festivals to talks, museums, fundraising comedy festivals, musical performances all in addition to the television and Netflix and Hulu, but rather these active conversations when you are participating, where you are seen, and heard.

On one hand, we have turned to this alternative form because we must, we feel an imperative to connect with others, to support each other while also drawing strength from each other’s mediated glitching presence, and in the crises, this is the only way. And maybe there’s even a bit of the trend at work here, because this is just what we’re doing now.

Perhaps you’ve heard the term Zoom fatigue, especially acute for those who are expected to follow a work schedule like the one from the before times, all online, and then find themselves using their off-work hours in the very same mode. Because it’s hard. I mean, really hard.

It’s hard when people who can’t stop talking for hours when hanging out on a back porch find themselves staring at each other through a screen and just don’t know what to say, and don’t have a clue why that is.

It’s hard when members of a group have different levels of familiarity with the norms the technology demands, such as knowing to mute yourself so that the video doesn’t switch over to you when you rustle papers, even though someone else is talking.

It’s hard when convenors of our online meetings don’t know about those norms either, and don’t know the additional facilitator labor required to ensure compliance so that one person can’t accidentally stomp all over the fragile emergent communal vibe.

And on and on.

I went to a professional meetup that included a fascinating recap of many of the technologies over the decades that have tried to connect people remotely over video so that they can collaborate. And yet the meeting began with the familiar fumbling aloud in search of the sharing screen button, the host squinting away from the camera, at a second monitor, navigating the intricacies of the interface while we waited patiently but increasingly felt disconnected instead of connected.

I went to a community meeting where someone gave a spiritual musical performance full of commitment and enthusiasm. I imagine that many of the attendees felt this was a beautiful gift, but honestly I was terribly self-conscious, knowing that I was on-screen myself and had to manage my own performance, my own reaction to something very intimate that I just was not prepared for.

I joined a family celebration where one house had such terrible latency (of which they were entirely unaware) that the conversational disconnect spiraled repeatedly into something highly comical and highly frustrating.

The flawed assumption that we can simply use this technology as a replacement for the ways we used to connect is more clearly revealed as time passes. As with anything, rather than replicating experiences, the technology transforms experiences. Right now we the users are mostly in problem-solving mode, and the problems we are solving are mostly implicit. The problem is the technology, but also it’s the user interface, it’s the intended use case versus the emergent use cases, it’s the social constraints and norms, and a whole lot more.

And this is what researchers do. We don’t come with answers, but we come with ways to look at complex problems that involve people, systems, and other people. My litany of complaints is an invitation to both incremental and innovative changes, but those changes have to be based on a deep understanding of the issues, and that’s the work.

And this is one of the things I do for clients, well, really, it’s what I do WITH clients. We unpack the hidden aspects of their products, current and future, in order to prioritize their next steps to delivering the kinds of experiences that meet their goals.

And this is something I can do for you and your team. I’d appreciate you reaching out to me to find ways that we can work together.

And now, my interview with Julia Nelson, the Director of Research at MOO.

Julia, it’s really excellent to have you on Dollars to Donuts. Thanks so much for being here.

Julia Nelson: Thank you for having me. It’s an honor to be here.

Steve: So let’s start. Yeah, talk about the work that you do and maybe talk about MOO, which is sorry, I don’t get to say very.

Julia: Absolutely. I know it’s an unusual name. Um, so mu is an online printed design company. We’ve been around since 2004. based in London, that’s where the company was started. But we have several offices in the US and we operate globally. And we’re mostly known for business cards, but our real passion is product design, particularly physical product design. So we’ve been expanding over the last few years into new things like notebooks and postcards and flyers and other things. things you can do with paper. We are a bit unusual, I think, in the sense that we in two ways in terms of how we’re structured as a company, so one is that we’re completely vertically integrated. So we control all of our manufacturing. And in fact, that is a core part of some of the, the software that we build and maintain is our own sort of infrastructure for communicating with our manufacturing operations. And we’re also a very flat organization in terms of how we’re physically structured so the headquarters in London has tech and product teams, but also most of our online marketing all of our in house industrial design team. We also have our own creative team who do all their photography, photography, videography, and customer services in house which is great because it’s super easy to walk over and drop in on a customer call anytime of the day. So I’m director of research. I’ve been at mu for about three years and four months now, I started as a head of in January of 2017. My team is quite small. Over the years, we’ve fluctuated between myself and one other researcher up to about four people in total. And I think we’ve had a bit of an interesting arc as a research team. So when I joined the company, we were more of a user research function. We reported into the design organization, which sat within the product team. And we were primarily focused on evaluative user testing on the website, but over my tenure there we’ve explicitly shifted the function to be much more of almost a research and insights function. So we not only handle user research website, but also cover design research from physical products. So in forming innovation around the things that our industrial design team is creating, and then we have sort of taken over a small portfolio of market research as well. So everything from claims testing to market sizing comes within our purview.

Steve: How did that change come about?

Julia: Um, I think that’s a really interesting question. I think it’s a combination of two things. One was sort of an internal desire within my team to make sure that we were doing more generative, more strategic work that had more of a role in setting product strategy within the company and eventually company’s strategy. So it was something we explicitly work towards, in the ways that we work with our Stakeholders internally was pushing towards that more generative research. But also we got really lucky, we had some sponsors within the company who really understood the potential of what our research function could be and really championed us with senior management. So we lucked out in that sense. But I think the combination of those two things and sort of over time being able to show people how you could leverage research insight earlier in the process earlier in the design process, or even more broadly into business decision making. It’s a it’s been part of just demonstrating the value of what we can provide and how we can provide it in different ways across the company.

Steve: All right. So you’re talking about kind of two key factors here. One is the team themselves identifying, hey, we want to work you know, If we have more than we can bring here, we want to Yes, work in this way. And then sponsors, people who, you know, were in influential decision making positions, who saw, as you said, saw the potential here.

Julia: Absolutely. Yeah. And I think it’s not just about potential, but I think I’ve been really lucky. I’ve worked with both people who have intimate knowledge of research and have worked with researchers before. And I’ve also reported into AI reporting to someone who isn’t a researcher but is really open to sort of having a dialogue about research and the value that research can bring and then scouts for opportunities for us to move into doing more strategic work that is more about core business strategy, about product strategy, etc. So I think it’s, it’s possible to have allies who may not fully understand what you do but understand enough to kind of be on the lookout for ways to leverage what your team can offer to the company.

Steve: Right. Well, when you say, I understand what you do, it makes me think about how, I mean maybe this is true of every group of professionals that have a shared practice, whether they’re accountants or engineers or you know, farmers, but I don’t know if this is unique for researchers, we like to talk about the how and not like the, the process and not the outcomes. Hmm. I don’t know if that’s if you agree with that characterization.

Julia: I do think we love I think researchers are people who love details and who enjoy the mechanics of the work. You know, I think most people I know who get into research love the process, because that’s part of what fulfills them. So we naturally gravitate towards talking to talking to the process. And I think, I think when you’re talking to people when you’re talking to state holder’s from, you know, whether it’s people in different departments or senior managers, I think you picked up on something right, which is the language of outcomes? What is it going to deliver? How is it going to help us? What choices that going to enable us to make? How is it going to reduce risk speaking in the language of business tends to resonate more? Right? It’s the How will this be applicable to this challenge or problem or decision that I have to make, as, you know, the VP of marketing or the VP of product, etc.

Steve: And this comes up, I think, a lot in you know, when researchers get together they talk about, you know, having impact and having influence. Mm hmm. What you’re saying is, you know, probably the number one opportunity for researchers to do better, but I want to go back to one of the things he pointed out, I think, which is the is the consequences of doing that. Well, I guess I don’t, I don’t want to imply causality. causality to say that but you’re describing Where people may not know what you do, but they know what kinds of outcomes you can provide. And they’re on the lookout for those opportunities. So it’s not we should get those folks to do some research. It’s those folks can provide this kind of information that we need right here.

Julia: Absolutely. I think that’s exactly the right way to think about it. And you’ve articulated it more clearly than I have before. Yeah, it’s the knowing what, what’s this? I don’t want to say tangible because research can be quite an intangible thing, but it helps articulate that outcome that it’s going to give them that way. It’s going to enable their work. So even if they don’t understand the mechanics of how the research will get done And I think that’s almost easier for some people to some people, for other stakeholders to run with, you know, even if they can’t explain how the research will be done or the method or the you know, sample size or any of that. Being able to say this will help us do x within the business I think is incredibly powerful and persuasive not only for them, but for them to make the case on your behalf to others. So I often when I talk about the value of research, I often talk about risk and decision making. Rather than what specifically we might be doing.

Steve: I mean, I so I agree, and then I feel anxious when I

Julia: Ah, okay.

Steve: And so some of this is just I can make statements like as researchers, we and I may just be talking about my, what my bad habits are. But I don’t think we can just dig into this, sir, the framing here because I feel like I or researchers, you know, we do talk about the details of the process as a way to I mean, sometimes we call it we call this quality or sometimes we call it, you know, yeah, rigor. Mm hmm. Because there are so as a researcher, you know how to make process decisions that will get to the right. The right outcomes. Yes. And, and I think we’re often given requests that include process details.

Julia: Yeah.

Steve: Here’s the kind of people we want you to talk to, here’s how many, here’s how quickly, here’s what we want you to do with them. And so you can find yourself pulled into a process discussion. You know, if we don’t do this, then we’re obviously setting people up to be give us favorable responses this way. You know, that’s the guidance that we’re asked to provide.

Julia: Absolutely.

Steve: So, what’s my question for you? Is and maybe you’re just Describing a more more mature state where not having those conversations where the, you know, the request is for outcomes, and the responsibility for process decision falls to you.

Julia: Yeah, I think we’ve been really lucky in that sense. I think that’s been one of our triumphs over the years that I’ve been there is really consistent. I didn’t mean to negate the need to speak to process. But I think we’ve managed to, in some ways, educate our audience that what we really need to be asking ourselves is what do we want to know? And what are we going to do with that information? And we’re happy to talk your ears off about process but we’re going to go away and make a decision about the options for that process that are rigorous enough to give us confidence and in the quality of our output. And there’s always trade offs, right? They’re always time pressures, there’s always resource pressures. So our job as researchers is to help Deliver the insights that you need to address the questions that you want to answer. But also to figure out the how of how to do it and how to how to getting off track here a little bit. Let me restart on my answer there. I’m sorry, said that not to negate processed in it. And I think I’m speaking to more of the advocacy around research. So I think arming people to speak to the outcomes that we can provide, to make the case for the role that we can play within the organization is a different thing from speaking of what are the goals of the research process, or goals of research and the process behind it? So I didn’t say that to negate the importance of the process aspect. But yeah, I think that’s been one of our successes. Over my time there is we actually have gotten people to to let go of process a little bit more We certainly get pressure around time that sort of everyone always needs their insights yesterday. And we have to manage that as researchers, but I think I had a just illustrate it. I think that’s one of our successes within the culture. So I was in a meeting where another researcher was talking a group of stakeholders through a sort of brainstorming around, you know, the research objectives and the research questions and then spoke a bit to potential process. And I’m saying if you see the word ethnography, I get really excited. So I started geeking out and every single stakeholder in that room, who is not a researcher turned around and said, Julia, you lecture us about not prescribing process, you’re not allowed to prescribe process here. So we’ve kind of managed to at least train them a little bit to let go of trying to design the process for us. And to trust that we’re going to walk them through how and why we chose the approach. If we need to go for something lean Where we’re making decisions about how to take a leaner approach, etc. And I think that’s been a bit of a cultural win on our part. I hope that answered the question.

Steve: Yes. And I note that you don’t keep them out of the loop around. Yes,

Julia: absolutely not. Yeah, the more they’re in the process, the better. Not only, you know, just keeping we walk them through the research design, but we try to engage our stakeholders as much as possible in the process, I think is every every research, you know, here, I’m gonna generalize community, but there’s so much power and having those stakeholders in the room. So as much as possible, we try to encourage them to get involved.

Steve: Right, well, so I think you’re, you’re identifying it’s not a contradiction, but it’s an interesting challenge. And maybe this is why we’re not it’s not accountancy or so. other profession? Because I don’t know, because you’re very clear, this isn’t a black box. Right? It’s not we want these outcomes. Okay, we’ll go get them for you know, and yes to collaborate and have them participate helps the results for all the reasons that we know. But you’ve maybe, you know, as you talk about this education process has clarified where the responsibilities are where the what your value add is here, what your unique skill set is, is not is in being able to plan how to best meet their goals.

Julia: Absolutely. Absolutely.

Steve: So this is I mean, I feel like we’re hearing the, you know, the evolution towards a really strong positioning, which is still collaborative and still engaging but is relieving them of the responsibility relieving stakeholders to the responsible for things that just make their job harder, and that are actually, you know, within your, within your skill set.

Julia: Yeah. And it’s not been instant, it’s definitely come over time. It’s definitely a trust that we had to build. And also, you know, a lot of the stakeholders who we work with in different parts of the company, we’ve now been working with them for one or two or three years. So we’ve managed to, they’ve seen enough of the process and they’ve seen enough of our decision making, and we’ve walked them through a lot of that to I think, have a level of, of confidence that we’re going to define something that is both that is rigorous and will deliver what they need to know. So it’s an earned trust. It’s not an instant trust. And people are still anxious about process. I think we try and take time to explain why we’re taking the approach that we’re taking. Why we’re not considering something else, or why We feel this is the best option. So I think really speaking to that, why have we made these choices is really helpful in building their confidence around our decision making around how we design the work that we do

Steve: this this dynamic, you know, the, the trust that you’re earning, and then how you, you know, take ownership of getting towards outcomes, but you engage them in the way that you see variations. I mean, in our conversation so far, you’ve described like, a range of types of business that the organization has, and yeah, different and different work that you’re doing towards those. So, you know, how might it be different if you’re working on the industrial design of a, of a notebook versus maybe something more strategic versus something on the websites, etc, etc. Does that lead to variations in what we’re talking about?

Julia: Absolutely. I mean, I think I think there’s different cultures within a company, right? And there’s different points at which people start in their understanding of design and of design or user research. So I think we struggle. I wouldn’t call it struggle. I think we have a natural sort of common language with folks in UX design and product to have a specific experience around user research and the role that it can play in product development. Whereas the language that I speak with industrial designers is sort of, we have a similar understanding of process from in terms of the framework of design thinking and the need for sort of deep customer empathy to inform your ideation and prototyping etc. But a lot of industrial designers don’t necessarily come from a human centered design background. So even within our industrial design team, they have different levels. have experience doing research with users of a product or buyers of a product from a market research perspective, so, you know, with there’s different sort of pockets of education that need to happen there about how do we translate what we’re understanding into choices about form and the tactical experience of a product. Which are different conversations that I would be having elsewhere, partially because of subject matter, but also because they’re starting with a different understanding of some of the basics of Human Centered Design, etc, and how it plays out in practice. Whereas on the market research side, you know, they have, there’s parts of the company that are much more used to market research surveys, focus groups, etc, which isn’t necessarily what we do. Although it’s, you know, it’s parallel in some ways. Many of the techniques that we use are also used in more market research. But you know, their understanding of what research is and how it can be used and what do you get, what’s the insight you get out of it is it is entirely different from the industrial designers and the product design portion of the company. So I think we have different conversations and also different challenges with those different cultures within the company in terms of how we communicate and advocate for what we do. And also speak to the limitations of what we have what we can deliver.

Steve: So I’m thinking about, you know, maybe two dimensions here one is time, and you’ve kind of talked about, you know, the Earned trust over time. And the other is maybe the horizontal axis of in my in the chart, I’m waving my hands to illustrate. Yeah, I mean, one way, maybe I’ve screwed it up because time should be horizontal and the, the, the different parts of the organization that are at have different cultures and sort of think differently about it. And so you approach each of them differently. And this is all evolving over time. Yep. So, in the moments in any particular moments where you have to you and your team have to bring something different to bear maybe more patience or simplification or more speed and more camaraderie or kind of shared language, as you said,

Julia: Yeah.

Steve: How do you do you have a way of keeping that all in balance or, you know, the word that comes to mind for me, is, uh, is around patience. But that’s, that’s my, that’s overlay on to what your experiences mean. How do you When, when, when things go when there’s probably an optimum situation here, these people are easiest to work with and they have been? Yeah, you know, once you’ve tasted Did that and sort of, you know, achieved something, you know, like a micro Nirvana for the researchers role. You know, how do you remember? How do you create for yourself the long view? I mean, you’ve been doing this for a few years in this organization? How do you kind of put the individual experiences in context of that long view that you’ve been working towards?

Julia: Oh, that’s a tricky question. Tell me more about what you mean, my long view.

Steve: The, you’re describing a change over time, when you talk about trust, yeah, means at a point in time a, it’s going there’s gonna be more stress or concern in the party, collaborating. And you might be thinking, yeah, you know, I’m here right now with this team, and I Want, again somewhere else with this team in the future? And I mean, you’re kind of saying when you say earn trust, yeah, that makes me think there’s not a shortcut you have to work and kind of get better and get better and get better.

Julia: Yeah.

Well, I think sometimes it’s just about a lack of experience with research. It’s sometimes hard to visualize what something can tell you before you’ve, you’ve been through it. So try to think about how we’ve tackled this. I mean, I think we’re lucky in that one victory makes you more confident about the next victory. But I think I think what helps me is putting it in the perspective of the field as a whole and I talked to a lot of designers about this. Which is that in some ways design, while also very old is a very young function for many companies. And similarly, although user research or design research is an established field and is establishing many companies, it’s not as ubiquitous as like, as you said accountancy or HR are easy to understand in a way because they’re sort of ingrained in the institutions we work in. Whereas research and design are I want to say younger, but maybe more amorphous for people. So I think thinking about the fact that this is a field that is still new for many companies, that is still new for many people in the workplace, who maybe haven’t interacted with researchers before. And then it’s also rapidly evolving. I think that helps me get a bit of perspective on you know, when I may be feel challenged, working With a set of stakeholders is to just put it in that context of, we have different levels of experience with a, with a, with this arena or this discipline. And that’s, that helps me step back to, you know, I have to start with the baby steps, I have to start with ensuring we have shared assumptions and shared language and a shared understanding of what user research or design research can be and can do. So I think, I think that for some reason, that helps me with the frustration to step up to the bigger picture. And to put it in context to the wider discipline as something that is not as embedded or institutionalized as an HR function. And just to be a bit Kinder about you know, it’s it’s going to take time to educate people and for people to wrap their minds around it.

Steve: Right I think your point about the hit the youth or You know, duration of the field versus in the particular company? Yeah. Right. And, you know, movie said 2004, I think, right? Yes, it’s a 16 year old company, which is, maybe for a lot of people, an old company in a lot of it. Hmm. So you could be a researcher in a company where your career as a researcher exceeds the duration that company has been in existence.

Julia: Exactly.

Steve: And maybe the professional life of the people that you’re working with? Yeah, cuz we have people of all ages working in companies. So if for five years, your colleague may only have been out of school for two years. Yep. So yeah, I think that that’s a really good point to kind of contextualize where everybody is at.

Julia: Yeah. And it goes both ways, right. I think on the marketing side, there’s a much more Establish discipline around market research, which shapes certain expectations around the nature of the qualitative research that we do, which aren’t, you know, people come in with preconceived notions of what research looks like because of experience with that discipline. So it’s not always that they don’t have experience with the field, you know, they might be younger, but it’s out to that they might come from maybe a more traditional background or have less exposure to a more agile way of working or to, to the discipline as a whole.

Steve: So what are ways for, you know, people in the come up in the design research or user research tradition and people in the market research tradition? What are ways for them to collaborate and coexist effectively?

Julia: Hmm, I think it’s about knowing about how you fit together. So it comes a bit back to methodology and to understand that difference between sort of opinion data market research and more of the user and the design experience and how they can complement each other rather than conflict. So I think having an intimate understanding, and I’m still working on this, I’m not a market researcher by training. So my growth over the past couple years has been immersing myself more in that field and trying to understand their disciplinary heritage and also understand methodologically. What can those methods bring to the table? And where does that end? And where does design slash user research begin? I think being able to speak to that and to have conversations with people about that fit, whether they’re market researchers or they’re people within your company who kind of need both, or maybe need one and not the other. I think getting really clear on how you explain that to people that complementarity and where one begins the other ends is really helpful to communicating how they fit and when one is appropriate when the other is approved. Corporate how they can complement each other, and etc.

Steve: So we’re talking, I think, I think you’re talking about processes or methods, right? Yes. complementarity?

Julia: Um, yes. Well, knowing what each of them deliver. So, you know, my market research is going to give me insight on things like, potentially, you know, stated behavior and on opinions and that sort of thing. And what is it that that can help me understand, and what are the limitations of that work? And then how can I leverage that alongside something that’s coming from more of a design research perspective? How can we leverage the strength of both of those disciplines to work together? So So yes, from a methodology perspective, but more than Uber level?

Steve: How that how those two things fit? And then how does that look from a team perspective? Have you have the activities of market research? And yeah, I mean, you can talk more about kind of what your team looks like. But is there a person who I where the market research hat and I, where’s the user research hat? Because that’s, that’s, I think, key to making, what you’re talking about happen is, is the individuals and how they can guess, you know, complement each other.

Julia: Yeah. So the current structure of my team is, is a sort of a mix of generalists and specialists. So we are, we are because we’re very small. We are a centralized research function. And also because we not only service our product teams and crews, but also we work across the company. So myself and our other core researcher, at the moment are more generalists. We work across the board on a variety of different projects. I spend about 50% of my time as an individual contributor as well as manager. And then we have a couple people who are disciplinary specialists who are who are part time and who help sort of lead or advise. They not only do the work, but they provide that methodological expertise. So we have a part time market researcher we work with out in New York, and a very experienced design researcher for physical products who we work with out of Boston. And so that kind of odd some of the the experience and the methodological, it complements sort of the more generalist skills that myself and the other senior design researcher bring to the team. They can provide that injection of disciplines specific perspective and experience.

Steve: You’ve used generalists and specialists. Hmm. This came up in a discussion I was in recently Someone was asking as a researcher, should they be a generalist or specialist and it just made me wonder, like, what do we mean by that? So you Great question. What what are those terms mean for you?

Julia: Ah, for me, um I think it’s a combination of I’m thinking about the two people on my team. So it can be either skillset or experience. So, my market researcher has a specific set of methodological skills and also skills around survey design that she brings to the table which strengthen that element of our practice. Whereas our physical product researcher has a lot of experience thinking about sort of the physical medium of design research. So he brings perspective on what we’re looking for. From a from an industrial design perspective, what we need to understand how we need to explore some of the physical dimensions of those products in ways that are different than what we would do online. So that’s a bit more less methodological and more experiential expertise that he brings to the table. So I think it can be in either of those two dimensions. I mean, I think over the course of a career, you inevitably become a bit of a generalist in the sense of at least I find that that researchers are always hungry to learn new things and to explore new methods and to understand how they can utilize them in their work. So I think I think we all have a little bit of a gelatin generalist tendency within research.

Steve: So I’m going to ask the magic wand question. Okay. Apologies for asking a researcher the magic one question. With a genie appeared and offered to wave a magic wand and said you could add You could add two people to your team. How would you start to characterize what you would look for?

Julia: generalist or specialist? Do you want me to speak more generally to how I approach hiring or more team structure?

Steve: Well, let’s start with how you approach hiring.

Julia: Hmm. Okay. How do I approach hiring? Um, I think I, and this might be a personal preference. So I think that the key things that I look for when I’m hiring someone are, I focus a lot on research design. So do people understand can they elicit the objectives of a project can they design, a process a project that will address those objections? That’s something really, really critical to me in terms of rigor is making sure that people are flexible and adept at really thinking through that aspect to the process and defining the objectives of research, developing a design that’s, that’s rigorous and and effective and delivering the kinds of insights that we want. That we need to know. And I look for but a lot of it beyond that. So certainly there’s core skills around interviewing and observing and stuff like that, which I think are were difficult to assess in most traditional hiring processes. But you get a sense of that through how someone interviews and how they interact, the questions that they ask in the hiring process and through their portfolio, but I also tend to look for A couple of intangible qualities. I look for a growth mindset. Because research is such an experiential field in some ways, certainly. It there’s a lot more formalized education around user research, then there was more than then there has been in the past, but a lot of what you learn, you learn on the job. So I think having a hunger to learn a hunger to experiment, a desire for constant improvement. I think that really set someone up for success as a researcher because it’s not ability to learn something that the desire to try new things that allows you to expand your skill set. So that to me is a really strong signal of someone who I want to have on my team because of the potential for growth that they bring to the table. I also look for people who…I think this comes back to it, but who but who turned failure into learning so that this is a bit of growth mindset as well, but who can show me that they’ve taken something, applied it and improve their practice. And then I think somebody who can adapt who’s flexible when it comes to methodology, who can think through practical challenges, like what happens if I have half the time and half the money? And, and can adapt their approach to to those constraints? And then stakeholder management is the last one is this someone who has a has dealt with challenging stakeholders, even if it’s not in a research context, but who has sort of the emotional intelligence to think through how do I meet this person where they are How do I build a relationship with this person? How do I? How do I educate and advocate with them for how research can play a role in the work that they do day to day? So I look for kind of those three things growth, mindset, ability to learn from failure, stakeholder management and adaptability.

Steve: So for those last those last four, how’re our people communicating that to you? How does How do you find that from someone? Hmm.

Julia: It’s a lot of behavioral interviewing, it’s a lot of talking about past experiences and how they’ve dealt with past situations. Even if again, it’s not necessarily in a research, research traditional research role. Some of it we get to through, we do do task interviews, but they’re always I dislike giving people take home tasks, because I think that’s a bit unfair. We do do a thinking exercise with them, where we talk about how they might approach different challenges in the context of designing a project. So it’s a lot of behavioral questions, and a lot of just seeing their thought process about how they might approach a problem that’s presented to them.

Steve: So the other piece you said we might explore is the composition of your team.

Julia : Sure, at the moment, it’s extremely tiny. So the composition and the structure of my team has changed over time when we started and we were more embedded within the product function. We still weren’t large enough to necessarily be embedded in crews. There were two of us for for 12 crews. So we had to take almost by default, a bit more of a centralized approach. But we did try and sort of service different sets of crews. So in that instance, we were set up more around the product structure. So with different people looking after crews who kind of work on similar aspects of our website and our creation experience, etc. So they can build that depth of knowledge around that particular element of the product in the website. As we’ve sort of shifted within the organization, we actually moved out of the product function under the VP of strategy for a while. And then under the VP of commercial planning and strategy, we’ve evolved our structure to adapt to broader changes within the business. So mu has been working hard to align the company a lot more cross functionally around some of the key outcomes that we’re trying to achieve. So now we’re structured around there’s three major outcomes the company is pursuing and there’s a researcher and Is strip at the strategic level within each of those outcomes, which have steering groups that look after the outcome. So with that person is sort of responsible for overseeing for being engaged in the thinking and the decision making of the of the strategic group that’s guiding that outcome within the business. Being able to identify research needs, being able to work to shape that research and define projects and then engage with me and with a broader team to figure out how we deliver on that. So we’ve moved into something that is a little bit as a company, we’ve transformed in a way that has sort of transformed the nature of the team. So yeah, so we’ve got those three people embedded in each of those outcomes or two, and we’re seeking a third and then the specialist support kind of sits across all three of those. So for our part timers who work on through quantitative market research and our physical product researcher, they they service the needs or they work with A researcher who’s sort of leading the work for each of those outcomes to deliver what we need to.

Steve: So the person that’s embedded in one of these outcome groups, they’re involved in planning, assessing what research needs to be done and planning for that, is that the only or the primary way that what research is gonna happen gets determined is that is that what’s happening? I would

Julia : say the majority, but not all of it. So we also get asked to do research by our Executive Committee. So there are sort of business level strategic projects, which are fewer and further between, but they will make specific requests advice. And then we also for sort of day to day evaluative research, usability testing, things that need to be done and more of the executional side of design. We work extremely closely with our UX design team, to sort of empower them to do some of the research and also to oversee the work that they do or to support them on that work, depending on the needs of the designer and their experience. With UX research there, we help them drive that sort of more tactical aspect of work, or we step in to do that where there’s, there’s not someone available to do that.

Steve: I want to go back to something you’ve said, I don’t know, maybe 15 seconds ago, when you were talking about understanding the point of view of people who maybe have less experience or just different levels of familiarity with research, you made the observation that despite its you, as a field, you commented that research is rapidly changing. And I just want to go way back to that and say, in what ways do you see it rapidly changing?

Julia : I think there’s a few different forces of change that I think about well, so independent of your context, and what’s happening. I think there is an aspect of as a researcher, you’re always evolving and changing within your own practices of researcher. I’ve often said to people that I don’t think anyone comes into this field as a full stack researcher. You don’t know all the methods. You have different levels of experience with different aspects of research. You know, you might be an excellent interviewer, but project management is a challenge. So I think even within our own practices, we are always evolving as researchers. And I also think it attracts people who love that process of learning and evolving their practices they go. I think the other ways that I see things changing are, some of them are organizational. You know, going back to what we were saying earlier about people’s understanding of design and research or innovation, there’s a cultural evolution that happens within companies with their experience with those fields. So I think as you mature within an organization that also feels like a source of change. And then I think, you know, technology wise, we’re moving into new technologies and to voice into a high into these new areas and bit sorry, not Bitcoin. I’m completely blanking on what the Bitcoin technology is. I’m so sorry, blockchain blockchain. So understanding these new contexts and these new interactions is another source. A change in evolution in our discipline, learning to how our work applies or can can enable those things, I think is another continuous education element to what we do.

Steve: You had this great line that, you know, no one comes into the field full stack researcher. Yeah, leads me to ask you, was there a point at which you learned that this was a thing? This was a field or a practice?

Julia : Yes. And I also embarrassingly sort of was very up close to its emergence without really realizing it was there. So I did my undergrad at Stanford, and I remember cycling past the D school probably 1000 times and I think I went in there exactly once. my undergrad. So right at the height of when, you know, the D school was emerging as a voice and design thinking, I sort of missed the birth where the emergence of that is a mainstream discipline. I was there and I missed it. But I was sort of aware of design thinking and innovation and design research as a discipline, I think tangentially from that point. And I spent about 15 years in the Bay Area. So I had a couple friends who were one was a design researcher at IDEO. And the other was a user researcher, who is not quite senior in Seattle. So I think in some ways, I got exposed to the idea that this field existed through just those friendships and knowing a little bit about what they did. And I remember feeling that tweak of interest when talking to them about what they did and what they enjoyed about it. But I didn’t so I always had it kind of in the back of my head is something that I might potentially be interested in professionally, but it didn’t really click for me as something that I wanted to make part of my career until 10 years into another career. So I did my undergrad work in anthropology, specializing and looking at the intersection between culture and environment. So I was always interested in people and how people interacted with their physical environment and sort of used the natural world around them. And then I went into sustainability for 10 years working first as a consultant. And then in house in the corporate social responsibility and sustainability were my work was largely around doing extensive qualitative fieldwork. So I specialized in community engagement and community development around mining oil and gas and renewable energy projects. So I remember so I was about 10 years into this career. And I remember having a moment when I was working in house for one of the largest Canadian gold miners in the world. And my job was to go to different sites and engage in field work with the communities around these mine sites and really understand the impacts and the effectiveness of how the company was managing social impacts on these communities. And I just remember having this thought we were talking about local procurement and using procurement to encourage economic development around mine sites. And I just thought we could be doing so much more. We’re doing a lot of stuff that’s about managing risk, and mitigating impacts. But we’re not talking about opportunity. And we’re not talking about innovation. So I had a bit of an aha moment around that and just thought, I really want to be doing something that’s more on that innovation side of the picture. So I went back to school in my early 30s, and I decided to go get an MBA because I knew I wanted to do something around strategy and business and innovation and I wasn’t quite sure what it was and ended up at Cambridge, which is why I’m in the UK today. But I took a number of classes from professors there who specialized in service design and sort of innovation more broadly. And I really enjoyed that and that felt like the thing that I was looking for, and we did a few small designs. research projects in that context and not that really gave me a sense for, oh, this is a thing that I can do. I’m enjoying it. it leverages all this past experience that I have doing qualitative research in a different context. I think this is something that I want to pursue. And that was what led me to move. Well, I ended up finding something that was the right fit for taking a dive into the field.

Steve: In addition to the experience with field work that you pulled forward, are there other things from other points in your career or your education that you see yourself adapting or leaning on in the work that you’re doing now?

Julia : Absolutely. I think a lot of the business elements of what I did the training that being a consultant gave me around speaking to strategy and speaking to a business audience is something I leverage all the time. I think all of my managerial experience comes from managing teams and managing reports in that context. And a lot of the experience and guidance that I got from other people during that time is something I use a lot in managing my own team. Now, I did a lot of intensive facilitation work when I was a consultant, large cross functional, multi party, sort of international Working Group type of stuff, which I think gave me a lot of confidence around running meetings and facilitating decision making, and a bit of that conflict management elements. So all of that comes to bear in my stakeholder management. What else? I think all of that I think it gives me a bit of perspective on some of the organizational and business challenges that we face as researchers sort of like having a like a spare in your toolbox. You can draw on this different wealth of experience in a completely different fields, but that ends up being surprisingly applicable in what you do.

Steve: Is there anything different about managing researchers Specifically.

Julia : Hmm, I think I think we have some inherent organizational challenges. I think knowing what the long term career path is for researchers seems to be challenging for many of my peers, right? Talk to in the industry, like there’s the managerial path. So moving into managing researchers, but what is a career look like for someone who is an individual contributor? What is that arc of growth look like over 10 1520 years? How do we help people figure out what that looks like for them? Sometimes they hear from my peers that there’s a sense that what we can do as researchers is capped, that there’s a limit.

Steve: So you are having these conversations with your peers about what’s the career path for researchers and it sounds like this is a question mark in the field right now.

Julia : I think it does feel a little bit that way. Is that is that something you feel like you’ve heard from others more broadly?

Steve: You know, I hear phrases like career ladders. Yeah. And so on, and that these are starting to get defined for Yes, I haven’t looked at them. Mm hmm. And I don’t know that I have this sort of expertise to assess them. Mm hmm. You know, and I think about your point about the field rapidly changing, or I think about my own career where the field that I’m in now is not the field that I came into. Yes. Field 10 years ago is not the field. But I don’t know what the what the period of that transformation like. But yeah, if you say, well, what’s what is an individual contributor going to be doing in 10 years? Well, in what field? Will they be working is? Yes. Like this one? Yes. Yeah. I find that challenging. And so so I think your point is like that’s, that’s a challenge to manage a team manage individuals on your team be successful because the path for them is hard to plan for?

Julia : Yeah, I think so. And I think I think it’s also quite individual. And this might be because of the nature of research as a distance. To plan so if you were working in say, operations or in factory work or as a lawyer, there’s a very defined sort of ladder that you might go up if you’re a lawyer, you know, you start as an associate and you work your way up to partner, Managing Director, you know, that’s very well understood in a very clear sort of hierarchical path. I think for researchers, there’s, there’s many, the career path might look a little bit less linear. There’s certainly like a hierarchy of experience and potentially responsibility if you’re on the managerial side. But I think something that I find both sort of freeing and challenging when matching researchers is it’s very individual what you find fulfilling and motivating, I think to a greater degree in the field in our field rather than another fields because of the breadth of what we do. You know, you can grow in your skills constantly. You can grow into new skills, you can develop experience in new areas you can try out new methodologies. But because there isn’t a defined career path to sort of hang your aspirations on, it becomes much more individual to figure out what motivates you as a researcher, are you motivated by learning how to manage others? Are you motivated by the advocacy element of what we do? Are you motivated with by the joy of learning new methodologies that you can apply to your work? Are you more excited about being able to work in different products or different aspects of the business? So it ends up being? I think it’s both joyful and also challenging when managing a team of researchers to support that diversity in people’s aspirations and to create opportunities for their growth that are going to look different from person to person. So to make this more concrete, I have managed researchers who Junior researchers who’ve realized that they’re much more interested in the ideation and maybe a little bit of the design element of what we’re doing and so for them focused On ideation skills is the thing that motivates them on a day to day basis. I’ve had researchers who’ve gone into UX design because they realized they were passionate about not only getting the insights, but the translation into a solution. And I’ve had others for whom you know, what motivates them as being an expert. So what they want to do is be given opportunities to master something new. So I think learning to support those ambitions and to carve out opportunities that support those different ways of growing is a bit more challenging in our function than maybe in more traditional fields,

Steve: in maps really well to what you said about looking for a growth mindset in meeting candidates and you listed a number of different ways that that candidate might have shown in the past or indicate currently that they are interested in growing and I think this is something that’s, that’s extended even further, you listed even more ways that someone could grow and I’m sure nowhere near an exhaustive list. Yeah. You know, when you describe this, and makes me want to create this really awful matrix, which is like some grid of all the expertise is that a team? We’ve been given the challenges and so on, so that it’s balanced and you don’t? Yes. And you have like a bit of everything. Yeah. And I say awful because you made a really important point that this is very individualized. So yes, you can’t put people in those little slots of that grid. So your your team ends up looking more organic, or there’s a different model that comes out of helping individuals follow that, yes. seems too hard to plan for or control or undesirable to control.

Julia : Yeah, but I think there that’s also where the magic of how you build a team comes in, right? We know while there’s sort of, to me, there’s more of an individual, maybe career pathing that that tends to happen. It’s also about identifying what are the complimentary skills that each member brings to that team and how do you bring in people who bring who filled I don’t To say gaps, because they’re not necessarily gaps, but who add that additional, whether it’s expertise or skill set or experience that complements the other people within the team. So I think that’s where you’re not just helping guide people down their individual career paths, but you’re also playing conductor for the team as a whole and figuring out how do you bring together those different strengths and and leverage them as a group so that you take advantage of how they can complement each other and how they can fill out your practice as a whole.

Steve: Is there anything that we should talk about today that we haven’t got to?

Julia : The ideal question, I think, I feel very passionately about encouraging people who don’t necessarily have traditional backgrounds to jump into this fields. I think all researchers say to some degree that they don’t necessarily have a traditional background when they when they come into research, but I think there’s a lot of strength in it. welcoming people with different perspectives on to your team. So someone who used to be a designer or someone who maybe comes from a more academic background or someone who comes from a completely different application of qualitative research, I think there is an element of resilience and perspective that that lens to a team, which is sort of the sum is greater than the parts. And I think that’s something that is crucial to seek out on a research team because I think it’s strengthens you and strengthens your practice as a team.

Steve: Certainly, I see a lot of I’m trying to transition into into UX research. And what comes after that usually is how do I do that? Yeah, you’re pointing out that it behooves the team to welcome those people. Is there more like how does how does the field create that welcome?

Julia : How does the field create that welcome, I think there’s a lot more we can do to create opportunities for people to get their feet wet. So I’ve started to see a lot more Associate researcher, we’ve created sort of a more junior level within our career level of associate researcher, something that makes it easier for someone to to get experience within the field we’ve taken on interns or an intern, which I think is is a nice door into the discipline. So I think creating some of those formal opportunities, but I think being willing to mentor as well facilitating connections for for people who are just trying to get into the field, even if it’s just for informational interviews, or maybe informal projects is something else we can definitely do. And I think also, as someone who came in from the outside, I think it’s possible to I think we underestimate how much it’s worth really sitting down with someone who has been a week. When you’re on the outside, it’s a bit more difficult to understand what this wall looks like what the day to day looks like. What are the skill sets. So when you’re trying to think about what are my transport skills that bring value to this field that will help me transition into design or into research. I think just being able to talk to someone who is a researcher who works on the day to day and to have a conversation with them about how your skills fit into this picture is completely invaluable. So I think making yourself available to people who are really trying to understand what this work is about how they can leverage the skills and experience they already have, making it concrete for them and helping them figure out how to articulate that value is a big part of giving them the confidence to take the leap and an understanding of how they can they can bring those skills over the fence into research.

Steve: That’s great if some very concrete advice I think for people in the field and looking to get into the field. I think that’s a great place for us to to wrap up here. It’s been really interesting and enjoyable to speak with you Julie. I really appreciate you sharing so much with us.

This episode of Dollars to Donuts features my interview with Chris Kovel, the Head Of Research at First Abu Dhabi Bank (FAB).

I look at needs as proximate needs and ultimate needs. An ultimate need is why the product exists in the first place. And then the proximate need is the experience of using that product, right? So if you take for example, a hamburger, that’s a product and the ultimate need is the hunger that it satisfies, right? We as humans need to eat things and hamburgers are one of one of those things that we can eat. That’s the universal need that we that it solves. But then there’s also the needs of actually getting it, getting to the to the place, getting to the restaurant, then there’s the needs of having a good experience in line, being able to read the menu, being able to take it to go if I wanted to. So there’s these nested needs within the greater need of why the product exists. Both are important. But I think that product teams don’t always take both into account. – Chris Kovel

Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.

Steve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.

Noticing something about my headlong progress through my slice, my friend paused and looked up from his own cake, curiously asking me, “So…you don’t save the best for last?” This was a new concept to me and I stared blankly, crumbs leaking out the corners of my mouth. His mother pops by to affirm, “Yes, Stephen, we save the best part for last!”

Beyond surprised, I was enlightened. Of course, some parts of the cake had more value than others. A forkful of plain ol’ cake wasn’t as good as cake and frosting which itself wasn’t as good as that mouthful of corner megafrosting. But the big news was that I had an option to eat to an end goal. I could eat differently – portioning, partitioning, and planning – suffering through not-the-best bite, so that down the road a slice I would have the absolute best bite. That would become my reward, that was something to work towards. Gratification delayed was gratification improved.

In a fundamental way, this exchange broke me. Decades later I am still struggling to effect repairs. I absorbed a significant lesson – that even a tiny in-the-moment decision can have consequences, risks, and downsides. This week, even, I noted how, with hardly any conscious thought, I was actively managing how much salsa to put on a tortilla chip, weighing that if I took too much from the little container, I could wind up chomping on dry, salsa-less chips by the end of the meal.

As eaters, we face moments for optimizing (or at least risk-mitigating) constantly. Food designers don’t seem to consider this aspect in their apportioning. Whether it’s Lunchables, or an airplane snack box, a meze platter, or a charcuterie plate, there’s an intended combination (nominally meat or cheese or spread on bread or cracker), by design. But the quantities are rarely calibrated so that there’s a cracker for every morsel of cheese, etc. Of course, we can eat things in a variety of ways, and an extra apple slice isn’t really a problem. But, a solitary dollop of marmalade is absolutely a fail. And so, when designers don’t consider this, it falls to us, the concerned eater, to plan, calculate, adapt. For some of us, that isn’t an enjoyable part of the experience. Indeed, the affordances, the aesthetics in a bundle of base/toppings is that we are consuming a set of choices, a purposeful selection of quantities of elements. So the mismatch is just that much worse when the cues tell us to expect coherence.

Going beyond what may be dismissed as my compulsiveness, this is really about being present. It’s hard to be present with your food, or indeed be present at all, if you consider what may happen in the future, worrying about potential microannoyances. Navigating the pleasure of being present with the pressures of what lies ahead is the burden – and freedom – of being an adult, of constantly having to make choices. At my most hopeful, I might reconsider learning about the-best-for-last less as fracture to my equanimity and more as an important and inevitable loss of innocence.

Here’s another zero-budget episode to share with you. And I want to remind you that you can support Dollars to Donuts by supporting my small business. Here are some of the ways I work with companies.

The first is User Research – I work well in situations where the team knows some things about a set of users, often they each know something different that they strongly believe, and have no way to get from their anecdotal familiarity to an actual framework to make decisions. “Oh, we’ve talked to our customers many times, but we’ve never heard THOSE things from them…” This is often research about people, rather than “a product”

The second is a Player Coach where I’m working with “People who do Research” or maybe there’s a junior research person on a project, where they can use some of my expertise but they are going to own the project from start to finish, so I come in to advise some parts of the process, roll my sleeves up for other parts of the process, depending on what the team needs.

There’s Team Coaching where I’m an available resource with office hours or other format so that people can come with questions, seek feedback, etc. I usually not “doing” anything, just there to talk and review documents

Training is for people who are moving to do more research, this is a half-day to two-day curriculum on planning research, interviewing, analyzing data

A Master Class is for a research team that has a particular area of interest, this is a customized combo of lecture, exercises, and facilitation

And finally, Alignment Facilitation where I’m bringing together a group of people to identify and sort through issues, often around optimizing and improving process

If I can help your team, please get in touch.

All right, let’s get to my interview with Chris Kovel who heads UX Research at First Abu Dhabi Bank.

Chris Kovel: Pleasure to be here, Steve.

Steve: Chris, thanks so much for being on Dollars to Donuts. It’s great to have the chance to speak with you. Why don’t we start By Why don’t you introduce yourself?

Chris: Yeah. Hi, I’m Chris. I head up UX research at First Abu Dhabi Bank.

Steve: How long have you been in that role?

Chris: So, First Abu Dhabi Bank is a is the biggest bank in, in the region. So I work in Dubai, and I’ve been here for about 14 months.

Steve: And did you join the organization to take on this role?

Chris: Yes. So within the organization, there is an innovation lab. So fabric first Abu Dhabi bank Research and Innovation Center. And in that position, I head up the research and that position was created and I decided to take the plunge and go out to do By and it’s been a great experience.

Steve: Do you have I mean, this is an unfair question, but you may have some context since this happened before you got there. But do you know what led to the identification of the need for this role creation of the role, you know, the, the conditions that were established that led you to start talking with the organization and join?

Chris: So yeah, the history of the lab or the fabric, It started about two years ago. So, my boss, the head of research for fab guy named Gavin, he was brought on to really try to help with the digital transformation. First of Adobe banks, a really big bank, like, like I said, biggest bank in the region. And in terms of technology in terms of digitizing the products and the services. That was really the mission of the innovation lab or fabric as we call ourselves now. And in terms of research, Gavin kind of comes from a software engineering and design background, and knew that research was going to be very important in terms of creating products and digitizing products. And the other kind of core mission other than digital transformation with that fabric has is kind of spreading a culture of innovation within the bank. And that means empowering other departments, not just the innovation lab to be innovative, to think kind of like designers to be problem kind of obsessed, and all that and kind of requires research and hopefully we can get into kind of how that how that happens.

Steve: Well, that’s no time like the present. So how does that happen?

Chris: Yeah, so there’s a lot of different ways to kind of come at it. At least from my perspective, in terms of kind of developing products, you know, I think when research we like to follow design thinking principles comes out of the D school, most famously in Stanford. And but it traces all the way back to a guy named john Arnold, who create who came out with a kind of a book pamphlet, I think, in the 50s, or 50s, or 60s, called creative engineering. And it’s just a way of kind of problem thinking about problems, right. And, and, and needs, right. So when you think about problems when you think about needs from a customer point of view, that will allow you to kind of frame issues and then solve them through products and services. Most of the time when research kind of gets embedded in my opinion into this design thinking framework either up front and towards the end of the product likes, lifecycle. So I kind of think of it as bookends. And up front or kind of upstream is a lot of times when you’re trying to find a need and then downstream is a lot of times just smoothing out the experience or smoothing out the product through testing we do both at the Innovation Lab. But I’m very passionate about the upstream kind of research because it’s really where these the tricky problems are. You know, trying to understand human motivation is not a not a simple thing sometimes

Steve: From your perspective, or maybe from all the folks in fabric, you know, the value of understanding human motivation, in terms of how it applies. To the kinds of products and services or decisions that have to be made in a financial institution, maybe that’s clear to you. Does, how does the rest of the organization see that relationship between, hey, we need to understand human motivation? Because here’s what we’re going to do as a bank.

Chris: Yeah, yeah, it’s a learning experience. To answer that, that question because that was something that I kind of have found, wherever I’ve worked throughout my career as other people who kind of don’t, you know, live, you know, live the UX kind of world of the user research world or really being obsessed about the customer and making the product kind of fit the customer’s needs. is a difficult one. I mean, it’s about education. It’s about culture, you know at fabric. We have four core components within the You know, the lab. So one is research. The other one or one is research. One is accelerate, which is kind of more about more design. One is FinTech. We can get into that if you’d like. And then the last one, and arguably the most important one is culture. And that was that it’s just an education piece, right? It’s getting people to come and walk through the lab and show them how and the pods that we do user testing. And with, to kind of do roadshows to kind of get the word out there. We’ve done hackathons where we bring people together for kind of, you know, interactive brainstorming. We do talks and kind of fireside chats about design thinking and why they why people should care. So it’s, it’s difficult, right? Because culture is kind of a amorphous, intangible thing and to change it. It’s hard to change it if you can’t even put your finger on it sometimes. So it’s a work in progress. But at the end of the day, it’s it really is just about people and about just talking with them and kind of getting the word out there. And, you know, in the beginning, the bank kind of, you know, we had some Slow, slow days to begin with, but slowly but surely after that we started, especially within research after we start delivering value. And they kind of see the power of answering questions and have been kind of evidence based, then I think that there’s two ways I think it’s the education and then it’s also delivering high quality real projects.

Steve: One of the things that, you know, we talked about in researches is in addition to, you know, doing quality work is, is having people is having influences having people sort of be willing to hear that information that maybe challenges their beliefs, and then ideally, take some action based on that. And so, you know, you’re talking about driving this change, what have you seen on that side of things in terms of, you know, I mean, believing engaging with taking new perspectives on and then, you know, having that influence the decisions that are being made.

Chris: Yeah, so, a couple things on that point. So first off, you know, it’s, it’s never, you know, most of the time, I probably argue, arguably all of the time when a group comes to us to get some research. It’s usually a question that is pretty polarizing. You know, it’s research and also in many ways, I found is kind of a way for people to kind of settle internal conflict or settle internal debate. So, you know, these the results that we provide really matter and they really kind of, you know, take it to heart, but that’s a good thing. But the other kind of bad thing sometimes is that when you’re in the room, and you’re kind of, you know, for example, if you’re testing an application, and you had peep designers and product managers kind of working really, really hard to making things good. And in many cases, the research is the questions that we ask kind of the data that’s presented in many cases is really negative and sometimes and not negative. But that it’s kind of supposed to be that’s why that’s why the research is there. And to kind of manage that manage that in the room is something that it’s a it’s a learning experience, but You know, we’re kind of, we’re kind of human we get we get defensive, or the people who are in the room are human, and they get defensive. When, when the research is being is being presented sometimes so that’s just something you have to be kind of conscious of, you know, I tell that the younger Junior guys, that’s, you know, there’s a certain way to kind of couch the reports and the and the data. And when kind of infighting begins in the room, you kind of have to just take a breath, and really just look back at the rely on the methods. And that’s why, in my opinion, kind of the methods and the way you present and the quality of the research is so important because if it’s done well and it’s executed properly, then the data is the data and you can kind of just kind of, you know, sleep well at night, even if it’s caused some internal strain So, so that’s kind of kind of the first point The other one about taking them taking it forward, right is a kind of an interesting one, it’s me and me and Gavin, my, my boss, we kind of go back and forth about this a lot. And in my kind of philosophy is that the data is the data. Sometimes they want us to recommend, and sometimes they don’t. But in terms of solutioning, that’s kind of their I mean, we’re the researchers, right? And they kind of other product owners. And if the if the research and the project is scoped up in a way that’s pretty narrow, and they just have a couple of questions when we kind of provide that that information if they kind of because of politics, or because of infighting or because of, you know, other bulwarks. If they don’t take it forward, then it’s kind of out of out of our hands, which is kind of arguably good and bad. I don’t know how to how do you see it?

Steve: Yeah, I mean, I think you’re hitting a really This is a personal hot button of mine. You know, how do we measure our own success? I’ll tell you I had a kind of a galvanizing experience a number of years ago with a friend of mine who had worked with a number of design agencies. And we were just we were, we decided to meet and share our capabilities presentations as a way of just learning about each other and learning from each other. These were public, public documents that we could share comfortably. And, you know, I had all these stories were research, it uncovered something really insightful and potentially impactful to these business areas that my clients were working on. And I kept saying, you know, unfortunately, this didn’t happen or you know, they weren’t ready or they decided not to and you He kind of stopped me and said, like, Steve, you’re not in the business of them shipping a product, you’re in the business of highlighting opportunities. And that, and that, that was really grounding for me. I think, as I, you know, I want my success stories to be success stories and not failure stories, but we can’t control everything. You know, and I think it’s, it’s a little, it’s a little distressing to hear researchers tag their own success to a thing that’s ultimately out of their control.

Chris: Yes, yeah. That’s Yeah, it’s interesting. I, I agree. I agree with all that. Everything that you said there. And I think it’s kind of if you look at kind of the different levels of the macro and the micro, it’s like, I think if you’re, if you think about it, in the macro level, right. So the product that I’m researching isn’t going to be a success or failure. Then that’s, that’s definitely It can be kind of disheartening, right? If you do all this really good work and then the, the product kind of dies or it goes away or it takes a different direction, a direction that the research isn’t what you know, kind of away from the research findings, but if you look at it in the macro and just kind of, you know, look at each project as each project and as long as you know, you hit this hit ticked all the boxes in the scope, and you kind of went out and you know, you did the, you know, the research the way, the way it should, should have been done the way that you wanted to wanted it to go and then reported it and then you can kind of I think rest easy if you just kind of look at it with kind of net very narrow with kind of almost with blinders on but it’s hard sometimes. Right.

Steve: And I, you know, it changes and I think you’re kind of alluding to this. A little It changes based on, you know, how emotionally invested we are as individuals. And, you know, I work as a consultant. So, I love my clients and I’m super engaged, but at the end of the project, you know, they have to move forward without me so I have a, I have a way to detach my success or failure or, you know, how identify my own success or failure because project ends, even if a relationship continues. There’s a point where the that’s, that’s over, you know, you are part of an organization even though fabric is, you know, a center. It’s in the name. I wonder if I mean, I wonder if different researchers just based on sort of where they sit in the organization or you know, how they feel about their own ownership of that product or their own ownership of that team. I wonder if their own assessment of what success looks like very for them.

Chris: Yeah. I wonder if for me if I’m kind of okay with just delivering and then and then kind of moving on to the next thing and kind of judging the success project by project, not kind of the overall life cycle. I’m wondering if I it was kind of imprinted on my brain kind of from the early days when I first got into doing research. So I, I worked originally or fresh out of college in Manhattan, at a human factors engineering firm, very small, kind of boutique firm. And we would work on interesting and cut with big, big clients like Google and Amgen, and others, Nike, but it was literally like the brief would come in and you know, a month or two and we would do the research and then we wouldn’t hear anything because of you know, confidentiality. situations and what have you. So it was very much just like here’s the here’s the findings, here’s the data, and then on to the next thing, and I got very comfortable with that. So it might just be I don’t know, maybe how you get your start. But I wonder how if there are teams out there where researchers are kind of, you know, embedded into the whole product from kind of soup to nuts from cradle to grave, and if they prefer that, or if they prefer for me I love one of the reasons why I love research is because one week you can be working on digital wallets in the next week on a personal loan app. And you know, I love I love that for the versatility and the diversity of working on many different types of projects and kind of moving on to the next one.

Steve: Yeah, I’m thinking of several things here at once. Now. I mean, it’s interesting. You know what? Yeah, you make a really good point that, you know, sort of how you were raised up, how did you get your start and what sort of model formed for you versus, you know, the experiences we had versus someone that’s, you know, part of a team and maybe sees themselves as being tied to the success of that product. And, you know, I wonder, and again, I’m super biased because of the, you know, how my, how my approach was formed and how I operate now. But I wonder about, like, what’s good for research, and I feel like the truth to power role that we play, you know, is essential, and, and being self contained or self sufficient, like we’re going to do great work, and then we’re going to help you as much as we possibly can but that next part is up to you. That seems that mindset to me seems good for truth to power, you know, role but, you know, what I’ve come to admire with people who are embedded is you know, and maybe, you know, fabric is different than my business, you are part of a larger organization. He talked about culture, but, you know, the being embedded and working long term with people kind of day to day sitting with them. You those researchers, they really know those people, they, they can influence. I mean, literally from the inside, right? They’re part of that team. So I don’t know what you’re seeing with that fabric as you build relationships.

Chris: Well, for me, I think I’m kind of torn right because I can see the value and kind of embedding researcher through from the beginning of the product to kind of The end to when shipping or you know, when it’s becoming live. But I think there’s also value in kind of the outsider kind of role that sometimes we play as being brought into a new project or on a new product team. And kind of not having all those internal biases and kind of the curse of knowledge, right, that research kind of, you know, it’s just another it’s another set of user eyes, new fresh eyes of seeing, seeing the product and kind of looking at it from different point of views, different point of views. I don’t know I think that there if you really kind of take a step back and kind of look at a product or a service lifecycle, and no two are the same, but I really like kind of following the I mentioned it before the D school design thinking kind of philosophy and if you look at that The circles there’s five different steps or five different kind of parts to the design thinking model, kind of empathy is usually up front, and then testing is at the end. And I think having researchers in involved in the process for those two things is essential. I feel like it’s a lot of times, you know, we people and businesses and, you know, marketers or product people look at research as kind of a nice to have, and I really do kind of think that it is at least in those two areas upfront and then at the end, it really is essential. It’s, it’s a must have.

Steve: And what’s what is the research doing for the process or for the team or for you know, what’s the what’s the impact of research in those two different stages. of the process.

Chris: Yeah. So I think the easy one is the testings kind of down downstream, right? So when a product kind of comes into the business from and starts having, you know, getting some traction and people start becoming, you know, working on it. You know, usually a prototype is made, it depends on what kind of product for us doing UX research, most of the time, it’s digital, it’s, but, you know, that’s not to say that it’s always the case, it could be a physical product, it could be an experience, it could be a service of some, some, some sort. And usually, after a prototype is made, then the researchers come in and that’s when that person prototype is kind of smoothed out. So usability test tests are kind of bread and butter for UX researchers. But in many cases, I think upfront, you know, the research hasn’t, you know, the teams don’t bring it in. The teams don’t bring researchers into the process to begin with. And I think that there’s a real impact and a real value that away for, you know, product teams to kind of set themselves apart, and making sure that the product that they’re creating has been tailored to a specific need. I mean, I kind of see it’s, you know, some simple terms, that a product is just a sad product is basically satisfying a need and the way that you find needs, and it’s a very tricky, tricky kind of business, but the way that you find needs in my, in my opinion is through empathy and through going out and kind of applying the methods that we as researchers use to kind of, you know, make things a lot clearer and distill down the essentials in terms of what the problem areas are, what the need areas are no jobs to be done. framework, I think gets this right to where, you know, it’s kind of I was just watching a webinar, I think, and I really like this, this phraseology that they use they, they were thinking about milkshakes. Right? And like, why milkshakes were being sold more in the mornings, during, during the workweek than any other time. And the way that I remember that the way it was framed was, you know, why do people what are they hiring, a milkshake for? And I really like that kind of reconceptualization about that. Thinking about products and services and what do you Why are you know, hiring your computer? or Why are you hiring your water bottle? And I’m thinking about it that way I think really can kind of generate interesting new ideas and that’s kind of the heart of innovation in my opinion.

Steve: When you at fabric, have these initial conversations with teams at the bank. What are they? What are they looking for? Like? Do they have a product? Are they looking for needs? How are they kind of framing that initial interaction with you?

Chris: Yeah, yeah. So most of the time they come they have a product that’s kind of either out there and kind of in the wild or, or an MVP or a prototype and they want us to kind of help them bring in users have them you know, put that put that product in front of them and kind of observe their behavior and kind of get back to them with the findings about whether they the users kind of could understand whether there are any kind of issues with the content issues with the lexicon, this these kind of things. And so, you know, we work with the digital team, we also we work with other departments within, within, within fab, on kind of lack of productivity. And so, a couple of months after I started, I started interacting and working with the head of marketing. And we kind of created a market research branch off of fabric. So now, any type of market research initiatives come through us and we, we test ads we do kind of more discovery type work about products, you know, for example, there was one credit card. So fab has 22 different credit cards if you can believe that. And the one that was performing the worst was a card called gems card and we were able to bring in people who were using the card and kind of getting their sense about why they’re not using it more. Why kind of other confused about the point system or you know, kind of just getting them to kind of talking and Having an informal kind of discussion about problem areas and things that we can improve and kind of the whys as to the underperformance. So yeah, was that Did that answer your question?

Steve: Sure. And it raised another one. So that that says we’re going somewhere we want to go. So I guess that raises a question for me around the definition. So I think you’re describing that as a market research project. How do you define or delineate what’s you know, what’s UX research? What’s market research? Yeah, think about it.

Chris: It’s interesting. I think of it all as kind of the same thing. I mean, this is now kind of being in the industry for as long as I have been. I’ve seen I’ve kind of seen different titles or worked with the from corporations, you know, things like human factors, things like user research, UX, kind of design designers. And I think that at the end of the day, it’s really all kind of coming, coming from the same place or I guess the it’s all looking towards the same thing when it comes to all these, you know, you know, human factors or UX or you know, market research. Just at the end of the day is trying to answer questions with data from your actual your customers. And it’s trying to get at the need, I think it’s trying to find needs and try and discover needs, and then trying to help the people who make the products or make the services, make them better, make them more, not just making you know, Making people that’s not just, you know, listening to people about their problems of, you know, with, with a product or their problems with their day to day life, but really trying to understand the motivation and understand their feelings and their desires and kind of satisfying them with services and, and products.

Steve: So I want to ask a different question, following something you said before you talked about the sort of four areas and where culture was like a really important one. Yeah. And so you, I mean, from where I sit in California, I’m curious about culture in your organization, because not only are you know, in a specific industry, which, you know, based on my experience, like banking, has a certain culture, sort of historically around and you know, maybe that’s very much an American perspective. But, you know, a regulatory a regulated industry with certain history means there’s a certain approach to decisions and a certain approach to, quote, risk and so on. But you’re in a very different region of the world than I am. And I’m so and that makes me wonder, you know, what observations do you have about, you know, the regional business culture? And how that how that responds to innovation? And the changes that innovation is asking for, as well as financial banking kind of services culture as a factor that characterizes the culture that you’re working in.

Chris: Yeah, so I think culture is changing the culture is one of our core missions with one of fabrics core missions. So it’s a work in progress. You know, I think fabric and then fabric versus fab. I think within fabric, we kind of have in our own culture, it’s a little bit different than kind of the corporate, you know, buttoned up tie wearing jacket wearing kind of traditional stereotype of the bankers. I don’t see too much of that, that stereotype within the bank, you know, meeting with people in different departments. Um, but yeah, I think work I’ve worked in the corporate setting a couple times. And I think kind of, you know, there is there is a difference. There is a division between kind of the traditional, you know, normal kind of roles that you would think about when you think business versus kind of the UX or user research designer type people. And it says it’s, it’s something that yeah, it’s I feel like there should be more in integration, I would, I would like to see more integration, you know, in the future and to just from a business point of view, I think it’s good business to kind of have people like us at the table. And in many cases, it’s there. You know, we’re not we’re just not there yet. So, and it’s something I think that organizations that, you know, struggle with about how to kind of you hear a lot of people, you know, a lot of people in the industry, I don’t know if you get this as well, but talk about when they’re working in a big corporation like to evangelize, you know, that user research or just kind of UX in general, has to be evangelized, right. And kind of the, it’s kind of a little bit that you know, first off, it’s not great that it even needs to be evangelized. And then the second part is like How do you actually what’s the best way to, to evangelize? Yeah, we talked a little bit about, you know, it’s educating, I think that the way the approach that we have is kind of through education, you know, events, these kind of kind of cultural events and then and then just producing, you know, kick ass work.

Steve: Can you What do you think would be different if, you know, you’re at fabric within fab, but you were, this was a company in Manhattan, or this was a company in London. Do you think there’s any regional shifts that you would see?

Chris: Yeah, it’s an interesting question, um, regional shifts. So you know, it’s funny, I when I first moved out here, I was expecting it to be a lot more of a culture shock than I was like, then it actually was so. Yeah, I’m sure there would probably be some differences without a doubt, less sand. But I, I think I think for the most part though, it’s you know, it would there wouldn’t be much it’s funny Dubai actually has I think it’s 8080 or 85% expats right so there are very few locals and nationals UAE nationals. So the bank is comprised of a lot of people from a lot from all over the world, right so it’s a very diverse very multinational community. Not a not too many people from the States but a lot of people from England, a lot of people from Australia. India is there’s a high percentage so yeah, it’s kind of a melting pot, and it’s a really cool and interesting environment to work. But I think at the end of the day, yeah, it’s People are kind of people. So

Steve: do you have any sense of, you know, the business community whether it’s you know, outside your organization, but in the region you know, what the awareness or adoption of, you know, user experience user experience research, you know, elsewhere outside of fab.

Chris: Yeah. So it’s something that’s that is, I think taking off and growing within the within the region. I know for us personally, what we want to do is fab not only serves the UAE, but it also serves other countries in the MENA region like Egypt and like Saudi Arabia, and others, and I know that we as fabric have kind of goals to take certain elements of our of our toolkit in our service Kind of export them into, into these regions, right? So research being one of them, I think because of its kind of universality. And its way for, you know, you can kind of use it in many different contexts. So, so that’s definitely exciting. But, you know, it’s funny, I think user research and UX in general, is kind of a very niche, you know, industry. So even within the United States, you know, a lot of people don’t really know about it, whereas in this region, it’s almost like I think there’s a lot of people who don’t know about it about it as well. But I think that there is receptivity, they’re much more receptive to kind of embrace these kind of new and maybe kind of non traditional ways of approaching work. And I think that’s really exciting.

Steve: Yeah, sounds very positive. And we’ve talked in and around this a little bit, as you’ve talked about some of the ways that, you know, that fabric is working with teams, you know, throughout the bank and so on. When you start working with these teams, what kinds of disciplines or functions or roles do you most? Or is there a typical, you know, set of rules that you are collaborating with most directly?

Chris: Yeah, I would say I mean marketing for one because, you know, fabric and marketing, there’s kind of this dotted line between fabric, fabric reach research team and then the market research that we do. So that’s definitely a division that we work with very closely. We started working with product a lot. We’re doing some kind of foundational fundamental research project that we’re really, really excited about, on personal loans that hopefully, it can kind of spread and we can do this this kind of research, research stream with all different products that really start from the ground, ground floor and get those needs and kind of assess and evaluate the current state of the products and then kind of kind of move forward involved in evolving from there. We work with the digital Team A lot, a lot of times that the guys will come and request meetings for, you know, to do kind of benchmarking testing, like I said before, a lot of usability testing, touchpoint testing, sometimes card sorts, every now and again, some kind of upfront research about kind of ideas and content concept testing and this type of thing. But for the most part, it’s kind of its product, it’s the product testing. Let’s see this. Yeah, I mean, we’ve been working with a lot of different departments, which has been great. I think we had 25 or 28 different research projects completed last year. vast majority of them was from ppg, which is personal Banking Group. We have corporate banking, we haven’t been working with corporate all that much. But that might be changing in the next couple. next couple weeks, weeks and months ahead. But yeah, I think that’s weird. So there’s also a team that that that head that basically is single product team with it’s called pay it and it’s a it’s a digital wallet and we’ve been doing a lot of different types of ethnography work, you know, competitive landscape research, and you know, customer interviews and these types of things. So yeah, that’s been exciting too, because I think that the digital e wallet space is a new one. And I think it can be. It can it can there’s a lot of value in kind of thinking differently and really trying to figure out the needs of people and why kind of a digital wallet exists, or why they exist, and then kind of helping them shape and smooth the experience.

Steve: And are there situations where fabric is proactive, say, as opposed to responding to requests from a from one of these teams initiating some effort, some program from within fabric does that happen?

Chris: I think at the beginning, once we started kind of ramping up, we were so busy that we were just focusing on the products that I’m sorry, the projects that were that were kind of coming to us and There were times when we had to kind of deny or you know, tell people sorry, we’re kind of we’ve hit capacity, we have no more bandwidth. But I think with this kind of Corona situation, we’ve had a time to kind of level set and start thinking a lot more about, you know, spinning out projects internally. The prot, the personal loan project that I just that I mentioned in my last my last response to your last question, was something that we kind of were developing ourselves and when the product came to us, they were just like, What’s on your mind? What what’s kind of some, some new stuff that you got? What’s kind of what do you guys thinking about how, you know, we kind of want to kind of work with you like what can you kind of offer us you know, there wants to want to have this relationship and that was something that was internal, an internal idea or an internal model, turtle project that we kind of then took off the shelf and applied it to, to product when they kind of came knocking. So that was kind of kind of cool. I you know, it was to give a little bit of a background, it’s a new and innovative, new and innovative way to think about personas. And what we really wanted to do was we wanted to I mean, I think most people within the industry when they hear persona, it, they kind of, you know, they’re, I think it’s a very ugly, ugly words in some people’s minds. You know, but I think that there’s, there’s, there’s use and there’s value to them, but I, I see them as being kind of a little limiting or limited, right. So the idea for this project that we’re working on with product is kind of to do the personas the traditional way, which is where you go out and you get together. Either through focus groups or in depth interviews, a bunch of a bunch of users, a bunch of customers within that specific product. You talk to them about their lifes, you talk to them, you would get you understand, you know, their demographics, you understand their kind of, you know, their motivations, why they’re the other products, they’re using, you know, kind of a comprehensive 360 degree view. And from all these interviews and focus groups, you know, personas kind of emerge out of them. So the idea is, is to kind of take that traditional method and kind of mash it together with a method that’s kind of been now being developed over the past couple of years with using machine learning and AI by feeding all of you know, customer data into And an algorithm and have the algorithm then create the personas, you know, mathematically, and then if there’s some way for us to kind of merge those two, where you get kind of the, the best of both worlds of approach, because you know, in many cases, the traditional UX. Personas through research are kind of limiting. I mean, you can ask a lot more questions than you can with a datasheet, you can get a lot more kind of subjective internal motivate motivations, you can kind of capture those but the power of generalizing that is not the same as the power that the machine learning model has. And if there’s a way for us to kind of, you know, understand that those two are, there’s a way for us to kind of push those two methods together. I think that would be a really huge impact for the bank. So it’s kind of a combination of that persona piece with this idea of service design, which, in many, in my, in my opinion is very, very much a just another method of, of user of user research, where you go, and you talk to people, this is for personal loans, you talk to people about their experience, you know, taking out a personal loan, so usually people who, within the last, you know, three to six months have taken out a personal loan with fab. And you kind of map out the whole journey from kind of conceptualizing, to kind of get you know, that kind of motive motivation moment of like, I’m gonna need a personal loan, why they need one, and then going through the experience of, you know, coming to fab going, setting, you know, signing all the documents, and then, you know, lastly making the payments and kind of having that map and looking for gaps looking for pain, areas of pain, things that can be improved upon. And if we kind of have those two models, and we do those that piece of work for every product in the bank, we think that, you know, hopefully, we can really change the trajectory of the products and make them more about kind of user needs and user solving user problems then, then kind of how they’ve been set up so far.

Steve: That’s an amazing approach. I haven’t heard of anything like that before. You’re talking about? Yeah, just like you said about market researcher kind of blurring the boundaries or eliminating the boundaries between things that were sort of separate islands. And here you’re pulling in two very different kinds of things to create. One, one better approach. It’s very exciting to hear Yeah, it’s

Chris: it’s funny you say about marketing because I remember when I was meeting with so the merchant team came to us came to fabric. And we did just kind of a normal, you know, textbook, piece of user research on some credit card designs and then also asking them some questions about what would get them to sign up for a credit card. And, you know, they were really happy with the results. And then they were like, okay, we want to kind of grow the team, we want to kind of have you hire some dedicated researchers and kind of have this this dotted line between marketing and between UX and I was kind of like, wait, I, I don’t I don’t know like, I don’t know marketing. This is not my world. Like I don’t know if you have the right guy, but the more I kind of learned, and the more I talked to the marketing people and the kind of the decision makers, I learned that you know, there really is not bright lines between product and marketing, and, you know, and research and just kind of thinking holistically about how products Come about thinking about, you know holistically about the economics, you know, why? Why do products succeed? What makes what are they? For what, you know, what makes a good one? What makes a bad one? You know, at the end of the day, it all starts with understanding your customers. And that’s what we as researchers, do, we try to understand all the different facets of people and what would make them buy and what even what is it what is a product doing? What purpose does it serve? When you start there, then I think the marketing and the product and all of the those other parts of the business kind of just take care of themselves, it becomes much more easier to navigate and make decisions based on this fundamental understanding of the people that you’re serving.

Steve: And right now that thing that activity is But we call it research. It’s, and you know, people that do our job are the ones that do that. Yeah. And, I don’t know, I feel like in some ways, it’s a historical accidents like you when you describe, you know, this is what the process should be. And this is, you know, in order to get to these ideal results, and here’s the activities that kind of started off you know, the way that research is as a discipline has done evangelizing has done has, you know, invited other people to participate and sort of said, Hey, you know, everyone needs to do this. I mean, if you mentioned the D school right there D school is like classic for saying, Hey, we’re going to train everybody to do this not people with you know, the researcher title. This is everybody’s job.

Chris: I mean, I think it it’s a it’s a way of reopening Kind of reimagining kind of what businesses are doing, and why they exist and what their purposes are, right? I mean, it’s Yes, I think at the end of the day, it’s like, it has to do with monetary gains, and, you know, stakeholders, you know, you know, making sure your stock is going up and these types of these kind of economic questions, but, you know, I think, you know, when you say things like product, we talk about products, we talked about services, we talked about needs, and they can be really defined kind of in very broad strokes. And when you when you define them in broad strokes, then then the power and the use and the kind of impact and the relevance of understanding your customers, which is what as you’re saying is what we do, becomes, I think Paramount, right, because that’s the way that you can separate yourself from the competition. It’s not just about innovating for innovating sake, right, it’s innovating to me is to just find a need and solve it in a new way. And if you can do that, I think you’re, you’re poised to have to have success. The problem, of course, is some of the bureaucracy, some of the politics that kind of happens in you know, in the different layers within the business. And also the kind of tricky thing of, you know, researching humans, it’s, I love this quote, it was a physicist, guy named Lawrence Krauss said, I do physics because it’s easy. Right? Well, it’s clearly it’s not easy, right? But I think what he was getting at is when you’re when you’re researching humans, they’re tricky. They’re complicated. It’s a little bit messy, right. He kind of love He was his point was the elegance of physics. Right? You can describe parts of the world in very elegant mathematical ways. And it will always behave that way. Whereas people were unpredictable. And kind of getting at the root causes and those root needs and those root desires is tricky. But if you get them then then i think i think everything falls into place.

Steve: I would love to change directions. And, you know, hear a little about what your trajectory has been, you know, how you found research and you know, what kinds of roles you’ve had that kind of, you know, we all know about Manhattan a little bit, but yeah, describe more of that story.

Chris: Sure. So I kind of I kind of fell it fell into this accidentally, actually. And yeah, I love this question asking Others in the industry this one because it seems like you know that there isn’t a clear path into kind of the world of UX in the world of user research. Which maybe would be maybe that’s the case for all industries. I don’t know. I have a hunch that this one is a little bit different though. People get into it through kind of untraditional ways. So yeah, I was. I was going to grad school in New York. And I was getting interested in fascinated with science and I started working in a behavioral science lab. And I graduated, and I moved to Brooklyn and I didn’t have a job. And I was kind of running out of money and I was and I wanted to do research. I really liked the idea of kind of trying that out as a full time kind of career and the first job I interviewed ATS was a, like I was saying a small boutique firm on the Upper East Side of Manhattan that did Human Factors engineering research. And I saw the research paper the human factors for I had no idea. And it was like kind of a pretty low level, entry level position. And I went interviewed, I got the job. And I kind of just fell in love with it like, and I never really looked back since and kind of this idea of merging science and research and data with design was something to me that I never really thought about and, and kind of, you know, you know, for me when I when I thought when I think of design, I always would think of kind of just someone sitting in a room, you know, you know, twisting their moustache and coming up with ideas. I didn’t realize that you can actually kind of use science a US kind of methods, research methods to influence how products Shape products, right? Which is really what a lot of times what human factors engineering does. It, it’s, you know, if you if you looked at the D school model, it was the kind of lives in the testing area for, for a lot of for most of what it does. So I did that and I kind of after six months of working there, it was, like I said, a small a small firm, the my direct reports left kind of suddenly and the next in line. So, she reported to the CEO, a guy named Charles Morrow who has been in the industry for 45 years. And then so that left a big wide open role that he you know, said that now it’s mine and I was kind of again in moment of crisis like I don’t know if I can if I can handle this right like I’m only six months in, it was a very much like a sink or swim moment. And now here this, you know, person who really didn’t go to school for engineering and research is now kind of managing these huge projects for Amgen and Google I was I yeah, it was definitely harrowing, to say the least. But I kind of kind of made through it. And then a couple of years after I decided to move to the Midwest and I took a job working for more, more human factors, stuff working for a company called Medtronic that makes medical devices I worked in the neuromodulation. Department, working on a couple of different products kind of flew around the world, around the country, mostly talking to doctors, kind of getting their feedback and trying to help The shape certain medical devices. And then I kind of switched gears and got into the world of UX and UX research, actually that same mice the same directed my direct report of my first direct report, Emily, lady named Emily actually suggested we kept in touch after she left and she suggested that I start looking into this, this thing called UX research, it’s very similar uses the same methods of kind of the methods of that we employed for human factors employees. And I kind of thought that it was Yeah, it was I was getting a little bit bored of the physical products a little bit bored with the medical devices. You know, tech was kind of booming and it was kind of an exciting place to work. I thought that it’d be cool to kind of open up my tools. And start working on screens. I did a little bit of screens, kind of within the human factors, but definitely not as much. And I started then working in in a few different innovation labs, I worked for a company in Minnesota to call I’m sure you’ve heard of it Best Buy, how many people here have heard of it, and that was my last. My last role was Best Buy was before I moved out, moved out here to Dubai. To do to help them kind of creates UX research department within an embedded into the bank, which has been lots of fun, I think.

Steve: So outside of fabric and the work you’re doing for the Banco? Are there other things that you’re working on in your profession?

Chris: Yeah, so, um, since the past year, year and a half, I’ve been kind of kicking around an idea concept of trying to think about, you know, things like UX and jobs to be done theory and market research and human factors, all these different kind of disciplines that seem to be kind of putting a lot of a lot of weight on understanding the customer bringing the customer into the design process. Being customer obsessed. And it kind of kind of all kind of filled. I was thinking about writing a book about you UX research more technical, then I was working in Best Buy. And I kind of struck up a friendship with a guy named Jay reader who was a graduate from the Stanford D school and he kind of introduced me I broadwalk brought him the idea of the book and he kind of introduced me to different material, different kind of literature material that the D school was kind of putting out in their philosophy which I had heard of design thinking but I hadn’t really dove too deep in it. So we had a lot of interesting conversations about you know, design thinking and you know, books like need finding by dead panic, and the books by, by the Kelly brothers kind of I started kind of getting interested in that and, and design thinking I felt was very cool approach to kind of what user researchers were doing already, right when we do user research. We’re trying to make better products and we’re trying to come design help, you know, collect data to help designers and to help product people shape products and make and make really better experiences. And, you know, I found I found that, you know, it was really hard for that will that you know, a way that when you look at the design thinking framework, in many cases research is not so much talked about when you get kind of down into the details they do. Talk about how empathy which is kind of the first pillar the first step requires kind of a journalistic kind of anthropological mindset kind of going out and observing people and asking them questions and watching how they use the product and seeing the context and all of these kind of kind of more research stuff. And I thought that yeah, that this was a really cool model, but I thought that it kind of needed a little bit. It was kind of maybe outdated, a little bit. In many cases, it was it was used for physical products. And now I think, you know, because the world is becoming digitized. So there’s so many now, so many services and products that are now on a screen. And I think that kind of user research, especially for digital products, you know, I think that looking at them as a product I think in at the end of the day, I think some people when they look at, you know, a journey for example, have, you know a bunch of screens to get you to sign up for a loan that that is actually a product if you define if you can, you can actually define it that way. And when you start looking and seeing all the different products, you know that almost if anything, is customer facing and you know, you can use it or, you know, Facebook as a product or Candy Crush as a product, all these things, you can look at them through this kind of new lens as Okay, this is a product, this is a service, then the kind of the design, the D school design thinking kind of applies. And the problem I think one of the problems with the D school is that it just doesn’t go far enough into the methods and into the kind of deep dive that user researchers kind of do the kind of, you know, the ways that we kind of understand problems and the knowledge that we build through our methods is something that can Really, I think benefit the upfront beginning empathy part. So if people, and this was kind of the kind of the theory or the thesis of the book is that if you do your due diligence, and you do your research upfront, before, before, before you even have an idea about what the product is, that will actually help, you know, go waiting out into that kind of unknown area, and kind of relying and believing in research. That, that, that that’s the way to kind of kickstart these products, and then they these projects. And that’s kind of the kind of the idea.

Chris: Yeah, so it’s still pretty new. I’m kind of it’s all outlined right now. That kind of the writing is began, but very slowly, so it’s the tentative title is the universal lens. And yeah, yeah, hopefully I, you know, I can kind of get it get it to completion I kind of have some more time on my hands now with everything that’s happened with kind of the work slowing down because of, you know, Corona and, and all this and social distancing. So hopefully by next year we can I can have something to share, you know, with the world so we’ll see.

Steve: What is the universal lens?

Chris: Yeah, so the, the universal lens is kind of the metaphor is you know, looking at the world and as I was rambling on, you know, before, if you look at the world as everything is a kind of a anything could be a product or service and kind of The way that that products are just basically solutions to needs, then that is kind of the universal lens that this kind of this kind of framework is applicable to whether you’re designing a medical device or your you know, you have a shoe running shoe or a bank app or a chair, it’s all kind of at the end of the day, understanding user needs. Then from based on what those needs are solving them with a service or solving them with a product that will that will at the end of the day, hopefully make that that product successful. You know, we talked a lot about experience. And I like to use the term smoothing the experience. But a and many, many times when you know we as researchers work on these projects, the Product themselves didn’t really seem like it was vetted. It seemed like it kind of like a you ask the question, what is the product for? And, you know, I think that there’s kind of I look at needs as, you know, proximate needs and almost, you know, kind of like. So proximate needs an ultimate needs and an ultimate need is kind of why the product exists in the first place. And then the proximate need is kind of the experience of using that product, right? So if you take for example, a hamburger, that’s a product and the ultimate need is the hunger that it satisfies, right? We as humans need to eat things and hamburgers are one of one of those things that we can eat. And that’s the kind of the universal need that we that it solves. But then there’s also the needs of you know, the needs of actually, you know, getting it, you know, getting to the to the place getting to the restaurant, then there’s the needs of having a good experience in line, being able to read the menu being able to take it to go if I wanted to. So there’s this kind of kind of nested needs within the kind of greater need of why the product exists. Both are important. But I think that product teams don’t always take both into account.

Steve: Well, Chris, thanks so much for being on the podcast. It’s great to hear what you’re working on what you’re thinking about, and just some of these bigger picture ideas, I think are really very interesting. So I’m thanks for taking the time.

Chris: Thanks, Steve. It was a pleasure.

Steve: Thanks for listening! Tell your colleagues about Dollars to Donuts, and give us a review on Apple Podcasts. You can find Dollars to Donuts on Apple Podcasts and Spotify and Google Play and all the places where pods are catched. Visit Portigal dot com slash podcast to get all the episodes with show notes and transcripts. And we’re on Twitter at dollarstodonuts, that’s D O L L R S t o D O N U T S. Our theme music is by Bruce Todd.

This episode of Dollars to Donuts features my interview with Noam Segal, the Director of Research at Wealthfront.

Everyone from PMs to designers, researchers, obviously, engineers, data scientists, marketing, we’re all trying to to understand our clients, we’re all taking part in that process in some way, shape or form. And so I view my role and user research’s role as an enabler, as a coach, as augmenting other efforts already happening in in the company, and really maximizing the returns we get on on the research we do. – Noam Segal

Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.

Steve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.

I just watched an interview with Alice Waters, who describes herself as a chef, author, food activist, and the owner of Chez Panisse Restaurant in Berkeley. She’s often referred to as the creator of California cuisine, back in the 70s. At one point in the conversation, she explained how when she meets people, she will sometimes begin by bringing out food, say fresh fruit, and taste the food together with the other person. She explained how this shared sensory experience was an alternate, and perhaps more effective way of creating a connection between people.

It really made me think, wow, is that something we could do in user research? Could you begin an interview by sharing a sensory experience with someone? It could be taste, via food as Alice Waters did, but it could be a touch experience, a moment of smelling, a shared point of listening. I would want to understand more about what Waters believes this accomplishes and if it’s conjecture or there’s any more evidence, and I don’t really have any idea how to introduce something like this into an already hesitant dynamic, that initial moment.

And with all (hopefully all?) research happening remotely at the moment, is there some sort of shared over distance aspect of this, a sensory experience that both parties could initiate, maybe it could be entirely pedestrian, such as feeling the glass of a mobile device, versus something celebratory like a piece of fruit selected by Alice Waters.

I don’t know what this would lead to but I’ll be curious to hear what happens for people that try it. As well, it serves to remind me that all too often I neglect including all the senses in how I process the world and how I engage with others.

This is another episode without either professional editor or transcriptionist. This podcast is my way to contribute at this particular moment, but I hope you can keep me and my practice in mind for collaborating on research, for coaching, for training, and other work to help advance the maturity of your organization’s research practice, wherever you’re at currently.

Now, let’s get to my interview with Noam Segal, who is the director of research at Wealthfront.

Noam, thanks for being on dollars to donuts. So it’s really great to have you here.

Noam: My pleasure. Thanks so much for having me.

Steve: So my typical way of beginning which we can do today is just to ask, ask you to introduce yourself.

Noam Segal: Sure. So my name is Noam Segal and I am currently on parental leave, actually, with my six month old, baby boy Dean. But when I’m not on leave, I’m director of user research at Wealthfront. So that’s what I’m currently up to. I’m originally from Israel, although I don’t sound like it because my parents are both English and somehow I inherited the British accent. And I moved to the US in 2012 with my wife for graduate school Originally, I got my PhD in psychology at the University of Illinois, Urbana Champaign. And when I graduated, we decided to move to the Bay Area and both pursue positions in tech. And I’ve been in UX research ever since and have worked for Airbnb, Intercom and again these days Wealthfront.

Steve: How did you find out about UX research?

Noam: So, as an Israeli, we have mandatory military service, and I served in a unit within the Israeli Air Force for most of my, my service. That particular unit is in charge of Israel’s missile defense systems. And when I left the military got discharged, I decided to join the military industries. Kind of the Israeli equivalent of maybe Lockheed Martin in general dynamics, those sorts of companies. And I focused on developing the UX, but we didn’t call it UX at the time of these missile defense systems, primarily the our weapon system, which is a system designed to protect as well, from medium to long range missiles. This could not be more different than what I do today. Very little relationship, but that’s when I really started appreciating and and desiring to be in this field of UX more, more broadly, that was my focus, I focused on the interface. I focused on understanding from the operators of these systems, what they what they need, and it was an incredible six year learning experience. So that’s kind of how I got into it in the first place. Later on I wanted to improve my my research abilities and knowledge and I was definitely considering going into academia for a while. So that led me to the Ph. D program. But at some point, in year four of the PhD I realized that I love two things. I love research and being a researcher and I love technology. And there is a profession that marries those two things quite perfectly. And that is UX research. So I decided to shift my focus back to industry and and going to work at first with Airbnb and carried on ever since.

Steve: Did you finish the PhD?

Noam: I did. I did. It was a hectic time in my life. I was in the middle of my fourth year in world Typically a six year program. That said, it’s typically a six year program because of how difficult the academic job market is rather than how long it actually takes to finish a dissertation. And I realized and there was some events in our lives, both personal professional that I want to take a different path from academia. And so I interviewed with several, especially Bay Area companies and was fortunate enough to get a couple of offers and decided to fast track the rest of my my dissertation. And I proposed the research I was going to do in January, and I defended my dissertation on June 1 of 2016. And less than 24 hours later, we were on a plane to San Francisco to stop life in the Bay Area. So it was a crazy period in our life. And I’m rather happy it was over because it was quite difficult. But yes, I did. I did graduate, it was very important for me to finish the program. And I benefited greatly from my studies, my relationship with my advisor, and all of my fellow students at Illinois, really appreciate that time, it taught me more than I could ever imagine.

Steve: So maybe I can just go back. So you kind of described this. And we have like, book ends, maybe have a narrative here, where at the beginning, you are, you know, in the, in the army and kind of looking at those things. And at the end of this chapter anyway, you’re getting on an airplane to San Francisco to enter the, you know, Bay Area, user research roles. But I’m wondering, going back to when you were in the army, you started as you said, You didn’t refer to it as UX. So I’m wondering, when did the research part like that this was a thing that needed to be done that could be done that you could pursue as an actual job. Is there a point at which that started to enter your consciousness?

Noam: Absolutely. So when I was in the military industries, I wasn’t in a research role, efficiently. And I also wasn’t particularly aware of the realm of UX research. But UX research is not really an existing profession in Israel, a separate function typically, in Israeli startup companies, it will be the founders or perhaps the designers that do the research. And so I was not in in a research role, but I was working in a very small team, trying to develop systems that were completely new to the world. incredibly complex, and the stakes couldn’t be higher. These are systems that are enabling Israel to defend itself from potential nuclear attacks. It really is life or death. I mean, in industry in tech, perhaps you’ve heard this before, we often use militaristic terms. Like we have war rooms if we launch a new product, or we conduct a post mortem, when something goes wrong and engineering have to figure out what happened. But I really was in in a company in a team where the stakes were incredibly high. And that requires research, whether you’re in that official role or not. I was in a unique position, where on the one hand, I was working in this team as a civilian, speaking to people who operate the system thinking about the user experience. This is the system where people have, in some cases just a few seconds to make very critical decisions. And failure is simply not an option. But on the other hand, I was also an operator myself. So I was still in the reserves, still serving in the military still doing one or two days a month in some cases in the unit as a reserve soldier. And so I got to see and experience these systems from those two, two sides of the coin, which was very, very interesting. And obviously, this happens to to us, as well here in the Bay Area, often we are using the systems we’re also developing, that’s quite common. But that was the first time that happened to me. And I think one thing that really struck me developing these systems and I think it’s part of the reason I decided to move on is that I really, truly believe that failure is a critical component of learning. And failure is not really an option when what you’re developing is missile missile defense system. Um, and that’s a very heavy weight to carry on your shoulders. For anyone who is part of that sort of team. It’s a huge responsibility. It’s not the type of intuitive work that we do in the Bay Area, you cannot fail often, and, and learn, you know, move quickly and break things. You can’t do that when you’re in that type of industry. And so on the one hand, it was an incredible experience and really my first venture into UX. And on the other hand, I really needed to move on in order to become a better researcher, and a better professional within the UX research community. So that’s a bit about my time there.

Steve: Yeah. And what did you do for undergraduate?

Noam: So I went to a small kind of liberal arts college in Israel. And I studied psychology as my major. And I took kind of Media Communications as my, my minor. I was in the first cohort of psychology studies at this college, they just opened the psychology school the way I started. And maybe maybe it’s a personality thing, but I love to be part of new programs, new things, new ventures. And then I had other options to go study at more established universities, which you may have heard of Tel Aviv University as an example. I really wanted to join this this new program, and I have to say it benefited me greatly. The learning experience was incredible. We were a small cohorts of people. We have very close relationships with our professors. And the fact that I went there was a big part of I think, why I ended up getting into graduate school in the US and being able to make this move and ended up where where I am today. So it was a wonderful undergraduate experience, very small campus. My first time moving out to my parents house and and experiencing a lot of lots of new things. I really appreciate that. That time and the psychology the foundations of psychology gave me which were very helpful later on. And until today, really.

Steve: So tell me what kind of company Wealthfront is.

Noam: So Wealthfront, I would say has been through a few different periods, different stages. It began as a platform where you can innovate vest your money passively, passively invest your money in in the markets later on Wealthfront added planning features to enable people to plan for their financial future. And most recently, Wealthfront has added a cash account, a high interest cash account. And later on in the year, we’ll also be launching a bank account for our clients. And so I would say in general, what we’re trying to do at Wealthfront is provide people with incredibly sophisticated tools to manage their finances, manage their investments, manage their cash, plan for their future, and hopefully, achieve their their life goals. Through through well planned and invested finances.

Steve: Yeah, what’s the research practice like at Wealthfront.

Noam: So it’s interesting, you should use the term research practice because research at Wealthfront is quite different than what I experienced elsewhere. First of all, when I joined Wealthfront, the research team was not the UX research team. It was the team in charge of the algorithms and the math powering our passive investment engine. And these days, that team is called the data science And there’s a research component to that. So in FinTech research actually means something different than in most other companies and that took getting used to for me. But research UX research at Wealthfront is an interesting topic because as a practice, it has existed since the company started. And everyone from the founders of the company to later on PMs and designers at Wealthfront have all been conducting variety of research, project methods, and staying connected and trying to understand and best help our clients and our potential clients. But user research as a function didn’t really exist that Wealthfront before I joined just over a year ago. So I’m the first person at Wealthfront with user research in his title. That’s maybe not completely accurate because my manager, a picture, who is VP of design at Wealthfront actually started out as a researcher, but she transitioned into a design role quite a while ago. And I was the first person since to join as a researcher. So as a practice, it’s existed for for over a decade now, as a function, what I’m trying to do Wealthfront is really build out research as a function, which means a bunch of a bunch of things. But really, the point is that research at Wealthfront is incredibly democratized. And I very much support that approach. And so by democratized I mean that we’re all running research trying to better understand our clients and potential clients. Everyone from pm to designers, researchers, obviously, engineers, data scientists, marketing, we’re all trying to to understand our clients, we’re all taking part in that process in some way, shape or form. And so I view my role in many cases and user research his role as an enabler as a coach, as augmenting other efforts already happening in in the company, and really maximizing the returns we get on on the research we do.

Steve: What are some things that you do in coach examples or it’s such a powerful word? What are ways that you are coaching?

Noam: Yeah, so we are currently two user researchers on the team. seem to be a third. Hopefully, we’re currently in a, in a weird situation all of us as we’re recording this with COVID-19 out there and a global pandemic. But we are growing the function of user research. Until then, and regardless of that, we hope to enable especially designers and PMs more than anyone else to do better research. And to do that twit were using a few different techniques. One of them is creating lots of materials in our internal company wiki around best practices for a variety of methods and research techniques. There’s no reason to reinvent the wheel every time and rethink how we should conduct a usability test or what best practices interviews are or what type of satisfaction metrics we can use, etc, etc. And so the first step there is to make sure those materials are out there for everyone to consume. And that’s a work in progress for us. The second thing is running workshops or interactive lessons, webinars, whatever whatever it takes to engage with various teams within the company, and enable them to learn in a group setting, about research best practices, and how they can level up their research. And then another technique or another way we do this is indeed, just one on one coaching, with with designers with with pm. We started doing this a bit. I hope when I get back from parental leave to continue doing that. That’s all of interest from many people in the company to have such sessions and get feedback on on their research sessions and what could be better, and what should what should remain the same because it’s already great. Our designers and PMs are well versed in research and they’ve conducted countless research sessions of all sorts. But there’s always something to learn for all of us. And so through those various methods, whether it’s on an internal wiki or whether it’s in a group setting, or whether it’s in a one on one coaching session, we believe that through those various ways of coaching and teaching, we can all do better research. Not to mention, teaching is maybe one of the best ways of learning, in my opinion. And so one benefit I feel of being a coach and an enabler and a teacher is that you have to constantly reevaluate and rethink what you’re doing. And then about the methods you’re using, about the best practices about the principles behind your research, everything you’re trying to, to accomplish. And I’m really happy about that because I’m not sure as a community within user research we’re doing that enough necessarily so i think it’s it’s a it’s a positive side effects of of taking that coaching approach.

Steve: So what is it that we’re not as a community overall doing enough?

Noam: Debatable, but I’m, I’m referring to revisiting the methods and techniques that we’re using as researchers. Use the research or UX research method Dollar discussions

Steve: Anything there anything specific that you’re thinking of that is unexamined that needs to be reconsidered or revisited?

Noam: Well, I’m curious to hear your opinion as well, I, I feel like many of the techniques or frameworks were using in user research. And one specific example I can give is Jobs To Be Done, which I really learned the most about in my time at intercom. Most of those techniques and methods are taken from other realms such as marketing. In the case of in the case of Jobs To Be Done, it really started out more than it was kind of imported into the world of software development and and product And I wonder if I do often wonder if there’s more room for user research specific methodological discussions and innovation around the methods we use? I think broadly speaking, I haven’t seen much of that. And I think it’s debatable whether that’s needed, or, or not. But I think that as we develop new tools, new ways of working remote research is a classic example right now, given the global pandemic, when there’s also a need to establish potentially new methods to go with that reality with those technologies and think about how we do how we do research. The fact that many of our methods and frameworks are imported from other realms, is maybe just partially a sign of how long UX UI Such has been going on as a distinct profession and function. But maybe it’s time for us to to branch out ourselves and dive more deeply into into those methods we use. So hopefully that answers some of the question them.

Steve: What’s the granularity of a method like Jobs To Be Done to me, and I know you’re talking about methods and frameworks, you know, I’m just thinking there’s, like remote research is an umbrella term for a certain set of things, Jobs To Be Done is. And I’m thinking about, you know, he talks about innovating in the methods. I’m just not sure. We all that’s such a great word and problematic because we all think it means something different. So. Right, I’m thinking about some of the things that have come up in remote research when people were saying and I’m not gonna have exactly right, but how can I see my screen and the other person’s screen if they’re on a mobile device, and then people have come up with various solutions and sort of technologies and ways to hook up different pieces together so that you can remotely have a certain kind of interaction is what we consider that and innovation in the methods.

Noam: So, I view that example as more of the medium than the method. Yeah. So if we take eye tracking, as a method, because I do consider eye tracking a method and to maybe one way to define the method given that is a way to, to I’m kind of losing my own train of thought here, but kind of, I was gonna say like a way to gather insights, but that’s also the medium right? So let’s kind of scratch that for And then. So, thinking about eye tracking, eye tracking can be done in person and it can also be done remotely. So, that component the in person or remote component that to me is not an inherent or core part of eye tracking as a method. I think what makes eye tracking a method or the point is that this enables us to glean certain insights by by measurement of where people are focusing their attention, what people are prioritizing what people are going back to, etc. and we can take that a few steps, a few steps further. We know already that what people say is only a citizen part of the picture and Other parts like body language, and other ways in which people convey sentiment are not conveyed in, in, in verbal form. And so I’m personally very interested in kind of sentiment analysis within interviews, because I think that would add a lot of richness to to what we do as researchers, and we’re starting to see the technologies that could enable such methods of sentiment analysis within qualitative research to happen. I haven’t seen much of that research go on, I’m certainly not doing it myself. I don’t think we have adopted the technologies that Wealthfront to conduct that type of research, but I find it intriguing. So that would be an example for me of other methods in this case, sentiment analysis. It’s a novel implementation of that method, because we’ve been doing sentiment analysis for text for a while now and and we can take a bunch of tweets or take a bunch of feedback from the App Store and analyze it for full service and sentiment, but within one on one interviews, whether remote or in person, I don’t think I’ve seen much if any of that happening. And so that would be an example of a methodological innovation in my mind, that we could be thinking about moving forward. That there are also you know, in the realm of psychology, for example, in academia, I mean, there are a bunch of researchers who are dedicated to to methodology, essentially, or things like developing new statistical methods or packaging. For platforms like AR, to enable others to analyze their data better, and do more rigorous science and get better results. We don’t really have that function. I would suggest in UX research, we have researchers, research ops has been on on the rise for a few years now as and is an incredibly important component of research. I don’t think I’ve seen anyone claim the title or show particular interest in UX research methodology as a focus as a professional focus, whether it’s within any particular company or research team or whether it’s, you know, a freelancer who doesn’t who isn’t affiliated with any particular company. So that to me is another another potential step our field could, could move towards to advance our methods. Not sure I planned to dive so deeply into that so tell me if you find it interesting.

Steve: Oh, I find it really really interesting. And, you know, it evokes for me, I mean, just your proposal that we’re not innovating within UX research around methods is just, I mean it’s a strong point of view and I don’t disagree I just hadn’t really ever thought of that. It just reminds me of, you know, in my trajectory, a very early era for me when it felt like there were lots of innovative methods. And so a couple of things come to mind that maybe I’ll just throw out and see, you know how this fits into your into your kind of hand waving. I’m waving my hands kind of an architecture of war. You know, what’s innovative? What’s sort of? What’s a method? What’s a medium? I’m thinking about Bill Gaver, who I think was at Xerox PARC or at EuroPARC at the time, I can’t really remember. I mean, those both might be wrong. But he came up with something called cultural probes. Which is, and I’m also not going to get this right. But it was, I mean, a form of a diary study, but you were kind of creating kits to send to people. So it was asynchronous. It was at a distance. It was analog, but not necessarily analog. I’m recalling it properly. And you know, it was just, and maybe some of those things. We don’t you know, something’s an innovation and then it just becomes every day. Like, I mean, I’m describing, putting a packet together of things and sending to people well You know, if you go back to when we had before digital cameras for a while there were disposable cameras and they were cheap and you could like just buy them and send them off to people and they would take pictures and send them back and you know, now we have discount there’s a platform for recording your own video. So the technology changes and you know, this idea of cultural probes which is like a pretty great term. I always thought it kind of it doesn’t feel innovative anymore, but it was I think it was you know, came out of came out of academia or kind of the academic labs in industry there was a group in Europe I wish I don’t I don’t off the top of my head know what the sources but I’ll look it up. They had a method called moving with a magic thing. I wish I hope I have that right. And that was basically if you want to understand that behavior, you handed somebody a block of wood. And I think you either described to them what functionality we have, or they don’t remember whether you determined or they determined, but they would carry it around for a while and then come back and report on. You know what it changed for them even though you were so just now we have, you know, we have a minimum viable product or we have, you know, a paper prototype, but we have lots of ways of sort of creating the appearance of capability. But this was very mean. Sort of a little hand wavy and very European and sort of rooted in a in a period in the, I don’t know, maybe the late 90s early 2000s those felts exciting to me at the time, and then they they are so I don’t know, I guess My question, you know, I don’t know if you’ve heard of those things or how they kind of fit into your, you know, thoughts about what methods are and what it means for methods to be innovative. And is that changing just you know, as decades fly by?

Noam: So it’s a question. I’m not even sure I have the answer to, I think we have to ask ourselves, where is the ball or what’s the tipping point where we can say that we have innovated on a method and diary studies are a perfect example you just mentioned them. Diary studies have been going on for decades now. And as you mentioned, they started out with people sending physical packets to to other people’s mailboxes and waiting for responses and just doing that, you know, a few times until they get the results they need. And indeed, these days, we Have modern fantastic platforms like dscout for diary studies, and they have completely digitized all of the process and have also added a bunch of functionality that was obviously not available to two research shows back in, in in, in previous decades such as video responses, for example, which offer an incredible richness to, to our, our clients, our users responses and to our understanding of what they need and what they want from our product. But I think that also begs the question, is that a fundamentally different method or is that an innovation on on the method that was used previously by sending people a letter or a packet in the mail or not? And I’m not sure what the answer to that is. I think it could be very, very subjective in my opinion. It’s it’s possible that let me rephrase that, in my opinion. I think it it’s fair to say that diary studies haven’t fundamentally changed in a while now. should they? I don’t know. But as a method, I’m not sure. It’s, it’s it’s evolved enough where we can say that we’ve completely innovated or we’ve reached a new era in diary studies.I wonder how you feel about that.

Steve: Yeah, I agree. And I think you’re I agree with you. Acknowledging that some of this is not it’s not answerable. At least, you know, I mean, I’m trying to I’m trying to get you to answer that’s my job here. But you also acknowledging like, yeah, we can’t really answer that, and I don’t disagree at all. And I think that’s a good, you know, I’m starting to server sort of see, like, yeah, diary studies or diary studies, and we do them differently and they’re richer and we have, you know, the pace is different and so on. But it’s still, it’s still the same kind of interaction with people. We’re trying to get a certain kind of information. You know, talk about sentiment analysis versus just, you know, an in person interview. That’s a very different type of data that’s being pulled into, into the research that, that I think we’re not getting now.

Noam: I’ll tell you, I’ll tell you what, though. I’m when I was in graduate school, I was in the social personality division within the psychology department, and obviously social and personality psychologists differ quite a bit in what they find most important social psychologists have focusing on the effects of the situation and the context people are in. Whereas personality psychologists are focusing on individual differences in characteristics and how they drive variation in in behavior and outcome in in real life. I think one thing that has potentially greatly affected and and maybe this is an innovation in, say diary studies, is the fact that people are still mostly self reporting, whether it’s in text or video, but they’re doing so on a mobile device. And they’re doing so in various places and contacts, which we know because, you know, potentially we can get that information as well. So these days, as we can all imagine, there’s a lot of information about each of us out there, online and our behaviors. It’s a very rich data set. It raises great ethical questions, security questions, other questions about how we all want to live our lives. But the fact is today, more than ever before, there’s a lot more data out there, not just about what we’re reporting and and what we’re answering in, whether it’s those written packets from the past or modern diary entries in the present through platforms like discounts. There’s also all of the context around that when it happened, wherever it happened, and countless other things. And to me that is potentially an innovation because it bridges between the two realms I was once part of and will always consider myself pause which is the social aspect and the individual aspects. And of course, we can add to that the cultural aspects and so forth and so forth. So, potentially, if, if, obviously, we get those data and people agreed to provide those data, we can get a much richer picture than ever before, about people’s behaviors about the context in which they happens in so many other things. And, and maybe that is a very meaningful step forward in how well we understand people.

Steve: It makes me realize and, you know, depending on your background, this is not going to be particularly novel observation, but a lot of research that, you know, we do in the field is focused on an individual. And, you know, we might occasionally look at a group of people, if we’re looking at something that takes place in a family or in a household, maybe some work processes, but we don’t seem to research across a network necessarily. I got at least I haven’t come across it.

Noam: That, yeah, this is something that absolutely resonates with me. Again, going back to my my academic days, I was working with an advisor who was doing a lot of what we call dyadic research or couples research, because we were working on romantic relationships, dating, online dating, attachment, stuff like that. And fast forward to to today at Wealthfront. Where we work where our work is about people’s financial lives. Then if you’re if you’re married or in a long term relationship, and you have joint finances, with your partner Now, then, obviously many of the questions on your mind, many of the needs you have are joint needs and joint questions. They are not individual needs or individual questions. But all too often, including at Wealthfront. And everywhere else I’ve worked, we do do the research at the individual level. And that is a very partial picture and a very biased picture of of people’s people’s realities. You can imagine that if you have to two people who are in a long term relationship and you ask each of them separately, about their finances and where they’re at and where they want to be. You may just get very different answers that don’t take the other person into account enough. You’ll also get very interesting answers when you ask each member Have that family to report on the other member of that family and what they think the other member believes about, in this case, finance. So, I definitely agree that one of the ways in which we can think, among other things about methodological innovation is the level at which we’re conducting the research. Is it the individual level? Is it the couple or dyadic? level, the group level? And when should we be focusing on that particular level? And also, should we be can we develop methods that are uniquely tailored to that level of of research? How can we conduct something like a joint interview in a way that will yield more robust insights? And does it need to be substantially different than how we’d interview Each of the individuals within that relationship

Steve: If I was just gonna wave my hands and imagine anything is possible, you know, having done a little bit about this topic and in my career as well. Seems like, you know, the family childhood family upbringing is a huge determining factor for a lot of beliefs and sort of why couples have different approaches because they have different backgrounds. I’m oversimplifying it, but boy if you wanted to, you know what, if you could maybe not literally go back in time, but it’d be interesting while we’re just you know, imagining Anything is possible to you know, interview the group that sharing the decision, but also learn about the, you know, the origins of those assumptions, rules, expectations, beliefs, and so on. From let’s just say it’s the family that they grew up in, you know, that I have no idea how you do that, from just a like a recruiting point of view or a logistical point of view or making sense of the data point of view. But just thinking about people as networks and where there are people as being in networks and where a lot of things that drive their behavior come from are not at the individual level. It does open it up in a way that I don’t know how to address.

Noam: Well, I too, struggle with that. And I, I struggled with even imagining what the methods would look like in order to get at the historical context of an individual or family, but I can tell you this. I have two children. And I have more photographs of my children than I do know than I ever imagined. I would have I have more data essentially about my children and I will always have that That data, then then I ever imagined would be would be possible that the richness of what I know about my children and could potentially share is, is crazy. Or another example is we can all go to Google or Facebook and look at years and years worth of history of where we’ve been, what we were up to, what you know, what we ate, and who we hung out with. And so, the next generation of researchers will certainly have access to an incredible trove of historical data. Hopefully, they get the proper permissions to access those data, and and respect the privacy of the individuals they’re researching. But assuming they do, I actually think it’ll be easier than ever before. To get such an incredible picture of what led the people wanted researching to the point where they’re at in their life. If you combine all of those data together with advanced analysis methods where we’d have the support of various fancy machine learning AI driven algorithms, then I actually do think that in just a few years, it will be feasible to understand people at a level where we’ve never been before, in particular, what you were just mentioning around where where everything began for those people and how they were raised and, and, and, and how they got to that personal and professional point in their lives. So I do see glimmers of or a glimpse into this potential future. I often wonder whether it will be useful or not, and how we’d analyze such data. And is this truly the future of our profession, but what I’m sure have is that too It will be a possibility, given everything that we we know about people and how we conduct ourselves these days with technology.

Steve: We’ve talked about a couple of things. So far and just very broadly, one is, you know, how research is working, you know, the roles that you’re playing, what you’re trying to help people accomplish at Wealthfront. And then, you know, your, your broader kind of observation that, that in terms of our methods, we’re not really focused on them as a practice, you know, as a profession, are there things that you can do? I like what you said, you know, this, that innovation and methods could come from anywhere. But is it something that there’s an opportunity for, for for Wealthfront, for you to be pushing methods ahead.

Noam: So I think that one of the beneficial results of democratizing research as extremely as possible and enabling everyone in the company whether it’s designers, PMs, marketers or others to conduct incredible research, and I’m fortunate enough to have colleagues who have amazing research chops and they do incredible research that frees up part of the time of the user research team. And and us as user researchers to focus on a some of the more strategic research projects that we have in the company. And it also will be it’s enabled us to focus on on research innovation, and either implementing tools that allow for better research to happen, for example, better knowledge management, of the research we already have, and are already doing and it also enables us potentially to focus More on developing tailored methods or frameworks that work for our context. And FinTech is in many ways, different or unique compared to to other areas. And there are certain things that we care deeply about. And so I think that the result of democratizing research as you assuming you accept that, that is a viable route to go down is that you are able to focus more on on other things, including that innovation I mentioned before. So we will definitely continue to try on the user research team to either implement new tools and platforms that are coming out that make us all better researchers, or innovate within the company. And and of course, if we can share those innovations with the broader community In order to develop methods that are better, a better fit better tailored to what we do, I can give. I can give an example here of, you know, something we care about deeply I would say in FinTech especially, is, of course, around the issue of trust. And the measurement of trust, which has, I would say, been a tricky thing, historically, for researchers, both in academia and industry. At Wealthfront, we are often handling people’s entire life savings investments. That’s a very big responsibility which we take incredibly seriously. And we want to understand how we can earn people’s trust, how we can build on that trust how we can augment it. We want to monitor the trust in us and and understand it deeply. And the measurement of trust. Well, measurement in general, is incredibly complex. And so developing great measurements for trust is definitely a goal, which I’m not sure we’ve, well I actually am sure we haven’t yet reached at Wellington, we aspire to do that. I’m not sure if anyone has. I think that as a research community, we can barely agree on how to measure satisfaction. And often we use deeply flawed methods just to do that. And in my opinion, measuring satisfaction is much easier than measuring something like trust. So more to come in the future, I hope including from us.

Steve: Maybe in the we have a few minutes left, it would be interesting, hopefully, to hear you. I’m going to start again in a few minutes that we have left I would love to hear a little bit about maybe a compare and contrast. The other places that you mentioned you’d worked Airbnb and intercom, you know, what are some of your perspectives on, you know, research being applied in very different organizations and what you experienced as a researcher in these different types of companies.

Noam: I’d love to do that. And by the way, I have as much time as you need because I feel like we covered a very specific thing for most of this conversation though I do find it fascinating. So for me, Airbnb was an incredible experience. I think what drew me to Airbnb more than anything was the mission at the time, which I believe is still Airbnb, his mission of belonging anywhere as an immigrant to this country, and as someone who’s traveled quite a bit and be To a few places around the world, I definitely know what it feels like not to belong somewhere. And I definitely know what it feels like to feel belonging. And I think that today more than ever before, it’s really important to break out of one’s bubble experience other cultures experience other places and hopefully experience that belonging. So, to me as a researcher, Airbnb, I had a very strong connection to to the mission. And there are some very unique aspects of research at Airbnb, for example. Airbnb is one of those products where the company’s aim is for people to use the app essentially as little as possible. And and venture out into the world the way Airbnb makes money. And grows its platform. And its benefits to the world is when people go out to travel not engage with with the app. And so a lot of the research at Airbnb isn’t just about the online app or web experience. It’s about the offline experience on the ground. It’s about the travel itself. It’s about things like the check in and the checkout experience, or what it’s like to go on an experience hosted by someone who lives in that city. And they happen to be experts at making pasta and they want to share their food passions with with the world. And so researching these offline experiences in a cross cultural settings in almost all countries across the world, is a very different experience than some people I would say have been in the UX research industry. Also, I credited with Be personally as a company founded by two designers and one non designer, but it’s a very design driven company. And in my mind, Airbnb is one of the best design schools in the world without officially being a design school. I learned most of what I know about design. And a lot of what I know about research for my time, at Airbnb. And the travel space is, is very challenging and very interesting. And I hope that in this time, when people aren’t traveling at all given a global pandemic, I hope we get back to that soon because it makes all of our lives a lot richer and a lot more interesting. And it’s some of the most fascinating research I’ve ever done. Another thing about about Airbnb, which is also a question I think in our industry, is how does the function of research operate within the larger organization? For example, at Airbnb researches is part of the design team. Whereas at Intercom for example, research was reporting into one part of the product function. We also have a bunch of different titles in our industry from experience research to design research to product research, user research and so many other things. And I know we discussed this a lot, but it does make a difference to one’s experience. Whether they work more closely with product and PMs or more closely with designers and, and design leaders. It really influences one’s experience, the things you can learn your your relationships with those functions and so forth.

Steve: But there’s a caveat to that I think it in looking at the two companies that you’re comparing and having not worked at either the missus, you know, definitely outside or hypothesizing but I could see be if I worked at Airbnb, I’d want to be working with him. He said, it’s a design. It’s a strong design company. And I think of intercom as a strong product company. And that, so it’s different where it’s different sort of who you report up to, or what organization or what roles you’re working closely with. But also that’s that is different in different organizations based on just the strength and maturity of those different practices.

Noam: Yes, I agree.

Steve: so just to clarify, You know, there’s a lot of different titles or the titles connected, is there a relationship between, you know what, what your job title is. And this this issue sort of, you know, who you report up to where the research organization sits, is that impacting the range of how titles are, are kind of applied to researchers.

Noam: So I think the titles we get as researchers are a function of a few things that are a function of the organization, their functional, its philosophy. They’re a function of how the organization itself is designed, and what the emphasis is on and what type of relationships the organization wants to cultivate more and how the company wants to be perceived, as well. I want to mention something about our work. At Wealthfront these days, some relationships are, in my opinion, so natural that it’s surprising that they don’t just occur within companies and don’t just happen and don’t just thrive without any particular intervention or organizational design. And yet more often than not, those relationships are missing at Wealthfront. We recently created something we call the research Guild. And the research guild is composed of user researchers like self consumer insights, which is typically the side of research that kind of belongs organizationally to marketing or perhaps in some cases, product and data science. Now, again, the relationship between user research, consumer insights and data science, in my opinion, is natural and obvious. And yet all too often, it’s not a thriving relationship, it might even be a very broken relationship. And so what we’re trying to create wealth and these days is is a much closer relationship between those functions Because ultimately, we are all trying to do the same thing, which is enable every single person in the company to understand our clients and potential gains and better to be closer to them. To hear them in a variety of ways. Whether qualitative or quantitative, it doesn’t matter. high level, we all have the same mission. And, and so maybe more, maybe something more important than the titles we have and constant discussions around what UX is or what UX isn’t or who is or who isn’t doing researcher, I think we might want to focus more on the collaboration between functions and the relationships between functions. And to really amplify these truly important crucial relationships between, for example, data science and, and user research. It makes all of our jobs better, it creates better outcome to create a better understanding of our users and our clients. And it’s, it’s one more thing I haven’t always experienced in my career and I wish I had so definitely trying to, to correct that now at Wealthfront and and create those, those beneficial cross functional relationships.

Steve: So how will this guild enable those relationships?

Noam: So what I’m again when I’m not on parental leave and actually working we have both leadership meetings for that guild with the managers or directors of each each group to keep each other in the know about projects and things going on and what we’re doing and how we can help each other and collaborate better. And we also have meetings between all of the guild members where we can provide each other with feedback on projects, whichever stage those projects are at, or ideas for new new directions or support, where support can be, can be provided. And anything else we can come up with. And just by having those couple of meetings, I think we’ve already seen so much benefit in terms of every project we conduct being that much stronger thanks to much broader feedback, much deeper feedback. And a variety of perspectives from people who have a very diverse set of skills, academic backgrounds industry experience, and to focus on on a variety of things in in their day to day.

Steve: And how did you come to use the term guild to describe this?

Noam: So I’m, I’m all about etymology, and very into into words, I think what drew me to the term guild and the reason I like that suggestion is that in most dictionary definitions of Guild, you have the notion of an association of people who are in the pursuit of a common goal. And I think that’s what’s important to remember that data scientists, consumer insights, or marketing research people, user research professionals, we are all in pursuit of a common goal at the, at the high level. And we should be collaborating as closely as possible in order to, to to reach that goal. And I think all too often we don’t understand each other’s work well enough, we don’t collaborate well enough and leverage each other’s work, we might sometimes repeat each other’s work or or you know, have like dual projects which are quite similar in their, in their outcomes. And so by focusing efforts on that common goal, and leveraging each other’s knowledge and skills. I think there’s something very powerful in that, which definitely exists in a bunch of companies in, in, in, in tech and otherwise, but could be could be stronger for sure.

Steve: So the thing about podcasts, I think, is that they, you know, we’re having a conversation on the day that we’re having it and someone may come to this at any point, you know, in the very near future or later on. So, you know, you’ve you’ve referenced our particular moments in time a couple of times or in the midst of a global pandemic. It’s really changing well as You know everything about everything for everybody right to work from home. People are sheltering in place right now in the Bay Area. And it’s forcing a lot of questions. Or, you know, for some and maybe not for others about research. You know, what we’re doing right now sort of how do we even think about how do we think about research our profession in a time where these other things are going on? And I’m wondering if that’s something that you’ve reflected on if you have a point of view about today about research today?

Noam: Yeah, I’ve definitely reflected on this, especially about a couple of points. The first is that I’ve, I’ve been passed off and seen lots of discussion around research ethics in this kind of period of time. People are potentially in also distress, a lot of anxiety. And so there’s the question of, it’s perhaps a philosophical question even of whether we should be approaching our users at a time like this. And what can we make of people’s responses when they’re in such a different and often distressed state of mine, given given the pandemic? My view on this is that it’s it’s crucial to continue to be such that such at such a time, certainly with more sensitivity and and a realization that there are probably limitations to to the research. But as someone who works in financial technology at a time when the financial markets are incredibly, incredibly volatile, I am personally happy that we are continuing at welfare to be attuned to what our clients need and what they hope for the future and to provide that for them. Because it’s exactly this type of situation where we want to to best serve our our clients and help them avoid potential mistakes that could be very costly to them. So I think there’s a need for much greater sensitivity at this kind of time. But I think researchers have an important role in pushing forward with research as much as possible. Maybe taking on additional projects that that benefit the health or medical services, community, and so forth. The other thing is, I think a big part of this past maybe decades and use research has been around research Ops, and how we can conduct research in a more lean fashion, fast paced, lightweight. For many companies, in person research travel, it’s a core part of the research practice. And yet today, we’re having to grapple with with this fundamental problem that we can’t do that anymore. And we’re forced to develop new tools and new ways of of getting to and connecting with our users, even when we can’t do that in person. And I think it’s important for every every company and every team and every researcher to think about those different contingencies and What can we do in a situation where we want to connect with the user, we want to understand them, but we have to do so in a in a remote interview or some other technique, which doesn’t include meeting in person. And this goes back to our discussion around methods and methodological innovation and what we can, what we can do around that. I personally hope that in the future, when maybe this time, augmented reality, virtual reality, other techniques become more of an established practice that we will be able to share spaces and share ideas and feel together and conduct kind of pseudo in person research even if we’re not. And I think there’s something very, very powerful in that. In fact, maybe that will drive part of the growth finally in in those types of methods, this type of event, I think it’s going to be very interesting to see the impact of this double pandemic on how we conduct our, our research, because ultimately, our profession is all about connecting with people and understanding them. And that is much, much harder to do when we’re all quarantined in our own houses, waiting for this to, to end.

Steve: Well, I thank you so much for your time. And, you know, it’s fun to go really deep into a few things. I appreciate you, you know, going all the way, all the way into the nooks and crannies of some of these ideas that you’re thinking about gives us a lot to chew on. So thank you so much for being on dollars to donuts. Thank you as well. It was a real pleasure.

Noam: I’ve been waiting for this. And I’m really glad you invited me. I appreciate it. Thank you.

Steve: Thanks for listening! Tell your colleagues about Dollars to Donuts, and give us a review on Apple Podcasts. You can find Dollars to Donuts on Apple Podcasts and Spotify and Google Play and all the places where pods are catched. Visit Portigal dot com slash podcast to get all the episodes with show notes and transcripts. And we’re on Twitter at dollarstodonuts, that’s D O L L R S t o D O N U T S. Our theme music is by Bruce Todd.

In this episode of Dollars to Donuts I speak with Laith Ullaby, the Head of Research at Udemy.

I’m really into the idea of questioning what we do. That can be the methods and that conversation about getting out of our comfort zone. It can be thinking about our relationships with stakeholders and trying to reimagine and iterate on those. And it can be thinking about the historical trajectory of the field and the legacies that that has imbued us with. And I think that being ready to iterate and question the assumptions that a lot of those things are built on, is the thing that I’d really want folks to come away with. – Laith Ullaby

Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.

Steve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.

I had a profound moment this week. I was facilitating a meeting, not a user research setting at all. But we were having a fairly open-ended conversation. One of the people in the meeting shared some very specific advice for the rest of us, offering almost an impromptu speech or pep talk, filled with passion and encouragement. It was very inspiring. But as the facilitator, perhaps because of my work as a user researcher, I wondered if maybe that wasn’t enough. This person articulated their richly-realized state of being, and it may be easy for anyone else to dismiss it, oh, that’s just so-and-so, they’re just like that. But, we’re not born with insight about ourselves, or clarity about a new way of being, so I was curious – and I thought it would be helpful – to understand more about how this person accomplished this. How did they get here? And even though I wasn’t asking followups in this session, it seemed like an opportunity to shift my role slightly. And so I asked “How did you get to this stage, where you have this clarity in your approach?” This person paused, and said, “Well, if you really want to know” and then proceeded to share very personal details about their life, from hardship through to today and what they had overcome and how that had shaped them. The details aren’t important for my story, though, and they really belong to the individual who chose to share them, not me. Anyway, it was an impactful moment for all of us in the meeting. And it reminded me about a fundamental in user research, that you don’t know what’s behind a door, it could be something banal, I mean, it usually is, right, but you don’t know until you open it. And sometimes what you’ll find is new, surprising, and impactful. But if you assume that it’s always banal, you’ll never try to cross those thresholds, and never have the opportunity to hear those significant perspectives. I’m describing this as a doorway and that what we do as researchers do is step into that space, but I don’t think that’s sufficient to capture what it is that we do. Yes, we are stepping into something ourselves, but also, and probably most importantly, is that we’re making space for someone else, choosing to ask that question, to be interested, invites someone else to take a step forward themselves. You can call this a holding environment, or a safe space, but I think the additional point I want to make is that it’s not just what you do as a user researcher, it’s what you do for someone else.

I’m putting another episode here done without the benefit of a professional editor or transcriptionist. I’m focusing in the near term on how I can contribute but of course this is a difficult time for many people including those who operate a small business. It’s a challenge right now for Portigal Consulting. If you’re in a position to consider me as a partner, that’s appreciated. I lead research projects, I help companies build a more research practice including assessments, training, and coaching. Thank you!

Okay here we go with my conversation with Laith Ullaby, the Head of Research at Udemy.

Well, Laith, thanks so much for being on Dollars to Donuts. It’s great to have the chance to speak with you today.

Laith Ullaby: Pleasure to be here excited for the conversation.

Steve: All right, well, let’s, let’s get into it. Why don’t you introduce yourself?

Laith: Sure. So my name is Laith Ullaby. I’m currently the head of research at Udemy, which is an online education platform. I have a academic background in ethnography, and have bounced around the UX research world for a few years now. And then I’m also really passionate about teaching. So for the last few years, I’ve been teaching at the UC Berkeley School of Information and at the UC Berkeley Extension.

Steve: So what does it mean to have a degree in ethnography? Not? I feel like that’s not a phrase that everyone can say exactly.

Laith: Sure. So technically, the field that I studied was ethnomusicology. Which is using ethnographic methods to look at the intersection of music and popular culture, and so my focus was on how governments in the Middle East, were using technology to disseminate national identity through folk culture. So it was a little niche. But, you know, it was fun to sort of build that foundation and ethnographic research and think about how sort of the interplay between popular culture and technology.

Steve: Do you? Are there echoes of that in, you know, your daily practice? Like, can you draw the threads back to what your what that program was about for you what that research was about?

Laith: For sure, you know, I think that the, the, you know, when I think about user research, it’s really combining the strands from a lot of different places, different academic places, different you know, parts of sort of the applied world. And one of those is sort of the design anthropology world which I think is really steeped in ethnography and for me that really raises some of these really interesting questions about identity and community and how we sort of come to understand ourselves, as well as power relationships. So working on a marketplace for education, thinking cross culturally about, you know, who feels empowered to be a creator of knowledge, I think really hearkens back to some of the core areas of exploration that you might encounter in an academic program like that.

Steve: Maybe, can you say more about, you know, the organization where you’re at? What, what, what Udemy does, and a marketplace for education as a compelling phrase, maybe you can impact that?

Laith: Sure. Yeah, I’d love to. So, um, as opposed to some other, you know, awesome companies in the space. We’re not a publisher, we’re a marketplace. And so we have a platform for instructors to come and to find learners. So just to get a sense of the scale, we have 50 million students, 57,000 instructors and over 150,000 courses. So, you know, in other models, we might say, Hey, you know, here’s sort of the Python course that we think is cool. And in our model, you know, any instructor can come on and make any variation of a Python course. And then it’s those sort of marketplace dynamics like reviews and number of enrollments and social proof and those kind of things like you might do on an e commerce platform that folks would be familiar with, in which students then get to evaluate what’s the right course for their learning need. So it’s really exciting to sort of see that that proliferation and that there’s so many different approaches to you know, finding the right student, the right course with the right instructor at the right time. It can be overwhelming you know, you there’s obviously I’m sure a lot of folks minds are going to that, you know, moment of overload of too many options. So, it We do try and help out with that. But that’s sort of the the core value proposition of what Udemy does.

Steve: And so as a researcher in a marketplace company, I’m going to guess that there’s research focused on when you describe two players in the marketplace, right, the instructors and the students credit research is, would you describe research as kind of focusing on, you know, ultimately, one or the other?

Laith: Yeah, and that’s kind of the fun part is, you know, I think sometimes if we’re thinking about sort of the very, you know, naive version of user research, it’s, you know, we’re trying to help out the one user. And, you know, for us, it gets more complicated, because you have to think about that balance. And so, you know, you can make something a lot easier for the learners, but that could put a huge burden on the instructor. And so you have to be able to triangulate and sort of balance those. Similarly, we also have a b2b offering. So in that world, there’s also an administrator role or sort of a buyer role that can enter the fray as well. So we have to then sometimes think about the the balance and trying to find Win Win wins for any kind of change that we do. So it’s definitely a bigger challenge at times, but it’s a fun one.

Steve: So in that third example, the b2b so someone, this is a scenario where that Python course might be used by like a learning development person inside an organization to bring that in for training set the scenario.

Laith: Exactly. And so we have soft skills, we have technical skills, there’s there’s a whole range and that is a subscription model. So they would get access to our courses through that subscription model and make it available to their employees.

Steve: So what kinds of research you know, what kinds of research are you and your colleagues doing?

Laith: Um, one thing that I really love about Udemy is over two thirds of our learners are outside of the United States. So we really have this global presence and as a researcher, you know, just as a nerdy researcher that that presents a lot of fun stuff to look at. But it also means that we tend to rely a lot on remote research just to be able to get that that that global perspective, that global reach for the studies that we do, so remote as a lot, but it’s interesting, you bring this up, because I’ve been talking with my team a lot about how, you know, I think sometimes both individual researchers and organizations can kind of fall into a comfort zone with where, how and when they do research. And so we’ve been talking a lot about ways to kick ourselves out of our comfort zone and make sure that when we’re designing research, it’s, you know, truly the best way to answer that particular, you know, immediate research question that stares us in the face, rather than something that has a lower logistical overhead or just is something that we’re more confident and we’re comfortable doing.

Steve: So how do you create the conditions where, when you’re talking about risk, you didn’t use that word, but outside your comfort zone? How do you create the conditions where it’s okay to go outside the comfort zone for you, your team, the stakeholders that you work with?

Laith: Absolutely. And I think, you know, you hit the nail on the head with risk, you know, it’s not that folks oftentimes don’t want to do these things. It’s that, you know, they, they are used to doing things well, and they want to maintain that that same level that they uphold themselves to, I think there’s a few things that help there. One is, as I’ve helped to build upon my predecessor, who previously lead the research team and build the team further, really tried to have complimentary and overlapping skill sets within the team. So hopefully, it’s not something that no one on the team has done. It might be something that one person hasn’t done or a few people haven’t done. So this way we can share and teach each other. I’m a big fan of when they talk about in the medical practice, they talk about See one do one teach one. And that’s kind of how you know and and, you know, a new procedure in medicine. So I really tried to embrace that and say, Hey, folks, you know, how can we teach each other because you might have seen someone do something, and then you got to do it yourself. And now you get to teach someone, that’s when you’ve really kind of got that level of mastery. So when you’re constructing teams, I think it’s interesting to think about where those opportunities are, and that you’re creating the environment for that. I think we’re also lucky to work in a place where there’s just a lot of trust in the research team, folks have just done a good job of delivering high quality research, and that that gets you a tremendous amount of goodwill. And so I think that’s allowed us to have a little higher risk tolerance, because people know that, you know, we’re trying our best and we’re trying to deliver good research and if something, you know, by chance, didn’t go super great. You know, it was done with the sort of the best faith and intent and just generally the leadership, the product leadership and other folks are trying to you know, Encourage that general behavior in terms of the features that we’re launching or the the ways that we’re interacting with our marketing messages or things like that, too. So it helps to be at an organization that generally understands, you know, that risk is something there, and that’s where you get a lot of growth.

Steve: And buried within that as a, there’s a specific nugget that you just said, I want to kind of bring back up. You know, you implicitly answered How do you build trust? And you said, like, deliver great work and paraphrasing you but is that do I have that correct?

Laith: I think that’s a big part of it, for sure.

Steve: But then you kind of talked about goodwill, and maybe that’s maybe good work in genders goodwill, which then gives you some permission to not and no, no, no, I talked about failing but you’re talking about Yeah, getting outside the comfort zone and having trust to do that.

Laith: Yeah, I think that’s part of it. You know, hopefully there’s a little bit of a virtuous cycle, you know, that happens there. And I’d say the other part of it is, sometimes research can be a black box. And so I think that as people understand the component parts that go into research, it helps to build that empathy. And I know you know, empathy is a little bit of a loaded term and has been critiqued in some ways. And, and, but I think that just sort of at that base level of having a better understanding of what other folks are doing. And I think that you know, even just having folks observing research sessions or participating or the research team, facilitating non researchers, doing research helps to build up that kind of understanding of the complexity and the nuance that goes into, you know, really high quality research. So I think that’s part of it as well.

Steve: When you when you try to move research from a blackbox to something that has nuance. You’re asking more of your audience of your stakeholders, I want to I want you to understand that this is there’s more to this than what you knew. And maybe this is the benefit of working in a company that has learning as kind of part of its mission. But you’re asking more of your collaborators, if you want them to understand what you do is more than a black box. That is, that’s more work. That’s more brain cycles that you’re asking of them. I feel like this comes up in the, in the discussion of, you know, research is a thing that everyone’s doing. You just talked about, you know, facilitating teams guiding them and doing their own research. We we’ve, as a practice, we have invited a lot of people In. And yet, we still want to highlight the difference between what a non researcher can do and what we can do. But that means showing somebody that, hey, it’s not a black box. There’s a lot of detail here. I’m probably asking like eight questions at once.

Laith: Do you know where I’m going? I think I do. And I think this is something you know, every research or guy Ben has struggled with this in one form or another. So I think I get the gist of it. Um, I think for me, there’s a few things. The first is, it’s been a bit of an evolution for me, I think coming in as an someone with a bit of an academic background. And originally being more on the consultancy side. We sort of positioned herself as like separation of church and state of like design and research. And so we really kind of saw ourselves as on the user side. Maybe, you know, the metaphor I might use is something like we were the scientists or we were the sort of the testers in a way. And I think as I’ve moved In house and just fundamentally tried to had different relationships, I’ve had to reformulate how I think about my relationship with product teams. And I’m, you know, I’m someone who can’t code. I’m not a designer, I love all those kind of things, but I just don’t have that skill set. And I think I never, for a long time, I didn’t see myself as a builder. And one of the big evolutions in my career has has been saying, Hey, I’m part of a team that are building things. And so it’s not me sort of on the outside, you know, as a referee, but I’m part of this team that’s building something, and there has to be that kind of give and take. And I think that repositioning of myself has also helped to allow folks to reposition themselves in relation to me, and and has opened up some of those kinds of things. But you’re absolutely right. You know, researchers can’t just sort of demand that everyone drop all the things that they’re doing and have all the bandwidth in the role, world for, you know, extensive kind of things, because they are very good time intensive, especially the more elaborate research projects. So there’s a few things there, you know, again, echoing that idea of, you know, not just being empathetic with our users, but being empathetic with our colleagues and our co workers, I think is super important. I think that the principles of Human Centered Design in what we asked them to do is super important as well. So if you’re soliciting feedback on a discussion guide, um, have you made that a really easy to read? guide? Have you set up what kind of feedback you’re asking for them in a really clear and manageable way? Have you have you tried to maybe anticipate their workload? You know, I think all the things that we might think about doing in terms of building products we need to be able to do for our colleagues in different ways. But I think it also just shows that you know, for research to be really supercharged and have the impact that it can you know, it’s it’s a it’s a team sport, and you know, it’s So when we think about sort of those models of UX maturity or research maturity, you know, part of it is that the organization is willing to invest in it. When I, when I think about democratizing research, it’s it’s not just about letting a designer go, and sort of being the gatekeeper and letting them go do some usability sessions, but it’s about, you know, having them more involved in the process and feeling that we’re all designers. We’re all researchers, we’re all product managers. We’re all engineers. Now. We have specialized positions within that, but we’re building together and and i think that helps, but it’s, it’s still tough, you know, I don’t mean to act like I’ve nailed it, and I have all the answers, but that that mindset shift has at least helped a little bit.

Steve: Right, because you started off saying that, you know, you never thought of yourself as a builder. And but I hear you evolving that identity for yourself. And I heard two things. So let me actually want to check in with you. One is, is the collective view right? So, we are building something and I am part of I am part of the we, and if we are building then I, in this case, you take on some of that mantle of the identity of a builder because you’re part of that team after that, and I also heard that, you know, in, in living up to the principles of empathy for our colleagues, you are designing experiences for them through the experience through the artifacts that you’re creating, and the ways that you’re, you know, trying to make it effective for them to work with you. And that’s something that’s being built, you are a builder in the production in facilitation and production and artifacts, and all those sort of bits and pieces that make research happen.

Laith: So so like a lot of tech companies, you know, we do a lot of playing off the sort of name of the company so you know, people that work at Udemy or Udemy, mates and things like that. So, one thing that my team has worked on over the last few quarters is To make a an internal self serve research process that we call a u du u, so it’s, you know, a play on the U and the name of Udemy. But it’s sort of the idea that you can do it yourself. And we approached it in very much the same way that a product team might approach it, you know, in terms of, we basically had one of the researchers act as a PM, and we put together materials and we decided, you know, what was in scope and what was out of scope. We designed out, we sketched out the user flow, and what it might be, we tried to figure out how the different pieces of the experience would fit together, and try to, you know, create a portal through which folks could do this with as little friction as possible. And we very much approached it, how a design team or you know, a product team might make a product because we wanted to, you know, sort of practice what we preach and and embrace those principles, and we’re iterating on it, and we’re doing user testing on it and you know, we’re trying to do all those things, so that it can evolve and be the best way that folks can do research. themselves?

Steve: How did it feel for the team to do that to take on these, you know, put on some different hats and kind of act as if you you were these other types of roles?

Laith: Well, I will say, just because you brought up the issue of feeling, which I think is important, is I think that, you know, for different folks on the team, this was more and less out of their comfort zone. You know, I, I know that there can sometimes be that that gut reaction when we think about sort of opening the gates and letting non researchers, you know, run wild and so there was some good fun discussions about, you know, how we saw ourselves and how we talked about research and some folks really kind of had to grin and say, Hey, I’m going to trust the process and see how this goes. So that was a fun kind of thing to go through. Um, but definitely, it was great to see folks taking on these other roles, and you know, maybe putting some of their design chops to use that they maybe don’t think of themselves as a designer, but they’ve picked up some of those skills along different ways or put some of their chops together and and kind of see you know, put into practice what they’d seen great pm doing. So it was a really fun process. And I think it was a fun bonding process, we kind of did it as part of an off site. We have one awesome researcher, but she’s located in Dublin, Ireland, so we don’t always get to be as much spend as much time with her. So we did this, the bulk of this work sort of did a sprint, when she was visiting, you know, the main office in San Francisco. So it was also just a really fun way to get to know each other and our working styles because we’re in an embedded model. And so folks are working with teams. And so it was it was just really fun to be able to work on a project together and kind of learn from each other about how we approach product design and product development. So just as a team bonding experience, I thought we got tremendous value out of it on top of sort of the the output in addition.

Steve: You mentioned empathy for your colleagues, you know, trying to do the work of a world They don’t normally hold, I think would might illustrate that as well for everybody. Absolutely. What are the so just maybe shifting gears a little bit, you know, thinking about the, the populations, the user groups and sort of this, this space of learning of teaching and learning, you know, what’s are their unique ways that you have to approach you know, beyond the the global aspect, whether it’s methods or just, you know, the, the mindset you have to bring to these particular areas that you want to understand. Was there anything specific there?

Laith: Yeah, um, I think that there’s probably a similar thing that you might find with financial services, some forms of that or the fitness world where folks have a lot of aspiration to do a lot of learning But it takes a pretty big investment to, you know, learn a new skill, especially if it’s in a more self directed environment. So, a lot of times, it’s that classic thing of, you know, someone might if you talk to someone right after a purchase, they purchase a course, they’re imagining the sort of future where they’re, you know, diligently doing, you know, three hours a day in 10 hours on the weekend, and, you know, things like that. And, you know, unsurprisingly, you know, a lot of folks struggle with that, whether it’s their current work responsibilities or home responsibilities, or, you know, they hit a stumbling block, and they’re not sure how to how to get over it. So it can be a challenge to parse between sort of how people imagine themselves behaving and then reconciling that with maybe what we see in the data analytics or or sort of what they do. You know, there’s some interesting behavioral science things about the more you can empathize with your future. The more effective you can do at helping that future state. So, you know, there was a company in Mexico and they would digitize an aged version of your face when you were presented with, like how much you wanted to contribute to your retirement fund. And sort of being able to empathize with yourself in the future made people more likely to contribute to themselves basically their own retirement fund because they could sort of imagine themselves as not being distinct in the future, but being the same person. So there’s a lot of challenges there around motivation and intent versus behavior. That’s a little bit of a rambling answer, but it’s it’s a complicated research area to dive into.

Steve: What about the you know, how do you approach the the instructor, community or segments?

Laith: Yeah, um, You know, definitely, marketplace dynamics can be a challenge. You know, if you have an entrenched instructor who’s very popular and has, you know, hundreds of thousands of enrollees, enrollments, it can be tough to figure out well, you know, what if a new instructor comes on, so helping folks to kind of feel that they have a fair shot is super important. And then also, there’s a few elements, I think, that are really important about helping instructors get the right kind of feedback in both Udemy and other places I’ve worked, you know, if you ask folks, if they want more analytics, and you start listing options, they’ll just say like, yes, yes, yes, yes, yes. You know, people just feel that they want more data and they want more information. But at some point, it can be overwhelming, or if they don’t know what to do with it, sometimes folks can kind of feel guilty and you know, this numbers changing and I don’t know what I’m doing differently and things like that. So there’s a real challenge in helping to not just present in information on a dashboard. But the context, the understanding the nuance, and hopefully equip them with strategies or suggestions along the way. And they’re not just sort of twisting in the wind to sort of decipher the, the different data visualizations, you know, how they might do without that assistance.

Steve: Right. And there’s, there’s something maybe lurking underneath your stories about, you know, the gap between I mean, maybe it’s simplistic level, like what people say they want or how we can get people to agree that features are, are things that they want versus, you know, serving a serving a deeper need, which is harder to harder to extract and kind of have a deep understanding of, that’s absolutely the difference of what you’re trying to accomplish with research.

Laith: Absolutely. And, you know, that’s kind of the you know, in some ways, the Holy Grail A lot of the work that we do is is figuring out how to get at that, you know, underlying unmet need and things like that. And I’ve even seen, it’s been interesting recently, you know, a few teams working on similar product offerings or features, and then realizing, when we took a step back that, hey, these might actually be trying to solve some similar things, we might be able to combine these efforts. And I think that’s one thing that research was helpful in is sort of highlighting what the actual value people were getting out of some of these prototypes. And it helped to sort of triangulate you know, what sort of, you know, had the most juice, you know, that we thought could really deliver something unique.

Steve: You we talked about your team kind of indirectly Can you describe Kind of, yeah, what kind of what kind of skills and backgrounds you know, make up your team and, you know, in future what do you what do you look for when you When you meet prospective candidates for your for your team?

Laith: Yeah, it’s a great question. I think that we have a pretty diverse range of backgrounds, which is pretty cool. At Udemy the way we’re just organized, it’s market research and user research together. And, you know, this is something I think is really interesting in the industry is all the different permutations of org charts. And, you know, you know, where folks sit and things like that. I think sometimes it can get, you know, over indexed as something that’s important because I think sometimes it’s often just legacies of how companies grow. But it’s interesting and I think it is good to think about, so we have one market researcher, and then four user researchers and then myself and it’s actually been really fun because to to have that market research lens as part of the user research team. Because it really helps to highlight things about brand voice and brand positioning and Sort of that that more holistic experience of how someone might enter the product through different campaigns, and and sort of thinking about that broader journey. So it’s been fun to have that perspective, most of the places I’ve worked at before, you know, market research is in a different building. And it’s kind of a black box even internally. So it’s, it’s cool to have that that together. And that person tends to bring a little bit more of that the survey chops and and some of those more market research kind of things, not to say that user research doesn’t do surveys, but you know, sort of bread and butter and approaching some some different kinds of questions. And then on the user research team, you know, we’ve got a couple folks that have a, you know, master’s degree in HCI. And then other couple folks that, you know, came to the industry through through different routes, including someone who formerly used to be an engineer working on renewable energy in that kind of format. So, it’s cool to that we all bring in different things, and we’ve been at You know, some different organizations and sort of trying to figure out the DNA of those places and what we want to sort of copy and and what we hope to innovate on or what we see that you know, someone might be bringing in a different mousetrap. When I’m when I’m looking at candidates Udemy is very mission driven, even though it’s a company, a lot of folks really just believe in education and the power of education, and the global nature of what we do. So, you know, always want folks that are really bought in to, to the idea of what we’re doing. And then I think beyond that, it’s, you know, someone that can bring in something new and that can be new methods and new perspectives, new approaches, but um, you know, I don’t really see it as that there’s like a one single profile that is gonna is gonna mesh well. Collaboration, of course, is super important. And having that that flexibility To think about, you know, for each project what what sort of permutations or iterations you know is this something that’s going to be a little more quick and dirty is is something that’s going to be a little more rigorous. The last area that in hiring that I’ve sort of formulated in my mind is, I think about written communication, verbal communication and visual communication. And I really want to find folks who are really comfortable, at least one of those, like really good, and then hopefully, you know, decent at the other two, and, you know, willing to kind of work on them. But I think it’s important that you have one modality of communication that you know, you’re just really good at, because you know, so much of what we do is communication.

Steve: I want to loop back to you said all sorts of all sorts of interesting things. I want to loop back to one of them. Because you mentioned a few moments ago that that that people are embedded. But you also talked About the sources. Watch one. Sorry, see one. Do one teach one. Is that right? Yeah, yeah. Okay. So I’m just wondering about, you know, especially you got this market researcher with a, you know, a pretty unique skill set among your team. And you have all this diversity. So how the the researchers are are embedded they have product teams they’re specifically focused on. So how do you balance that focus and then harness the hardest that diversity and let people influence each other and teach each other and draw from draw from that team? You know, that you have the centralized part and you have the embedded part. I’m wondering what that looks like.

Laith: Yeah, and I you know, I think this is a challenge for for for folks in some organizations to sort of be a functional leader, but not a project. Leader, where the folks on my team are just so much more expert in the details of the features and pods that they’re working on. So, you know, sometimes I feel like, you know, oftentimes I feel like, you know, they’re teaching me or they’re, you know, showing me how things are working. And so I have to think about other ways that I can contribute or add value, obviously, you know, that can be with being a sounding board for thinking about methods and approaches. But But to your point, there’s also about sort of how I can create that sense of a research team, even though most of our day to day is kind of often embedded with, you know, awesome designers and engineers and marketers and you know, folks doing other things. So we have a few mechanisms. And, you know, I hate to say that the answer to everything is a meeting, but some of it is meetings. So we have a team meeting. We also have a bi weekly session. That’s a we call it researchers talk shop. And so we have a spreadsheet where people just throw in topics and we For a lunch and we try and dive into different research topics. So that’s a great way for, you know, someone’s been doing a lot of workshops recently, and they can kind of share what went well and what didn’t go well. And we do some of that in our research team as well. We have a office hours as well, so people can just stop in and talk with us, but we, you know, sort of hang out in the conference room. And that’s just an opportunity for us to kind of catch up a little as well, you know, because sometimes, you know, 10 people show up to office hours and sometimes it’s just one and just kind of hearing each other helping other folks and office hours you kind of get to hear a little about their approaches and things like that. Another mechanism that we have is we do try and pick some some bigger, juicy or questions that we are sort of more researcher initiated, and we do those collaboratively so that they you Edu initiative is a is an example of that. And we’ve tried to do a few other collaborative researcher initiated projects. Another thing is we have a little podcast club, so We’ll listen to podcast episodes and then just set up a quick little half hour to, to talk through it in different ways. And it’s just fun, you know, a chance for researchers to to talk about things through that lens and try and grow and get those other perspectives. For me, the nice thing about podcasts is it feels really low friction to just find like a 20 or 30 minute podcast. And, you know, folks can do that when they’re commuting to or from work and you can still have a really lively discussion, I found historically, it can be a lot harder to give folks PDFs or articles, you know, it can be harder to sort of find the time. So those are some of the mechanisms that we have to try and develop that that sense of community and exchange.

Steve: That’s there’s a lot of them. And so I know you’ve got one person in Dublin, for the people that are in San Francisco. Are you sitting in a co located or whatever the better way to say that would be

Laith: we’re all sitting on the same floor but you know, it’s a decent sized floor. So, you know, there’s folks that aren’t in visual visual range. So we’re kind of sitting near the product teams that we work with, we’re sort of the seating chart is by, you know, product teams rather than by functional teams. But you know, you know, you run into people constantly just in the halls and at lunch and running between meetings. So, definitely, I see everyone, you know, I’d say, at least a couple times every day, you know, even if it’s just passing.

Steve: And so for researchers that are embedded in these product teams, how, how is sort of the the, the plan for what they’re going to work on determined?

Laith: Yeah, it’s a great question. And shortly before I joined the company, or a little before I joined the company, the research team had switched from that more centralized model to an embedded model. And so some of this has been an iterative process. The other challenge is, you know, there’s just not infinite reading resources. So is the ratio of researchers to designers and PMS. You know, there’s usually one researcher juggling at least a couple PMS if not two, or three or even more. So it makes prioritization really a challenge. And I think before we’ve been a little reactive, and it’s hard to make sure that you’re working on the right things, if there’s just sort of like a queue coming in, and you’re trying to keep your head above water. So one of the things we’ve worked on and iterated on it quite a bit over every quarter is just to do a quarterly planning process, where folks go out as as teams are putting together their, you know, quarterly okrs. Researchers are also evaluating what the research asks will be over the next quarter. The researchers then take that all together, you know, from across the teams, you know, think about they do a little bit of a T shirt sizing, you know, what’s a small, medium and large project, think about their bandwidth, sort of think about timing And come up with sort of a proposal. You know, it’s very aspirational. But it’s still a really good, you know, exercise to go through. Now, what I think makes it has made it really effective is we then do a readout of these plans to leadership. So I’m really lucky to have a very actively involved CEO and CTO and head of product. And so it’s those folks and a few others in a room, and each researcher goes through what they’re intending to do over that quarter. Now, it’s really great for research just in terms of getting visibility, and, you know, getting some recognition because we also do a section where we sort of share what we thought were some of the winds, and do a little bit of a retro on the last quarter. So get some of the researchers a chance to highlight some of the great work that they’re doing to leadership. And then also just to get that feedback where they might talk about what they’re prioritizing, and then to get that color or get that really strategic perspective, from the CEO talking about, you know why he’s excited to hear us diving into this area or that And then the last part that I think is kind of powerful as well is then if there’s kind of push back a few weeks later, as a researcher is explaining, like, Oh, I’d really love to get to that, but I’m doing this, you know, they can kind of say, Well, you know, this is what our, you know, leadership team, you know, thought was really important. And it gives a little more gravitas to that sort of prioritization process. And, of course, we build in, you know, feedback cycles, you know, throughout as we’re developing this as well. So, it really hope hopefully gets folks on the same page. Um, you know, there can be a little overhead to this. So we are trying to make this as spelt and, and sort of low overhead. So that’s part of what we’re iterating on. And this might not scale if we were a team of 15 because there just wouldn’t be time to, you know, have the CEO sit there through 15 people’s presentations. So, at least for now, it’s been really valuable and, you know, we’re gonna have to think about as the team grows, how to modify it.

Steve: A lot there. There’s a lot. So the researchers themselves are, that are they’re the ones putting together their individual roadmaps.

Laith: Yeah. You know, going back to that question about, you know, building teams, you know, I want folks who who are, you know, I want to find awesome folks and empower them to do the work. I’m not, you know, overseeing or micromanaging or sort of telling them to do this or that, you know, they tend to bring me stuff and we chat about it. And I’m a sounding board, and we strategize and I give my perspective. But, you know, they’re the ones that are mediating the relationship with their, with their stakeholders. And so, you know, they really have to believe in whatever choices are being made about what they’re working on. And I also want them to feel that they have agency, you know, that they’re not just sort of, uh, you know, designers sometimes talk about sort of being like pixel pixel pushers. I don’t know what the what the right alliteration would be for researchers, you know, we could work on that, but I don’t want researchers to feel like they’re just sort of being dictated to about do this usability test or do this You know, the generative, you know, exploration or getting a survey, again, making them part of the process, and and they’re contributing to the OKR process, as well. So it’s really should be a symbiotic relationship. We’re not quite there, but I think we’re really moving in the right direction. And I’m excited about that, that that back and forth and the interplay

Steve: I think it’s a compelling vision that you’re, you’re illustrating, and I want to go slightly meta here. Sure. Maybe to meta I don’t know, you know, you. I mean, you said something that I think is quite common, right? The, the, the research resources are limited, or they’re, you know, to put another way, demand exceeds supply. And I think that’s common and I wonder, you know, outside of Udemy, like, if that’s sort of a truth in at least for tech, you know, in the kinds of organizations That means there’s lots of kinds of research, but in sort of tech overall, I think that’s a common, you know, you have different team sizes, you have different organization, you know, approaches to research, but in general, I hear this a lot. What do we what do we think of that like that? That’s a that’s sort of a universal truth that there’s a more there’s more needs than there are resources. You know, why? So what’s what is? Do you have any reflection on that?

Laith: I mean, there’s definitely I think we should be self critical enough to say we’re biased as researchers. So, you know, I see research need where perhaps other people don’t and I think that’s, that’s totally fair. And and I think it’s also the the tricky part is, are you optimizing for sort of peak demand load or average demand load, and that can be a hard thing to disentangle. And then Lastly, I think it’s what, where and how can research impact things? So, you know, I think, you know, are we already sort of cleaning up things and grinding out a lot of usability studies? Are we doing really conceptual, you know, blue sky kind of research? What kind of impact do those do different things have? are we are we, you know, what sets of stakeholders are we interfacing with? Um, you know, I think it’s a very different situation, if, you know, you’re getting into some really interesting things that overlap with back end engineering, and you’re doing interesting things with PMM. And, you know, local operations, you know, what I’m saying instead of just sort of the more core conventional stakeholders that you might have, I think that that exploring those kinds of things, or maybe things that I’m seeing that I wish I could do if I had more bandwidth, but the organism might not necessarily see that yet or see the value. And so it’s a matter of sort of picking the opportunities to try and demonstrate how big their menu of research is, and all the places that we can impact. So it is a tricky one. And you know, it’s probably better to have a few too many researchers, sorry, a few too few researchers then to go the other way.

Steve: And, you know, I think about I certainly when I started the field, there just there wasn’t a lot of demand. I mean, there wasn’t a lot, there wasn’t as much awareness there wasn’t as much, you know, process that includes these these tools and kind of incorporates what we bring. And it’s been interesting to see, you know, over time, how the demand is high. The, you know, the the The amount that that our work is integrated with, you know, other work that’s happening in the company is as grown and I think you’re just you’re articulating, you know, a future state perhaps where there’s even more, you know, more aspects of the work that a company is doing, are places where we can bring value. That’s like a, I don’t know what the shape of the curve is over time, but you can sort of see it evolving from, you know, who are you guys to, we have a lot of work for you to, you know, in the future, there’s way more than we can do then you then people are maybe ready for or that that, that we have demonstrated, you know, as a practice, you know, as, as broadly.

Laith: Absolutely. I mean, it as I’ve grown in this field, you know, I’ve really found that, you know, Human Centered Design has kind of permeated my life. You know, let’s say I’m trying to I set up a time to hang out with some friends. And you know, it’s sometimes hard to coordinate everything with like three or four friends. You know, when I’m typing up that email, you know, I think about, oh, a little, little visual hierarchy in, you know, as I typed this out to make people understand the options and what I’m asking for and to reduce the language and make it easier to scan, you know, that goes into me putting together that email. So I think we kind of have to embrace that, that philosophy of Human Centered Design is really powerful. And it’s been fun to your point of seeing some of the bigger companies where the internal, you know, human resources team, or the you know, real estate team might have a user researcher because they’re thinking about that that’s a product you know, how people access the benefits internally or what workspaces are like at a company or all these different kinds of applications. And that whether we call it user research or service design or you know, any of these kind of but that that that fundamental approach Can can benefit organizations in so much of what they’re doing. It is fun to think about that, that that future state and all the places that we can, you know, deliver value.

Steve: You know, and use that up top, you know, you came from academia, you had sort of just a variety of things that you’ve done along the way, in research and kind of before research, and I’m wondering if you kind of look at your history, but also, as you you know, have you as you grew into the field, you I think learned about the, the history of the field of user research. Do you see? I mean, are there seeds of what we’re talking about? If you look backwards in, you know, where research had been applied, how it kind of evolved or, you know, what you see when you look backwards?

Laith: Yeah, this this is a topic that I’m just continually fascinated by, and I would love your perspective on this, because I’m sure you know, a lot of this very well. And it’s been interesting. You know, I think sometimes When I’m doing my teaching, I almost get the sense my students kind of feel like this is how it is. And this is how it’s always been. And you know, sometimes I kind of have to remind them a little and be like, this is actually a very rapidly evolving and changing field. And you all are going to have a huge say, in sort of what this, you know, ultimately looks like or sort of what it what it might look like in a more mature state. And it really has undergone these different shifts in thinking about all the different strands that go into it, at a very, very simplified level. And not to say that these phases are not overlapping or exclusive, but it feels like, you know, previously there had sort of been the innovation labs, the Xerox parks, and that that had been sort of where a lot of this kind of work was being done. Obviously, there’s a older history when you’re thinking about World War One and two and human factors, and that’s another sort of strand in this. And then it feels like some of this work then became a little more focused around agencies. You know, you’ve got your radios and things like that. And we started to see that shift to in house research happening and calling it UX and things like that. And I think around then is also when we saw this shift to much more consumer oriented products. Before a lot of it, you know, had been very enterprise e or, you know, business oriented, where maybe someone, all their exposure to human computer interaction would be the one computer at work or something like that. And so we over this time, we’ve evolved to this place where technology is being much more omnipresent. And you’re not just dealing with a technology one manufacturer, but you’re starting to have to think about that the design changes on your phone are affecting the design changes on your computer, and those might be made by different software companies and hardware companies and all these kinds of things. Now, I think originally, a lot of those in house teams were embedded, or sorry, we’re centralized, like I sort of talked about a little before and then we’ve seen Seeing the shift to the more embedded teams and the real just tremendous growth that’s happened over I don’t know, the last five to eight years, and weaving together all these different strands from psychology and design and Information Science and anthropology and you know, more than I could sort of a name. And we’ve also seen the pipeline really emerged at these really great masters programs, you know, a lot of programs in library science, or have rebranded themselves as an Information Science program and are contributing to this. And of course, you know, human computer interaction programs and things like that. So it’s interesting to think about sort of what comes next. I started to see some more innovation labs kind of things popping up. So it’s funny how that might be coming back into vogue. And then the sort of rage of democratizing research and, and sort of figuring out how to scale the impact of research teams beyond just embedding more and more researchers interesting. As well as you know, sort of the the global nature of this, you know, feels like there’s a bit of a different UX culture in some parts of Europe, and you’re seeing emerging UX culture in in other parts of the world too. So that’s kind of my take that I’ve gathered through just a few articles and chatting with folks. But I’d actually be curious if that aligns a sort of a very rough arc with sort of your take of the field.

Steve: So this is the part of the interview. This is the danger of interviewing an interviewer. Yeah, I mean, I would love to sort of, maybe someone has done this, but I’d love there’s a visualization, it’s kind of it’s kind of a timeline you know, around with some of these major things happening and you know, as you talk about it, you know, the the emergence of design and consumer products as has elevated to design, right? The, the way the jobs for designers changed once there was an iPhone, I mean, going back the jobs are designers change once there was like a Macintosh computer, but and that, you know, we’re, I have felt for a lot of my career that and not necessarily a bad way but that that research is kind of following in the wake of design. And so when I joined the field it was it was following in the wake of industrial design, because that’s where all the innovation was happening. And, you know, they kind of they kind of weren’t, they were the innovation labs. But then the web comes along, and there’s just so much growth and you know, things that you talked about. And so, you know, user experience comes along as a phrase and suddenly we’re user experience researchers and researchers because we’re, we’re working on that in those industries. We’re working on digital stuff more exclusively. You know, and I think we, I mean, in your narrative you are, you’re any good history has to do it includes a lot of different as you call them strands. And sometimes I feel like it makes, it’s necessary, but it’s hard to see the research strands like you don’t want to decontextualize it. But I also I do want to decontextualize like I want to how do we talk about research without talking about anything else? Just as a you know, as a, as a as a thought exercise, like just to look at how research has changed, even though these other things have changed, and that’s why research has changed, but I’m certainly interested in the like, I think you’re going to your point about, you know, a lot of the work moving in house and as someone that’s still a consultant, I can tell you that’s that’s definitely true. It really has changed, my professional life changed dramatically. As that trend took hold, so I feel that one very specifically. And that’s a very research specific one, even though I think you see lots of creative professions that were historically done, those consultancies are done as agencies got moved in house. I feel like there’s that Apple caused some sea change in the ad industry when they Yeah, brought something in house. And I don’t know if that was sort of the the change point that led to what you’re describing with, you know, research overall.

Laith: I think that if I were to focus a little more on the research strand, because I think it’s a fair call out, I would say there’s two things and one is thinking about the position that the researcher has these of either product or what they’re working on. And it’s funny because I think Yeah, back in the day, and I’ll contradict myself a little about not about focusing on research, but you know, if you were a designer, you’d be like, Oh, I want to go work for fjord or tea handed lax or You know, these kind of big design agencies and they all got either acquired or Accu hired not all of them, but a lot of the really prestigious ones. You know, and and you saw that happening a little with research too, with with bolt Peters, you know, and things like that getting brought in house by the big companies Adaptive Path and things like that. So it has been interesting to see that, that that that transition, but I think we’ve inherited certain ways of approaching things, or at least I have, I know, coming from that consulting side, it was kind of a formal relationship that I had with the product and with the product teams. And you know, it’s sort of like having the big, bloated deck, you know, that Edward tufte A would, you know, you know, shake his finger at me for putting together and I’ve had to think about ways of thinking about like, Hey, you know, I don’t really need to go through a lot of that front matter that I might have had in this very formal deck for clients, with people that I have a much more casual relationship with. Similarly at Udemy when we first transferred from that embedded sort of Central To embedded forum, they’ve been sending out surveys, sort of satisfaction surveys to see like, how did the research project goes. And ultimately, you know, there are people that you’re in meetings with all day and you’re sitting next to them and all these kind of things. It feels kind of odd and distancing. To send that satisfaction survey. Now, we still need to have mechanisms for feedback and to figure out ways to do better, but we kind of needed to adapt the ways in which we work to reflect these different, you know, relationships. And I guess the other way I’d flip it around is to sort of to take a research, design thinking approach, which is, you know, if we were to redesign, design research today, and could somehow throw away the baggage or sort of the history of all these things, how would we design it differently? How would we redesign the way we work and our relationship with products, you know, allowing that there will probably be a spectrum of those things depending on the company and those kind of things. You know, it’d be, you know, as a further thought experiment, it would be kind of fun to think about, what are the things that we’ve carried forward? Because of these histories coming out of industrial design and out of human factors and out of these sort of evolutions? And what are the things that we might do differently if that if we’d been on a different path, or we’re just starting a new today?

Steve: And one of the, I think, one of the legacies and you know, whether it’s a strength or a weakness or both, is the the, the diversity of paths that have led people to research? You know, it’s still true today. And it was, yeah, like, I think, you know, if you go back in the olden times, there were few sort of credible paths, they had to sort of invent a path and now there are, I think, many paths that are somewhat credible, but it’s So lots of people have to find their way in from something else. Right. And I think there’s there’s, we’re the field welcomes more people from more places, it’s still a struggle to get in. And then I guess, just there, it was a different struggle in the past, but the that lack of definition of like, what background does a researcher have? Or what backgrounds plural does a research team need to have? I think that’s very exciting. But it also, it also seems like a huge amount of legacy code that we spent a lot of energy and emotion on that if we were looking at things to sort of toss away and reinvent from scratch. That would be that would be one that I would I would lobby for.

Laith: Yeah, it’s tough. And, you know, I think a consequence of that is the research culture at different organizations can be so different, you know, if you have one place A lot of the folks are master’s degrees in human computer interaction and not saying that that’s a bad thing, but it’s just a set of things. And you have another organization that, you know, more folks coming out of sort of design anthropology or applied anthropology, you know, they’re gonna have very different ways of framing problems and approaching things. And you might have another organization where it’s a lot of designers and engineers, who you know, learned and got into research, you know, through work experience, those would all have very different flavors. And yeah, to your point, it’s kind of like a great thing and it’s an exciting thing and it’s an enriching thing. But it can lead to folks talking past each other, or, you know, getting in over their heads in certain situations. You know, I do sometimes my main concern can be about the ethical conduct conducting of research, and that, you know, I really hope that everyone who gets to do this awesome line of work has the the proper you know, ethical, you know, You know, training to sort of keep their eye out for getting into situations that are not good for them or not good for the industry and are not good for the research subjects.

Steve: And where does that? I mean, I think this this speaks to the multi discipline. I guess that’s what you’re saying. So I’m just reiterating your point for you. Sorry. But like, yeah, where does that come from? How do research teams work, regardless of what their you know, what their composition is? How does? How do we mature the practice of the ethical aspect of the work?

Laith: It’s tough. I mean, I think that there there needs to be a degree of self policing, you know, whether it’s on slack communities or email listservs You know, I think it’s it’s okay to discuss things that have popped up and say, Hey, you know, I’m seeing the survey coming out from this or invitation to participate in research like this. And you know, I don’t know how I feel about About that, definitely, I’d say larger teams, which I think is where a lot of more junior folks are able to get their start. I would love to see them have a sort of a formal cadence of doing some some ethics training. I think it’s super important. And But yeah, I think there there is a bit of a vacuum there. You know, there’s some good articles and it does get covered in you know, the, the seminal, you know, works like the Baxter courage cane understanding user needs. You know, there’s great discussions in there about, you know, ethics and things like that. But um, I definitely think as an industry, we could be doing more.

Steve: And this is goes back to the strands thing, like the conversation about ethics in user experience, progresses a certain way and it’s easy for us as researchers to get sort of swept up in that but the issues are different. I think there are specific things as you’re kind of getting at.

Laith: Yeah, I think that the the things you’d watch out for if you’re doing academic psych experiments in grad school would be very different from if you’re doing anthropological field research, you know, let’s say with a marginalized community, but yet in industry, because of the wonderful interdisciplinary Enos, you know, you might have those folks sort of doing the other kind of research and not being fully abreast of the, you know, you know, if they have very advanced ethical training and research ethics training in their home discipline, you know, as they move and sort of get to experience the other strands, they might not be totally, you know, up to speed on on those other kinds of areas. So, even as folks that think they have really great you know, you know, Human Centered research ethics training, as we move into this UX research world, there needs to be something sort of specific for us. I will say that that you know, the UX pa has guidelines market research organizations have ethical guidelines. And I think those are a really fantastic starting point for the work that we do, because they’re, they’re focused on sort of this more applied or industry setting. But, um, you know, there’s a lot more that I think we can do collectively.

Steve: What do you see in your students in terms of their awareness or, you know, hunger for this kind of guidance?

Laith: Um, you know, I, I don’t know how representative Berkeley is, you know, I think it tends to approach sort of attract a little bit more of the mission driven or idealistic kind of folks. So, I know that whenever I’ve mentioned, you know, doing work with different kinds of communities or crossing socio economic status, you know, sort of working with folks in a broader spectrum, you know, not necessarily just, you know, Bay Area folks that have an iPhone, you know, 11 You know, I think that they really get excited to think about Those kind of things, and they really embrace the responsibility and recognize the privilege that they have. I have to say, I’m just so impressed and blown away by how sophisticated the students are, you know, I feel that they that they’re really attuned to some of these kind of things, and you just have to open the door a tiny bit, and they tend to really, you know, jump at the chance to sort of think through some of these really complex issues. And think about the ramifications of, you know, sound design of people that, you know, have different accents, or AR VR for accessibility and things like that, you know, they’re they’re connecting the dots really quickly. I just think it’s really important to give them sort of the frameworks through which to process these kind of things. So that’s kind of what I see to be my challenge.

Steve: Your challenge is to give them those frameworks.

Laith: Yeah, exactly. So they think it kind of helps. To see like the bigger picture a little easier.

Steve: Yeah. Well, this is super interesting. And we could obviously dig into all of this two times more than we already did. I guess, just to throw it to you like, what did I ask you about? Or do you want to talk about that? We didn’t get to?

Laith: I don’t know. Uh, you know, I’m a nerd. I can talk about anything for forever if you can’t tell from the conversation. But, you know, I appreciate you give me the chance to talk about some of the historical stuff. And just quite frankly, like outside of the interview would love it if you had any more perspective on that to share just because I think it’s interesting to think about that legacy. But I don’t really have any other you know, access to grind or anything. If you feel you you’ve got enough.

Steve: So I’ll throw one more question at you and then I’ll try for a perspective on you know, on the strands conversation I guess so so. So if this was, if our conversation on dollars to doughnuts was going to end up in podcast club, what would you want people to get out of it?

Laith: I think I’m really into the idea of sort of questioning

what we do. And, and, and again, that can be the methods and that conversation about getting out of our comfort zone. It can be thinking about our relationships with stakeholders and trying to reimagine and iterate on those. And it can be thinking about the historical trajectory of the field and the legacies that that has imbued us with. And I think that just sort of being ready to iterate and question the assumptions that a lot of those things are built on, is the thing that I’d really want folks to come away with.

Steve: Excellent. Can I said I would, I would try to offer something else. I’m the sort of looking forward looking back on the field piece. You know, and so, my, my perspective is both informed and I guess hampered by just time. Like I have been doing this for, you know, more than two decades, which is weird what to say when it comes out of my mouth, but it’s, it’s true. And I have just been feeling for the last little while that, you know, we’ve been on this trajectory, that sort of this gentle upward slide in terms of, I don’t know, skill building, like figuring stuff out building best practices, like a lot of things that we were, you know, making up as we went along, have been formalized, have been shared have been documented. There are, you know, for a talk I was doing, I just tried to gather up as many books about research that are out there and there is a lot like There’s so many of them. And that’s when it’s a good thing we’re, we’re talking about this expertise is being documented and shared. And I’ve just been feeling in the past year or so. And, you know, again, this is a stone, it’s only my perspective, it’s not a declaration of what’s real or true, but I’ve been feeling like that we’re like any sort of, I mean, maybe you have a better take on this as someone who helps people build skills or do learning, right, you, you advance at a certain rate, and then you kind of reach a plateau, and then you and you sort of you cruise at that plateau for a while, and then you hit the next, the next rung or the next, you know, movement upwards. And I feel like that in you know, in, in the profession that I’m part of, as I see it, I feel like that’s what’s about to happen or is happening. It’s happening in little pockets, and we’re just starting to to be bring it into the mainstream. And I mean, I am I mean as as as, as the day we have this conversation or a number of days away from a conference I’ve been involved in called advancing research. That’s part of the Rosenfeld media, advancing research community, which anyone is welcome to join. And I’ll put a link in. And I think the name is really great. And you know, that we we talked about, how do we position this conference and what does advancing research mean? And, you know, what are all there’s lots of great stories and great work that gets shared. But we’ve also heard it a lot over and over and over again. And you know, yeah, if you’re new, maybe you haven’t heard some of this, but there’s so much stuff that we haven’t heard that we need to be talking about, and you brought up ethics, you know, earlier in this conversation as one that like yeah, it’s here. It’s there, but it’s we haven’t really made it part of our it’s not part of the the baseline, it’s not part of the kind of the core fabric of what we all practice, it’s like a thing we need to do. And I’m, I feel like, you know, as we move from the plateau to like the next incline, that we’re making a collective concerted effort to bring these under recognized pieces in. And what’s exciting for me is you don’t have to look very far to see, oh, that thing that we’re not really doing well, somebody is working on that somebody has a point of view, somebody has built something or built a process or built a framework and that it brings me at least an opportunity to to, you know, to iterate or to to move to that next, you know, that sort of 2.0 3.0, whatever. I don’t know what the numbers are, but I feel like we’re on the cusp of that. And again, that is such a, that is my perspective. I can’t say that anyone else would feel that way. But it’s something that I am drawing a lot of excitement from, because I feel like oh yeah, I know how to do a lot of this stuff. And there’s things that I didn’t even know that I didn’t know how to do, that I should be doing. I don’t even know where considerations I’m not talking about, like, you know, our ever shifting library of tools and infrastructure, but, you know, really the, the, like, read like an iteration of best practices to be more inclusive to be more empathetic of our colleagues. And I feel so I feel very excited about the near term because I feel like that stuff is is coming out. Yeah, I don’t know if that if that how that matches with your own trajectory, which is obviously different than mine.

Laith: It’s so interesting. I’ve been having some really great conversations with do you know Steve Fadden.

Steve: Yeah, yeah,

Laith: yeah. So um, in you know, I think this is something that like organizations have struggled with where Like, what’s after a senior researcher level is a little bit like there be dragons, you know, it’s like kind of easy to articulate like an associate to a researcher to a senior researcher. But you know, if you don’t go the management route, you know, what makes you you know, what a principal or a staff researcher or whatever that organization calls it, you know, the language tends to get a little mushy. And what would distinguish that senior researcher from that next level, I think it’s something that we as an industry are like, to your point, we’re still we’re getting closer, we’re kind of grappling with it. And it’s a little tough to, to talk about, you know, what are those those things that really go into having that impact? That would sort of denote a level beyond senior. And you know, this is a little beyond the, I think some of the title inflation that might happen in the in the in tech as well. But there’s some real interesting challenges there. And I think there’s also a set of unmet need here where I’ve met a lot of folks that have kind of hit that plateau and it’s can be a little hard, especially if they don’t have a really involved manager to know about to figure out on their own, you know, what are those things are? How do I get the training? Or how do I get the experiences, to get out of my comfort zone or try the new things or be aware of the things that I don’t know that I don’t know, to figure out how to get to that next level, it can be a little bit of a glass ceiling. And to your point about inclusiveness, I think that that’s also an area where we’re bias can really creep in, in terms of, you know, allowing people to express what they bring to the table and make a case for themselves, that they’re having that impact or however they would be measured to sort of go on to that next level. So it I like that you brought that up and it’s a it’s share the perspective that’ll be exciting to sort of see how we collectively grapple with some of that.

Steve: I know that just today and again, our today is not the today of the person that’s hearing this but anyway The research ops community just published. I think it’s called like a research skills framework. So I saw that it was posted I haven’t seen what it Oh cool, what it consists of but I think at least it whatever it contains, it validates through another source the the issue that you’re identifying, and that this is a this is a thing that’s being considered and, you know, worked on by the research community at large.

Laith: Yeah, that’s awesome. And I love that, you know, a lot of times companies can’t share these kind of frameworks externally, which is which is a shame. But I love that that they’re able to come up with something that they can share publicly and it’s one thing I do love about this community like the the listserv is so helpful. People pay it forward, people pay it forward to me like yourself when I was first coming into this, this field, and I’ve been really lucky to help a few folks here and there as they have tried to enter the field and things Like that, and I just love that there’s such a propensity towards knowledge sharing, and I think it’s just going to benefit us all in the future. So that’s one thing I am very proud of, in terms of our community.

Steve: I think that’s an extremely positive up note too. So yeah, thanks late for taking the time and sharing so much interesting stuff and really for a great conversation. I really appreciate it.

Laith: Thank you for the invite. Really fun conversation and I appreciate the opportunity.

Steve: Thanks for listening! Tell your colleagues about Dollars to Donuts, and give us a review on Apple Podcasts. You can find Dollars to Donuts on Apple Podcasts and Spotify and Google Play and all the places where pods are catched. Visit Portigal dot com slash podcast to get all the episodes with show notes and transcripts. And we’re on Twitter at dollarstodonuts, that’s D O L L R S t o D O N U T S. Our theme music is by Bruce Todd.

In this episode of Dollars to Donuts I speak with Kathryn Campbell, the Director of Research & Insights at Ticketmaster.

Whenever there is availability of somebody that might normally work on the marketplace side, they might tag team on an account manager project and that helps to inform them about that product. It gives them a little bit more purview. It facilitates internal sharing of learnings because we are a very large, complex organization. So that flexibility is both more satisfying to the researchers, but also benefits the product teams in the long run. – Kathryn Campbell

Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.

Steve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.

Over the past few years I’ve been volunteering with MoveOn a progressive organization. My contribution is to be part of their texting team, where they identify an issue or a cause and a group of us volunteers send text message about that topic to their membership. Last week I was part of a campaign raise awareness of best practices for protection against coronavirus.

We don’t send text messages from our phones, it’s through a browser interface. A texter sees a pre-addressed, and pre-written message which they send, one at a time, usually thousands in total. When people respond, the volunteer can classify that response, which will produce a relevant response that we might customize to ensure it makes the most sense.

MoveOn collects responses and future campaigns (say, for someone to call their representative about an issue, or to get out of the vote) a built based on what they learn from looking at their data.

So the next day when I was texting, I noticed two changes. The “thank you” response had been changed to “you’re welcome” which was a big improvement. Even though the “yes” and “no” responses were labels for what the person told us, if they said “thank you” as their response, your brain isn’t looking for a “thank you” label in order to send the text “you’re welcome” – you start looking for “you’re welcome” because that’s what you’re trying to say. The other change was a new response category for “I’m a medical professional.”

Well, here’s another episode! As before, this was assembled quickly, with the budget I have available for production, which is zero. There’s no professional editor, there’s no professional transcript. The audio quality is the best we could get, when people are working at home, without access to all of their gear, using an Internet under strain, and so on.

And just a reminder that you can support this podcast by supporting my business. This is an uncertain time, and many small businesses are feeling the pinch and don’t know what’s coming. I will appreciate you keeping me in mind now and in the future as a collaborator in conducting user research, and as a resource for helping grow and develop your company’s capabilities. Please reach out!

All right, let’s get to my interview with Kathryn Campbell, the Director of Research & Insights at Ticketmaster.

Steve: Kathryn, thanks so much for being on dollars to donuts. I’m looking forward to the chance to speak with you.

Kathryn Campbell: Hi, Steve. Yeah, thank you. I’m looking forward to the conversation as well. I’m a big fan of your podcast.

Steve: And now you’re on the podcast. So

Kathryn: living the dream.

Steve: Yeah, living the dream. Well, we’ll see. We’re starting to live the dream. I would love for you to introduce yourself to begin.

Kathryn: Okay, great. Kathryn Campbell. I am the director of research and insights for Ticketmaster, which is a subsidiary of Live Nation, the world’s largest live entertainment company.

Steve: So in a minute, maybe we can talk about the research that you’re doing. But can you set the context a little bit about Ticketmaster itself?

Kathryn: Absolutely. So most of your listeners are probably familiar with Ticketmaster and if Had some interactions with us over the years. And we’re a 40 year old company, and a long time innovator and leader in the live event ticketing space. So most of us think about Ticketmaster, from the perspective of the fans, and the fans are a critical user group for us very near and dear to our hearts. And the people at Ticketmaster. Mostly come there because we are fans ourselves. But we also have a number of other customer groups that we serve. That includes the artists and their managers and promoters. It includes the venues and the surfaces that work with the venues like parking and refreshments. And we also have direct relationships with most of the major sports leagues in the US and another country. So football, basketball, hockey, all those kinds of things. So as you dig into it, you realize there’s actually a very Complex ecosystem of customers to serve.

Steve: Oh, so you do research with fans, artists, venues, leagues? I think those were the kind of groups you mentioned. What? Yeah, what does it look like? I mean, you know, within what you can describe, I’m wonder if there’s any aspects of some of those types of research that maybe are things we wouldn’t have heard about.

Kathryn: Sure. One of the things that I really enjoy about my job is working. We have a subsidiary called tm music that works directly with the artists and their teams. And it’s trying to get more money into their pockets rather than into brokers or scalpers pockets. So working with them on innovative solutions that allow them to ensure that people that say members Their fan clubs can pre order tickets, or receive priority in the waiting room or queue versus other people that we have not identified and might have reason to believe could be resellers. So that’s one area where we get to work. We also get to work with venues. And of course, we tend to think of the big arenas. But there’s also the relatively small theaters that do a lot of touring productions and helping them be more efficient and have a better relationship with their subscribers as well as one time buyers. And then with the lakes, it’s helping them get to know their customers better. One of the things that takes you a while to realize about ticketing is that in the old world, you only actually knew who about a fifth to a fourth of the people in your stadium or arena were most of them you know, one person might buy four tickets. So football teams might be selling out, but they don’t really know who a lot of the people are that are in that stadium, how far they’re coming from, what the composition of their group is. So there’s lots of interesting dimensions to the research that we do with those different segments.

Steve: Great. So a ticket buyer is not the same as a event goer.

Kathryn: Exactly. Yeah, it’s a different

Steve: two different stages there. Oh, well. What? You know, how do you given the different types of I mean, customers, users, etc, that you’re trying to learn about? Does that impact, you know, how you organize your own group and in order to work with different different parts of your organization and then different customers?

Kathryn: Absolutely. And, you know, this is an ongoing debate, I think, in the research world is whether you have a centralized group, you know, Internal agency, which is what our model is, or if you embed researchers in the lines of business so that they can work more closely with the stakeholders. What I feel is best for both the company and for the team members is kind of a hybrid model. I really like the mutual support and training benefits and the opportunity to stretch and grow of having everyone on a centralized team. But some of these areas are very complex. And so our internal stakeholders would like to have some continuity over time and work with the same person and not have to re educate them about the complexities of their product each time. So what I’m always trying to do is to maintain continuity to the greatest degree possible on the products and stakeholders, but also allow the individual researcher to feel like they are growing and stretching and developing new skills. Being exposed to new kinds of projects. So that is a constant balancing act.

Steve: So this may be a really annoying semantic question, or not even a question, but okay.

I liked the phrase that you use when you talked about embedded, you talked about embedding in lines of business. And sometimes when people talk, right, embedded, they talk about embedding on product teams. And those may be the same thing. But in terms of what they what picture they paint in my head. It I don’t know, is a product team and a line of business when we think for the purposes of embedding, is that a distinction without a difference, right?

Kathryn: So Well, not necessarily. I mean, you’re right within a line of business, there’s going to be multiple product teams and some of our products cross multiple lines of business. So for example, the person product that we use that support season ticket holders is called account manager. And it is basically a white label for the leaks or the theater theaters to manage their subscription customers, right. So that’s within one line of business, which is actually our enterprise, our b2b line of business. However, those people, the individual seasons, subscribers, season ticket holders are still fans, and they also want to buy maybe individual tickets, right? And they want to be able to manage their tickets in a centralized way. And they can easily get confused if there’s a different experience and account manager than there is in their account on ticketmaster.com. So, you’re right that it gets muddy. It’s not the same thing. But in terms of the underlying question about How do you balance that? One thing that we try to do is let’s say, we have one person who is an expert, which we do we actually have two people that are experts on account manager. However, whenever there is availability of somebody that might normally work on our what we call the marketplace side, the b2c side, they might tag team on an account manager project and that helps to inform them about that product. make them more aware of the complexities for a fam that has membership, you know, an accountant both and they can also inform that account manager team about some of the things that we’re doing on the marketplace side, right. So it gives them a little bit more purview. It facilitates internal sharing of learnings because we are as you’re beginning to get a sense we are evolving. Very large, complex organization. And we’re very fast moving. And so that flexibility i think is both more satisfying to the researchers, but also benefits the product teams in the long run.

Steve: It’s an interesting, it’s a, it’s a, I think, a brilliant idea to do sort of knowledge transfer or knowledge distribution, there’s probably a smarter way to say that by moving people, the people contain information, and you move the people around and the information, the knowledge, the expertise gets shared that way. As opposed to, you know, I feel like sometimes we will. There’s a there’s an idea, we should extract the information that’s in people and then centralize that, but you’re kind of, you’re distributing the people. Because the collaboration, you know, lets that expertise come to the surface and have impact in ways that you wouldn’t necessarily be able to predict.

Kathryn: Yeah, I think that’s absolutely true. I remember listening to someone talking about how that was a strength that Google the way that people tended to move between different groups and the informal knowledge sharing and transfer there. I think with large, complex organizations that whether we like it or not that informal knowledge sharing is at least as important, if not more important than the more formal documentation, which is always going to be less rich and behind the times, right. And I also think, you know, at the risk of digressing that this whole issue of really caring for your team and their career is one that I think is too often overlooked. You know, even on the agency side, it’s always easier to satisfy your client, whether it’s an internal or an external client by having them keep seeing the same face. But no, who wants to be that person that only knows about One thing and keeps doing the same thing year after year. And you know whether or not you want to build out your portfolio because you want to eventually leave or maybe you just want to stay interested in engaged and challenged. I think it’s really incumbent on us as leaders to facilitate that growth. And that is a lot easier to do. If you deliberately encourage this kind of movement across different project types and skill sets.

Steve: What else do you do to take care of your team?

Kathryn: Oh, gosh, we do a lot of things. You know, in our weekly meeting, one technique that I developed a few years ago when I was working with a boutique agency primitive spark was to start the weekly team meeting by really sharing on a more personal level. What’s present for you, you know, are things going great and you want to share something? Do you have a sick kid at home And you’re worried about them. We really try to bring in a bit of that humanity to our professional understanding of each other, because it helps you understand where people are coming from. We also really try to encourage thinking as a team in all respects, we actively try to load balance. So you know, with research, you have the best laid plans, right? have everything laid out and then a product gets delayed, or it gets accelerated. And so you might find yourself either with three major projects that you need to do interviews on this week or nothing for three weeks. And so we’re constantly encouraging the team members to chip in and help each other and again that facilitates that ability to knowledge, share and maintain both personal growth and professional efficiency. Time,

Steve: whenever there. I mean, it seems like in some organizations there’s, you know, a lot of measurement of where you spend your time or where you don’t spend your time that may have the it may work against what you’re talking about, right? What How are people incentivized or just incentivizes spend their time? So, I’m wondering, I don’t mean I don’t know sort of how it works at Ticketmaster, you can obviously, say this is the goal. But is there are there measures in place to reward is the wrong word, but to support that kind of operationally?

Kathryn: Yeah, it’s a really good point because we get back what we incentivize and what we measure, right. So I do not keep a tight track of hours per week, or, you know, we don’t have billable hours because we’re an internal agency that that Not what the focus is on so much as a couple things. One is what kind of research we’re doing. I’ve heard you in prior podcasts, ask about focus on generative versus evaluative research. And so that is a focus for us. We want to continuously move upstream where we can have more value in terms of, you know, the quote to build the right it before you worry about building it, right. So we do have quarterly, okay, ours, objectives and key results around what percentage of your work is generative. To that end, we have embarked on a democratizing research program over the last year where we’re training the product designers. In how to do their own UX research, you prototype research, which has a number of benefits for everybody. One is it’s faster for the designer, they can do quicker, smaller studies, if they don’t have to wait for one of us to be available. They get closer to their end customer and get to understand them better. And it also means that we don’t get into the situation that happens with many UX research departments, where we’re just constantly every week or two churning out new studies, but never moving upstream to the bigger product concepts, right? So this way, we can kind of push off some of that more routine, prototype testing work. And we’re now focused on doing concepting focus groups or surveys to size market interest in something or facilitating a design workshop or or those kinds of

Steve: So I wanted to ask them about where projects come from. You’ve made space kind of made bandwidth for your team by empowering others to take on a certain set of projects. And now you listed a bunch of things that, you know, you’re free to go and do. Where do those projects originate?

Kathryn: Right? It’s a great question. And I think that feeling empowered to take some ownership of that process is probably the inflection point in terms of really adding value to an organization with research. So I am extremely fortunate because ticket masters president Amy Howe, and our cmo cat Fredrik out when I joined two years ago, both of them explicitly said to me, I do not want you to be an order taker. I do not want research to Be order takers. And so that said to me, You are empowered to go out there. And so we do a couple things. One is as projects come in, and we have a process by which things come into the queue, and then they’re reviewed and groomed weekly, and that sort of thing we talked about, okay, what here best aligns with a key corporate priority? And Ticketmaster is pretty good about sharing what the key business priorities are that people should be focused on. But then occasionally, we also say, Gosh, I wonder I keep hearing about this new thing, and it seems like a really important thing, and it seems like something we should be doing research on and nobody’s asked us to do a search. So we feel very comfortable pursuing those stakeholders and saying, gosh, you know, wouldn’t, wouldn’t it be a good idea to get some end user validation that this is the right set of requirements before we built this, or to help ensure you know that that we’ve captured all of the requirements here. We can do that we also have built in a model of periodically checking in with the most senior leadership at the company to tell them what we have been working on. And also what’s coming up, and to say, are the projects that we’re planning on doing for the next quarter aligned with where you think the real value for the organization is? And so that can also be a point of alignment. But yeah, increasingly, we find ourselves suggesting things that really should have research done on them. And sometimes, in fact, we just go ahead and do stuff. And I think that as long as you’re not just pursuing, kind of cool vanity projects, but you’re using as your North Star, what the origin This nation says are truly the key business priorities. I don’t see that that whatever turn out badly for you.

Steve: So in the situation where you are, you know, you’ve identified something that a group is working on. And you approach those stakeholders and say, Hey, we we could do this research that would help you at this point. I guess my, the question I’m really after is, you know, that this thing that a lot of researchers Express stress about the, you know, no one listens to me. They don’t take our recommendations, we can’t kind of impact the team, they go ahead and do what they’re going to do anyway. So how did the How does the How does the taking action on the research part play out when you are, you know, not being an order taker when you’re being proactive and kind of coming to them? What do you see happening as a result of the research?

Kathryn: I think that we as researchers need to be willing to be flexible. I don’t like to encourage stakeholders who constantly come in at the last minute when they could have allowed for more planning time. But sometimes things really do emerge at the last minute or become very urgent. And so we need to be willing to say, hey, what can we do to try to make research work within the time constraints and the other needs that this project has. And I think that if you are willing to be an empathetic partner in that respect that that earns you some trust in the relationship and they can see hey, you’re knocking yourself out to make this work. So I should go ahead and use it. And then the other thing is some stakeholders just get it more than others do and one of the best pieces of advice I got from analyst at Forrester Research when we were talking about how to build a culture that respects research, what she said, Hey, you know, work with the people that really value you create fabulous case studies, and then use them as advocates because that will have a lot more impact on the recalcitrant stakeholders then you constantly saying, Hey, why don’t you use research? research is good, right? So, you know, when I have to prioritize who to make room for I definitely prioritize people that historically have acted on our recommendations. Yeah.

Steve: That’s, that’s really good. I just want to highlight the first part of that, and make sure I understood that. You’re saying that if you adapt and you know, are flexible and are sort of demonstrably digging in to support someone, that that sets them up to be more receptive to the results of the work that you’re doing for them.

Kathryn: That’s that’s been my experience, you know, let’s face it 90% of the time. The reason someone does not embrace research is, well, sometimes they say to themselves, well, I won’t learn anything new. But the biggest fear is normally this is going to take too long. I’m already behind. I have to hit this deadline. That’s what I’m going to be measured by. It’s back to what we talked about earlier. people behave based on what they’re measured on. And product owners and certainly project managers tend to be measured on Did you deliver on time, right? So if I go to that stakeholder and I said, Look, I am really worried about the unknowns in this new product and rolling it out too quickly to a large audience without doing some more user testing with the beta product. We’re willing to take our whole team and work in the evening or over the weekend and pull this together because we feel so strongly that that would be a good thing to do. Well, that kind of puts the stakeholder in the position where if they say no, now they’re going to be held accountable if big things show up, right? They’re making a very conscious choice to say no. And the good news is, you know, I won’t say how long I’ve been in research, but it’s been quite quite some time. And in all of that time, there has literally never, ever even one time than a research project where I didn’t learn anything. There’s always a surprise and in our business, sometimes that little surprise can turn out to be something quite significant. So for us, you know, now in the last year a big product innovation for us has been around secure mobile ticketing, which means that your phone is your ticket, right? I don’t know if you’ve had experience now with going to events and everything’s on your phone. Well, if you happen to have, let’s say, an Android device with a heavy cover on it, it turns out that that impedes the scanner working correctly in certain circumstances. So there may only be if we do a quick beta test with 100 people, there might only be two or three people there. But when you scale that up to all of the people that are going to any event in the next year, all of a sudden you’re talking about having a big impact on our call center, for example. So you know, highlighting Oh god, I’m glad we combat. And and you know promulgating that story of how we went to bat and how we found those insights. You know, I think it builds a sense that it is in everyone’s best interest to work with us. Now, the flip side is, I also constantly am tracking down and hounding the Senior Product people to say, what’s coming up this quarter. What’s coming up this half of the year? Have your priorities shifted, let us know as soon as possible because I don’t like living in a state of chaos. I don’t like constantly having to shift all the priorities because of something being coming up at the last minute if in fact, we could have known about it in advance. So now when they see me in the hallway, they immediately go through their mental checklist of, oh, is there anything that I should tell Kathryn and her team about?

Steve: Right, because you want people saying Oh, when they So you’re coming down?

Kathryn: Well, in a good way,

Steve: in a good way, the good kind of Oh, here comes Kathryn you’re painting a picture for us. I mean, the you mentioned that, you know, when you first started talking with the leadership and protect the master about taking on this role, they had a they framed the position as something where you had agency where you were kind of going to do things as you said, not being an order taker. I mean, this, I would imagine, you have to guess here, but what what do you think created the conditions for them, that they understood the power of research and someone like you to, to work in order to work that way? How did they see that light?

Kathryn: You know, my hypothesis about that is both of those women are very smart, very data driven executives. So they We’re not particularly knowledgeable about research, we do have Ticketmaster has an extremely robust analytics group. So we invest a lot in site analytics and marketing analytics and have very in depth reporting and good turnaround times for learnings. So I think that they just wanted to extend that from the behavioral, you know, what are people doing data that we were very rich in into the Why are people doing that side that we were not rich in? So I think that having a data driven organization that only had half of the insights that they needed is probably what created that willingness to or that embrace of the research group.

Steve: And how does the work that your team does? How does that set With the the people or the work of of analytics?

Kathryn: Oh, that’s a great question because it’s kind of an unusual setup and yet I do think it’s the direction that the industry should move over time. So both micro, which is research and insights primary research, we sit next to an equal to the site and marketing analytics group. So that together we form a combined insight center of excellence that reports up through marketing. And Forrester has been promoting this idea of an insight Center of Excellence for some time of bringing together all of those different sources of information about your customers. The site data, marketing data, UX research, survey research, qualitative research into one place And when I hear the stories, the war stories from my colleagues at other companies, it really helps me appreciate how wonderful this construct is. Because just the other day, I was talking with a friend who’s in charge of user research for a global company, and he said, you know, any time our work starts to move in the direction of maybe doing a survey, all of a sudden the marketing research team wants to get involved, but they’re a completely separate group reporting to different people. They’re not as knowledgeable about the background on the study, you know, it becomes a big deal to engage them and yet, they don’t want us to do anything that looks like a survey if they’re not engaged. So you create these territorial issues and obstacles to free, flowing information and you’ve been unmuted. For a long time, you know how it is it’s kind of like unwinding the sweater where you start out in one place, and then you learn something interesting. And so you start to follow that down. And so things keep evolving, you know, the appropriate methodology and keeps evolving. So having everything centralized, the only thing that’s not centralized that I think is a key part of the insight into our customers is fan support and client support. And we work very closely with those groups. We exchange reporting, and we also check in with each other when an issue seems to be arising to see you know, what each other is hearing. But pulling all of that together, I think makes the company just much more insightful and more nimble to act on insights.

Steve: Maybe we can shift a little bit be interesting to hear about, you know, the makeup of your team, whatever, what kinds of backgrounds that people have coming in? Yeah.

Kathryn: So, you can imagine given the wide scope of research that we do, it’s, it’s a mosaic of people with different backgrounds and different skills. There’s a certain core skill set that everyone has to have in terms of You know, basic one on one interviews, usability testing, and straightforward survey research design. But beyond that, we definitely have areas of expertise, we have one person who really loves and embraces research Ops, and that is a godsend to all the rest of us, because it just makes everything much more organized and all the rest of us more efficient. We have a couple people, including my number two person who come from design backgrounds, and so they are especially good at thinking through how to make the information design as impactful as possible. So we’ve really been up leveling the quality of our reporting to make things more aesthetically pleasing and also to help focus the reader on the couple of key numbers or quotes or Whatever that they should really come away remembering. We have we just brought on someone who is expert in advanced analytics. So like I said, What I’m looking for is everybody has a basic, a couple of the same tools in their tool belt. And then across the team as a whole, were able to offer richness of skills in a wide variety of areas.

Steve: And when you talk to prospective hires are the kinds of things maybe beyond you know, the skills necessarily, but I don’t know. How do you think about trying to assess if somebody would be a good addition to your team?

Kathryn: Yeah.

I think that really the personality characteristics are more important in most cases than the hard skills although sometimes you need a particular Hard skill. It’s really important that people be smart and think quickly on their feet, you know,how how intensely you have to focus to shift when an interview shifts or to look at data and think about what it’s really telling you and whether or not there’s alternative explanations for that. So that that kind of skill set a mental nimbleness is very important. It’s very important to me, in particular, that they show initiative. I don’t want to be a micromanager especially, you know, when you’re working remotely and right now a lot of us are working remotely, you need to trust that people are going to pick up things and run with it. I think that the basic people skills likability and very strong verbal and written communication skills are pretty key. Once again, where we’re servicing Senior stakeholders across a wide variety of businesses and sometimes our insights might mean multimillion dollar shifts and investment. So a person has to be credible and take feedback seriously. You know, one of the things that we practice a lot is it helping our clients recognize the actions that are implied in an insight, but not overstating what the research itself told us, right? So it would indicate that this is a good new hypothesis for us to test and explore, maybe through research, maybe through a B testing, you know, but without saying, Oh, yeah, this is what fans want us to do if that was not exactly what the research says. So that kind of discernment is pretty important to maintain credibility as a researcher.

Steve: Among the researchers, today, what kind of relationship do you all have to? I guess live entertainment, right? The business that the ticket master is in, what’s the culture among your team relative to that?

Kathryn: It’s pretty fun. I have to tell you, it really is, um, you know, the hand employee handbook says, you know, in terms of dress code, it says, you know, come to work is your authentic self. And so, you know, people come to work and they’re in band t shirts, or they’re wearing a jersey for their favorite sports team. We have dogs, you know, dog friendly policy. So a lot of people bring dogs to work, when there’s a major sporting event, you know, the World Cup or something like that. We throw it up on the TVs in the office. So we are all fans and we really do empathize with and more To maximize the joy in the live event experience. And the flip side is we work. It’s a very, very, very high velocity environment. Ticketmaster is actually one of the largest e commerce companies in the world. And as we sometimes say, you know, every Tuesday is like, Black Friday, because, you know, when you have one big global music musician after another coming out with their tours, and you have 20 million people trying to get 400,000 seats all at the same time. You know, there’s a lot of stress on systems. So, yeah, it’s an intense environment, but it’s also a fun environment.

Steve: Do people, do people still line up to buy tickets that’s up maybe showing my age or a nice

Kathryn: They do but they line up virtually. they line up in a virtual waiting room. And yeah, and it’s it’s intense out there, you know, they’re stressed out and we have a lot of compassion and empathy for that, you know, somebody is so frustrated because they’re, you know, biggest star is announced a tour and they really want to see them. And the reality is that even if the systems work perfectly and very frequently, the systems don’t work perfectly. There’s just you know, demand outstrips supply. So, in certain circumstances, you know, when BTS goes on tour when Taylor Swift goes on tour, not everybody is going to get tickets, at least not immediately, you know, maybe eventually but it’s not easy. So again, working through fairly sophisticated tools to try Try to root out bots we actually get we turn away more than 1 billion with a B bots a month. You know, resellers and bad actors are constantly hitting our systems trying to scoop up all the tickets so that they can resell them at a much higher margin and we are spending all of our time trying to outwit them and trying to deliver real fans tickets to the events that they’d like to see within their pocketbook if at all possible.

Steve: So demand you know, for successful artists demand is always gonna outstrip supply, but you’re trying to make sure that as much supplies as possible is available for the for the fans.

Kathryn: That’s the goal.

Steve: I’d like to I’m going to shift gears.

Kathryn: you’re the host, you get to do that.

Steve: All right. I would love to hear a bit more about, you know, we’ve talked about, you know, your work most recently, but can you talk a little about your background, how you found research and how you found, you know, the kind of research that you’re doing now? Sure.

Kathryn: It’s an unusual background for UX research, but I think it is the perfect background for what I’m doing now and the direction that I think research is moving. I actually started out working for advertising agencies doing marketing research, doing segmentation studies, message testing, focus groups, you know, various brand measurements and became a brand strategist. I was is actually a partner for Ogilvy and Mather ad agency over brand strategy. And what I realized as the internet changed from being viewed as just a media channel to being really the whole business for some companies, I realized that the biggest problem with some of the brands I was working on was their poor user experience. And so I wanted to learn more about the elements of good user experience and what made a site intuitive as part of understanding how to align that experience with their brand promise. And over time, I just got more and more interested in user experience. And so that became both through self education and reading and taking classes. I never got a degree in HCI But I migrated into UX research. And because I had the prior background, I could also recognize when some questions came up, and I would say, Hey, this is not, this is not the kind of thing that Jakob Nielsen meant when he said you only needed five users. This is actually a survey research question that you’re thinking about here. And this is how we should go about that, or this is a conjoint pricing question. So developed an unusually broad research skill set, which is, as I said, perfect for what I’m doing today. I’ve given a lot of thought to why, what the obstacles are to having more integrated research departments like the one that we’re building at Ticketmaster. And I think the problem is, people come up through different silos. I mean, the research mindset, broadly speaking, is appropriate across all these different techniques, it shouldn’t stop with, you know, at the edge of one tool and now you need a completely different kind of training, researches research at a fundamental level. But I think the fact that some people start like I did in marketing research in the marketing department and some people start in UX research either in it or design or the product group, and they just don’t have much exposure to the other side. I think that that legacy of corporate organization is really what is holding us back more than anything intrinsic that says that we couldn’t work more closely together.

Steve: Hence the insight Center of Excellence.

Kathryn: Exactly.

Steve: Yeah. But then I think about you know, your, your earlier point about, you know, people, people moving themselves between different Different groups is learning for them, but also, they are pollinating or cross pollinating. And so you’re someone that’s moved from the advertising world in the marketing research world to the, as you said, the user experience world right now they are separate worlds. Yeah, I guess my question is just one that reiterates what you said, which was that it was, it was a, it was good for you to be able to do that and set you up. So

Kathryn: I think it’s good for all of us. Really, you know, I think the fact that we tend to transition careers is not just a fact of life, I think that it actually strengthens us. You know, so Dustin connais. At Ticketmaster, he’s Senior Research manager and he actually had started a pod of user researchers in the enterprise Design Group at Ticketmaster. Before I arrived, Dustin was a designer. And he took Hillary beanstalks class over at I think it’s Cal State Fullerton, and got so interested in user research that that became his passion. And he transitioned into becoming UX research and talked the design leader into letting him hire more people in that area. So now he’s part of this larger group. But he’s one of the people I was referring to that really helps elevate all of us. A lot of researchers are good at identifying insights, but we’re not fabulous at the way that we communicate those insights certainly visually. I’m one of those people. And so Dustin has been developing templates and guidelines so that our work isn’t ugly and tedious. To make it through, people are proud to share our decks and the key information jumps off the page at them. And, you know, we wouldn’t have that capability if Dustin hadn’t migrated his career in that way.

Steve: Well, he sounds like you to a certain extent, sort of discovering something that’s adjacent, getting very excited about it, investing in developing it themselves and then bringing, bringing not just the skills that they’ve learned, but that larger perspective, that larger set of skills and mindsets to help because research is a multifaceted thing. I think we keep talking about that. And so the diversity of skills is only enhanced by people kind of coming in from elsewhere and bringing their their previous kind of wardrobe or kits with them.

Kathryn: I agree. 100%. You know, I think that people that go into Research are almost by definition intellectually curious. And, you know, so we all benefit from the other things that are interesting to them, right?

Steve: I mean, one of the things that I’ve reflected on, you know, just given that I’ve been in this profession for a long time, and you know, I was in this profession when no one really knew what it was and including me that now you can’t now there are undergraduate programs or graduate programs, there’s a thing with a name that people can find, and they kind of come into it with a definition of what they what it is and what they’re going to bring to it. Of course, that doesn’t mean so that sort of the shape of the field has evolved because people are choosing it earlier, as opposed to finding it later. And I

Kathryn: yes, I mean, I agree with that. And I am right there with you. I share that that surprised at how Oh, wow. This is now Real Thing and none of us that started out, you know, came to it with that kind of background or at least very few. And yet, you know, if you go to research meetups, like I actually am the organizer for LA UX research meetup. Here in Los Angeles, there are still quite a few people that want to enter the field from adjacent fields, right. People that were designers, people that were content strategists, people that did maybe academic research. And so even with that clearer path and training opportunity, I don’t think it will ever necessarily end that there will be people trying to figure out how to leverage the life experience that they’ve had that does still bring a lot of value of a transition into a new skill set.

Steve: I mean, I feel like life experience is, and I don’t mean the amount of life experience you have, I just mean, the, you know, back to your, you know, your meetings where people talk about what’s going on with with them, you know, a culture you work in where people can bring their authentic selves, you know, that, to me seems so supportive of reach research, because it’s a mean depending on your method, but there’s a large amount of human to human that’s in it. And it is that makes it messy, but also very real. And if you don’t have space to be yourself at work, if you’re not creating an opportunity to kind of connect, you know, backstage in front stage or whatever your metaphor is, then that makes it hard to I think, to do research as well as it possibly can as you possibly can. So, as you said, your life experience whether that’s, you know, anything, I mean anything, any life experience that you have had, as well as sort of discipline and education professional experience to me, feel Kind of feeds the whole beast

Kathryn: 100% 100% you know, we also have another Senior Research manager on my team Brent Lowe. He came from a theater background and you know, so he brings an energy and exuberance to facilitating brainstorming sessions and meetings and helps to coach some of the people that are perhaps a little bit more introverted by nature and how to maintain that high level of energy that, you know, facilitates people contributing lots of ideas, right? So it leveraging whatever background and passion you have and being able to find a way to nurture and express that in your profession to me, that’s that’s a win win for everybody. Right.

Steve: And within your team, people, you know, and you said this before, we were talking more about about maybe methods or projects, I guess, but that also is an opportunity for each other to learn. I mean, work with a theatre person on the same project, you’re going to just you can’t help but be exposed to and inspired by different approaches to anything.

Kathryn: I agree.

Steve: And so you mentioned that you you had when you were working in advertising, you know, you you spent time in the agency world. How do you contrast that a life inside a corporation versus life inside an agency?

Kathryn: Yeah, that’s a good one. It was a big transition for me. I spent a lot of my career in agencies either large or small. And you know, the fun thing about agency life is usually everybody’s very smart, very creative. You’re constantly learning something new because you have Learn about the industry for each of your different clients. And that’s all fun. And I was I was nervous to be completely honest with you when I went to work for a huge corporation, like Ticketmaster Live Nation. And the good news is that ticket master is as I was explaining earlier, so complex, I don’t think I’m gonna get bored there anytime soon. You know, there’s, there’s still lots and lots and lots to learn. And it is kind of fun to rather than just tell somebody what they should do and give them the idea and hand it off to them. It can be it can be frustrating and it can be time consuming, and it can take a lot of perseverance to see that through. But there’s also a real joy in that Being part of something that comes out the other end, then you say, wow, you know, I played apartness, building that product in the first place, or I made a major contribution to how user friendly it is. And so that creative aspect of it is a little bit different. It’s not as much time on the blue sky brainstorming and a little bit more on the Okay, so how do we still pursue this given these technical integration considerations or these other challenges that in the real world, you have to work through, right. But it’s been very fulfilling. It’s been a great experience so far.

Steve: Is there anything that we should try to cover in this conversation that we haven’t got to?

Kathryn: Those were a lot that you hit on a lot of great questions. So it’s been a real joy chatting with you. I’m just kind of curious, you talk with so many people. Are there any trends that you find emerging or find very interesting in user research right now?

Steve: I mean, part of the fun of doing this podcast is talking with people who are I mean, it’s the design intent of the podcast, people who are who are in leadership roles. And you know, at so where the organization has invested to a certain point, because I think if you sort of hang around with researchers as I do, you hear a lot of the same concerns. So it’s interesting to talk with you and hear how either you’ve built around them or the conditions were created differently. So you know, you’re describing a situation where the research team is proactive. But that’s by design. It’s. So you are sort of you were given that opportunity. And I don’t mean to diminish your accomplishment that way, but the culture, the culture, the leadership kind of started off there. And I think a lot of people want to get to that stage. And it just makes me think about you know, like, there’s a lot of there’s a lot of frequently asked questions, I think that and we talked about them, maybe using some different language, but you know, having influence getting time, you know, having buy in isn’t there’s a lot of anxiety about that. I, I hear that from people in so many different environments that I go into. And I feel like more and more, I’m realizing, and maybe everybody else already knows this, but I’m realizing it’s like it’s hard to manage up on those things. It’s hard to, you know, create a culture that isn’t there. It’s hard to sort of change leadership. shifts mind. It’s that the there’s a lot of these are big picture things, how do we get everyone working on the same thing? Or how do we not duplicate research? Well, we have to devote resources to managing that. And I feel like because there’s a lot of scrappy research practices out there. Sort of under resourced, without leadership without guidance, you know, not completely bought into there’s a lot of hope for you know, magic fixes to systemic problems. So, you know, when I hear how people have worked hard to either find or build, you know, cultures, investment supports, that doesn’t have those problems. I don’t mean to say that everything is perfect for you or anybody that that I talked to, but it sort of drives home that you kind of can then there should be some folksy saying about how you you Can’t get the thing that you need without really going without really having what it takes to pull that off. So, if you’re a solo researcher and you work kind of taking, you know, taking orders, you know, for some designers and you’re by yourself, you can’t really change the corporate culture and buy in around research in the belief because that, you know, often needs to be there beforehand. So it’s a it’s a heavy lift for those folks. And I know they want it. So yeah, yeah, I mean, I’m telling you what I learned from you, I guess it how it fits,

Kathryn: right. Well, no, I, I really agree with what you’re saying, you know, there’s that old saw. Pick your parents carefully or pick your parents? Well, you know, the secret to so many things in life is good genes, right. So I think that what you’re highlighting for folks that are looking for work is You know, especially earlier in your career, it’s easy to get caught up in the title and the salary and that kind of thing. But there’s, there’s some of these bigger issues that you’re touching on Steve that are really so much more important. You know, what, what is the attitude toward research? Are people looking forward to you and what you’re providing? Or are they crossing their arms and being defensive about it? Do they view research as being an essential ingredient in success? You know, I sometimes say to people, I know that you’re just focused on getting out the door as quickly as possible, you know, moving fast and breaking things, but I genuinely believe that I am the fastest path to you launching a successfully adopted product in the marketplace and by I mean the research team, right. So you know, looking at Those things asking some questions kind of testing that out. If you are earlier in your career, and you can find a company that will really value and encourage you in that career, then that’s going to be a lot better for you in the long run and a lot more satisfying, personally, than making an extra $5,000 or having a senior in front of your title or or that kind of thing. I mean, that’s my perspective. What What do you think?

Steve: I mean, I would say yes, and some people may like the challenge, some in but you know, I guess go into these things with your eyes open. Yeah, so if this is not a place, go ahead.

Kathryn: Oh, no, no, I was just going to say I do think this is one of the drivers behind the trend in the last couple of years of teaching UX designers and UX researchers to better understand the language of business. I imagine you are familiar with, you know, these workshops and things that are starting to crop up. And there is some truth to that, that if a person in product leadership is focused on, you know, we were talking about before, what are you measured on? Okay, well, you’re measured on, you know, did you hit stay within the budget for this product at launch on time? What was the initial adoption? And the initial adoption is going to be the last thing to be measured? You find that out after the fact, right? So if we only ever talk about user experience as well, we want our fans to have a positive user experience that makes it seem like a nice to have, right? Yes, if time allows, of course, we’d like to do that. So I do think that if you’re in one of those less hospitable, less enlightened environments, that learning how to frame your work in a way that makes it clear that it is directly part of that team’s success and that you are all about I use terms like mitigating risk, enhancing adoption, reducing wasted dev effort, right? Those are the ways that I talk about research because it makes it really clear. And, you know, one thing I left out of my background is along the way, I got an MBA in business, you know. So it makes it a little bit easier for me to think about the way to talk about what I do in the benefit oriented terms that my stakeholders will understand.

Steve: I feel like what pulled research along you know, for many years, when we started having phrases like UX, which we didn’t always have, was was designed was user experience, and that that was kind of the foot in the door for a lot of technology companies. And that it, those people in that work highlighted the need for research and so we’ve kind of been pulled along. And, you know, I think you’re highlighting another what feels like a transition to me that research is part of the organization, whatever part of the organization, all of the parts of the organization, or can impact all of those. And that’s the reframe, you’re talking about, from, hey, research, can we can help create a good user experience too. We can help, right, you know, change the how the business model we can help. Right, reduce inefficiencies in engineering. Those are all that’s not that’s research sort of stepping out of the shadow of its big brother and saying, like, Hey, we’re, we’re a function that does all this stuff. And that’s I feel like that is part of the future that you’re that you’re looking towards and kind of helping to move to feel towards

Kathryn: Yeah, a great, good way to put it.

Steve: All right, well, if we’re almost at wrap up, I want to tell you about my experience working at Ticketmaster.

Kathryn: Okay.

Steve: Because I worked as an undergraduate computer science student I, I worked at a subset was a subsidiary It was a company in in Toronto called bass, which was best available seating service. Bought bikes got bought by Ticketmaster long ago. It was bass ticket master for a while then just became like, part of ticketmaster.ca when I was a night run operator, so there was no there was no internet. So it was it was lining up at the record stores that had terminals, and it

Kathryn: right right.

Steve: And it was the phones. So I was in a little office that had like the phone banks. You know, if you think of like a 70s movie with like a computer room with like the raised floor and the glass walls, you had to like to like badge in into it. I worked well, I worked in like an office outside there. And so at the end of the night, like I would go in and was like a, I don’t know, like a 6pm to like 1am shift. And you basically had to wait for all of the nodes of the network to close like so whenever the club sold its last ticket or the planets ran his last show, all those things would go offline. And then there were all these like men and women who had to run scripts that would then print out I think every record for the day. So you just print out these huge so that was their audit trail, I guess. I mean, I didn’t have any mental model at the time, but you just go print out all these things like 132 character tractor, whatever the metric was the super wide. Super fast printers that were in this room and you have to, you know, tight tape all these reports and take the tractor feeds off them and staple them and put them everybody’s mailbox and then do a disk backup, which was these dishwasher sized appliances you’d open the top and there was a harddrive in there which looked like like a giant kind of cake container. unscrew them physically move, you know, disc a to there was two there was a some kind of backup thing, you’d move one to the other and take one off the shelf and like run this, like 25 minute job to kind of back everything up. And it was I mean, I don’t know if it was archaic at the time, but it’s obviously just like a weird history thing there. But of course even even the tech people were like so into music, so into bands just exactly like what you’re describing there. They had that same culture. And it was fun to be part of that. It wasn’t fun to be moving disc packs around it, like 1am. But

Kathryn: anyway, pretty cold, right?

Steve: Yes, temperature control.

Kathryn: Oh, that is great. That is a great story. Well, you’ll have to, I’d be happy to get your ticket master t shirt. It sounds like you’re an OG

Steve; like old old OG. Yeah.

Steve: Well, this was really fun. Thank you for taking the time and, you know, sharing your perspective on all the things that you’ve accomplished and kind of your your vision for the field. I think

Kathryn: I really enjoyed this. Thanks. Thanks so much. Yes.

Steve: Thanks for listening! Tell your colleagues about Dollars to Donuts, and give us a review on Apple Podcasts. You can find Dollars to Donuts on Apple Podcasts and Spotify and Google Play and all the places where pods are catched. Visit Portigal dot com slash podcast to get all the episodes with show notes and transcripts. And we’re on Twitter at dollarstodonuts, that’s D O L L R S t o D O N U T S. Our theme music is by Bruce Todd.

In this episode of Dollars to Donuts I speak with Laura Faulkner, the Head of Research at Rackspace.

I’ve never just sat and done just what I was asked to do. I’m always looking for something new, something else. It’s probably just part of how I’m built but it’s also a conscious choice of, of just doing my my current job is simply not enough for me and even just that extra 10 minutes of curiosity or desire to see something else or learn something new while still doing my job and delivering on that, it’s just opened so many doors helped me step into a lot of opportunities and learn new things. – Laura Faulkner

Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.

Steve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.

I just watched Martin Scorcese’s 2010 documentary “Public Speaking” a profile of author Fran Leibowitz. I honestly didn’t know very much about her, but she’s a smart and acerbic social critic, whose persona, and perhaps her life is built around her strong opinions and her skill in expressing those opinions. She repatedly emphasized in the film how there are too many books, and thus too many poorly written books; she’s pretty critical of the ability of most people to have an interesting story to share and to have the ability to communicate it in a valuable and valid way. While her career is indeed public speaking in one form or another, for the rest of society, she advises public listening. But really this is because she wants to be listened to, she doesn’t necessarily want to listen, she wants to talk. This is in no way meant to be an indictment of her – she very clearly exists in a world of her own making. While the mode of the times, and indeed given the work that many of us do, emphasizes the importance of everyone’s stores, and we put real effort into inviting people in and giving them space. I mean, this approach, this mindset is at the core of user research! But I found it remarkably refreshing to hear a different point of view.

It reminds me of an experience I had in the field last year interviewing a head neurologist, someone who was very successful, who had written multiple books and taught surgery around the world. Before this project, I was kind of warned to expect a lot of ego from surgeons especially, and I guess in some ways this particular interview subject was the platonic ideal of that. He would not sit through my carefully scripted concept presentation. He interrupted. He pointed out his bookshelf of published books, he told us about his children attending an Ivy League school. He gave examples of where he’d been influential, how many times he’d taught, how much money he had saved the hospital with his process innovations. I had to adapt my approach for him so that we could get through the session. After we left, my colleague, who worked my frequently in this domain half apologized and half-checked-in with me to see if I was okay or not, given how quote awful and difficult the interview had been. But I didn’t have that feeling at all. The interview was amazing! We learned so much from him! He was incredibly accomplished, and very very opinionated. And like Fran Liebowitz, a total New Yorker about it.

But I wasn’t having lunch with him, I wasn’t trying to be friends with him. I was trying to learn from him. And I approached him – as I do everyone I interview – as an expert. He was just more able to act in that role that most people are.

I am very sensitive to people who don’t listen to me, or who let their ego run all over me, or otherwise disrespect me. But this was an interview situation, and his so-called arrogance and ego was amazingly helpful and honestly didn’t feel disrespectful in any way, so it was just fascinating to see how someone else in the same session felt about it.

I’m glad there are people like Fran Liebowitz and this surgeon. They are in the world to do amazing work, to inspire people, to make a difference. That doesn’t mean I want to hang out with them and try to tell them about myself. But that’s okay, isn’t it?

So, we’re back with some more episodes. As of this recording, we’re in a particular bizarre period. This episode was assembled quickly, with the budget I have available for this, which is zero dollars. I’ve foregone my professional editor and my professional transcriptionist. You will hear some digital noise and distortion on my part of the upcoming interview, but it comes and goes, and I’m not talking too much, so I hope you don’t get too bothered by it.

It is a challenging time for businesses of all types, and certainly for small consultancies. It’s a challenging time for my own practice. I hope if you’re listening to this that you’re in a position to consider me as a potential collaborator for user research, training, coaching, and facilitation. Thanks!

All right, let’s get to my interview with Laura Faulkner, the Head of Research at Rackspace.

Steve: Laura, thanks for being on Dollars to Donuts. It’s great to speak with you.

Laura Faulkner: Thanks, my pleasure.

Steve: So I think it’s a great starting point would be to ask you to introduce yourself. Tell me and everybody what about your work and we can kind of dig in from there.

Laura: So I’m Laura Faulkner. I have a doctorate in research psychology focusing specifically on human computer interaction. I got that from the University of Texas about 10 years ago as I was already working in the field. As of just a few weeks ago, I celebrated 24 years in this field. So I was in it before it was a job. And when we were just doing it and practicing it alongside whatever else we were doing, especially in technology now I’m the head of research at Rackspace, which is rather fun in in a couple of ways. One is that it’s an enterprise organization. And we do really deep technologies essentially provide the internet and manage that for folks. So we’re doing really complex interfaces. And I love that I love that problem space with all those bizarre technical details and add lots of complexity to manage. And our to make that easy for customers who like are sitting in their chairs, working hours and hours every day, and often in high stress situations. It’s just really fulfilling. So I’ve been here about three years, three and a half years and our team works as as an internal agency really, for the entire organization of Rackspace. So we’ll do pretty much any kind of research for anybody who asks

Steve: That’s a nice position to be in or not? I don’t know. We’ll do research for anybody who asks you, how did that come about?

Laura: So within the organization, we were we sit alongside design and where we work with them and and we have some close partners with a content team. So we’re like always at the ready for them. But that ends up being just a fraction of our work. We will, will work for product to the any of the product managers or product teams that need to want to market test their product ideas, and the directions that they’re trying to go or to prioritize their products to discover demand in the market, what’s going to make things easy and best for them. And then we’ll do larger scale market research. We’re currently running a large scale market survey, because it it lets us flex to whatever questions that that any of right anybody in Rackspace has that is that where there’s unknowns and they can’t move forward to make a decision that we’re all about that we’re our mission statement is to spark fast, confident decisions. And so that’s where we come in. And sometimes it’s just to confirm unknowns that certain teams are having, or to break ties. We break ties a lot with the data and and also to support business cases based on what’s best for our customers and our users.

Steve: Stay over the 24 years. I was gonna ask about this later, but now you’re making me think like the arc of, you know, your, your experience in the profession over 24 years, but also your observation of what the professional looks like. you’re describing an interesting state. I think, you know, this this mission statement and the The positioning and the interest and the provision, I guess, to help the whole company kind of wherever this, this, you know, critical decision, information is needed. You’re there to do it. You know, how, if you look at that positioning where you are now and kind of what you’re able to provide? How, how would you describe the arc of just this one part of it? Like, who and how do we work with and what’s kind of how are we? I keep saying positioning, but but I guess that’s really what it’s about. How do you think that has evolved over 24 years?

Laura: That’s a great question. And I really I get really excited about this question because I see this as one of the possible futures of our profession. So we we began and even the DNA of the team I currently lead began holy just like as design testing kind of organization at the beginning of my career. It was really a unicorn position. It’s like there was the design and the research. And I, I like I have a special like passion for the research part because I have degrees in sociology, anthropology and psychology. And so I just love love love studying human beings and I love research methods is like finding that that question and applying the best method to it. So as the the need has evolved, and William say as the field and the practice have evolved as a as the need has evolved, for answers from people who begin to be seen, as reliable, as valuable as as somebody who can really help and help get the right answers, the freedom to apply the best method possible. Is is what’s so exciting about how we’re working today and I see this as that potential future one potential future for our field is the opportunity to drop it having to everything having to be specific methods and having this one toolkit that’s just user experience and open up to the broader possibilities of any kind of method that’s going to help deliver the best experience in the end. And and I’ve just having watched that happened and I also just, I landed in rather a dream position here at Rackspace in that I, I inherited a team that was very deeply educated, there’s masters and PhDs is is what we just happen to have. It’s not what scare anybody about that. That’s what you need to practice in the field because that’s not accurate, but because we had a lot of deep experience in a lot of different kinds of methods and the ability to understand applying a finding a best fit method for Whatever question is out there and whatever way that you need to prove that are convinced it’s true. It’s just really opened up our possibilities and it’s also accelerated our cadence. Because we don’t we get very little pushback on the back end once we’ve delivered our results because we’ve chosen a great method at the beginning that actually gets to the heart of the questions.

Steve: So in this this futures, that we’re talking about, where do these methods come from?

Laura: a broad range of of research capabilities for me, they came from my education in sociology, anthropology and psychology. So, all of those different approaches such as being able, the very fact that you can know how to ask a question well, means that you can put it in multiple different formats. Perhaps it’s a large scale surveys, because you need, you need a you need a shallow data set that just has a whole lot of yes and no answers upfront. Or maybe the best way to answer the question that you come up with is to have a deep qualitative interview where you generate a large number of, of what I call them vertical data points, which is that you create these deep, deep pits of thousands of unstructured data points. So, gosh, don’t remember did that? Did I answer your question? They’re asking me again.

Steve: Well, I’ll say I mean, you started to I mean, it’s where did the where did the methods come from? And I think your answer starts to beg the other question, as was what sort of the granularity of the phrase method? You know, are there millions of methods are there eights like? What is that what’s underneath the term method?

Laura: Wow, that is such a great question. And now I have to like take my take an expert mind and try to quantify it because it’s Something that after all these years I do rather naturally. And so I don’t know if you’ve read Malcolm Gladwell blink, but it talks about, experts can’t always put a word on how they do something. So let me see if I can parse that out. What do I mean by method? So we’ll take the problem, what’s the problem to be solved? Like from a business perspective, what’s the business value that needs to be delivered? We look at that from a project perspective. And then we then we drill down on that, well, what is holding you back? What are the unknowns holding you back? And maybe they just they need to know what are like to get some sentiment and they just need some words on some sentiment. Maybe they need to know about does this workflow work or does it break down and that would be qualitative tests where you’re watching people or a remote test, where you get a lot of data where you’re gonna, like recorded online, and you look at multiple people having done task testing and that sort of thing. And then maybe it’s, look, our CEO just doesn’t believe this because it it only has a few people and and he’s just really wanting to see the numbers. And so then we would convert whatever questions and methods we have up to. Okay, let’s, you know, let’s we’ll, we’ll do a survey, which by the way, surveys are hard. I think it’s one of the hardest kinds of research to get right. And it takes the longest it takes the longest to get it right before you release it, even though it’s fast on the data collection end and the analysis end. So does that begin to get at what you’re asking about methods? does it begin to codify What is it my head?

Steve: Yeah, it was. So maybe I can reflect back a couple of things. And then maybe I’ll try I’ll try asking you a question with hypothesis, which we might call leading, or you might call good interviewing. I don’t know, we’ll see how we go.

Laura: Sounds good to me. I love that you’re asking this by the way, love it.

Steve: Cuz I feel like the when you started off talking about the team and the, you know, the deep level of education and your own backgrounds, and then even you saying earlier on, like, Look, we do market research, we do all these things that sort of sometimes exist under other umbrellas. It seems like, you know, approach a in, you know, building up a super set of methods is look at a range of disciplines, whether they’re professional disciplines or academic disciplines. And you have, you know, your set to choose from just gets larger. I guess the, the leading question is, is was to ask what you think about inventing methods, generating new methods.

Laura: Oh, my gosh. Absolutely love that. And and it’s really necessary to this approach and and can get you like the best possible thing. So a great example of that recently, we we did a study where that we were studying a concept of domain trusts, which is where you have ad administrators have to hand over the permission to do administrative tasks to other people. And so what they wanted to know what would people trust this with? Would they would they be willing to do it? And so we, the, the original method, and the the, the, the stakeholder who came and asked for in that case was very adamant about how they wanted to do it. And in that case, we went ahead and rolled with it and then they will also wanted to like do some of it themselves and so we part of how we scale it. In the organizations that sometimes we will also mentor DIY research. And this person has done some sessions. And so we were willing to do it. So we went with it, but we weren’t sure that the task testing method was going to get to that underlying question of trust. They believed that it would come up naturally as they were going through and that if they really questioned it, and it was kind of a way that they thought it was a way to fake them out, if they really questioned that, that these system administrators would trust somebody else to have the permissions and then not just trust somebody else, but somebody else is somebody else that they might not know about. So like, I pass it off to to Joe and Joe passes it off to Mary, but I don’t know that Joe ever passed it off to Mary. And so so that that was the original method and it was one of those cases where it failed miserably. It looked it on the surface. It looked like they all trusted it. But it was very clear that they didn’t quite get it. And by showing them the interface in the interfacing eat seeming easy, because it was beautifully designed, the workflow was great. There, it was misleading. And we were quite sure that something was missing. And they came back and said, Well, this isn’t they didn’t answer the questions we needed. So we went into a room and we just brainstormed and thought through, okay, how can we get at the psychology of this trust thing, without showing them something tangible like or having them work through a tangible workflow that that’s beautifully designed to this easy so that we’re not testing the design. And so suddenly, we started, like mapping out on the whiteboard. And we came up with this method of creating diagrams of people and these apps. diagrams of people and messages and connections and created these stories around it. And we had them put themselves in the story. So we would show like this one box and there’s a little icon and it’s them and we put their name on it as they told us their name. And then we said, okay, who is somebody that you might turn over permissions to, that you might allow to? And so they would say that and we built out the diagram as they talked like, Okay, well, here’s these other icons, and these are, these are these other three people, you’re going to turn it over, so like your level B people. And so we went from there and continued the story and we had them tell us a story of their own organization, we would live map out the organization as they talked about it. And so then we got down to the really down to the heart of it. And we we had, like person number five in their in their first first degree connections. Say Jay Okay, you’ve Turn it over to them. Now they’ve turned it over, they have these, these now, third degree connections from you. And they’re turning it over to them. And they said, Well, that’s probably okay. Because they wouldn’t turn it over to anybody that wasn’t trustworthy and this and that. And they then they said the key word, and I, and I assume I would know who that was. And I’d be able to see who that was, because then I could go back and check and see who that was. And that was the key. That was the key that broke the whole thing open, which is that the truth was, they were not willing to have the mechanism work in that manner. Were without without their permission, though, that was another piece that they would often just volunteer and say, it’s like, well, I’m assuming that my first degree connections will come back to me and say, Okay, I want to grant permissions to this person, or you’re going to notify me that they’ve done that and tell me who and and we blew up in the whole model that way and It turned out to be exactly what they needed. So that was like live creating a model. And that we actually did live in an abstract way through the combination of these diagrams and the storytelling.

Steve: And were you showing the this beautifully designed interface in this round of sessions?

Laura: We, we did it at the very end as a like a bonus piece, like, okay, now given that, and then what we did was we said, okay, now assume that those problems were solved. Now, let’s experience this. And so we that we just got extra data for them about the about the design in that process.

Steve: Can you say a little more about, you know, going into the room with a whiteboard and thinking through how like, you know, the process of creation of this new method? You know, and maybe that’s another unkind question for me, but I don’t know. Can you reflect on that process?

Laura: Not at all, not at all that that really begins to get to it. So this is where you have some understanding of how a human is built and how a human works and thinks just really works and comes in handy. It’s like and then also understanding one of the key aspects of, of any research design in any research method is being able to, to know that you’re measuring the right thing. And so how do you get to that you’re measuring the right thing and you’re getting at what the actual human behavior is going to be. And so that particular day, it’s like what we were looking at is like, okay, we knew that showing the interface was getting in the way of getting at where where’s that fear button in there? It’s like we were trying to see Is there a fear button in there and have we pushed it and so it this was, this was a combination of like the the minds in the room that understood how human beings work. We have a couple people with Human Factors degrees, and and then others are just, you know, have experienced so much experience in the user experience field. And then just some common sense is like About what? what would what would get at that that fear that concern that which is what we’re actually trying to measure. So that’s really the big question there is like, what are we trying to measure? Looking at your various methods and saying, Okay, if we use this method, whoops, this thing might get in the way. Or if we use that method, this thing might get in the way. Just like when you do surveys, the order of your questions can change the results that you get. So there’s one bit of guidance that I like to apply, which is that in a survey, you don’t ask the demo, or if you can avoid it. You avoid asking the demographic questions up front that have you telling them about themselves, because that can actually bias how they think about what they’re getting ready to do because you and they have Suddenly put them into a certain box and they’ve stepped into that point of view, you can get us may be able to get a slightly more pure response. If you if you just dive in and do the, the the other the basic questions that you’re trying to get at first and then add the demographic on at the end. So yeah, that’s that’s some of how we do it is make sure that that you’re getting at what you need to get at and that whatever you’re doing is not getting in the way of getting at what you need to get at. Mm hmm.

Steve: And then part of the story, I think is, you know, is your collaboration with the the other your stakeholders, your clients, I mean, however you kind of label them who, who have something in mind who have a goal in mind for this research. And so how do you how do you have that conversation? You know, maybe if we, you just told the story, but like, maybe we could talk about what happened before the part of the story that you’ve told But how to how to projects like this get formed.

Laura: That that is absolutely one of my favorite favorite things. So after many years of pain of, of studies like that, that again is a really good example of one where this happened and that that we didn’t do it. We didn’t set the foundation well up front. For, for both what was really needed we believed we’d gotten it that but turns out as we had the conversation after the first study failed, we we we went back to our disciplines conversation. So after some years of having those kinds of experiences in my career of studying, not getting what they needed after we did all of this work, is that I now ask I have a series of questions that I ask up front, it’s a weakness as we call it, our research intake process. We do it in about 20 minutes. So anytime somebody comes to us for research Was it? Oh, that’s great. We’re so excited about what you want to learn. Let’s have a 20 minute talk. And that’s we keep it really simple and lightweight for them. And there are six areas that we ask about and drill down on the first really being the most important one. The first and most important question, and I’ll drill down into each of these in a little more detail is what problem are we trying to solve? So a lot of times they’ll, they’ll think that they have one thing that they need, and turns out there’s another so what problem they’re trying to solve? Who is asking them to do it? Who are they accountable to? Because that can change the questions too. Are there any constraints to the to like the solutions or anything? Is there? Is there something that that’s in the way is like a technical constraint? Or is there a business constraint? And then are those negotiable? Are there any risks? What do we know or already have because sometimes they don’t need a test at all because they already have it. And then what is the timeline on this? And are there dependencies in that timeline? So, so those are those are our six, what What problem? are we solving? Who’s asking you to do it your stake your stakeholders? constraints, risks? What do we know? Or have? And what’s the timeline?

Steve: Yeah, that sounds like that’s 20 minutes to, to cover that.

Laura: Yeah, we’re really good at it now. I think our first ones maybe took an hour. But we’ve really got it down. Because the first one we are able to get down to, really to the heart of it very, very fast. And it’s the single most important if we get to nothing else, the one that we have to have in terms of a conversation that you can’t just ask over email, you have to really hold their feet to the fire is that what problem are we trying to solve? So it will sometimes have people come in and say, well, we need to deliver this new ticketing application or we need to make improvements to that ticketing application, or we need to test our design. So that’s not really the business problem that you’re trying to solve. That’s, that’s what you’re wanting us to do is is test those or, or that’s what you’re going to deliver. But what is the business problem that that application is trying to solve? And so in this domain trust case, it’s like we we needed to get down to what their real concern was not was, would they do this? Would they be willing to do it? And you know, is does this workflow work? It was more, does this frighten them? Is there something about this that they’re worried about? That could go terribly wrong? And then so we really have to hold them to to that on the way out, and what business value does this deliver? What are you trying to do? Who are you solving it for? So there’s kind of three questions and what problem we’re trying to solve. what problem are you trying To solve, that’s the that’s question number one, who are you solving it for? So are is it for a certain kind of customer or an internal person or a certain kind of user? Who’s the what’s that persona? And then what is the desired outcome? What do you hope will happen not as a result of the test, but as a result of actually, like releasing this thing that you envision that you want to release out into the wild? So if we do nothing else, we’re really going to get them down on that. And they often think they can do that in a in a one line email, and then they find out in the conversation that it’s not quite that simple. I think one of the best compliments that we’ve had is that we have one, what one repeat customer, he’s a product manager who comes to us all the time. And he said, Okay, so I need you to ask me those questions that make product managers cry.

Steve: Is that just take those All these questions are is where to where’s the tiers for them here? That problem one seems really interesting.

Laura: Yes, that’s definitely that’s the first and biggest. And if we get to nothing else that one can deliver all sorts of value, then we’ll also you know, then we also get them to this, the stakeholders who are who’s asking you to deliver this because what they really need can often change with, they may be driven by something external. One of the frustrations that we sometimes have, when we don’t ask this question is that we’ll go and do this and deliver it. And they’ll go a certain direction, but really, they or they don’t believe you or they won’t take our they won’t, they won’t. They won’t take our findings and they won’t do anything with it. And we find out after the fact, oh, their boss asked them to do this specific thing. And so if we know both who what, who they’re accountable for like with this product manager, it’s like After a while, it’s like I got to know who his boss was, or whether the his boss or whether the engineers were asking him to do this thing. And based on those two things, we could structure the study where it would actually get the answers that would convince that person. So we actually do that skip over of our stakeholder by asking them who’s your stakeholder, and then we arm them with the data that will help them be convincing to their other to their person. So that’s, that’s really another layer.

Steve: And you mentioned this, but I feel like it bears reiterating that. I mean, none of these questions or anything about the how we will approach this and, and, and I think, and I think you said this as well, that’s often the starting point for people who approach you, right, we need to do step x, or task y. And you’re, you’re setting that aside and asking a lot of very specific whys and how’s and whens and who’s That’s correct.

Laura: So yeah, this this is all developed not just out of pain in user experience testing over the years, but also out of having gotten to be a program manager and being able to think from that space and in various times in places where I had engineers as well as as user experience and other folks working with me and for me, and and so it was really this this concept of the, who are we accountable for? What are the constraints? What are the risks that that sort of thing does actually come straight out of business management experience? But now we’ve armed researchers with Yeah,

Steve: I can’t help but think that there are, you know, either small teams or you know, research teams of one or just lots of researchers who their role is very much To take assignments, and that may be a methodological assignment or problem assignments. And you’re describing, you know, a context for research at Rackspace, where that dynamic has, has changed entirely and you’re there to, you know, I guess work as a partner and, you know, former foreman approach together, you’re guiding them, you’re facilitating, you’re not being directed. Go do this. So we need it tomorrow. If I’m characterizing where you’re at, right now, I’m wondering if you have thoughts about, you know, how does, is there a way to get from A to B, like, you know, if A is Yeah, delegated assignments and B is facilitative, empowered partner who’s you know, who has a mission statement? What is what does it look like to go from one to the other?

Laura: So, so we have found that even our most beginner, Junior researchers are able to have this conversation because one, it’s kept very short. And we don’t reveal these things. It’s like we asked them more from that empathetic perspective, like Rao, what do you what do you need to do in this? And what do you know about this? Just as a discovery, it may not change a whole lot of what you do, but it can provide really valuable context for what you do. So somebody may have asked you here, can you do a usability test on this really quick for me? And, and yeah, we just need you to run this and run these workflows to go ahead and ask these questions up front. It really builds out the goals and objectives sections of your test plan, so that you get to a real goal of what is the ultimate goal from of this thing that you’re delivering, it could actually end up changing you Your script and how you ask those questions. So even if you’re a very frontline researcher who’s just doing what people asked this, this can really inform your your test planning and give it great context.

Steve: It’s interesting to think about that case, you sort of not, like you said, You’re not really revealing how this works. It’s not Well, before we agree to do the usability tests, we’re going to we need to ask these questions. It’s it’s kind of a yes. And that’s great. Tell me Tell me more. Tell me more. Tell me more.

Laura: Yes, it’s taking what we do from a we asked great questions, perspective and from but from this this at the beginning planning point of view and place In the fight in the study,

Steve: yes. And would there be a scenario where, you know, this frontline researcher is being asked to do usability tests, ask these questions. You know, might they identify a different approach? And if so, what would they do?

Laura: Absolutely. And so this is where you really get to shine is where and and, you know, because a lot of what we’re about is keeping people from making really serious mistakes and delivering things that don’t work. And so this is where you really have that opportunity to, to guide encourage mentor your stakeholder, even if they’re bigger than you by by saying, okay, so if we, if we do that, that user experience test, here’s what might happen is that you would get this and these kinds of answers. But if we were to test this this other way, and it’s good to kind of build it out. First thing and get an idea in your head, we don’t typically reveal the method at that time, we typically have them know the method within about an hour after this after we’ve said and thought about it. And sometimes we know instantly and we’ll reveal it, but rarely do we actually reveal the method in that situation. We’re taking in we’re being a sponge, and we’re encouraging them to think in a different sort of way. And, and, and it’s only it’s then when we’re looking at how that might work and and how it might have a method that they were asking for isn’t going to quite work. And that happens a lot like when we have people come to us say, we need a survey. And we we did there were times when we used to just say well, actually, you probably don’t need a survey, that that didn’t work, telling them that they’re wrong. And redirecting them without context didn’t work. hearing them first. So by asking these curious questions and diving down, honestly, we found those people really enjoy the conversation. Because when they get to talk about their work and their challenges, who they’re accountable to, and what success looks like for them in this project or this effort, and sometimes they’ll actually walk out of that those just those few questions, thinking about their own work a little differently. So that’s the first part is that we’ve listened and we applied empathy to that, and we’re very kind when we do this, this experience experiment and this experience and so that’s, that’s really the first foundational piece is that we’re, we’re giving them an opportunity to speak about things that they nobody thought to ask them. And so that’s, that’s step one in the success measures of this and then second is that then if we develop another method, that that that we then go have another conversation like okay, so looking at this, it looks like you’re This is going to get you better what you’re going to want and they may or may not buy that No, I just need this, this run like this. It’s like, okay, run an experimental session, run a pilot session with both and see how they work. Sometimes the stakeholders are right. And I think we’re, we’re more on point most of the time simply because that’s, you know, that’s our superpower. We were the were the methods people. But there are times when, when we thought it needed to be a certain way, and we were wrong. But when we are more effective when we’re delivering something more effective, we can show it, then we have this opportunity to do it. In that particular case with our domain trust case study. It was simply that we went back and told them we didn’t say, okay, you all screwed up by saying, making us do this method and all of that we actually took it on ourselves. We said, Look, we we heard you, we believe that that maybe there should be a different approach. We and we didn’t follow that up strongly. Would you be willing and we’re going to try this other experiment, we’re going to go do these other studies, and we’re going to see if we can get you exactly what you need. We’ve developed this other method. And so in that case, they felt the pain and they were willing and by the time we got there, they were like sponges with with with our data. And yeah, really answered the question. So that’s a that’s a Those were the steps, which is that empathetic listening upfront and really using it the conversation as an opportunity to discover and to hear. And then, and then, if they’re insistent, go ahead and do it, perhaps run a pilot of the alternate method and see if those look different and if that gets them closer to what they want. And then if there is a fail, then you have the new method in your back pocket and kind of apply it at that time.

Steve: You know, I have this not particularly brilliant theory, but I guess this observation that user research, given who’s in the profession, how it works. It’s a helping profession. You know, the way that reference librarians are or you know, some medical people or something, and I mean, I hear that in your story that that, that this is not about being adversarial, towards whatever client stakeholders people in other departments or disciplines, you’re, you’re really approaching them to help them and it becomes you just it’s framed very collaboratively. It seems to me that, that starting point of that kind of philosophy, like you said, of being very kind and using the empathy and we use those words in research all the time, it sort of gets thrown around a lot, but when you describe how it’s implemented and how it’s sort of a philosophy to live by and work by, it’s really exciting to see how that creates a working relationship that doesn’t have a lot of the a lot of the challenges a lot of researchers raise in their own Work, you’ve kind of created a culture or context of cooperation that just is flavored very, very differently because of that.

Laura: That’s That’s right. And thank you for seeing and recognizing that, that that is how that works. And we actually have we have one more like, one of our superpowers in our toolkit for that, is that after we had a big push back situation, where product managers got really upset by our results, and then we found out that that he was under other pressures. And that was that we didn’t even know about and the fact our results like they were correct, and they were right, and they needed to be applied and used and they knew that but it they they just had a real reactive situation to it. And in that case, this this is on another project than that one. In that case. We really see sat down. And we did a post mortem ourselves about that interaction and how that there was some unfortunate Bad Blood really created between us and that, that person and and he was just really wounded for a while. And so we, we set but we were bright, but we were right about it, they and they needed to do it and they and they have to do this that they and they did they applied the results. But still it was like we had that relationship fail. And so we we wrestled with this for some hours and an offside over my kitchen table one day. And we finally came up with so we were what we’re doing is like coming up with with core value approaches that we could apply to to solve a situation like this and to respond to it. And and even before that, how could we seek from not creating it in the first place. And the big thing was, is that we wanted to do that without giving up our power, our authority. as researchers and our knowledge, so it wasn’t just about caving and saying, okay, you’re right now our data is wrong or whatever, or you can take it or leave it or whatever it is that that we might approach. It’s like, Okay, what can we do in a situation like this that’s proactive without giving away our authority. And the two words that we finally came up with, which are our watchword, our bill, whether we’re be curious. Be curious. That is one of our research superpowers is that when a stakeholder is pushing back on our results, or on our method, or anything about that, we might start getting defensive on the inside. And we will stop and recognize that it’s like, okay, we don’t want to give up our authority as the methods experts as the data experts as the findings, experts. But in that moment, what can we do? It’s like, oh, there may be Something more going on for this person or in this situation that we simply don’t understand. And there’s a reason why they might be pushing back in that way. And so we turn it around into a question. It’s like, wow, that’s it, you know that that seemed hard for you What? What is that about that? What are you afraid will happen? Is there somebody that might come back at you or push back on you? Is there a deadline you might not make? Is this going to make things more complex? So we we pause and and seek to be curious, and that has made a huge difference in our stakeholder relationships. So even the most basic researcher can be curious about the people who are asking them to do something. It’s a simple tool.

Steve: That’s really wonderful. It reminds me of so many of my own experiences where I have an interaction that is weird or uncomfortable or unpleasant, you know, even a minor thing and I might Hold on to it for a while and then try to have that realization that it’s even though I’m experiencing it, it’s probably not about me. Right? There’s something going on. And so, I mean, I think your team is, has got a really lovely framing for it. And then we can take the opportunity to answer that question like find out be curious, what is that thing that’s going on to, I would imagine there’s some self care in that because it sort of proves that they don’t hate you, or they don’t think you’re stupid or whatever the thing we do to ourselves when we get those signals, but also back to your kindness and your framing around helping if you understand what the issue is for that person, or even giving them the chance to articulate it, maybe they haven’t articulated it before. you’re providing value in every direction and a number of different levels with that with that mission.

Laura: Thank you. And that that that is correct. It’s like having that, that opportunity to speak about those. And we learn so much more about the business and about how it works and or about the technology and how everything works or the product direction by doing this. The other element of that that was starting to talk about a little bit was that retaining your own power in this. So we also in that same offsite session really looked at, okay, how do we retain and keep, you know, so yes, we’ve listened, we’re curious now. And the data is still the data. It’s like the answer is still the answer. And so there’s a couple of things that we do is that one, we’ve now taken the heat off of that by by using this phrase, the data so the data is telling us, the data is telling us and so instead of saying Yeah, but the users are saying this or we found this in the study, and this was this was the the output and whatever it was that and really starting to get our backup about that. We we use that more generic and in really, like research based answer of, yeah, I hear you with all of those challenges and things. And this will be complex. And we just want you to know that the data is telling us this, the data is telling us this. And that way, it’s it’s agnostic, and it doesn’t say you have to go do this. It’s like we also the third thing we do is we treat them as grownups who are accountable for their own jobs. And so the ultimate decision on what to do we leave to them. And so then the final thing that we do other than So, you know, saying this is what the data is telling us, and then leaving Leo letting them giving them the courtesy of making their decision. And and living with that, whatever it is, the last thing that we do is we retain our data. And we will, sometimes we will even take special care after a study is done on one that somebody is pushed back on. And we’ll make it even more attractive and pleasant and more complete of a report. So even if we’ve done like just a quick turn report will actually turn it into something more formal archival. And then we save that. And I’ve actually had as a situation with this actually happened more than once. But there was a large one that for a large corporation that I was consulting with. And this was a large scale refactor of the main navigation of their whole site. And it was intended to increase click throughs to their product sets. And we tested this and we actually traveled nationwide tested, you know, 18 year olds to 80 year olds and, you know, all walks of life and everything. And so we gotten all this stuff. And the data told us that there weren’t these critical fails in this and that it was it was going to fail. And they were they were on their timeline. There were business reasons they wanted to release it. And so they they rushed and did it. So, you know, we told them here’s the findings. We let gave them the courtesy and and here’s the bad things that might happen. We gave them the courtesy of making their decision they did. And then they went back to look at the at the analytics and discovered that not only did click throughs not go up by the 30% that they estimated. The click throughs went down by 30%. So something had gone drastically wrong. And they spent like the first 18 hours or so scrambling with technology and all sorts of things and all sorts of finger pointing and fight since then somebody said maybe we should ask research and so they came back to research and they said Do you have any ideas about what may have gone wrong? We said yes, actually. So we we documented in this report. And here are the things that that were risky that that you can look at. And they looked at those as like, well, there was some dissent in the organization about how did these get through, we didn’t go there. We just gave them the data, the information, we helped them interpret it helped them come up with designs to fix it. And they did fix it. And within 36 hours, the click throughs had not only corrected, but they had gone up the original 30% that was intended for the project. And we never went back to that we never said I told you so we lead with the data says tells us this, gave them the courtesy of making the decision and then coming back and being ready in case the risk the bad thing does happen. That that we we gave it to them without judgment and and say yeah, here’s how to fix it. said this is this is not only very humane it’s great business

Steve: there’s something here about you know, when are the moments when people are willing to hear like we are we are sometimes change agents we are sometimes telling people things that yeah the break their worldview.

Laura: Yes, that’s a great way to say it.

Steve: So, you know, you can sometimes you can choose your moments and I mean, in that story, you couldn’t really choose your moment it was kind of the moment came to you and you were you were prepared to respond. But your earlier your earlier attempts to to influence and sort of change minds. I mean, really didn’t didn’t land but you were ready when the when it came back around.

Laura: That’s correct. And most of the time where the the research results are supported, And the bad thing does happen. And then sometimes we you know, because we’re intentionally observing this, and we’re really going for high quality for our customers and end users. Almost every researcher has discovered that there are times when it maybe that bad thing happens, but it doesn’t have as big of impact as we thought it was. Would if they didn’t, that was with somebody not fixing something. And so sometimes, yeah, sometimes that happens, too. And, but most of the time, that’s, that’s what we’ll do. And actually, we we do find because we’ve had that in tech conversation up front. We, we have the tech conversation upfront, we review test plans with our stakeholders before we run them, and we invite them to the sessions so that, you know they’re in there too. We’re very transparent. We even put all of our working documents in shared directories, even our working reports and things we don’t hide anything from them. Most of the time, they don’t go look at the details, but the fact that we’ve made it transparent is created a lot of trust. And then ultimately, you know, sharing the data with them most of the time, they’re convinced because we’ve set up such a good foundation up to there. And then when they’re not we have this failsafe mechanism for for how to handle when they don’t take the advice.

Steve: In US, you said a couple times, no, you’re treating them like adults, the decision about what to do or not to do is theirs.

Steve: And I think you’re surfacing. You know, what is what does success look like for researchers? And you’ve, I think you’ve articulated in a specific way that I haven’t necessarily heard that there’s a lot of there’s a lot pressure we put on ourselves as a practice to have influence in you know, in that word means so many things, but one of those is people doing what we think they should do. But saying, oh, that that they’re adults and that they’re going to make their own decision and there’s factors that we don’t know. right i mean even your mission statement about where you use the word spark is kind of what you’re doing. I mean spark spark plays a certain role in a fire being led or not led but it’s it’s right there kind of outside to make it happen but it’s not the fire right the fire is the is the next thing that happens.

Laura: Yes, I’m so that’s the spark fast, confident decisions, because that’s what we’re about. And so that’s actually another the fast part is another element of the method is that sometimes we need a faster method to to get at at the same sort of thing. So that would be another consideration. But ultimately, yeah, it’s like we’re trying to find where is the unknown in the audience. is keeping them from moving forward and moving forward to the right thing in the right direction. And yeah, once we’ve discovered that we, by setting it up that way up front, and having that in mind that, that we have more success influence on the back end.

Steve: When you spoke earlier in this conversation about, I think bringing tools and mindsets from business management to research, because we’ve, that’s what we’ve talked a lot about, right is sort of the management of making research, effective and successful inside your organization with all of your colleagues.

Laura: Yes. So the two pieces I brought specifically from business were the constraints and the risks, because those are two things that you really have to well and then what is the business outcome when we ask about the desired outcome in the problem solving conversation, and question we’re going for what is the business outcome, like no, not that just that we need to release these ticketing changes. It’s like, Why? What do you hope will happen as a result of those? So that’s one aspect is what’s the business outcome? Or the organizational or whatever it is that, you know, that’s driving the outcomes, the bigger ones? And then the constraints. It’s like, what are the technical business constraints? And are they negotiable? It’s like this is that that’s comes basic, from basic business management that really anybody can begin to can learn to understand and begin to understand is that there are sometimes there’s just you know, there’s there is a hard schedule or budget constraint or legacy technology constraint. We all hit that with a lot, don’t we? But then then the question to ask is, you know, not not just to say, well, that’s just going to work and you can’t give them what they want. Before you go there, you say, or, you know, give them what they need. The users are going to have to have this bigger investment you say, okay, is that negotiable And to what extent is that negotiable, that technology or that, and then that you, you begin to get them thinking in those other directions. So then then the third one, then is is the risk. So I use a basic risk management model, which risk risk assessment management model, we asked what are the potential risks of the of this solution? What is the likely you know what bad things will happen? So, that’s a simple way to ask it, what what bad things might happen if this gets released? Because a lot of times when they’re asking you to test something, they’re not necessarily thinking about the the potential implications of it. Or maybe that’s what they’re actually trying to get at in the test. They just didn’t really say it. It’s like, we’re afraid that this workflow will, will break down at this point. It’s like, well, what bad things will happen? What’s the risk? What, what’s about bad things would happen because they don’t always articulate that part. They’ll just say that it won’t work. And then Okay, what’s the likelihood of that is that really Something that would happen, is that a high medium or low likelihood that it’s going to happen? And then what? What would be the impact? It’s like, okay, yeah, that bad thing might happen, like, there’s this usability problem, and it would be there, and they’d have to work around it. And they would still have another way to do it. So that would be a low impact. There’s not a big, yeah, the bad thing might happen. But the bad thing that will happen as a result is is really not so huge. Or they might have a data loss or the system will crash or somebody will die like that those those would be really high impact. So once we have that high, medium, low likelihood, high, medium low impact, and we know what that risk is, then we can have that conversation. How do we mitigate that? So how can we mitigate that and sometimes I’ll actually change their solution live while they’re talking to you simply by asking these questions. And sometimes we even ask ourselves these questions about our own studies. It’s like is there a risk of this study so so we asked the the risk The likelihood high, medium low, the impact high, medium low. If both of those are high Oh, then we may need to add more participants cuz, because if this is a really risky thing, we need to find all possible capability reasons that are possible failures. It’s like maybe the first 15 people won’t reveal it. So we need number 16. So if it’s a really high high impact that that’s one of the decision points it tells us is we need to add more participants potentially, if it’s a really low impact, sure, we might be able to get feedback from six, we might and we only get feedback from three or hallway testing if it’s a low likelihood, low impact. And then how can we mitigate that? How can we fix that? So? So yeah, it’s it’s a great little conversation to have, and it’s not really complicated. Risk Management sounds like such a big, scary business II thing, but the risk, the likelihood impact and mitigation. It’s really rather rather simple. It’s so valuable

Steve: not just to step back a level a little bit, and we started off, you know, you talked about your, your, your degree in research psychology and you mentioned sociology, you mentioned anthropology and by the end of the conversation, we’re talking about business. Business. And so you know, for you in your career, like where did that come from? How did you pull that in, in addition to all the other things that you’ve kind of incorporated in from the social sciences? Where did the business stuff come from?

Laura: By taking opportunities that came along and by being curious, so that specifically for me how that happened was that I was in a defense consulting organization with the University of Texas at Austin, the Applied Research Laboratories, a very old and respected laboratory that started with the Navy and then expanded to other services. And there was a there was a pitch that was We got to do just people coming in. And I thought, well, I want to pitch user experience to them. And so I managed to weasel five minutes in with these people who were looking for high value, things that they could invest in toward this specific effort that they wanted. And have that whole day and the little five minutes, I was able to wrangle out of somebody else’s 30 minutes session, because I came in late to the game. They were lit up by this whole user experience concept, and it exactly matched on what they needed. And so because they had a need, and I presented it, I brought the contract and so I started asking my boss, how do I do this? How do I get a contract? What do we do? And so I started learning about the business aspects just because I went put myself out there. There was an opportunity, I’ll tell you, I was terrified. But it was it was like but I was also like, thrilled exhilarated like being at the top of the hill on a roller coaster, and the best way and So I took that on and and by doing that I began to, to have to learn that sort of thing. So that was the start of it. But the really big pivot was then when, in addition to that I, they they had a project that had a manager who left and they needed somebody just to pick it up really fast and keep it going. They thought it was going to die in six months, because it was not going well. The stakeholder wasn’t happy. So again, I was curious, it’s like, Okay, why is the stakeholder not happy? Well, it’s because things were not delivering on time or and it was like what all these requests for like a year overdue there were $100,000 over budget, and they just didn’t see the value anymore, except for this one guy doing little ad hoc fixes for them. So I started looking at their contract and seeing what they signed up for and discovered that, that we were doing a whole lot of things that were out of scope, but we weren’t doing the things that they signed up to And it was really simple way to begin to learn business and budgets and, and to that sort of thing was I was just curious. And I just started asking the question like we do as researchers, why, why? Why is why are they unhappy and I had empathetic conversations with them. And so when I when I started having those empathetic conversations with the people who were in charge of these projects, and even as a researcher start asking them well, so why are you concerned about that? What that what’s that about? What kind of pressures are there? That’s where I began to learn about this and in in, like, I go out, try to find, well, how do you manage this and this sort of thing, and I paid attention when they somebody did a risk study at some point and I was like, Oh, what’s that? So I went in and said, Can I listen in on your on your post mortem, you know, on your on that risk that happened, that bad thing that happened? And I that’s where I learned those those basic steps. Even though they weren’t basic in that situation, but because we’re user experience people, to boil them down to those basic questions, that was how I did it. That’s how it’s in this one tiny little part of this conversation and that really anybody can do it. So it’s like taking the best of who we are as user experience people, and especially as researchers, which is paying attention, asking questions and boiling down things to their simplest, simplest forms, and then applying those to make a difference. I’m really proud of myself. I’m glad you asked all these questions.

Steve: You know, we keep coming back to how be curious. It’s just as a mantra for good collaboration and for developing skills and you know, we we bump up against all kinds of stuff that may be intimidating. But you’ve, you’ve showed how to learn by being curious. And as you said, you know, paying attention and boiling it down. So all the skills of a researcher being applied to all these different contexts, it’s a really it’s a it’s inspiring, and I think but also very actionable. I really like it for that.

Laura: Yes. And then I, I’ve never just sat and done just what I was asked to do. It’s like I’m always looking for something new. Something else is probably just part of how I’m built but it’s also a conscious choice of, of just doing my my current job is it’s simply not enough for me and that even just that extra 10 minutes of curiosity or desire to see something else or learn something new while still doing my job and delivering on that, that that that’s just opened so many doors And, and and, you know, helped me step into a lot of opportunities and learn new things.

Steve: Is there anything we didn’t talk about today that we should

Laura: we could talk about all sorts of things. So that this this is a lot. And this is really a lot of the heart of of how my own practices developed, how I would love to see the field develop just in terms of, of one, you know, that that applying the best possible methods in being free to use any or invent them based on the real questions and to know those real questions to ask up front. And then how to treat people along the way. And grab the opportunities along the way, I think. Yeah, that’s that we’ve summed up a lot of my practice really, really well. Well, it’s, it’s really beautiful. As I said, I find it really inspirational. So thank you, Laura, for being so open and sharing so much good information. I really enjoyed the chance to speak with you today.

Laura: Oh, thank you so much for asking. It’s my pleasure indeed.

Steve: Thanks for listening! Tell your colleagues about Dollars to Donuts, and give us a review on Apple Podcasts. You can find Dollars to Donuts on Apple Podcasts and Spotify and Google Play and all the places where pods are catched. Visit portigal.com/podcast to get all the episodes with show notes and transcripts. And we’re on Twitter at dollarstodonuts, that’s D O L L R S t o D O N U T S. Our theme music is by Bruce Todd.

In this episode of Dollars to Donuts I chat with Colin MacArthur, the Head of Design Research at the Canadian Digital Service. We talk about bureaucracy hacking, spreading the gospel of research throughout government, and embedding researchers in complex domains.

Often the idiosyncrasies in people’s research and the sort of surprises that don’t fit within the template are the most important things that our researchers find. – Colin MacArthur

Follow Dollars to Donuts on Twitter and help other people find the podcast by leaving a review on Apple Podcasts.

Steve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization.

I just read the 2011 book “It Chooses You” by filmmaker and artist Miranda July. It’s one of the best books about ethnographic research that isn’t really actually about ethnographic research. In the book she describes a period of her life where she was creatively stalled in finishing the screenplay for her film “The Future.” As a way to either get unblocked or just avoid what she should be working on, she develops another project, to call people who have placed ads in the free classified newspaper the PennySaver, and arrange to come to their homes and interview them.

She reports on each of the interviews, including excerpts of the transcripts, and some amazing photographs. The interviews are sort of about the thing being sold, but because she’s getting outside of her cultural bubble, she takes a wider view, asking people about a period in their life when they were happy and whether or not they used a computer (since even in 2011 a newspaper with classified ads was a relic of a previous era).

These interviews are confounding, hilarious, disturbing, touching – everything you’d hope. And July is honest about what makes her uncomfortable, about her own failures to properly exhibit empathy when it’s needed, or her challenge in exercising caution in some dodgy situations while still being open to connecting with strangers. She incorporates her feelings about her own life as she hears from people about their hopes and their reflections back on their lives, lived well or not so well. She articulates her own judgements about the people she met and how that informs her current thinking about her own life and her aspirations for her future.

I can’t believe it took me this long to finally read this book, and I can’t recommend it highly enough.

Through Dollars to Donuts, I’m gathering stories about maturing research teams, looking for best practices, emergent approaches, and insights about organizational culture. This is of course highly related to the services I offer as a consultant. In addition to leading research studies in collaboration with my clients, I also help organizations plan and strategize about how to improve the impact research is having. Whether that’s working as a coach for individuals or teams, or running workshops, or advising on best practices, or leading training sessions, there’s a number of different ways you can engage with me. Please get in touch and let’s explore what the ideal collaboration would look like.

You can email me about this, or with feedback about the podcast, at [email protected].

Coming up soon is the first Advancing Research conference. Rosenfeld Media is putting on this conference March 30th through April 1st 2020, at the Museum of the Moving Image in Queens New York. I’ve been working on the curation team helping to assemble the program and it’s looking like a really fantastic event. I’ll put the URL for the conference in the show notes. You can use my discount code Portigal dash ar10 to save 10 percent on your registration. I hope to see you there!

All right, on to my interview with Colin MacArthur, the Head of Design Research at the Canadian Digital Service.

Well, thanks so much for being on the podcast. I’m really happy to get to speak with you.

Colin MacArthur: My pleasure. Thanks so much for having me.

Steve: All right, let’s just start with who you are? What role do you have? What organization do you work for? I’ll just throw all the launching stuff onto you and we’ll go from there.

Colin: Absolutely. My name is Colin MacArthur. I’m the Head of Design Research at the Canadian Digital Service. We are an office of the Treasury Board of Canada that works with departments across the Canadian federal government to improve the way they serve their constituents. We do that on my team using design research and by helping people inside the government get closer to the folks who they’re trying to service. We do that in partnership with designers and engineers and product managers and policy experts on interdisciplinary teams, but the perspective that the researchers bring is a hard and fast focus on the people who we’re trying to serve and what their experience with our services is like.

Steve: Why is this under Treasury?

Colin: (laughs) Good question. The Canadian federal government, or government of Canada is organized with departments and then with a number of central agencies. And Treasury Board is one of those central agencies. Its name is a little odd in that it’s not actually the department of finance. It’s not the Ministry of Finance. It plays a central role in sort of managing and consulting with other departments about how they run themselves. It’s a management board of government. And so we are in kind of an interesting place because from Treasury Board we have a view of lots of interesting things happening across government. We’re positioned kind of naturally to give advice and also learn from departments that we work with and then to work across lots of different departments in a way that would be a little more unusual if we were nestled inside a dedicated department itself. So, Treasury Board is sort of one of the central agencies of the government and that’s why we’ve ended up where we are.

Steve: Is that where the Canadian Digital Service originated?

Colin: That’s exactly right. So, we’re relatively new. Founded just a couple of years ago and we started right in the same place we still are, inside Treasury Board, reporting to Canada’s first Minister for Digital Government who was also the President of the Treasury Board.

Steve: What’s the relationship between digital service and, I don’t know, what would you call like regular service. Like the things that government does. Because you included policy in kind of the mix of people that you collaborate with. So, everyone else you listed seemed – design, engineering, research, PM – seems very – yeah, this is how sort of software is made. How digital services are made. But policy – and this is from someone outside government, so maybe it’s a naïve question, but policy just sort of begs the question for me, like oh what’s digital versus just services?

Colin: Yeah. What a good question. I think the way we choose to look at it is we’re interested in improving services, period. So that means the elements of those services that are online, but also the elements that are offline and drift into paper processes and drift into things that are more typical policy problems. But in this day and age it’s pretty hard to have a meaningful discussion about service in general without talking about the digital pieces of those services. So, when we put together teams to work with departments we absolutely come with a digital emphasis. That’s one of the strengths that we can bring. That’s one of the pools of expertise that we have. But we’re just as interested in looking at the non-digital sides of that service. And in reality, they all fit together and attempts to kind of separate them out into the website and the paper part are often pretty hard and don’t end very well for the people we’re trying to serve. So, we kind of view ourselves as tied up in both and we try to staff our teams with expertise that allows us to do that. That said, our name is the Canadian Digital Service and I think often our entrée is our digital skills. But we try to be more than that. We don’t think digital problems can really be solved by just looking at the technology piece.

Steve: That seems to me analogous with so many efforts to do transformation or introduce new services or new products – you know private and public sector around the world where – I mean you’re kind of hinting at the power of the like the construct of digital. That it invites maybe a different mindset in approaching a problem that can exceed beyond the boundaries of what is digital. Or, as you said, they’re intertwined.

Colin: Yeah, absolutely. I think we view the computer part as only part of the digital mindset and approach that we try to bring. I think departments appreciate that. I think that we are often able to look at a problem from different angles because we’re not just looking at the IT systems involved. We’re looking at all the pieces related to the problem they have.

Steve: So, what is your role specifically?

Colin: My role as head of design research is to first of all lead and coach the researchers on our team. So, I help them as they’re embedded in product teams chart the direction they want to take with their research, improve the quality and the quantity of their research. Make sure their research is kind of fitting into the product lifecycle in the ways that we hope. And also make sure they’re developing as professionals. We are happy that folks don’t just come here to build services. They come here to build skills and we do that with our researchers, like with all of our other disciplines. So, helping and supporting the researchers is one big piece of my job.

Another is to convene the government wide community practice around design research. So, we know that design research happens in places other than CDS. We know that research is a tool that lots of other departments are using. But what I noticed when I first arrived was that there were very few conversations between people doing research in different departments, or even within a particularly large department. So, I use my role and my limited additional time to convene folks across government of Canada who do this kind of work and talk about the challenges of doing it and talk about ways that we’ve worked through some of those challenges. The joy of that is getting to see all of the sometimes hard to spot places where people are doing interesting things. You know taking interesting methodological approaches, having new discussions that aren’t visible if you don’t get them all in a room and get them to start talking. So, convening that government wide community of practice is another key part of the job. And related to that is trying to create some tools and some policy change that helps that whole community move forward. We try not just to talk about what the problems are and swap stories on the solutions. We try to learn from that community and then use our position and knowledge to nudge forward policy changes or government sort of practice changes that can help researchers across government be more effective.

Steve: Do you have an example of a policy change or process change?

Colin: Sure. So, one of the central pieces of research regulation in the government of Canada is something called the public opinion research rules and guidance, known as POR. I’ll try not to use that acronym, but public opinion research is this kind of class of research that the government has a process for managing – very thoughtfully and deliberately over a relatively – I don’t want to say slow, but certainly a longer timescale than most user research or design research would happen on. And so one of the key areas of confusion we saw when we started doing design research with partner departments was wait, can you really do this without going through the whole process for public opinion research? Isn’t what you’re doing just public opinion research? Why are these things different? Why should we believe you that they’re different?

And so we went to our colleagues in another part of Treasury Board who actually own the policy on public opinion research, and we said to them, “look, these are the kinds of things we do. These are the kinds of questions we ask. If these are the kinds of questions involved is this really public opinion research?” And their response was, “if that’s really all it is then probably not.” And what was great was that we could then work with them to write a clarification to the guidance around the policy which really just meant like updating a webpage to have a section that talks about POR relates to user or design research and kind of further explain to departments that they could be doing user design research that didn’t need to fall within this typical public opinion research cycle. And that kind of guidance is so important to loosening structural challenges to doing research across government, right? That kind of guidance from the center is what helps people trying to do research in the edge of their departments make the argument to their manager that this isn’t quite as risky as they thought it might be.

Steve: The perceived risk is in doing the research? Is that what you mean?

Colin: Exactly. And I would say risk is probably a strong word. People say “well we have these rules for doing public opinion research. I don’t really know about any other kinds of research, so you better follow these rules.” And I think that what we’re trying to do is say “well there are actually some other kinds of research that are useful and not quite the same thing”. And think about those and realize that maybe what you’re trying to do falls within those umbrellas and thus doesn’t need to go through the same process and make management and the executive layer of departments a little more comfortable with that fact.

Steve: I think implicit, and maybe you said this explicitly, the kind of processes and procedures and things that would be required to do “design research” are less onerous than to do public opinion research. Is that correct?

Colin: That’s right. I think that public opinion research is – is I think often interested in policy and government level questions about public opinions, right? And design research is often focused on questions like what’s it like to interact with this department or this service? And those are different things that require different methods and different rules of evidence and so what we try to do is keep people from getting them mixed up in one another. Now I think if my colleagues from other parts of Treasure Board were here, they would remind us that there are some forms of design research that can kind of verge into typical public opinion research and then it’s important to engage the public opinion research process. But there are also lots of things like usability testing of a new service interface that are clearly not in the realm of public opinion research and therefore we’re really happy to encourage departments to do.

Steve: So, in this convening that you describe, sort of finding these, you also mentioned that there are maybe hard to find – I may be putting a word in here – surprising…

Colin: Yeah.

Steve: …sort of areas of research. How have you come to find those people and that work?

Colin: Well you know it’s funny Steve. I think as we were setting up the community of practice, I realized that recruiting participants for research was kind of the best practice I could have for recruiting members of a government design research community of practice. So, like when you’re recruiting people for research, you put out a call for the kinds of folks you’re interested in, but you also – you snowball, right? So, we started with people that we knew and we said hey, do you know anyone else that does this kind of work, or is interested in these kinds of questions and do they know anyone else and do they know anyone else. Often the third or fourth degree from there, you get to folks who are like, “oh, I didn’t even realize what I was doing was design research, but it is and I’m excited to find this group of people who’s all trying to do something similar.”

Steve: So, by bringing these people together, some of whom wouldn’t even have identified with the labels that we would put on what you’re doing and creating a chance to share and improve practices, this seems like it spreads far beyond what design research within Canadian Digital Service would be involved, right?

Colin: Right.

Steve: The reach is much broader.

Colin: That’s absolutely right and that’s the reason why we did it, right. So, our mandate isn’t just to build services with departments. It’s to continue to improve how the government delivers service more generally. And one of the ways we do that is through finding the folks involved in doing what we do and trying to enable them, right? Give them more tools, whether they’re policy tools or methodological tools. Whether it’s – give them a space to vent or give them a space to celebrate. One of the hard things, I think, about pushing something like design research within government is it’s – it can be hard to find a place where there’s a group of people like you who are really excited about the kind of work you’re doing. And so I think there’s some practical benefits of the community and there’s also some sort of emotional support that happens in the community of practice that’s really heartwarming.

Steve: Are there ways to find – this is maybe the counter example, or the other part of the set – so that teams or departments or groups that should be doing this, that maybe don’t know that it exists, or don’t know that it’s accessible to them, that it’s reasonable or feasible, but would help them in their efforts to improve the way that they serve the people that they’re serving? How do they fit into – I mean it’s kind of a boil the ocean question since you’re already finding everyone that is doing it, but sort of opportunities to build the practice, whether you all are providing that, or you’re enabling them the way you’re enabling the people that you’ve convened? How do you think about that?

Colin: Yeah. It seems like your question is really like what about the people that aren’t doing research and should? And they are also an important group. So, I’ll say a couple of things about them. I think that most major departments do have some groups of people trying to do research by some name, right? So, they’re trying to do client experience improvement, or your digital transformation group has a group of user centric specialists, right? So, there are people trying to do something sort of related to design research, at the least. I think what we try to do is find those folks and then we try to introduce them to the body of knowledge that’s specific to conducting research. Like we believe research is a craft in and of itself and it’s hard. And it takes some work, but it’s also accessible and something many folks can learn. So, with that in mind we try to locate the folks who are working in the generally related areas and inspire and equip them to do this kind of work, or continue to improve how they do this kind of work. You know we talk about the challenges to not only design research, but sort of digital best practice, at multiple levels. One of them is the policy or sort of structural level and we – things like our POR guidance clarification, the public opinion research clarifications, those certainly help folks, but they’re not enough, right? They – folks also need the skill to do the work and they need opportunities to learn. We try to create lots of informal opportunities to see what other people are doing. And then, you know, the bottom layer, beyond just policy and skill, is, for lack of better word, inspiration. Showing that it’s possible, right? Like showing that this work can happen within the government, within all of the unique factors of the government, and that it’s useful. And so I think we try to not just give folks kind of on the edge some policy tools, but we also try to expose them to the skillset and we frankly just try to show it’s possible and continue to cheer them on as they push it forward.

Steve: That’s good. I think we’ll probably come back to the ways that you’re working with teams and departments, but I want to go back to one of the other things you said early on. You were kind of describing the two main things that you’re involved in. We’ve talked about sort of convening the community of practice aspect here, but you also talked about leading and coaching researchers. You were really emphasizing building skills was almost a core value. Like this is a thing that you’re really thinking about being an outcome for people that work in research. Where does that come from? That is not a universally held belief I think in groups of researchers.

Colin: Yeah, well I think part of it is born out of the broader mission that we’ve been discussing, right? So, I think we are not just here to do research that helps create better services. We’re here to help the gospel of research spread around the government. And so in order to do that I think I need to care both about continuously building the skills of our staff and then taking those same tools and making them available across the government. So, for me it’s hard to figure out how I could say, “ah, broader government you should be building your skills in this way” if I wasn’t also practicing that as a leader in our own community. And I’ll say we’re not just interested in sort of helping folks build their skills, but helping – and helping researchers within our group build their skills. We’re interested in them learning about how to teach other people about research. So, I think we’re trying to build the empowering and enabling and teaching ethos into the folks that come into our group that makes it easier for them to interact with our partner departments. It makes it easier for them to have useful conversations across the design research community of practice. So, I think that from a mission angle is why that’s been a central part of how I view my work. I also just fundamentally believe we can all continuously become better researchers and that one of the ways to do that is focusing on skills building as a continuous improvement approach, not as a sort of set it and forget it professional development activity.

Steve: There’s often service design happening, of various kinds, without there being research happening. Even though we should all clutch our pearls at that idea, it still happens. So, at some point research is given a title, given a mandate. Someone like you is involved. Can you talk a little about sort of your role and your own trajectory? Where it came from and how it got to where it is today?

Colin: We’re relatively new and relatively young. I think we are blessed to have a dedicated research group and research leadership, given our size and our age. So, how did we get there? You know as – when CDS was a young and scrappy startup or a handful of people, I think the roles were not as clearly defined and people did whatever they could do to help the partnerships move forward. And luckily one of the skills in that mix was research. There were people there who thought that design research was important and who liked going out and talking to people about services. And so from the very beginning that was a part of who we did our work. And so as we grew, and then we had to kind of start dividing into more discrete communities of practice, and being a little clearer about what people’s roles were, I don’t think there was ever a question that research was kind of an important dedicated skill. I think it’s very related to design and so I think that we’ve gone through some iterations of figuring out kind of exactly how we relate to design. But right now we’re a parallel community at the same level as design or product management or engineering. And I think the more we exist that way, the more we like it because researchers need different kinds of support and coaching than other folks do, right? Research is a really different skillset then designing, at a service or an interaction or even a visual level, right? And so I think our researchers are really happy to have a dedicated group where they can get feedback on their craft and have managers that are rewarded and selected by not just their expertise sort of in general design, but in doing research in this world. So, I think we’ve been happy with how that’s shaken out. And I think we’re just – we’re lucky that research was kind of part of CDS’s way from the very beginning, down to the people who were founding the team.

Steve: Did you come into CDS in the role that you have now?

Colin: I did. So, I came to CDS with experience at a similar organization in the U.S. federal government. And when I arrived it was to lead the design research team with a recognition that that was a distinct team with sort of distinct support needs and a need for a manager that knew research and knew about doing research in government. So, I know that that’s not always true and so I feel very blessed to have ended up in a situation like that.

Steve: So, there was a team, or a nascent team, and that team needed leadership?

Colin: Yes. It was a small team at that point. I think there were three of us when I arrived, and we’ve grown substantially. I think we’re now 10 people or so, and continue to be a part of the growth plans for the org.

Steve: So those 10 people – you sort of described early on about how they would be working closely with a specific department or team that they were kind of part of.

Colin: Yeah.

Steve: Maybe you can talk about what that cycle looks like or sort of how projects or jobs or roles and researchers are matched up over time. What does that look like?

Colin: Sure. Yeah, so researchers are embedded on interdisciplinary teams. We call them product teams. And those product teams work with partners through some phases. So, the first phase is a discovery phase and that is a phase where researchers really lead in open-ended research about the nature of the problem that the department is trying to solve. We usually get set up with departments who know they have an issue and are interested in kind of our different approach to things, or they have a goal and they’re interested in us helping them achieve it. But researchers lead their teams in unpacking what that means, particularly for the humans involved, right. There’s important technical discovery that happens in parallel, but we really try to get the whole team involved in talking with both the members of the public implicated by the service, but also the staff involved in delivering a service. You know government services are this wonderful like sociotechnical network of people and we try to understand how those all fit together in discovery. It’s very rare that we would just talk to the public. I think we would spend a lot of time doing that. We really emphasize that, but we also really try hard to see the other side of the service and how all those people work together to create the experience the public sees. So, that’s discovery, right. Getting the lay of the land and understanding perhaps some of the possible roots of the problem.

And then a team transitions into alpha and beta phases of building something. What that is varies tremendously based on what the problem is. I think one of our product teams with Immigration, Refugees and Citizenship Canada did a bunch of work on a letter that was involved in a rescheduling process. That was sort of classic paper content design, tested by a researcher. Happened in conjunction with some digital work they did, but was just as important. So, that could be part of an alpha or a beta, or it could be about building a new digital service like we did with Veterans Affairs Canada building a new directory of benefits for veterans. Those products take different shapes, but regardless the researcher is trying to bring the voice of all of the people we talk to in discovery back into the product development cycle. and so that can mean usability testing, or content testing. As our products get more mature, we also use quantitative methods. We run randomized control trials. We try to use analytics on things that are out in the wild. They use all of the methods at their disposal to try to keep bringing the voice of the people we’re serving into the product process. And I think what that looks like, as I say, just varies tremendously and we kind of like it that way. I think that’s one of the cool parts of being a researcher at CDS is we say well you have this team, they have their needs, you need to serve them and make sure they have the right information about people to make good product decisions, but there are lots of different ways we can get at those answers and we agilely – probably to overuse that word – assemble the methods and the timelines accordingly.

Steve: So, a researcher working on, for example, the Veterans Affairs, for the duration of that program that’s the thing that they’re working on?

Colin: Exactly. Exactly. So, we’ve been really fortunate to basically have one researcher per product team and to be able to keep them on the same team. Which – and sometimes that’s not possible, but you know research, as hard as we try to not do this, sometimes becomes a practice of implicit expertise, right? I think researchers who have a long history in an area, or with a product, kind of know things about how people will respond to the service that are hard to articulate or sort of systematically articulate in reports or the other ways that we try to codify that knowledge. And so we see huge benefit to people having some longevity in the product teams and thus in the domain that they’re working in. Not always possible, and I understand not possible everywhere, but for us I think researchers really enjoy kind of building a deep expertise that comes with doing 10, 15 or 20 studies for a particular product in a particular area.

Steve: Depending on the type of organization, the carryover from one product team to another – I mean in my mind government is sort of an example of a category where there’s a lot of different things that you’re doing.

Colin: Yeah.

Steve: And obviously there’s cultural things and organizational things. Whereas if you’re working in some commercial enterprise that maybe makes a lot of different products and maybe serves different customers, the breadth might be less.

Colin: Yeah.

Steve: So the value, I guess, if I’m doing the math in my muttering – the value of that sort of hard won knowledge is maybe necessary to preserve or cherish it in a different way, just given the breadth of what you’re doing.

Colin: I think you’re right. And one of the fun things about getting to support these folks is that I can have conversations on a given day that range from how the Royal Canadian Mounted Police want to handle cyber crime to how Canada Revenue Agency processes tax returns for low income people to how the Employment and Social Development Canada issues benefits, right. And those are incredibly different, both business processes and missions, but also involve very different people and very different kinds of concerns on the parts of those people. So, it’s pretty neat to get to hear about all of that, but it also creates a challenge in that when we start on a new team, or a new researcher comes to a new team, I think there’s some sort of basic government knowledge they’ll bring with them, but often they’re trying to come up to speed on a pretty complicated area, pretty quickly.

Steve: When you described earlier your role is to – you know in that kind of coaching relationship you have with the different researchers – but it sounds like you’re the one that has the overview or has the window into these different product teams and what they’re doing. Are there researchers – what kind of interaction do they have that isn’t through you necessarily about what each is working on or what kind of challenges they’re facing?

Colin: Yeah, absolutely. I am – one of the things I learned very early on was that I was not the best conduit for all of that information between them and each other. And so I increasingly view my role as creating opportunity for them to share directly with each other and across the org. So, what that boils down to – there are first of all researcher standups, where we talk about what research we have done, every week. And we try to focus on not just kind of what we did, but what we’re learning. And that’s usually not enough to create understanding, but it’s enough to create a hook, right. It’s enough for one person to say, “oh, that person is really talking about the experience of submitting a form that’s an application for benefits and that’s actually kind of similar to what we’re doing, so maybe we should go have a more deep conversation, or I should ask for some documentation.”

We also have rotating, dedicated research critique groups. So, researchers form groups of 3 or 4. They meet weekly to give each other feedback on their work. It’s a little trickier to give critique on research than it is on design artifacts, but just as important. So, they’ll go through research plans with each other. They’ll go through recordings of interviews and sort of talk about the approach that the researcher took and different ones other people might have took. They’ll go through reports and deliverables. And although the kind of stated purpose of those sessions is to help people grow and share skills with one another, I’ve observed that a real common output is better cross product knowledge between them, right. If you’ve spent the time really thinking about the pros and cons of someone’s research approach you tend to understand their product a bit better. So we have critique groups.

And then one of the other kind of structures we have at CDS is these research community meetings that are actually open to everyone from the organization. So, every other week we host these 45 minute kind of brown-bag, lunch style meetings where researchers give talks. They give talks on either their product work, when they’ve recently finished it up, or on their – or on new or interesting kind of methodological or government logistical things that they’re working on. So, we try to create lots of channels for that information to flow. I think that we’re still kind of a size where some of those meeting and interaction driven approaches work. As we grow, I suspect we’ll have to get more creative.

Steve: Right, critique circles for ten is different than doing that for 80, or brown-bags. Which is crazy that I would throw that number around and everyone would nod, like yeah, that’s the size of research groups in some organizations now.

Colin: Sure, sure.

Steve: It’s not that long ago that that was an absolutely ridiculous idea.

Colin: Absolutely.

Steve: So, at ten, you can have some kind of communal knowledge just based on – well, you’re putting formal things in place for sort of semi-formal knowledge exchange.

Colin: Yeah. Yeah, that’s right. I think that we’re certainly beyond the totally informal everyone talks over lunch about what’s happening size. I think we are at the edge of what works well for critique groups and sort of community meeting exchanges. But again, it’s a little trickier for us because our domains are so different, right. And so it’s harder to say we’re building a common understanding around the user of a particular product that we all study. Often, we’re studying very different things and there are things to learn across them, but there’s also real differences between them.

Steve: I’m going to switch gears a little bit. I think it builds on some of what we’re talking about, but as you talk about the team having grown, and maybe growing into the future, what do you look for? What makes for a good researcher for CDS?

Colin: So, we focus broadly on a couple of things. I think the first is craft, for lack of a better word. So, when we hire people, and when we’re going through interviews, we really dig into the details of their previous work. You know what – how did you construct the research questions? Why did you pick those questions? How did you pick the methods that you used to answer those questions? Why did you pick those methods? We spend cumulative hours working through that with folks because we really believe that the basic skills are so important because they’re hard to – if you don’t have them down, they’re

And I think the other piece of it is broadly what we call bureaucracy hacking and that is being able to navigate a bureaucratic process to get your work done within the time that we need it to get done and with enough cognitive flexibility to kind of think through what the really important pieces are, where there might be alternative routes to the really well trodden one, and ultimately get a result despite a complicated, multi-person, multi-process situation. So, we look for people who are excited to do that and who have some demonstrated skill navigating situations like that.

Steve: Is the public opinion research story you talked about earlier, is that an example of bureaucracy hacking?

Colin: I think it’s an example of institutional level bureaucracy hacking. I think that product teams themselves and people who are working on those teams often have to do the same thing, but kind of at a lower level. So, they’re working with a partner who says, “oh, like we have a departmental process around talking to this group of people. We need to go through that process in order to do this research. That process usually takes nine months. Your research is supposed to take 3 weeks, how do we make that happen?” And it’s kind of an interesting skillset, right, that’s required to navigate those situations. It’s not just sort of knowing what good research is. It’s also your ability to think analytically about a process and understand the reasons for the pieces of those processes and then look more broadly. I will say, as we grow, we try to do less of that at a departmental level and more of that at kind of a broader institutional level, but that’s still a work in process.

Steve: It’s interesting, and maybe just coincidental, but you talked about looking for recruiting skills, very specific kinds of skills, in talking to researchers, and then this bureaucracy hacking example that you gave is around the logistics of recruiting participants.

Colin: Yeah, yeah.

Steve: You know and – I mean I’m glad we’re highlighting recruiting. I think it’s sort of a neglected – just get people and talk to them sort of is sometimes the belief in research. You’re talking in some cases about getting to maybe harder to find groups of people, or groups where there’s a specific relationship. And we also talk a lot, in research in general, about operationalizing some of those things. We haven’t talked about that. What’s your view on – whether it’s the recruiting part or just in general, in the context that you’re in, how does that idea play out?

Colin: Yeah. So – I mean I think that it’s not surprising and certainly important that the industry at large is increasingly focused on operationalizing processes like recruiting and thinking about I think ways to devise that work and to scale it more efficiently. That seems totally reasonable given the size of research teams that are now part of the modern organization. I will say, at CDS I approach it with some caution, and I’m worried I’m going to sound a little like old and cantankerous, despite not being much of either…

Steve: You have me though on the call. So, just compare to me. I can be old and cantankerous. I’ll take the heat on that.

Colin: I appreciate that, Steve. No, I – for a lot of our work, the work of doing the recruiting is part of the research itself, right. So, actually going out and making connections with community organizations or professional boards, or senior centers, or all of the places where we do our research, that’s part of how we understand who we’re trying to serve and the social structures that surround them. And so I – when I think about how we’ll scale, I try not to think about ways that we would totally take that out of researchers’ hands because I think they would lose part of the picture of who they’re trying to study if they were to do recruiting in its own right. If they were to sort of separate that into a different role. I think that – the other thing that strikes me is that so little – like when I look at the substance of our research, at the substance of what we do, some of it would I think be possible to kind of mechanistically do faster, right? Like if we had better templates and better knowledge stores and sort of better processes to string those things together, like it could speed things up a little bit. But often the idiosyncrasies in people’s research and the sort of surprises that don’t fit within the template are the most important things that our researchers find. And you know perhaps that’s a sign of our organization’s maturity, but I get worried about kind of whisking all of it into a well-oiled, sleek process for – you know I wonder about the sort of edge of whiteboard, straggly sticky-note on the margins insights that we miss. And those – and for us, in our work, those are so often key, right. Those are – we’re still building an understanding of the space and so often those things that don’t fit well within your predetermined framework are the most interesting parts for us. And I worry about losing that. That said, making the logistics of scheduling people and where they go easier, I am all for it and we continue to look for ways to do that. But I think we have to be thoughtful about what we automate in our industry, just like all knowledge workers should be, I suppose.

Steve: You made a comment a few minutes ago when I asked – when you just said research in general – you said, “it’s hard.”

Colin: Yeah.

Steve: I just want to – I want to go back to that. I don’t know if I agree or disagree. Can we just reflect on that notion of research is something that’s hard? Like what about it is hard? Should it be hard? Is that a bug or a feature?

Colin: Yeah. As it came out of my mouth, I’m like oh that’s kind of a complicated statement, I wonder what you meant by that – to myself? I would say this. In some ways research isn’t and shouldn’t be hard. I think one of the things I love about this work is that I can sit down with someone from any one of our teams and explain some basics and they can go out and start learnings things in a more systematic way with some good pointers from a good 30 minute discussion. And that’s great. And so I don’t want anything I’m about to say to be read as kind of discouraging making research accessible to everyone. We say everyone at CDS can be a researcher and I really believe that. I will say that research is also a – one of those things that’s very easy to do mediocrely. I think that there’s a lot of subtleties to how you make people comfortable in research sessions. There’s subtlety to how you digest lots of qualitative information. There’s subtlety to how you arrange your research, so it influences your product in the most responsible, but impactful way. That’s all – that’s all – I think there is subtlety and trickiness to that and so I think it’s important to have an appreciation for people who are really good at doing those things and to – I don’t necessarily count myself as one of them for all of them – and to recognize that that is a real skill and that is a skill that we should – that we should celebrate in kind of our broader industry community. That is, I think we can do that without also saying hey, there are some basic things that people can do that are easy and quick and help make more people gather more data to make better decisions. I think those are sometimes placed in a false binary in the Twitter sphere, for example, and I am not as interested in that. At CDS, and I think my comment was somewhat related to the Canadian Digital Service in particular, there are some things that are quite – I won’t say uniquely hard about research, but are specific to our context. You know one of those is the sort of incredible domain switching that we expect researchers to do every 3 to 9 to 12 months. And that’s, I think, not unusual in consulting, but it is somewhat more unusual in a product driven organization like ours. There is also, I think, the hard work of figuring out how to fit your research into a interdisciplinary team. You know we don’t just write reports and make presentations and give them and sort of hand them off to designers. We don’t let people off the hook at that phase. They’re kind of in the trenches with all those other people as the product is being developed. And we expect the way that teams run their agile development process to reflect research and for researchers to be a voice in that. And I think there are lots of organizations that have that expectation. I think we’re trying to do that in combination with domain switching and with the third part of it which is, again, we’re trying to build skills. Not just in ourselves, but in our departmental partners. So, you’re trying to learn a new domain, fit your research skills into this rapidly evolving, interdisciplinary team, and help a partner learn the basics of research and appreciate research. I think that’s hard. I think it’s fair to say that’s a difficult thing to do.

Steve: I want to switch a little bit. I would love to hear if there was a point in your own personal, professional path when research was a thing that you identified with? Like I want to do that, I am that, that’s for me? I don’t know, is there a moment or a stage at which you connected with the field that you are in now?

Colin: Yeah. There was. I conducted my first usability test when I was in 7th grade. I was – this was pretty early in the days of such things. But I was working on the school’s website. That was sort of my hobby. I helped the computer teacher with that. And I was reading Jakob Neilsen’s Homepage Usability book. There was this beautiful book with multiple sort of printed out homepages and he talked about his methodology in the back and I read that. I was like oh, this usability test is sort of something we could do. So, I got someone into the computer lab and tried it out and there was this moment of like wow, when you ask other people to try something, and when you ask them questions about what they’re thinking and what their goals are, you challenge yourself in ways that you – that I didn’t expect. And it’s also incredibly rewarding. It’s an incredible high. I don’t know how else to describe it. And it continues to drive me and the folks on my team. So, from that moment on I knew that I wanted to do this kind of research in some way professionally. There were lots of steps between there and here, but I knew pretty early in life that I really loved learning about how people used computers and services and understanding their approach. And, you know, I’m really blessed to work alongside lots of other people who are similarly jazzed to learn those things about people.

Steve: And that’s an astonishing story both – to me – in the early point in your life at which this happened and the specificity when it happened. I think often these stories are about oh I went to a farm and then I realized I wanted to be a marine biologist. Like the connections are more diffuse. But you, at a very young age, were doing exactly, or pretty close to exactly the thing. I mean it’s not even a metaphorical discovery.

Colin: Yeah.

Steve: You literally discovered the work.

Colin: It’s odd to me too. And I do look at friends’ and family’s career trajectories and I’m like hah, I didn’t think that’s how it would work out for me, but I really – I just loved it. And I looked into other things, but kept coming back to this, this being what I really just enjoy doing.

Steve: Are there any points, whether it’s 7th grade or older or younger – what kinds of things can you look to in your earlier parts of your life that were, I don’t know, maybe weak signals or kind of nascent superpowers that connect to what you’re passionate about and what you’re spending time in now? Does anything exist earlier in your life?

Colin: You know, one thing that has always been somewhat odd about me, and I do think relates to my passion for this work, I have always loved seeing the metaphorical or physical backroom of the process. So, like when I’m flying, I’m always like hah, like what does the computer system that the, you know, gate agent used look like? And like what do they do and when do they do it and why? And I’ve always – I’ve always been like that to a point of I think some well natured teasing from my family about my desire to understand the details of how the parking ticket machine dispenser system works and what the numbers on that thing mean. So, I think, especially to do this work in government, you have to kind of have a love of uncovering process and humans and how they interact with that process. And the trick is you don’t have to love like process, but you have to love kind of post modern process. Right? You have to love the fact that like it is different things to different people and kind of is this great mirror game of clarity and unclarity all at the same time. That’s something from an early age that I’ve always really enjoyed. I also would say that I – so early in my career I worked for the U.S. National Park Service and one of the things that I did there was being a park guide. So, be out on the trail talking to people and listening to them and answering their questions and seeing how they interact with a place. And there – it’s real joy – I feel real joy to seeing people in an environment and both helping them discover things and see how they discover things. And often I feel like my work today isn’t that much different, right. I think I’m more open ended than park interpreters are, but it’s still here is this new land and this new world that you are encountering, and I get to be with you as you’re doing that and help you think through what you’re doing. And so I think the kind of odd areas of joy I found in that work certainly kind of echo through to what I’m doing now.

Steve: Is there any example of a notable success? Something that, you know, maybe that your researchers have done and kind of completed that you feel proud of? Or just kind of a notable example to share?

Colin: Yeah. I think that – it’s funny. You know we’ve done a lot of research that I’m proud of and I’m proud of our researchers for doing at CDS. But the things that make me most excited are the moments when I see our researchers teaching and coaching people and departments to start doing research themselves, right. So, for example, with the Veterans Affairs Canada project they – the department identified someone who would be a researcher and sort of continue on the research with that product after we handed it back off to them. And I, you know – those moments when those people were working together and getting to support them through that, I think those probably make me feel prouder than any particular deliverable or product that we’ve gotten out the door.

I will say the other exception there is I’ve looked – I was looking at our sort of internal database of research before this interview and I’ll say the other thing I’m particularly proud of our team for doing is becoming more and more diverse and creative in the types of research it does, right. I think when we started, we were trying to make the basics happen and so we did lots of interviews and we did lots of usability tests. And those are great, but there are other things to do too. I think we’ve become not just qualitative, but mixed methods. I think that we have embraced the complications of trying to kind of test services in more live ways, right. So, not sort of just usability test prototypes, but wire something up so that someone can complete an actual transaction with it and see the back end experience and orchestrate kind of a Wizard of Oz experience for them. Those, those increases in our skill and our ability to mix and match tools responsibly, that’s incredibly rewarding as a researcher at heart.

Steve: Is there anything that I should have asked you about that I didn’t, and you want to make sure we talk about?

Colin: No. I think your questions were well rounded. Nothing comes to mind from my list.

Steve: Do you have any questions for me?

Colin: What’s the most surprising thing you’ve heard during one of these interviews?

Steve: Well, this is going to sound like I’m pandering, but the fact that you did a usability test in 7th grade is – I’ve just never come across that in my career, let alone in one of these interviews. But actually – I will say – I said I would take the weight of being the old curmudgeonly person. So, here we go. I don’t like the what’s the most surprising thing question ‘cuz it presupposes that the person – and I realize this is not exactly research since we’re doing a podcast, but as an interviewer in general, I’m not trying to be surprised. I’m trying to be focused and interesting and sort of the most benign sort of factual thing should also be exciting and triggering of interpretations. So, you know what I mean? Like the surprise is me bringing my judgement into it. So, of course I gave a concrete example. I was surprised by something that you said, but I also…

Colin: Yeah.

Steve: People ask researchers that, right? When you did this study what’s the most surprising thing? And I think sometimes that question can be good if it provokes what are we learning, or where are our assumptions being challenged? But, how do we – I also want there to be space for like hey I didn’t come in here with assumptions. I came in here with just curiosity.

Colin: Yeah.

Steve: So, I’m kind of talking out of both sides of my face here, I guess. I do that.

Colin: Well, I agree. And I do think there’s a real risk to boiling research into a set of surprising and non-intuitive insights, you know. Like the idea is that we should be manufacturing surprising little blobs of knowledge that can be passed off into the ether. And I worry about that. I worry about the work getting boiled into that. And so I hear you. And I guess I agree with your curmudgeonly instincts.

Steve: Aren’t I a nice person? Any questions for me? I mean I’m going to slap you for the first one that you asked me. So, I’m just living up to the curmudgeonly mantle I chose.

Colin: I expect nothing less. Well, that’s good. No, I would say I am curious as to what your views on the burgeoning research ops movement are? As someone who I think has been in this world for many of its evolutions, what’s your reaction to the increased operations emphasis?

Steve: I mean I have – learning about research ops provoked a uninformed critical reaction. And you said it really well when you talked about the value that the recruiting can bring to the researcher. And, you know, we hear researchers talking about the value that they get out of doing their own transcripts. Personally that’s not a thing that I could deal with, but immersing yourself in the subject matter and the people in the community and the data and the conversations is super labor intensive and the idea that we’re going to fix that by kind of operationalizing things scares me because it takes away the quality and – you talked about subtlety, I think. But if you look a little deeper into what research ops people are championing, they’re not championing that. They’re – that’s my chauvinistic misinterpretation. And I think just by creating that word, I think it invites other people to also do that and maybe even implement that. But the people who are championing it are thinking about process and thinking about how does this exist? And, you know, having spent my entire career as a consultant, so not embedded in the organization, not building these processes – maybe advising on them, or kind of giving feedback, it’s sort of an interesting thing to watch from the outside. It comes up in a lot of groups. I talk to people who are kind of early on in their organizational maturity. You know I read something recently – oh, some research ops people were sort of declaring that the first researcher hire should be an ops person which, you know, from the outside seems like well of course you would say that. That’s sort of a discipline that you are evangelizing for. But I found myself working with a group where I just mentioned that idea as an idea that exists. I wasn’t in a position to say they should do this, they should not do it. But I said, given what you’re talking about your barrier here is ops, is infrastructure, is recruiting, is not reinventing the wheel. And I said, you know – they had headcount to hire one person and I said well what proportion, from zero to 100, is that person going to be dedicated towards this? You know, if the same way that you are – that you talked in the beginning, about your organizational change efforts, you know convening people, finding processes, sharing best practices. I think a research ops person can also be doing that, you know at a different level and using kind of different processes than you’re talking about, but…

Colin: Yup.

Steve: I mean that is championing the growth of research as a thing that we have skills to do at various levels efficiently. That’s what you’re – that’s one of the things you’re doing. That’s one of the things a research ops person could be doing. Taking away the hard, labor intensive, inefficient tasks that lead to the subtlety that makes research great, that you described -that’s not a thing I want to do, but I – it doesn’t have to be what ops is, so.

Yeah. If there’s nothing else, then maybe we’re at the end. Really great conversation. We covered so much interesting stuff and I learned a lot and I’m really so happy that we had the chance to speak today.

Colin: Likewise. Thanks so much for your interest. It’s been a pleasure.

Steve: Thanks for listening! Tell your colleagues about Dollars to Donuts, and give us a review on Apple Podcasts. You can find Dollars to Donuts on Apple Podcasts and Spotify and Google Play and all the places where pods are catched. Visit portigal.com/podcast to get all the episodes with show notes and transcripts. And we’re on Twitter at dollarstodonuts, that’s D O L L R S t o D O N U T S. Our theme music is by Bruce Todd.