Dollars to Donuts

In this episode of Dollars to Donuts, I speak with Marianne Berkovich, Head of User Research & Consumer Insights at Glooko. We talk about doing research through leadership changes, setting up opportunities for self-critique, and how to build empathy, especially in health technology, by experiencing some aspect of the condition and treatment yourself.

It really bothers me when smart people go out and build things and spend a lot of time and energy to build things that are not for humans. And I’m like, erh, why didn’t they do that? For me it’s more about empowering people who have that energy and who have that entrepreneurial spirit to make the things that are right. Not just make stuff, but actually, make the right thing. It’s a set of skills that I think maybe everybody should have and maybe once everybody has those skills and can do it well, maybe the role of researcher doesn’t need to exist. Until then, I feel like it’s my duty to go out and spread the gospel, as it were, of this is how you talk to users. – Marianne Berkovich

Follow Dollars to Donuts on Twitter and help other listeners find the podcast by leaving a review on iTunes.

Steve Portigal: Well, hi, and welcome to Dollars to Donuts, the podcast where I talk to people who lead user research in their organization.

I was reading the New York Times today and I noticed something unusual, although I’m seeing this sort of thing more and more. After the byline, but before the article starts, is some italicized text in square brackets. It reads “What you need to know to start the day: Get New York Today in your inbox.” This is copy that only belongs online, in the app or the website or an email newsletter. Presumably you would click on something. But I’m looking at the newspaper – I feel like I’m starring in one of those YouTube videos where a toddler is trying to swipe a magazine and can’t figure out why it’s not a touch screen. I see the New York Times making this kind of error in their print edition every few weeks, and it’s kind of appalling, because it suggests a lack of detail that I don’t expect from a high quality product like the Times. When you get an email that has the wrong name in the salutation, even though we’ve all done it ourselves, it brings your appreciation down a notch or two. They clearly aren’t taking the care that they used to, and that we would hope for.

And it’s especially interesting because I remember when the opposite used to be true; when the experience we had online, with news especially, was a not-quite-there translation of a print experience. And now it’s flopped. The print edition readers are not the primary customers. The organization has identified a different key user and it’s not us.

I can’t help but wonder about any of the users we learn about, are they in a less desirable category or perceived that way because of changes in internal processes or organizational structure? Do we care about their experience, or are we making them into what we call “edge cases” which is a fancy way of dismissing them. It’s hard to imagine the print reader of the New York Times as an edge case, but hey, that’s where we are.

I want to remind you that I’m looking for ways to be able to keep making this podcast for you. Here’s how you can help. You can hire me! I plan and lead user research projects, I coach teams who are working to learn from their customers, and I run training workshops to teach people how to be better at research and analysis. I’ve got two books you can buy- the classic Interviewing Users and Doorbells, Danger, and Dead Batteries, a book of stories from other researchers about the kinds of things that happen when they go out into the field. You rating review this podcast on iTunes, and you can review both books on Amazon. With your support, I can keep doing this podcast for you.

Let’s get to my interview with Marianne Berkovich. We did this a little differently – as part of a live event. We closed out San Francisco’s local edition of “World Information Architecture Day” with our interview, live on stage, in front of an audience. Then when we got off stage, we sat down in a somewhat noisy room and talked a little more to cover some of the things we didn’t have time for. We’ve cleaned it up as best we can but the audio may be a little bit different from how things normally sound. It was really fun for the two of us to speak on stage, and I hope to have more opportunities to record episodes of the podcast in similar settings.

Marianne is the Head of User Research + Customer Insights at Glooko. She’s worked as a consultant and for Google and Adobe.

Thanks for agreeing to talk with me, Marianne. Why don’t we just start by having you introduce yourself. What do you do? Where do you work? Tell us about that.

Marianne Berkovich: My name is Marianne Berkovich and I am Head of User Research & Consumer Insights at Glooko and we’re an online diabetes management platform.

Steve: What is an online diabetes management platform?

Marianne: That’s a great question. So, diabetes is a condition that’s a lot to do with numbers. You want to keep your blood sugar not too low, not too high. So, it’s really conducive and lends itself well to technology and checking your numbers. We have an app for the person with diabetes. They can track all sorts of things there that could affect their blood sugar and then they can also send that information to their clinician who can further see patterns of you’re high in the mornings and what are we going to do about that? So, it’s a platform that both the clinician can access and look at those things as well as for the patient themselves.

Steve: Where in the history of this company and its product did you come in, to bring in user research?

Marianne: So, I am the first user researcher. The company was founded in 2010. I’ve been there for about a year and a half and I am learning all sorts of things of what it’s like to be, first in a healthcare company, or a health tech company, and also being the first researcher.

Steve: Okay, let’s see how easy this is. What are some of the things you’re learning about being the first researcher and working in a health tech company?

Marianne: First of all, as a researcher, it just feels very weird to be answering the questions and not asking the questions. Maybe I answer questions in a way that makes it easier for the researcher to ask the next follow-up question. So, but you know what a great researcher Steve is, so this is showcasing his talent too, right! So I think some of the things that I’m learning is the role of advocacy. That even though there was a need for hey it’s time for a researcher, we need someone full-time – before they were hiring vendors or kind of doing it ad hoc. And so, it was time to have somebody who could be dedicated and who’s trained in this. But at the same time there’s a lot of – I don’t want to say – in some cases there was resistance, but I think in a lot of cases it’s more of just not knowing what good research looks like, or part of the reason my title is so long is that people had an impression that oh, user research is just usability studies. And so, I talked with my manager and we put in the “& Consumer Insights.” Head of User Research & Consumer Insights – so it’s like it’s everything. It’s going to be going out and doing field visits. It might be surveys. It might be a lot of different things. So, to kind of help with that advocacy. And I feel like I do spend – I mean, I think as researchers we always spend a lot of our time in advocacy mode, but I’m not surprised – or maybe surprised – it’s just taking a lot more of my effort to do that advocacy work of what research is and how can it be used.

Steve: I just want to clarify. You’re advocating for user research?

Marianne: Yeah. So, I think even though the company talks about being patient centered and user centered, that what does that really mean? I think one of the things that I’m finding out – so, there’s a lot of people who have been in the field for a long time and they’re like we understand diabetes. We’ve been in this for many, many years. And to have somebody come in – I don’t have any experience with diabetes; I don’t have diabetes myself – to come in with these, you know, different insights or different ways of doing things and people are like, we’ve been there, we know how to do this. And so, to advocate for, hey, we went out and talked to some people. We learned this thing that’s different and knew and it’s maybe against the conventional wisdom, how might we use this? And maybe some of that resistance of like, well we’ve always done it this way. Or conventional wisdom says this other thing.

Steve: So, the other part that you were learning was working in healthcare tech. It’s a new industry for you. So, talk about maybe what that’s been like.

Marianne: Yeah, so – and before that I was – I did a fellowship and we were working in the legal space. And the legal space, the health tech space, are slower moving than just straight up consumer products. It’s like hey let’s build something and we’ll launch an app and do these things. You know regulated spaces of like what is the FDA like? I had a little bit of background in human factors testing, but we have a product that actually got FDA approved. But what does human factors testing look like? How is that different from just regular usability testing? What does it mean to understand the whole ecosystem of payers and employers and insurance companies and PDMs and all that type of thing of moving that needle and kind of inching things forward when you’re in this space that can’t be disrupted so easily because there’s good reasons why these constraints exist.

Steve: So, the company was about 8 years in existence when you joined. So, how much of that were you building from scratch? Describe a little bit about what you encountered and what you had to build around sort of the tech and regulatory aspects maybe.

Marianne: So, actually, luckily the regulatory aspect is just one part of what we offer. It’s the Mobile Insulin Dosing System. So, when you start on insulin, how do you come up to taking the right dose of a certain type of insulin. So, the product does a lot more than that. So, this was just one small part of it. And it was actually – that was already in the works when I joined. So, the app was already out. The website was already out. MIDS was already sort of in progress. So, I think for me it’s been a role of a little bit of back to basics of like let’s learn about our users. How are people with Type 1 diabetes different from Type 2 diabetes. So, in fact, I was telling Steve before this, I’m doing an ethnography right now and so I was in Fresno this week talking to folks with Type 1 diabetes. I’m in Phoenix on Monday, but kind of building that basic understanding and actually building personas and really having a deeper understanding rather than like, yeah well, we have people now who are Type 1s, they know that they need. To move beyond that, to really have some foundational knowledge, and it’s causing us to go back and be like okay, well, is this – is there things in the app that are working the best way. Maybe we can make the clinician experience more efficient. Maybe there’s things that we learn when we go to visit a clinician – be like oh, that’s not how we thought that worked, can we rethink that. So, I think it’s a little bit of back to basics and rethinking certain things rather than to build anything from scratch.

Steve: Before you talked about advocacy which was kind of saying hey, we need to learn these kinds of things, but in some ways doing that reveals the most scary thing of all – oh we need to rethink assumptions and yeah, a 10-year-old company, it’s entrenched. So, what happens when the things that you’re uncovering are inviting, challenging – they’re inviting the challenging of established belief structures around the product and what it looks like and everything. How are you doing that?

Marianne: I think it leads to another kind of wrinkle of working in a startup is that we’ve had a lot of turnover at the leadership level, actually. So, the CEO who was the CEO when I joined was not the founder. He was the second CEO. And he reached a point where he was like, “you know what I really like getting the company to where it is now, but now that we’re really – it’s time to scale and grow things, it’s not what I want to do.” So, we got a new CEO. We got a new head of product. We got a new commercialization officer. So, all that advocacy I had done and sort of like when we did they Type 2 ethnography when I first started, like all that – you know taking people on the road and showing them and doing these empathy workshops and all that – whoosh, out the window because we’ve got a whole new cast of characters now. So, I think that was part of it too. It was a little bit humbling to have to re-sort of create my credibility again with – it’s a whole new cast of characters and a whole new set of people to influence. Luckily my manager has been a great champion of that. And so, I think it’s finding ways that we can leverage and sort of maybe it’s not the right time for things. So, we’ll go out and do this research and we have this foundational stuff. We sort of did a big aha, then those people went away. Can we save that and find another time to sort of bring that to the fore when it might be a little bit more accepted? So, I think it’s both sort of pushing for it and finding the right time to sort of introduce those things.

Steve: So, what does a manager do? How does that championing work?

Marianne: I think, – and I don’t know if this is true, but I feel like as a woman you know I can say a little of things if she’s also a woman, but like women in general, like it’s much more effective if instead of me saying it, I say like, “yeah, what Steve said.” Or, you know, what somebody else said. So, I think because of that, I think it becomes more effective. So, she’s not tooting her own horn. I’m not tooting my own horn, but she is sort of pointing to the work that I’m doing. Plus I think we have different kind of communication styles, and sometimes she can sort of, – I’ll say a bunch of stuff and she can sort of synthesize it in a really nice way and it sort of becomes a little bit more impactful that way, that like I can spend all my time sort of yelling and screaming – metaphorically speaking – and then she can sort of bring it home in a very succinct way.

Steve: So, part of what she’s doing then is also – you talk about finding the right moments. Or, what should we be doing to have impact on the organization as the layers above us are changing. I don’t know. I’m wondering about when new people come in, and this is going to vary on a case by case basis, but you’ve gone through this whole series of trying to open people’s minds up a little bit and say hey things are different. Those people leave. New people come in, but is there an opportunity there? Do those people – what baggage do they have, or do they bring in? I’m not asking you to like slag anybody individually, but as you see these kinds of changes happen in an organization and you’re trying to craft a story that’s about how the world really is vs. maybe what we hope or assume – how does the – what changes when new people come in, with or without baggage, around what the truth is?

Marianne: I think part of it is understanding where people are coming from. Because I think in my last role – I was at Google for a long time and I think like certain assumptions I made that like everybody knew what user research was and everybody knew this and I kind of got used to that, that that was the norm. Like product managers of course know what I do, and everybody knows how this works. And so, I had to figure out like, oh, okay, I can’t assume that people know what these things are, or that when I say user research and what you think in your experience with user research, like maybe that was focus groups. And they’re like yeah, we did those and it was great and I was like okay, great that’s a start. So, I kind of know where I’m starting from with folks. So, I think that’s part of it. And I think also, because I am the only researcher and things move a little bit more slowly, like we did the stuff around Type 2s and now we’re doing the Type 1s. And so now I’m using all that stuff that I learned last time of like what was effective in terms of running a workshop and getting people to come on visits with me. And it’s like it’s a little bit smoother again this time, but we’re doing that again this time around. So, it seems that having that opportunity to redo that foundational research has resurfaced itself. And so, to take advantage of that opportunity, that it’s like well, let’s look at Type 1s now.

Steve: There’s something here about you’ve had lots of experience. This is not your first job. You’ve been at lots of different organizations, you’ve done lots of research and influenced stakeholders and product teams and so on, but it sounds like that there’s a good measure of learning in this job which is about how do I do the thing that I know how to do to be effective in this context. Does that ever go away for researchers, do you think?

Marianne: I hope not. I think it’s – and I think that that’s part of the reason that like, you know – one of the reasons I left Google is like it took me a while to realize the thing that I was doing there and the things that – that, kind of really I’m passionate about are not things that I could do at Google. But like who leaves Google, right. So, I think finding ways to find my path and things that are not sort of the traditional way of moving up in, you know, you’re a junior research, then you’re a senior researcher and then you’re a manager and all these things. To find my own path and sort of be okay with that and find different ways to learn in different contexts. Like the fellowship that I did with Blue Ridge Labs which is a social impact incubator. And I was again, the only researcher, and so I got to do more mentoring. And so that was interesting. So, I think finding ways to both follow my passion and find ways that like what does this organization look like? And I think we’ve probably heard this all before in terms of doing sort of user research or bringing that lens to our stakeholders or our people that we’re working with as well. I think that part never goes away and it’s always changing because it’s always a different set of people.

Steve: Right, more so – is the landscape changing more in a place like Glooko than in a place like Google?

Marianne: That’s a good question. I think it’s very different. Google’s obviously a very – there’s just many people. And so, I think the dynamics of what it’s like to have an organization that big and when I joined I knew all the researchers and now the scale – each product’s team is much larger than that. So, I think it’s a different type of – it’s a different set of issues. So, I think wherever you go – like I’m from the East Coast and people ask – they’re like well what’s better, east coast or west coast, or all this stuff? And I’m like they’re different, right. I don’t think there are things that you can really compare. So, for it’s, – and this is my first time working in a startup and so I don’t have another startup to compare it to. So, maybe if I went to a different startup I could do more of that comparing and contrasting, but in some ways, it feels like apples and oranges of a large organization that’s established and, you know, is well funded and all that vs. a smaller one that is working in a very different space.

Steve: Just hypothetical – I know we’re not supposed to ask – apparently you aren’t supposed to ask hypothetical projection questions in user research, so good thing this is not user research. If you were to look for a job at another startup, I’m just thinking about you joining an 8-year-old company, is there a maturity that you would look for, or a timespan that you would think differently about in the next stages of your career?

Marianne: I think it’s less for me about that. It’s more about really getting jazzed about the problem that I’m solving. So, for me diabetes is a very big issue. Thirty million people in the U.S. and growing. And so that feels like a real meaty issue and that gets me excited. So, I’m like even if some days it feels like I’m pushing a boulder uphill, it seems like a worthwhile thing and that’s what gets me up in the morning. So, for me it’s much more about that and going in with eyes wide open of like well what kind of organization in it – is it, and like okay do I want to take that on. And if the answer is yes then working around those constraints because that’s just the nature of the beast.

Steve: You talked before about figuring out the right job title that would describe, to the rest of the organization, the way that you were going to work. As you – can you say more about sort of what that process was, what those conversations were that identified an opportunity, that helped get you excited that this was something you wanted to do?

Marianne: Well honestly, I think part of it was naivety, that like this was – I think it was actually the way it was posted or sort of written up was Manager of Consumer Insights and I’m like okay, whatever. You know, I’m not on a ladder. I’m the only one, so it doesn’t really matter. So, for me, I was so excited – in fact I was consulting before and so Glooko was one of my clients and I got hired. So, we crafted the role a little bit based on what I was doing. And so for me it was less about negotiating the right title and in fact I think only after I joined – because I was like, sounds great, like it sounds like this is a good fit, sign me up, and then only when I started and I was like hey, I want to make business cards, but I don’t really like this Manager of Consumer Insights – we had that conversation after I joined. And so that’s when my manager started telling me about like hey, this is some of the perceptions because I had gone in very naively also of like everybody knows what user research is. Like, we’re in Silicon Valley, like everybody knows what this is. And so, it didn’t even really occur to me that that advocacy, or the extent of that advocacy that would need to be done.

Steve: So, what – you worked with the organization as a consultant and then came in-house to kind of lead the effort. What was similar and different about – you know, the before and after that transition?

Marianne: So I think one thing is, it was very intimidating. I didn’t know anything about diabetes and so I ran my first study and like everybody showed up and so they were listening in on the call and I’m like how’s this going to go? Like I hope I don’t say anything stupid. And so, you know I think as a consultant, coming up to speed on a different domain very quickly, and I think to me it’s about asking those open-ended questions and asking questions in a way that it doesn’t really matter if you know the domain. I mean it’s certainly much better if you do, but there are ways that you can sort of cover that up, and especially if you’re going in with an apprentice mindset and all those types of things, it kind of helps the situation along a little bit. So, I think for me that was definitely intimidating to not know the domain and have everybody show up.

I think another difference is as a consultant, and I wasn’t a consultant for very long. It was about a year or so before I decided to take the role with Glooko , but I was very cognizant of how I was spending every single hour and whether it was going to lead to making money or not, because is this going to generate a lead? Is it going to generate a sale? All those types of questions were very much top of mind whereas when you’re in-house I think you can spend the sort of hours – like you’re spending your capital in a different way. You’re building relationships and that’s kind of how you’re sort of making money. So, to me it’s sort of a different way that you don’t have to be so aware of every hour leading to money.

Steve: I’m going to switch gears a little bit and maybe we could just rewind. I’d love to hear you describe maybe your path. How did you get into this stuff? What brought you to where we are sitting today?

Marianne: So, I was an English major for undergrad and after I graduated I didn’t know what I wanted to do. So, I spent a couple of years – I actually at that point did work. I was in D.C. and I worked at the National Museum of American Art. We were digitizing the collections. It was a very long time ago and so I was actually cleaning up all the photos and I’m like eh, I’m not really into this technology thing. My Dad also was a computer science professor. I’m like definitely not into that technology thing. Like that’s stuff that my Dad does. Boring. So, I spent a couple of years kind of bumming around a little bit, decided to move out to Denver and I started working as a technical writer. And I was working at a financial services company and so my role was to write the help text for this complex financial software. So, I would sit with the developers. They would explain to me how the complex software worked, and I would write it up. And then I had a thought. I’m like if we just made the software easier to use I wouldn’t have to write anything. So, that led me to first get a certificate in technical communication. And in that I started kind of looking around a little bit more and found the whole field of human centered design and human computer interaction. I looked around and I found a program at Carnegie Mellon. And at that point I had actually switched over to Lockheed Martin, which is a great role – you can ask me about that in a second. Sorry, I just keep getting the questions to ask. And, but I – you know I decided that I really needed a degree in this thing. That just kind of reading books about it or whatever wasn’t enough. So, I decided to go to grad school and I remember just seeing that description of the master’s program and I was like that’s exactly what I want to do and like all my life had been leading to that point to be like that’s exactly what I want to do and having that feeling of like, yup, this is the direction I want to head in.

Steve: Which master’s program? And then keep going. Yes, please.

Marianne: It was a master’s in Human-Computer Interaction at Carnegie Mellon University. So, the role at Lockheed Martin. So, I’m old. I’m definitely over 35. And this was sort of in the olden days and what we were doing is taking – so, you know how we have wildfires and all that type of stuff? So, the resources to manage all those things, like sending the air tankers and the trucks and the crews and all that – that was being done by hand. So, Lockheed Martin got a contact to turn that into – it wasn’t even online. It was just kind of a digital program to do that. So, we would interview subject matter experts and I was like a requirements analyst. Like we didn’t have roles of like designer researcher. Like that wasn’t a thing yet. And so, for me it was like super exciting to solve a real problem. We got to actually go to Boulder and like see the command center and see some of those things. I mean I really felt like I was making a difference and I knew what this was about. But I felt very much at a disadvantage of like, I don’t know what should the icons be? I don’t know. Nobody has best practices around icons yet. And so, it felt like early days and that’s why I really wanted to go back to school and like learn some stuff because I knew there was a lot that had been done already.

Steve: So, you come out of that program and where do you go?

Marianne: Consulting. Yeah, I actually stuck around Pittsburgh for a while and I again ran my own consulting thing. I do that to sort of – when I’m in exploration phase. Or, when I’m dating somebody who just started a PhD program and I can’t leave Pittsburgh. One or the other. So, yeah, and the guy I was dating, he was very entrepreneurial, and he was like such a great cheerleader. He was like yeah, we can figure this out. He had been running his own startup before he started the PhD program and he was like, yeah you can do this. I’m like, yeah, I can do this. So, he did help me a lot and so I was finding projects and was kind of getting my feet wet with that. But then I found that it was time to leave Pittsburgh. That it wasn’t what I wanted to be doing. I really wanted to learn from other people and so I started looking for other opportunities.

Steve: What were those opportunities that you found?

Marianne: You can take it in a different direction. So, I actually interviewed at Google at that point and they didn’t want me, I didn’t want them. It was a very different company at that point. But I also interviewed at Adobe. There’s actually a pipeline from Carnegie Mellon straight here to Silicon Valley. I mean seriously, just pull up a bus, just put the graduates on it and bus us all out here. So, there were a bunch of people who I had gone to grad school with who were at Adobe and I went to interview there, and it felt like a really nice fit. I remember actually – after the Google interview I came home to my hotel and I just turned on the Simpsons and like all I could do was just sit there and like not move. But after the Abode interview, I actually – it was one of my first times out here in Silicon Valley and like I went hiking. I was like I was that inspired and that energized that I went hiking. I’m like this is a sign that maybe I should take this role, that this is a good role for me.

Steve: We can cue up some Simpsons for you after this conversation. You know, just put you in a dark room. Um- so, when you talk about the work that you’re doing now, you’re doing field work, you’re doing ethnography, you’re taking people out. When did you learn how to do that?

Marianne: That’s a good question. I feel like I should credit CMU with setting me on the path and teaching me about contextual inquiry – thank you Bonnie John. And I think for me one of the things that they told us at Carnegie Mellon is – it’s a one-year program, so if you didn’t come in as a computer scientist you’re not leaving as a computer scientist. If you didn’t come in as a designer, you’re not leaving a designer. But we’re going to give you enough exposure to coding and designing and having your stuff on a wall and getting critiqued and to do some social science stuff. And so, for me, having been an English major and I minored in art history and theology, I mean I was as liberal arts as you could get. I wasn’t going to come out as any of those things and I didn’t gravitate towards it. I was like I’m not a designer. So, I think I just gravitated towards research. And I think maybe the English degree kind of prepped me for that in terms of asking good questions and thinking about things and looking for patterns. So, I think just naturally I’m a pattern seeker and I think honing my ability to sort of ask questions just came with time. So, I think it was having a good solid foundation and then just kind of having a natural affinity for it. And I read your book – much, much later, but I did read your book.

Steve: Right. By the time my book came out you’d been doing this for a while. Can you talk about the environment at Adobe? What was there there to develop the practice of user research? In the practitioners like yourself that are coming in with X amount of experience, what did they do well that let you get to the next level of your craft?

Marianne: I think actually I was really lucky at both Adobe and Google, being surrounded by really smart people. So, that made it okay of like I don’t know how to do this, and can someone help me? Like I learned how to do surveys through some of the really awesome people at Google. So, I think for me it was just being surrounded by really supportive people and I think that was one of the things that was part of the culture at Abode. Like we would go to my manager’s house. She would have like all the researchers at her house and we would do these offsite – and it really felt like family. I mean it sounds a little cheesy, but – and so it felt like it was a place to grow and learn. So, I think for me that was a great stepping stone as my first sort of real job after graduate school, to be like oh, okay, this is what being a researcher is like. Oh, this is how you interact with product teams. So, I think it was just learning all those basics. But I think one of the things that – kind of coming back to the question of learning to do ethnographic work and all that is once you’re out – and especially you’re probably one researcher. You’re rarely lucky enough to work with another researcher who can sort of observe you can say like hey, why are you asking that question? Or, maybe we could do it differently. One of the things that I found, when I did have some opportunities at Google to work with other researchers, I’m like we don’t all have the same skills. Like what happens between you and a participant is different. And so, one of the things that having noticed that is I started a class at Google to critique ourselves. So, give researchers an opportunity to observe each other moderating and to give feedback to each other. And then I read a Medium article about it. It’s called Don’t Leave Data on the Table, if you want to look it up. I think that’s one of the things too is that we don’t get that critical eye anymore and we assume that like every researcher is the same and it’s like well actually, you know – and those things can also creep up on us. Like I see myself all the time, especially now that I’m sort of teaching other designers at my company more of like, ah, leading question. Yes or no question. You know. So, I find myself still doing that, even if I’m aware of it. But I think having those things pointed out to us is also really helpful too.

Steve: What’s the structure of the critiquing class?

Marianne: So, it was slightly different when I taught it externally at CHI, and internally. So, internally all our videos are available to each other. So, basically people would send in clips from research that they did. We would make small groups, so it lets you work in a group of five, and we would watch a segment of somebody conducting a study. And at any point people could stop the video and be like hey, I noticed this. It was like when we were actually practicing to teach it, one of my co-teachers, she actually had her pen and she would do a lot of like pointing and sort of like do a lot of things that indicated high status. And she was like, I never noticed that – until she saw herself, until we talked about it, she wasn’t aware of that body language and what it was conveying. And so basically we’d spend 5/10 minutes, or however the time divides up, to watch that video and stop if we see anything. And we asked people to rent that idea that like, again, we don’t know the context of what came before. We don’t know the context after. But this is what I’m noticing. And that’s what we do as researchers is we notice. So, it could be a pattern, it could not, but at least invite people to consider something that they’re doing. And also, by seeing other people, just having that conversation, seeing what other people – and be like okay, I didn’t show that, I didn’t do that in my clip, but like I’ve definitely done that before too. So, learning from each other as well has been really valuable.

Steve: It sounds like a way to create kind of a safe space where it’s okay to critique is that everybody is kind of up for that critique and we’re all being exposed together. Is that kind of the way to make it safe? Because this is our workplace and now we’re talking about how we’re not living up to sort of the high standards of our profession.

Marianne: Yeah. And I think that’s one of the things that we tried really hard to make it a safe space. And one of the things that we did was actually we had – spoiler alert – we’d give out like balloons, which are also like these cords that you pull to sort of stop the production line. But just having a room full of balloons made it a little bit more fun. When I taught at CHI we had to create the videos and so I had these little stuffed animal guys that people could put their phones on when they were doing the videos. So, then we had little stuffed animals on all the tables. And so, – and also modeling it ourselves that – I don’t remember if we actually showed some of the videos, but we definitely told some of the stories of like when we did this ourselves – it’s like I’ve been a practitioner for how many years? Like I’m still learning. I still screw this stuff up. So, making ourselves vulnerable was part of making that safe space too. And also, I think what we tried to do was, when we did it within Google, was not have everybody who was on the same team be together. So, it wasn’t like anybody who you worked with directly.

Steve: You used the phrase “sort of screw this up” and I think like the pen example and I’m sure there’s others that are clearly things we shouldn’t do as researchers. To me it seems like there’s a certain amount of stuff that’s subjective that you and I would do it differently because we’re different and we have different personalities and different energies. For me, critique about that would be really interesting because I like to see how other people handle things. My question here I guess is, is there – does the critique approach look at alternative approaches? How do you talk about that vs. best and worst practices?

Marianne: I think we also gave an opportunity for people to stop the cord on themselves. To be like I didn’t know what to do here. Like help me brainstorm. So, there’s that opportunity too of like for the person themselves to be like how will we handle this? And to hear different perspectives on – there’s, you know, I could have done this, or I could have done this, or in situations in the past I’ve handled it differently. So, I think having just that variety of there’s not one right way to do that, that it depends on the context. And maybe sometimes somebody explains like well this is what happened earlier in the session and so it led me to do it in this way and we’re like, oh, okay, well that makes a lot of sense. And so, I think having a little bit more of that context helps too.

Steve: Are you able to bring any of this into the environment you’re in now where, if anything, you’re teaching people and leading them.

Marianne: Yeah. And I think I’m learning a lot more about screwing it up also. While I’m out in the field doing these Type 1 ethnography, like I’m trying to help designers to be able to do their own research. And so, one of our more junior researchers is actually doing one on one interviews. So, I helped, and we talked about it and I wrote the discussion guide and the study plan. We talked about it and she was all on board and off to the races we went. And then I actually had a cancellation, so I was able to watch one of the sessions and I was like oh no, I didn’t – I screwed up. I didn’t prepare her well enough. It was a little bit more than she could handle and that was squarely on me, you know. That because I’m so used to doing it and it comes so naturally to me that I didn’t recognize that she was in a different space and she would need a different level of support and that I just hadn’t prepared her. And so, we role played a little bit and we practiced a little bit. But I still feel like – I feel guilty. I’m like she’s got more sessions this week and like I don’t know if she’s really prepared. So, I think that part is on me as well to make sure that the things that I’m modeling and the things that I’m sort of teaching are doable for where the designers are and not just hey it’s easy, anybody can do this.

Steve: In my opinion there are some things in life that we do that we maybe learn more from screwing up than we do by succeeding. I don’t know what falls into that category and what doesn’t. I wonder if, as much as you feel for this person and feel responsible for them, I wonder just ultimate perspective, did you set them up for a lot of learning they might not have otherwise got to by – of seeing the edges of what they’re able to control or what they’re able to execute on?

Marianne: I can’t speak for her.

Steve: Good. Alright. Well we hit a dead end. Let’s see what other questions I have written for you here. If we were to have this conversation again in a year or two years, what kinds of things would you want to talk about that you would have achieved in this organization?

Marianne: I hope that there’s less sort of friction. I think we’re already starting to see the seeds of it, that you know product managers who have been at the company for a long time are saying like oh, I’m doing this new thing, you know I need user research. They don’t know what it is. They don’t know why exactly, but they’re sort of like it’s a good thing generally. So, I think to have more of that. Honestly, I think like some of the things that are more – a little boring, kind of more infrastructure stuff – like I hate recruiting. I’m a terrible recruiter. I’m terrible with like time zones and all those details, that type of stuff. So, like I’d love to have a recruiter. I’d love to have another researcher on board. And some of the things that I feel like I have a great success under my belt, that it looks like we’re moving away from NPS (Net Promoter Score), hooray, thank you, thank you, and moving towards satisfaction. You know asking satisfaction question and how often we’re going to do that. You know, that’s a really interesting thing that I’ve been thinking about is can we pop that up in an app. I mean this is an app that people use for health and like they might be having a low blood sugar. You know this is a stressful time that people are using the app. Like nobody is excited about managing their diabetes. So, to pop up something in the app, to be like how are we doing, that’s not the right thing. So, I haven’t figured out like how we might introduce those satisfaction questions and when is the right time to do those things. But at least moving away from net promoter and to asking the satisfaction questions and asking that on a regular cadence so that we can see over time like are we getting better and kind of what the trajectory is, I think that would be a huge thing to have accomplished as well

Steve: That’s a great note to end it on. So, thank you very much. It was a great conversation. Thank you. Thanks, everybody.

Okay that was the end of the interview on stage, and now here’s the remainder of our conversation. So, welcome back Marianne.

Marianne: Thank you. It’s the bonus track, huh?

Steve: Right, this is the director’s cut version of the podcast. So, one of the questions I wanted to follow-up with you on is you talked about the work you’re doing on the Type 2 and the Type 1 and really trying to drive change in the organization. I’m wondering, sort of tactically or practically, what are ways to bring those experiences that patients have into the organization itself?

Marianne: I think actually one of the most fundamental ways that it started is using our app itself and so – I don’t have diabetes myself, but for the first 2 weeks that I was – or first couple of weeks I was at Glooko, I sat down with our certified diabetes educator and we pretended that I had just been diagnosed. And so, she talked me through what somebody would hear. How they would be introduced to pricking their finger and measuring their blood glucose. And so, I spent 2 weeks and I logged all the food that I ate, all the exercise that I did. I measured my blood glucose twice a day just to get that experience of what it was like and like yeah, it was pretty crappy, even after a couple of days, to prick your finger. And to imagine that’s for the rest of your life. Like, I had the luxury of stopping. And so, for me it was a really small way to understand what it was like for somebody who was newly diagnosed with diabetes to do that. And I also presented that back to the team and I sort of, encouraged other people to try. So, we instituted a program. So, anybody who starts at Glooko now can get a blood glucose meter and to try it out themselves and to see what that experience is like. So, it’s one small way to at least start building that empathy.

Steve: Are there others that you want to describe?

Marianne: Yeah, actually, so I wrote a Medium article about different ways to build empathy when working in health tech. So, one of them was experience it yourself. So, actually to try the app and try measuring your blood glucose all the time. See for yourself. So, when we did the ethnographic visits, to bring people out so they could actually talk to and see people. And listen to others. So, some people that I work with actually do have diabetes and so I wanted to hear their stories. So, we started something called Life with Diabetes Storytelling. So, just a couple of folks each time and they’ll either tell their own story of how they were diagnosed and what it was like. Or some people have children who have diabetes and that whole experience. Or some people have parents who have diabetes. So, it humanizes it even further because these are people that we actually work with. So, it’s one thing to visit somebody’s home and you never see that stranger again, but these people that we see all the time and we think nothing of somebody at the lunch counter, or lunch table, pricking their finger and taking their blood glucose, or pulling up their shirt and dosing insulin right at the table. Like that’s just kind of what happens. And to ask those questions and making it okay to ask like well when did you tell your wife? And what is it like to do sports and kind of do those things that you used to do? So, it’s been a really great way to create empathy and even bond as a team, to hear each other’s stories. We put them up all on the website so people who start new can watch the videos themselves and learn about people stories.

Steve: And what’s – the format is videos. Are there are materials that are – I guess describe what this looks like?

Marianne: So, I was a little ambitious at first. I was like we’re going to have three people who speak once a month and I realized the company’s not that big and a lot of people actually don’t want to talk. And even I was encouraging people to – even if you don’t have diabetes, you don’t know anyone with diabetes, like there’s lot of this stuff on the Internet. Like watch some videos. Read some people’s blogs. Like bring that in. Haven’t had any takers on that yet. So, now kind of what we’re doing is we just have two speakers and we do it about every other month. And I just give people the floor. I say you’ve got 15-20 minutes, tell your story. Some people take less time. Some people are a little bit more hands – they take a little bit more time. But all I’m doing is giving them the floor, just creating that space. And it’s been wildly popular. Like engineers will come. Like people – we actually fill the room where it is. So, it’s just an opportunity for people to tell their story. And I also wanted to make it a little bit special, so the JDRF has these little bears called Rufus the Bear that teach kids who are diagnosed like how to do their insulin and all that. So, I’m riffing off that. So, I have these little bears that I give people after they speak, and it says, dear so and so, thanks for sharing a diabetes story. So, it’s personalized and people sit them on their desk so that when you walk around you can see who shared a story. So, I wanted to make it a little special of like, yeah, thanks for sharing and we know whose stories are out there too.

Steve: That’s great. So maybe a different question, following up on some of the things we talked about earlier. You know thinking – we talked about how you learn research and kind of your process through these different environments that you were in. You know at this point, do you have a super power?

Marianne: I think my super power is complexity busting. So, I think that based on what I’m hearing – and I think qualitative research gets poo-pooed a little bit, but I found that being able to suss out those patterns from just a few interviews and turn that into a robust framework of – you know I did a thing when I worked at Google around ads and what are the characteristics in ads and that framework is still being used. So, some of the things that I’m doing now in terms of persona research – or developing personas and also the Bingo card framework of the needs of people with diabetes. So, there’s a couple of columns, there’s a couple of rows and people have different needs and sort of fill that down individually. And those types of things that lead to action and sort of like synthesizing, encapsulating like here’s the things you need to know. Like, I’ve thought about it and I’ve taken all this chaos and I’ve put it into something that’s usable and actionable that the teams can use. And I think that’s one of the things that really makes me happy when I’ve figured something out and then turned it into a product that the team is like aha, we can run with it. This is actually useful for making decisions.

Steve: Right. I think those two pieces are really interesting because I’m with you on the finding patterns rapidly and being able to tell a new story about them, but I think you’re productizing them and communicating some kind of communication design around them so that other people can understand that, I think that’s two superpowers in one, maybe the way you’re describing it. Yeah. Do you have a way of thinking about your own brand, your own identify as someone that – as a researcher, or someone that works in tech, works in these kinds of spaces? How do you think about yourself that way?

Marianne: I think for a long time I thought I’m sort of like everybody else and like we’re all sort of doing the same things. And I have an opportunity now – I’m part of this incubator for women leaders and it’s having me sort of rethink what is my brand? And that one needs to have a brand. I think I also maybe started thinking about it when I started having my own consulting practice and I’m like how do I stand out and what do I offer and how do I package these super powers, and kind of bring that to the fore? So, – and part of this incubator that I’m in, it just started somewhat recently – but the women who are in it, like it’s not just people who are in tech. I think a lot of us are in tech, but we have like doctor and we have some lawyers and we have some other people who are doing other things. And so, it’s great to be with that different community. And so, one of the things that we did in our first session was really think about ourselves as a company. It’s like we all work for companies. We’re all like what’s the vision statement? What’s the mission statement? Every company, you’ve done SWOT analysis. So, we’re turning that lens on ourselves. And so, part of our homework for this month is to figure that out. And I think one of the things that I was thinking about is my mission is really user centered everything. That user centered stuff can be applied to any domain and having worked in finance and wildland firefighting and consumer stuff and with creative professionals, I think I’ve seen it there. But that’s all been within tech. But really, applied to things outside of tech. So, it can be applied to parenting. It can be applied to government. It can be applied to just a lot of different things. Like in fact I joined a gym recently. It’s called 9Round Kickboxing. And I realize how user centered they are. So, I’ve got to riff on this for a little bit. So, some insight somebody had along the way was people don’t want to spend a lot of time at the gym. They want to go at any time they want to. They don’t want to wait for a particular class and they like the special attention from a trainer. So, what does 9Round offer? You can literally go at any time. There’s 9 rounds that you do in order, but you can start as soon as you get there. Each round is 3 minutes. So, you’re done in 30 minutes and it’s one of these high intensity workouts. And there’s a trainer there who can like help you and spar with you at a particular moment. I’m like this is a fabulous example of being user centered and really understanding what your customers need. And I was like I signed up for the year. I’m like done. This is exactly – I’m your target user. You’ve figured out my needs and this is fantastic. So, thinking a lot more about human centered everything and how can we apply that to domains and sort of just everyday life because when things are human centered it’s more humane.

Steve: Is there a source or a reference or an inspiration for that human label, kind of in what you’re looking at?

Marianne: I think it came from you know having worked in a lot of different domains. I think what really crystalized it for me was I read a book called More Human by Steve Hilton and he’s married to a Silicon Valley exec, but he was in the UK and then he moved here and he was like – went to – started hanging out at Stanford and came upon design thinking and he’s like aha, all these things I’ve been doing and thinking about policy and applying it to business and all these things. And so, he really crystalized it and I’m like, yeah, I believe that. Like what if our businesses were more focused that way and what if like the policy of how do we get rid of homelessness actually like prototyped for us, rather than just spending tons of money and doing a particular project for 5 years. So, I think he really captured that human centered everything for me.

Steve: So, as you think about that as your brand, how are you going to operationalize that brand – to use a terrible phrase. But how does this go beyond a concept for you? What do you think you’re going to be doing?

Marianne: So, I think it’s something that I’ve been doing in terms of mentoring, like working with entrepreneurs – so when I left Google, I was sort of exploring two paths. One was more mission oriented something or other and the entrepreneur side of everything. So, right now my full-time job is much more in that mission driven kind of aspect of it, but I’ve also been doing mentoring for entrepreneurs. I’m mentoring a woman in Venture for America right now. I’ve done kind of office hours for a bunch of things. I’ve taught at Nasdaq Entrepreneurial Center. And so, I think those are ways where I can find different ways to apply it and find ways that – how do you find intrapreneurs and entrepreneurs to give them the tools to be able to ask questions and go out and talk to users to apply it in a lot of different ways? That is a way to make that happen. And so, I think that for me, whether it’s my full-time job, or kind of like the balance of those things may change in the future. So, maybe I’ll be spending more time and I’ll find a role that lets me do more of that working with entrepreneurs and helping them develop that and maybe less in this oriented stuff, or maybe I’ll be able to find ways to combine the two. But I think for me, in sort of the next phase, it’s kind of how do I find the overlap, or sort of finding the balance of those two things?

Steve: You know sort of the thesis of this podcast is around – I don’t know thesis, but just the people we want to have as guests are people who are user research types and I’m wondering, as you describe this sort of human centered everything, and these mentorship roles, are you being a researcher, as a way you would define it in doing that work? We have labels and so I want to ask you to kind of unpack the labels and see what you’re sort of framing on this work you’re exploring vs. maybe the work that connects you and I and why we’re having this conversation?

Marianne: Yeah, that’s a really interesting question. I think for me, it’s – I think once I get sort of hot under the collar about something, like everything else goes out the window. You know I was talking earlier about once I found Glooko and I’m like wow, they’re working on a really big, juicy problem. This is something I wanted to do, who cares what the actual label is, or what the title is. So, I think for me, like it really bothers me when people – smart people – go out and build things and spend a lot of time and energy to build things that are not for humans. And I’m like, erh, why didn’t they do that? So, I think for me it’s more about empowering people who have that energy and who have that entrepreneurial spirit to make the things that are right. So, not just make stuff,

Steve: So, it’s a really articulate lovely reminder of why we do research? It’s what we’re trying to have happen and you’re looking at achieving that goal through research, or through mentorship, or through advocacy and all the things that you’re doing are putting stuff out in the world that helps people – I don’t want to put – now I’m putting a lot of words in your mouth, but there’s kind of a producing of things ideal that you are a champion of. So, I see research – as you explain it, I can see research as a part of that, but not the only one.

Marianne: I think you summed that up really nicely. So, thank you for making me sound more articulate.

Steve: That’s the reflecting back technique. Okay, so now maybe wrapping up our epilogue, anything to add in this? Anything I should have asked you about?

Marianne: I was just thinking another user centered everything. I’m doing some volunteering and working with a foster child and I’ve been able to turn that into a user centered thing too. Because she doesn’t live with me, I just see her for a few hours each week, so I think very intently about the activities I’m going to do and being a crafty person also, I have spent a lot of time on YouTube videos and Pinterest and whatnot and making activities that I’m like well we’ve got to work on like fine motor skills. We’ve got to work on like counting or matching and stuff like that. And I’m having such a great time designing these little activities for her and also seeing how she responds and like, oh, she doesn’t like those types of things. Or, she needs more of these types of things. So, I feel like bringing it to – like I think I just naturally bring it to all aspects of my life and I see the power that it has, and I just feel I’m like – I just need to be a champion for those things because I see the value of it and I just want the world to know.

Steve: That’s a lovely note to wrap up on. So, thank you so much, Marianne.

Marianne: Thank you so much.

Steve: Well! That wraps up this episode of Dollars to Donuts. Go to portigal.com/podcast for the transcript as well as links for this episode. You can follow us on Twitter, and subscribe to the podcast at portigal.com, or iTunes, or Spotify, or Stitcher, or anyplace excellent podcasts are distributed. You can buy my books are available at Amazon and rosenfeldmedia.com. The amazing theme music was written and performed by Bruce Todd.

Welcome back to Dollars to Donuts. This episode features Leanne Waldal, Senior Director of Product Research at New Relic. We talk about establishing research in an organization for the first time, building up a diverse set of research collaborators, and the pleasure of taking on certain types of challenges.

I’ve seen hopeful examples in startups recently where even though there are only 3-5 people they bring in someone for research and that’s highly unusual. Usually, if you’re starting up a company you have seed money, or you have your first round, the first people you’re hiring are engineers. You’re not making one of your first 10 hires a researcher. I know some example now around the San Francisco Bay Area where they’ve actually brought someone in whose role is to do research really early in the company. And that either points to a certain amount of humility around, oh, just because I am the user it doesn’t mean I know the users. Or, because they’re going into a space they don’t know. – Leanne Waldal

Follow Dollars to Donuts on Twitter and help other listeners find the podcast by leaving a review on iTunes.

Steve Portigal: Well, hi, and if you’re a new listener, welcome to Dollars to Donuts, the podcast where I talk to people who lead user research in their organization. otherwise, welcome back to Dollars to Donuts, the podcast where – okay, you got it.

It’s been a little while since the last episode, and I’m really happy to be back making new episodes for you. In a little bit I’ll talk about how you can support me, and the podcast, but first I wanted to mention an interesting article I’m reading. It’s from The New Yorker magazine, from the January 28, 2019 issue. It’s an article by Robert Caro who is an author of among other things, a multi-volume biography of Lyndon Johnson. This article is adapted from his memoir called Working: Researching Interviewing Writing, and his recollections about those activities are what caught my eye. He describes going to do research at the Johnson Library and Museum and going through boxes and boxes of papers. It’s the hard and tedious work of investigative journalism and what struck me was that amount of inferring, and cross-referencing and delving he was doing – he was coming up with facts and perspective that were not there for the taking but were from analyzing and synthesizing the information that he did have as well as what he didn’t have. Later in the article he talks about the importance of “place” in his research, that to understand Johnson he had to leave New York and really spend time in the Texas Hill Country. He writes “As soon as we moved there, as soon as the people of the Hill Country realized we were there to stay, their attitude toward us softened; they started to talk to me in a different way. I began to hear the details they had not included in the anecdotes they had previous told me.” And finally, he talks about silence in interviewing, citing two characters from fiction – Inspector Maigret and George Smiley, and their tactics to quote keep themselves from talking and let silence do its work unquote, where Maigret fiddles with his pipe and Smiley uses his tie to polish his glasses. Caro has his own brilliant technique: “When I’m waiting for the person I’m interviewing to break a silence by giving me a piece of information I want, I write SU (for Shut Up!) in my notebook. If anyone were ever to look through my notebooks, he would find a lot of SUs.”

Speaking of the connections between journalistic interviewing and user research, I want to recommend the podcast from 2017 “The Turnaround” – it’s interviews with different kinds of journalists about doing interviews, about getting to the story. I learned as much from the contrasts between our different objectives as I did from their best practices that apply directly. Check it out!

Now, obviously, I’m back with more episodes of Dollars to Donuts, I’m taking more of an open-ended approach to new episodes, and they’ll appear as they’re ready, without any specific frequency. It takes a lot to do this podcast and rather than taking advertising, or doing crowdfunding, I want to ask you to support the podcast – and me. You can hire me! I plan and lead user research projects, I coach teams who are working to learn from their customers, and I run training workshops to teach people how to be better at research and analysis. You can buy my books for yourself and your friends and your colleagues – I’ve got two books – the classic Interviewing Users and more recently, a book of stories from other researchers about the kinds of things that happen when they go out into the field – it’s called Doorbells, Danger, and Dead Batteries. You can also give this podcast a star rating and a review on iTunes, and you can review either book on Amazon. With your support, I can keep doing this podcast for you.

Let’s get to my interview with Leanne Waldal. She is the Senior Director of Product Research at New Relic. She has led research at Autodesk and Dropbox as well as running her own research agency for almost 17 years!

Thanks for being on the podcast.

Leanne Waldal: Absolutely

Steve: Let’s start with a basic introduction. Do you want to tell us who you are? What kind of work you do? What your company is?

Leanne: I’m Leanne Waldal. I am Senior Director of Product Research at New Relic. New Relic does application performance monitoring. It usually makes people’s eyes glaze over. Basically, we have a tool that developers and engineers, primarily that group, uses to monitor the traffic on websites and mobile apps. So, if you think about streaming media, online retail, financial services – tons and tons of data and traffic – their site reliability team, or engineering team, might use us to monitor that to make sure nothing goes down. I’m based in San Francisco and I’m a part of the design team which is a part of the product team. The product team is mostly in Portland, Oregon. We have a smaller portion of it in San Francisco and we also have a team in Phoenix and in Barcelona.

Steve: Can we talk a little more about New Relic and who the customers are? You said to make sure that it’s not down, but I’m assuming it’s not just that binary state, like is Amazon up or is Amazon down?

Leanne: It’s to monitor it. So, be watching to see if something happens that’s slower. Usually it doesn’t matter if something is faster. It means you’ve optimized performance and things are running really well. So, if things are going slower or you notice that people are having longer wait times for page loads or checkout loads or media streaming, or whatever sort of thing they’re trying to do in your app. You would have spent a considerable amount of time setting our products up, basically, to then do all that monitoring and raise alerts when something goes wrong.

Steve: Okay. So, it’s not up and down. It’s slow and it could be any part.

Leanne: Exactly. Lag time. Checkout’s not happening in the amount of time that you expect it to.

Steve: Did you use the phrase reliability engineers?

Leanne: Yes. So, the new role that basically Google created about 10ish years ago is the site reliability engineer. They wrote a book about site reliability engineering. So, our target user is usually called a dev ops engineer or a site reliability engineer. If you are a more modern company with the way that you’re keeping track of your website and your apps, then you probably actually have a development operations team, or a site reliability team. If you’re not quite there yet, then these people might be a part of an engineering team, or IT operations or something.

Steve: That’s good context. Then you said, you’re part of the design team which sits in the product team?

Leanne: Yeah. So, at New Relic, engineering and product and design all report up to the chief product officer. Different companies do it different ways. Engineering might be alongside product or design. Design might be alongside. But at our company, if you’re in the product org, you belong to the design team, the product team or the engineering team, or a few different teams that are in the org.

Steve: How long have you been in the role that you’re in?

Leanne: I’ve been at New Relic for six months. I did a similar job at Autodesk, before New Relic, for about a year and a half. And before that I had a similar role at Dropbox for a couple of years.

Steve: How do you compare and contrast those three organizations and what you saw and what the trajectory was in those roles?

Leanne: We could start with Dropbox. Dropbox was private, pre-IPO. It was very much a unicorn. When I started there it had 400-600 people. I don’t remember exactly how many. There was no research. They had a small design team. They had, as a lot of tech companies at that time, being a unicorn, had tons of engineers. And also, because Dropbox is a consumer app as well as a business app, everybody used it, so they felt like they knew who the user was.

Steve: Everybody that worked there?

Leanne: Everybody that worked there, yeah. So, I started setting up a research team there and grew it as the design team was growing. That’s different from Autodesk where I went next. A very old company, very fascinating products with fascinating customers and use cases. And people there who were researchers there, who had been researchers there for a long time. And what I was doing there was combining research and analytics together. At Dropbox, analytics was separate. And I had a small team that was more globally disbursed. At Dropbox, my team was all in San Francisco, to start. Here at New Relic, when I joined, there was one researcher here in San Francisco and one in Portland and a similar sort of tone around a lot of the people who work here at the company. Our site reliability engineers (or have been), so they know a lot about the market and a lot about the users. So, we’re basically here to help the company understand all of the new users it’s acquired since it started because it’s a different company now than it was 10 years ago. Ten years ago, New Relic was a company that was mostly reaching out to developers and engineers to use its product and now we’re mostly focused on enterprise and we have lots of enterprise customers now.

Steve: There’s that classic – you referenced this, right. People think they know they’re customer because they are users.

Leanne: Yeah, we all do.

Steve: But you’re kind of saying yes, to a certain extent. I don’t hear you shutting that down?

Leanne: It’s like yes and the marketplace out there has changed and maybe there’s something new that we’re not looking at right now. And also I’ve noticed that for companies where they maybe used to do contextual inquiry and go out and visit their customers and users and understand them deeply, in the last 10-15 years companies have tended to move to surveys and remote interviews and stepped away from that sort of like deep understanding you get from being in a room with someone, or on a construction site, or on the road with them when they’re using your mobile app, or whatever it is. So, you can see everything around them while they’re using your product or your app. And when you move away from that you lose sight of that holistic story of the customer experience. And then research can come in and do that for you.

Steve: Why do you think there is that movement towards the remote and movement away from the contextual work?

Leanne: Contextual work takes time and effort and can be exhausting. Exhausting to do it and exhausting to come back and know what to do with all that data, how to tell a story out of it and how to decide how it can have impact or value? Or sort of the like, now what do we do? I think also as humans we get really familiar with where we are, and we neglect to notice that the world around us has changed.

Steve: So, that’s back to your point then. We started this company based on work that we’d done and skills that we had and users that we knew, that we were, and the world moves on and shifts and new customers…

Leanne: And it’s not just New Relic or Autodesk or Dropbox. That happens at any company. You can be in banking or in legal. Ask anybody who has been there a long time, compared with a person who just started. They’re having much different experiences of it.

Steve: You described, especially with Dropbox and New Relic, this point at which the company had little or no research, and that is not unique. We know lots of companies like that.

Leanne: Oh yes.

Steve: Do you have a hypothesis about what’s going on? There’s a point at which someone like you starts talking to these companies and starts saying, “hey, here’s what I would do if you had me come in and work with you.” What’s going on beforehand? What’s the point of need or pain that’s identified inside these companies when they realize…

Leanne: Sure. So, what I’ve noticed at the companies I’ve worked at, as well as companies I was interviewing at for this sort of position, and companies I know the story around when they hired someone in sort of like a director or higher role in research, usually something changes in revenue and that change in revenue sets forward sort of like how do we figure out what’s going on. And if they have someone who is currently working there in a role who has had experience in the past with doing market research, or doing churn research, then those people will start to raise the flag of like oh, we have to do research. And then it’s how do we do it. So, that’s one reason that someone brings someone in to do it. Another reason is that there’s sort of like a groundswell of engineers, product managers and designers who are telling the people who make hiring decisions, “we need someone to be in charge of design.” Or, “we need someone to be in charge of research. We’re doing a lot of this ourselves and we don’t feel like we’re doing it well enough.” Or, “we, for example, do lots and lots of interviews. We don’t have time to synthesize the data. So, could you please hire a researcher to work with me.” So, sometimes that happens. Another way it happens, which is how it happened here at New Relic, is that the company decides to hire like a VP or SVP of design and then that person who knows, oh this is what a design team looks like – you have someone who is focusing on content design and language and someone who is focusing on interaction design and visual design. And I also need someone who is focused on research. Sometimes it’s from the top down. Sometimes it’s from the ground up. Sometimes it’s there used to be this role at the company before and someone left or was laid off and then they just didn’t backfill it and then after a while realized oh that’s really a role that’s really important.

Steve: We need that.

Leanne: Yeah, and it’s not – I used to think that maybe it was sort of like a trend, like everybody is starting to be customer centric, or user centric, or whatever that means. So, then they were like, “oh, well Joe down the block is doing that, so we need to do this too.” And what do we do, we hire someone to run research. But, as I have been interviewing for jobs over the year and talking to different companies and doing these jobs, I’ve realized, no, there’s actually something that happens inside the company that causes that change. It’s not so much like copying a competitor or another company that you care about. You might do that with engineering or something, but research is sort of like the first thing to go if you start running out of money. And, I’ve seen hopeful examples in startups recently where even though there are only 3-5 people they bring in someone for research and that’s highly unusual. Usually, if you’re starting up a company you have seed money, or you have your first round, the first people you’re hiring are engineers. You’re not making one of your first 10 hires a researcher. So, I know some example now around the San Francisco Bay Area where they’ve actually brought someone in whose role is to do research really early in the company. And that either points to a certain amount of humility around, oh, just because I am the user it doesn’t mean I know the users. Or, because they’re going into a space they don’t know. They just had a really good idea about it and they really need to understand the space.

Steve: That’s an encouraging sign.

Leanne: Yeah.

Steve: Is there a distinction between – you gave a number of different scenarios where research comes in – do you see a distinction in sort of the context, or the action that’s being taken when bringing in a person to do research vs. bringing in a person to lead/manage/run/build research?

Leanne: Yeah. So, if you’re brought in as a person to do research you’re usually reacting to the things you’re being asked to do. You usually don’t get to pull your head up and do strategic work. You’re doing lots of like compare this design with this, do this survey to answer this question, help me with like interviewing these six users in remote interviews. If you’re being brought in as someone to say like where should research fit into the company, it’s more challenging, but more interesting and exciting because then you’re being told like tell us what we should know? Like where should we go with this? Are you going to focus more on the market research side, or the sales research side? Are you going to focus more on the product design side? What are all the things that we need? What kind of programs do we need? How do we get access to users? How do we interact with users and engage users? There’s just all these little pieces of how a research process and practice runs inside of a company that is great fun to put together because you have to do it based on who you’re working with and the culture of that organization. It’s not just a cookie cutter that you put into a company.

Steve: Is there a trajectory from one to the other? Like someone that’s hired because the need is to do research. Is there a trajectory for that person to start answering the kinds of questions that you’re talking about?

Leanne: Yeah, yeah. I think if that’s sort of like the way you work, or your personality, or the way you’re motivated, you can get there. For example, I have someone on my team right now who I mentioned to the other day like – she’s a researcher and I said, “you could make some day, if you wanted to, a really amazing research operations manager because you’re really good at all these pieces around managing the research and communicating it.” Because half of research is PR and sales. You have to find the people to engage with it, to sell it to at the end. You know you want your research to have impact on product or a marketing campaign or whatever you’re doing. So, you have to gather the people around you. And you have to keep track of, particularly in a B2B company, who are all the people who have to be involved if I want to do visit customers. So, in a consumer company you just go out and find people. You talk to them and you do whatever you want with them. At a B2B company you have to be engaging with customer success, with sales roles – whatever they’re called – account execs, account managers – whatever that company calls them. You have to often also be engaging with product managers, designers, design managers. Engineers, didn’t used to be, just as a stereotype, interested in research and now at some companies they are very interested. And so managing all those types of people that you either have to get approval from, or you need to make sure that they’re bought into what you’re doing, or you need them to come along with you and do it, takes a lot of work. So, there’s this whole new role of research operations that usually helps keep track of that. And I’ve noticed some researchers have a real knack for sort of like managing all those details around their project and then shining a light on it and sort of going out and selling it, sort of like a marketing campaign. They just know who to schmooze with and talk with. And some researchers are really good at doing the research and writing the report and sharing it. I pointed out to her, “this is something you are doing for our team without being asked. Like, that’s a real superpower.”

Steve: Does that start to change the – I think you said this, at least indirectly – does someone who is shining around operations, like your colleague here, does that start to change how research is perceived or experienced by others in the organization?

Leanne: Yeah. Especially if you’re an organization that’s not used to research. Or they’re used to doing it themselves. So, they’re not used to partnering with someone else and they’re not used to having someone say like, “oh that’s not really a survey. That’s more of an interview.” Or, “oh, that’s not really an interview. We need to actually go watch people to do this.” Or, “this can’t be answered in an interview survey, or watching people. We need them to be in a diary study.” They aren’t used to having someone bring in all these different methods. And they aren’t used to having someone sort of like find some insights and then tell them. So, that’s why it’s so important to have a collaboration. I want you with me while I’m doing this research so that when I tell you the pens need to be blue, not red, you were there to hear everyone say they like blue better than red, instead of throwing the report over the wall.

Steve: I hadn’t really thought before of operations as a – I’m not sure you’re saying this – but, almost a Trojan Horse. That there are these sort of tactical objective problems to be solved under operations. How are we going to find these people? What is it going to take logistically? What is it going to take legally and so on? That starts to change some of the conversations maybe that can happen internally. Like you said, “this is your problem. We recommend you go about it this way.”

Leanne: Yeah.

Steve: I hadn’t thought of that under an operations lens myself.

Leanne: It’s really more in a B2B company than a consumer company because you have to manage so many relationships and oftentimes a researcher wants to focus on sort of like the details of the research. Like ask any researcher, they don’t want to do all the details around recruiting and scheduling people – or, most don’t.

Steve: Right.

Leanne: And so, you want someone who can do that. And then sometimes researchers also aren’t the people who know how to best sell their work or edit it down into like the three bullet points for an executive presentation. And it’s helpful to have somebody who knows how to do that. You know just sort of like editing and presenting and PR and marketing. And also knowing who to market it to? Like, oh, I know that there’s this sales VP in this office in Toronto who is going to be interested in this. That mentality of like keeping all these dots rolling together. And if you have someone who knows how to do that and also understands how to do research and what goes into research, that’s what I see makes a really good manager from an operations perspective. There’s the people management side too. Being able to mentor and coach and take care of people. But there’s more and more in B2B research a need for someone specifically in this operations role to just sort of like help everybody and raise the brand of the team and the company, or in the product team or the design team.

Steve: It’s so cool that we’re at a point in this field and this practice where we acknowledge, as we always did, that there’s a number of very different kinds of skills and expertises that are required. And you just outlined a whole bunch of them. But that we’re able to sort of say, and not be laughed out of the room, like oh, that might be different people. You talked about collaboration and collaboration with people with different strengths or superpowers.

Leanne: Yeah. And another expertise and skill that’s being introduced more into companies is the academic anthropologist who is now working inside companies on a research team. Like EPIC used to be primarily all academics and now there’s a bunch of people from business involved in EPIC, and also people who used to only be academic anthropologists and ethnographers working on research teams, inside tech companies or consulting agencies, or whatever. So, I recently hired a few anthropologists on my team. One who is a former professor, one who had applied anthropological research practices for market research. Another who had done research with an NGO, but also worked inside of a company recently. And I think that when you have that academic background and you bring it into a company, it’s like a special sauce. You also bring in a special sauce if you’ve been working in tech for 15 years and you know the ins and outs of how a product team works, but I like mixing those two things together because people learn off of each other.

Steve: You like to mix the special sauce.

Leanne: Yeah.

Steve: So, it’s an extra special sauce.

Leanne: Yeah. So, for example, we decided as a team that we wanted to look at all of the internal blog posts and other things that had been written about customers and users, from the point of view of people inside the company, because my team is new to this company. We were like okay, let’s take a look at everything that everybody else wrote. And I was trying to figure out how we would do that. Like, who knows how many internal blog posts there are and how many things that have been written about customer visits and things that different people know about users that they report out internally. And then I realized, I have some former academics on my team, academics who are now working as researchers in industry. They have skills in literature review. So, brilliant. I took them and had them put together like how are we going to review this and code it – special sauce, like they absolutely know how to do that. It was really easy for them to put it together. And then took other people who had more industry experience, or application focused experience, and said, “okay, now you can do what was set up. You can do the review of following the system.” And I have people who know how to set up systems set up the system.

Steve: So, that’s an example of the special sauce. They know how to deal with this.

Leanne: Yeah. And if you have a team who has mixed experiences like that, then they just sort of like, they lift each other up and they teach each other things. And that’s something that you can’t learn if you’re the only researcher, or you’re just one of a few researchers who all have exactly the same background. So, you all come out of the – whatever the programs are now, at Carnegie Mellon, Stanford and Berkeley and University of Washington and on and on. And you go and get a job and you do remote interviewing and some focus groups and some other things But everybody around you is doing exactly the same thing. Then you don’t get a chance to sort of like up level yourself. And you also, like if you’re the person managing that team, you don’t get a chance to learn new things from people on your team that you hired, that like you don’t have the same background as them. So, it just improves humanity basically.

Steve: So how do you think about like what the – if the team is kind of eclectic now and you have all these different mixes of skills and backgrounds and aptitudes, how do you think about what the collective should be? How do you think about the mix?

Leanne: Of a whole team?

Steve: Yeah

Leanne: So, I do miss the team I used to have which had both research and analytics. I think being able to have people on your team who do data science, or who are more like business analysts who know how to count things up and make charts, or someone who knows how to mix data together and come up with reports out of it, you get a much faster feedback between the analytics people and the research people. Analytics is like oh I did this, I want to know why? Research is like I can go out and find out why. Or research says, “I need some access to people who have certain characteristics, can you find me a list of them through analytics? Thank you so much, I’ll go out and talk to them and then I’ll come back to you.” In most companies those roles are separate. So, it is here. Analytics is a separate team. We work super well together, but sometimes when you have them both on the team it’s really nice because you’re both at the same team meeting and you’re both seeing what they’re doing. From a research perspective, having people who are all quant or all qual, or halfway in between or know both, is useful because anybody who’s in the job market right now, when they look for their next job the requirements are going up and up and up. We don’t want you to just be somebody who does interviewing. We need you to know how to design a survey. We need you do know how to do SQL queries. We need you to know how to do – like there’s more and more requirements showing up in job descriptions now. And so, if you can be a part of a team where somebody else knows that thing you don’t know, it just – we’re all going to like leave and get another job someday. Nobody stays somewhere forever. So, it will just help you when you’re going out to look for your next job.

Steve: So, skill development, kind of through proximity to your colleagues.

Leanne: Yeah. And it’s also diversity. Like any time, you have a diverse team, whatever you’re defining diversity as, just makes for a better team environment because all these different perspectives around the table from all the different places they came from, just makes for like different things to feed into feedback about a project you’re working on, or something that’s going on in the company. You know, we didn’t all come from the same school or the same program, so we don’t have a difference of thought in the way we approach a problem.

Steve: So, for you, thinking about the teams that you’ve put together – because you’re talking about different kinds of – I don’t know if diversity is the right word, the way you’re using it, but qual and quant are different orientations, or…

Leanne: And also, academic and industry. Or in tech, because I’ve worked in tech for 20+ years, I look for people to come work at the tech company I’m working at who haven’t necessarily worked in tech before. So, look for adjacencies. Like you did research, but you did it for a car company. Or, you did research, but you did it for an agency. So, that they can bring different things into the team. That’s what I mean by diversity. You know research, but you used to be a professor. You know research, but you were working on an astrophysics PhD and decided that’s not what you wanted to do anymore, and you realize things that you learned as a part of your graduate program – so you totally know how to do surveys and how to talk to people and how to do research.

Steve: I had this Twitter conversation with someone the other day and they asked me where is a writing course that I can learn to be better at writing the kinds of things I have to create as a researcher. They can’t find things like that. They can find persuasive writing which is for sales. I kind of scratched my head and said, “I don’t know that that exists.” I went into this little sort of pontification, as Twitter invites you to do, about learning from adjacencies like take a creative writing class. Take a journalism writing class. And in doing that, you would start to see, oh here’s how the lessons of doing this adjacent thing well could apply. That was the best advice that I had. I feel sort of affirmed, from what you were saying, that adjacencies for certain kinds of things bring a certain kind of value.

Leanne: So, the Luma Institute, who put this framework together around design thinking methods so that it’s easy to teach and easy to learn, calls that alternative worlds. And I think that’s a useful way to think about it, that for your person who is trying to figure out how to write a research report for whatever audience they’re trying to serve, well that’s storytelling. It’s sales, it’s marketing. It’s slide development. It’s all these other things. And when you start thinking of it as how would I think of this from the point of view of a salesperson? Or how would I approach this from the point of view of a marketing person who has to put customer stories on the web and sell that to somebody? Or how would I do this as the point of view of the person I am presenting this research to? Like what do I want to hear? It gets you out of the sort of like oh I have to have 10 slides and one has to have my methodology and the end one has to have like recommendations. You can start thinking about it more as like I want to tell this story so that they land on that naturally. Or, they ask me questions about it when I’m done that will answer all those things, so I don’t have to have it all in my report.

Steve: I feel like this is a transgressive notion of expertise though, especially in tech.

Leanne: Yeah. It would be nice for all researchers if they could just come in and do their research and research was valued and anything that researchers said they were learning from customers/users was taken and put into the roadmap. I don’t know anywhere that happens. And because there isn’t anywhere that happens and the way the culture in these teams work, you have to have this sort of like marketing and collaboration and partnership mentality around your work. If you’re a consultant, you get a project, you do it, you collaborate, you sell it a little bit – you’re done. You don’t have to still hang out and make sure it’s being used and follow-up.

Steve: It’s just all sunshine and roses being a consultant, right!

Leanne: Exactly. That’s why I don’t do it anymore.

Steve: I had a coach advising me, in my consulting business, and what they said was – they said many things, but one of them was anytime you start talking with a prospective client, show them – show me how you’ve helped someone like me. That adjacency kind of framing – the advice was that wasn’t as persuasive. So, that could be what have you done in my industry, in my vertical?

Leanne: Yeah.

Steve: You know, and I think researchers, or anyone that just likes to sort of – that gets deep into different kinds of problems, you know we see connections between things that are – you know we know why this problem is like this other problem, even though it’s different.

Leanne: And you want the other people to see that and sometimes it’s hard to show them a lens that will help them see that the way you saw it. Yeah.

Steve: I think the advice was that’s not where you start the relationship. You start the relationship a little more close in. “Oh, this thing that you’re doing. Yes, I have done that.” Even though every problem is new in ways that – it’s very hard to do that.

Leanne: Yes. Yes.

Steve: Sort of starting the – showing relevancy right at the outset of the conversation.

Leanne: When I ran a consulting agency, that’s how I started it. Somebody asked me if I knew how to do something and I’d never done it before and I said I knew how to do it. And all of a sudden, I had a consulting.

Steve: So, your career is just built on lies.

Leanne: Yes.

Steve: Is what we’re saying here, okay.

Leanne: Yes.

Steve: And we’re leaving this in. Yeah, well I think that’s true too. There’s the confidence to explore. I mean can you do this? Yes. Have you done it? Well, no, but I can do it.

Leanne: Yeah. And sometimes the question never comes up, have you done this? It’s just the can you?

Steve: So, that’s – so, we’re talking sort of requests in the consultant side of things. Inside organizations, in any of the ones you’ve worked in, like how are projects formed? Who decides?

Leanne: So many different ways. So, I would say that if you’re beginning research in a company, you’re taking requests and you’re probably using your boss or some other key stakeholders you recognize to decide which to work on. And you’re probably prioritizing those in some sort of backlog, if that’s in a spreadsheet or it’s a Jira or whatever. So, you’re showing people sort of like here’s what’s in my backlog. Here’s what’s coming up next. If you’re a team, the way I prefer to run a research team, which is the way I’m gradually getting it set up here – how it was set up at my last couple of jobs – is to have researchers who are paired or embedded, or whatever you want to call them, but they’re primarily working with a product area. A product team, a feature team, a product area, however your organization is set up. And they are servicing that team and partnering and collaborating with that team. And they own the research for that. But because they’re a part of the research team they also have a holistic view across the organization, because they’re also meeting with other researchers who are doing the same thing and seeing where their work, or where their product area or features sort of overlap, or interlock with each other. And then having some people who are more senior, principal researchers, just working on special strategic projects. So, that could be, depending on a company, that you have someone who is more focused on market research or competitive research. It might be someone who is focused more on research to understand where we should go next? So, sort of like future casting and sort of like looking at what’s going on, finding edge cases, seeing spots for innovations and other sorts of insights. And then also the operations side. People to manage panels of users and recruit people. And then there’s always, depending on how many people there are, some sort of management tier for people management and coaching and that sort of thing.

Steve: And when you talk about embedded you – it’s certainly a frame that you hear described with some regularity, but your words are a little different. Some people talk about embedded, that that researcher is on that team, sort of fully on that team. You’re almost describing like a workstream or a customer or something.

Leanne: Yeah, because the way people work on teams now is not the way that HR software recognizes. So, HR software, for example, recognizes I have a boss. He’s a VP of design. Recognizes I have people who report to me, all the people on my team. Um, maybe recognizes that I have people that might be good for 360 reviews. So, my direct colleagues who also report to my boss. But, that’s such a limited view of what I call my team because – I’ll just talk for me. Like I work with lots of different people around the company who, if I was going to be reviewed, I would want them to weigh in. People on my team – actually, we had a research team meeting this morning. I had them all bring a stakeholder map because I wanted to see – now that I have people that are sort of like embedded or paired within teams, like who are they working with and what does it look like, compared with people on my team whose work is sort of nebulous around all the different people that they’re managing and all the different work that they’re doing. And it turned out that way. So, if you’re a researcher on a team – a lot of people refer to the three-legged stool, but if you have a researcher it’s a four-legged stool. So, you have someone from research, from engineering, from product and from design, all working together with like a common focus on how are we going to get this out the door and make sure it meets customer needs, solving a problem, all that sort of stuff. If you are somebody in operations or a principal researcher working on a strategic project, you just have a mess of people around you. And so, all these people sort of like vacillate in between teams. Designers do the same thing. Designers are a part of the scrum team and they’re a part of the design team. Engineers are a part of a team for some sort of product or feature. They’re also a part of an engineering team. And if gets even more complicated when your manager is in one city and you’re in another city. Because if you’re enough time zones away you probably have somebody who is sort of like your dotted line manager in that city to help you with like all the things around your job. For example, I have a researcher in Barcelona. We can’t look over each other’s shoulders. There’s only two hours a day when we’re both at work. So, there’s a director of design in that office who is her sort of – who is her manager there. So, if she has questions about how does this work? Or how do I do this? Or who is in charge of this? Or help me review this? He can do that. But no HR software recognizes that. It doesn’t recognize she basically has two bosses. It doesn’t recognize that I have one boss, but actually these three other people are kind of my bosses too. It doesn’t recognize that that designer has a boss for whoever is running that scrum team, but that designer also reports to a design manager or director. So, most people work that way now. There’s very few people inside companies anymore who will say they belong to one team and that’s all they belong to. Everybody belongs to like multiple teams and we have this archaic structure around this person decides like your performance review and hopefully gets input from all these people. But, just lines people up in like lines and doesn’t realize that it’s a little more messy than that.

Steve: So, does that gap between the reality and the structure of the software, what’s – are there consequences to that for how we work?

Leanne: I don’t think there’s consequences. I think it’s just the way it is. I think there’s more 360 reviews now, just in my experience, then there were 20 years ago, because you work with so many other people now. I can remember jobs from early to mid-90’s. Nobody was doing 360 reviews of any of the jobs I had then. Like I worked with a set of people in a geographical space. I didn’t work with anybody else. I think it’s just more like it is what it is and things have come in to help manage and understand that, but we, the people who sort of like do to the sort of organizational management, haven’t figured out how to accurately represent that in a structure of the software that manages the company. Some companies try to go – what was it, like Zappos, that decided nobody had a manager. That seems like part of a reaction to that – Holacracy, that’s what it was called.

Steve: So, we got into this because we were talking about what does it mean to be – there sometimes seems a tension between researchers are embedded, or researchers are centralized. And I think what you’re saying is they are neither, or both. And that that’s how work happens.

Leanne: Yeah, and you have to look at the culture of your org. How does your org work? Who wants to participate and who wants – it’s that old RACI model. Like who wants to be involved? Who wants to be informed? Who wants to partner with you?

Steve: So, wait.

Leanne: Do you know the RACI model?

Steve: I know it, but I need to clarify it.

Leanne: It’s where you define – I might not get this right. It’s like where you define who is responsible/accountable, who is interested in consulting on it and who just wants to be informed. So, if you’re thinking about a project, you always have people who sort of fall into those categories. So, I just think it depends – some cultures are very top down and as you go up the ladder they only want to be informed and not as involved. Some cultures are more participatory and as you go up the career ladder, everybody wants to participate and know and be involved. So, it just depends where you land and get a sense of who just needs an inform, who actually wants to come along and sit with you while you do a remote interview, and travel 3 blocks away to visit the customer. You know how far up the ladder does that go?

Steve: So, coming into these organizations that have done very little with research to date – like understanding the culture, it seems like that’s got to be an initial step to get the lay of the land?

Leanne: Yeah. Who’s making decisions and what data are they using to make those decisions? Do they want any more data? Usually the answer is yes. Anybody wants to know more to make better decisions and, in a corporation, make more money. So, figuring out who those people are, where the gaps are, what they need to know. And then another thing you can do when you start at a company is be the researcher yourself of that org because what you’ll start to notice, just like any research project, is that you’ll start hearing the same thing from multiple people. So, you’ll start hearing like, “the one thing we’ve never figured out is whether people really like blue or red better.” You know there’s always something that’s like the one thing where people aren’t aware that everybody has the same question, but because you got them to open up and talk and had casual conversation and used all your interview tips and tricks to interview multiple, different people who don’t work with each other, you’ll start to see a theme around something that everybody wants to know, but nobody knows, but nobody is like trying to figure it out because they don’t know everybody else wants to know. I’ve seen that at every company. We’re working on a project like that here right now where there’s basically two things that I heard across the entire company when I first came here. So, I was like okay, that’s where I’m going to start some key research projects and then I’m going to bring in some people who are interested in it. And then as I hire people in, I’m just going to start attaching researchers to different product areas and work with the people who run design to sort of think about which designers need the most help right now. Or, what meets the business directives, or the key priorities, or the goals, or whatever – the product org of the company.

Steve: Are there stages of evolution? I don’t know if it’s milestones or some framework that at these points you’re kind of coming in where there’s very little and you start building in a way that is specific to what you’re learning about the culture, are there stages that you could identify that you’re passing through or moving towards?

Leanne: Oh, yes. So, once you get sort of like programs and tools set up you don’t really have to do as much anymore. So, programs and tools like is there decent survey software? Do we have access to users? How do we access them? Is there a panel? When we do research, do we have standard agreements that we sign with the humans that we’re doing research with? Repositories – where are we storing everything? How are we storing it? How are we accessing it? Do we need transcripts? Do we have a tool for that? Do we need to do card sorting? Do we have a tool for that? Just sort of like sorting out all the tools and processes for the practice. What was the question? I got lost on a rabbit hole of process and tools.

Steve: You were answering the question. So, what are the stages or milestones?

Leanne: Oh yeah, okay. So, looking at what’s needed that is more like capital improvements. So, like we need a new roof. Okay, we’ll get a roof, it will last for 20 years. Done. So, things like how you pay incentives and research agreements, that’s sort of like the roof. Then you need like everything that pays the electric bill and the gas bill all the time. So, those are projects. So, what projects need attention right now. And you learn that from interviewing sort of like all the key stakeholders you’re going to work with, which in my example earlier was once you interview a whole bunch of them you start to find out that there are some key things that everybody wants to know. And then figuring out, either with your boss, if she or he knows, or with key stakeholders, like if I was to start bringing in researchers one at a time, where would we start? Who would benefit the most? What would be most impactful? Or said another way, sort of like what are the things launching soon, or later? What are the business goals, or what’s in the strategy? Or what are the business directives? Or all those sort of things that make a company run. And that helps you because then you’re making sure you’re focusing on something that is actually going to eventually affect the company’s bottom line. So, when the research comes out and it helps that, then you’ve been like okay, this is how we do it at this company. We’re not academic. We don’t go sit back and sort of think about things and just, you know, go research things out of curiosity. We look at where the company is going to make sure we’re helping it and helping it do better with the research that we’re doing.

Steve: And then what’s a five year – that’s a presumptive question. They’re all presumptive questions. What’s sort of the horizon for which you would have vision? The time horizon?

Leanne: For the times that I’ve come into a company and there’s been no research, or one or two, like here, individual researchers, what I’ve told people is it’s going to take 12-18 months, with budget and with head count, to get this into a modern research practice. And here’s how we’ll do it incrementally and here’s how we’ll check in and benchmark and measure it along the way. When I was at Autodesk and I inherited a team and then I sort of hired in and added on to it, it was much different. These were really old products that had a really amazing install base. And some of them were market leaders. It wasn’t like companies that were two years old, five years old or ten years old, or fifteen years old. Like, once you’re at 30 years you’re settled in a certain way. So, I did different things. I tried to better understand the products and better understand the company to see where we would could add value. And then also, because that was a much larger company, see where research was happening across the company to see if we could sort of like combine efforts.

For example, I went to some company that’s like 25 years old, that has say 1,000 researchers. Because there are some companies out there now that have that many. And I was plopped in as a director of research with 20 to 50 direct reports. It’s a much, much different role than coming into the company and setting something up. They’re you’re coming in and you’re saying how is this done? Is there any way it could be improved? Is there any way I could sort of like do – no, there isn’t? Okay. So, how do we prioritize things? How do we have impact? How do we collaborate and share out and be good corporate citizens and be a part of the team?

I’ve realized, with this third job that I’m on now, after doing consulting for many years, that I do really like growing things. There’s a challenge to it that like gets me into work every day. Like, oh, we have to figure this out. Oh, there’s lots of problems to solve. Oh, how are we going to prioritize this? Or how are we going to solve for this? You know, how am I going to find someone to fill this role?

Steve: You just listed a bunch of things as positives that someone – if you’re in a different emotional state or a different sort of mindset – could frame those all as negatives.

Leanne: Yeah. Yeah. You have to understand the challenge you like to work on. Is the challenge I want to be the person who works with the team to help them figure out what they need to solve? Or do I want to solve an organizational challenge while also working on business priorities and solving problems?

Steve: So, this is something that kind of sparks you?

Leanne: Yeah. And I didn’t realize that until – it was like 3 months ago or so. We hosted a breakfast here and somebody was asking me what I was doing, and they were like, “Well it sounds like you’re really good at that because you’ve done that three times now, and you started your own company.” And I was like, oh, that’s the first time I realized like this is my thing that I like to do.

Steve: Yeah. That’s great.

Leanne: Sometimes you have to be almost 50 to have finally found yourself.

Steve: Right. It’s the journey, not a destination, right.

Leanne: Yes.

Steve: I’m sure you’re going to keep finding yourself.

Leanne: Yes.

Steve: Anything else that I should have asked you about that you want to share with us? The collective us. I know it’s just you and I in a room.

Leanne: Well, there is one other thing I’ve noticed in the past few years. I have used a community platform, that’s sold by a company and meant for marketing and brand teams, and have used it for product and design research. There’s something about – so, support always has its help forums. Marketing and brand usually do focus groups, but now sometimes they run online communities where you can get badges and win things for participating in their research, or telling stories, or whatever. There’s a space in between that people who are on research teams at companies sometimes use and sometimes don’t. So, a typical team might have a panel, particularly if you’re B2B and it’s hard to get access to people. Or even if you’re a consumer-focused company and you just want to make sure you have like your 10,000 people who you can access at any time to easily invite to research. But I think there’s something about the way that people on social media interact with each other in a group in one-to-many fashion and many-to-many fashion that is happening in some ways in research teams, and sometimes not. So, if you’re a researcher and you can, at any time, poke into a community that belongs to you – this is my panel, but your panel, they can’t all see each other. In a community they can see each other. And so, it acts a bit like a support forum, a bit like this marketing and branding, sort of like what you think of our new colors sort of thing. Or this is our new video for our marketing campaign. But instead it’s for product and design. I’ve noticed there’s a certain fear of it – oh my God, if we get our users and customers to talk to each other, what will they say about us? But I think if you get over that fear and you open it up and put together a private space where they’re all talking together and you and your product design, engineering, marketing sales, whatever partners are also in there, you create this sort of like brand loyalty and product loyalty that you don’t necessarily get just from marketing brand and sales and just from support. So, I’ve done that in my last two companies, created this sort of community where people could see each other, and we could interact with them in little like pop-up sort of projects. Haven’t done it here yet. But when I talk to the vendor that I’ve used for doing this, the last time they demoed it for me, I saw that the examples they were giving me were for a product and design team. So, I was like huh, you’re now selling to product and design teams. So, there’s something going on with people picking that up and doing that.

Steve: How do you set people’s expectations, the people that are participants in this community, for what – as you said, it has elements of other kinds of communities that are maybe more familiar…

Leanne: Well, you have to make sure, particularly in business settings, that – you know don’t share anything that you wouldn’t want your competitors to see, because you don’t know if your competitor is in this community, or not. We’re focusing on sort of like your work and your use cases of these sorts of technology. Don’t disclose anything that you shouldn’t be disclosing. But, if you’re in a space where people are using a technology where they get a lot out of finding out how other people are using it too, then you create relationships with each other and then you create your own relationship with them. Another downside is you can get bullies, so you need to have guidelines. We can kick you out if you behave in one of these ways. And with panels, because it’s the company to the user or customer, and that’s it, it’s harder to find sort of like an intrinsic incentive for someone to participate in research. In a community you usually don’t have to have any incentives that are hoodies or cash or gift cards or whatever because the intrinsic incentive is that they’re getting value from other people being in there. And then every once in a while being able to talk to a product manager or talk to a designer or talk to somebody else. So, I think that – I don’t know if that’s something that will keep going, but I see it as a way, particularly in B2B research, to provide value for your customers that actually helps out your sales people and gives your product people and design people a way to interact with users and then also give the user support that they’re probably also getting over in the support forum, but a different kind of support because you feel like you’re part of a group.

Steve: It’s compelling to me to think about just blurring the different kinds of interactions that companies have with people.

Leanne: Yeah

Steve: Like you said, there’s marketing, there’s research, there’s support and we structure the companies around those kinds of functions and the tools and so on.

Leanne: Yeah. And it can make it hard to see the end-to-end customer experience. Yeah.

Steve: What – maybe we can rewind. Like how did you end up in research as a profession?

Leanne: Back, when I graduated from college, in the early 90s, I went to work for a startup cellphone company as a statistician. My background is in statistical computing and economics from my university education. So, I was analyzing what were then considered very, very large sets of data – terabytes of data. I was running a neural net program on a Sun SPARCstation. I don’t even think it had a gigabyte of RAM. Yeah. It had two processors. It was amazing. It would take like 4 to 6 days to run something, to spit out a model. And if I got it wrong I had to do it all over again. What I was doing was I was working in a marketing organization and I was there to help them like figure out things like who should we sell call waiting to? Like should we just sort of try to sell it to everyone? Or should we target people? So, I was figuring out targeted lists based on behavior that we saw in cellphone use data, to help the marketing team understand who to sell things to. When they did that, what I realized was I didn’t know why the people were buying it? Like, just because I could predict like because you have these like usage analytics, it makes you more likely to buy, but why is that? And a couple of things happened then. The company I was working for got bought by AT&T, became what is now AT&T Wireless Services. When you get bought by a much larger company, a bunch of chaos ensues. Your job often changes. Your boss often changes. I was very young. I was in my early 20s. And at that exact time Mosaic came out and the web became graphical.

So, there were all these things happening that occurred at this point in my life where I was like, huh, this is my first job out of college, but now I’m interested in knowing why these things happen? I just taught myself HTML. I figured out how to put together a webpage. Oh my goodness, there’s this thing called Match.com and I met this girl and she lives in San Francisco and I want to date her. Oh, I’ll just go take this job at this company in San Francisco, this web company that has like 20 people and they’re making websites and they’re from all different sorts of places. And I became a QA manager there. Once again, sort of being asked, “can you do this?” Not, have you done this. And I said, “yes.” And then from QA realizing that we didn’t really know what the users were doing. And I, at this point, had never heard of user research. But I was friends with people who were doing that at different companies. So, I asked them about it. And then I left that company and went to a startup. Was also a QA manager. Got laid off when the company didn’t get its next round of funding and went and did some consulting. Doing like web development for the summer. Set up a company that I had intended to be a company that did QA for web apps and somebody came to me and said, “can you do some researcher interviews with users while you’re doing browser testing?” And I said, “yes, we can.” And I was like here’s my chance. I’ve always wanted to know why people are doing things. And so, I happened into it that way. And in the mid to late 90s anybody could do that. You could teach yourself PHP. You could declare yourself to be a web marketer. You could say I’m an information architect. Like you just said what you were, and you could just do it. So, I quickly went and found some friends and said, “I have this consulting project. I don’t know what to do, help me out.” And they were like well you do AB…like the things I just mentioned. You have to do A and B and C and D. I was like okay. So, I did A, B, C and D. I did the project. I got paid. I thought it was amazing. I was like I want to do this again. I was like a kid who gets ice cream for the first time. Like, I will have more of that. And that’s how it got started. In consulting, what you need, from my experience, is you need one or two projects as reference projects and then that helps you get the next project and the next project and the next project. And then it was 17 years later.

Steve: Wow. You’re saying that there was a time when you could sort of declare yourself to be this thing.

Leanne: Yeah. When the web was new. When this new technology was coming out and consumers were grabbing onto it and companies were seeing that money could be made there and – people think now that there are more jobs than people, but then there really was more jobs than the labor market could supply because the labor market didn’t have the skills and the education system wasn’t educating people with those skills. For example, in looking for research interns for this coming summer, I interviewed people who were getting degrees in user research. Like a Bachelors of Science in user research. And I was like wow, like companies are now preparing people and there’s a whole education system out there in these D Schools and I Schools and everything now that didn’t exist in the 90s.

Steve: Right. I sometimes feel like oh I’m – ‘cuz I came up through the same system, or lack of systems, that you did.

Leanne: Yeah. If I wanted to do this job now – say, if I had gotten my Masters at the Berkeley I School, or whatever, and I went and got a job as like a senior researcher at a tech company, and I worked myself up, way up, it would be much, much different than my experience of teaching myself and figuring it out.

Steve: What’s lost or gained in that evolution, if anything?

Leanne: Well, I can clearly see that my sort of like – YouTube didn’t exist then, but it would have been like the university of YouTube, the way that I went about teaching myself how to do research and starting up a whole consulting agency around it – it would have been much different if I had been schooled in a particular way in how to do research. I think some things would have been lost because it wouldn’t have been sort of like me and a couple of other people just sort of figuring out how to do something. You’d come up with new stuff. But I think some things might have been gained. I might have more confidence – like I have a Masters in this, so I know how to do it. Or, I did my thesis on this and I am – yeah. There’s a certain amount of imposter syndrome that goes around, even if you have that education. But sometimes when you don’t, and you make it up yourself, you can doubt whether or not you really know what you’re doing. Fake it ‘til you make it.

Steve: I remember just being in a consultancy kind of around that period of time and that every prospective client that came in we would have to write a proposal for what we were going to do. And it just felt like writing a Master’s thesis every time. Or some just impossible thing to give birth to. Like explaining what it was, deciding the steps, trying to articulate those steps. We just didn’t have any point of reference and so we were sort of figuring it out every time until I think at some point we started to settle in and like oh, here’s kind of what – what it looks like. And now like writing a proposal is not hard. There’s many other hard things.

Leanne: But now you have templates and you can copy it.

Steve: And you know – I mean I have a narrative that runs through my head that oh yeah it kind of goes like this. We have more tribal knowledge as well as…

Leanne: Yes. More people to reach out to, to get examples from too, than used to exist.

Steve: Right. In this era, this consultancy, not only were we figuring it out, we were doing so in pretty much isolation. Because you knew a few people, but most of us at that era – there wasn’t a community of researchers to kind of connect to.

Leanne: Yahoo Groups did not exist yet. Google Groups was not there. Google wasn’t even here.

Steve: Yeah. This is great. It’s a very interesting conversation. I learned a lot.

Leanne: It was great chatting with you Steve.

Steve: I appreciate you taking the time.

Leanne: Absolutely.

Steve: Anything else? Are we done? Any last thing?

Leanne: It’s fantastic. Yes. Perfect ending.

Steve: Mic drop. Alright, thank you.

That’s it for this episode of Dollars to Donuts Go to portigal.com/podcast for the transcript as well as links for this episode. At the website you can also check out all the other episodes, and subscribe to the podcast on iTunes, Spotify, and all the usual places. My books are available at Amazon and rosenfeldmedia.com. Our theme music was written and performed by Bruce Todd.

In the final episode of the season I speak with Monal Chokshi, Head of User Experience Research at Lyft. We discuss traditional paths to a user research career, creating routines for meeting different types of users, and the emergence of leadership roles in user research.

In this episode I speak with Kate Lawrence, Vice President of User Research at EBSCO Information Services. Our conversation covers where to place user research in the organization, emotions in fieldwork, and empowering others to advocate for information literacy.

This episode features Pree Kolari, the Senior Director of Design Strategy and Research at eBay. We talk about the career arc of a researcher, having impact on the product, and breaking down organizational walls.

This episode features Gabe Trionfi, the Manager of Research at Pinterest. We discuss the evolution of user research, collaboration between disciplines and the journey versus the destination.

In this episode I chat with Elizabeth Kell, the Senior Director of User Research at Comcast. We talk about the growth of Comcast's user research practice, essential soft skills for research candidates, and putting a human face on the people that use your products.

Today I chat with Kavita Appachu, the Senior Manager of User Experience Research at Kelley Blue Book. She describes the different roles she's had in different organizations, moving from design to research, and explains the change effort underway at Kelley Blue Book.

In today's episode I speak with Aviva Rosenstein, the Senior Manager of User Experience Research at DocuSign. We explore how to make all types of research actionable, the benefit of doing your own recruiting, and the evolution from building a usability lab to having an in-house research capability.

We kick off the second season with Judd Antin, the Director of Experience Research at Airbnb. Judd and I speak about their model for embedding talented generalists with product teams, skill-sharing among researchers, and just what exactly makes research "sexy."