Experiencing Data w/ Brian T. O’Neill

Today I’m joined by Vera Liao, Principal Researcher at Microsoft. Vera is a part of the FATE (Fairness, Accountability, Transparency, and Ethics of AI) group, and her research centers around the ethics, explainability, and interpretability of AI products. She is particularly focused on how designers design for explainability. Throughout our conversation, we focus on the importance of taking a human-centered approach to rendering model explainability within a UI, and why incorporating users during the design process informs the data science work and leads to better outcomes. Vera also shares some research on why example-based explanations tend to out-perform [model] feature-based explanations, and why traditional XAI methods LIME and SHAP aren’t the solution to every explainability problem a user may have.

Highlights/ Skip to:

• I introduce Vera, who is Principal Researcher at Microsoft and whose research mainly focuses on the ethics, explainability, and interpretability of AI (00:35)

• “Human-Centered XAI: From Algorithms to User Experiences” Presentation

In this episode, I give an overview of my PiCAA Framework, which is a framework I shared at my keynote talk for Netguru’s annual conference, Burning Minds. This framework helps with brainstorming machine learning use cases or reverse engineering them, starting with the tactic. Throughout the episode, I give context to the preliminary types of work and preparation you and your team would want to do before implementing PiCAA, as well as the process and potential pitfalls you may run into, and the end results that make it a beneficial tool to experiment with. 

Highlights/ Skip to:

• Where/ how you might implement the PiCAA Framework (1:22)

• Experiencing Data Episode 106 with Jeremy Utley

Today I’m wrapping up my observations from the CDOIQ Symposium and sharing what’s new in the world of data. I was only able to attend a handful of sessions, but they were primarily ones tied to the topic of data products, which, of course, brings us to “What’s a data product?” During this episode, I cover some of what I’ve been hearing about the definition of this word, and I also share my revised v2 definition. I also walk through some of the questions that CDOs and fellow attendees were asking at the sessions I went to and a few reactions to those questions. Finally, I announce an exciting development on the launch of the Data Product Leadership Community.

Highlights/ Skip to:

• Brian introduces the topic for this episode, including his wrap-up of the CDOIQ Symposium (00:29)

• “I think generally what’s happening is the technology continues to evolve, I think it generally continues to get easier, and all of the people and cultural parts and the change management and all of that, that problem just persists no matter what. And so, I guess the question is, what are we going to do about it?” — Brian T. O’Neill (03:11)
  

• “I also would say on the data mesh topic, the feeling I got from people who had been to this conference before was that was quite a hyped thing the last couple years. Now, it was not talked about as much, but I think now they’re actually seeing some examples of this working.” — Brian T. O’Neill (16:25)

• “My current v2 definition right now is, ‘A data product is a managed, end-to-end software solution that organizes, refines, or transforms data to solve a problem that’s so important customers would pay for it or exchange something of value to use it.’” — Brian T. O’Neill (19:47)

• “We know [the product is] of value because someone was willing to pay for it or exchange their time or switch from their old way of doing things to the new way because it has that inherent benefit baked in. That’s really the most important part here that I think any data product manager should fully be aligned with.” — Brian T. O’Neill (21:35)

• Episode 67

Today I’m answering a question that was submitted to the show by listener Will Angel, who asks how he can prioritize and scale effective discovery throughout the data product development process. Throughout this episode, I explain why discovery work is a process that should be taking place throughout the lifecycle of a project, rather than a defined period at the start of the project. I also emphasize the value of understanding the benefit users will see from the product as the main goal, and how to streamline the effectiveness of the discovery process. 

Highlights/ Skip to:

• Brian introduces today’s topic, Discovery with Data Products, with a listener question (00:28)

• “Discovery work, to me is something that largely happens up front at the beginning of a project, but it doesn’t end at the beginning of the project or product initiative, or whatever it is that you’re working on. Instead, I think discovery is a continual thing that’s going on all the time.” — Brian T. O’Neill (01:57)
  

• designingforanalytics.com/podcast: https://designingforanalytics.com/podcast

Today I’m chatting with Peter Everill, who is the Head of Data Products for Analytics and ML Designs at the UK grocery brand, Sainsbury’s. Peter is also a founding member of the Data Product Leadership Community. Peter shares insights on why his team spends so much time conducting discovery work with users, and how that leads to higher adoption and in turn, business value. Peter also gives us his in-depth definition of a data product, including the three components of a data product and the four types of data products he’s encountered. He also shares the 8-step product management methodology that his team uses to develop data products that truly deliver value to end users. Pete also shares the #1 resource he would invest in right now to make things better for his team and their work.

Highlights/ Skip to:

• I introduce Peter, who I met through the Data Product Leadership Community (00:37)

• “I’m a big believer that the majority of analytics in its simplest form is improving business processes and decisions. A big part of our discovery work is that we align to business areas, business divisions, or business processes, and we spend time in that discovery space actually mapping the business process. What is the goal of this process? Ultimately, how does it support the P&L?” — Peter Everill (12:29)
  

• “The important way that data and analytics can help an organization firstly is, understanding how that organization is performing. And essentially, performance is how well processes and decisions within the organization are being executed, and the impact that has on the P&L.” – Peter Everill (20:24)

• “The great majority of organizations don’t allocate that percentage [20-25%] of time to discovery; they are jumping straight into solution. And also, this is where organizations typically then actually just migrate what already exists from, maybe, legacy service into a shiny new cloud platform, which might be good from a defensive data strategy point of view, but doesn’t offer new net value—apart from speed, security and et cetera of the cloud. Ultimately, this is why analytics organizations aren’t generally delivering value to organizations.” – Peter Everill (25:37)

• “The only time that value is delivered, is from a user taking action. So, the two metrics that we really focus on with all four data products [are] reach [and impact].” – Peter Everill (27:44)

• “In terms of benefits realization, that is owned by the business unit. Because ultimately, you’re asking them to take the action. And if they do, it’s their part of the P&L that’s improving because they own the business, they own the performance. So, you really need to get them engaged on the release, and for them to have the superusers, the champions of the product, and be driving voice of the release just as much as the product team.” – Peter Everill (30:30)

• On hiring DPMs: “Are [candidates] showing the aptitude, do they understand what the role is, rather than the experience? I think data and analytics and machine learning product management is a relatively new role. You can’t go on LinkedIn necessarily, and be exhausted with a number of candidates that have got years and years of data and analytics product management.” – Peter Everill (36:40)

• LinkedIn: https://www.linkedin.com/in/petereverill/

Today I’m continuing my conversation with Nadiem von Heydebrand, CEO of Mindfuel. In the conclusion of this special 2-part episode, Nadiem and I discuss the role of a Data Product Manager in depth. Nadiem reveals which fields data product managers are currently coming from, and how a new data product manager with a non-technical background can set themselves up for success in this new role. He also walks through his portfolio approach to data product management, and how to prioritize use cases when taking on a data product management role. Toward the end, Nadiem also shares personal examples of how he’s employed these strategies, why he feels it’s so important for engineers to be able to see and understand the impact of their work, and best practices around developing a data product team. 

Highlights / Skip to:

• Brian introduces Nadiem and gives context for why the conversation with Nadiem led to a two-part episode (00:35)

• “So, data product management [...] is a combination of different capabilities [...]  [including] product management, design, data science, and machine learning. We covered this in viability, desirability, feasibility, and datability. So, these are four dimensions [that] you combine [...] together to become a data product manager.” — Nadiem von Heydebrand (02:34)

• “There is no education for data product management today, there’s no university degree. ... So, there’s nobody out there—from my perspective—who really has all the four dimensions from day one. It’s more like an evolution: you’re coming from one of the [parallel business] domains or from one of the [parallel business] fields and then you extend your skill set over time.” — Nadiem von Heydebrand (03:04)
  

• “I think every data leader out there has an Excel spreadsheet or a list of prioritized use cases or the most relevant use cases for the business strategy… You can think about this list as a portfolio. You know, some of these use cases are super valuable; some of these use cases maybe will not work out, and you have to identify those which are bringing real return on investment when you put effort in there.” – Nadiem von Heydebrand (19:01)

• “I’m not a magician for data product management. I just focused on a very strategic view on my portfolio and tried to identify those cases and those data products where I can believe I can easily develop them, I have a high degree of adoption with my lines of business, and I can truly measure the added revenue and the impact.” – Nadiem von Heydebrand (26:31)

• “As a true data product manager, from my point of view, you are someone who is empathetic for the lines of businesses, to understand what their underlying needs and what the problems are. At the same time, you are a business person. You try to optimize the portfolio for your own needs, because you have business goals coming from your leadership team, from your head of data, or even from the person above, the CTO, CIO, even CEO. So, you want to make sure that your value contribution is always transparent, and visible, measurable, tangible.” – Nadiem von Heydebrand (29:20)

• “If we look into classical product management, I mean, the product manager has to understand how to market and how to go to the market. And it’s this exactly the same situation with data product managers within your organization. You are as successful as your product performs in the market. This is how you measure yourself as a data product manager. This is how you define success for yourself.” – Nadiem von Heydebrand (30:58)

• Mindfuel: https://mindfuel.ai/

The conversation with my next guest was going so deep and so well…it became a two part episode! Today I’m chatting with Nadiem von Heydebrand, CEO of Mindfuel. Nadiem’s career journey led him from data science to data product management, and in this first, we will focus on the skills of data product management (DPM), including design. In part 2, we jump more into Nadiem’s take on the role of the DPM. Nadiem gives actionable insights into the realities of data product management, from the challenges of actually being able to talk to your end users, to focusing on the problems and unarticulated needs of your users rather than solutions. Nadiem and I also discuss how data product managers oversee a portfolio of initiatives, and why it’s important to view that portfolio as a series of investments. Nadiem also emphasizes the value of having designers on a data team, and why he hopes we see more designers in the industry. 

Highlights/ Skip to:

• Brian introduces Nadiem and his background going from data science to data product management (00:36)

• “The product mindset already says it quite well. When you look into classical product management, you have something called the viability, the desirability, the feasibility—so these are three very classic dimensions of product management—and the fourth dimension, we at Mindfuel define for ourselves and for applications are, is the datability.” — Nadiem von Heydebrand (06:51)
  

• Mindfuel: https://mindfuel.ai/

Today I’m chatting with Kyle Winterbottom, who is the owner of Orbition Group and an advisor/recruiter for companies who are hiring top talent in the data industry. Kyle and I discuss whether the concept of data products has meaningful value to companies, or if it’s in a hype cycle of sorts. Kyle then shares his views on what sets the idea of data products apart from other trends, the well-paid opportunities he sees opening up for product leaders in the data industry, and why he feels being able to increase user adoption and quantify the business impact of your work is also relevant in a candidate’s ability to negotiate higher pay. Kyle and I also discuss the strange tendency for companies to mistakenly prioritize technical skills for these roles, the overall job market for data product leaders, average compensation numbers, and what companies can do to attract this talent.

Highlights/ Skip to:

• Kyle introduces himself and his company, Orbition Group (01:02)

• “I think data products, obviously, there’s starting to get a bit of hype around it, which I’ve got no doubt will start to lead organizations to look down that route, just because they see and hear about other organizations doing it. ... [but] what it’s helping organizations to do is to drive adoption.” — Kyle Winterbottom (05:45)
  

• Kyle’s LinkedIn: https://www.linkedin.com/in/kylewinterbottom/

Today I’m chatting with Phil Harvey, co-author of Data: A Guide to Humans and a technology professional with 23 years of experience working with AI and startups. In his book, Phil describes his philosophy of how empathy leads to more successful outcomes in data product development and the journey he took to arrive at this perspective. But what does empathy mean, and how do you measure its success? Brian and Phil dig into those questions, and Phil explains why he feels cognitive empathy is a learnable skill that one can develop and apply. Phil describes some leading indicators that empathy is needed on a data team, as well as leading indicators that a more empathetic approach to product development is working. While I use the term “design” or “UX” to describe a lot of what Phil is talking about, Phil actually has some strong opinions about UX and shares those on this episode. Phil also reveals why he decided to write Data: A Guide to Humans and some of the experiences that helped shape the book’s philosophy. 

Highlights/ Skip to:

• Phil introduces himself and explains how he landed on the name for his book (00:54) 

• “If you take success in itself, this is about achieving your intended outcomes. And if you do that with empathy, your outcomes will be aligned to the needs of the people the outcomes are for. Your outcomes will be accepted by stakeholders because they’ll understand them.” — Phil Harvey (05:05)
  

• “I wanted to write code; I didn’t want to deal with people. And you feel when you can do technical things, whether it’s machine-learning or these things, you end up with the ‘I’ve got a hammer and now everything looks like a nail problem.’ But you also have the [attitude] that my programming will solve everything.” – Phil Harvey (14:48)

• “This is what startup-land really taught me—you can’t do everything. It’s very easy to think that you can and then burn yourself out. You need a team of people.” – Phil Harvey (15:09)

• “Let’s listen to the users. Let’s bring that perspective in as opposed to thinking about aligning the two perspectives. Because any product is a change. You don’t ride a horse then jump in a car and expect the car to work like the horse.” – Phil Harvey (22:41)

• “Let’s say you’re a leader in this space. … Listen out carefully for who’s complaining about who’s not listening to them. That’s a first early signal that there’s work to be done from an empathy perspective.” – Phil Harvey (25:00)

• “The perspective of the book that Noelia and I have written is that empathy—and cognitive empathy particularly—is also a learnable skill. There are concrete and real things you can practice and do to improve in those skills.” – Phil Harvey (29:09)

• Data: A Guide to Humans: https://www.amazon.com/Data-A-Guide-to-Humans/dp/1783528648

Do you ever find it hard to get the requirements, problems, or needs out of your customers, stakeholders, or users when creating a data product? This week I’m coming to you solo to share reasons your stakeholders, users, or customers may not be making time for your discovery efforts. I’ve outlined 10 reasons, and delve into those in the first part of this episode. 

In part two, I am going to share a big update about the Data Product Leadership Community (DPLC) I’m hoping to launch in June 2023. I have created a Google Doc outlining how v1 of the community will work as well as 6 specific benefits that I hope you’ll be able to achieve in the first year of participating. However, I need your feedback to know if this is shaping up into the community you want to join. As such, at the end of this episode, I’ll ask you to head over to the Google Doc and leave a comment. To get the document link, just add your email address to the DPLC announcement list at http://designingforanalytics.com/community and you’ll get a confirmation email back with the link. 

• Join the Data Product Leadership Community at designingforanalytics.com/thecommunity