Inside DuckDuckGo

In this episode, Jeffrey (Operations) and Anne (People Ops) discuss how performance reviews work in a distributed leadership model, and how the process has scaled as we’ve grown.

If you have feedback on Duck Tales, or episode ideas, email us at [email protected].

Disclaimers: (1) The audio, video (above), and transcript (below) are only lightly edited and may contain minor inaccuracies or transcription errors. (2) This website is operated by Substack. This is their privacy policy.

Jeffrey: Hello everyone, and welcome to Duck Tales, where we go behind the scenes at DuckDuckGo. We discuss the stories, technology, and the people that help build privacy tools for everyone. In each of these episodes you’ll hear from the employees about our vision, product updates, engineering, or things like approaches to AI.

Anne: There’s a lot of

Jeffrey: My name is Jeffrey and I’m on the internal efficiency team at DuckDuckGo.

Anne: Yeah,

Jeffrey: And today I’m joined with my colleague Anne. Anne, do you want to introduce yourself?

Anne: Hi, I’m Anne. I’m on the People Ops team at DuckDuckGo. It’s been about five years now, which I can’t believe, but lots of exciting stuff over that time.

Jeffrey: Great. Today we’re gonna be talking about our internal performance reviews and how we run them. And how would you describe our performance review process to someone who’s never heard of DuckDuckGo?

Anne: Well, I think the important thing to start with is sort of describing DuckDuckGo’s distributed leadership model, because with distributed leadership, that’s going to feed into performance reviews. And so the short answer I would give with that is that there’s no one decider of performance. There’s multiple different ways, multiple different people that give input, as well as a really strict requirement that it be evidence-based. And we’ll get into this a bit later, I’m sure. But we’ve done a lot of work on the evidence-based piece and being an asynchronous organization really helps with that because mostly everything is written down. So that’s that’s the high level that I would give.

Jeffrey: Who do we how do we decide what collaborators get involved with each person’s reviews?

Anne: Yeah, this has been a really interesting question over the years. And I would I would just love to brag a bit about how we used to do it when it was, you know, we were a smaller organization and then talk about how we’ve evolved that because well, I think way long ago it was the organization was small enough to just kind of know. You knew who should be giving input for who. And then the organization grew to the point where we needed help figuring out, right? You can’t one person couldn’t hold it all in their head.

And so you very generously helped us create an algorithm that put out a spreadsheet that basically it ingested information about people’s leadership relationships to one another, their work overlap with one another, and their experience giving feedback, among other things, calculated all of those relationships, also taking into account load.

Because you know, maybe one person should review 50 people, but that’s just too many, right? So balancing those trade-offs and then making a suggestion, which was really cool. And then those suggestions obviously had to be reviewed by humans and still, you know, load adjusted and everything. And there were several rounds of human review. So that was great. And then that also came to the point where we outgrew that because even those suggestions were gonna start fraying at a point, and the human review was cumbersome.

And so we’ve moved, I think quite naturally toward leaning on functional calibrators who are closer to the functional work and then pulling in the objective side of the house, which is sort of like where work is getting executed and having those be the default inputs for people. But yeah, that’s kind of how that’s changed over time. And it’s really, it’s been a really exciting problem, fun problem to be a part of.

Jeffrey: For the most part it’s scaled with as the company has grown and we’ve just had to make small changes here and there.

Anne: Yeah, I it’s as a people person, I also say, you know, people say how do we keep our culture as we grow? And this is similar to that where you say we don’t keep something, right? We s how do we scale something in accordance with our values as we grow? And then this I think would be a really good example of that in a really unique one too.

Jeffrey: Cool. What makes evaluating performance here different from a typical company?

Anne: Well, similar to what I said before, right? Distributed leadership means you have to figure out who it’s not like a managerial chain where the manager writes a performance review with peer feedback and then that gets approved by their manager and that gets approved. You know, it’s not a chain of approvals, it’s a collaborative process, which is also why that was one of the main reasons. There were several, but one of the bigger but bigger blockers for us purchasing performance review software from a company was that none of them had the functionality to have the level of collaboration that we have, like back and forth collaboration. And so there’s that. And then the real just like I said earlier, reliance on data. We are an asynchronous organization that really leverages processes in our project management tool, Asana, to fill the gaps of what a lot of live meetings would be. And I would imagine other Duck Tales have talked about no meeting Wednesdays, which is like for real. At other companies, it’s kind of in air quotes, but I would say here, not all of their companies to be clear, but oftentimes that can slip. But here it’s pretty, pretty legit. And so yeah, we’re generally meeting light, and so everything is in Asana, and you can pull that in to work summaries. And that’s also something you’ve been involved with, is really having a set of validated summary of work over the period of the year with links, right? So you can actually go and dig into the work over the course of the year as needed. And yeah, that’s the other thing I would really be I would say is is unique that the reliance on evidence and links and things you can actually see and look at and review is pretty high.

Jeffrey: Right. You touched on this a little bit, but can you go into how the process has scaled as we’ve grown? Or evolved.

Anne: Yeah, so yeah, that was one of the things is obviously sw figuring out how to fill those collaborative evaluator roles. Another is automation. We’ve really, you know, built our own system that integrates with where work gets done for us, which is Asana. And as a people operator, I feel blessed that I’m just given IT resources to be like, hey, you need this system to work. Here’s some really awesome skilled IT folks to help you make it a reality. And we have some really cool people on the team working to build our own performance review system that integrates exactly with where work happens for us. And so I’d say that is one that we’ve built that system and it’s evolved over time alongside other systems, right? So we have things like our daily asynchronous personal stand-up.

Right. So everybody does a stand-up for themselves every day and that gets recorded. And so then that can filter back into this whole performance review process because you have a record of what you were focused on every day. You have a record of the project you were focused on every week. You we have a whole summary of the relationships of your project advisory, career advisory, everything. You know, we’ve basically been automating more of these things to be centralized in a way that’s digestible for users. And those two things have really evolved together in in a way that’s pretty cool.

Jeffrey: I’m really proud of the our daily stand up we call our daily one must, but it’s been with us for a while now and it’s all asynchronous where people can fill it in in the morning and then everybody who needs to know can see what their stand up is. Cool.

Anne: Yeah, that’s the transparency part too, right? You can kinda go see what anyone’s working on at any time, which is pretty cool.

Jeffrey: How is this process suited for our remote asynchronous first culture?

Anne: Yeah, I mean I think similar to the things I’ve said before too, it’s it’s it fits the culture because for one, we have all of this substrate to work with, right? We have all of the the asynchronous culture is really reliant upon things being written down. And when things are written down, you can really shortcut a lot of the you know, forgetting what happened or whatever, or even just misremembering. And so you know, we have like bias checks that can happen on the data because the data are actually there, obviously all reviewed by people. But yeah, so it’s fit for us for that reason because of what you’re actually working with. And then the collaboration, too, is I think this is just part of asynchronous work in general. If you’re in a live meeting discussing something, it really just kind of depends on your processing speed in the moment for what you’re thinking and deciding on.

And we don’t make decisions as a general rule live. We can have discussions about them. Or maybe if it’s a really gnarly one, but generally decisions are happening in a really thought out way. And that’s the same thing with performance conversations, right? So it’s not just having a live debate about something in a calibration, right? You’re actually looking at, you’re having time to consider things and look at the actual evidence as opposed to just having a live conversation with all of its associated biases. So that’s that’s one thing I would I would say.

Jeffrey: Default to having things written down helps with the evidence. Yeah. What are the challenges with the process?

Anne: Yeah, I guess the flip side of having the everything written down is it’s not actually everything written down, right? Obviously, we still there’s a lot of that happens with like context and in a chat or in a live call. And we don’t want to over rotate on, well, if it wasn’t written down, then it didn’t happen. And that just relies on people’s judgment. And so that does rely on people actually taking the time to look at things and make those judgments. I think we have really smart capable people doing that. So it’s a challenge that just is something that they need to keep in mind. It’s not really something we’ve had to solve for necessarily. And then as we’ve grown, the question, you know, a lot of this is very time-intensive, right? There’s a lot of conversation around each of these pieces and a lot of consideration that goes into it. And then, you know, just the question organizationally is: how can we make this easier for everyone?

As much as you know, we all really value performance reviews and also it can be if you’re someone who has a lot to do, it’s a lot of work. We very much appreciate everyone who puts the work in because everyone cares about doing it well, and doing it well takes a lot of time. So that’s been a challenge for us is how can we keep the quality bar while, you know, making it easier and easier to do every time.

Jeffrey: Mm-hmm. It kinda leads into the last question of what our future plans are. Do you want to go into any of that?

Anne: Yeah, so high level. We’ve done a lot of work over the years to streamline, like I was talking about all of the data, and then it feeds into, you know, we have up to five different like categories of ratings. We call them professional dimensions. And we’ve done a lot of work to sort of categorize those things and how so to streamline the amount of work you have to do as someone who’s evaluating. And all of that work really we’re trying to take it to the next level of having that happen in a real focus time period rather than in combination with other work. And also, we used to give so well, I’ll finish that thought first, which is we’re going to move to a shorter cycle, which is going to be about three weeks. That’s the goal. Where everyone, we talked about our daily standup earlier, where ever if you have a task to do that day for the performance review, your daily stand-up will be write my self-reflection.

Everyone at the company writes it that day, and then they can move on to other things. So that it’s not this kind of organizational cost that’s dispersed across some time. And it also gives people the freedom to really focus on it that day because it is their one must. So you can really put all your attention there. And that follows through the rest of the process. The other piece is, you know, we are we used to do these annual, you know, shout-outs and focus areas, right? Which was the sort of written part of the review.

Sometimes they could be quite long, and that takes a lot of time to review, also. And as we thought about it scaling, and we’re just thinking also about scaling our career advisor function and coaching and you know, leveling up career advisors as we scale, which I think is a very normal company scaling thing, is our annual shout-outs and focus areas right for everyone? Like really, we shouldn’t just be doing this once a year. And of course, we should all be getting coaching conversations in an ongoing way. And so we’re actually building the process around ongoing coaching that happens in folks one-to-ones regularly as opposed to sort of having this heavy text written piece in the performance review. So that’s something we’re experimenting with. And I say experimenting with because everything at DuckDuckGo is an experiment and changes in response to evidence and how it’s going and the needs of the organization, but that’s what we’re looking at next.

Jeffrey: I’m really excited about the shorter time period for it and the focused week that it’ll be great.

Anne: Someone who was one of the highest loaded folks at the organization, I yeah, that you’d be very excited.

Jeffrey: Yeah. All right. I think that covers all of our topic today. Thanks for joining me, Anne. See y’all later.

Anne: Great to see you, Jeffrey. Bye.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit insideduckduckgo.substack.com

In this episode, Beah (Chief Product Officer) and John (Support) discuss our customer support process, why we read all tickets (good or bad), and how we actually use them to improve all aspects of the user experience.

If you have feedback on Duck Tales, or episode ideas, email us at [email protected].

Disclaimers: (1) The audio, video (above), and transcript (below) are only lightly edited and may contain minor inaccuracies or transcription errors. (2) This website is operated by Substack. This is their privacy policy.

Beah: Hello, and welcome to Duck Tales, where we go behind the scenes at DuckDuckGo and discuss the stories, technology, and people that help build privacy tools for everyone. In each episode, you’ll hear from employees about our vision, product updates, engineering, or approach to AI. Today you’re gonna hear about how we do support at DuckDuckGo, and my guest is John. John, do you wanna introduce yourself briefly?

John: Yeah, for sure. Yep, I’m John. I work within our marketing and communications team. And for the past few years, my main goal has been managing and improving how our support operates for the DuckDuckGo subscription and beyond.

Beah: Sweet. So if you’ve used DuckDuckGo support, perhaps you’ve gotten a message from John or one of the other teammates who we’ll learn more about in this call or in this pod. So I’m Beah. If you haven’t met me, I’m on the product team at DuckDuckGo. All right, let’s jump in. I’m gonna ask John some questions and we’ll go from there. So tell me John, just to kind of set the scene, what sort of things do we provide customer support for?

John: A lot. The bulk of the support that we offer is through the DuckDuckGo subscription. So for us who handle support, it means handling things like billing questions, assisting with activation across multiple devices, questions about the plans that we offer, and more technical troubleshooting related things. The subscription can be bought a few different ways, each with its own different path to activation. So a good bit of the support that we offer is helping our users navigate these paths to get more comfortable with our subscription and its features. So some of our great front-end team members built a support form where most of our tickets run through. It was built to be kind of as self-service as possible, but a category is picked and depending on what the selection is, a potential solution gets surfaced. This way, plenty of users can get an answer without really needing to submit anything or wait for us to respond to them. But basically, our help pages kind of sit behind this and there’s links to that support form on every one of our subscription help pages. It’s the first stop for most of these tickets. We’ve been piloting an AI agent on a few help pages and I’m sure we’ll get into that as well.

Beah: So like if we don’t, if we can answer, if we can get people an article right then and there that answers their questions, that’s the best case scenario for everybody because they get an immediate answer, but if not, then we have a process to kind of hand it off to a human or an in-between, which is like AI assisted answers.

John: Exactly. That’s right.

Beah: And maybe I should say like before we had the subscription, which is the first paid product that DuckDuckGo offered, and the only one we offer, we didn’t really have a support team or support process. I mean, we had emails for different things, but like effectively we didn’t have to worry about customer support. And it was the introduction of the paid subscription that sort of like brought this problem to our doorstep and to you. Is that fair to say? Yeah, yeah.

John: Yes. Absolutely. Yeah. Yeah. Perfect.

Beah: Okay, cool. So tell me a little bit more about like how this all works like team-wise, who’s working on this, how have we structured that?

John: Absolutely. We have a small but mighty team. So when that subscription first launched, there were a few of us that kind of rotated handling these support tickets. After a few months, we streamlined it significantly to where I was the only one who was handling things, which probably seemed like it was unfeasible and it really was. Early last year, we brought on an awesome specialist as...

Beah: It, I mean it, it worked, you just didn’t sleep great.

John: Yes, yeah, it worked. It worked. But there was a lot that was dependent on me and it caused a lot of lead time for some people to get responses. It was a lot to manage basically. So we brought on an awesome specialist as we continued to grow. She’s been awesome in the organization and the optimization of our entire end-to-end process. She’s got more support experience than I do by far and is a great Swiss Army knife of skill sets. So with her help, we, along with dozens of other teammates throughout the organization, we kind of overhauled the whole support structure at DuckDuckGo. And then recently, earlier this year, we brought on two more support rockstars to kind of help shoulder the ticket load to ensure that our users aren’t waiting too long before getting a real human response. So they’ve been a great addition. They’ve been working fantastically. And they’ve added an additional support layer to the mix. But yeah.

Beah: Got it. And you, you, you mentioned like lots of other people. I mean, so there’s like the people who are actually writing the emails back. And then I think part of what you’re getting at, tell me if this is right, is like, you know, they are likely in many, not in all cases, but in some cases, like need to go talk to an engineer to understand what’s going on, or like a product manager or somebody else who’s been working on the product directly day in and day out and like they’re just kind of interfacing between like this huge, you know, pool of resources who know different things about DuckDuckGo.

John: Yes, yeah, and that, the whole concept of working out in the open really helps make that a lot easier. So we have engineers and developers that really plug in whenever a support signal points to the need to escalate for a better response that we don’t know or we can’t handle.

Beah: It has been for you.

John: So that direct line to the actual builders that are behind the scenes ensures that we go straight to the source and we don’t have to wade through multiple layers of people to get an answer that’s accurate.

Beah: Gotcha. Cool. So tell me, like, kind of riffing off of that, how does this fit into like our broader approach to customer feedback loops? Like we have this email support system, but we also like interface with users or get feedback from users like on social media or, you know, all these various ways, which I would say for context for listeners, like we really, we like really read that stuff and really pay attention to it. Like we’re pretty, I’d say we’re pretty obsessed in fact with understanding. Like if you’re sending us, if you’re a user and you’re sending us signals about your experience, like we are pretty hungry for that. So yeah, tell me about like the boundary, the interface boundary between like that kind of feed, those customer feedback loops and the support that you’re talking about.

John: Yeah, so it’s a big circle. So essentially, the support piece to me, from what I’ve seen, I feel like I’ve been in every aspect of support, internal and external, that we offer. And this support for the subscription that we do is truly one of the most honest and raw feedback channels that we have. Most of the time, people write in when there’s a problem. Like, you’re not going to be contacting support to be like, hey, great job. Each ticket gets categorized in those categories.

Beah: Though feel free to.

John: Yeah, you’re more than welcome to, yes.

Beah: Yeah.

John: But we don’t even, we don’t handle things that are just a subscription, we handle any and all questions that come in, which I might get, we might get a deluge of tickets with me saying that. But, that said, we get a really wide range of feedback. It’s truthful. It’s unsanitized. It’s anonymous. So one thing that I’ve insisted upon since we launched this is really reading that rougher stuff on purpose. So I thrive on reading through our negative feedback. A good bit of it relates not just to the poor experience with the support team that happened from time to time, but also their experiences with the products that we offer. So both halves of those go to those who can action them. When we see things, multiple complaints about a specific piece of the product, we’ll let the people know who could actually work to fix it. So actions like these help drive rewrites to our help pages that might seem confusing. Clear responses to our users in the future. Updates on our products, bug fixes.

Beah: That is a problem that would not be using my C.

John: And this was also the catalyst to including suggested solutions on our help, our subscription support form. So sometimes when you select something, you’ll see a suggested solution that bumps you to a help page where you can clear things faster, like we mentioned prior. But yeah, things like social media posts. If you’re on Reddit and you see a response from someone that’s like, hey, it’s John at DuckDuckGo. That’s me. And app store reviews, those get watched by our team and others. And then they’re routed to us when they’re support related. So it ensures that those feedback loops are all connecting together.

Beah: Nice. All right. So tell me a little bit, like, as our support function has grown and needed to scale as the subscription has grown, like what changes have you had to make?

John: Yeah, absolutely. So for a little bit of inside baseball, in late 2022, DuckDuckGo acquired the company that I co-founded, bringing my friend and I on board to help build out the personal information removal piece of the subscription. But one of the things that drew DuckDuckGo to our company of Removaly was our stellar customer support. And that was something I really wanted to ingrain in this subscription. So early on, I made sure to take a leadership role in building that out, which starting from zero is pretty... It was pretty fun to go from zero to where we are now. But at first it was really, really bare bones. We actually just ran it through an Outlook email. I personally replied to over 15,000 emails in the first year and a half that the subscription existed. Recently we switched over to Zendesk to help kind of manage a growing user base and get analytics and things that we could use to help improve. So between that switch and the addition of these other fantastic support team members, it’s really allowed me to step back and focus on the more operational aspects of how we support our users, but I still try to sometimes get into the weeds.

Beah: Yeah. That’s the basic problem to be for.

John: And we use those Zendesk analytics to inform continued growth for support and ensures that we scale effectively in parallel with the subscription and even.

Beah: Nice. Fifteen thousand. That’s pretty good, John.

John: Yeah.

Beah: All right, what are some of the main challenges of this system?

John: Privacy. Privacy is, while it’s fantastic here, it can be the bane of existence for support.

Beah: Yeah.

John: We don’t hold any information about our users by design, so we troubleshoot with a lot less context than most companies have. It helps make us better at asking questions and writing clear documentation, but it’s a real constraint, honestly. There’s also the consistent balancing act between speed, self-service, and satisfaction. And in many cases, we get to pick two of those, which negatively affect the third. So every improvement that helps people find answers faster, we have to weigh it against the experience of the person who still needs a human to respond to them. We watch that side really closely. So for the many, fast can never mean worse for the few right now. Of course, there’s also the typical ordinary difficult stuff. We don’t close down support over the weekends. So we cover seven days a week with a really small team. We have volume spikes frequently when we launch new areas, when we introduce new features, when we go viral on social media, when we start large-scale advertising initiatives like we did a couple weeks ago. We’ve got multiple billing platforms that have different financial processes and platforms we build on shipping really quickly that we have to absorb to make sure that we’re not giving incorrect or outdated information. But despite this, our really small but mighty team has managed to continue operating at a really high level of customer satisfaction. So that’s pretty sweet.

Beah: Nice. So you’ve touched on using AI once or twice. Tell me a little bit more about how you think about the role of AI in customer support. Yeah.

John: Yeah, so first I was really hesitant. I mean, I’ve seen and I’ve witnessed firsthand with other companies that I’ve interacted with that AI can be great and helps with scaling, but it could also like really frustrate people when they know that they’re chatting with someone that’s not going to be able to help them. It’s a bot. So we’ve done so.

Beah: Yeah. Yeah, I kind of forgot. I’d kinda forgotten about those early conversations, but you’re, you’re bringing it back. I think I remember being like, like I know what side John’s on.

John: Yeah, I fought it a lot. But we’re working on it. We’re trying things out. So we piloted kind of a privacy preserving AI agent through Zendesk on a couple of our support pages. And it was built to only answer from our own help page content. It handled some real volume. But we’ve had to kind of be honest with ourselves about the results. They’ve been OK. But there’s a lot of nuance involved in the questions that people are asking us. So we’re still ideating on it. We’re tweaking it. We’re refining it more. But that’s something where instead of shipping quickly, we want to make sure we move slowly and do things the right way. And then AI and the customer support space in general is really touchy, as I mentioned. It’s got to reduce the user effort without touching the quality of the answers or negatively affecting the privacy that our users expect from us.

Beah: Yeah.

John: So we tool that to configure conservatively for privacy. Help content is really kept as consistent as possible. And we’ve used AI a bit to kind of make sure that we are keeping our help pages the latest and greatest.

Beah: Okay.

John: Internally, we use our own AI tools where they really actually will help. But we still make sure that a human owns every reply that goes out that’s not specifically driven through that AI agent.

Beah: Yeah, that’s interesting. I was thinking about like, I mean, I feel like for us to give everybody the information they need quickly, it’s kind of a no-brainer that we should be experimenting with getting AI help on that. It feels like it, it could be quite, quite powerful, but I think right now I’m on a very frustrating customer support thread with, I’m guessing, mostly AI driven responses for like a piece of furniture I purchased.

John: Mm-hmm.

Beah: And it’s responsive, like it’s sending me a lot of emails, but they’re just like such fluff, you know? Like it doesn’t have any good answers. It’s not really taking my like email all that seriously. It’s just kind of holding me off, you know? And like every time I get an email, and it’s like, and it’s also like super apologetic. It’s like, like we’re so sorry. That must be so frustrating. And it like, by the like third message like that, I’m like, what? This is like so disingenuous.

John: Exactly. Yeah. Yeah. Yes.

Beah: Like, and I’m like, are you just AI? Like I think you’re just AI, right? And yeah, it’s, it’s tricky. But I do, but I mean like I, yeah, I want a response fast and I want it to be meaningful and in theory, like there should be a good application of AI to do that.

John: Agree. Yeah, it’s really critical to find a good balance of that. So you’re not providing a really fast service that’s not answering the question effectively or that’s providing that fluff that’s making people like, obvious that maybe this company doesn’t care about me as much as I thought if they’re not willing to have someone on the other end actually having this back and forth.

Beah: Yeah. There’s also like a little bit of a sense of, I guess like worst case scenario, like deception, you know? Like I’m like, are you, are you a robot pretending to be a human? Like that’s weird. I d, that’s just weird.

John: Exactly. Yep. Exactly. Yes. Yeah. So I’m slowly coming around, but I’m still pretty dubious about it in its, in its current state. So.

Beah: Yeah. Yeah, nice. All right. Well, to be continued, maybe, yeah, we can do a V2 version of this episode a year from now and see if, see what kind of progress you guys have made.

John: Yeah.

Beah: But let’s close on a more positive note. What aspect of our support, our customer support are you most proud of, John?

John: Man, honestly, I feel like we’ve been able to keep or improve our set-by-ourselves quality bar while we’ve continued to scale. So we’ve been able to survive that increase in volume without negatively affecting the quality of service that we offer. At Removaly, I mentioned my co-founder Kyle and I, we did the customer support. It was a two man crew and we did live chat and it was difficult. But that closeness to our users shaped how I wanted that quality bar to be reflected at DuckDuckGo, which, it’s not easy pivoting from a very small startup to one of the most visited sites on the internet. Three plus years and a lot of growth later, every ticket here still gets a specific human answer. Negative feedback gets read by people who can actually drive the change to the products that people are saying that they’re having issues with.

Beah: But I won’t make that. Yeah.

John: And our support is consistently having a seat at the table in product decisions. We’re actually being listened to when it comes to like, well, what do our users actually think about this? Are we getting feedback? What do you guys recommend? So right now our support team is four people. But we’re a small team whose effectiveness is so strong that honestly I’d really put it up against support organizations with entire support structures 10 times the size of ours. And I think we’d be able to beat them.

Beah: Yeah.

John: So being able to keep that quality bar despite scaling quickly and rapidly over the course of the past few years has been wonderful to see.

Beah: Nice. Well yeah, I mean, DuckDuckGo I think has like some DNA in a time when we weren’t one of the most visited sites on the internet and were like, Gabriel was, you know, personally super focused on, you know, talking directly to, to customers, to users, and I think we still have a lot of that DNA. I mean, we need to on the one hand find ways to make it scale, but I think sometimes like caring inordinately about an e, a single angry individual can be a really powerful tool for improvement and motivation. And yeah, so thank, thanks John for like all the work that you and the team do on that front and also to listeners who provide us with feedback. Thank you guys. I, we really try to make good use of your time.

John: Absolutely, yeah. Thanks for having me, Beah.

Beah: All right, later.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit insideduckduckgo.substack.com

In this episode, Marc (Engineering) and Zach (Data Science) discuss how we use AI internally: our most common uses, how we use it privately, and why our principles matter more when using AI, not less.

If you have feedback on Duck Tales, or episode ideas, email us at [email protected].

Disclaimers: (1) The audio, video (above), and transcript (below) are only lightly edited and may contain minor inaccuracies or transcription errors. (2) This website is operated by Substack. This is their privacy policy.

Marc: Hi, and welcome to Duck Tales, where we go behind the scenes at DuckDuckGo and discuss the stories, technology, and people that help build privacy tools for everyone. In each episode, you’ll hear from employees about our vision, product updates, engineering, or approach to AI. My name is Marc and I’m on the engineering team. I have with me Zach. Zach, you want to introduce yourself?

Zach: hi, yeah. I’m Zach Deane-Mayer on our data science team here at DuckDuckGo. I am still a Duckling, so I’ve been here for about ten months. and I work on building our internal AI systems.

Marc: Great. yeah, so today we’re gonna talk even more about AI, which is Zach’s specialty. I think today we’re gonna talk specifically about how we use Duck or sorry, how we use AI internally at DuckDuckGo. so yeah, I guess to start out, as a company, you know, we’ve adopted AI pretty pretty heavily internally as a tool to help us move faster. can you talk a little bit, Zach, about the the approach that we’ve taken here? Given that I think this you were in on the ground floor with this, right?

Zach: Yeah, yeah. So DuckDuckGo as a company takes I I think a somewhat unique approach to AI as we take a unique approach to everything we do. We very much have a culture of of things like first principles thinking and and you know our own spin on things.

Marc: Right.

Zach: And a very important part of what we do are our privacy and security. so before we rolled out many of these tools, we spent quite a bit of time internally getting comfortable with their failure modes and the blast radius potential for for those failures. One of the first places we started using AI a lot was with coding tools. So right off the bat with Cursor and then Claude Code. And you know, one, the models are very, very good at writing code. But then two, if you are effectively using tools like git and GitHub, an agent that’s limited to writing code can only do so much damage if your code base is backed up in GitHub and you don’t let, for example, people force push to main or whatever like that. Like if your GitHub is properly configured. And so we had a lot of questions like that early on. And then, you know, once once we were confident, we really were able to kind of let open the floodgates and let people really start start using both Cursor, Claude Code and now to a lesser degree Codex to to write code and automate as much of engineering as they can.

Marc: Yeah, yeah, that’s great. you mentioned privacy, right? Like that is obviously it’s a core tenet of our products. It’s also a core tenet of how we operate here. and yeah, to your point, we do tend to do things a little bit differently. so I’m curious how the privacy kind of shaped our internal workflows with respect to AI, given that, you know, w we even though we don’t collect user data, we do still have things that we obviously don’t want to to share more broadly.

Zach: Yeah. Yeah, absolutely. so our privacy engineers very much think of like a an information hierarchy of stuff that is you know very I don’t know what the word is I want to use for the scale, but there’s a there’s a high to low scale. and so there’s certain information that we don’t even save to disk in our production servers. There’s certain information that is stored in memory and is never retained anywhere in our systems.

Marc: Right.

Zach: and that is fundamentally the safest form of data. If you don’t if you don’t store a piece of information, it’s it’s very hard to accidentally store it in the wrong place and in a way that you don’t you don’t want to do. and and that kind of principle thinking applies to AI too. Like if if an AI system doesn’t have access to certain data, it It can’t do the wrong thing with that data. and one of those ways to prevent access is just by by fundamentally not not storing the data in the first place. As you go through that hierarchy, you know, there there’s information that we’re more and more comfortable with working on internally, like for example, you know, time series traffic of you know how many users are on our various websites, like that sort of aggregated data that can’t be tied to an individual person. That’s a lot safer to start saying, hey, I’m gonna, you know, I’m on the data science team, I’m gonna do some data science work and look at, you know, whatever, whatever it is that the set of questions I have. and then I think the kind of the bottom piece that is incredibly important is we pay a lot of attention to our data processing agreements that we have with all of our vendors, but especially our AI vendors. and the contractual agreements you have with people about what they are and aren’t allowed to do with your data are are also very important. so so yeah, there’s there’s a spectrum of things and you go talk to any privacy engineer, I mean you can very quickly learn how just how deep the problem goes.

Marc: Yeah, yeah, it’s interesting you mentioned principled thinking, right? Like I I think that’s something that is a core part of the culture here. And I I think it’s fair to say that we have, you know, spent a lot of time talking about how to adapt AI to our principles as opposed to adapting our principles to AI, right? which I feel like you see a lot in the industry right now.

Zach: Yeah, yeah. Yeah. Yeah. And and and and that’s I mean that’s a that’s a great point. Like two of our principles that matter a lot to me personally are questioning assumptions and validating direction.

Marc: Absolutely, yeah.

Zach: and you know, I’m I’m like the the AI, you know, waving, you know, the flag and the the pom-poms and saying, let’s go AI internally.

Marc: Right.

Zach: but like I I find I have to like, you know, remind people, just because we’re using AI doesn’t mean we throw our principles out the window. Like,

Marc: Absolutely.

Zach: I am you know, again again as a leader here, I am more interested in like like I want to validate direction too. And so I am very interested to hear from people places where AI isn’t working. Like, like I want to see, you know, the PR that’s a 7,000 line of code wall of text where a team’s going, whoa, like we can’t we can’t review this, let alone ship it.

Marc: Right.

Zach: And those those kinds of failure modes, like AI is rife with them. And we don’t, you know, we don’t just look at an AI’s output and say, well, an AI made it, so let’s ship it. Like you you should use AI as as you know, a lever against, you know, with your principles, not not as a as a reason to throw your your principles out the window. But like again, like just questioning assumptions and validating direction, like you know, AIs are notoriously bad at questioning assumptions.

Marc: Right.

Zach: Like this will happen all the time, even with the latest models, where you’ll be like, hang on a sec. Like you said X and then you said Y, which implies not X, like, but you never connected the pieces together in in in in the chain of logic.

Marc: Sure, sure.

Zach: and so I I think I think your principles matter more when you’re using AI.

Marc: that’s interesting. Yeah. Yeah, that I think that’s really well put. actually leads us into our next question to a degree. there’s yeah there’s a lot of hype right now, obviously with AI. I mean to put it mildly, right? the hype cycle is has been super interesting. there’s a lot of concern about AI in the workforce today, right? And we’re certainly not immune to that, although we don’t really see it that way. as a company. So maybe can you talk a little bit about how we’re how we’re getting so many people to use it effectively and and you know where we’re trying to get to.

Zach: Yeah. Yeah, so AI is a very powerful tool. and it’s also in some ways a a slippery tool. Like it’s it’s changing so fast. It there’s not like a long history of like I don’t know, I I can’t I can’t think of an equivalent technology, but there’s there’s not like The field has changed so fast, everybody’s brand new to it.

Marc: Yeah.

Zach: And there’s a lot of surface area of of unknown unknowns, of things you don’t know that that can go go wrong.

Marc: Well.

Zach: I I heard a joke I really like that like working with AI, it’s like a cross between Amelia Bedelia and the guy from Memento.

Marc: That’s good.

Zach: and and it’s a great joke, but it’s like, you know, one, interprets everything you said too literally. and then two, it forgets tomorrow everything that you said to it today. And like, like those two problems, like, you know. It’s you know, I I’ve been playing around with Fable a lot recently. It’s an amazing model for one-shotting really difficult programming problems. But you still get that sometimes over literality, like like what you say wasn’t quite interpreted the way you meant it, and you have to be careful of that. And then the other side of like, you know, you come in tomorrow and you have to curate context that you take over between sessions. And I I think honestly that that latter part has been the part for me that has been the most difficult to adapt to is just what do you remember and how do you remember and how do you persist that memory?

Marc: Sure, sure.

Zach: And that’s very much something you have you have to build for yourself. and and you know from the individual level to the team level, like how are we storing information in, you know, when an AI goes wrong Where do we put that learning so that we don’t, you know, forget about it again in the future?

Marc: Right. Right, right. Yeah, I think from an engineering perspective, I mean, you know, we’re both engineers, so I I I think it’s safe to speak about first hand experience there. it’s definitely changing the field, right? Like everybody acknowledges that. I don’t think it’s obviating the field. I’ve heard so many analogies about how, you know, AI is like this or like, you know, whatever.

Zach: Yeah.

Marc: One I heard that resonates a little bit again from a software engineering perspective is like nobody hand compiles code anymore, right? Generally.

Zach: Right.

Marc: And I feel like, yeah, it maybe not a perfect analogy, but it didn’t make like that didn’t obviate the need for strong engineers, right? It just changed the nature of what they’re doing on a daily basis. And I think at at most that’s what we’re looking at here potentially, but obviously it’s a hot topic.

Zach: Yeah. Yeah. Yeah. No, so I I was literally just listening in a podcast, I’m on a podcast, I was listening to podcast, I love it, this morning with Fiona Fung, who’s the manager of the Claude Code team.

Marc: Mm-hmm. Okay.

Zach: And she had just like this wild and one-off anecdote. She’s like, my old boss, like my first engineering manager I ever had, you know, called me up the other day and and she said, This guy started his career programming in punch cards. Like

Marc: Yeah, right.

Zach: Programming in punch cards.

Marc: Right.

Zach: And the guy called her up, was hey, I’m using Claude Code and you run the Claude Code product team. I just want call you tell you how cool Claude Code is.

Marc: Ha ha ha. Yeah.

Zach: and I thought that was like a really great piece of perspective. Like just like imagining the transition between programming in punch cards to programming in Python. And like I’ve never, you know, I’ve never, you know, I didn’t experience that because I wasn’t I wasn’t around. but you you forget how long the history is of of computing and how much it already has changed.

Marc: Absolutely. Absolutely. To think about people. I mean, I think most of us have at least tried some assembly, right? Like it I don’t know if they still teach that in university today, but they certainly did. even if it was just to expose you to it. But yeah, that’s not something that, you know, the average person or the average engineer is gonna be doing. But anyway.

Zach: Mm. Yeah.

Marc: So yeah, I guess we could start wrapping up. I think what we’ve talked a lot about engineering examples. I’m I’m wondering if you know, we have some non engineering examples of how we’re using this at the company.

Zach: Yeah. one of my favorites is is spreadsheets. I spent a lot of time early in my career building models in Excel. and I definitely developed like an aesthetic for like what I want an Excel model to be like. and then I stopped using Excel and both the interface changed, but like my muscle memory, you know, I never got to like the keyboard shortcut level is always point and click. But like I I’ve been frustrated a lot recently with trying to build a model in Excel and I can’t find the commands I want or necessarily even remember them. and what’s really interesting to me is is a lot of Excel models, you can you can give a data set to Claude, you can describe the problem you want to solve, you can describe the shape of the model that you want. You know, I want one tab with my input parameters, one tab with my data, and one tab with the calculations and output or something like that. And it’ll do a r a remarkably good job of building things like that. and there’s you know, there’s a lot of work that still happens in spreadsheets. and it’s really nice to be able to go to Claude Code and give it a spreadsheet and say, Hey, can you change the formatting? I’ve got all these cells where the important information is bolded and I want to remove the bolding and replace the bolding with like a red background. And that’s really tedious to do with pointing and clicking. and Claude Code, like it’s just reading the XML. And so it’s just like it’s very good at, you know. A a spreadsheet is an XML file and it can just say, Cool, yeah, I’m just gonna change around the XML structure and now instead of bolding, you’ve got red highlight.

Marc: Yeah, yeah. Got it. Yeah. Yeah. No, that’s interesting. I th I feel like yeah, as much as I’ve embraced it in, you know, my daily workflow, I still have not yet gotten the nerve up to trust it with something like a spreadsheet. But I know that a lot of people do, so I probably need to get over that.

Zach: Yeah. I mean, if your spreadsheets are in version control, as mine all are, because I’m a huge nerd, like there’s there’s limited room for it to really, really screw screw something up. Yeah. The other one so I just

Marc: Yeah. Sure. Sure. All right, great. Well, you know, please go ahead.

Zach: w one more is just like summer camp forms. Like I’ve got these forms, like I look actually no, it was a school form. I got a form from my kids’ school and it was like it was like a scan. But it like a little bit sideways, so all the fields are a little bit off. And so, you know, it like filling out a PDF form is annoying enough when there’s like the little boxes that you can type in. But it’s like,

Marc: Right. Yeah.

Zach: okay, I don’t want to print this thing out, handwrite everything. My handwriting’s terrible. Scan it, send it back. And I was like, I actually did, I said, you know, Claude, I said to Claude Cowork, here’s the form. I dictated to it what I wanted in each field. And I said, can you just iterate? Actually, it was Claude Code, not Cowork. Can you iterate on you know, take my text, turn it to text box, put it in the PDF, and then, you know, line them all up correctly with the fields, which was not easy. And it built itself a little loop where it was like, you know, it got like my information to bullet points is pretty easy. Map them to the fields is pretty easy.

Marc: Yeah.

Zach: And then it just did an iteration of okay, write them out to the spreadsheet and then just like adjust the X and Y coordinates of the boxes until it lined up well enough that I could email it back to the school. Yeah. Yeah.

Marc: Wow, that’s awesome. Yeah, I have not I wouldn’t even have thought to do that with AI. So I feel like I need to be a little more ambitious.

Zach: Yeah. Yeah. It was the same where like I I spent five minutes talking to Claude, gave it the file, sent it off in the background and it came back and it was done and I sent it. I just, I checked the data, I was like, Yep, that is correct and you know, emailed it to the school and instead of, you know, I mean, who knows? It would have been probably twenty minutes of printing it out, filling it out, scanning it, sending it back, plus a bunch of steps and I didn’t want to do

Marc: Yeah, sure, sure. All right, great. I yeah, I feel like we could talk about this all day.

Zach: me too.

Marc: but we should probably wrap up. So yeah, thanks a lot for your time and thanks for everyone else for for listening in.

Zach: Yeah. Great, great talking with you as always, Marc.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit insideduckduckgo.substack.com

In this episode, Konrad (Engineering) and Shilpa (Engineering) discuss Personal Information Removal: what is a data broker, how we remove your records without ever sending your data to our servers, and the ‘whack-a-mole’ of constantly changing data broker sites.

If you have feedback on Duck Tales, or episode ideas, email us at [email protected].

Learn more about the DuckDuckGo Subscription here.

Disclaimers: (1) The audio, video (above), and transcript (below) are only lightly edited and may contain minor inaccuracies or transcription errors. (2) This website is operated by Substack. This is their privacy policy.

Konrad: Hello everyone and welcome to Duck Tales. We will look under the hood of DuckDuckGo, talk about stories, technology, and people who are building the privacy tools that you all use. And in each of the episodes, you’ll hear from different employees about our vision, updates to our products, how we do engineering, or how we approach AI. So today I’m your host. My name is Konrad Zwinell. And I’m working at DuckDuckGo for seven years now. I work on the front end team at the very beginning. Then I run privacy engineering team. And now I’m back on the front end team working with Shilpa, who is my guest today. And Shilpa is running a personal information removal tool team. And she will tell us more about it today. Shilpa, why don’t you introduce yourself?

Shilpa: Yeah, hi. I’m Shilpa. I have been in DuckDuckGo for, I think, a year and a half now. I joined in on the Apple team working on iOS and macOS. And for the last few months, I’ve been driving the team that does works on the personal information and mobile product.

Konrad: Nice. OK, so first question that people may have after hearing this intro is, what exactly is personal information removal? Could you tell us, high level Shilpa, what’s the product and what it does? What’s the idea behind it?

Shilpa: Yeah, so in the US, this is predominantly a US-based product. In the US, there’s entities called data brokers that basically harvest your personal information from various different sources, your online activity, social media, what have you. And they collect that information, and they basically buy and sell it as a commodity for advertisers and different entities. And our product basically goes and looks at all of these data brokers, looks for your personal information. If it’s there, then it submits a request to remove it. And if the data broker is complying, they will go ahead and remove it and we will be able to verify the removals for you as well.

Konrad: Nice. So basically reducing your footprint on the web, reducing availability to your personal information, such as address, phone number in some cases, relatives, if I remember correctly.

Shilpa: Yeah.

Konrad: So they collect quite a bit of information about you and make it really easy for advertisers, but also other people to look it up. For FV,

Shilpa: Yeah.

Konrad: yeah, so that’s a bit scary. How does it work from the user perspective? Let’s say you want to use PIR. You want to try it out. How is the experience looking? Could you share with folks how it looks UI-wise and what you have to do to try it out?

Shilpa: Yeah, so today, actually in some platforms, you can even go try it out for free, part of the experience on macOS or Windows, where you can go ahead and put in your personal information, some basic stuff, like your name and date of birth and city and you will be able to go see, we will go ahead and do a search on the various different data brokers that we support. And it should be able to show you all of the different brokers that have your information, the information that they do have. And the first time, if you see this actually, it’s a little bit, it’s actually quite a bit scary to see how much of your information is out there. And literally anyone can do a search and pull up a lot of your information about you, your family and stuff. So yeah, once we do the search and then we’ll go ahead and go to those, once we know that your records are there on those specific brokers, we go ahead and submit requests to opt you out.

Konrad: Thank

Shilpa: And then there may or may not be some steps in between all of which we will take care of for you. And then usually there’s a delay from a couple of days to a few weeks maybe. Then if the record is removed, we will go ahead and do a scan again and make sure that your record is removed. And then we’ll report to you that that record’s been flagged as removed. And not just that, we also do periodic scans to make sure we do like maintenance scans to make sure that your record is removed and it continues to be removed. Because the way that data brokers work is once they remove it, they’re not very intentional or deliberate about these records. They’re just collecting and gathering from whatever sources they’re getting. So if it’s removed, doesn’t mean it stays removed. Sometimes it goes, sometimes it comes, sometimes it’s merged from other sources. So it’s important to continue to keep, keep removing it.

Konrad: Yeah. And we keep on adding new brokers, right? Meaning, you keep the PIR running, we will add new brokers and we will opt you out out of those brokers. So it’s not only like insurance that your records don’t pop up on the same brokers, but as we find new brokers starting business and appearing on the web, we will try to opt you out from those too, right?

Shilpa: Yeah, yeah. It’s the whole world of data brokers is very fluid. There’s new ones coming up. There’s mergers happening. There’s some of them are going defunct. Some of them are temporarily defunct. They come back. So we’re all constantly keeping an eye out for how things are changing, fixing it and adjusting for the changes they’re making, as well as constantly looking out for new ones that are coming up, seeing how we can support removals on those sites.

Konrad: That’s right. And I guess that’s our biggest challenge, right? Like fighting with this ever-changing websites and brokers kind of even fighting back on purpose, trying to kind of avoid our product, like make it not work. Would you say it’s the main challenge? Do we have like some other challenges?

Shilpa: I would say that is pretty much the main challenge. They are deliberately or not constantly changing things on their side. And since they don’t have a very clean, clear request that we can make to them, just to say, here’s my record, go remove it. But it involves usually going through and scraping the website. It’s not a very clean process, which means that we have any minor changes on their end leads to us breaking things on our side. So we have to constantly keep an eye out and monitor, make changes, fix. So this constant whack-a-mole, I think, is one of the biggest challenges. It’s also a good engineering challenge. So we are always thinking of various different ways to innovate and try to be as flexible as possible and see how we can actually get ahead of them. That actually is like the main engineering challenge related to this product for us.

Konrad: Yeah, we both like a challenge. That’s why we are on this team, right? OK, let’s see. So we are both engineers.

Shilpa: Absolutely.

Konrad: So maybe let’s do some shop talk and talk how this work under the hood. So yeah, can you tell us a bit about that?

Shilpa: Yeah, so doing a little bit of engineering talk, normally you would imagine that, you know, like we take the personal information from the users and then we go and make a nice clean API call directly to data brokers and they give us a response and we can say either success or failure. But that is actually not the case. Data brokers do not make our lives easy. What we actually have to do is basically what you would do if you were to manually go ahead and remove it. You would go to their website, type in your information, do a search, see if the record is found, and then go to another form. Again, type in an opt out request, and in between there may be various different stages. Depending on the data brokers, you may have to solve some captchas. You may have to get a confirmation code from the email or click on a link in the email. So we do all of that in the entire lifecycle of search and remove a record. So yeah, and I think one unique thing to the way we do versus our competitors is that when we take your personal information, it stays on your device. We do not send it off to our servers where it stays forever and potentially vulnerable to various other attacks. So we’re very particular about keeping all of your personal information within your device. So we just make the request from your browser within your device and keep it in your device so that it’s a lot less liable to be breached in any other way from server side.

Konrad: Right, because that’s our approach. Because the competition existed when we first started building this product, but we didn’t just copy the existing model. We thought it through and decided that what they are doing, it’s not going to work for us because we don’t want to receive personal information from our users. We don’t want to process that. We don’t want to deal with it. And so we came up with this model where everything happens on device, which has some pros. It has some cons. But yeah, we are making it work. And yeah, as you said, it’s challenging. But it’s the right thing to do to keep your data private. OK, so we talked a bit about what’s under the hood. We talked about the challenges, how those brokers are fighting back and by also going offline and changing their websites all the time and changing those forms. And yeah, so that has been our focus for a bit. You started talking, you mentioned like a bit about like flexibility, but we also do monitoring. We do other things like how we are adapting to this like well, complex environment.

Shilpa: Yeah, so constantly we have various different kinds of monitoring set up. So we’re always keeping track of how many of the brokers are responding correctly to the scans. We’re seeing what percentage of the scan attempts are going through, what percentage of the opt out request submissions are going through. Anytime we see more than a reasonable amount of variation in these, we always have one person that is on call watching out for any of these anomalies, any anomalies come up, we go do investigation to see what fixes need to be done. And also, we’re also trying to see how to reduce the amount of reactivity in terms of fixing this. We are trying to see how can we incorporate as much flexibility into our products, so there’s less reacting and more, we’d be staying on top of things when in case things do break. Currently, major part of our roadmap is actually focused on how to be more flexible as a product. And we’re also exploring to see how we can like incorporate AI, it’s fairly well suited to monitor things and maybe even try to attempt to fix it even before a human takes a look at it. So we’re pretty excited to kind of start working on all of that in the near future.

Konrad: Cool. Yeah, what’s next for PIR? What’s on our road?

Shilpa: I think one of the biggest requests from our customers has been to add the feature, make it available on mobile. We’ve had it on desktop for a while now on macOS and Windows. And there’s been a lot of requests to add it on iOS and Android because as expected, not everyone wants to run this on their desktop or has access to one all the time. So yeah, we’re currently working on that. We are excited to have it released on mobile, which should be coming up in the near future. There are challenges associated with making this work on mobile, especially since we don’t just send everything off to a server that will take care of things for us. On mobile, we need to make sure we have the connectivity. We don’t drain the battery. We are using the resources reasonably and efficiently. And we may not get a whole bunch of time from either the iOS or the Android operating system to run things as much as often as we want. On a desktop, we don’t have to worry about Wi-Fi connectivity or battery as much, but on mobile, there’s all of these additional challenges that we have to take care of. The team has been working for the last year almost to figure out ways around this and it’s a very good challenge to be working on.

Konrad: Yeah, the word ‘challenge’ comes up a lot when we discuss PIR. But thankfully, we have some great people on board and the work is moving smoothly forward. So yeah, look out for PIR on mobile. OK, and if anyone after hearing this discussion wants to try it out, even without becoming a subscriber, because this is like a subscriber feature. You mentioned they could do freemium, right? So how one does that? Where do you find it in our?

Shilpa: Yeah, so in the browser, if you’re not a subscriber, you should be able to find it in Settings, or sometimes you might just get a pop-up in a new tab when you open asking you to go try a free scan. So do try that. And if you live in the US, it’s going to be quite shocking. I will warn you of that to see how much of your personal information is out there. So it’ll ask you for a few things about your personal information, three to four fields. If you fill that in, it’ll do a quick search across the number of data brokers we support vary time to time because of the various reasons I said how they’re coming up and down all the time. But we support a maximum of about 60 currently. So we go do a search across all of these brokers. And you’ll be able to see how much of your information and even your family’s information is associated with you. And if that seems like a reasonable thing for you to invest in, then you can go ahead and subscribe in order to go for, allow us to go make the request to submit a request to remove your records from those sites.

Konrad: Perfect. I think we covered most of this stuff. Any kind of a closing thoughts, anything that we should have mentioned?

Shilpa: I think the only thing I would say is, yeah, it is a little bit scary to go see your personal information everywhere, but rest assured that we are constantly scanning and constantly monitoring. And a lot of the users also often of these products often mention how the number of scam calls that they get on their phones also goes down. So it’s really a very worthwhile product to be investing in. Check it out, check out the free scan version and we’d love to hear from you as to how things are working. And we’re also making a lot of improvements to our UX to make it more user-friendly and to make it more clear as to what’s going on behind the scenes and what stage of the various processes that we mentioned. End to end, each of your records is in with each of the brokers. So we’re excited to make all of those user facing improvements in the future.

Konrad: Perfect. Thanks so much, Shilpa. That was great. I really appreciate this discussion. And yeah, that’s all folks. See you in the next episode, I guess.

Shilpa: Thank you.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit insideduckduckgo.substack.com

In this episode, Gabriel (Founder) and Beah (Chief Product Officer) discuss our focus on delight, discovery, and dependability (the 3Ds), what they are, and how they shape our day-to-day work.

If you have feedback on Duck Tales, or episode ideas, email us at [email protected].

Show notes: Listen to our episode on Hack Days here.

Disclaimers: (1) The audio, video (above), and transcript (below) are unedited and may contain minor inaccuracies or transcription errors. (2) This website is operated by Substack. This is their privacy policy.

Beah: Hello and welcome to Duck Tales, where we go behind the scenes at DuckDuckGo and discuss the stories, technology, and people that help build privacy tools for everyone. In this episode, we are going to be talking about what we internally call the 3Ds: discoverability, dependability, and delight. And if we If you haven’t met me, I’m Beah. I’m on the product team and I’m here today with Gabriel, founder and CEO of DuckDuckGo. Anything anything you want to add to your introduction, Gabriel?

Gabriel: No, I do not. If you’ve watched this at all, you’ve already seen me a bunch. So and they’ve seen you as well, so

Beah: Yes. Cool. Okay. So yeah, let’s just jump in. 3Ds. What what’s the deal? What I mean, I said what each D stood for, but like contextualize this whole thing.

Gabriel: Yeah, so part of the way we run DuckDuckGo as a company, just like internally, like how we operate, is we have a what we call our top priority, which is like the main thing that we think we need to focus on as a company. And that’s changed every few years. But like right now we’re in the middle of and we’ve been doing it for I think around 18 months, something like that, this 3Ds focus on discovery, dependability, and delight as you pointed out. And those are product things. So it’s really a focus on making our product better in those kind of three areas. And just to walk through them briefly, discovery is we actually have a lot of stuff in our product that people do not know about. So there’s the basic thing that a lot of people still don’t even know DuckDuckGo exists and we have a product and we’re a privacy company. But when you get a little beyond that, you know, we’re a search engine, we’re a browser, we have a subscription now. In our browser, you can sync your passwords and bookmarks. There’s all sorts of features, and like the holistic product is a lot bigger than even most of our users realize. And so it’s kind of discovery of those features. Dependability is a little more straightforward. That’s like, okay, you’re a user, the product should be very dependable. But I mean that means like reliable. You can switch to it easily. All the workflows work well. There’s no rough edges in it. Obviously, that’s a tall order when you run a search engine and a browser and a VPN and all these other things we do. But we’ve been nail we’ve been working down the list of all the little issues that people you know complain about, rightly so, that might have been issues with their bugs in our product and we’re trying to fix them. And then the third one is delight, which is, you know. I think most intuitive, I guess, but even harder to actually articulate how we get there.

Beah: Yeah.

Gabriel: But it’s like you have a delightful product experience. It’s those products and experiences that you are unexpected, really bring you joy, like you really want to share with your friends and family. Like we want to bring that delight into using DuckDuckGo.

Beah: Yeah.

Gabriel: So the focus has been on all three of those, because we think we need to do all three to make a great product. And so we have projects and people working on, you know, basically all three at all at all times at the moment under our top priority.

Beah: Yeah, just to add to the discovery point, I mean we get feature requests and feedback. Feature requests for features we have. Like people just straight up asking for features that exist. Yeah. Private browsing. Yeah. Yeah.

Gabriel: Yes. I would argue that might be our top request. Things like, yeah. Yeah. This is a core discovery issue. Yeah, and even more fundamental than that is like a lot of a lot of search engine users, because that was our first product, and we have the most search engine users, don’t even know we have a browser. And our most browser users on mobile don’t realize we have a desktop browser.

Beah: Yeah.

Gabriel: So regardless of features within all those things, like just at the very basic level, people don’t even know we offer a browser. And they should. Yeah.

Beah: Yeah. And like sometimes it’s as simple as like changing like making something more visible and then other times I think it’s about the language we use or like the way we characterize something so that people know that it’s what they’re looking for. So yeah, there’s like that one’s yeah.

Gabriel: Yeah, one more point on that too. Like that one’s subtly hard as well, because like you don’t discovery means that you have to get people to know about it to your point. And you maybe think of something, which is that you have to get in front of people then with a message, but that can be annoying. Like it look can look like an ad, right?

Beah: Mm.

Gabriel: And so that’s a delicate balance. It’s like, yeah, you’re a cer you’re a private search user, you probably want our private browser because it’ll actually make the benefit of privacy much more you know, better in your life if you have them paired together. However, you don’t want a thousand messages of, hey, we have a browser downloaded now. So there’s there’s like this delicate balance of like trying to get awareness to get the people who will who want it who want it to get it without annoying people who don’t want it.

Beah: Mm. Totally. Yeah. And there’s just like it’s the competition of many things. Like you probably want our browser. When in our browser, you probably want your passwords and your bookmarks. And then you probably want those synced across devices. And you probably want this thing that’s opt in turned on. And like, yeah, you can just we have to I think we spend a lot of time trying to make sure that we’re not like bombarding people with too much messaging or promos and it’s but then but then again we get feedback requests for things that are sitting there somewhere behind the scenes. So yeah.

Gabriel: So just before we move on to that one practically, then what that means is we’re constantly running experiments basically of like how best to do these in product kind of promos. And and and I think to your point, like it’s better to do them. The naive approach is to shove them all in onboarding, which people then whiz by. But the better approach is like very contextual, like maybe like when you do a bookmark for the first time, ask people if they want to sync. And then the other part of discovery, just to end that part, is is just our external marketing. So like not in talking a lot of in-product stuff, but we also spend a lot of time just doing general marketing, especially in the US, about educating people about our different experiences.

Beah: Yeah. Yep. So we ended up talking a lot about discovery, but just another like zooming out question. How how did we land on these besides the fact that they all share a D, why did we pick why did we pick these three things?

Gabriel: That’s that’s clearly part of it. Yeah. Yeah. Yeah, I mean if you think about the whole product kind of marketing funnel, which is you need awareness in a product, then you need to know a little bit more about its benefits and value proposition to consider using it. They call that consideration. Then if you kind of are interested in that, like in our case, like privacy benefits, you may trial and then that initial trial point on like is kind of an activation in the funnel. And then you want to retain users and then ultimately a referral. You want users to be happy enough that they’re referring you. So that’s the traditional like marketing funnel. And different companies have issues you know, at different parts of the marketing funnel where they would want to focus. I think it was pretty clear that we had some gaps in in in kind of all parts of the funnel almost equally. And the 3Ds from a product perspective map onto that pretty well. So, like the discovery part is just we keep measuring these surveys, but most people still don’t know about us, and most people don’t know about our different products. So there’s that’s an obvious gap. The dependability one, we’re constantly running user surveys.

Beah: Mm.

Gabriel: And so once people are activated, you know, our retention has gotten a lot better in the last three years because we’ve really improved the dependability of all our products. But we could tell it was a little bit of a leaky bucket, like people had come in and really wanting to love us and try us, but having an issue switching, or you know, we didn’t have, you know, sync or, you know, with their bookmarks. And so we felt that that was also deserved a lot of a focus because we could see that people weren’t always staying when they were activating. And then the third part is word of mouth. So we had grown mainly through word of mouth. The most most of our users have come through word of mouth. But a lot of that was on the back of privacy being one of the biggest stories in tech. And I I hope that’s coming back, and it should, because AI has a lot of privacy issues. But you know, since the pandemic that has been less so. And so to get word of mouth without a big news cycle all the time, you need delight.

Beah: Mm-hmm.

Gabriel: And so it’s really mapping these to a funnel and making it a product focus of that funnel.

Beah: Yeah.

Gabriel: And then something that has either if you’re gonna pick names, you either have to rhyme or they have to have some alliteration. So this is alliteration.

Beah: Yeah. On the delight thing, I also think like for a I think can because we are not the default in many senses. I I think delight pre plays a pretty important role in retention too, because like there’s sort of this FOMO factor to retention where if you run into a little bit of friction on DuckDuckGo, I think you’re more likely to churn than if you run into a little bit of friction on like what people what is actually literally the default or people perceive as the default, you know, and I mean literally the default. Like if you have an Android phone and go, you know, Google provides your default browser, your default search engine, like you people run into friction or, you know, bad dependability like in any product. But I think like their tolerance for it can be lower initially in DuckDuckGo than in other than in Google, basically. So I think delight is an important antidote to that as well. Like you have to have a reason to be willing to endure occasional friction. I mean, we’re gonna keep trying to reduce that friction and make the product better and more dependable, but like it’s just never gonna be perfect. Like we could be, you know, yeah, no no product is ever perfect. There’s always rough edges. So I think delight also just helps on the retention front too.

Gabriel: Yeah, I mean I I think you’re making a good point. I presented it as mapping on different parts of the funnel, but I think you’re absolutely right. Delight really can map onto all parts of the funnel, even the top where you can have delightful marketing and people, you know, we hope that might be wanna be associated with our brand and our product just because not just values driven on privacy, but also we’re presenting that to the world in a way that resonates with them, you know, like they want to be associated with us because they like the way that we make our product and present our marketing and present our, you know, our app store page and stuff like that. I know you thought more about delight from the product side as probably anyone else in the company. Do you have like any deeper thoughts on like from the product side, like how to even think about going to going to incorporate, you know, that kind of experience?

Beah: Yeah. Yeah. I mean, the way I think about delight usually is is basically addressing the emotional needs of people, not just the functional needs. So you can have a feature or an interface that just that like has that works in some basic sense, but it’s not actually pleasurable to use. And like, you know, we’re humans, and emotional beings and pleasure is worth something. So yeah, I mean, I think that can take so many forms, and that’s part of what’s difficult about it. And sometimes the most like sort of the lowest hanging fruit stabs at delight are not in fact delightful at all. Like you can bombard people with confetti, you know. And I don’t think that’s usually delightful. I mean there might be moments where some kind of celebration in the product is is good and appropriate, but like there’s a lot of ways to be annoying too that that if you just try to inject some personality into the product or some delight like you that can take you down some some bad paths. So I think one of the tricky things about delight is that it like requires maybe several ingredients. One is a point of view. Like I think there has to be some consistency. Like it I do think personality matters and brand matters. And if like the product’s just like throwing all these different sort of personalities at you, that like that’s actually like disconcerting, not delightful. To another ingredient that I think is really important, maybe most important, is just like the amount of like rigor and finesse and discretion that has to go into delight features. Like sometimes it means obsessing over a single word, like way more than is obviously worthwhile. Or I don’t know, just like fine tuning the details until it feels just just right. The good news is I think you can usually test for it qualitatively pretty well. And we have a lot of good qualitative testing tools, and we can kind of like you know, put it and and the other thing is once you’ve spent so much rigor internally on something, it it is like awfully hard to look at it the way a new user would. And so I think qualitative testing is is important. But the bottom line of all of that is I think it is actually like often pretty expensive to do well in terms of people’s time and effort. And so it it requires like a real commitment, which is I think, you know, speaks to why it is in this company top priority is it is a statement of commitment to putting in the time to do this, do this well. The other thing I’ll say is that I also just think it’s like more fun to build products that are delightful. Like personally, even though it is so hard and time consuming, it is like one of my favorite ways to spend my time at DuckDuckGo. And I think people who are good at it and like it generally agree. So yeah, maybe that’s kind of a good segue actually to another question, which is how we actually operationalize the 3Ds. Like we say this is the company top priority. Like beyond that, how does it work its way into our day to day or week to week or month to month operating rhythms?

Gabriel: Yeah, no, that that’s a good segue. I mean, I think to some extent every product would like these three things, right? They’re like mapped onto the funnel and they need to be thought of to some extent pretty much by everybody making any product. But to your point about delight, despite wanting it in our product and knowing how important it was, I don’t think we were getting it until we set the top priority. And so one answer to that is and when they get to how to operationalize it, but like just saying we wanted it was not working, you know, like and even with dependability, it’s like, yeah, we have lots of bugs and we would want to fix them all, but we weren’t systematically, I think, addressing dependability in some areas as well. As we needed to, and we were seeing that in different metrics, say like crash reports and stuff like that. So I think operationalizing it could look different in different companies. For us, you know, we often operate as objectives, so we have like maybe 25 different objectives, and so we’re thinking about what are the top discovery dependability delight issues that we have, and then making an objective around that. Which each objective will have its own then set of projects to go solve that objective. And so we’ve been doing that for about eighteen months. And then within each pro

Beah: And you and like we’ve started in that time we’ve seen objectives that have started stating their problem to solve in terms of the the 3Ds, which is cool.

Gabriel: Exactly. And that and and and I think you’re right that it takes that amount of time to like get it in people’s heads to start thinking that way. And then even one below on the project level, you know, we’re tagging projects with the 3Ds. So is this a project about one of them or multiple of them in some cases? And even the criteria, I think to your last point, of individual projects are often related to one of these. So like dependability has KPIs depending on different contexts, but like crash reports is a good example. Like let’s get this crash number down in this way. Delight we’ve to your measurement point earlier, we’ve developed our own internal like KPI survey methodologies to to like actual test delight. And discovery, we can run surveys as well on our users and the general public to understand like is this working? Are we raising awareness of these different features? But essentially we’re operationalizing it by literally tying the company objectives and projects directly to these 3Ds and making the success criteria about.

Beah: Yeah. One other place that it shows up it is we just recently f had Hack Days, which is a convention at DuckDuckGo where everybody there’s a few days in a week that basically people can work on any project that they think would be interesting to work on, dedicate three days to outside of our normal like prioritization rigor. And at the end everybody who wants to demos in a company meeting the the project that they worked on. And we give out some awards and special mentions. And those are grouped in terms of like, you know, best of in those three categories dependability, delight, and discovery. So yeah, it feels like over the last year it’s really kind of woven its way into the the fabric of the company. And I think that’s been I think people really like it too. Like it just, you know. People come to DuckDuckGo from different functional backgrounds and skills and it like gives us some shared language to reason about product decisions and prioritization. I think it’s I think if you surveyed people it it would be a you would find that like it’s been a really rewarding shift for the company.

Gabriel: Yeah, I think people in particular like the permission, like you were saying earlier, for delight, you know, to be able to spend a little extra time and on polish and trying to do something really cool that they’d like to their friends and family to see that they’d be proud of. You know, like I think that’s where you get to when you even go beyond dependability into delight. Also just wanna say that we had an earlier Hack Days episode, so if that is any interest, you can go look at that in the archive.

Beah: We’ll put that in the show notes. I get to say that because I don’t write the show notes. So let’s see.

Gabriel: Yeah, well they’re just a transcript of the whole thing, so effectively effectively you’re right.

Beah: Cool. Okay, well that sounds like a good place to wrap. All right.

Gabriel: Cool. Well thanks for you.

Beah: Thanks, Gabriel. Later.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit insideduckduckgo.substack.com

In this episode, Zac (SVP, User Insights) and Mario (Marketing) discuss our study on the AI privacy problem: from stats and surprises, to AI hot takes, and how DuckDuckGo can help.

Show notes: Learn more about the study and our findings here

If you have feedback on Duck Tales, or episode ideas, email us at [email protected].

Disclaimers: (1) The audio, video (above), and transcript (below) are unedited and may contain minor inaccuracies or transcription errors. (2) This website is operated by Substack. This is their privacy policy.

Mario: Okay, there we go. Hey everyone. Welcome to Duck Tales, where we go behind the scenes at DuckDuckGo and discuss the stories, the technology, and the people that help build the privacy tools for everyone. In each episode, you will hear from employees about our vision, product updates, engineering, marketing, but also our approach to AI. Today you’re going to hear about some brand new research, very fresh, that we ran about what Americans actually think about privacy and AI. And I’m here today with someone who happens to have the exact same haircut as me. And that means luscious locks, of course. Zac, do you want to introduce yourself?

Zac: Yeah, hey everybody, I’m Zac Pappas, SVP of User Insights. I’ve been at DuckDuckGo for 14 wonderful years, and I’ll be talking to you about this study today.

Mario: Wow, fourteen years. Yeah, we’re we’re very lucky to to have you here. you’ve been, you know, working in Insights for a long time. You’re one of the first, very first DuckDuckGo employees, is this correct? Yeah.

Zac: Yeah, I think it was technically the second employee back when we were just a few people around the Paoli office here in Pennsylvania. So it’s yeah, it’s been a long fourteen years. but I think it’s been really exciting and things changed so so frequently that it doesn’t really feel like we’re doing the, you know, same insights or kind of same research over and over again.

Mario: Yeah. Well that’s that that’s pretty good. But it’s also useful because it means you also have about as long as a view as anyone on how people feel and think about privacy, you know, what they want protected, what they say versus what they do. So I think this is gonna be an interesting conversation. And yeah, by the way, you’ve introduced yourself. I’ve been talking, I have been rude, I haven’t introduced myself. My name is Mario, I’m in the on the marketing team. I’ve been here for almost four years now. a fun fact about myself is that I like to tell people I once came in fourth in the Eurovision Song Contest. For those who don’t know, it’s like it’s big music kind of song festival and competition in Europe. So yeah, I like to say I once came in fourth, but kind of. So not exactly. And I’ll just leave it at that. what about you, Zac? Do you have any fun facts you want to share? Or obnoxious facts? Also loud.

Zac: nothing that interesting or stand out. I guess y like most people, when I have the time I will go out to my garage, fire up the forge, and do a bit of blacksmithing. so pretty standard stuff.

Mario: Wow, okay, that’s that’s quite cool. I mean, this is this is Duck Tales, not Zac Tales. If it was Zac Tales, I would ask you a lot more questions, maybe put in a request. So

Zac: Next episode, yeah.

Mario: yeah, next time. Okay, let’s let’s jump in. So we as a company and you particularly, since you’ve been here for so long, we’ve been studying how people think about privacy for close to, I know, in this case like 15 years or more. You know, what’s what’s new is doing that specifically for AI. in June, our user insights team surveyed 1944 US adults, all quota balanced to census on age, gender, region, et cetera, about their AI privacy concerns, what they know, what they don’t know, what they actually do about it. you’ll find the methodology on the show notes if you’re interested. And this is this is what we want to talk about today. So, Zac, let’s let’s start with the juicy stuff. To you, what was the most surprising thing in this study?

Zac: I think given the fact that AI has been so pervasive and around for so long, most surprising thing was that I think forty-three percent of everyone knew none of six basic facts about how how AI chats are handled. So that includes a third of active AI users who can’t, you know, recall or kind of signal what basic data privacy and data handling practices that the AI services that they’re using are actually adherent to.

Mario: Well— well, can you tell us a little more like what facts you’re talking about?

Zac: Yeah, I actually wrote them down so I can get them correct for you. So the facts we asked people about were AI chats can be subpoenaed by courts and law enforcement. AI chats can be made public in a lawsuit. Employers can access work or enterprise AI accounts. Settings to delete or stop storage vary by provider. Free accounts often have weaker privacy than paid accounts, and humans can review your chats. Some of these maybe you would expect most people who haven’t dug too far into AI or privacy policies, they maybe wouldn’t think of those things first, but certainly human employees. Employees can review your chats. We’ve seen instances of this in the past with other companies in data collection when there’s a breach or when there’s a mishandling of data. Those are some pretty basic things that I would have assumed people at least had some suspicions about, but it’s surprising that they didn’t.

Mario: Yeah, this is this is a little yeah, it was a little unexpected the the level of of at least to me, you know, listening to this. I I was looking like looking a little bit here at the at the details. Like I saw, for example, on the that the AI chats can be subpoenaed by the government, for example, which means they could then made accessible in public because there were a lot of news about this, about some cases about AI companies. It was about only twenty five percent of people knew this. So why do you think people don’t know these facts given it has been you know, so much on the news. I know this isn’t in the study, but just curious about what you would you

Zac: Yeah, I I mean i it it’s definitely a different technology. I think it’s been a lot more opaque, the way that it’s been integrated and with the speed that it’s been rolled out, and the kind of the breadth that it’s been rolled out to most products that people are using. I think they get a different flavor and a different version of AI everywhere that they look with, you know, incomplete data practices, meaning they haven’t, you know, execute at the executive level, most companies haven’t really figured out how to deploy AI safely. So what they’re getting is say a rushed experience, but something that probably wasn’t built in the best interests of data privacy or kind of long-term sustainability in that regard. so here, yeah, it’s a it’s a maybe a combination of it being kind of a pervasive, like broad applic broad applied technology as well as how fast it has risen. and maybe my own personal experiences, there isn’t that that much useful information for a mainstream person, maybe who’s not a developer or a coder who’s you know gonna be really in the weeds of the technology. It’s kind of hard to find some of the better, you know, more mainstream or general approaches than how to use it safely.

Mario: Yeah, yeah, I can s I can see that. Thanks for that. So yeah, we know what’s the most surprising thing about the study for you, but like asking the bigger question, like, are people worried about privacy in AI in the AI context?

Zac: Yeah, absolutely. Seventy-five percent of people are worried about privacy when using AI. And that has been consistent for years as we’ve been running these studies. And we’ve seen this with external research as well. invasion of privacy, just the discomfort and uneasiness of using it and not knowing how the data’s being handled, and surveillance are usually around like 50% of what worries people the most when it comes to AI, and usually something privacy specific is in like the top two factors among those that are concerned about or like withholding use and not using it at all as a c it’s a contributing factor to why they’re not.

Mario: Yeah, and and yeah, and I I remember some of these studies that are published, like outside of our own, and it’s always like main concerns about AI and there’s usually one that’s like, you know, economic impact or job losses and then privacy is usually hanging there on the top. So I I definitely review myself.

Zac: Yeah, absolutely. And you know, I I think that gets lumped in with a lot of things. particularly the thing that I think most people do end up worrying about is that it’s going to have some impact on their financial situation down the line. That’s your personal details being involved in a breach, credit card details, somebody getting into or unfreezing your credit account and trying to open a new line of credit, like a lot of these things at the end of the day, I think, get back to the comfort and stability that people have with either their finances or their way of life. is it gonna jeopardize you know a job application that I have in the future to find something that you know I’ve put publicly online or didn’t know that I was putting publicly online through use of a service? So I think that we should expect that it’s going to be a top issue, especially as we kind of continue into the future as we integrate AI with all of these other services that are being used, whether it’s other apps or subscriptions.

Mario: Yeah. Yeah. And and I also remember other studies that that you guys have run and where it isn’t like there’s a one single thing about privacy that that people are worried about. There’s a broad concern about privacy. And and for some people it’s generic on like I don’t like the surveillance. I I just I don’t it’s an invasion of privacy. It can be a little a little more amorphous. For others it’s a bit more concrete, but it’s it’s the main thing is that there’s a broad general concern on this on this topic. And

Zac: Absolutely. Yeah.

Mario: So there’s a big concern about privacy. Seventy-five percent of people worry about it when using AI. What are they doing about it?

Zac: It’s a tough story to tell. not tough because we don’t have the data or that we don’t understand it. It’s just like kinda hard to to piece through. So about forty percent have taken no action at all to protect themselves. but what that means is sixty percent have, and about half of them of that sixty percent, have like just either, you know, avoided using AI or maybe toned down a question because of privacy concerns. And so that kind of nets out to about one in three of the US adults that we polled at least have held back from using AI in cases where they’ve had a privacy concern. You can imagine, given the volume, the number of people using AI, that is a lot of potential interactions every day that people are either withholding or kind of transforming into a way that they feel more safe in using it. Others are actually seeking out the privacy options that exist. So there are some things like temporary library chats, switching between multiple accounts or services so that you’re not, you know, contaminating, so to speak, one AI with some context that you might not want it to have. But where there aren’t strong defaults, we can expect that to be an uphill battle as we saw this with search privacy and browser privacy, is that people tend to default to whatever the first thing is that they see in the search that they’re using. So it’s no surprise that the number isn’t higher because it’s not easy for people to find, even though they are demonstrating a lot of pent-up demand and action on it.

Mario: Yeah, I I this is interesting. I I I remember once being in like a a qualitative study and I’m we’re just talking to people and you know they’re talking about their concerns. And and I remember there were a few people that they were saying they would, for example, they would often when they’re asking something to AI, they’ll even change what they’re writing. So they will not write what they want to ask, they will use some code words for example. So there’s there’s people are are are kind of editing themselves, even when they’re

Zac: Yeah, for sure. Another big part of this that we didn’t study specifically here, but that I can imagine plays a large part in the psychology behind this is not having a very clear policy even on the like work-administered AI or if there’s an organization that you’re a part of, if they’ve given you access to a you know company tool, usually there’s some concern there about what the admin of that tool can see, what you know, how the company is collecting all of that data even from their own employees. So it kind of goes back to the idea that they’re not given the the right instruction or the right success setup to get them started with AI in a way that’s actually going to help them long term and abate this issue of them having a concern about their privacy the whole time that they’re using it.

Mario: Mm-hmm. Yeah. Yeah, of course, of course. so yeah, people have these concerns. Do they trust the AI company?

Zac: Yeah. No, I knew you were gonna ask that. No, it’s fourteen percent of respondents say that they mostly or completely trust AI companies. I think it’s thirty-nine percent have no trust at all. and that’s been, you know, so one of the more consistent findings. especially I think it gets worse as companies continue to demonstrate some pretty poor handling of the rollout of AI, both from a privacy perspective and also just a UX perspective. I think it it contributes a lot to the mistrust that people have in the tool when they can’t really understand why it’s being integrated into their product experience. but if you look at something, you know, different than that, a place where AI hasn’t really been integrated that much, at least in a salient way, you can look at a stat, this US government trust in AI was even worse. It was 48% have no trust at all in the US government to mostly or completely or to to prep protect their data.

Mario: Hmm. Okay. well. Yeah. I’m I’m I can imagine it’s not just US government specific, I can see and being generalized even in other places, even though this is just a US study. Wow, okay.

Zac: Yeah, I mean maybe my o my my other personal take on this is there really aren’t a whole lot of good actors or, you know, role models in the space. I hope DuckDuckGo can can really stand for that and show people how to do it well. But if you think about people who are doing, you know, good AI rollouts, particularly the way that they’re handling, you know, user data or other types of data, there aren’t that many great examples. I can’t, you know, even think of any right now.

Mario: Yeah. Yeah. Okay. So we have three quarters of people that are, you know, of people that are concerned about privacy when using AI. We have about two-thirds of the US using AI in some way, shape, or form. So how do you kind of, you know, how do you how do you think about these two, you know, similarly contradicting, you know, stats if folks don’t trust AI with their data? Like how do you how do you explain that?

Zac: question. That’s the classic privacy paradox. But as with the past, I think it’s kind of a a load of BS in the sense that people still use things that they don’t trust, especially when they don’t have another choice. So a good example might be like your cable company. I don’t think most people you know have a really high affection for their cable company, yet they are forced to use it. They, you know, use it in mass. So oftentimes there aren’t other options. And if DuckDuckGo is an example, like people would rather use something that you know doesn’t feel like they’re paying for it with their data. And I think it was just the stat in our study that said 58% of AI users agree that like privacy keeps them from doing more with AI than they currently do. And goes back to that stat of the one third of Americans kind of like preventing themselves from putting something into AI in one of those cases. Like that’s a lot of this pent-up demand and uneasiness that shows you people are still using the tool, but they could be using it, say more more happily or in a more delightful way that doesn’t come with that that trade-off to their concern.

Mario: Yeah, yeah, yeah, totally. I I can see that. And so okay, clear. There’s concerns. you know, there’s a privacy paradox. Is there anything like when you looked at this broad data about concerns trust, is there any pattern when you look at specific segments? Is there you know, are these are these concerns like common across, you know, different segments or

Zac: Yeah, it’s funny, is this was a question that was asked a long time ago when I first started because we we used to look you know at the same thing with regards to search privacy or browser privacy. Is privacy you know more pernicious or a greater care for certain demographics or certain segments? and and no, at least here, like same thing with with broad privacy, it stretches across gender and political affiliation, urban versus rural, psychographics, demographics, and really something that I think everybody’s concerned about. If you you know apply some some basic rigor to that. I think you could say, you know, people who are less technically inclined, maybe older folks, or people who are less, you know, educated might have more of an aversion to it in certain cases or for certain applications. But broadly it is something that stretches across all walks of life that we that we’ve at least that we’ve studied.

Mario: Thank you. Thank you for that. So, okay, moving from this this this area. was there like when you go through the data in this study, was there any other facts that you think would like surprise people if they would hear about it, anything that came out of the study, any insight that would surprise anyone, you know, reading through it?

Zac: Yes, and we are gonna put something up on our Spread Privacy blog about this. the stat that I thought was most surprising was 32% of all AI users have shared something with AI that they have not shared with a family member, friend, or like a doctor. so that just kind of goes to tell you that there is this large contingent of people who are kind of giving away more to AI than they would even to the most trusted confidants that they have.

Mario: Yeah, so so in a way they are using AI as a confidant more than humans, sometimes. So

Zac: Yes.

Mario: yeah, it’s not surprising. I mean, you know, just you know, just the talking to people and and and seeing the news and I c I can see that, but is this something again I’m gonna ask you the segment question, is this something that is more prevalent on certain groups and less so on others?

Zac: well earlier you were asking about privacy concerns, or privacy concerns are are prevalent. Across pretty much every psychographic and demographic, and like I said, something I think everybody feels. On usage and the way that they’re using it, the interesting thing that we found was that parents are actually confiding in AI almost twice as much as non-parents. So among parents that use AI, 43% have told them something that they would have never shared with another person, and that number is only 25% for non-parents. So there’s this you know large gap, and you can kind of think through some of the reasons why that parents would versus non-parents would be kind of confiding more in it or giving it more you know lucrative information from them. partly I think because you know there’s a a lot of certain life stages where that level of expertise is both like expensive or inaccessible. So parents probably are asking a lot of questions that might be a little bit embarrassing or a little bit too difficult to reach out to somebody about, maybe requires a lot of context. And so here you see this behavior with parents kind of confiding more in AI compared to non-parents. And I do expect that if we— So,

Mario: Yeah, and and and that is a very interesting stat. And it reminds me once when I was talking with a few let’s let’s say AI users who were were parents from another study, and I remember them talking about the tension of sharing things with AI because it’s useful in many ways, but at the same time there’s also concern. So that’s it’s just interesting to have this one. Thank you. Thank you for this one. So you mentioned also before AI training, like AI chats being used to train AI. So i y you mentioned that. you mentioned that in general people didn’t know didn’t know about all o any of these facts about AI, but specifically about this one, about training. Like do people how many people know about it? How do people feel about it? Because this is something that’s talked about a lot.

Zac: Yeah. Yeah, so the earlier stat was, you know, forty percent of people can’t name even like one of the six basic stats about how the data is handled. but if you look specifically at the stat or the the the criteria of like whether or not people are aware that AI services can train based on your data, 53% of people didn’t know that. Or or they weren’t sure that their conversations were used for AI training. And when they learn about it, of that 58%, it’s, you know, about half of them or more than half of them are really uncomfortable with it. And I think 30% are very uncomfortable. So it’s something that I think as people do realize it or find out how it’s being applied, and learn that these, you know, that if you take that group of people who had no idea about any of these six concerns, the more exposure that they get to it, the more that they learn about it, the more concerned that they’re likely to develop.

Mario: Yeah, yeah, and my awareness of the of the issue and it’s it’s so important. Okay, that’s that’s interesting but also good good to to understand. so we’ve been talking a lot like about the study and like I think we can almost finish the start finishing the conversation here on on the study. But I’m just curious, especially with your you know, long experience studying privacy and technology, how is studying AI privacy different from studying privacy in other technology or studying the the consumers and users’ opinion about it. Like what’s what’s different from what you’ve been studying for for many, many years?

Zac: that’s such a great question, Mario. there’s a lot that’s different. There’s also a lot that’s similar. I guess one of the hard things is there’s all there’s always a background rate of people’s misunderstanding between where certain products and and others begin. So search to browse was historically one of the kind of boundaries that I think people had a hard time with. Where does Google Chrome stop and Google search start would be kind of something that we would kind of try to get our heads around. Here at the simplest level, people just give AI more. It is a, you know, chat box with a lot of text entry. It makes it very easy to just give it kind of more information. And they also do that through voice prompting and being able to use your microphone, connecting services through either MCP or other kind of connectors and just giving it tons and tons of context that I think lead people to providing more deeply personal information, a richer set of data. So if you think about what the AI company sees on the other side of that, they’re just getting a lot more. And so if you can kind of go back to the risk that people are rightly concerned about, they’re giving a ton of data. There’s a ton of interpolation on top of that about what that data means. It’s very opaque, if not completely unknown, how the the AI companies themselves are handling or storing that data. And then you come back to that risk. If there is a breach, if there is some bad actor or an employee who has access to this, what they have access to is a completely different level of detail and context than than prior. So in researching this, something that we try to take into account is one, this this kind of background rate of people who, you know, share either a deep cons like share a deep concern as well as a like low understanding of exactly— exactly what the concern should be about. And like that, sussing that out a little bit more, I think, is something that we’re gonna have to get a lot better at over time because the products are gonna get more complicated and, in my opinion, more homogenous. almost every app that I use today has some kind of a chat box in it. They all look very similar, they all act somewhat similar. and at the end of the day, I think what people are really trying to get back to is a solid understanding that their lives are getting better and they’re not trading off something extreme like their privacy or their— yeah, their wellbeing as a result of that.

Mario: Yeah, the the black box. you know, this is this reminds me of like I remember once watching Chanel, one of our researchers, and and she was talking to to someone and this person was was saying that with with AI, it’s it they were saying if if someone sees what I’m typing on a search box or on my browser, they’ll know what I do. But if they know what my AI chats are, they know how I think. So there is this feeling of a lot more intimacy and sensitivity and exposing your putting your brain on a paper, like it was even an expression I’ve heard someone use. That yeah, I think it it connects well to what you to what you just said.

Zac: Yeah. Yeah, exactly that.

Mario: Yeah. well, cool. Well, Zac, I think this ends the study discussion. You know, it’s good you give us this context, but you know, this is a little it’s a little it’s a little dire. Like where does this leave, you know, internet users and and people, you know, who just want to use it to, you know, improve their lives and and and you know and yeah, essentially. What’s do you have any words for us on that?

Zac: Yeah, hopefully these people end up finding out about DuckDuckGo. because we’ve made AI optional and private by default. So this is something that, you know, because this is our bread and butter, we’ve tried to understand it and really make a product that serves the interests of of you know mainstream people looking for useful AI that doesn’t come with all of these privacy trade-offs. More so though, like this is about trust. And we’ve built our products around raising the standard of trust online. And I think a lot of companies are going to have to meet users in the middle. Otherwise people, as they normally do, will vote with their dollars or kind of walk out of these products in search of something better.

Mario: Yeah, yeah. Well, okay, that’s it’s good to to end on this on on a trust and hope. And I’m glad we’re you know as a company that we’re trying to help in in making things better for for everyone. So okay, so my you know, I think like if I could summarize this, people are worried and check me and check if I’m saying something wrong. People are worried about AI and privacy. they’re worried, but they still use AI, even often as a confidant that they’ll say things that they don’t say to others. when you do inform people about the specifics of how they are at risk, they don’t like it at all. people that are already concerned about privacy, many are taking action. For example, by not asking certain questions or using AI less. and this is kind of like what I’m what I’m taking from it. And again, it’s good to know that, you know, companies like DuckDuckGo, but others as well are building tools, are building tools to solve this this issue. And this is a better place to be than, you know, nobody cares that there’s there’s people acting on it and that people are also taking action for it. and yeah, so Zac, I thank you for this, thank you for this overview, but I don’t wanna, you know, I don’t want to end on this on this note. So we’re gonna try to do something we have never done on Duck Tales. Are you willing?

Zac: yeah, this is this is not my first Duck Tales. I don’t know what you’re prepared for. So let’s let’s do it. I’m excited.

Mario: Okay. Well that’s okay, very good. So you’ve heard about Subway Takes, like popular online series. A lot of fun. Yeah, Subway Takes, hot takes. Yeah, yeah, or hot takes. Yeah, that

Zac: Like hot takes, set of things like hot takes. Okay. Yeah, yeah, yeah.

Mario: that works, that works well. So yeah, so we’re we’re gonna adapt this format a little bit. I’m gonna give you a topic. You’re gonna give me your hot take, and then I’ll tell you if I agree with it or I disagree with it. Okay. The most important that’s honestly the most relevant part. So we’re gonna

Zac: The most important part. Okay. Yes, yes, yes, yes.

Mario: call this Zac Takes. Okay, it’s a new segment. Very good. Ready? Okay,

Zac: Love it. Yes.

Mario: so what’s the most overrated thing in AI right now?

Zac: frontier models. Do I have to explain these by the or should I explain them? Lightly.

Mario: Slightly, just slightly.

Zac: Frontier, I just think if what you were using yesterday was now all of a sudden invalidated by something that was released today, then we kind of lost the point. And I understand this is a little bit of a loaded question and I don’t get enough time to respond to it, but I’m gonna say frontier models. I don’t think they’re as big of a deal in terms of my experience.

Mario: Yeah. Very good. Fifty percent agree. Fifty percent agree. That’s where I’ll leave it. What’s what’s the most underrated thing in AI right now?

Zac: Yeah. same this is where I was turning— task completion. I think that is the thing that we should be talking about and whether or not people are actually getting something better than they were getting before. Can they order pizza faster or better, more consistently? Can they converse with their family about things that they couldn’t do before? I think we kind of lost sight of the basics of building good product is satisfying task completion.

Mario: Well, I’m gonna give you a I’m gonna give you a another fifty percent agree on this one. I would counterpoint with I think giving people control of if, how and when they use AI. For me, that’s like an underrated feature, let’s put it like that, that I think we could all we could all benefit.

Zac: That is good. Yeah.

Mario: So in five years, another one. In five years, do you think private AI is gonna be its own category? It’s gonna be a checkbox inside every big AI product.

Zac: I I would say both. I think even I mean now it is a table stakes feature I think that people are are expecting, even though most companies aren’t delivering it or delivering it transparently. so I would say that it’s both. And I do think before long we’ll see a lot more vertical integrated AI where somebody’s gonna try to make a play either for privacy or some level of trust, which is the big differentiator.

Mario: Okay. I can’t not say a hundred percent agree on that one. So final two, what’s an everyday object that most deserves a privacy policy?

Zac: the easy answer would have been like mirror or something, but I’m gonna say hotel key card because and this is just a personal thing. Every time I have a hotel key card, it is I I don’t know if I should throw it away. Do I return it to the desk? Should I shred it? I’ve heard that there’s nothing on there. It’s just my I d I don’t know, but at this point I would just like to know when I receive one. Yeah, if there’s just maybe a little disclaimer on the back of the card that could just tell me exactly what’s on there. I’d feel a little bit better about how I can.

Mario: But information is important. You’re confusing. Yeah. Very good. Finally, last one. Last one of probably the last episode of Zac Takes Ever. What is the single most correct way to spell Zac?

Zac: There’s no way to know. That knowledge was lost centuries ago. I did in high school, I don’t know if if this this person’s gonna be watching this, but I did in high school try to make a play for Z-A-Q as the way to spell Zac. Yeah. Yeah.

Mario: Zac, I’m gonna say a hundred percent agree. That’s it. That’s it. We found the answer. We found the answer. No way I need it.

Zac: I haven’t seen anybody pull it off. I don’t think I am the type of person to pull it off, but if I if I would have started in the early two thousands, I think by this point it would have been fashionable.

Mario: There you go. Very good. Very good. Well, nicely done, Zac. Anything else you want to add before we wrap this?

Zac: Nothing for me. Thank you so much, Mario. This is great.

Mario: Very good. So Zac, thank you so much for joining Duck Tales. And to everyone listening, if you’ve got feedback on the show, episode idea, you want to complain about the host, want to know more about this study, for example, write to us at podcast at duckduckgo.com. That’s podcast at duckduckgo.com. We read all of them. We will listen to them if it’s an audio recording. We will see the video in case you’re sending us a video. We’re multimodal. So thanks, Zac, and see you around.

Zac: Thank you.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit insideduckduckgo.substack.com

In this episode, Beah (Chief Product Officer) and Chris (Partnerships) discuss the DuckDuckGo Subscription: the core protections, private AI, and how the new Pro-tier serves AI power users.

If you have feedback on Duck Tales, or episode ideas, email us at [email protected].

Disclaimers: (1) The audio, video (above), and transcript (below) are unedited and may contain minor inaccuracies or transcription errors. (2) This website is operated by Substack. This is their privacy policy.

Beah: Hello and welcome to Duck Tales, where we go behind the scenes at DuckDuckGo to discuss the stories, technology, and people that help build privacy tools for everyone. In this episode, you are going to hear about our Pro subscription plan. And my guest today is Chris Calvi. Chris, do you wanna briefly introduce yourself before we jump in?

Chris: I— I would love to. Thank you for having me back. I was on— a quick ad for people who have not listened to the original subscription overview Duck Tales episode. Gabriel and I did that six months— I don’t know, actually probably nine months ago at this point. So go find that. But I’m coming to you from sunny New Jersey. And I’m a member of the partnerships team here at DuckDuckGo. And I’ve been at DuckDuckGo for about five years. I love it here. And most recently helped run efforts to add this new Pro plan to the subscription, which we’ll be talking about on this episode.

Beah: Nice. I’d like to think of this episode as the Pro edition of the previous podcast episode that you recorded with Gabriel.

Chris: Okay, that’s fair enough.

Beah: Just to set the bar.

Chris: Yeah.

Beah: All right, and I’m Beah, I’m on the product team, if this is your first time meeting me. And yeah, I think we can kind of jump in. Like, let’s start with— can you just explain, Chris, what even is the subscription, what we’re talking about?

Chris: Yeah, and I always like to start with, like, why the subscription first, and then I’ll say what the subscription is. So the why: you know, DuckDuckGo has a ton of fully featured, excellent, free-to-use, no-login-required functionality. And we’ve had that for, like, what, fifth— 17 years? How— how long has DuckDuckGo been on? We’ve had it for a very long time. And— and so we, though, realize that there are— there’s extra functionality and features and protections that we would like to offer to— to our users, that you can’t offer— either can offer free, for free, because they have a material c—, like, a substantial cost to us, or it can’t be, like, ad-supported like the search. So— so two years, a little over two years ago, we launched the DuckDuckGo subscription. And— and that includes four protections— that are aligned with what I just mentioned, things that we sort of, like, can offer for free. And so there’s the identity theft restoration, which internally we call IDTR. And that is essentially, if you—

Beah: Not me. I call it identity theft restoration because I try to avoid—

Chris: You— you don’t like to use acronyms.

Beah: I’m anti-acronym, yes.

Chris: As a culture, we typically are. Though I like to break the rules. I break the rules.

Beah: I continue.

Chris: So in the event that your identity is ever— is ever compromised, your n—, your identity stolen and is used, for instance, to open up another credit card— for— that’s not— that you didn’t open, we have a phone number that you can dial into. Actually, let me share my screen and I can show you what I’m talking about.

Beah: You can do a live demo of calling a phone number and see what happens. Just don’t do that.

Chris: That would be risky. No, I’m just kidding. It would be good, but it would also be boring. So let me know— does this work? Do you see?

Beah: I see.

Chris: All right, so to get here— after— this is kind of like jumping ahead, but let’s say you have a subscription to DuckDuckGo already, you would log in, you’d— you’d come to your— if you have the app installed, you’d come in and you would select DuckDuckGo subscription. And in subscription settings, if you didn’t already have this device connected to your subscription, you would have to do that as a first step. But once it’s connected, then you have it and you can just come in here and go to the element of the subscription that you want. So let me walk you through those pieces of the subscription. The first one I was mentioning was identity theft restoration. And so what you would do is— in this example I mentioned where somebody opened a credit card in your name, and you’d want to re—, you wanna deal with that, you’d come in here and you would call our— this has some information for you about how it’s gonna go. You will click “talk to an advisor” and then you would make a simple phone call and they would help you make things right with your— your credit card companies, your banks, your credit bureaus, things like that. So that’s identity theft restoration. The— the subscription, as I mentioned, has four key pieces of functionality— we call it four protections in one. The second one is personal information removal. So this would also be available from your— your settings screen. You’d click on here and you’d click “open personal information removal.” And similar to the identity theft restoration, this would bring you to a page where it shows you what— what it’s going to do, what it is and how it’s going to help you and how you get started. And what personal information removal is: in the US, we have this insidious issue where data brokers essentially aggregate personal information about you, about everyone in the country, and they put it on the internet or they— they’ll sell it. Sorry, they’ll put part of it on— on the public web and then sell part of it. And this includes things like your name, your address, your phone number, your date of birth. And even relatives, like— that, like, lived in the household with you. I’m sure you’ve probably seen this if you’ve ever searched for yourself. And so with personal information removal, we go through and we opt you out of the listings on over 60 of those sites. And we’re adding more all the time. And so you would just come in here, you’d click “get started,” and this would ask you a few things, a few pieces of information which stays on your device and does not go to our servers. That’s important to mention here. And then goes through a process that’s run locally from your machine and takes some time. It takes, like, anywhere from a few hours to a few weeks or longer to get you fully removed from all of these sites. So some of the sites can act quicker than others, but this will go through and— and opt you out from being listed on those data broker websites. The— the— and interrupt me with any questions you have about this, any of these functionality, but we went through—

Beah: No, this is good, this good.

Chris: We went through the identity theft restoration, the second one being personal information removal, and then the third one is the VPN. And the VPN is— stands for, since we don’t like acronyms, virtual private network. And what it— yeah.

Beah: I guess I do allow VPN. I’m gonna allow it.

Chris: I mean, but— what it— what a VPN is, is it— is it, we set up a— anonymous secure server, and we have them in 30, or over 30, countries around the world. So wherever you’re at, you can connect to the closest one. And— and what it will do is securely— it encrypts your traffic and securely and privately tunnels it through these anonymous servers and then out to the resource that you’re requesting on the other end on the internet. So, like, all of your internet traffic is basically being tunneled through this— this private secure server and then out the other end. And so this— why— what, like, what this does is it protects you in— in— in two— two, like, key ways. One is your ISP, the internet service provider that you’re connected to, won’t be able to see the sites that you are accessing, like what sites you’re visiting. And that helps you because they’re not then able to— either it makes it more difficult for them to censor the content that you can access, and/or build an ad profile about you, for like an advertising profile, and then like resell that, for instance.

Beah: Okay.

Chris: And then on the other end, it— it protects you— the sites that you are visiting, they’re going to see the IP address, like, kind of like who you are, of the server. And so they don’t actually know who you are unless you were to, like, log in, for instance. So it gives you protection from both— in b—, on both of those cases. But, like, the— the— the use case that is, like, most common for— for the VPN, and I should have switched over to showing you this, is the— is when you’re at, like, a public Wi-Fi location, in an airport or a hotel or, like, a— a coffee shop, you don’t need those— you don’t need them seeing what websites you’re accessing. So VPN gives you that extra security. And I— as I had mentioned, you can change the location that you’re— the server, where the server is. I just keep it on “nearest location” and I’m on it right now. I keep it on all day. I use it all the time. And so I think that’s, like, the key thing about VPNs. I don’t know if the— if I did that justice.

Beah: You did. You’re doing such a good— I hate to rush you along, because you’re doing such a good job of explaining these features, but we have to get to the Pro plan. So I think you’re kind of getting to here.

Chris: The key thing. Okay, so let me just get you— yes, let me get you that last thing. So the—

Beah: Yes.

Chris: So the fourth feature is Duck.ai. And Duck.ai is free. But if you want higher limits and be able to do more on it, then— if you have a subscription, then you get higher limits, and that’s the fourth piece of functionality in the subscription. So there you go. That’s our subscription, those four things.

Beah: And explain the Pro plan versus the basic subscription versus free with regard to Duck.ai.

Chris: Okay, so free is— we don’t— we technically don’t ha—

Beah: And do you want to?

Chris: We call it a free plan. It’s not a free plan. It’s not, like, a tier in itself. DuckDuckGo— yeah, I should turn off— I could turn on— stop sharing.

Beah: Yeah.

Chris: So, okay, are we back?

Beah: I’m right back.

Chris: All right, so the— the f— DuckDuckGo, I mentioned before, it’s like everything is free on there until you get a subscription. So there’s— there isn’t really a free plan. You don’t log in. And then the— the— with the sub—, with the Plus tier, or the Plus plan, of our subscription— that was— that includes all of the functionality that I just mentioned. And— and then any— it— with Duck.ai you get increased limits over— and new models compared to what you would get on our— if you were just— just went to Duck.ai and didn’t have a— a subscription with us. The—

Beah: So when you say new models, you mean, like, basically fancier models that are more expensive to run.

Chris: Correct, yes, yeah, that’s exactly right. So just more capable, more f— more advanced models. And— and then the Pro— the Pro plan of our subscription is our newest plan. It just launched two months ago and is— is very much AI-oriented. It is a— it’s for the AI power user. And so for those who are using Duck.ai and want even more capabilities than you can get in the Plus plan, and higher limits to be able to be doing more work in there, then that’s— the Pro plan is for them. And so it includes everything in the Plus plan, but higher limits on AI and a—

Beah: But wait, there’s more.

Chris: —a Pro-exclusive model, which is Claude Opus, and better— like, better reasoning effort you can also enable in there, for when you need to— to kind of, like, go deeper on something. And I should mention the Plus— Plus plan is $9.99 a month or $99.99 a year. And the Pro plan is— is double that. So $19.99 a month or $199.99 a year.

Beah: Yeah, nice. Well, I am a daily user of our Pro plan features. I use Claude Opus via Duck.ai, I think, daily. So I can vouch for it. Well worth it. Can you tell me, Chris, were there any particular challenges with building the Pro plan?

Chris: Yeah, I mean, it was— like, kind— was like we started this in late fall, I would say, is when we started— the idea had been there, but in— in, like, mid-late fall is when we started actively working on this. And it was like adding a new floor to your home. So we— like, the home was built to be one way, and then we were like, well, we actually need to do this too. So kind of— just there was, like, the technical nuances of having that, right? This, like, all of the— every app version that we have, whether you’re using a mobile— you know, if you’re using iOS or Android or your desktop DuckDuckGo apps, all of them had to be updated to be able to recognize these new sub— the new subscription tier, and also to be able to help you get signed up to that, or to upgrade to that if you’re on the existing Plus plan and wanted to jump up to Pro. So I think that was sort of, like, the main challenge. But the— kind of the— to be more specific, the one— the one challenge that— is may surprise people f— is the— the actual subscrip— subscription flow— signing up or upgrading— is— is— is— was the biggest, I think, challenge. Because it’s— the subscription’s in 30— it’s in 30 countries. So we have to account for all of that. We have to account from where you’re coming from when you sign up, so that we can understand which features make sense for you to see, and, like, the— because we want it to— everything, all in this process, we want to be simple. And then adding that sort of second— second plan makes it complicated. You know, you want to be— be able to— to— to be clear to people: these are the things you’re going to get in each plan. And so you have to be very careful there when you’re— when you’re modifying things like that.

Beah: Yeah, you’re going from a “buy or not buy” decision to a “don’t buy, buy, which kind of buy” decision. And that’s going to vary a little bit— what you’re getting is going to vary a little bit by country— and me trying to be simultaneously transparent, extremely transparent, about all of these details and not overwhelm users with details they can’t absorb. Is that fair? Yeah.

Chris: Exactly. Yeah.

Beah: Okay, how about— were there surprises for you, like, after we’ve been live with the program for a couple months now? Like, any interesting feedback?

Chris: Yeah. Yeah. I mean, I think that the, like, biggest— there’s— there’s really two surprises. One is the pace of adoption of this— of this plan, the Pro plan, is much faster and gr— is growing quicker than I expected. And— and so that’s, like, on the— the plus side, the good news. On the other end—

Beah: Or on the Pro side.

Chris: On the Pro— the Pro, yes, to continue that— the Pro side. And then on the— I think, in terms of, like, the constructive feedback side of things— we launched this, and, you know, like, right out the gate when we launched this two months ago, you’d just— like, has been an issue before, I think— if you hit your limits, you would sort of hit a limit for your day and you’d be like, “no, like, I didn’t get any warnings, I didn’t get any, like, educational, what this means.” And so we received fair feedback on that. And so we’ve worked to remedy that. There’s now a usage— for, like, a progress bar in your settings when you have a subscription. So you can go in there and you can see, like, where you’re at compared to your limits. And we have also added some more proactive notifications. So it doesn’t just wait till the end, it gives you a few sort of, like, heads-up, like, you— you might want to consider switching to a— to another model that won’t home your— your budget as quickly. And so that’s— that’s kind of the area where we are spending a good amount of time at the moment, and trying to improve that, and how to be more— you know, it’s going back to the simplified thing, like how to be more simple and transparent with people about when they make decisions, like which models are good for everyday use and which you should be more using— more sparingly— when you really need that extra effort.

Beah: Yeah, yeah, like, limits are annoying on any plan. But if you’re— if you hit a limit and you’re, like, deep into a conversation and you just, like, lose all that context, that’s frustrating for anybody. But if you’re, like, paying for this thing, it’s like, that’s a new level of—

Chris: Yes.

Beah: —right? Like, your expectations are different.

Chris: Very ac— very astute point. And— and I— I think that, to be clear, we are— we are doing more there too. There’s some stuff that— hopefully— I’m not sure when this is gonna— when this is gonna air, but— but maybe by then there’ll ev— there’ll be even additional things that we’ve done to really improve the experience so that people don’t have those issues anymore.

Beah: That’s a good segue. Is there anything that you can share that’s upcoming for Pro that people have to look forward to?

Chris: Well, we are— across, like, both t— both plans, we are always looking to consider what new functionality should we be including there. Does it make sense? And, you know, I— in the— in the— for the Pro plan, I think the big thing at the min— at the moment are the— around limits, right? Around how we can just be— be— peep— make people’s workflow a little more efficient so that they don’t hit these stops, or they, you know, they can just work around them. So that’s— that’s kind of— I think that’s, in the short term, some of the stuff that we’ve got coming.

Beah: Gotcha. What if you are interested in all this but you do not know if Pro tier is for you? Can you try it for free?

Chris: Yes, so both the Plus and the Pro plan have a seven-day free trial, if you have not already used a trial. So if you’ve, like, tried Plus— the Plus prep plan already— and you want to try Pro, you’ll have to buy it for a month to just try it. Or if you— if you’re new, you can try either one for— for seven days for free.

Beah: Nice. Where does somebody go if they want to do that?

Chris: Yes, so you can go to duckduckgo.com/subscribe and you can see more details on everything I talked about today and— and sign up for a trial on either of those plans. But hopefully you’ll give Pro— Pro plan a— a try.

Beah: Yeah, I recommend it. Okay. Before we close, Chris, I want to ask you— this is an experiment. Haven’t done this before at the end of the Duck Tales episode, but we’re going to do it. You know how it’s, like, a popular thing at the end of podcast interviews to do a lightning round, like, fire off some quick questions? I have three questions for you.

Chris: Okay.

Beah: You get— you have to answer. My rule is going to be five words max. Can you it?

Chris: That’s gonna be challenge.

Beah: Okay. Let’s try it. Let’s see what happens. All right. What— model— we’ve been talking about model choice and Pro unlocking new models. What model do you personally use the most in your life, Chris?

Chris: I use Sonnet the most, and then I move— move up— this is really exceeding five words— but I move up to Opus as needed. So, like—

Beah: Okay.

Chris: —usually, like, at least once a day I’ll jump to Opus.

Beah: Sonnet, Opus as needed. I’m giving it to you.

Chris: There you go. Thank you. Yes.

Beah: It’s like a training cycle for the next question.

Chris: Yeah. We’ll see— yeah, let’s see if I can do this.

Beah: What was the location of your last functional team meetup? So, Chris, you mentioned you were on the partnerships team. We do— for those who don’t know, we do meetups where everybody gets together in a given location with their functional team. Also do that company-wide. Where was the last location of your partnership team meetup?

Chris: Rome, Italy.

Beah: Very good, very good. Okay,

Chris: Ha ha.

Beah: —this last one might stump you, but let’s try it. If you were Dax the Duck— Duck, Duck, Duck, DuckDuckGo’s mascot— what color bow tie would you wear?

Chris: Blue.

Beah: Okay, down to one word. You went from, like, 12 to four to three, I can’t remember, to one. It’s very good.

Chris: There is a— there is a, like, Haiku joke to be made here, ‘cause Haiku’s a model, but I’m not gonna— I’m not going—

Beah: That’s a very good audience.

Chris: —there.

Beah: I think you just did. You just nailed it. All right, well, thank you, Chris, so much for joining Duck Tales and talking about the Pro plan. And listeners, I hope you give it a try if you haven’t already. And yeah, see you around.

Chris: Thanks for having me.

Beah: All right, bye.

Chris: Bye.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit insideduckduckgo.substack.com

In this episode, Beah (Chief Product Officer) and Mihai (Design) discuss how the Duck.ai team prioritizes what to build, how we use research and user feedback, and a demo of recent UX updates.

If you have feedback on Duck Tales, or episode ideas, email us at [email protected].

Disclaimers: (1) The audio, video (above), and transcript (below) are unedited and may contain minor inaccuracies or transcription errors. (2) This website is operated by Substack. This is their privacy policy.

Beah: Hello, and welcome to Duck Tales, where we go behind the scenes at DuckDuckGo and discuss the stories, technology, and people that help build privacy tools for everyone. In each episode, you’ll hear from employees about our vision, product updates, engineering approach. And in this particular episode, you’re going to hear from Mihai, who is here to talk about the various UX improvements that we’ve made to Duck.ai recently. Mihai is on the design team. And do you want to just give your give a quick introduction of yourself, Mihai?

Mihai: Sure, thank thanks for having me. My name is Mihai. Originally from Moldova, lived across the world in many places, now based in Switzerland, joined DuckDuckGo maybe around three years ago, and I’ve been on the Duck.ai team almost from the very beginning, so I’ve seen the product evolve since then dramatically. So so yeah, we’ll talk about it today.

Beah: Nice. Well thanks for joining. For anyone who hasn’t met me, I’m Beah. I’m on the product team here. I’ve been around for a while. So yeah, I’m gonna go ahead and ask Mihai some questions about his work on Duck.ai and we’ll just dive on in. So I think I said we’re talking about improving Duck.ai UX, and I’m not sure I said that if you’re not familiar, UX is just a fancy abbreviation for user experience, even though we all know experience doesn’t start with an X. So this means like, you know, making the features that we have and the functionality that we have more usable, tweaking the functionality to better like suit people’s needs and desires, and just make the product more delightful and easier to use. You feel free to amend that definition. But just to kind of like set

Mihai: Feel free to amend definition.

Beah: the set the tone a little bit, Mihai, tell me a little bit about like how we even decide to work on something. Like how do we prioritize UX user experience changes in Duck.ai land?

Mihai: Yeah, as designers, we’re basically I I tend to think of myself as the diplomats that we we try to talk to as many people to get our work done. But we use various signals to inform these decisions. Some of some of them are more perfect, some of them are imperfect. There’s no There’s no reality where we’re working with perfect information. So sometimes we form assumptions, sometimes we rely on research. User feedback, internal feedback. Sometimes we have strategic bets that we place as an organization. We have dashboards with metrics. We have just no we know the competitive gaps that we have, and sometimes we just rely on instinct. So we

Beah: That’s good.

Mihai: we bundle all of these together in our decision-making process. Some of these signals they surface a problem,

Beah: Yeah.

Mihai: others validate the direction and it’s on us to use our own human judgment to determine the right trade-off for for these things. And in the end it narrows down to to looking at where the biggest overlap is with the user problems that we have and where we’re taking the product.

Beah: Yeah, or do all the hand and just then follow it.

Mihai: Yeah, maybe this is more kind of like general response, but no one decision was the same.

Beah: Yeah, right once. Yeah, yeah.

Mihai: So it’s it’s it’s all based on experienced people in the room making the the calls based on the information that we have.

Beah: Yeah, I mean one thing that I’m guessing people I I bet a lot of people would underestimate the degree to which we’re obsessed with user feedback. Like you can’t completely rely on user feedback to tell you how to build a product because, you know, hopefully the people building the product have additional insights about what’s possible and what good looks like that users can’t articulate or shouldn’t take the time to articulate. It’s not on them. But that aside, like we obsess pretty hard on feedback. Like we read an insane amount of feedback. We have systems for categorizing it and making sense of it. I mean, I’ve been caught up in like long discussions where we are really deeply trying to interpret a comment somebody made on Reddit or in the app store and like try to figure out the best way to address it. And so I’m just pointing that out because I think that might not be obvious. Like I think there’s probably a lot of, you know, for a tech company that serves millions and millions of users, I’m not sure that’s I’m guessing that’s not the norm. So that is a big part of it. But then like beyond that, you know, there’s a bunch of other signals we use, like Mihai said, and I know in Duck.ai on that team, like you guys have a rubric, right, that you’re using to actually like evaluate in a structured way a bunch of project ideas and feedback is one of the inputs to that rubric, but there are others as well. So also very maybe distinctive of DuckDuckGo is I think it’s a hyper structured and rational decision making process. Would you agree with that, Mihai?

Mihai: Yeah. It’s it’s quite unique and I think the the great thing is that a bunch of us, everyone here, they’re scoping their own projects, they have their own ideas, they uncover different things that they’re pitching in and that goes into that rubric. And at certain times we those categories of how how we score the rubric, they’re weighted. So sometimes certain elements are more important than the others. And that help us kind of look at what what’s important to us right now and what we’re getting from the users and all the projects that are strong candidates for us to tackle tackle in the next roadmap, for example.

Beah: Cool. Yeah, so let’s talk about something that did bubble to the top recently. Like, can you give me an example of an a UX improvement that you’ve worked on?

Mihai: Yes, so the first example is dictation. I’ll share my screen real quick. Cool, so this is our interface here. So users keep asking for dictation features, so it’s this microphone in the input field. There’s certain people that for them is they would use the input field. They’ll rather talk than type. It’s either a prompt is really long, or you they just want to do it hands free. And shipping this was actually great because it it got us in line with the competitors as well. But internally we we were kind of i it it was the group was split in terms of like if we wanna do it until that user feedback kept coming back, kept coming back. And we decided to to pull the trigger. For from the UX standpoint, what was exciting for me personally is voice interfaces, they have a big affordance problem. That it’s really hard sometimes to know what’s the system status for that and to avoid the jargon is basically you need to know what the what the app is doing at any point in time. And that was an interesting challenge on top of just delivering the feature. And if I if I trigger it, we we do three things here. We label things clearly for people to know what’s going on. You also get a visual cue, this waveform that provides feedback, and as you see me pause, that waveform reacts to my voice, but also the input field has changed its its shape not the height or anything but what’s inside it now is geared toward towards a different input which is not text and we’ve added this nice glow animation to it which is which is really cool so and then it’s trust transcribing and giving back the the text. Another one should should I should I give a few more? Alright.

Beah: Nice.

Beah: Yeah, yeah. Give me another example, please. That’s good.

Mihai: Alright. Another one is batch deletion, which is another feature that was requested by by users. This one is a little different because our we’re famous for our fire button. You you can basically burn all your data and stuff like that. But in Duck.ai specifically we had the reality that you could either delete one chat at a time or all of them. And nothing nothing in between. And we we decided to that we need to give people more precise control. To do it in just one step. Whenever you have 30 chats, for example, and you just want to delete seven at a time, it’s quite annoying to go one by one. So this one, it’s not groundbreaking or anything. It’s been a pattern on the web for quite some time. How you can batch select the different things. This is a great example, thinking back to the signals that we spoke. It’s a great example when you need to know where to follow the process for something when whenever you’re designing something that’s more unknown and whenever you need to trust your instincts more. And this is a project that was basically that where we we used our experience more and the project was short this was delivered and and shipped quite quickly because of that. And what the feature does you can go in and select many chats you want to delete. We have different entry points to how to do it. Usually

Beah: Yeah.

Mihai: we know people click the fire button to open it so we surface a button there for people to actually not delete everything but select multiple ones or for maybe power users whenever something’s selected you can shift click and enter this mode and delete delete your chats. So yeah, this one the reason I’m I’m asking is we spoke about all these signals and how complex things are, but this one was th three designers, we got in the room, we did a thirty minute thing, then I spent maybe two hours on it on edge cases, we put it to review to a wider group of stakeholders and it went straight into implementation, which is which is great because we have features that are a little bit more complex which require all all of that

Beah: That’ll be true.

Mihai: kind of digging, research and all those other things. But this is a great outlier maybe where we

Beah: Yeah.

Mihai: can just roll with differently a little bit.

Beah: Yeah, yeah. Also, there’s yeah, I j while you were talking, I was thinking about we ha we also added not too long ago another another way to delete chat level sort of at your cadence, which is like when you hover do you wanna show your screen again real quick, Mihai? When you hover over each chat in your chat history, you get a fire button. So you can also just tap that. So if you want to do one at a time, but you know, from a list, that’s another way to do it. I think that was a nice win. And then I don’t know, just for a little like real time inside baseball, well, truly an hour ago, Mihai and I were on a a thread where we were talking about with some other folks whether what the equivalent best equivalent of that on mobile is where you don’t have hover, you know. So we just have to if we want I I personally love and use those that per chat fire button. You’re like, I don’t need that, I don’t need that, I’m gonna clean it up real quick. On mobile we don’t have that. So yeah, like is it worth the real estate to just always show the fire button? Is that too noisy or people gonna misclick it? So we were kind of debating that this morning and gonna get it into an internal build and like kind of feel it out ourselves and then you know if it feels good we’ll we’ll release it. But we’ll watch any feedback coming in as well to help guide us on whether it’s working as we we hoped it would. So hot off the process. Alright, do you wanna do you wanna talk about one more we talked about maybe the the model picker might be interesting to show ‘cause that was a nice and nuanced recent UX improvement.

Mihai: Yes, yeah, let me share my screen again. Yeah, this one this one is not the one I designed, so I don’t want to upset the designer who actually done the hard work. But

Beah: Mm-hmm.

Mihai: I I advised this project so I stayed close to it. So I’ll it’s it’s Esteban. So thank you Esteban for

Beah: You wanna give him a you wanna name drop? Give him a shout out.

Mihai: a better model thinker. We’re all grateful for for that. Yes, it’s basically our our team has grown, so it’s it’s great that we we can cover more ground. And Esteban is not new on the team, but whenever whenever I started it was just me. So it’s great to see all of these things basically. Ship and all these amazing features that We can put out much faster. Now this one is basically speaking of old stuff, the old mode model picker had a whole lot more details in in in it, and through research we’ve uncovered that people just could not understand what’s in there. And they were stalling, trying to make sense of it a little bit. Power users have no problem, but I don’t know like general public, people who are actually not experts on just just want to use the tool. It it was a thing that was giving them pause. An another thing that we’ve uncovered which was useful insight is once they got past that confusion, they really valued it and it was the thing that they would recommend

Beah: Yeah.

Mihai: to friends and stuff. And on top of it we also as a company, Duck.ai specifically, we provide all these L LMs from different providers. So this is Some sort of a our own differentiator where you get the private versions of all of these. So we wanted to fix this old problem of people stalling and being a little bit confused about it. And the solution was basically again based on insights, we we realized that much people knew what they wanted to use or had some general idea and they they

Beah: Again, you know, I we like that people looking on a TV.

Mihai: just use what’s giving them a good response. They just stick to it and majority don’t really switch. There are use cases for some that that switch for different particular reasons, but for this particular model or now a drop-down they just needed something much simpler and the job of this is to show that there is choice because we provide all of these things, variety, but also we wanted to remove the guesswork for for people. So

Beah: And then you’re going to

Mihai: Esteban has done an amazing job of condensing this into showing that variety. But we switched what we are doing here to for the user is we’re providing now a recommendation of what we think and what we think that they’ll use to have a great time and a great experience on on the platform. For each subscription tier we have Free, Plus and Pro. We recommend two models, one always good for everyday use. Based on their tier and another one which will give even a maybe a stronger, more comprehensive response, but will use more limits or it’ll it’ll exhaust those limits faster basically. So yeah now it’s kind of it’s much simpler and much more enjoyable at least for for for me and

Beah: Yeah. I mean

Mihai: based on your the research for users to use.

Beah: Yeah, one detail is like before it was a button that you pressed and it opened a modal. Like this it opened a new screen effectively with the with your choices and then you select your choice and then click start new chat. And like even just it’s a small thing, but like the the difference between what feels like kind of a drop down and something that feels like a new screen where you have to pick and then say start new chat, like I I I knew how it worked, and it’s not like one click takes so so much time, but the sort of like cognitive friction of like picking and then and then clicking something again, like start new chat, and like usually I hadn’t even started a chat. Like it just it didn’t feel easy peasy to switch around, you know? It felt like higher friction than you needed to so like you everything else is that even just kind of moving that like changing the UX pattern, I think is a nice, a nice little win. Sweet. But that’s one that probably is like counter to your batch delete example. Like we really did kind of agonize over a bunch of nuances because like with any design decision there’s trade-offs and there’s different cases that you have to account for in different states and like it did it took a while to to reason through it all. Speaking of which, can you give me, Mihai, an example of like a difficult or non-obvious design decision that you’ve had to make recently?

Mihai: A recent one. We always I think as a designer here I o I’m always reminded of how little we know about our user in comparison

Beah: Yes.

Mihai: to everything else we’ve experienced. I remember like we just don’t know the count of how many users we have just because we don’t track them. And stuff like that. Everything is very much on those guesstimates and kind of like I’m people sometimes use phrases like I’m confident in this assumption, like eighty percent and stuff like that. They give a percentage and it’s it’s a wonderful constraint because it’s it’s why I joined. I’ve recently so

Beah: Because it just to like because it’s privacy respecting is what you’re getting at. Yeah, we don’t track.

Mihai: Yeah. Exactly. So and and that goes with everything we touch whenever we’re designing it. It’s you you design something and it then then it’s through that lens of of respecting the the data and and providing a a great experience without needing all of those other things which other companies do. And I maybe don’t have a very good example to give that is recent one that is maybe an older one is whenever you’re getting a response how you’re switching through different models which is something Which is currently being worked on and where you will be evolving that feature soon. But that one, what wasn’t too obvious about it is that switching models is not a new paradigm in user’s mind because this AI thing is somewhat new. That’s why it takes the shape of a chat interface because it’s familiar. And switching a model in other places we’ve seen very much is has different pagination approaches and stuff like that, which is now learned but not natural based on what we’ve uncovered. And every time when I spoke to people, like I’ve seen so many confused face faces about like the pagination of answers and stuff like that. And what

Beah: Button on the natural work is everything natural.

Mihai: What we did, which wasn’t obvious obvious, we basically w took a more linear approach to how those responses are getting surfaced, which feels more natural into a chat interface, a thing that we’ve been using for decades. So

Beah: The patterns are a thing that we have.

Mihai: we kind of went against the grain there. And it wasn’t obvious at first.

Beah: So what would a paginated response look like? Like if we went that route, we would have like what, opened a whole new chat or something?

Mihai: It it no it it creates the same response but it gives you like a counter, like a page for the responses and then you paginate and it gets forked in different streams, almost like a decision tree that you can paginate back and continue conversation in different so that that

Beah: Yeah.

Mihai: even like I spoke to engineers who use AI, like they use Cursor, they use all these like advanced tools. And for some it wasn’t apparent that to to some that this feature worked that s certain way. So even at advanced people who are used to these tools were a little confused by the conventional norms, so to speak. So it made me feel really proud a little a little bit because we kind of took a different approach. And I think we get more than 300K uses on week from that switcher, which which is cool to see.

Beah: And three... coming.

Beah: Nice. Alright. Last question, Mihai, before we sign off, can you preview any upcoming improvements that people should look forward to?

Mihai: It’s the project I just wrapped, which is still being worked on. So the future may still evolve, but one thing we’re trying to add to to the platform is are you seeing my Figma, Beah?

Beah: Yeah, I can see.

Mihai: Okay, perfect. So this is very very rough. I think my is hiding my some of it. Basically we wanna add the share functionality to your responses, something that we we don’t do currently and we wanna do this at the response level but also at the entire conversation level. And once you once you click share, you’re basically sharing a response. You create a public link. That we’ll we’ll leave on our encrypted server with the encryption keys and all those other things that engineers are working on to make that super secure and everything. And then you can copy and and send it to to to people to to get that that response. And you’ll be able to to see it’s as someone who opens the link, you’ll be able to see what was shared and you you’ll be able to delineate what’s yours if you choose to reply or if you choose to add it to your own chats. So this is how it was designed. Sometimes we evolve the work as we play with it live when it’s when it’s in code. But it’s something that yeah should help a lot of people send a link as opposed to send a bunch of text.

Beah: Nice. That eggshell white is looking pretty sharp. Is that what it’s called, eggshell?

Mihai: It’s actual actual TW10 specifically.

Beah: Ooh, eggshell 10. Nice. Awesome. Alright. Thank you, Mihai. Anything you want to add before we before we close this out?

Mihai: This is great. I yeah, this is great. Thank you for having me.

Beah: Yeah, thanks for being on Duck Tales. Alright. Thanks, Mihai. Thanks everyone.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit insideduckduckgo.substack.com

In this episode, Beah (Chief Product Officer) and Marc (VP, Engineering) discuss how we create, communicate, and challenge operational defaults to make better decisions at speed.

If you have feedback on Duck Tales, or episode ideas, email us at [email protected].

Disclaimers: (1) The audio, video (above), and transcript (below) are unedited and may contain minor inaccuracies or transcription errors. (2) This website is operated by Substack. This is their privacy policy.

Beah: Hello and welcome to Duck Tales, where we go behind the scenes at DuckDuckGo and discuss the stories, the technology, and the people who help build privacy tools for everyone. In each episode, you’ll hear from employees about our vision, product updates, engineering, or approach to AI. Today, you are going to hear about how we work, I guess, or one facet of how we work, which is using operational defaults and I have with me Marc, who is going to who I’m going to interview about operational defaults. Marc, do you wanna introduce yourself?

Marc: Yeah. Marc, I’m on the engineering team at DuckDuckGo.

Beah: Okay. That’s about all you need to know. and I’m Beah, I’m on the product team if if you haven’t seen me before. okay, cool. Well let’s jump in. What even is an operational default?

Marc: Yeah. Yeah, so defaults are I would say a deceptively simple concept. at at DuckDuckGo we’re always trying to leverage mental models to help us move more quickly and think more clearly and and things of that nature. in that context, the default effect is seen as a negative, right? And and I think we’ve all seen that in apps and in different places. Here we’re trying to leverage the the upside of that, where we’re basically taking any decision that you can make ahead of time that generally is true or has a right answer eighty percent of the time, we’re gonna call that a default. for that other twenty percent, we wanna leave room for our folks to be able to make exceptions, ideally very thoughtful exceptions.

Beah: When you say default is a negative in the app, you mean like, you know, if you get an Android phone and the default browser is Chrome and the default search engine is Google and it’s really hard for another player like DuckDuckGo to to break through th those defaults. Is that right? Yeah. Okay.

Marc: yeah, exactly. There’s there’s in Gabe’s book of mental models there’s the default effect and again it’s generally negative, but like I said, in this case we’re trying to use defaults as as something that we can leverage to help us go quicker.

Beah: Yeah, yeah. So how is that how is it different than a rule?

Marc: I mean it’s conceptually the same thing. I think rules are just more binary, right? Like this is something you do or you don’t do pretty much all the time. Where here we’re saying that these are things that are generally true. Like I said, eighty percent of the time is just the number that we made up. and that allows people to potentially make faster decisions. but we also want to include encourage people to think about what they’re doing and not just go on autopilot. And I think that’s the danger of catching something as a rule is that yeah, people will just kind of blindly follow it by default.

Beah: Yeah, yeah. I think that distinction is super important. Like, first of all, there are a lot of things where rules just have exceptions and therefore they’re not really good rules, you know? like you have to be pretty sure that there has to be a really good reason to make something a rule because it’s going to fit different situations differently. And yeah, it’s like once it becomes a rule, people like lose power to do anything differently, so they don’t think about it, they don’t feel empowered, and like with a default, there’s still a burden of judgment because you can overturn the default and choose a different path. but the idea, I think, would you care would you say this is a fair characterization is like A, don’t spend brain power thinking about like reinventing the wheel or thinking about something that doesn’t really warrant brain power, like there’s a good default, and B you know just to n to nudge people in the direction of the best average answer, you know.

Marc: Yeah, right. It’s leveraging decisions that we’ve deliberated over in the past and you know have come to a good decision on. and exactly what you said. We don’t want folks reinventing the wheel. That said, we do want I mean, we hire really smart people at DuckDuckGo and we wanna, you know, leverage and celebrate that. So yeah, everybody’s encouraged to challenge the norm, so to speak, or you know, pre existing defaults. And in fact that’s a big big part of this. whole concept is coming back and challenging these at times. But yeah, th they’re, you know, i in a lot of cases it’s it would just be wasted effort to be rethinking this from scratch. And as much as there are some that are that are rule or that you know might make better rules because they are more binary, there are others that really don’t make great defaults either. we had a decent example where it was like, yeah, there is something that was true, say fifty percent of the time. I don’t remember what the actual number was, but it just didn’t make a great default. I honestly don’t remember what it is, but yeah, d just to say that not everything c falls into that framework framework.

Beah: Yeah, that that’s a good segue. Yeah. T t give me some examples of things that we do think are good operational defaults or that we treat as such.

Marc: Sure, yeah. I mean we have a bunch I you know, one of the things that’s so powerful about this is that I think we all do this all the time. so for example at DuckDuckGo we try to prioritize quick wins, which I guess the other thing I should say about defaults is that they are somewhat self evident, like to the point where that’s why I said they’re kind of deceptively simple because there are things that you don’t necessarily feel like you should have to put a lot of time into, a a lot of thought into. But quick wins are one where, you know, we at the company we’ve been, you know, trying to push people toward that. Right. Like the idea of a quick win is something that is relatively low effort but has high impact. And I mean you would saying it out loud, you would say, of course that’s something we should prioritize. But I think it’s also important to acknowledge that people don’t think that way. instinctually in a lot of cases. They wanna sometimes they want to work on, you know, the hardest thing or what they see as the most impact when, yeah, maybe less would do. so that’s one. one of my favorite ones, and I think a lot of people at this company would agree is we do no meetings on Wednesdays. so that’s a very simple one that, yeah, you we we don’t I I I w wouldn’t call it a rule, whereas we don’t you know, you can have a meeting on a Wednesday and sometimes you have to, especially as if it’s with somebody external. But generally you shouldn’t be scheduling meetings on Wednesdays.

Beah: See, that’s interesting because I would have called that one a rule. We I specifically I would have called the rule no

Marc: I mean

Beah: If you want to hop on real time and talk to somebody internally about something as it comes up, fine. If you have like an external call, we’ve also accepted that. But I I try to enforce that one as a rule because I think that one, if it’s a default, people just don’t observe it. They break it way too much. That’s my opinion.

Marc: Well, I yeah, but I think you just gave a few exceptions that are good, right? Like I I’m not gonna necessarily schedule something ahead of time, but I will yeah, reach out to somebody and say, Hey, it would be quicker if we talked about this ad hoc. So maybe I mean, yeah, maybe it’s splitting hairs, but

Beah: Yeah, it’s interesting though, because I think of those as part of as like the rule is constrained. So I guess making a rule is is safer the more you constrain it. And you could you could arguably ha tackle one operational problem and either make a highly constrained rule that has exceptions and carve outs or a broader default and maybe sometimes both at the same time. So yeah, okay. All right, that’s a good one. give me you got another example.

Marc: Yeah, fair enough. Sure. Sure. Yeah, I I can give you another one. so this one we came up with recently and that’s working incrementally, which again, you would say and when I say work incrementally, I basically mean you know we’re we’re trying to ship smaller features more frequently or even larger features and smaller increments. you’re going back and you’re you know, making sure that you’re on the right track, etc. Which again sounds like obviously that’s something that you would want to do. But anyone who’s ever collaborated with anyone on anything, I think would would realize that, yeah, that’s I don’t think that’s a human default, right? Like I think we all, you know, want to have a task, go build whatever we’re supposed to build, and then show the awesome thing that we did. And yeah, that’s I think that’s a good instinct to have. But there are many cases where if you misunderstood something initially or new information pops up, yeah, it’s super important to make sure that you’re raising that up and collaborating with others on

Beah: Mm-hmm. So you s you said that we made that one recently. Like what does it mean to make an operational default? How do we codify or communicate these?

Marc: Yeah, that’s a great question. And I don’t think we’ve actually figured that out yet. We have a lot of defaults across the company. This one was was communicated to to several objectives. but yeah, it’s not I would say it’s written down, but it’s not there’s no like internal glossary or internal directory of of defaults or anything like that. It’s basically just a reminder for folks. And again, it seems like it’s something that you should obviously do, but I think it’s important to remind people of these things.

Beah: Yeah, so like we basically we write things down in the form of like messages and or or really I mean anyway, but and I think that can happen maybe one thing that’s worth saying is that that that can happen at different levels of the organization. Like sometimes you have a group of five people working on an objective and they might adopt their own defaults or ways of operating. And then on the other hand, once something once we like believe something to be

Marc: Sure.

Beah: Sorry, I don’t know if listeners can hear that, but my dog is drinking some water right now. So we’ll see if that comes through. you know, once we’re really confident in something and that it’s like broadly applicable across the organization, then we might communicate it at like a higher level and codify it a little bit more more strongly as a default.

Marc: Yeah, and that’s kind of where that’s kind of where we are with this right now. We’ve communicated it to a couple of objectives. You know, objectives end up having their own culture, intentionally or otherwise, and this is yeah, an attempt to kind of test it on a smaller care scale before we start communicating it more broadly. and I I have one other I can give. This one’s much more engineering centric, but so we’re all working on AI quite a bit these days.

Beah: Yeah. Okay. Yeah.

Marc: And I think anybody that’s ever worked with, well, anybody that’s ever used AI will tell you that or has observed that you don’t get the same answer twice, right? It’s it’s you know so-called non-deterministic output. so in order for us to build products leveraging AI, it’s really important that we have evaluations up front, right? That you start with, you know, say a list of if I have question why. or sorry, question X, I’m gonna get answer Y, or at least something in the general with the general gist of of answer y. and yeah, because you can’t always super accurately predict what’s gonna come out, it’s really important that before we start building something, you know, you you have this eval or evaluation data set so that you know when you’re doing a good job or or when you’re when you’re not. I think that one honestly that one’s fairly close to a rule in my book, in terms of the engineering expectations we have, but there are definitely cases where it, you know, it doesn’t make sense or it’s too much effort to start there.

Beah: Yeah. And so like because that’s a default, even if people most people kind of think it’s a best practice, codifying it as a default means that if somebody decides not to do it, they might get the question, why? Are you sure? And they have to like defend basically if you don’t use a default, you have to have a good answer.

Marc: Yeah, that’s exactly right. Yes, you have to have yeah, there’s there’s we encourage people to make exceptions, but it’s gotta be well reasoned.

Beah: Up there. Yeah. Yeah. Yeah, yeah. Cool. Can you think of any defaults that we’ve adopted as an organization or on a team and then like decided were the wrong default and changed them?

Marc: Yeah, I thought about this one a bit. I I can’t think of one where we actually decided this was the wrong default. But I can there there are several that we have you know, potentially evolved over time, and that continue to evolve. The one and and yeah, this one is also super engineering specific. But the one that that comes to mind is, you know, initially all of our back end engineers were the default for all of them was to use Perl for our back-end services, the Perl programming language. But we made exceptions there, you know, because no language is the right tool for every job. So, you know, if we were developing something that Perl just wasn’t the right tool for, we gave people the space to make a case to not use it. That and that was true for a number of years. I think now I wouldn’t call that a default anymore. I don’t think that that that is the starting place for most folks, like when we’re developing a new a new service. so maybe that’s one that as you would either say is evolved or or maybe retired is a better way to put it. But yeah, I I can’t think of one where I think we were actually outright wrong, but definitely new information and things come in and we change them.

Beah: Mm. Yeah. One that kinda one that comes to mind to me that maybe went from having a not really having defined a default intentionally, but let kind of the flow of the stream define itself was we we, you know, as some listeners may know, our company is organized into a a bunch of objectives, like teams working towards some important goal for the company. And those objectives are expected to hold what we call an assessment every so often to kind of like redefine the strategy in the roadmap. And we I think f for the first maybe f few years I was here, we didn’t have an explicit expectation on how often that happened, but I think the sort of adopted default was a pretty long period, like maybe on average, I don’t know, five months, six months. something like that and then at some point we what’s that?

Marc: Yeah, when you felt like it was important. I said when you felt like it was important, it was very I think very subjective.

Beah: Yeah. Well and it was like easy to procrastinate if you were leading an objective, you know, like ‘cause it felt like a time like a thing you’d have to put time into and put yourself out there. And so it just kind of was like getting long. But in reality we observed that when people ran assessments more often, we had better roadmaps and better decision making and, you know, sort of less leaving the track without, you know, the help to get back on the track and

Marc: Yeah.

Beah: And so at some point we I think maybe said like let’s default to every eight weeks and then we maybe t brought that back to six. Right now it’s it’s six weeks is the default. and I think that’s been a nice improvement to make that explicit.

Marc: It’s actually a it’s a good example of working incrementally over a you know, different horiz time horizon. But yeah, for sure.

Beah: Yeah. Cool. Anything else you want to add, Marc, before we wrap this?

Marc: I don’t think so. I think again, I feel like I have to keep saying that a lot of these are self evident because it just sounds so obvious in a lot of cases, but it is yeah, I think I think it leads to better results and and hopefully we’re gonna see that very soon in these objectives that where we’ve introduced it and yeah, r really important to challenge them as well.

Beah: Yeah, there’s probably a lot of obvious things that people don’t do on the regular because of some cognitive, you know, glitch or habit or, you know, your predictably irrational behavior. So yeah, just making it explicit I think can be powerful. Cool. All right. Well, thank you, Marc, and see you around.

Marc: Yeah, yeah. Yeah, thanks.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit insideduckduckgo.substack.com

In this episode, Cristina (CMO) and Zack (Senior Director, Marketing) discuss our new Duck.ai ad, how we develop, test, and measure creative and campaigns, and why we do a lot of the work in-house.

Disclaimers: (1) The audio, video (above), and transcript (below) are unedited and may contain minor inaccuracies or transcription errors. (2) This website is operated by Substack. This is their privacy policy.

If you have feedback on Duck Tales, or episode ideas, email us at [email protected].

Cristina: Hi, and welcome to Duck Tales, where we go behind the scenes at DuckDuckGo and discuss the stories, technology, and people that help build privacy tools for everyone. In each episode, you’ll hear from employees about our vision, product updates, engineering, or approach to AI. Hi, I’m Cristina. I’m on the marketing team, and I’m here today with Zack. Do you want to introduce yourself?

Zack: Yeah, my name is Zack Einhorn. I’m on the marketing team as well at DuckDuckGo and been here for nine years now. and I live in LA. Happy to be here.

Cristina: Happy to have you, thanks for joining. So today we’re gonna talk about our new ad campaign. On June 29th, we launched Prying Eyes, the name of our first national brand campaign for Duck.ai. And for those of you who don’t know what Duck.ai is, it is a way to chat privately with ChatGPT and other AIs, all in one place for free, no login. So today we’re gonna go behind the scenes, how it came together. Our unique development and production process, our views on AI, and more. Before we jump in, I just want to say to everyone listening: love it, hate it, have ideas for a new episode, please write us at [email protected]. That’s [email protected]. We will read all of them and would appreciate any feedback. So, first question: Zack, why why a brand campaign for Duck.ai and why now?

Zack: Yeah, well, you know, for years we had been focused on you know performance marketing for as an alternative to Google and Chrome and focusing on search and browse. And we’re introducing a new product market that has virtually zero awareness. And we’re really confident about the product market fit. We know that there’s a lot of built-up anxiety around AI and a desire for private alternatives. And so we know it’s time to let people know that one exists and from a company that you trust and have been trusting.

Cristina: So I find that when I chat with people about what I do, they think the the making the ad portion of my work is the most interesting, right? And they’ll ask, like what is it like to be on set and all these kind of things? So I think a lot of people find that industry but interesting. But even for industry veterans, it’s maybe interesting to hear about how we at DuckDuckGo make an ad, because it’s a different bit different than how most companies do it. Can you dive into that?

Zack: Yeah, absolutely. so you know, most companies, and in my in my experience, the prior companies, you know, they kind of hand everything over to one agency. You know, everything is sort of with one big house strategy, research, testing. and the way that we do it is a little bit different. Like you said, we’re we tend to we like to be really hands on. and I think, you know, one thing that makes us pretty different too is that, you know, we’re we you know we’re some of like the foremost researchers on this space at the company. And so we you know, whereas it might be really useful if you’re in like a you know a a a category, like you know, consumer package goods, to have like the expertise of an agency that may have many, many clients in that space and can and can, you know, give you strategy and insight into areas that you may not have. but because we really you know, we have a lot of that research and talent in house, the strategy really you know comes from us. and you know, we tend to lean on agencies more for like additional initial ideation. and then from there we really take things into our own hands. We have we have freelancers that we’ve gone back to a handful of times to help us develop ideas. We’ve got internal talent that helps us develop ideas. and then we have production companies that we work directly with. And that I think is a real for us. You know, we we save a lot of, I think, you know, we save a lot of time in the game of telephone that you usually play with your with your agency by being able to, you know, handpick the director, be you know deeply involved in the post-production, like attending edit sessions, coloring. and I think that that gives us a advantage because you know not only do we get to accelerate by making those being hands-on in those choices, but we also continuously build those skill sets. And so you know we’ve been doing it now, we’ve been doing it this way and kind of building on that process for five years now. And we’ve just we’ve built up a a a ton of expertise internally and done a lot of hot like you know, a lot of good hiring to fill the the areas of expertise that we didn’t have, expanded our network of partners. yeah, we’ve really kind of built up this this in-house team.

Cristina: Yeah, thanks for walking us through that. So internally, the team has been calling this ad Prying Eyes. Like, what does that mean? Where did the idea come from?

Zack: Yeah, absolutely. So it came from one of our one of our agency partners. You know, we we’d we went through a process where you know we realized one of the things that we weren’t super strong at internally was coming up with the initial idea. and so we ran through a process where we worked with a series of agencies and freelancers to to come up with ideas. One of them came up with this idea that was really sticking with us where we had you know giant photorealistic eyeballs following you know c following people and and you know in you know invading their space when they’re doing asking AI sensitive questions. And we had a few directions we were looking at, and we had done a bunch of research. And the thing that we kept coming back to was this thing we called exposure anxiety. And the crux of it was that we realized that when we got a level down into the specific things that people were worried about being spied on. you know, giving away when they’re when they’re using AI, that it became like a bit polarizing. Like, you know, they once you used a specific example, people are like, well that’s not quite really me. But we found the thing that really stuck for everyone, and that was the most visceral thing, which I think is the most important because I think that’s what sticks with you it emotionally in an ad and will make you think of it later and motivate you, was just the actual sensation of of being watched. That was the through line that whether you were worried about, you know, asking AI about something related to, you know, health, or whether you were asking about an issue going on at work, or you know, whether you’re asking AI you know for symptoms of, you know, something that your that your kids dealing with, that that no matter what it was, everyone shared this this this feeling of exposure anxiety and you know what what can make you you know what is more exposing than feeling like there are giant eyes behind you watching everything that you that you type.

Cristina: Yeah, this was this was a fun concept. you know, we considered a lot of different ideas, and for me, just seeing and thinking about the eyeballs was just so arresting, right? In a really clear visual way to encompass the feelings that we had heard in research, right? We’re a very data-driven company, and so research is really important to us. We’re not just thinking of something that looks cool, it needs to be grounded in an insight that we think relates back to the problem we’re trying to solve.

Zack: Yeah, absolutely. And I think it also just there was a lot of connectivity to to like our overall message, too, like, you know, in in terms of of privacy and and and prior campaigns. I think that we, when we went in, I think we really thought we want something that is uniquely AI. but I think what we found is that a lot of the anxiety around AI is kind of an evolution of the of the anxiety that already existed. So just like the way, just the way that AI is the newest way that people are are using the internet and and how it’s you know it’s more personalized and you know tailored than than than search and browse was it’s also that much more viscerally you know uncomfortable you know you’re you’re a lot more you know, likely to have sort of these uncanny experiences where, you know, AI will bring up a prior conversation or mention a tidbit that you thought, you know, wasn’t being logged, you know. and and so I think it really helped draw that draw that through line.

Cristina: Yeah. Yeah, I I personally feel very passionate that if someone is going to use AI, they should use Duck.ai because it’s hard to anticipate just how this data will be used from other companies in the future. But I think we can all anticipate, probably not in a great way, right? In terms of privacy. On the last episode I was on, we talked about how not everybody wants AI, right? And that’s why we feel so strongly about making it optional. So how do you think about making an ad for AI?

Zack: Yeah, I mean I think that’s a very real subset of people. I look at, you know, the way that I look at DuckDuckGo’s mission is, you know, I think we, you know, I look at the goal of DuckDuckGo to provide for people the the best quality possible version of the internet that you are used to using, but private. And so I think, you know, that is the that is the the the mass of people that we saw that that we that we serve. AI adoption is, you know, well over fifty percent and growing. And so I think you know it’s really important that we serve that audience. at the same time, you know, we firmly, while while we believe, you know, in providing the pro in private AI, we also, you know, we also take exception to the way that AI is being sort of jammed down people’s throat by other by other companies. and you know, we take pride and we think it, I think I we think it’s you know goes along with our overall mission of build of raising the standard of trust online. to give people the ability to control AI and use it on their own on their own terms. And so even in our in even in our our advertising of our AI products, we’re very clear you know in in our messaging, like for example, the television ad in our end card and VO to say that you know Duck.ai is is from DuckDuckGo and that our our policy here is that AI is private and optional. And so we you know we want to be clear that that is all you know that is our brand’s stance.

Cristina: Yeah, absolutely. So we’re spending all this money on TV and digital and all these channels. How do we know if it works? Like what what are we trying to achieve here and what are those signals?

Zack: Yeah, I mean it is, you know, raising brand awareness is meaningfully harder to read than than you know performance performance marketing. and it takes time. you know, we we I think the bottom line is we won’t know, we won’t know right away. the what we’ve seen from the initial tests is you know, we we’re able to get a read on awareness on a two to three month kind of timeline to see if we’re moving that. usage usually lags behind that. and so I think, you know, a lot of times when you’re when you’re looking at awareness, I think what you’re looking at is is, you know, you have you have immediate you know you have immediate key performance indicators which for us are all are all centered around awareness you know we’re doing pulse checks for every four weeks we’re we’re revisiting a survey and we’re seeing if we’ve if we’ve raised it. and then in terms of driving actual growth, I think it’s really more of a higher level question of running that strategy for long enough to be able to create the the flywheel and seeing how it impacts growth holistically. And so I think that is something that’s probably on a longer you know, timeline closer to, you know, six months to a year where you can really see the the big impact of focusing on on brand awareness. And that’s that is similar to what we saw you know, several years ago when we were doing big brand awareness push for the core product.

Cristina: Yeah, absolutely. It’s a bit of a patience game, right? Wait waiting to see if these metrics go up and maybe leaning on some early indicators, but it’s not that instant gratification that you get from a direct response camp.

Zack: Yeah. Yeah. Yeah. We are looking at the direct response metrics on but but trying to set expectations that that you know we’re in a we’re in a teach people our name place.

Cristina: Yeah. So so you and the team have been working hard on this for months. Maybe you get to take a little bit of a break now while, you know, the big channels are out there and you’re waiting for some of the data to come in, but I’m sure that’s not all. So do you want to speak to what’s next?

Zack: Yeah, I mean I think you know, not really taking a break. trying to, you know, I think we had real we have really you know, ambitious goals for this campaign. I think, you know, we we’re we are ramping up in all senses of the word. You know, we’re starting out with we’re starting out heavily in a in a in a handful of channels, very, very video centric, you know, CTV, YouTube, and and social, but we’re really looking to build out the 360 of the campaign. We’re looking at a handful of other channels inclusive of out of home and audio and looking to expand overall coverage pr we’ve got we’ve got a bunch of a bunch of stuff deck to to help fill out this campaign and and bring it to life on a larger scale.

Cristina: Yeah, absolutely. I’m very excited that this is finally out there now and working to not only teach people the name, but hopefully get people using it. So again, if you haven’t yet, try Duck.ai. Again, it’s free, no login required. So if you want to do an AI search, but you want to do more privately, give it a go. Zack, it was really great chatting with you today and kind of reliving the work of the last few months. Thank you and the team for all of your hard work. and again, very exciting that the ads are out there now.

Zack: Mm-hmm. Thank you. Appreciate you having me. I’m very excited.

Cristina: Alrighty. Yeah. Okay, well that’s all for now and stay tuned for future episodes of Duck Tales. See you later.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit insideduckduckgo.substack.com