In the very old days, if you needed more storage for your database, you had to walk into the data center and install another server. Now you can just log into your cloud provider’s console and click a few buttons. Voilá, more storage. So easy! But what if you’re replicating that storage configuration for hundreds of databases at once? Suddenly, that’s a lot more clicking. Not so easy! (Plus, very tedious and very error prone.) So instead of living with this “ClickOps” approach, we developed a declarative infrastructure model — our very own “infrastructure as code” solution for managing cloud resources at Spotify scale. Instead of manually configuring each resource, developers just describe the state they want. And once we adopted declarative infra, we unlocked ways to improve not just how we manage resources, but also how we update policies, manage dependencies, and make other changes to code across our entire fleet of repos — quickly, safely, easily. In other words, programmatically.

Host Dave Zolotusky talks with David Flemström — who went from pushing the limits of Spotify’s infrastructure as a feature developer to working on the platform team in order to improve infrastructure for all of our developers. The two Daves discuss what declarative infrastructure means at Spotify, our journey to adopting it (going from Puppet to cloud consoles, to something better than both) and why we did it, how our model works (Kubernetes!), how it changed the relationship between our feature teams and our platform teams, how this shift helped enable Fleet Management at Spotify, and where we’re going next with abstracting infrastructure so that it helps our engineers do more, more easily.

Learn more about declarative infrastructure and Fleet Management:

• NerdOut, Episode 12: Fleet First

Read what else we’re nerding out about on the Spotify Engineering Blog: engineering.atspotify.com

You should follow us on Twitter @SpotifyEng and on LinkedIn!

Last episode, we talked about ABBA, our first A/B testing tool. We used it to test UI changes, new features, content recommendations — anything and everything we could think of. ABBA was so good and worked so well for so long…that we decided to get rid of it. Years of using ABBA taught us what makes for good experimentation, and we eventually realized we needed a better tool, built from scratch. Listen to find out why we pulled the plug on ABBA and how Spotify’s Experimentation Platform was born. And in case you missed it, a version of our internal platform will be available to the public as Confidence, a new enterprise product for developer teams — read today’s announcement: “Coming Soon: Confidence — An Experimentation Platform from Spotify”.

But first, let’s talk buttons. Everyone always has so many questions about buttons. How do you know which color they should be? Or how big they should be? Or whether the corners should be round or square? The easy answer: an A/B test! But if only all product experimentation was as simple as testing buttons. Senior staff engineer Mark Grey returns to talk with host Dave Zolotusky, along with senior engineer Dima Kunin — he helped build Spotify’s Experimentation Platform and was the guy who had the honor of finally retiring ABBA. They discuss the ins and outs of enabling experimentation at scale, including targeting criteria, controlling eligibility, the importance of measuring exposure, using properties instead of feature flags, the advantages of separating your app configuration from your experiments, fallback states, sample ratio mismatches — and all the other questions you have to answer about your experimentation process before you can even ask something as simple as “what color should a button be” — let alone “will this machine learning model consistently provide recommendations users appreciate over the next year”.

Plus, did you definitely, positively, absolutely eat the bread? Or did you just buy the bread? And a bonus trick question: What’s the difference between “treatments”, “variants”, and “groups” — and why is it always so hard to name things?

Learn more about ABBA and its successor, Spotify’s Experimentation Platform:

Plus, find out lots more about how we do experimentation at Spotify on our engineering blog — including a little light reading on automated salting and bucket reuse, choosing sequential testing frameworks, comparing quantiles at scale, and how we scale other scientific best practices across the org.

Read what else we’re nerding out about on the Spotify Engineering Blog: engineering.atspotify.com

You should follow us on Twitter @SpotifyEng and on LinkedIn!

Back in the day, Spotify built a custom A/B testing tool called ABBA. It was great. The platform enabled lots of teams to try out lots of ideas for new features to see what worked and what didn’t. With ABBA, we went from doing tens of experiments to hundreds of experiments. But we didn’t just learn what color button users liked better: the more tests we ran, the more we learned about our testing methods, including the limitations of ABBA itself — which eventually led us to a new, better way to test. Here’s the story of ABBA, our very first experimentation platform, and the lessons we learned about doing product experimentation at scale.

Host Dave Zolotusky talks with Mark Grey, a senior staff engineer and 10-year Spotify veteran. They discuss Spotify’s earliest efforts at product testing, our early infrastructure for data and data processing (using Hive and Hadoop), how migrating to the cloud unlocked more processing power (and more testing), the difference between using tests to design the color of a button and using tests to inform the very next user interaction via machine learning, feature flags and holdout groups, all the things we learned about conducting scientifically sound experiments, how we built a culture of experimentation among our software development teams, and what finally drove us to sunset ABBA and build its successor: a bigger, better internal experimentation platform. Plus, progress bars and lightsabers.

Read more about ABBA and how we do product experimentation at Spotify:

Read what else we’re nerding out about on the Spotify Engineering Blog: engineering.atspotify.com

You should follow us on Twitter @SpotifyEng and on LinkedIn!

We’re often focused on features that improve the experience for users — letting them do something better, faster, smarter. But if you don’t consider accessibility issues, many users won’t be able to use the feature at all. From color contrast and text size to alt text for enabling screen readers and voice commands, accessibility issues can affect everyone, whether you have a permanent disability or just an armful of groceries. So how do you get developers and designers to adopt an accessibility mindset and make it a fundamental part of the development process?

Host Dave Zolotusky talks with Dani Devesa Derksen-Staats, an iOS engineer on Spotify’s accessibility team. We’ll hear how Dani went from a five-year computer science program, where accessibility wasn’t mentioned even once, to becoming so passionate about the topic, he wrote a book on it. They also talk about the basic things we forget to consider when we don’t consider accessibility; how we can all benefit from accessibility improvements, whether that’s getting up a curb in a wheelchair or while pushing a stroller; and ways to address accessibility issues into the development process, from adopting a multimodal approach to UX design to integrating accessibility tests into your CI/CD.

Spotify recently introduced an Accessibility Center. Have questions or concerns about accessibility? Contact us:

More on accessibility from Dani:

Read what else we’re nerding out about on the Spotify Engineering Blog: engineering.atspotify.com

You should follow us on Twitter @SpotifyEng and on LinkedIn!

How do you make an AI-generated voice feel more like a real person? You give it a real personality. Spotify recently released a new feature in the US and Canada called DJ. Turn it on, and you hear a curated selection of music and recommendations, along with a fun, friendly, knowledgeable voice telling you more about what you’re listening to (you know, like a real DJ) — except everything you hear is personalized just for you. Learn what makes this DJ feel so realistic and meet the people behind the technology — including the DJ himself.

Host Dave Zolotusky talks with product director Zeena Qureshi and director of engineering John Flynn — together they lead Speak, the team at Spotify responsible for DJ’s realistic, expressive voice. Hear how the AI voice technology they pioneered for Hollywood blockbusters and triple-A video games now brings Spotify’s personalized DJ to life, how they record a range of emotions to create deeper datasets for modeling DJ’s unique personality, the technical challenges of generating and delivering these dynamic performances at the press of a button, and how having your own personal audio guide brings a totally new dimension to the Spotify listening experience. You’ll also hear from the person who provides the raw ingredients for the AI DJ’s voice and soul: the real-life Xavier “X” Jernigan.

Learn more about Spotify’s personalized DJ and the technology behind it:

Read what else we’re nerding out about on the Spotify Engineering Blog: engineering.atspotify.com

You should follow us on Twitter @SpotifyEng and on LinkedIn!

What happens when the standard tooling for iOS and Android just doesn't cut it anymore? What’s it like to maintain an app when there are literally thousands of commits every week? How do you develop your feature without worrying about everybody else’s feature, when at the end of the day, you’re shipping a single, massive app? In other words, what’s it really like to build apps at Spotify’s scale?

Host Dave Zolotusky talks with Patrick Balestra, a staff engineer on our client platform team, whose job is to make sure every Spotify developer has a great experience building, no matter what Spotify app they’re building for. They talk about adopting Bazel, reducing local build times with remote caching, managing multiple monorepos and zillions of dependencies, doing big tech migrations (Objective-C to Swift, Java to Kotlin) without slowing down, open sourcing our tools, and contributing to the Mobile Native Foundation.

Learn more about how we build mobile apps at Spotify:

Read what else we’re nerding out about on the Spotify Engineering Blog: engineering.atspotify.com

You should follow us on Twitter @SpotifyEng and on LinkedIn!

Whether you’re working at a major financial institution or an audio streaming platform, it turns out engineering teams face a lot of the same challenges to doing their jobs effectively. In this episode, you’ll hear Spotify’s head of technology and platforms, Tyson Singer, discuss developer experience with engineering leaders from Expedia Group, Bank of the West, consulting firm Booz Allen Hamilton, and digital telecom TELUS — all adopters of Backstage, our open platform for building developer portals. This panel discussion took place on December 15, 2022 at the launch event for Spotify Plugins for Backstage, our new developer software bundle.

At the event, Tyson is joined on stage by Guillermo Manzo (Expedia Group), Boyan Vassilev (Bank of the West), Jason Miller (Booz Allen Hamilton), and Nate Axcell (TELUS) to talk about the costs of context switching, breaking down organizational silos, how to get your developers started and how to get them unstuck, open source, inner source, golden paths, and the importance of making sure the best way to do something is the easiest way to do something otherwise it just doesn’t get done.

Learn more about Spotify Plugins for Backstage:

Listen to more episodes about Backstage:

Or check out the links below:

Read what else we’re nerding out about on the Spotify Engineering Blog: engineering.atspotify.com

You should follow us on Twitter @SpotifyEng and on LinkedIn!

There are thousands of developers building great Spotify experiences — who don’t actually work at Spotify. From smart speakers to game consoles to car infotainment systems, how do we support this diverse, creative community building on top of the Spotify platform? Along with providing APIs, SDKs, and other developer tools, it starts with something even more fundamental: listening to and understanding their needs. That’s where our developer advocates come in.

Host Dave Zolotusky talks with Serah Kiburu, a Spotify engineer and senior developer advocate. They discuss how and why Spotify supports third-party developers, the value of being “customer zero” for our own developer tools, our most popular APIs, the importance of listening before you start building, Serah’s journey to Spotify in Stockholm, and how she’s preparing for the robot apocalypse.

Learn more about the resources we provide third-party developers:

Read what else we’re nerding out about on the Spotify Engineering Blog: engineering.atspotify.com

You should follow us on Twitter @SpotifyEng and on LinkedIn!

From Norway to Nebraska, there are individual developers keeping the open source projects that we all depend on going. Last year, Spotify started an annual fund to help support maintainers of free and open source software (FOSS) — the Spotify FOSS Fund. We earmarked 100,000 Euros and then gave it away to nine projects, no strings attached. One of those recipients was Byte Buddy, a project with millions of users, but just one volunteer maintainer — its creator, Rafael Winterhalter. Hear what it takes to sustain something that started as a passion project in 2014 and then grew to become much more.

Rafael talks with our host Dave Zolotusky and Per Ploug, Spotify’s open source tech lead, who helped set up our FOSS fund. They discuss how Spotify decided which projects to fund, why giving away (and accepting) free money is harder than you might think, and what it’s really like maintaining a popular open source project all on your own.

Learn more about the recipients of the the Spotify FOSS Fund and other ways we’re working to create a more sustainable open source ecosystem:

Read what else we’re nerding out about on the Spotify Engineering Blog: engineering.atspotify.com

You should follow us on Twitter @SpotifyEng and on LinkedIn!

Recorded live at midnight EST, December 6, 2022, from Boston, Gothenburg, and Stockholm, via the Spotify Live app. In this special Advent of Code episode, we try to solve the Day 6 puzzle the moment it’s released. Host Dave Zolotusky and Spotify’s chief architect, Niklas Gustavsson (aka, ngn), play against Spotify’s reigning champ, the one and only, unstoppable Jimmy Mårdell. Entering today’s challenge behind ngn in the standings, will Jimmy reclaim his rightful place atop the Spotify leaderboard? Play along with us and find out!

Plus, Spotify engineers share what makes this friendly, low-stakes coding competition so much fun every year, Jimmy answers questions from the live audience (including offering some speed tips), and more.

The Advent of Code season is upon us once again:

Special thanks to Eric Wastl for creating such a fun thing.

Read what else we’re nerding out about on the Spotify Engineering Blog: engineering.atspotify.com

You should follow us on Twitter @SpotifyEng and on LinkedIn!