Software at Scale

Akshay Buddiga is the co-founder and CTO of Traba, a labor management platform.

Apple Podcasts | Spotify | Google Podcasts

Sorry for the long hiatus in episodes!

Today’s episode covers a myriad of interesting topics - from being the star of one of the internet’s first viral videos, to experiencing the hyper-growth at the somewhat controversial Zenefits, scaling out the technology platform at Fanatics, starting a company, picking an accelerator, only permitting in-person work, facilitating career growth of gig workers, and more!

Highlights

[0:00] - The infamous Spelling Bee incident.

[06:30] - Why pivot to Computer Science after an undergraduate focus in biomedical engineering?

[09:30] - Going to Stanford for Management Science and getting an education in Computer Science.

[13:00] - Zenefits during hyper-growth. Learning from Parker Conrad.

[18:30] - Building an e-commerce platform with reasonably high scale (powering all NFL gear) as a first software engineering gig. Dealing with lots of constraints from the beginning - like multi-currency support - and delivering a complete solution over several years.

The interesting seasonality - like Game 7 of the NBA finals - and the implications on the software engineers maintaining e-commerce systems. Watching all the super-bowls with coworkers.

[26:00] - A large outage, obviously due to DNS routing.

[31:00] - Why start a company?

[37:30] - Why join OnDeck?

[41:00] - Contrary to the current trend, Traba only allows in-person work. Why is that?

We go on to talk about the implications of remote work and other decisions in an early startup’s product velocity.

[57:00] - On being competitive.

[58:30] - Velocity is really about not working on the incorrect stuff.

[68:00] - What’s next for Traba? What’s the vision?

[72:30] - Building two-sided marketplaces, and the career path for gig workers.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

James Cowling is the co-founder of Convex, a state management platform for web developers.

Apple Podcasts | Spotify | Google Podcasts

We discuss the state of web development in the industry today, and the various different approaches to make it easier.

Contrasting the Hasura and Convex approach as a good way to illustrate some of the ideas. Hasura lets you skip the web-app, and run queries against the database through GraphQL queries. Convex, on the other hand, helps you stop worrying about databases. No setup or scaling concerns. It’s interesting to see how various systems are evolving to help developers with reducing the busywork around more and more layers of the stack, and just focus on delivering business value instead.

Convex also excels at the developer experience portion - they provide a deep integration with React, use hooks (just like Apollo GraphQL) and seem to have a fully typed (and therefore auto-completable) SDK. I expect more companies will move “up the stack” to provide deeper integrations with popular tools like React.

Episode Reading List

* The co-founders of this company led Dropbox’s Magic Pocket project.

* Convex → Netlify

* Convex vs. Firebase

* Prisma

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Josh Twist is the co-founder and CEO of Zuplo, a programmable, developer friendly API Gateway Management Platform.

Apple Podcasts | Spotify | Google Podcasts

We discuss a new category of developer tools startups - API Gateway Management Platforms. We go over what an API Gateway is, why do companies use gateways, common pain-points in gateway management, building reliable systems that serve billions of requests at scale. But most importantly, we dive into the story of Josh’s UK Developer of the Year 2009 award.

Recently, I’ve been working on the Vanta API and was surprised at how poor the performance and developer experience around Amazon’s API Gateway is. It has poor support for rate limiting, and has very high edge latency. So I’m excited for a new crop of companies to provide good solutions in this space.

Episode Reading List

* Amazon’s API Gateway

* Stripe’s API - The first ten years

* Envoy

The Award

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Ted Young is the Director of Developer Education at Lightstep and a co-founder of the OpenTelemetry project.

Apple Podcasts | Spotify | Google Podcasts

This episode dives deep into the history of OpenTelemetry, why we need a new telemetry standard, all the work that goes into building generic telemetry processing infrastructure, and the vision for unified logging, metrics and traces.

Episode Reading List

Instead of highlights, I’ve attached links to some of our discussion points.

* HTTP Trace Context - new headers to support a standard way to preserve state across HTTP requests.

* OpenTelemetry Data Collection

* Zipkin

* OpenCensus and OpenTracing - the precursor projects to OpenTelemetry

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Or Weis is the CEO and founder of Permit.io, a Permission as a Service platform. Previously, he founded Rookout, a cloud-debugging tool.

Apple Podcasts | Spotify | Google Podcasts

Many of us have struggled (or are struggling) with permission management in the various applications we’ve built. The complexity of these systems always tends to increase through business requirements - for example, some content should only be accessed by paid users or users in a certain geography. Certain architectures like filesystems have hierarchical permissions that efficient evaluation, and there’s technical complexity that’s often unique to the specific application.

We talk about all the complexity around permission management, and techniques to solve it in this episode. We also explore how Permit tries to solve this as a product and abstract this problem out for everyone.

Highlights

[0:00] - Why work on access control?

[02:00] - Sources of complexity in permission management

[08:00] - Which cloud system manages permissions well?

[11:00] - Product-izing a solution to this problem

[17:00] - What kind of companies approach you for solutions to this problem?

[22:00] - Why are there research papers written about permission management?

[38:00] - Permission management across the technology stack (inter-service communication)

[42:00] - What are you excited about building next?

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Jon Skeet is a Staff Developer Platform Engineer at Google, working on Google Cloud Platform client libraries for .NET. He's best known for contributions to Stack Overflow as well as his book, C# in Depth. Additionally he is the primary maintainer of the Noda Time date/time library for .NET. You may also be interested in Jon Skeet Facts.

Apple Podcasts | Spotify | Google Podcasts

We discuss the intricacies of timezones, how to attempt to store time correctly, how storing UTC is not a silver bullet, asynchronous help on the internet, the implications of new tools like GitHub Copilot, remote work, Jon’s upcoming book on software diagnostics, and more.

Highlights

[01:00] - What exactly is a Developer Platform Engineer?

[05:00] - Why is date and time management so tricky?

[13:00] - How should I store my timestamps? We discuss reservation systems, leap seconds, timezone changes, and more.

[21:00] - StackOverflow, software development, and more.

[27:00] - Software diagnostics

[32:00] - The evolution of StackOverflow

[34:00] - Remote work for software developers

[41:00] - Github Copilot and the future of software development tools

[44:00] - What’s your most controversial programming opinion?

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Lee Byron is the co-creator of GraphQL, a senior engineering manager at Robinhood, and the executive director of the GraphQL foundation.

Apple Podcasts | Spotify | Google Podcasts

We discuss the GraphQL origin story, early technical decisions at Facebook, the experience of deploying GraphQL today, and the future of the project.

Highlights

(some tidbits)

[01:00] - The origin story of GraphQL.

Initially, the Facebook application was an HTML web-view wrapper. It seemed like the right choice at the time, with the iPhone releasing without an app-store, Steve Jobs calling it an “internet device”, and Android phones coming out soon after, with Chrome, a brand-new browser.

But the application had horrendous performance, high crash rates, used up a lot of RAM on devices and animations would lock the phone up. Zuckerberg called the bet Facebook’s biggest mistake.

The idea was to rebuild the app from scratch using native technologies. A team built up a prototype for the news feed, but they quickly realized that there weren’t any clean APIs to retrieve data in a palatable format for phones - the relevant APIs all returned HTML. But Facebook had a nice ORM-like library in PHP to access data quickly, and there was a parallel effort to speed up the application by using this library. There was another project to declaratively declare data requirements for this ORM for increased performance and a better developer experience.

Another factor was that mobile data networks were pretty slow, and having a chatty REST API for the newsfeed would lead to extremely slow round-trip times and tens of seconds to load the newsfeed. So GraphQL started off as a little library that could make declarative calls to the PHP ORM library from external sources and was originally called SuperGraph. Finally, the last piece was to make this language strongly typed, from the lessons of other RPC frameworks like gRPC and Thrift.

[16:00] So there weren’t any data-loaders or any such pieces at the time.

GraphQL has generally been agnostic to how the data actually gets loaded, and there are plugins to manage things like quick data loading, authorization, etc. Also, Facebook didn’t need data-loading, since its internal ORM managed de-duplication, so it didn’t need to be built until there was sufficient external feedback.

[28:00] - GraphQL for public APIs - what to keep in mind. Query costing, and other differences from REST.

[42:00] - GraphQL as an open-source project

[58:00] - The evolution of the language, new features that Lee is most excited about, like Client-side nullability.

Client-side nullability is an interesting proposal - where clients can explicitly state how important retrieving a certain field is, and on the flip side, allow partial failures for fields that aren’t critical.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Harshyt Goel is a founding engineer and engineering manager of Platform and Integrations at Loom, a video-messaging tool for workplaces. He’s also an angel investor, so if you’re looking for startup advice, investments, hiring advice, or a software engineering job, please reach out to him on Twitter.

Apple Podcasts | Spotify | Google Podcasts

We discuss Loom’s story, from when it had six people and a completely different product, to the unicorn it is today. We focus on driving growth, complicated product launches, and successfully launching the Loom SDK.

Highlights

[00:30] - How it all began

[03:00] - Who is a founding engineer? Coming from Facebook to a 5 person startup

[06:00] - Company inflection points.

[10:30] - Pricing & packaging iterations.

[14:30] - Running growth for a freemium product, and the evolution of growth efforts at Loom

[30:00] - Summing up the opportunities unlocked by a growth team

[33:00] - Sometimes, reducing user friction isn’t what you want.

[34:30] - The Loom SDK, from idea to launch.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Daniel Stenberg is the founder and lead developer of curl and libcurl.

Apple Podcasts | Spotify | Google Podcasts

This episode, along with others like this one, reminds me of this XKCD:

We dive into all the complexity of transferring data across the internet.

Highlights

[00:30] - The complexity behind HTTP. What goes on behind the scenes when I make a web request?

[11:30] - The organizational work behind internet-wide RFCs, like HTTP/3.

[20:00] - Rust in curl. The developer experience, and the overall experience of integrating Hyper.

[30:00] - Web socket support in curl

[34:00] - Fostering an open-source community.

[38:00] - People around the world think Daniel has hacked their system, because of the curl license often included in malicious tools.

[41:00] - Does curl have a next big thing?

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Sahil Lavingia is the founder of Gumroad, an e-commerce platform that helps you sell digital services. He also runs SHL Capital, a rolling fund for early-stage startups.

Apple Podcasts | Spotify | Google Podcasts

Sahil’s recent book, Minimal Entrepreneurship, explores a framework for building profitable, sustainable companies. I’ve often explored the trade-off between software engineering and trying to build and launch my own company, so this conversation takes up that theme and explores what it means to be a minimal entrepreneur for a software engineer.

Highlights

(edited)

Utsav: Let’s talk about VCs (referencing your popular blog post “Reflecting on My Failure to Build a Billion-Dollar Company”). Are startups pushed to grow faster and faster due to VC dynamics, or is there something else going on behind the scenes?

It’s a combination of things. People who get caught up in this anti-VC mentality are missing larger forces at play because I don't really think it's just VCs who are making all of these things happen. Firstly, there’s definitely a status game being played. When I first moved to the Bay Area, as soon as you mention you’re working on your own, the first question people ask you is how far along your company is, who you raised money with, how many employees you have, and comparing you with other people they know. You can’t really get too upset at that, since that’s the nature of the people coming to a boomtown like San Francisco.

The way I think about it, there’s a high failure rate in being able to build a billion-dollar company, so you want to find out reasonably quickly whether you will succeed or not. Secondly, we’re in a very unique industry, where equity is basically the primary source of compensation. 90% of Americans don’t have some sort of equity component in the businesses they work for, but giving equity has a ton of benefits. It’s great to have that alignment, and folks who take an early risk for your company should get rewarded.  The downside of equity is that it creates this very strong desire and incentive to make your company as valuable as possible, as quickly as possible. In order to get your equity to be considered valuable to investors, you need to grow quickly, so that investors use these models that project your growth rate to form your valuation.

Many people took my blog to say - it’s the VC’s fault, but that’s not true. The VCs let me do what I wanted, they don’t really have that much power. The issue was that in order for employees to see a large outcome, you need the company to have a large exit. As a founder, you’d do pretty well if the company sold for $50 million dollars, but that’s not true for employees, they really need this thing to work, otherwise, the best ones can just go work for the next Stripe. So you have this winner-take-all behavior for employees, and it’s ultimately why I ended up shrinking the company to just me for a while.

Utsav: So do you give employees equity in the minimalist entrepreneurship framework?

Firstly: avoid hiring anyone else for as long as possible, until you know you have some kind of product-market fit. I think It depends on your liquidity strategy. How are you as a founder about to make money from this business? The way you incentivize your employees should align with that. If you want to sell your company for a hundred million dollars, consider sharing that and giving equity. If you plan to create a cash cow business, consider profit sharing.

Utsav: What, if any, is the difference between indie-hacking and minimalist entrepreneurship?

They’re pretty similar. Indie hacker seems like a personality, perhaps similar to a Digital Nomad, where the lifestyle seems to be the precedent. I went to MicroConf in Las Vegas, and the attendee’s goals were fairly consistent - to buy a nice house and spend more time with their family. In that case, your goal should be to build the most boring but profitable business possible, for a community you don’t particularly care about because your goals have nothing to do with serving that community, which is totally fine. No value judgments from me. With indie-hacking, it seems more geared around independence. I tried living the digital nomad life - work solo, travel the world, no schedule, but I didn’t actually enjoy it. It wasn’t really satisfying. I like working on a project with many people, and things improve, and I get to learn from others, they learn from me, I like talking to my customers, who I can talk to frequently, and their lives are getting better because of my work. I enjoy that. So I wanted a middle-ground between the “live on a beach” mentality and the blitzscaling, build the next Facebook mentality. I like to think that with things like crowdfunding, this will get more and more feasible.

Even though my article went viral and the ideas often resonated, there’s this aspirational aspect to many humans - they want to build something amazing and big. It’s kind of the Steve Jobs “make a dent in the universe” idea, even though he might not have actually said that. To account for that, I think incorporating some of the indiehacker principles in the startup path might actually be the most applicable and accessible solution for people.

Utsav: One of the key ideas in the book that, that strikes out to me as someone who's a software engineer is that you can keep trying projects on the side. And eventually, if you're doing things right, if you're talking to customers, you will hit something that people want to buy or to use, right? You're not going to get it right the first time probably. Um, but I think that's a really important idea in this. Could you elaborate on that?

There are two kinds of people: one, who builds a lot of stuff but don’t know who for. Another to-do-list app, a meditation app, you name it. So you build it, but then you can’t figure out who’ll use it. The other kind is stuck in analysis paralysis, and can’t really hone in on an idea that they want to commit to. The solution to both these personas is to forget about business and immerse yourself in the communities you care about, and try to help them. Focus on contributing to these communities. These could be slack/discord communities. For me, it was Hacker News, Dribbble, and IndieHackers. There’s a bunch of subreddits for everything.

Start being a part of these communities, first by listening, and eventually by contributing. I can guarantee that if you become a useful part of the community, you share ideas, people will come up to you and talk about problems that they’re facing. For example, they’re getting paid by YouTube to produce fitness videos, but have to wait for the end of the month, and they’d really like to get paid instantly. Once a community trusts you, and you solve a problem for a specific set of people, you instantly can validate good ideas and deliver value. And iterating over ideas with this community can give you a good chance of success.

Listen to the audio for the full interview!

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev