Outages, bugs, and other production nightmares — every startup has them. Dee and Marie are on to talk about what the process is when something goes wrong at CodePen, and tips and tactics for how to deal with a production fire.

#277: Putting Out Fires

The CodePen team talk about the ins and outs of running a web software business.

Technology

Tech News

Podcasting

Gadgets

Software How-To

Marie and I jump on the show to tell y'all we're taking a little break! It feels like years since we've been eluding to the fact that we're working on a new major upgrade to CodePen. Rather than keep dancing around it, we're going to minimize or remove working on anything that isn't working on that. We can't wait to come back for episode 401 and tell you all about it.



Time Jumps





Sponsor: Split



This podcast is powered by Split. The Feature Management & Experimentation Platform that reimagines software delivery. By attaching insightful data to feature flags, Split frees you to quickly deploy, measure, and learn the impact of every feature you release. So you can safely deliver features up to 50 times faster and exhale. What a Release.



Start raising feature flags (and lowering stress). Visit Split.io/CodePen for a free trial.

400: Hiatus

There was a small problem in our database. Some JSON data we kept in a column would sometimes have a string instead of an integer. Like {"tabSize": "5"} instead of {"tabSize": 5} of the like. Investigation on how that happened was just silly stuff like not calling parseInt on a value as it came off a  element in the DOM. This problem never surfaced because our Rails app just papered over it. But we're moving our code to Go in when you parse JSON in Go, the struct type that you parse it out into needs to match those types perfectly, or else it panics. We had found that our Go code was working around this in all sorts of ways that felt sloppy and inconsistent. 



One way to fix this? Fix any bad data going into the DB, then write a script to fix all the data in the DB. This is exactly the approach I took at first, and it would have absolutely fixed this problem.



But Alex took a step back and looked at the problem a bit wider, and we ended up building some tools that helped us solve this problem, and solve future problems related to this. For one, we built a more permission JSON parser that would not panic on something as easy to fix as a string-as-int problem. This worked by way of some Go reflection that could tell what types the data was supposed to be and coerce them if possible. But what should the value fall back to if it's not savable? That was another tool we built to set the default values of Go structs to be potentially other values than what the defaults for their types are. And since this is all in the realm of data validation, we built another tool to validate the data in Go structs against constraints, so we can always keep the data they contain good.



Once all these tools were in place, the new script to fix the data was much easier to write. Just call the safe JSON function to fix the data and put it back. And the result is a cleaned up code base and tools we can use for data safety for the long term.



Time Jumps





Sponsor: Split



This podcast is powered by Split. The Feature Management & Experimentation Platform that reimagines software delivery. By attaching insightful data to feature flags, Split frees you to quickly deploy, measure, and learn the impact of every feature you release. So you can safely deliver features up to 50 times faster and exhale. What a Release.



Start raising feature flags (and lowering stress). Visit Split.io/CodePen for a free trial.

399: Data Munging

Stephen and I hop on the podcast to chat about some of our recent tooling, local development, and DevOps work. A little while back, we cleaned up our entire monorepo's circular dependency problems using Madge and elbow grease. That kind of thing usually isn't the biggest of deals and the kind of thing a super mature bundler like webpack deals with, but other bundlers might choke on. Later, we learned that we had more dependency issues like inter-package circular dependencies (nothing like production deployments to keep you honest) and used more tooling (shout out npx depcheck) to clean more of it up. Workspaces in a monorepo can also paper over missing dependencies — blech. 



Another change was moving off using a .dev domain for local development, which oddly actually caused some strange and hard-to-diagnose DNS issues sometimes. We're on .test now, which should never be a public TLD.



Time Jumps





Sponsor: Notion



Notion is an amazing collaborative tool that not only helps organize your company’s information but helps with project management as well. We know that all too well here at CodePen, as we use Notion for countless business tasks. Learn more and get started for free at notion.com. Take your first step toward an organized, happier team, today.

398: DevOops

I was asked about the paradoxical nature of CodePen itself recently. CodePen needs to be safe and secure, yet we accept and gleefully execute user-authored code, which is like don't-do-that 101 in web security.  Marie and I hop on the show to talk this through as an update from quite a long time ago. It's wonderfully-terribly complicated. Part of what complicates it is that there are many different kinds of worrisome code, from malicious, to distasteful, to spam, and they all need different treatment. This is a daily and never-ending war.



Time Jumps





Sponsor: Notion



Notion is an amazing collaborative tool that not only helps organize your company’s information but helps with project management as well. We know that all too well here at CodePen, as we use Notion for countless business tasks. Learn more and get started for free at notion.com. Take your first step toward an organized, happier team, today.

397: User-Generated Content Saftey

Robert and I jump on the podcast to have a little chat about open source generally and what we do with open source at CodePen. CodePen itself is not open source, aside from the small bits we've made public and the open-source things we include within it. But all Public Pens on CodePen are open source, so we certainly handle a lot of it! Enough that I felt comfortable making our Mastodon presence on Fosstodon, which is an open-source-focused instance.



Time Jumps





Sponsor: Split



This podcast is powered by Split. The Feature Management & Experimentation Platform that reimagines software delivery. By attaching insightful data to feature flags, Split frees you to quickly deploy, measure, and learn the impact of every feature you release. So you can safely deliver features up to 50 times faster and exhale. What a Release.



Start raising feature flags (and lowering stress). Visit Split.io/CodePen for a free trial.

#277: Putting Out Fires

Download our free app to listen on your phone