Last Week In AWS Podcast

About Corey Quinn

Over the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.

Links

• Retool
• N2WS
• @QuinnyPig

Transcript

Corey: Welcome to AWS Morning Brief: Whiteboard Confessional. I’m Cloud Economist Corey Quinn. This weekly show exposes the semi-polite lie that is whiteboard architecture diagrams. You see, a child can draw a whiteboard architecture, but the real world is a mess. We discuss the hilariously bad decisions that make it into shipping products, the unfortunate hacks the real-world forces us to build, and that the best to call your staging environment is “theory”. Because invariably whatever you’ve built works in the theory, but not in production. Let’s get to it.

This episode is sponsored by a personal favorite: Retool. Retool allows you to build fully functional tools for your business in hours, not days or weeks. No front end frameworks to figure out or access controls to manage; just ship the tools that will move your business forward fast. Okay, let's talk about what this really is. It's Visual Basic for interfaces. Say I needed a tool to, I don't know, assemble a whole bunch of links into a weekly sarcastic newsletter that I send to everyone. I can drag various components onto a canvas: buttons, checkboxes, tables, etc. Then I can wire all of those things up to queries with all kinds of different parameters, post, get, put, delete, etc. It all connects to virtually every database natively, or you can do what I did and build a whole crap ton of lambda functions, shove them behind some API’s gateway and use that instead. It speaks MySQL, Postgres, Dynamo—not Route 53 in a notable oversight; but nothing's perfect. Any given component then lets me tell it which query to run when I invoke it. Then it lets me wire up all of those disparate APIs into sensible interfaces. And I don't know frontend; that's the most important part here: Retool is transformational for those of us who aren't front end types. It unlocks a capability I didn't have until I found this product. I honestly haven't been this enthusiastic about a tool for a long time. Sure they're sponsoring this, but I'm also a customer and a super happy one at that. Learn more and try it for free at retool.com/lastweekinaws. That's retool.com/lastweekinaws, and tell them Corey sent you because they are about to be hearing way more from me.

Welcome to the AWS Morning Brief: Whiteboard Confessional. Today I want to talk about infosec. Specifically, an aspect of infosec that I think is not given proper attention, namely two-factor auth. Now, two-factor auth is important to enable but first, back up a second. Use a password manager with strong passwords for all of your stuff. Those are table stakes at this point. 

Now, most password managers will offer to also store your multi-factor auth codes, your OTP tokens, etcetera. I'm not a big fan of that because it feels to me, perhaps incorrectly, like I'm collapsing multiple factors back down into that same factor. Someone gets access to my password manager, worst-case scenario, I’m potentially hosed. That's not great. Now, the password managers will argue that this isn't technically true, yada, yada. I'm old fashioned. I'm grumpy. I'm an old Unix systems administrator that had certain angry loud opinions, so I'm going to keep using separate tools for managing passwords, as well as getting in as a second factor. May I also point out that SMS is terrible as far as a second factor. Don't use it if you possibly can, for reasons that go well beyond the scope of this show: we're not that kind of podcast. 

Now, let's talk about what happens if you, for one reason or another, lose your MFA device, or the app on your phone because this happened to a certain business partner of mine named Mike Julian. Now, Mike wound up getting a new phone, which is great because his was something from the Stone Age presumably some kind of Nokia candy bar phone. I hear someone dropped one of those things once the last time they were in mass sale and accidentally killed the dinosaurs. So, that's the kind of era of phone he was upgrading from to, I think, the iPhone SE, but don't quote me on that. I don't tend to pay attention to his taste in electronics. Personally, I question his taste in business partners, but that's all right; he signed on the dotted line; he stuck with me now. 

So, he inadvertently wound up losing access to all of his old MFA tokens and having to get them re-added in other places. Some systems worked super well for this. It was a matter of, “Oh, I'll just use my backup codes,” which he kept like a good responsible person. It let him in, he would then be able to regenerate backup codes, change over the device and everything was glory. For others, he wasn't so lucky and had to phone in and get a reset after identity verification. So, now he didn't have his multi-factor device, so it would fall back to using SMS because it had his cell phone. And he could not disable that with some environments. So, that becomes an attack vector, if you're able to compromise an SMS number which, surprise, is not that hard if you put some effort into it. 

This, of course, does bring us to Amazon. Mike needed to reset his Amazon MFA token. Now, when I say Amazon, I don't mean AWS. I mean, Amazon, and I'm going to go back and forth as I go down the story a little bit. So, this is an Amazon retail account, not an AWS account. And it turns out when you Google how to reset your Amazon MFA token, all the results are about AWS. 

So, “Okay, that's interesting,” says Mike. He Googles effectively to remove all results from aws.amazon.com. Cool. Now all the results are about things that are not Amazon stuff. Not anything helpful. So, there's no documentation in Google for any of this as applies to Amazon retail, it may as well not exist as a problem. This is less than ideal from Mike's perspective. He was able to reset his AWS multi-factor auth for the AWS account—that's for the same email address tied to that amazon.com account, but AWS and Amazon have completely separate MFA infrastructures. 

So, this is fascinating. He posts on Twitter, which is the number one way to get help when you have an Amazon issue and you run a company devoted to making fun of Amazon, and AWS support chimes in because they're helpful. Someone else says, “I've been trying to solve this problem for 10 years and got nowhere. Good luck, Godspeed.” And it seemed odd because it's an Amazon retail problem. Why is AWS chiming in? And this leads to a phone call. Mike finally winds up getting on a series of phone calls with AWS support. 

...

About Corey Quinn

Over the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.

Links

• Retool: https://retool.com/
• http://retool.com/lastweekinaws 
• http://snark.cloud/n2ws 
• @QuinnyPig

Transcript

Corey: Welcome to AWS Morning Brief: Whiteboard Confessional. I’m Cloud Economist Corey Quinn. This weekly show exposes the semi-polite lie that is whiteboard architecture diagrams. You see, a child can draw a whiteboard architecture, but the real world is a mess. We discuss the hilariously bad decisions that make it into shipping products, the unfortunate hacks the real-world forces us to build, and that the best to call your staging environment is “theory”. Because invariably whatever you’ve built works in the theory, but not in production. Let’s get to it.

This episode is sponsored by a personal favorite: Retool. Retool allows you to build fully functional tools for your business in hours, not days or weeks. No front end frameworks to figure out or access controls to manage; just ship the tools that will move your business forward fast. Okay, let's talk about what this really is. It's Visual Basic for interfaces. Say I needed a tool to, I don't know, assemble a whole bunch of links into a weekly sarcastic newsletter that I send to everyone. I can drag various components onto a canvas: buttons, checkboxes, tables, etc. Then I can wire all of those things up to queries with all kinds of different parameters, post, get, put, delete, etc. It all connects to virtually every database natively, or you can do what I did and build a whole crap ton of lambda functions, shove them behind some API’s gateway and use that instead. It speaks MySQL, Postgres, Dynamo—not Route 53 in a notable oversight; but nothing's perfect. Any given component then lets me tell it which query to run when I invoke it. Then it lets me wire up all of those disparate APIs into sensible interfaces. And I don't know frontend; that's the most important part here: Retool is transformational for those of us who aren't front end types. It unlocks a capability I didn't have until I found this product. I honestly haven't been this enthusiastic about a tool for a long time. Sure they're sponsoring this, but I'm also a customer and a super happy one at that. Learn more and try it for free at retool.com/lastweekinaws. That's retool.com/lastweekinaws, and tell them Corey sent you because they are about to be hearing way more from me.

Today's episode of the AWS Morning Brief: Whiteboard Confessional was supposed to be about a zero-day that I was disclosing. Cooler heads have prevailed and we will talk about that next week instead, once I've finished some conversations with the company in question. Sorry to disappoint you all, but I have something you might enjoy instead.

So, today I want to talk about getting fired, which is one of my personal specialties. I'm not kidding when I tell people that a primary driver of starting my own consultancy was to build a company wherein I could not be ejected on the spot by surprise. Since I can't be fired anymore, let's talk about the mechanics of getting fired from someone who's been through it, just so folks get a better perspective on this. In the United States, our worker protections are basically non-existent compared to most civilized countries. Barring a contract or collective bargaining agreement to the contrary, you can be fired in the United States for any reason or no reason, except based upon membership in a protected class. 

So, to be clear, my personality is certainly justification enough to fire me. I say this for our listeners in other countries who hear I was fired and equate that to a moral failing. “What’d you do, rob the cash register?” No, I'm just me; I'm difficult to work with; I'm expensive to manage, and my personality is exactly what you would expect it to be based upon this podcast. The way the firing usually works is that you get an unexpected meeting request with your boss. “Hey, can we chat?” 

Those meetings are so unnerving that even that intro leaves scars years later, my business partner and I—both of us can't be fired clearly. But we still get nervous when we tell each other, “Hey, we need to talk in an hour.” We have instituted an actual policy against this at our company, just due to the collective trauma that so many of us have gone through with those, “Is this how I get fired?” moments. So, you have an unplanned meeting with your boss. Nine times out of 10—or more: 99 times out of 100 that's fine—it’s no big deal: it’s about something banal. 

But on this meeting, you walk in and surprise, there's someone from human resources there too, and they don't offer you coffee. First. I want to say the idea of calling people resources is crappy. HR—whatever you want to call it: people ops—but regardless, they're there; they're certainly not smiling, and they don't offer you coffee. 

And that's the tell. When you're invited to a meeting that you weren't expecting and no one gives you coffee, it is not going to be a happy meeting. They usually have a folder sitting there on the table in front of them that has a whole bunch of paperwork in it. There's the, “This is the NDA that you signed, when you started your job here; it's still enforceable: We're reminding you of it paperwork.” There's a last paycheck and a separate paycheck of your cashed out vacation time in jurisdictions where that gets paid out, like California. And often, there's another contract there. This is called a severance agreement. The company is going to pay you some fixed amount of money in return for absolving them of any civil claims that you may have had during the course of your employment. I'm not your attorney, but let me tell you what the right answer here is. 

Whatever you do, do not sign that contract in that room, in that moment. You've just been blindsided; you don't have a job anymore; you're most definitely not at your best. And you're certainly going to be in no position to carefully read a nuanced legal document prepared by your employer’s attorney designed to constrain your future behavior. They may say, “Take all the time you want,” or they may imply they can't give you your last paycheck until you sign it. The Department of Labor would like a word with them if that's the case because that's not legal. 

Thank them, leave with your head held high and bask for a moment in the freeing sense of no longer having any obligation to your now ex-employer. All the projects you had in flight, let them go. All the things y...

About Corey Quinn

Over the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.

Links

• http://nops.io/snark
• http://snark.cloud/n2ws 
• @QuinnyPig

Transcript
Corey: Welcome to AWS Morning Brief: Whiteboard Confessional. I’m Cloud Economist Corey Quinn. This weekly show exposes the semi-polite lie that is whiteboard architecture diagrams. You see, a child can draw a whiteboard architecture, but the real world is a mess. We discuss the hilariously bad decisions that make it into shipping products, the unfortunate hacks the real-world forces us to build, and that the best to call your staging environment is “theory”. Because invariably whatever you’ve built works in the theory, but not in production. Let’s get to it.

nOps will help you reduce AWS costs 15 to 50 percent if you do what tells you. But some people do. For example, watch their webcast, how Uber reduced AWS costs 15 percent in 30 days; that is six figures in 30 days. Rather than a thing you might do, this is something that they actually did. Take a look at it. It's designed for DevOps teams. nOps helps quickly discover the root causes of cost and correlate that with infrastructure changes. Try it free for 30 days, go to nops.io/snark. That's N-O-P-S dot I-O, slash snark.

Welcome once again to the AWS Morning Brief: Whiteboard Confessional. Today I want to talk about, once again, an aspect of writing my Last Week in AWS newsletter. This goes back to before I was sending it out twice a week, instead of just once, and my needs weren't that complex. I would gather a bunch of links throughout the week: I would make fun of them and I had already built this absolutely ridiculous system that would render all of my sarcasm from its ridiculous database where it lived down into something that would work to generate out HTML. And I've talked about that system previously, I'm sure I will again. That's not really the point of the story. 

Instead, what I want to talk about is what happened after I had that nicely generated HTML. Now, I've gone through several iterations of how I sent out my newsletter. The first version was through a service known as Drip, that's D-R-I-P. And they were great because they were aimed at effectively non-technical folks, by and large, where it’s, “Oh, you want to use a newsletter. Go ahead.” 

I looked at a few different vendors. MailChimp is the one that a lot of folks go with for things like this. At the time I was doing that selection, they were having a serious spam problem. People were able to somehow bypass their MFA. Basically, their reputation was in the toilet and given my weird position on email spam, namely, I don't like it, I figured this is probably not the best time to build something on top of that platform, so that was out. 

Drip was interesting, in that they offered a lot of useful things, and they provided something far more than I needed at the time. They would give me ways to say, “Okay, when someone clicks on this link, I can put them in different groups, etcetera, etcetera.” You know, the typical email, crappy tracking thing that squicks people out. Similarly to the idea of, “Hey, I noticed you left something in your cart. Do you want to go back and buy it?” Those emails that everyone finds vaguely disquieting? Yeah, that sort of thing. So, 90 percent of what they were doing, I didn't need, but it worked well enough, and I got out the door and use them for a while. 

Then they got acquired, and it seemed like they got progressively worse month after month, as far as not responding to user needs, doing a hellacious redesign that was retina searingly bad, being incredibly condescending toward customer complaints, subtweeting my co-founder on a podcast, and other delightful things. So, the time came to leave Drip. So, what do I do next? Well, my answer was to go to SendGrid. And SendGrid was pretty good at these things. They are terrific at email deliverability—in other words, getting email, from when I hit send on my system into your inbox, bypassing the spam folder, because presumably, you've opted in to receive this and confirmed that you have opted in—so that wasn't going to be a problem. 

And they still are top-of-class for that problem, but I needed something more than that. I didn't want to maintain my own database of who was on the newsletter or not. I didn't want to have to handle all the moving parts of this. So, fortunately, they wound up coming out with a tool called Marketing Campaigns, which is more or less designed for kind of newsletter-ish style things if you squint at it long enough. And I went down that path and it was, to be very honest with you, abysmal. 

It was pretty clear that SendGrid was envisioning two different user personas. You had the full-on developer who was going to be talking to them via API for the sending of transactional email, and that's great. Then you had marketing campaign folks who were going to be sending out these newsletter equivalents or mass broadcast campaigns, and there was no API to speak of for these things. It was very poorly done. I'm smack dab between this, where I want to be able to invoke these things via API, but I want to also be able to edit it in a web interface, and I don't want to have to handle all the moving parts myself. So, I sat down and had a long conversation with Mike, my business partner. And well, before I get into what we did, let's stop here. 

This episode is sponsored in part by N2WS. You know what you care about? Many things, but never backups. At least until right after you really, really, really needed to care about backups. That's what N2WS does for your AWS account. It allows you to cycle backups through different storage tiers; you can back things up cost-effectively, and safely. For a limited time, N2WS is offering you $100 in AWS credits for setting up their free trial, and I encourage you to give it a shot. To learn more, visit snark.cloud/n2ws. That's snark.cloud/n2ws. 

So, we looked at ConvertKit, which does a lot of things right, but there are a few things wrong. There's still no broadcast API, you have to click things to make it go. So, I have an email background, Mike has an engineering background. And we sat there and we've decided we're going to build something ourselves to solve this problem. And we started drawing on a whiteboard in my office. This thing is four feet by six feet, it is bolted to the wall by a professional because I have the good sense to not tear my own wall down, instead hiring someone else to do it for me. 

<...

About Corey Quinn

Over the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.

Links

• nOps
• N2WS
• @QuinnyPig

Transcript

Corey: Welcome to AWS Morning Brief: Whiteboard Confessional. I’m Cloud Economist Corey Quinn. This weekly show exposes the semi-polite lie that is whiteboard architecture diagrams. You see, a child can draw a whiteboard architecture, but the real world is a mess. We discuss the hilariously bad decisions that make it into shipping products, the unfortunate hacks the real-world forces us to build, and that the best to call your staging environment is “theory”. Because invariably whatever you’ve built works in the theory, but not in production. Let’s get to it.

nOps will help you reduce AWS costs 15 to 50 percent, if you do what tells you. But some people do. For example, watch their webcast, how Uber reduced AWS costs 15 percent in 30 days; that is six figures in 30 days. Rather than a thing you might do, this is something that they actually did. Take a look at it. It's designed for DevOps teams. nOps helps quickly discover the root causes of cost, and correlate that with infrastructure changes. Try it free for 30 days, go to nops.io/snark. That's N-O-P-S dot I-O, slash snark.

Corey: Welcome to the AWS Morning Brief: Whiteboard Confessional. Today we're going to tell a story that only happened a couple of weeks ago. We don't usually get to tell stories about what we do in the AWS bill fixing realm because companies are understandably relatively reticent to talk about this stuff in public. They believe, rightly or wrongly, that it will annoy Amazon which, frankly, is one of my core competencies. They think that it shows improper attention to detail to their investors and others.

I don't see it that way, but I found a story that we can actually talk about today in a bit more depth and detail than we normally would. So, we get a phone call about three weeks ago. Someone has a low five-figure bill every month on AWS. That's generally not large enough for us to devote a consulting project to, because frankly the ROI would take far too long. It's too small of a bill to make an engagement with us cost-effective, but we had a few minutes to kill and thought, “Eh, go ahead and pull up your bill. We'll take a quick look at it here.” Half of their bill historically—and growing—had been data transfer which, okay, that's interesting. And as we've known from previous episodes, data transfers is always strange. So, they looked at this and they said, “Okay, well, obviously then instead of serving things directly, we should get a CDN.” 

So, then instead of getting a CDN, they chose to set up CloudFront, which basically is a CDN only worse in every way. And they saw no impact to their bill after a month of this. Okay, let's change that up a bit. Now, instead of CloudFront. We're going to move to an actual CDN. So, they did, there was a small impact to their bill. Okay, and costs continue to rise and what's going on? At this point in the story, they call us, which generally if you're seeing something strange on your bill, is not a terrible direction to go in. We see a lot of these things and if we can help you, we will point you towards someone who can. So, our consensus on this was, great. It is too small to look at the bill. 

But let's pop into Cost Explorer and see what's going on. We break it down by service and S3 was the biggest driver of spend. Now, that's interesting. Number two was EC2. But okay, we start with the big numbers and work our way down. This is apparently novel for folks doing in-depth bill analysis, but we're going to go with it anyway. We start taking a look within that S3 category of usage type, and lo and behold, data transfer out to the internet is driving almost all of it. The cost per request is super low. That tells us in turn that—because we've seen a lot of these—that there are large objects, but relatively few requests for them. 

So, all right, we're going to slice and dice slightly differently within Cost Explorer, AWS’s free—with an asterisk next to it—tool for exploring various aspects of your bill. That asterisk, incidentally, means that if you're doing this via API, it is one cent per call. If you're doing this in the console, it's free. Be aware, that can catch you by surprise if you write a lot of very chatty scripts. You have been warned. So, yeah, most of the spend was indeed on GetObject calls. So, okay, we know that data transfer spend was coming from an S3 bucket that was not going to a CDN. Otherwise, it's going to show up as a different data transfer charge in a different section of their bill. 

Okay, so now we know it's S3. We have to figure out what bucket it lives within. And this is an obnoxious process, and we tell them this. And they’re like, “Oh, yeah, we know what bucket that is.” This, incidentally, is where almost every software tool tends to fall down. We could spend some time tracking this down programmatically. Or we can just ask someone who already has the context of what their business does and how it works loaded into their head. Because otherwise, what we'd have to do is tag all their buckets, and then wait a few days for that tag to percolate into the billing system, and then query it again because the visibility into this sort of thing is terrible. It's a shortcoming of both Cost Explorer and S3 in that weird seam between the two. There's fundamentally no easy way to see at a glance which buckets are costing you money unless you do something fun with tagging. 

To that end, I'm a big believer in having every bucket tagged with a bucket name option. So, I can start slicing on that, and then you enable that as a cost allocation tag. So, great. Now, what is it that's getting requested? Well, you can also dig into this via access logs once you have the bucket, to see what's going on. Great. Now, we take a look at this, and sure enough—well, before I go into what it actually was, let's pause here for a moment. 

This episode is sponsored in part by N2WS. You know what you care about? Many things, but never backups. At least until right after you really, really, really needed to care about backups. That's what N2WS does for your AWS account. It allows you to cycle backups through different storage tiers; you can back things up cost-effectively, and safely. For a limited time, N2WS is offering you $100 in AWS credits for setting up their free trial, and I encourage you to give it a shot. To learn more visit snark.cloud/n2ws. That's snark.cloud/n2ws. 

Corey: So, this bucket was set to public access. Aha! it only allowed GetO...

About Corey Quinn

Over the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.

Links

• http://nops.io/snark
• http://snark.cloud/n2ws 
• @QuinnyPig

Transcript

Corey: Welcome to AWS Morning Brief: Whiteboard Confessional. I’m Cloud Economist Corey Quinn. This weekly show exposes the semi-polite lie that is whiteboard architecture diagrams. You see, a child can draw a whiteboard architecture, but the real world is a mess. We discuss the hilariously bad decisions that make it into shipping products, the unfortunate hacks the real-world forces us to build, and that the best to call your staging environment is “theory”. Because invariably whatever you’ve built works in the theory, but not in production. Let’s get to it.

nOps will help you reduce AWS costs 15 to 50 percent if you do what tells you. But some people do. For example, watch their webcast, how Uber reduced AWS costs 15 percent in 30 days; that is six figures in 30 days. Rather than a thing you might do, this is something that they actually did. Take a look at it. It's designed for DevOps teams. nOps helps quickly discover the root causes of cost, and correlate that with infrastructure changes. Try it free for 30 days, go to nops.io/snark. That's N-O-P-S dot I-O, slash snark.

Good morning AWS, and welcome to the AWS Morning Brief: Whiteboard Confessional. Today we're going to revisit DNS. Now, now, slow down there, Hasty Pudding. Don't bother turning the podcast off. For once, I'm not talking about using it as a database… this time. As you're probably aware, DNS is what folks use to equate friendly names for twitterforpets.com, or incredibly unfriendly names like Oracle.com, to IP addresses, which is how computers tend to see the world. I'm not going to rehash what DNS does. 

Instead, I'm going to talk about a particular kind of DNS problem that befell a place I used to consult for. They're publicly traded now, so I'm not going to name them. An awful lot of shops do something that's called split-horizon DNS. What that means is that if you're on a particular network, a DNS name resolves differently than it does when you're on a different network. For example, admin.twitterforpets.com will resolve to an administrative dashboard if you're on the Twitter For Pets internal network via VPN, but it won't resolve to that dashboard if you're outside the network, or it might resolve nowhere, or it might resolve just back to their main website, www.twitterforpets.com. 

And that's fine. Most DNS providers can support this, and Route 53 is, of course, no exception. This is, incidentally, what the Route 53 resolver, that was released in 2018, is designed to do: it bridges private DNS zones to on-premises environments, so your internal zones can then resolve to private IP addresses without having to show your private IP address ranges in public zones to everyone. So, the reason that matters is that this keeps you from broadcasting your architecture or your network layout externally to your company. Some folks consider doing that to be a security problem because it discloses information that an attacker can then leverage to gain further toeholds into your network. Some folks also think that that tends to be a little bit on the extreme side. I'll let you decide because I don't care, and that's not what the story is about. 

The point is that split-horizon DNS is controversial, for a few reasons, but in many shops, it is considered the right thing to do because it's what they've been doing. The internal DNS names either don't resolve anything publicly, or they resolve to a different system that’s configured to reject the request outright. But there is another path you can take; a third option that no one discusses because it's a path that's far darker, because it is oh, so very much dumber. But first…

This episode is sponsored in part by N2WS. Do you know what you care about? Many things, but never backups. At least until right after you really, really, really needed to care about backups. That's what N2WS does for your AWS account. It allows you to cycle backups through different storage tiers; you can back things up cost-effectively, and safely. For a limited time, N2WS is offering you $100 in AWS credits for setting up their free trial, and I encourage you to give it a shot. To learn more visit snark.cloud/n2ws. That's snark.cloud/n2ws. 

What I'm about to describe is far too stupid for my made-up startup of Twitter For Pets, so we're going to have to invent a somehow even dumber company, and we're going to call it Uber For Squirrels. It's like regular Uber, except it somehow manages to lose less money. Now, there's a very strong argument among the engineering community inside of Uber For Squirrels. Split-horizon DNS is dangerous is what is decided and argued for. And that's the proclamation because a misconfiguration could leak records in the wrong places, and theoretically take the entire online site for Uber For Squirrel down. There are merits to those arguments and you can't dismiss them out of hand, so a bargain was struck. 

The external DNS zone was therefore decreed to be uberforsquirrels.com, while the internal zone was configured to be uberforsquirrels.net. The uberforsquirrels.net zone was only accessible inside of the network. From the outside, nobody could query it. Now, this is, in isolation—before I go further—a bad plan all on its own. When you're reading quickly, uberforsquirrels.com and uberforsquirrels.net don't jump out visually to people as being meaningfully different. You're going to typo it in config files constantly without meaning to, and then you're going to have a hell of a time tracking it down because it's not immediately obvious that you're talking to the wrong thing; you might think it's a network problem. Your tab completion is going to break out of your known_hosts file, if you have such a thing configured in your environment, it's going to have to hit tab a couple of extra times to cycle through the dot net variants and the dot com variants. It's just a general irritant. 

But that's not enough to justify an episode of the show. Because wait, that is still some Twitter For Pets level brokenness. Why do I need to throw Uber For Squirrels under the bus? Well, because it turns out that despite using uberforsquirrels.net everywhere as their internal domain, they didn't actually own uberforsquirrels.net. It wasn't entirely clear who did other than that the registration was in another country, so it probably wasn't something that the CEO registered and then forgot about in his random domain list of things he acquired for companies he was going to start o...

AWS Morning Brief for the week of May 18, 2020.