Last Week In AWS Podcast

Links Mentioned

• Want to give your ears a break and read this as an article? You’re looking for this link: https://www.lastweekinaws.com/blog/cloud-repatriation-isnt-a-thing/ 

Sponsors

• A Cloud Guru: https://acloudguru.com
• New Relic: https://newrelic.com

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of August 17, 2020.

About Corey Quinn

Over the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.

Links

• Trend Micro Cloud One™
• @QuinnyPig
• ChaosSearch

Transcript

Corey: Welcome to AWS Morning Brief: Whiteboard Confessional. I’m Cloud Economist Corey Quinn. This weekly show exposes the semi-polite lie that is whiteboard architecture diagrams. You see, a child can draw a whiteboard architecture, but the real world is a mess. We discuss the hilariously bad decisions that make it into shipping products, the unfortunate hacks the real-world forces us to build, and that the best to call your staging environment is “theory”. Because invariably whatever you’ve built works in the theory, but not in production. Let’s get to it.

Corey: This episode is brought to you by Trend Micro Cloud One™. A security services platform for organizations building in the Cloud. I know you're thinking that that's a mouthful because it is, but what's easier to say? “I'm glad we have Trend Micro Cloud One™, a security services platform for organizations building in the Cloud,” or, “Hey, bad news. It's going to be a few more weeks. I kind of forgot about that security thing.” I thought so. Trend Micro Cloud One™ is an automated, flexible all-in-one solution that protects your workflows and containers with cloud-native security. Identify and resolve security issues earlier in the pipeline, and access your cloud environments sooner, with full visibility, so you can get back to what you do best, which is generally building great applications. Discover Trend Micro Cloud One™ a security services platform for organizations building in the Cloud. Whew. At trendmicro.com/screaming.

In almost any production environment, there's going to be a few tasks as your company grows that someone winds up having to perform in your production app. And in many cases, the people who have to perform those tasks are themselves not excessively technical, which means if you fail to properly invest in internal tooling, well, that means you're going to have someone who winds up getting this, effectively, printed out page that hangs in their cubicle—or equivalent during these uncertain times—where they wind up following a checklist of, step one: SSH into a production server. Step two: copy and paste the following command, which in turn, I don't know, spins up a Ruby on Rails console, or does some task on the database and returns a query. Now, this is universally recognized as awful because, for better or worse, most business users are not overwhelmingly comfortable when it comes to using SSH on the command line.

Now, in an ideal world with unlimited resources, you would be able to have an internal tools developer who could focus on things like that specifically for your teams. And in fact, most very large hyper-scale companies have entire herds of people doing nothing but that. But when you're building something from scratch, and you're a relatively small, scrappy team, it's much more challenging because you take a step back and have to make some unfortunate and challenging determinations of, “Okay, am I going to A) sit here and have very expensive people build tooling, or B) have them work on features, which, you know, bring money into the company?” I'm not going to sit here and say that people are wrong for not investing in internal tooling early on. 

But at some point, the longer you go without making those investments, the greater your risk is because someone is going to get something wrong. They're going to fat-finger a command somewhere; they're going to run it on the wrong system; a key pair is going to not do what it needs to do; some error-checking was not built into whatever script you're having them run, and a command is going to fail, but it's going to continue on as if it succeeded and potentially run the wrong thing in the wrong place. It effectively is setting up a recipe for disaster, and when this happens, as it inevitably will, the natural response is going to be to blame the poor schmuck who had to go ahead and run your crappy shell script command because you couldn't bother to invest in internal tooling. This is an area that's near and dear to my heart because it's something that I spend a fair bit of time worrying about myself. Again, I've built a ridiculous architecture that powers my newsletters, and I have a separate aspect of that, that lets my ad sales folks wind up injecting sponsor stuff into the newsletter for me. 

Fun fact that isn't super well known, I don't see any of the sponsor stuff that goes out in my newsletter until after I've already written that week's issue because I don't want to wind up finding myself having to change what I say to avoid irritating a sponsor, you know, like someone with a sense of self-preservation or an appreciation for maintaining their income might do. So, it's sort of an editorial firewall for me. In order for that to make sense, though, there was no way in the world I was going to get away with having people who are managing the ad sales portion, SSH-ing into a box, and running this arcane script that talks to DynamoDB. And, “Oh, yeah, just run this script; it invokes a lambda function, and—hey, where are you going? Come back,” is how that story is going to play out. 

So, my initial approach was to look into what it would take to pay someone who's good at building web forms and front-end tooling. It turns out those people cost a lot of money. My approach was to ultimately use Retool, which I've talked about repeatedly on this show, but there are a lot of tools in this space. AWS Honeycode, for example, is one of the worst examples of something like this. The value there is that it ties together a bunch of APIs with a drag-and-drop Visual Basic style interface that lets you build internal web apps. 

And their pricing model is such that you would never in a million years use this for anything public. But for internal tooling, it's a great approach. Sure, you need some developer time to set up the APIs, or the scripts that it calls on the back end, but it's really an accelerated function here because you don't need anyone to spend time on UI, past drag and drop. When it comes time to update something, you can wind up changing an API parameter or building a quick API on the other side and the interface remains remarkably consistent for users. There are a number of tools like this out there, and I'm a big fan of the no-code/low-code movement, specifically because it solves incredible business issues here.

This episode is sponsored in part by our good friends over a ChaosSearch, which is a fully managed ...

Links Mentioned

• Want to give your ears a break and read this as an article? You’re looking for this link: https://www.lastweekinaws.com/blog/your-disaster-recovery-plan-is-a-joke-written-by-clowns/ 

Sponsor

• New Relic: https://newrelic.com/

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of August 10, 2020.

About Corey Quinn

Over the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.

Links

• CHAOSSEARCH
• @QuinnyPig

Transcript

Corey: Welcome to AWS Morning Brief: Whiteboard Confessional. I’m Cloud Economist Corey Quinn. This weekly show exposes the semi-polite lie that is whiteboard architecture diagrams. You see, a child can draw a whiteboard architecture, but the real world is a mess. We discuss the hilariously bad decisions that make it into shipping products, the unfortunate hacks the real-world forces us to build, and that the best to call your staging environment is “theory”. Because invariably whatever you’ve built works in the theory, but not in production. Let’s get to it.

Corey: This episode is brought to you by Trend Micro Cloud One™. A security services platform for organizations building in the Cloud. I know you're thinking that that's a mouthful because it is, but what's easier to say? “I'm glad we have Trend Micro Cloud One™, a security services platform for organizations building in the Cloud,” or, “Hey, bad news. It's going to be a few more weeks. I kind of forgot about that security thing.” I thought so. Trend Micro Cloud One™ is an automated, flexible all-in-one solution that protects your workflows and containers with cloud-native security. Identify and resolve security issues earlier in the pipeline, and access your cloud environments sooner, with full visibility, so you can get back to what you do best, which is generally building great applications. Discover Trend Micro Cloud One™ a security services platform for organizations building in the Cloud. Whew. At trendmicro.com/screaming.

Welcome. I am Cloud Economist Corey Quinn, and this is the AWS Morning Brief: Whiteboard Confessional. One of the nice things about how I do business is that I don't actually know when I record these episodes, who is going to be sponsoring it. Today, I'm going to talk about secrets management. The reason I bring this up is that should whatever sponsor has landed the ad slot for this week be talking about a different way of handling secrets management, you should of course disregard everything I'm about to say, and buy their product and or service instead. That said, let's talk about secrets management and how it can be done in some of the most appalling ways imaginable.

There are a depressing number of you listening to this, where if I were to steal your laptops, A) you potentially would not have hard drive encryption turned on, so I could just pull things off of your system. That said, most modern operating systems do this by default now, so that's less of a threat. Now, let's pretend that I wind up instead surmounting an almost impossible barrier. That's right, getting a corrupted browser extension onto your system that somehow has access to poke around in your user's home directory. 

Think for a second about what I might find. Would I find, oh, I don't know, SSH keys that would grant me access to your production environment? Well, that wouldn't be that big of a problem because there's no possible way I would know what hosts they go for unless I look at the known_hosts file sitting right next to your SSH keys. But even that's a little esoteric because that's not something I would ever do at grand scale. Let’s instead consider what happens if I poke around in the usual spots and find long-lived IAM credentials, or whatever your cloud provider of choice’s equivalent is, which I believe is IAM in most cases unless you're using IBM Cloud, in which case, it's probably an old-timey skeleton key that is physically tied to your laptop. 

Now, the reason this becomes a common pattern is because it's honestly pretty convenient. You're going to need to be able to access production environments or your cloud environment, and have permissions that are generally granted to you, and ease of access is always juxtaposed with convenience. And invariably, convenience tends to win out. Sure, you can mandate the use of multi-factor authentication for those credentials to get into production, but that means you have to type in a code or press a button on a Yubikey, or something else. That fundamentally means you're going to be spending a lot more time pressing buttons or digging out passphrases than you're going to spend getting into production in a hurry. 

So, we make trade-offs; we cheat; it's human nature. And of course, once you get into your production environment, things are rarely better. It seems that you have a choice. You can either have the same password shared absolutely everywhere within an environment, or you have these incredibly secure key management systems, but in return becomes virtually impossible to rotate credentials. We've seen this before, and we've talked about this before. When we look at what happens when someone leaves a job unexpectedly, and suddenly the credential rotation causes four site outages in the next two days.

There's always a trade-off here. And the problem is, is that these elaborate multi-step secret retrieval processes that people can deploy are no stronger than their weakest link. I've talked about it in an early episode, but probably one of the most bizarre I've ever seen was for regulated data, where in order to start the database server, it required a long key that was cut into pieces, and then we needed to have multiple staff contribute and turn their key like we were launching a freakin’ nuclear missile from a submarine. And it worked, sure, but at the same time, it meant to restart a server, you needed at least two people nearby, and that became a little nutty. Let's also ignore for a minute the fact that this was just for encrypting the data at rest. 

Once the service was running, it was loaded into RAM. There was no real guarantee that this was going to be any more secure than anything else. And let's face it, we're living in an era now where people stealing the server out of our cloud-hosted environment is not the primary or secondary or tertiary threat modeling that anyone has to do. For better or worse, you can give an awful lot of crap to the cloud providers, but they've pretty much solved the ‘someone rams a truck into the side of the building, grabs a rack into the back of said truck and peels off into the night.’ Except IBM Cloud. So, what are some patterns that work for this? Great question. But first:

Corey: This episode is sponsored in part by ChaosSearch. Now their name isn’t in all caps, so they’re definitely worth talking to. What is ChaosSearch? A scalable log analysis service that lets you add new workloads in minutes, not days or weeks. Click. Boom. Done. Chaos...

Links Mentioned

• Want to give your ears a break and read this as an article? You’re looking for this link https://www.lastweekinaws.com/blog/multi-cloud-is-the-worst-practice/ 

Sponsors

• New Relic: https://newrelic.com/

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

About Corey Quinn

Over the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.

Links

• CHAOSSEARCH
• @QuinnyPig

Transcript

Corey: Welcome to AWS Morning Brief: Whiteboard Confessional. I’m Cloud Economist Corey Quinn. This weekly show exposes the semi-polite lie that is whiteboard architecture diagrams. You see, a child can draw a whiteboard architecture, but the real world is a mess. We discuss the hilariously bad decisions that make it into shipping products, the unfortunate hacks the real-world forces us to build, and that the best to call your staging environment is “theory”. Because invariably whatever you’ve built works in the theory, but not in production. Let’s get to it.

Corey: This episode is brought to you by Trend Micro Cloud One™. A security services platform for organizations building in the Cloud. I know you're thinking that that's a mouthful because it is, but what's easier to say? “I'm glad we have Trend Micro Cloud One™, a security services platform for organizations building in the Cloud,” or, “Hey, bad news. It's going to be a few more weeks. I kind of forgot about that security thing.” I thought so. Trend Micro Cloud One™ is an automated, flexible all-in-one solution that protects your workflows and containers with cloud-native security. Identify and resolve security issues earlier in the pipeline, and access your cloud environments sooner, with full visibility, so you can get back to what you do best, which is generally building great applications. Discover Trend Micro Cloud One™ a security services platform for organizations building in the Cloud. Whew. At trendmicro.com/screaming.

Hello, and welcome to this edition of the AWS Morning Brief: Whiteboard Confessional, where we confess our various architectural sins that we and others have committed. Today, we're going to talk about, once upon a time, me taking a job at a web hosting provider. It was the thing to do at the time because AWS hadn't eaten the entire world yet, therefore, everything that we talk about today was still a little far in the future. So, it was a more reasonable approach, especially for those with, you know, budgets that didn't stretch to infinity, or willingness to be an early adopter of someone else's hosting nonsense to go ahead and build out something in a data center. 

Now, they were obviously themselves not hosting on top of a cloud provider because the economics made less than no sense back then. So, instead, they had multiple data centers built out that provided for customers various hosting needs. Each one of these was relatively self-contained unless customers wound up building something themselves for failover. So, it wasn't really highly available so much as it was a bunch of different single points of failure, and an outage of one would impact some subset of their customers, but not all of them. And that was a fairly reasonable approach provided that you communicate that scenario to your customers because that's an awful surprise to have later in time. 

Now, I was brought in as someone who had had some experience in the industry, unlike many of my colleagues who had come from the hosting provider’s support floor and promoted into systems engineering roles. So, I was there to be the voice of industry best practices, which is a terrifying concept when you realize that I was nowhere near as empathetic or aware back then as I am now, but you get what you pay for. And my role was to apply all of those different best practices that I had observed, and osmosed, and had bluffed, into what this company was doing, and see how it fit in a way that was responsible, engaging, and possibly entertaining. So, relatively early on in my tenure, I was taking a tour of one of our local data centers and asked what I thought could be improved. Now, as a sidebar, I want to point out that you can always start looking at things and pointing out how terrible they are, but let's not kid ourselves; we very much don't want to do that because there are constraints that shape everything that we do and we aren't always aware of them. So, making people feel bad for their choices is never a great approach if you want to stick around very long. So, instead, I started from the very beginning, and played, “Hi. I'm going to ask the dumb questions, and see where the answers lead me to.” 

So, I started off with, “Great, scenario time. The power has just gone out. So, everything's dark, now how do we restart the entire environment?” And the response was, “Oh, that would never happen.” And to be clear, that's the equivalent of standing on top of a mountain during a thunderstorm, cursing God while waving a metal rake into the sky. After you say something like that there is no disaster that is likelier. But all right, let's defuse that. “Humor me. Where's the runbook?” And the answer is, “Oh, it lives in Confluence,” which is Atlassian’s wiki offering. For those who aren't aware, Wikis in general, and Confluence in particular, is where documentation and processes go to die. “These are living documents,” is a lie that everyone says because that's not how it actually works. 

“Cool. Okay, so let's pretend that a single server instead of your whole data center explodes and melts. When everything's been powered off, you turn it back on. That one doesn't survive the inrush current, and that one server explodes. That server happens to be the Confluence server. Now what? How do we bootstrap the entire environment?” The answer was, “Okay, we started printing out that runbook and keeping it inside each data center,” which was a way better option. Now, the trick was to make sure that you revisited this every so often, when something changed, and make sure that you weren't looking at how things were circa five years ago, but that's a separate problem. And this is fundamentally a microcosm of what I've started to think of as the bootstrapping problem. I'll talk to you a little bit more about what those look like in the context of my data center atrocities. But first:

This episode is sponsored in part by our good friends over a ChaosSearch, which is a fully managed log analytics platform that leverages your S3 buckets as a data store with no further data movement required. If you're looking to either process multiple terabytes in a petabyte-scale of data a day or a few hundred gigabytes, this is still economical and worth looking into. You don't have to manage Elasticsearch yourself. If your ELK stack is falling over, take a look at using ChaosSearch for log analytics. Now, if you do a direct cost comparison, you're going to say, “Yeah, 70 to 80 percent on the infrastructure costs,” which does not include the actual expense of p...