Last Week In AWS Podcast

Links

• Follow Last Week In AWS on Twitter
• AWS Outage Message
• "Kinesis Outage" by Ryan Frantz

Transcript
Corey: This episode is sponsored in part by our friends at Linode. You might be familiar with Linode; they’ve been around for almost 20 years. They offer Cloud in a way that makes sense rather than a way that is actively ridiculous by trying to throw everything at a wall and see what sticks. Their pricing winds up being a lot more transparent—not to mention lower—their performance kicks the crap out of most other things in this space, and—my personal favorite—whenever you call them for support, you’ll get a human who’s empowered to fix whatever it is that’s giving you trouble. Visit linode.com/screaminginthecloud to learn more, and get $100 in credit to kick the tires. That’s linode.com/screaminginthecloud.

Pete: Hello, everyone. Welcome to the AWS Morning Brief. It's Pete Cheslock again—

Jesse: And Jesse DeRose.

Pete: We are back to talk about ‘The Kinesis Outage.’

Jesse: [singing] bom bom bum.

Pete: So, at this point, as you're listening to this, it's been a couple of weeks since the Kinesis outage has happened, and I'm sure there are many, many armchair sysadmins out there speculating at all the reasons why Amazon should not have had this outage. And guess what? You have two more system administrators here to armchair quarterback this as well.

Jesse: We are happy to discuss what happened, why it happened. I will try to put on my best announcer voice, but I think I normally fall more into the golf announcer voice than the football announcer voice, so I'm not really sure if that's going to play as well into our story here.

Pete: It's going, it's going, it's gone.

Jesse: It’s—and it's just down. It's down—

Pete: It's just—

Jesse: —and it's gone.

Pete: No, but seriously, we're not critiquing it. That is not the purpose of this talk today. We're not critiquing the outage because you should never critique other people's outages; never throw shade at another person's outage. That's not only crazy to do because you have no context into their world. It's just, it's not nice either, so just try to be nice out there.

Jesse: Yeah, nobody wants to get critiqued when their company has an outage and when they're under pressure to fix something. So, we're not here to do that. We don't want to point any fingers. We're not blaming anyone. We just want to talk about what happened because honestly, it's a fascinating, complex conversation.

Pete: It is so fascinating and honestly, loved the detail, a far cry from the early years of Amazon outages that were just, “We had a small percentage of instances have some issues.” This was very detailed. This gave out a lot of information. And the other thing too is that, when it comes to critiquing outages, you have to imagine that there are unlikely to be more than a handful of people even inside Amazon Web Services that fully understand the scope of the size and the interactions of all these different services. There may not even be a single person who truly understands how these dozens of services interact with each other. 

I mean, it takes teams and teams of people working together to build these things and to have these understandings. So, that being said, let's dive in. So, the Wednesday before Thanksgiving, Kinesis decided to take off early. You know, long weekend coming up, right? But really, what happened was is that there was an addition of capacity to Kinesis, and it caused it to hit an operating system limit causing an outage. 

But interestingly enough—and what we'll talk about today—are the interesting and downstream effects that occurred via CloudWatch, Cognito, even the status page, and the Personal Health Dashboard. I mean, that's a really interesting contributing factor or a correlating outage. I don't know the words here, but it's interesting to hear that both CloudWatch goes down and the Personal Health Dashboard goes down.

Jesse: That's when somebody from the product side says, “Oh, that's a feature, definitely not a bug.”

Pete: But the outage to CloudWatch then even affected some of the downstream services to CloudWatch—such as Lambda—which also included auto-scaling events. It even included EventBridge, which was impacted, and that even caused some ECS and EKS delays with provisioning new clusters and scaling of existing clusters.

Jesse: So, right out of the bat, I just want to say huge kudos to AWS for dogfooding all of their services within AWS itself: not just providing the services to its customers, but actually using Kinesis internally for other things like CloudWatch and Cognito. They called that out in the write-up and said, “Kinesis is leveraged for CloudWatch, and Cognito, and for other things, for various different use cases.” That's fantastic. That's definitely what you want from your service provider.

Pete: Yeah, I mean, it's a little amazing to hear, and also a little terrifying, that all of these services are built based on all of these other services. So, again, the complexity of the dependencies is pretty dramatic. But at the end of the day, it's still software underneath it; it's still humans. And I don't want to say that I am happy that Amazon had this outage at all, but watching a company of this stature, of this operational expertise, have an outage, it's kind of like watching the Masters when Tiger Woods duffs one into the water or something like that. It's just—it's a good reminder that—listen, we're all human, we're all working under largely the same constraints, and this stuff happens to everyone; no one is immune.

Jesse: And I think it's also a really great opportunity—after the write-up is released—to see how the Masters go about doing what they do. Because everybody at some point is going to have to troubleshoot some kind of technology problem, and we get to see firsthand from this, how they go about troubleshooting these technology problems.

Pete: Exactly. So, of course, one of the first things that I saw everywhere is everyone is, on mass, moving off of Amazon, right? They had an outage, so we're just going to turn off all our servers and just move over to GCP, or Azure, right? 

Jesse: Because GCP is a hundred percent uptime. Azure is a hundred percent uptime. They're never going to have any kind of outages like this. Google would never do something to maybe turn off a service, or sunset something.

Pete: Yeah, exactly. So, with the whole talk about hybrid-cloud and multi-cloud strategies, you got to know that there's a whole slew of people out there, probably some executive at some business, who says, “Well, we need to engineer for this type of durability, this type of thing to happen again,” but could you even imagine the complexity...

Want to give your ears a break and read this as an article? You’re looking for this link.

Sponsors

• Veeam
• Linode

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Links

• Follow Last Week In AWS on Twitter

Transcript
Corey: This episode is sponsored by ExtraHop. ExtraHop provides threat detection and response for the Enterprise (not the starship). On-prem security doesn’t translate well to cloud or multi-cloud environments, and that’s not even counting IoT. ExtraHop automatically discovers everything inside the perimeter, including your cloud workloads and IoT devices, detects these threats up to 35 percent faster, and helps you act immediately. Ask for a free trial of detection and response for AWS today at extrahop.com/trial.

Pete: Hello, welcome to AWS Morning Brief. I am Pete Cheslock, and I am here yet again with Jesse DeRose.

Jesse: Hello. 

Pete: We here to talk about the best service announced not during AWS Storage Day 2020.

Jesse: So, close.

Pete: So, close, though. It was announced a few days after, and that is the AWS S3 Storage Lens service, which I think I've got that naming right. I know sometimes it's ‘AWS thing,’ sometimes it's ‘Amazon thing,’ and to be honest, I never know which is which. 

Jesse: Yeah.

Pete: AWS S3 Storage Lens is honestly one of the best new services that I've seen out, released thus far. I guess we're still pre-re:Invent announcements in a lot of this stuff. But what it is is a—from their site it says, “S3 Storage Lens delivers organization-wide visibility into object storage usage, activity trends,” blah, blah, blah, blah, blah, marketing speak. Basically, it allows you to get a view of your S3 usage across accounts. Which, that's mindblowing, right?

Jesse: Yeah. This feature has so much potential; I'm really excited to see where they go with it.

Pete: Yeah. And so when I first saw this blog post on Amazon’s site talking about it, my mind just started going crazy because again, we work in Duckbill Group as cloud economists with a lot of different clients, and because Amazon organizations may be the reason why, made it very easy to spin up new accounts, maybe also the adage, the design principle of creating many Amazon accounts to kind of segment workloads or to provide you to—segment your workloads in a way for cost reasoning or security reasons. But all of those things—somewhat related, somewhat not—have caused a lot of our clients to have lots of Amazon accounts. I mean, you could see hundreds, in some cases, of Amazon accounts. 

And the issue that I've always kind of had, and especially an issue we deal with in helping our clients analyze their costs and optimize their costs is how do you aggregate S3 usage? Because S3 is normally in the top five of services that we see in usage, how do you pull that together? And I guess we do that a lot of different ways. Jesse, maybe you can chat a little bit about what are some of the ways that we try to analyze this spend currently?

Jesse: Yeah. Pete, I think I'm really excited about this feature because AWS already offers aggregate looks at metrics for other top services by spend. Like, for EC2, you've got Compute Optimizer. We don't have anything for RDS yet, but I feel like that might be not far off, given Compute Optimizer’s existence. And we already have other tools that allow you to look across multiple accounts to look at metrics, especially if you're looking at Cost Explorer, for example, you can see metrics across multiple accounts, you can see spend across multiple accounts. 

So, I feel like this makes sense. I'm really excited to see that you can look at all of your S3 storage metrics in one place because right now, the only way that we're able to get any kind of representation of S3 usage is through Cost Explorer. And there are ways that you can go about filtering and slicing that data to get usage information and certain metrics, slicing and dicing on different filters for accounts and cost allocation tags, but it's all at the bucket level, or at the usage level, and if you really want to dig in deeper, you don't have a lot of options.

Pete: Yeah, it's a service that they're operating on your behalf. So, your only insight is what they give you insight into. Maybe some of that is CloudWatch metrics, there's obviously the S3 storage analytics that can give you some idea in your storage—based on access—that can help you kind of optimize, but nothing really again at the—ability to see it across multiple accounts is I think, really the big game-changer too.

Jesse: And I think what's really amazing here is that the majority of metrics that they're offering are free. And we'll get into that in a minute, but I'm really impressed that so many of these metrics are shared free of charge. You just have to turn it on. And then you have access to all of this great information that you can work with. 

Pete: Yeah. I think that's a great point that we haven't mentioned yet, that this is—the basic form of this is free. And the metrics that you can get are pretty useful in the free tier. Also, this is actually something that is turned on in your account right now. If you have an Amazon account, go into S3, it's actually under S3, it'll be on the left-hand column—at least it should be unless they go move stuff around—but you'll see a drop-down for Storage Lens, and you'll see an option for dashboards. 

And when you go into the dashboards, there will be a default dashboard already pre-configured with the free metrics enabled for your account. Now, that could be super helpful if, let's say, you just have one account, you can get some real good high-level metrics around your storage based on bucket. You can go into that dashboard and really quickly see total storage across all your buckets. You can see trend analysis with, day-by-day, week-by-week change comparison, how are things growing. There was one thing that I saw that I was really blown away by because this is something we deal with a lot is they have broken the metrics out in kind of a high-level summary, focusing on data protection, like being able to see data percentage replicated or encrypted, but also based on cost efficiency, too, being able to see if you have versioning enabled, obviously, there's a cost for that. 

How many old versions of this thing do you have, but also incomplete multipart uploads? That is potentially a large and in many ways, super hidden cost for some users of Amazon S3. If you are uploading a multipart file, and it fails, it lives in this purgatory, storage purgatory, where you're charged for it, but you may not see it in an obvious way. 

Jesse: And we see that with a lot of our clients who have multipart uploads and end up with these incomplete multipart uploads that just take up space. There's no clear metrics right now, prior to Storage Lens, that say, here's all of this stale multi-part upload usage that you're paying for, that's effectively just taking up wasted space. But now we have metrics for that; now we have information that can clearly tell us where they are, how much space they're taking, and you can actually do something about it.

Pete: Right. Yeah, it gives you this intelligence that you can act upon. To talk about those metrics, since we're kind of on that stage, when ...

Want to give your ears a break and read this as an article? You’re looking for this link.

Sponsors

• New Relic
• Linode

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Links

• Follow Last Week In AWS on Twitter

Transcript
Corey: This episode is sponsored by ExtraHop. ExtraHop provides threat detection and response for the Enterprise (not the starship). On-prem security doesn’t translate well to cloud or multi-cloud environments, and that’s not even counting IoT. ExtraHop automatically discovers everything inside the perimeter, including your cloud workloads and IoT devices, detects these threats up to 35 percent faster, and helps you act immediately. Ask for a free trial of detection and response for AWS today at extrahop.com/trial.

Pete: Hello, and welcome to AWS Morning Brief. I am Pete Cheslock, and I am here yet again with Jesse DeRose. Jesse, welcome back. 

Jesse: Thanks for having me, Pete. 

Pete: But it's not just the two of us. We have a very special guest: we are also joined with one of the newest hires to The Duckbill Group, Amy Negrette. Amy, hello.

Amy: Hello. And one might say the most special of guests; that person would be me.

Pete: The most special of guests. 

Jesse: [laugh].

Pete: Well, we are pleased to have you. So, in honor of Thanksgiving—American Thanksgiving, for anyone outside of the United States, or who doesn't celebrate. But this is the American Thanksgiving holiday week. We wanted to take a little different approach to this week's episode. And Amy, you were the one who kind of came up with this idea, and so that's why we forced you to join us because—

Jesse: One of us. One of us.

Pete: [laugh]. Because you had such a good idea, and we wanted to make sure that we just pulled this together and really did a Thanksgiving theme to this podcast. So, I don't know about either of you, but my family has some very clear requirements about what dishes do and do not constitute Thanksgiving. And you can always expect the turkey and the stuffing. It's just not Thanksgiving without those core components. 

Jesse: But then your cousin's boyfriend shows up with the candied vegetables that nobody asked to be candied. And, you know, you put a little bit on your plate because you want to be nice. You don't want to start World War III in the middle of Thanksgiving dinner. And you say, “Oh, yeah, this is good.” But then you're definitely giving those food scraps to the dog under the table and you don't go back for seconds.

Pete: I mean, a metric ton of sugar is probably the only way to make turnips taste good.

Jesse: Yeah.

Pete: So, with that in mind, we wanted to talk about what AWS services are those core services that you expect the customers kind of using to leverage the cloud, what services would kind of represent a Thanksgiving meal? Which ones constitute the turkey, or the stuffing, or the green bean casserole which, while preparing this, there seem to be some conflicting thoughts about the quality of a green bean casserole.

Jesse: There are some hot takes. Some hot, hot, hot takes in this discussion, putting this list together.

Pete: So, I'll kick us off with an easy, softball one because why not? But it's EC2, right? This is the turkey. It's the main course. And it's also what you'll be eating three to five times a day for every day for the next week or two because you're going to have a lot extra. It's just going to be around for a long time. 

Jesse: Yeah, I feel like EC2 is one that you're going to get in some capacity, anywhere. Whether it is straight-up EC2 instances, whether it is Fargate, ECS, you're going to be using this compute resource in some capacity if you're using AWS. I don't think I know of any AWS customer that is not using some level of compute with EC2. Except for the few people who have managed to move entirely serverless to Lambda, which I am thoroughly impressed if you've been able to do that. 

Pete: So, that is actually a great one which is Amy you do a lot with the serverless community. What do you think Lambda would be as a Thanksgiving side dish?

Amy: It is the canned cranberry sauce because everyone who I hear talk about it they seem to hate it, but I love it. I love not having to work for anything. It tastes the same and the sauce itself tastes like jelly and Lambda packages everything in a way where I don't have to deal with it, and to me that makes everything else super easy.

Pete: I think it's the slow oozing out of the can it does that really kind of makes me not want to like it, and those just too perfect ridges from the form of it. But I don't know what it is about it; when you just slice through that and put it on your plate, so delicious. And don't at me with your fancy homemade cranberry sauce, whatever. None of that can hold a candle. 

So, I actually think Lambda is the special smoked turkey. Because it's a new trend. Lambda being in the new trend, serverless is a new trend. And of course, everyone who is doing a smoked turkey or has a smoker just can't stop talking about it, much like serverless. They just can't stop talking about it.

Jesse: Yeah. I mean, I think that ever since you bought your smoker, you have not stopped telling us all about the meats that you're smoking on a recurring basis.

Pete: I mean, I got a 16-pound turkey for $14, and I got turkey for days. 

Jesse: What I love is that not only do you have a smoker and you talk about it, but you have a monitoring system that you set up so that you can monitor the temperature of the smoker at any given time. 

Pete: I'm a bit of a Luddite at home. I don't like IoT powered anything because I think they're all generally terrible, but for some reason, yeah, my smoker has a little whatever, cellular—powered, connects to my wifi, but I can get to it from the app on my cell phone, can check the temperature of the turkey, out of the store running errands. “Oh, got to get home soon, my turkey’s almost done.”

Jesse: Okay, I’ve got another easy one for us. S3 is your mashed potatoes. It's good, it's on everyone's plate, there appears to always be an infinite amount of it. Everybody's going to want some. And most importantly, if you leave a bucket of it open overnight, you're going to regret it. 

Pete: Yeah, that's going to turn to glue pretty fast, not Amazon Glue, which actually if we are going to talk about Amazon Glue and Lake Formation, and that weird amalgamation of Amazon services, we actually have one for that. This is something called the piecaken, which I had never heard about until I saw an Instagram ad because that's a thing. But a piecaken is a pecan pie—pecan or pecan? Let's not, do that.

Jesse: Oh, God, don't start.

Pete: Okay. Pumpkin pie, spice cake, and an apple pie filling. It's like three pies stacked into a cake. And that's what I think of when I think about the whole Lake Formation/Glue setup when you're trying to query or analyze your data lake. 

Jesse: Yeah, my arteries just clogged ...

Want to give your ears a break and read this as an article? You’re looking for this link.

Sponsors

• Gravitational
• Linode

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill