Last Week In AWS Podcast

Want to give your ears a break and read this as an article? You’re looking for this link.

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Links:

• Unconventional Guide to AWS Cost Management:https://www.duckbillgroup.com/resources/unconventional-guide-to-aws-cost-management/
• Building Successful Communities of Practice: https://www.amazon.com/Building-Successful-Communities-Practice-Webber/dp/095749193X

Transcript

Corey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.

Corey: Ever notice how security tends to be one of those things that isn’t particularly welcoming to folks who don’t already have the word ‘security’ somewhere in their job title? Introducing our fix to that, Meanwhile in Security. To sign up for the newsletter or to find the podcast, visit meanwhileinsecurity.com. Coming soon, from The Duckbill Group.

Pete: Hello, and welcome to the AWS Morning Brief. This is Fridays From The Field, hashtag-triple-F. I am Pete Cheslock.

Jesse: I’m Jesse DeRose, and I have a question: is it hashtag-triple-F, or is it hashtag-F-F-F? Are we spelling out triple F in this hashtag, or is it just literally three Fs?

Pete: The three Fs is a little triggering for me for me, with my high school grades, so let’s just stick to—

Jesse: [laugh].

Pete: —hashtag—

Jesse: —triple-F.

Pete: Triple-F, I think, just has a better flow to it. But that’s a good—it’s a good point in our continued effort to make triple-F—hashtag-triple-f a thing.

Jesse: All of our audience members were really concerned about that one because they’ve been trying to get us trending on Twitter, but they weren’t really sure, was it triple-F. Or was it F-F-F, or was it something in between?

Pete: Exactly. It’s just bad. But we’re going to keep trying at it, and we’ll see what happens. Well, anyway, we are back again to continue our Unconventional Guide to Cost Optimization on AWS with another listener question. And unlike the last time we did listener questions, this question actually came in during our Unofficial Guide, which means we actually have one listener this series. Because we can’t count the last one that was from way before. So, to this one listener, thank you, thank you for listening.

Jesse: Just that one listener. Just you. Thank you.

Pete: Yeah, just you. Everyone else, no, we’re not going to, we’re not going to thank you at all. But if you want to be our second listener, go to lastweekinaws.com/QA and give us a question. What do you want to know more about? 

What can we dive in a lot deeper on any of these topics we’re talking about? It’s complex stuff, and we’re all learning this, we’re all trying to figure out what works best. And not every company is the same. And that’s what I actually love about this question because this question actually came in from someone who didn’t put their name—but that’s okay—they work in the public sector, which is why they didn’t put their name in there. And they had a pretty interesting question. So, Jesse, maybe you can read this off for us and let us know what we’re going to be answering today.

Jesse: Yeah. This question is, “We’re an Azure shop, partly cloud on the way, however, we’re also becoming an Oracle OCI shop”—I’m so sorry—“And an AWS shop, and well, it’s public sector, so one-of-everything cloud provider. How do we convince management that cloud is a different thing than on-prem and needs some kind of cloud team? I dislike the phrase DevOps as a job title, but we need something to change the current model where nearly all of this work is outsourced to a quote-unquote, ‘managed service provider?’” Oof. I have so many feelings.

Pete: I would imagine. I mean, I was immediately—I felt called out, you know? Just @ me next time, public sector coward with the DevOps-as-a-job-title phrase.

Jesse: Yeah.

Pete: They often say that only a DevOps tool, I guess—wait, what’s the term? It’s like, “A DevOps tool would give themselves a DevOps as a job title.” Of course, that’s often said about me because I gave myself a title called ‘DevOps Director’ or ‘Director of DevOps.’ Either way, you phrase it, it’s all pretty bad.

Jesse: Yeah. So, there’s a couple of different questions in this, and we’re going to dive into each of them individually. But really, really quick, I want to talk about multi-cloud because that’s kind of the underlying discussion here; something that is not necessarily the focus, but let’s talk about multi-cloud. Why is multi-cloud a thing? Why is it an important thing that you should be thinking about?

Pete: Multi-cloud is an interesting topic that could go a lot of different ways. And I call multi-cloud a lot different than hybrid cloud. I think most people are probably doing hybrid cloud, meaning you’ve got some data centers—because it takes you years and years and years to move off of those—and you’ve also got cloud workloads, or maybe you’ve got some data centers and you’re bursting up to cloud workloads; that’s pretty cool, too. I think of multi-cloud as individual applications being deployed to the cloud vendor and cloud provider, based on maybe price or features or things like that. And honestly there, a lot of the cloud providers are getting closer in feature sets. 

But for example, I might want to use Lambda, but I may not want to suffer high cost of data transfer. So, can I build an application that leverages Lambda, but maybe leverages the extremely low cost of Oracle’s OCI data transfer? That made the news when Zoom signed that big contract with Oracle, it was largely driven by network data transfer. So, there are some reasons why multi-cloud might be a thing.

Jesse: And we’ve definitely seen multi-cloud in practice with some of our clients. But I also want to call out the caveat that the clients that were doing this were very mature in their cloud cost practices. So, kudos to those clients because they’re doing amazing, amazing work. But it takes time to really build up a mature, scalable, optimized, multi-cloud strategy.

Pete: Yeah, exactly. And I think the biggest challenge is that we see is, on the one hand, if you say to yourself, “I’m going multi-cloud, therefore, I will only consume core primitives like compute, block, store, object store, networking,” even though all the providers will provide you those services, obviously, the APIs to interact with them will be wildly different, but most importantly, the authentication models are going to be wildly different, how you authenticat...

Want to give your ears a break and read this as an article? You’re looking for this link.

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Links:

• Unconventional Guide to AWS Cost Management:https://www.duckbillgroup.com/resources/unconventional-guide-to-aws-cost-management/
• AWS Tagging Best Practices Whitepaper: https://docs.aws.amazon.com/whitepapers/latest/tagging-best-practices/welcome.html

Transcript

Corey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.

Pete: Hello, and welcome to the AWS Morning Brief: Fridays From the Field. I'm Pete Cheslock.

Jesse: I'm Jesse DeRose.

Pete: And we're back again, Jesse. We are back. But really have we gone anywhere to begin with?

Jesse: We've been making our way slowly but surely through this Unconventional Guide. Lots of really interesting recommendations, lots of really interesting feedback from all of you, which we really, really appreciate. We can't wait to dive into some of those ideas deeper in future episodes.

Pete: Yeah. And don't forget, you can give us additional feedback and questions at lastweekinaws.com/QA, feel free to add your name. Or not. Doesn't matter. It can be totally anonymous. That's fine with us. So today, we're talking about a topic that is very near and dear to our hearts. 

Jesse: Yes.

Pete: It is tagging.

Jesse: Yes.

Pete: Tagging your resources in Amazon, or I mean really any cloud provider; any place you can tag something you probably should. And we're going to talk a little bit about strategies for that, how people use their tags, just all the fun things related to it. Tagging, it's easy to do, right, Jesse? You just tag your resources and all your problems go away.

Jesse: Yep. Thanks, everybody, have a good night.

Pete: So yeah, if you've enjoyed this podcast, please go to—no, I’m just kidding. 

Jesse: [laugh].

Pete: Tagging is probably the thing that most companies are doing poorly, simply because it's hard, and it's an afterthought, and if you didn't have a really solid forced strategy to ensure tags and force compliance, you're probably not going back to fix it.

Jesse: Yeah. It's not thought about as something that's a first-class citizen in the cloud world. When you think about the things that are important to your business model, you might think about getting your application out the door and running, maybe talking about business requirements for availability, failover, data retention, but tagging is nowhere on that list. That's not something that I think any organization thinks about as part of an MVP, let alone future iterations of their products.

Pete: Tagging feels much like the same feeling I get when my doctor says that I should eat more veggies. 

Jesse: Oof.

Pete: I know they're good for me; I know we need to do this. They have vitamins, and fiber, and all these wonderful things. But in order to make those veggies something I want to eat, we have to learn to make it more delicious. Personally, I find duck fat works to make them more delicious. I wish we could apply a duck fat strategy to the tagging problem.

Jesse: Yeah, it's not an easy problem to solve. Or rather, I should say it is an easy problem to solve, but it's not something that anybody is quickly incentivized to solve. Tagging, just for the sake of tagging, it doesn't work.

Pete: Yeah, it's that there really are no incentives for it. No good incentives. It's usually because someone came over to your desk and said, “Hey, what's this charge for? And who's using it? And what's the deal with this?” 

And you're going into Cost Explorer, and you're like, “Uh, I don't know. It's in this one account.” And that's as far as you can go to figure out who did what and why that thing is the way it is.

Jesse: Yeah. There are so many different tagging strategies that we've seen. We've seen some clients talk about tagging as a way to potentially penalize engineers who aren't tagging or who are spending too much money. We've seen organizations who are tagging to reward teams that are tagging all their spend or keeping their spend optimized. Across the board, there are just so many different ways to go about this.

Pete: So let's assume you are like most of the companies that we've seen. Definitely not all: there are some rare gems out there that are making tagging a long term and continual process, which we're actually going to talk about in a future episode, how to do that. But let's say you're just looking at your bill, you're looking at your usage, and you're saying to yourself, “Okay. I need to be better at this.” What do they say, “The journey of a thousand miles starts with a single step?” What is that first step?

Jesse: Yeah there's a lot of different ways to go about this. I think there's a couple great places to start. Now, I will say AWS has a thrilling 24-page best practices white paper that we’ll throw a link in the [show notes 00:05:18]. 

Pete: Have you read that, Jesse?

Jesse: I will say that I have read parts of it. I have not read all of it, and so I want to make it very, very clear to all of our listeners, this is not a document that needs to become the holy grail for your organization. I think in the same way that you could read the SRE book from Google and have some good takeaways, you can skim through this white paper, maybe read through a couple of the sections that seem most applicable to your organization, and then start with those ideas, start with those best practices, and then build them over time organically; develop them over time organically.

Pete: I like to read it some nights when I'm just having trouble sleeping, and maybe by page two or three I’m just out.

Jesse: Yeah. There's a lot of content in there talking about what to tag, why to tag. I think the best place for any organization to start is to think about what are the important things that we need to tag. And that's a conversation that's going to involve not just engineers, but also finance, potentially IT, maybe also security teams, depending on how your organization is built. Because ultimately, what you want to do is understand what are the things that my organization cares about when it comes to our cloud usage?&n...

Want to give your ears a break and read this as an article? You’re looking for this link.

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Links:

• Unconventional Guide to AWS Cost Management: https://www.duckbillgroup.com/resources/unconventional-guide-to-aws-cost-management/

Transcript

Corey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.

Corey: Ever notice how security tends to be one of those things that isn’t particularly welcoming to folks who don’t already have the word ‘security’ somewhere in their job title? Introducing our fix to that, Meanwhile in Security. To sign up for the newsletter or to find the podcast, visit meanwhileinsecurity.com. coming soon from The Duckbill Group.

Pete: Hello, and welcome to Fridays From the Field. I'm Pete Cheslock.

Jesse: I'm Jesse DeRose.

Pete: And we're back, again. We're continuing our series, the Unconventional Guide to AWS Cost Management. And as always, if you have questions, as we are going through this series and want to learn more, go to lastweekinaws.com/QA. Thank you to all of those who have already submitted questions. 

Jesse: Yes.

Pete: Really great ones coming in. 

Jesse: Thank you.

Pete: We're going to take a couple of episodes in the future to answer those questions and really dive into them. So, keep them coming. We really love them so far. So Jesse, what are we talking about today?

Jesse: Today, we're going to be talking about one of my favorite topics, which is that humans are the most expensive part of Cloud. 

Pete: Yeah, we hear this quite a bit. I mean, not just in salary, right? This is the line that usually is mentioned when we talk to folks about their Amazon spend. They say, “Well, outside of salary, Amazon is our most expensive bill.” 

Jesse: Yeah.

Pete: That line has been repeated more times than I can count.

Jesse: But what's so fascinating to me is that this really gets at the idea of total cost of ownership. I think that's ultimately what I really want to focus on for just a second. Total cost of ownership is thinking about all of the spend related to your cloud costs. Now, when you think about cloud costs, you will generally think about just the usage that you have within AWS, maybe some discounts from either an EDP or PPAs. But are you thinking about how much time it's taking your engineers to manage all of that usage, manage that infrastructure, manage the deployment pipelines that are living within the cloud? Are you thinking about all of those components and the cost of those components alongside your usage?

Pete: Yeah, exactly. I think engineers are bad at this.

Jesse: Yeah.

Pete: Myself included. But this is something where we want to build things. That's why we're in this industry. And it's fun to build things. Maybe not so much fun to, kind of, ongoing manage those things. Looking at you, Cassandra and Elasticsearch clusters.

Jesse: [laugh]. Yeah, it's this idea that there are definitely opportunities for engineers to spin things up and manage things on their own when you want to build that Kubernetes cluster and learn how to manage a Kubernetes cluster, learn how to build a Kubernetes cluster. That's great. We don't want to stop you from building and learning at all. But when you're building infrastructure for your organization, for your teams, for your products, is it going to be more cost-effective for you to build this solution yourself, or is it going to be more cost-effective for you to leverage existing managed services within the cloud?

Pete: I like to call it operational FOMO, you know, the fear of missing out. And I think a lot of engineers suffer that when it comes to the new hotness, the new stuff. Kubernetes is a great example. I mean, I feel like a lot of those people were also equally like, “OpenStack is going to be the best thing ever.” And then it didn't. 

But I like to think of my time at a previous company where we deployed into the Cloud, specifically Amazon, and there was a fear that was, again, we've mentioned this before, it's an irrational fear about vendor lock-in. And that fear forced us into building forced us only using core primitives: S3, EC2, EBS, really. We really didn't use much more than that. I mean, obviously, the networks and stuff go in there. And the idea was, is that oh, well, we have this portability. 

And we—Duckbill Group, Corey, we've all talked about it, written about this. It's a fallacy. You're locked in for a lot of other reasons that I'm not going to go into right now. But because of that, we became very good at running our own databases and specifically consuming a large amount of time-series data. It was a security event application. 

And so one of the interesting flip sides of this outcome is that we ran our own monitoring infrastructure. I didn't pay for Datadog. They called me every single day and I was like, “My metrics infrastructure cost me $1,000 a month. You're going to charge me $50,000 a month. Even if you discounted that by half, I still am going to pay a lot more.” 

And the reality was, is that we became so good at managing these systems, we didn't need those services. But I always think back at like, at what cost? How much more time could we have invested in the application, the product, how we deployed it, availability, all that stuff, if we hadn't had to invest so much time into running our own Elasticsearch, running our own Mongo, our own Redis, our own Cassandra? We spent a lot of time doing those things.

Jesse: Yeah, there's a lot of opportunities to leverage managed solutions for those things. Because, again, part of it is this idea of your engineers don't have to spend time managing this infrastructure; they can spend time on other things. But also think about what are the other cost components of this architecture that you may be able to leverage by using a native or a managed AWS service? For example, if you look at Amazon Elasticsearch—is it ‘Amazon Elasticsearch?’ Is it—

Pete: I always forget if it's ‘Amazon Elasticsearch’ or ‘AWS Elasticsearch.’ And oftentimes, it doesn't feel like a rhyme or reason why they name it the way they do.

Jesse: Well, let me put it this way. If you look at the managed Elasticsearch service on AWS, you don't end up paying for some of the things that you might pay for if you were managing that infrastructure yourself, like data t...

Want to give your ears a break and read this as an article? You’re looking for this link.

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill