Last Week In AWS Podcast

Want to give your ears a break and read this as an article? You’re looking for this link. https://www.lastweekinaws.com/blog/Trillion-Dollar-Cloud

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Transcript

Corey: This episode is sponsored in part byLaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visitlaunchdarkly.com and tell them Corey sent you, and watch for the wince.

Jesse: Hello, and welcome to AWS Morning Brief: Fridays From the Field. I’m Jesse DeRose.

Amy: I’m Amy Negrette.

Jesse: This is the podcast within a podcast where we talk about all the ways that we have seen AWS used and abused in the wild, with a healthy dose of complaining about AWS for good measure. I feel like it’s just kind of always necessary. There always has to be just that little bit of something extra; it’s the spice that really makes the dish. Today we’re going to be talking about the ‘Cloud Cost Management Team Starter Kit.’ Now, in a previous episode, we talked about the ‘Cloud Cost Management Starter Kit,’ which was a little bit more generalized, and one of the things that we talked about, ultimately, was building a team that is responsible for some of this work, some of this cloud cost management work.

So today, we’re going to take that one step further; we’re going to talk about all of the things that your cloud cost management team should ultimately be responsible for, what it should look like, how you might want to start building that team within your organization. So, I’m going to kick us off. I think one of the first things that is so, so critical for any team that is going to be doing any work is buy-in at the executive leadership level. You need to make sure that engineering leadership, the C-suite leadership has your back in everything that you’re doing. You need to make sure that the work that you’re doing has been signed off at the highest level so that that leadership can help empower you to do your work.

Amy: And we’ve referenced this before, and really, every time we talk about things like what makes a successful project is that as the one executing that project, you probably need the authority and actionable goals in order to do that, and the leadership is going to be the ones to lay that out for you.

Jesse: Absolutely. If you don’t have the backing of leadership, whether it is your boss, whether it is the C-suite, whether it’s a VP suite, you’re not going to get other people to listen to what you have to say; you’re not going to get other people to, broadly speaking, generally speaking, care about the work that you’re trying to do, the work that you’re trying to incentivize and empower other people in the organization to do.

Amy: And that kind of leads us into the next portion of it where you need to know what the responsibilities are and have that clear delineation so that you understand the things that is expected of you, what the engineering teams, what they’re expected to do, and product teams, and finance teams. Everyone has to have a pretty much fenced-in idea of what they’re allowed to do and what they are expected to deliver, just like in any project.

Jesse: Absolutely. It’s so critical for me to understand what I’m responsible for, you to understand what you’re responsible for. I can’t tell you how many times I’ve been in a meeting where somebody will say something generally like, “We should do X,” and then everyone nods and goes, “Oh, yeah, yeah, yeah. We should do X.” And then everybody leaves the meeting and thinks that somebody else is responsible for it, and nobody’s been clearly assigned that work, or nobody knows that work is ultimately their responsibility.

Amy: And if you don’t assign it, people are going to assume that this is going to be a thing that if they have time to, they’ll get to it. And we harp on it enough that whenever work is not prioritized, it is automatically deprioritized. That’s just the way task lists shake out, especially at the end of sprint meetings.

Jesse: Absolutely. And I think that’s one of the other things that’s so important, too, is that it’s not just about assigning the work, but it’s about making sure that everybody who is involved in the conversation, everybody who’s involved in the work agrees on what those boundaries are, agrees on who is responsible for what actions, more specifically speaking from a task responsibility perspective. Because at the end of the day, I want my team, whether that is my individual team or a cross-functional team, to all be bought into who’s responsible for what parts of the project. We all need to be on the same page in terms of, “Yes, this is my responsibility. This part of the work is my responsibility. I will take ownership over this,” so that we can all help each other.

Get that project goal together. One of the other big ideas that is so critical to starting a cloud cost management team is identifying and socializing your business KPI metrics. Now, this is something that some engineering teams already think about day-to-day. They might have ideas of service-level agreements, metrics, maybe service-level objective metrics, but there might be other business metrics that indirectly—or directly—relate to engineering work. It could be number of users using your SaaS platform, it could be number of API requests, it could be the amount of storage that customers are storing on your platform. You want to identify what these metrics are, and start measuring your cloud spend against these metrics.

Amy: And as far as cost optimization projects go, the KPIs may not line up directly against how many servers you’re standing up, or how many users are coming through. They’ll be very indicative because you are spending money per user and per resource, but perhaps your business goals are different. Maybe you’re not looking at trying to save money, but better understand where that money is going.

Jesse: Absolutely. It’s not just about how many instances are running per hour, it’s not just about how many servers are running per hour, or how many users per server. It’s really about understanding what are the core driving indicators of your business? What are the things that ultimately influence and impact how your workloads, and servers, and API functions, and everything, flow and grow and change over time?

Amy: These metrics also can be influenced by things that are not architecturally specific, like savings plans, or the saving you would get through reservations, or some other contractual deal you get from your provider.

Jesse: Yeah, that’s one of the hard things, too, that we always hear from our clients. There is this idea that they think that they are spending a certain amount of money because they’re getting discounts from savings plans, or from reserved instances or from an enterprise discount program, and maybe their usage is a lot higher than that, but because they get these discounts, they think that they’re actually using a lot less than they actually are. And while this is not something we’re talking about specifically or directly in this conversation, it is something to be mindful of because there definitely can be a difference between your usage and your overall spend if your company is investing in thin...

Want to give your ears a break and read this as an article? You’re looking for this link. https://www.lastweekinaws.com/blog/the-key-to-unlock-the-aws-billing-puzzle-is-architecture

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Links:

• The cloud economist starter kit: https://www.lastweekinaws.com/podcast/aws-morning-brief/cloud-cost-management-starter-kit-2/

Transcript

Corey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.

Jesse: Hello, and welcome to the AWS Morning Brief: Fridays From the Field. I’m Jesse DeRose.

Amy: I’m Amy Negrette.

Jesse: This is the podcast within the podcast where we like to talk about all the ways we’ve seen AWS used and abused in the wild, with a healthy dose of complaining about AWS for good measure. Today, we’re going to be talking about balancing cost optimization work against feature work.

Amy: Buckle up everyone. I’ve got a lot of thoughts about this. Just kidding. It’s just the one: don’t.

Jesse: You heard it here first, folks. Don’t. Amy Negrette just says, “Don’t.”

Amy: Don’t. [laugh].

Jesse: So Amy, does that mean, don’t balance the work?

Amy: More like don’t choose. It’s always hard to make the argument to take an engineer off of feature work. This goes for 
all sorts of support tasks like updates and documentation, and as a group, we figured out that trying to put those off until an engineer has time to do it is not going to be a thing that becomes prioritized, it eventually gets deprioritized, and no one looks at it. And that’s why DocOps is the thing. It’s a process that now gets handled as part of and in parallel with software development.

Jesse: Yeah, I’ve had so many conversations in previous companies that I’ve worked for, where they basically said, “Well, we don’t have time to write documentation.” Or they will say, “The code is the documentation.” And, to their credit, there are a lot of places where the code is very cleanly documented, but if somebody is coming into this information for the first time and they don’t have technical knowledge or they don’t have deep expertise in what you’re looking at, they need documentation that is clear, understandable, and approachable. And it is so difficult to find that balance to actually make sure that that work is part of everything that you do.

Amy: And I think what the industry has decided is that if you make it a requirement for pull requests that if you’re going to make a change, you have to document that change somewhere, and that change if it has any kind of user impact, it will be displayed alongside it. That’s the only way to make it a priority with software. And cost optimization has to be treated in a similar respect.

Jesse: Yeah, so let’s talk about cost optimization as a process. To start, let’s talk about when to do it. Is this something that we do a little bit all the time, or do we do it after everything’s already done?

Amy: I know I just cited CostOps as a good model for this, even though that’s literally what we cannot do. We can’t treat cost optimization as something we do a little bit along the way because, again, speaking as an engineer, if I’m allowed to 
over-optimize or over-engineer something, I’m going to take that opportunity to do that.

Jesse: Absolutely.

Amy: And if we’re going to do project-wide cost optimization, we need to know what usage patterns are, we need to have a full user and business context on how any system is used. So, if we do a little at each step, you get stuck in that micro-optimization cycle and you’re never actually going to understand what the impact of those optimizations were. Or if you spent too much time on one part over-optimizing another part.

Jesse: It’s also really hard if this is a brand new workload that you’ve never run in the cloud before. You don’t necessarily know what the usage is going to be for this workload. Maybe you have an idea of usage patterns based on some modeling that you’ve done or based on other workloads that you’re running, but as a whole, if this is a brand new workload, you may be surprised when you deploy it and find out that it is using twice the amount of resources that you expected, or half the amount of resources that you expected, or that it is using resources and cycles that you didn’t expect.

Amy: Yeah. We’ve all been in the situation, or at least if you work with—especially with consumer software—that, you’re going to run into a situation where the bunch of users are going to do things that you don’t expect to happen within your application, causing the traffic patterns that you predicted to move against the model. To put it kindly. [laugh].

Jesse: Yeah. So, generally speaking, what we’ve seen work the best is making time for cost optimization work maybe a cycle every quarter, to do some analysis work: to look at your dashboards, look at whatever tooling you’re using, whatever metrics you’re collecting, to see what kind of cost optimization opportunities are available to you and to your teams.

Amy: So, that comes down to who’s actually doing this work. Are we going to assign a dedicated engineer to it in order to ensure it gets done? Anyone with the free cycles to do it?

Jesse: See, this is the one that I always love and hate because it’s that idea of if it’s everyone’s responsibility, it’s no one’s responsibility. And I really want everybody to be part of the conversation when it comes to cost optimization and cloud cost management work, but in truth, that’s not the reality; that’s not the way to get this work started. Never depend on free cycles because if you’re just waiting for somebody to have a free cycle, they’re never going to do any work. They’re never going to prioritize cost optimization work until it becomes a big problem because that work is just going to be deprioritized constantly. There’s a number of companies that I worked for in the past who did hackathons, maybe once a quarter or once every year, and those hackathons were super, super fun for a lot of teams, but there was a couple individuals who always picked up feature work as part of the hackathon, thinking, “Oh, well, I didn’t get a chance to work on this because my cycles were focused on something else, so now I’ll get a chance to do this.” No, that’s not what a hackathon is about.

Amy: You don’t hack on your own task list. That’s not how anything works.

Jesse: Exactly. So instead, rather than just relying on somebody to have a free cycle, kind of putting it out there and waiting for somebody to pick up this work, there should be a senior engineer or architect with knowledge of how the system works, to periodically dedicate a sprint to do this analysis work. And when we say knowing how the system works, we’re really talking about that business context that we’ve talked about many, many times before. A...

Want to give your ears a break and read this as an article? You’re looking for this link. https://www.lastweekinaws.com/blog/turn-that-sh—-off

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Links:

• Ted Talk: The Science of Productive Conflict

Transcript

Corey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.

Jesse: Hello, and welcome to AWS Morning Brief: Fridays From the Field. I’m Jesse DeRose.

Amy: I’m Amy Negrette.

Jesse: This is the podcast within a podcast where we talk about all the ways we’ve seen AWS used and abused in the wild, with a healthy dose of complaining about AWS for good measure. Today, we’re going to talk about the people side of technical projects, especially people who might introduce roadblocks for completing technical projects. Now, you might be thinking to yourself, “Jesse, Amy, that sounds like it is not about AWS.” But let me assure you that any project involving AWS is going to involve multiple different personalities approaching the project from different angles, who ultimately all have the same solution in mind, but have different ideas about how to get that problem done, or different ideas about what’s the right thing that ultimately get done to begin with. So today, we want to talk about that: we want to dive into how can you have really rewarding conversations with those folks? How can you better engage people who are intentionally or unintentionally difficult?

Amy: We want to be very clear that we are not trying to come after anyone. Every time that I’ve gotten an engagement, it isn’t because someone means to be difficult, but maybe there’s a project timeline, maybe there’s something else getting in the way of them being able to fully be present for that specific part of the engagement, and maybe that’s just what’s causing friction, causing speed bumps. And we’re all well aware of this. Jobs are hard, and especially this sort of work can be difficult. So, first of all, we totally understand, and this is just more about how to get everyone moving in the same direction at the same pace.

Jesse: Yeah, absolutely. I mean, especially with the pandemic going on right now, everybody’s doing remote work, some people have never actually met their teammates in person and they’re expected to work together efficiently, and quickly, and easily. It’s hard.

Amy: It also doesn’t help that when we do come in, we come in under the context of a cost optimization project, or some other efficiency-type title. And that sounds a little like the Bobs from Office Space, which I bring up a lot, especially during internal meetings. And it makes it sound like we’re going to come in to shake a bunch of things up and look for inefficiencies where there aren’t any, which is truly not the case. And it can cause a lot of insecurity, especially about how someone thinks that they’re doing their job, or that their job is somehow going to be impacted by what our suggestions are. It may not just be us, but it may be another migration consultation suite, where someone’s coming in to change the architecture that they’ve worked on for a long time, and that can put a lot of people in a state of 
unease.

Jesse: And I think it’s also important to note that it’s not just about an external party coming in like Duckbill Group or another external, third-party consulting service, or technical group. It could be an internal separate team. It could be your internal cloud cost management team that is starting conversations with development teams saying, “Hey, I want to better understand how you’re using AWS. I want to understand some of these cost optimization opportunities.” Even in situations like that where all of these conversations are internal within the company, even within teams, there are still multiple different personalities, multiple different people approaching the problem from different angles, and it’s still really, really important to make sure that you approach them collaboratively.

Amy: And ultimately, we wanted to be clear that what we’re going to be talking about is helping people think differently into a growth mindset, and being able to do this work without anyone feeling shame or embarrassment.

Jesse: Yeah. Growth mindset is so critical. It’s something that I love to talk about ad nauseum, and so I won’t dive into it too deeply here, but—

Amy: That’s another episode.

Jesse: [Laugh]. Exactly. Growth mindset is so important for folks in technology teams, especially in today’s technology era where there’s just so much constantly innovating. There’s so much new constantly going on around you, to new technologies, new teams, new ideas, new ways of doing things, new processes, new tools; it’s really important to be open-minded to learning those different things. You don’t have to use every single one of them, but be open-minded to different people approaching problems from different perspectives and different angles.

Amy: Having to face all of this uncertainty will cause some to not be the most cooperative when they have to start reacting into these situations, whether it is an internal change that’s happening, or if it’s an external consulting group; they can start coming back and taking a various sort of stance, and just like being back in middle school, sometimes standing up to a bully is simply how you have to succeed because it’s not about dominating, it’s about compromise and trying to find out what you’re trying to do and find that common ground.

Jesse: Yeah, absolutely. I think that’s the most important part here because when we talk about working with other personalities that are different than yours and having conflict, it’s not about dealing with them; it’s not about overcoming them from the perspective of winning the argument, so to speak. It’s about how do you compromise? How do you effectively find that common ground and move forward together? And sometimes it’s just about sharing context, it’s just about sharing that mental model that you have that might be different than the mental model that this other person has, or maybe the other team has.

Like for example, some teams that we’ve talked to can’t make a cost optimization change due to security, or legal, or product SLA restrictions, but maybe the person who’s coming in from the cloud cost management team or cloud cost management side doesn’t know that because they aren’t as familiar with the product.

Amy: But it can also just be a staffing issue. These projects take work, and if an engineering team is already stressed and stretched to the edge, they’re not going to have the resources, and they don’t want to be the ones to say we simply don’t have the manpower to do this.

Jesse: Yeah, absolutely. And it’s so, so important to be able to identify those bottlenecks or identify those constraints. Because ultimately, if you give a team that already has a ton of things on their roadmap new work and say, “Hey, cost optimizati...

Want to give your ears a break and read this as an article? You’re looking for this link. https://www.lastweekinaws.com/blog/the-17-ways-to-run-containers-on-aws

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill