AWS Bites

In the age of distributed systems we produce tons and tons of logs. This is especially true for AWS when using CloudWatch logs. So how do we make sense of all these logs and how can we find useful information in them?

In this episode we talk all about logs on AWS and we discuss the main concepts in CloudWatch for logs like Log Groups and Log Streams. We discuss how you can consume logs and how this used to be a big pain point with AWS CloudWatch logs and how now things are a lot better thanks to a relatively new feature called Log Insights.

Finally we discuss some best practices that you should consider when thinking about logs for your distributed cloud applications.

In this episode, we mentioned the following resources:

- Our previous episode on CloudWatch alarms: https://www.youtube.com/watch?v=rk4QMJf6R4U

- Analyzing log data with CloudWatch Logs Insights: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html

- CloudWatch logs insights query syntax: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html

- Pino logger for Node.js: https://getpino.io

This episode is also available on YouTube: https://www.youtube.com/AWSBites

You can listen to AWS Bites wherever you get your podcasts:

- Apple Podcasts: https://podcasts.apple.com/us/podcast/aws-bites/id1585489017

- Spotify: https://open.spotify.com/show/3Lh7PzqBFV6yt5WsTAmO5q

- Google: https://podcasts.google.com/feed/aHR0cHM6Ly9hbmNob3IuZm0vcy82YTMzMTJhMC9wb2RjYXN0L3Jzcw==

- Breaker: https://www.breaker.audio/aws-bites

- RSS:

Do you have any AWS questions you would like us to address?

#aws #logs #cloudwatch

CloudWatch is a great service for metrics. You get tons of metrics out of the box and you can also create your custom ones. One of the most important things you can do with metrics is to create alarms, so how do we get the most out of CloudWatch alarms?

In this episode we share our insights and cover the different types of alarms that exist, how to create an alarm, what to do when an alarm is triggered, a few examples of useful alarms and some of the drawbacks of CloudWatch alarms and how to overcome them.

In this episode, we mentioned the following resources:

- Our previous episode on CloudWatch metrics: https://www.youtube.com/watch?v=vwo2jXfyooQ

- SLIC Watch, a serverless framework plugin that generates sensible alarms and dashboard automatically: https://fth.link/slic-watch

This episode is also available on YouTube: https://www.youtube.com/AWSBites

You can listen to AWS Bites wherever you get your podcasts:

- Apple Podcasts: https://podcasts.apple.com/us/podcast/aws-bites/id1585489017

- Spotify: https://open.spotify.com/show/3Lh7PzqBFV6yt5WsTAmO5q

- Google: https://podcasts.google.com/feed/aHR0cHM6Ly9hbmNob3IuZm0vcy82YTMzMTJhMC9wb2RjYXN0L3Jzcw==

- Breaker: https://www.breaker.audio/aws-bites

- RSS: ​​https://anchor.fm/s/6a3312a0/podcast/rss

Do you have any AWS questions you would like us to address?

#aws #alarms #cloudwatch

CloudWatch is the main Observability tool in AWS and it offers a wide range of features: logs, metrics, dashboards, alarms and even events (recently moved into EventBridge).

In this episode we are going to focus on CloudWatch metrics. We are going to discuss the characteristics of metrics in CloudWatch: namespaces, dimensions, units and more. What metrics you get out of the box and how to create your own. How to access and explore metrics.

Finally we will compare CloudWatch to other providers like DataDog, New Relic, Honeycomb and Grafana + Prometheus and try to assess whether CloudWatch is enough or if you need to use other third-party services.

In this episode, we mentioned the following resources:

- How to send Gzipped requests with boto3 (which uses the PutMetricsData API as an example): https://loige.co/how-to-send-gzipped-requests-with-boto3

- CloudWatch service quota: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_limits.html

- CloudWatch metrics stream for DataDog: https://www.datadoghq.com/blog/amazon-cloudwatch-metric-streams-datadog/

This episode is also available on YouTube: https://www.youtube.com/AWSBites

You can listen to AWS Bites wherever you get your podcasts:

- Apple Podcasts: https://podcasts.apple.com/us/podcast/aws-bites/id1585489017

- Spotify: https://open.spotify.com/show/3Lh7PzqBFV6yt5WsTAmO5q

- Google: https://podcasts.google.com/feed/aHR0cHM6Ly9hbmNob3IuZm0vcy82YTMzMTJhMC9wb2RjYXN0L3Jzcw==

- Breaker: https://www.breaker.audio/aws-bites

- RSS: ​​https://anchor.fm/s/6a3312a0/podcast/rss

Do you have any AWS questions you would like us to address?

What are the benefits of Tags? You have probably seen that you can add tags to almost every resource in AWS… but should you really do it? And if you do it, what are the benefits?

In today’s episode Eoin and Luciano cover what tags are, some examples of how to use them and what kind of benefits they can give you and your team. Finally we’ll give you a bunch of ideas on how to build a tagging strategy and get value from tags!

In this episode, we mentioned the following resources:

- Resource group tagging (https://docs.aws.amazon.com/resourcegroupstagging/latest/APIReference/overview.html)

- Controlling access to AWS resources using tags https://docs.aws.amazon.com/IAM/latest/UserGuide/access_tags.html

- Enforce tagging via SCP (Service Control Policies): https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scps_examples_tagging.html#example-require-tag-on-create

- Use AWS Config to create rules that can alarm if some resources are not compliant: https://aws.amazon.com/premiumsupport/knowledge-center/config-resource-non-compliant/

- Example on how to use CloudTrail to automatically tag newly created resources: https://aws.amazon.com/blogs/mt/auto-tag-aws-resources/

- Archived AWS white paper on tagging best practices: https://d1.awsstatic.com/whitepapers/aws-tagging-best-practices.pdf

This episode is also available on YouTube: https://www.youtube.com/AWSBites

You can listen to AWS Bites wherever you get your podcasts:

- Apple Podcasts: https://podcasts.apple.com/us/podcast/aws-bites/id1585489017

- Spotify: https://open.spotify.com/show/3Lh7PzqBFV6yt5WsTAmO5q

- Google: https://podcasts.google.com/feed/aHR0cHM6Ly9hbmNob3IuZm0vcy82YTMzMTJhMC9wb2RjYXN0L3Jzcw==

- Breaker: https://www.breaker.audio/aws-bites

- RSS: ​​https://anchor.fm/s/6a3312a0/podcast/rss

Do you have any AWS questions you would like us to address?

#aws #tags #cloudformation

Should I use CloudFormation or should I use Terraform instead? If you are just starting to do Infrastructure as Code (IaaC) you probably have this question. In this episode we will discuss in detail how these two amazing pieces of technology compare against each other and what their features, weaknesses and strengths are. We will share our opinions based on our experience with these 2 technologies and guess what, for once we have a bit of clash of opinions! Can you guess who is in the Terraform camp and who is in the CloudFormation camp instead?

In this episode, we mentioned the following resources:

- A tutorial on how to create resources conditionally with CDK (and CloudFormation): https://loige.co/create-resources-conditionally-with-cdk

- An article to understand in depth how to use secrets management with SSM and SecretsManager together with CloudFormation: https://dev.to/eoinsha/3-ways-to-read-ssm-parameters-4555

- Ben Kehoe’s tweet about switching from CloudFormation to Terraform: https://twitter.com/ben11kehoe/status/1158758917515763712

- Terraform null resources: https://registry.terraform.io/providers/hashicorp/null/latest/docs/resources/resource

- CloudFormation Macros: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/macros-example.html

- How to workaround missing CloudFormation features (by Cloudonaut): https://cloudonaut.io/three-and-a-half-ways-to-workaround-missing-cloudformation-support/

- Org-formation: https://github.com/org-formation/org-formation-cli

- How to create accounts in an org with Terraform: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/organizations_account

- Control Tower Account Factory for Terraform: https://learn.hashicorp.com/tutorials/terraform/aws-control-tower-aft

- Pulumi: https://www.pulumi.com/

- Cloudonaut’s comparison of CloudFormation with Terraform: https://cloudonaut.io/cloudformation-vs-terraform/

- Cloudonaut’s free CloudFormation templates: https://templates.cloudonaut.io/en/stable/

Do you have any AWS questions you would like us to address?

AWS Lambda just got a big upgrade in ephemeral storage: you can now have up to 10 GB of storage for your /tmp folder! Before this was limited to “only” 512 Mb… But is this really useful? What can we do now that we couldn’t do before? Also, is this going to have an impact on price? And how does it compare with other storage capabilities that are available in Lambda? Eoin and Luciano are on the case to try to find some answers to these compelling questions, for the greater serverless good!

In this episode, we mentioned the following resources:

- Official AWS announcement blog post: https://aws.amazon.com/about-aws/whats-new/2022/03/aws-lambda-configure-ephemeral-storage/

- Will Dady on Twitter about 10GB of ephemeral storage now enabling interesting CI/CD use cases: https://twitter.com/WillDady/status/1507110176209322018

- Yan Cui’s post on Lumigo’s blog “Welcome to 10GB of tmp storage with Lambda”: https://lumigo.io/blog/welcome-to-10gb-of-tmp-storage-with-lambda/

- Lambda cost comparison with ephemeral storage spreadsheet: https://docs.google.com/spreadsheets/d/1_oGjLN0BeRR8CWfgdjeYiIknRTugdmJOhGkAjLCTStw/edit?usp=sharing

This episode is also available on YouTube: https://www.youtube.com/AWSBites

You can listen to AWS Bites wherever you get your podcasts:

- Apple Podcasts: https://podcasts.apple.com/us/podcast/aws-bites/id1585489017

- Spotify: https://open.spotify.com/show/3Lh7PzqBFV6yt5WsTAmO5q

- Google: https://podcasts.google.com/feed/aHR0cHM6Ly9hbmNob3IuZm0vcy82YTMzMTJhMC9wb2RjYXN0L3Jzcw==

- Breaker: https://www.breaker.audio/aws-bites

- RSS: ​​https://anchor.fm/s/6a3312a0/podcast/rss

Do you have any AWS questions you would like us to address?

Eoin and Luciano take you through the ways serverless can give you more security out of the box. We cover the tradeoffs between having more security control and the responsibility that comes with this power. There are always new security challenges so we cover some of the common pitfalls with serverless and AWS security in general. Finally, we share some tips to make your serverless deployments more secure.

In this episode, we mentioned the following resources:

This episode is also available on YouTube: https://www.youtube.com/AWSBites

You can listen to AWS Bites wherever you get your podcasts:

- Apple Podcasts: https://podcasts.apple.com/us/podcast/aws-bites/id1585489017

- Spotify: https://open.spotify.com/show/3Lh7PzqBFV6yt5WsTAmO5q

- Google: https://podcasts.google.com/feed/aHR0cHM6Ly9hbmNob3IuZm0vcy82YTMzMTJhMC9wb2RjYXN0L3Jzcw==

- Breaker: https://www.breaker.audio/aws-bites

- RSS: ​​https://anchor.fm/s/6a3312a0/podcast/rss

Do you have any AWS questions you would like us to address?

Leave a comment here or connect with us on Twitter:

Luciano and Eoin discuss their strategies and ideas to help new team members to start embracing cloud computing and get productive with AWS. What are the main concepts to focus on when bootstrapping this journey, how to make a plan and make sure it’s bespoke to the expectation of the new employee. How to do pairing sessions and make sure we can build hands-on experience. Finally we discuss building troubleshooting skills and make sure we put in place a virtuous cycle that can foster continuous learning.

In this episode, we mentioned the following resources:

- Our previous episode about AWS certifications and learning material: https://www.youtube.com/watch?v=qf0CuUOtPEI

This episode is also available on YouTube: https://www.youtube.com/AWSBites

You can listen to AWS Bites wherever you get your podcasts:

- Apple Podcasts: https://podcasts.apple.com/us/podcast/aws-bites/id1585489017

- Spotify: https://open.spotify.com/show/3Lh7PzqBFV6yt5WsTAmO5q

- Google: https://podcasts.google.com/feed/aHR0cHM6Ly9hbmNob3IuZm0vcy82YTMzMTJhMC9wb2RjYXN0L3Jzcw==

- Breaker: https://www.breaker.audio/aws-bites

- RSS: ​​https://anchor.fm/s/6a3312a0/podcast/rss

Do you have any AWS questions you would like us to address?

Leave a comment here or connect with us on Twitter:

Eoin and Luciano try to answer the question of what AWS accounts do you need for your team and how to organize them. In this episode of AWS bites we discuss common ways to organize AWS accounts and environments from the perspective of a company running production workloads. We try to answer questions like “how many accounts and how many environments?”. We also discuss how you and your team can be more productive by effectively managing AWS accounts and environments. Finally we explore some common security and cost-related tradeoffs that are common when it comes to organizing AWS accounts.

Thanks to David Lynam for suggesting this awesome topic!

In this episode, we mentioned the following resources:

- AWS Account vending machines: https://aws.amazon.com/solutions/implementations/aws-landing-zone/

- Org Formation: https://github.com/org-formation/org-formation-cli

- Terraform for accounts and organizations: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/organizations_account

This episode is also available on YouTube: https://www.youtube.com/AWSBites

You can listen to AWS Bites wherever you get your podcasts:

- Apple Podcasts: https://podcasts.apple.com/us/podcast/aws-bites/id1585489017

- Spotify: https://open.spotify.com/show/3Lh7PzqBFV6yt5WsTAmO5q

- Google: https://podcasts.google.com/feed/aHR0cHM6Ly9hbmNob3IuZm0vcy82YTMzMTJhMC9wb2RjYXN0L3Jzcw==

- Breaker: https://www.breaker.audio/aws-bites

- RSS: ​​https://anchor.fm/s/6a3312a0/podcast/rss

Do you have any AWS questions you would like us to address?

Leave a comment here or connect with us on Twitter:

Luciano and Eoin explore the wonderful world of data streaming using Kafka on AWS. In this episode we focus mainly on Managed Streaming for Kafka (or MSK) and discuss what are the main differences between MSK and Kinesis. We also explore the main features that MSK provides, its scaling characteristics, pricing and, finally, how MSK works in conjunction with other AWS services.

We conclude the episode by providing a decision tree that should help you to decide whether you should use Kinesis or MSK or avoid streaming services entirely in favor of something like SNS or SQS.

In this episode, we mentioned the following resources:

- Our previous episode on Kinesis data streams: https://www.youtube.com/watch?v=u_nR6up4Kvs

- Our series of Event services: https://www.youtube.com/watch?v=CG7uhkKftoY&list=PLAWXFhe0N1vLHkGO1ZIWW_SZpturHBiE_

- AWS MSK sizing spreadsheet: https://dy7oqpxkwhskb.cloudfront.net/MSK_Sizing_Pricing.xlsx

- Should My Startup use Kinesis or MSK? - https://www.youtube.com/watch?v=TJS19EuzH2k

- Intro to MSK (reinvent talk from 2018) - https://www.youtube.com/watch?v=9nKswHsLseY

- Running Apache Kafka on AWS (by Frank Munz) - https://www.youtube.com/watch?v=HtU9pb18g5Q

- Cloudonaut - Kinesis versus MSK - https://www.youtube.com/watch?v=kcBAKz0MPf8

This episode is also available on YouTube: https://www.youtube.com/AWSBites

You can listen to AWS Bites wherever you get your podcasts:

- Apple Podcasts: https://podcasts.apple.com/us/podcast/aws-bites/id1585489017

- Spotify: https://open.spotify.com/show/3Lh7PzqBFV6yt5WsTAmO5q

- Google: https://podcasts.google.com/feed/aHR0cHM6Ly9hbmNob3IuZm0vcy82YTMzMTJhMC9wb2RjYXN0L3Jzcw==

- Breaker: https://www.breaker.audio/aws-bites

- RSS: ​​https://anchor.fm/s/6a3312a0/podcast/rss

Do you have any AWS questions you would like us to address?

Leave a comment here or connect with us on Twitter: