AWS Bites

By AWS BitesTechnology

AWS Bites episodes

May 30, 2024124. S3 Performance
In this episode, we discuss some tips and tricks for optimizing performance when working with Amazon S3 at scale. We start by giving an overview of how S3 works, highlighting the distributed nature of the service and how data is stored redundantly across multiple availability zones for durability. We then dive into specific tips like using multipart uploads and downloads, spreading the load across key namespaces, enabling transfer acceleration, and using S3 byte-range fetches. Overall, we aim to provide developers building S3-intensive applications with practical guidance to squeeze the most performance out of the service.

💰 SPONSORS 💰
AWS Bites is brought to you by fourTheorem an AWS consulting partner with tons of experience with S3. If you need someone to work with to optimise your S3-based workloads, check out at ⁠⁠⁠⁠⁠⁠fourtheorem.com⁠⁠⁠⁠⁠⁠!
In this episode, we mentioned the following resources:
Our previous episode on S3 best practices: https://awsbites.com/83-bucket-list-of-s3-best-practices

“Deep dive on Amazon S3” (re:Invent talk from 2021): https://www.youtube.com/watch?v=FJJxcwSfWYg

More recent re:Invent talk on Amazon S3 (with updated data) - We discovered this one just after the recording: https://youtu.be/sYDJYqvNeXU

Multi-part upload user guide: https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html

Code examples from the SDK (high-level and low-level APIs): https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpu-upload-object.html

Node.js official helper library (@aws-sdk/lib-storage): https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/Package/-aws-sdk-lib-storage/

Example on how you can implement byte-range fetches: https://docs.aws.amazon.com/AmazonS3/latest/userguide/example_s3_Scenario_UsingLargeFiles_section.html

s3-getobject-accelerator library: https://github.com/widdix/s3-getobject-accelerator

aws-c-s3 library: https://github.com/awslabs/aws-c-s3

S3 storage lens: https://aws.amazon.com/s3/storage-lens/

Docs on S3 Transfer Accelerator: https://docs.aws.amazon.com/AmazonS3/latest/userguide/transfer-acceleration-examples.html

Performance Guidelines for Amazon S3: https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-guidelines.html

Performance Design Patterns for Amazon S3: https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-design-patterns.html
Do you have any AWS questions you would like us to address?
Leave a comment here or connect with us on X, formerly Twitter:
- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/eoins⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠
- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/loige⁠⁠⁠⁠
21 min
May 16, 2024123. What do you need to know about DynamoDB?
In this episode, we provide a comprehensive overview of DynamoDB, including how it compares to relational databases, when to use it, how to get started, writing and querying data, secondary indexes, and single table design. We share our experiences using DynamoDB and discuss the pros and cons compared to traditional SQL databases.

💰 SPONSORS 💰
AWS Bites is brought to you by fourTheorem. If you need someone to work with you to build the best-designed, highly available database on AWS, give us a shout. Check us out on ⁠⁠⁠⁠⁠fourtheorem.com⁠⁠⁠⁠⁠!
In this episode, we mentioned the following resources:
Our previous episode "122. Amazing Databases with Aurora": https://awsbites.com/122-amazing-databases-with-aurora/

Configurable Maximum Throughput on On-Demand tables: https://aws.amazon.com/about-aws/whats-new/2024/05/dynamodb-configurable-maximum-throughput-on-demand-tables/

Best practices for designing and using partition keys effectively: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-partition-key-design.html#bp-partition-key-partitions-adaptive

The DynamoDB Book: https://dynamodbbook.com/

Alex DeBrie’s podcast / YouTube (not about DynamoDB per se but still worth a shout!): https://www.youtube.com/@SoftwareHuddle

One of Rick Houlihan’s talks on DynamoDB: https://www.youtube.com/watch?v=xfxBhvGpoa0

Do you have any AWS questions you would like us to address?
Leave a comment here or connect with us on X, formerly Twitter:
- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/eoins⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠
- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/loige⁠⁠⁠⁠
36 min
May 02, 2024122. Amazing Databases with Aurora
In this episode, we provide an overview of Amazon Aurora, a relational database solution on AWS. We discuss its unique capabilities like distinct storage architecture for better performance and faster recovery. We cover concepts like Aurora clusters, reader and writer instances, endpoints, and global databases. We also compare the serverless versions V1 and V2, noting that V2 is more enterprise-ready while V1 scales to zero. We touch on billing and additional features like the data API, RDS query editor, and RDS proxy. Overall, Aurora is powerful and scalable but not trivial to use at global scale. It's best for serious enterprise use cases or variable traffic workloads.

💰 SPONSORS 💰
AWS Bites is brought to you by fourTheorem. If you need someone to work with you to build the best-designed, highly available database on AWS, give us a shout. Check us out on ⁠⁠⁠⁠fourtheorem.com⁠⁠⁠⁠!
In this episode, we mentioned the following resources:
Amazon Relational Database Service (RDS): https://aws.amazon.com/rds/

Amazon RDS Aurora: https://aws.amazon.com/rds/aurora/

RDS Aurora Serverless: https://aws.amazon.com/rds/aurora/serverless/

Cost calculator for cost comparison: https://calculator.aws/#/estimate?id=4f950b71be8b31438dd12c7aebc9beae3f88179e

Cloudonaut podcast and blog post about Aurora Serverless V2: https://cloudonaut.io/review-aurora-serverless-v2/

Do you have any AWS questions you would like us to address?
Leave a comment here or connect with us on X, formerly Twitter:
- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/eoins⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠
- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/loige⁠⁠⁠
29 min
April 18, 2024121. 5 Ways to extend CloudFormation
In this episode, we discuss 5 different ways to extend CloudFormation capabilities beyond what it natively supports. We started with a quick recap of what CloudFormation is and why we might need to extend it. We then covered using custom scripts and templating engines, which can be effective but require extra maintenance. We recommended relying instead on tools like Serverless Framework, SAM, and CDK which generate CloudFormation templates but provide abstractions and syntax improvements. When you need custom resources, CloudFormation macros allow pre-processing templates, while custom resources and the CloudFormation registry allow defining new resource types. We summarized recommendations for when to use each approach based on our experience. Overall, we covered multiple options for extending CloudFormation to support more complex infrastructure needs.

💰 SPONSORS 💰
AWS Bites is brought to you by fourTheorem, an AWS Partner that specialises in modern application architecture and migration. If you are curious to find out more and to work with us, check us out on ⁠⁠⁠fourtheorem.com⁠⁠⁠!
In this episode, we mentioned the following resources:
Ep. 31 "Cloudformation or Terraform": https://awsbites.com/31-cloudformation-or-terraform/

Serverless Framework: https://www.serverless.com/

SAM (Serverless Application Model): https://aws.amazon.com/serverless/sam/

CDK (Cloud Development Kit): https://aws.amazon.com/cdk/

Ep. 119 "The state of AWS 2024 (AnsWeRS community survey commentary)": https://awsbites.com/119-the-state-of-aws-2024-answers-community-survey-commentary/

Ep. 93 "CDK Patterns - The Good, The Bad and The Ugly": https://awsbites.com/93-cdk-patterns-the-good-the-bad-and-the-ugly/

SLIC Watch: https://github.com/fourTheorem/slic-watch

AWS SSO Utils by Ben Kehoe: https://github.com/benkehoe/aws-sso-util

JavaScript library to safely create custom CloudFormaion resources: https://www.npmjs.com/package/safe-cfn-custom-resource

Clouformation CLI: https://github.com/aws-cloudformation/cloudformation-cli

Cloudformation CLI docs: https://docs.aws.amazon.com/cloudformation-cli/latest/userguide/what-is-cloudformation-cli.html

Cloudonaut podcast "3½ ways to workaround missing CloudFormation support": https://cloudonaut.io/three-and-a-half-ways-to-workaround-missing-cloudformation-support/

Do you have any AWS questions you would like us to address?
Leave a comment here or connect with us on X, formerly Twitter:
- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/eoins⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠
- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/loige⁠⁠
31 min
April 04, 2024120. Lambda Best Practices
In this episode, we discuss best practices for working with AWS Lambda. We cover how Lambda functions work under the hood, including cold starts and warm starts. We then explore different invocation types - synchronous, asynchronous, and event-based. For each, we share tips on performance, cost optimization, and monitoring. Other topics include function structure, logging, instrumentation, and security. Throughout the episode, we aim to provide a solid mental model for serverless development and share our experiences to help you build efficient and robust Lambda applications.

💰 SPONSORS 💰
AWS Bites is brought to you by fourTheorem, an AWS Partner that specialises in modern application architecture and migration. We are big fans of serverless and we have worked on quite a few serverless projects even at a massive scale! If you are curious to find out more and to work with us, check us out at ⁠⁠fourtheorem.com⁠⁠!
In this episode, we mentioned the following resources:
Ep. 110 "110. Why should you use Lambda for Machine Learning?": https://awsbites.com/110-why-should-you-use-lambda-for-machine-learning/

Ep. 108 "How to Solve Lambda Python Cold Starts": https://awsbites.com/108-how-to-solve-lambda-python-cold-starts/

Ep. 104 "Explaining Lambda Runtimes": https://awsbites.com/104-explaining-lambda-runtimes/

Ep. 92 "Decomposing the Monolith Lambda": https://awsbites.com/92-decomposing-the-monolith-lambda/

Ep 64 "How do you write Lambda Functions in Rust?": https://awsbites.com/64-how-do-you-write-lambda-functions-in-rust/

Ep 65 "Solving SQS and Lambda concurrency problems": https://awsbites.com/65-solving-sqs-and-lambda-concurrency-problems/

Ep 10 "Lambda or Fargate for containers?": https://awsbites.com/10-lambda-or-fargate-for-containers/

Ep 4. "What language should you use for Lambda?": https://awsbites.com/4-what-language-should-you-use-for-lambda/

Ep 6. "Is AWS Lambda cheap or expensive?": https://awsbites.com/6-is-aws-lambda-cheap-or-expensive/

Article: "What do you need to know about SNS?": https://fourtheorem.com/what-do-you-need-to-know-about-sns/

Article: "What can you do with EventBridge": https://fourtheorem.com/what-can-you-do-with-eventbridge/

Article: "What do you need to know about SQS?": https://fourtheorem.com/what-do-you-need-to-know-about-sqs/

Yan Cui's video about using Lambda Destinations instead of DLQs: https://www.youtube.com/watch?v=hOetf6YN3zo

Profiling functions with AWS Lambda Power Tuning: https://docs.aws.amazon.com/lambda/latest/operatorguide/profile-functions.html

Blog post by Luca Mezzalira about hexagonal architectures for Lambda: https://aws.amazon.com/blogs/compute/developing-evolutionary-architecture-with-aws-lambda/

Ep 41 "How can Middy make writing Lambda functions easier?": https://awsbites.com/41-how-can-middy-make-writing-lambda-functions-easier/

SLIC Watch for automated dashboards and alarms: https://github.com/fourTheorem/slic-watch

Ep 35 "How can you become a Logs Ninja with CloudWatch?": https://awsbites.com/35-how-can-you-become-a-logs-ninja-with-cloudwatch/

Ep 34 "How to get the most out of CloudWatch Alarms?": https://awsbites.com/34-how-to-get-the-most-out-of-cloudwatch-alarms/

Ep 33 "What can you do with CloudWatch metrics?": https://awsbites.com/33-what-can-you-do-with-cloudwatch-metrics/

Do you have any AWS questions you would like us to address?
Leave a comment here or connect with us on X, formerly Twitter:
- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/eoins⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠
- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/loige⁠⁠
27 min
March 22, 2024119. The state of AWS 2024 (AnsWeRS community survey commentary)
In this episode, we provide commentary and analysis on the 2024 AWS Community Survey results. We go through the key findings for each area including infrastructure as code, CI/CD, serverless, containers, NoSQL databases, event services, and AI/ML. While recognizing potential biases, we aim to extract insights from the data and share our perspectives based on experience. Overall, we see increased adoption across many services, though some pain points remain around developer experience. We hope this format provides value to listeners interested in cloud technology trends.

💰 SPONSORS 💰
AWS Bites is brought to you by fourTheorem, an AWS Partner that does CLOUD stuff really well, check us out on ⁠fourtheorem.com⁠!
In this episode, we mentioned the following resources:
The 2024 Answers for AWS Survey results: https://answersforaws.com/2024

"GitHub Actions Feels Bad" by fasterthanlime (video): https://www.youtube.com/watch?v=9qljpi5jiMQ

"Doing serverless with Terraform": https://serverless.tf/

Our event services series (YouTube playlist): https://www.youtube.com/playlist?list=PLAWXFhe0N1vLHkGO1ZIWW_SZpturHBiE_

Our previous episode about machine learning and SageMaker "How to automate transcripts with Amazon Transcribe and OpenAI Whisper": https://awsbites.com/63-how-to-automate-transcripts-with-amazon-transcribe-and-openai-whisper/

Do you have any AWS questions you would like us to address?
Leave a comment here or connect with us on X, formerly Twitter:
- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/eoins⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠
- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/loige⁠⁠
40 min
March 15, 2024118. The landing zone: Managing multiple AWS accounts
In this episode, we provide an introductory overview of AWS's best practices for managing infrastructure using multiple accounts under an organization. We discuss the advantages of this approach and how to get started creating your own multi-account environment, or "landing zone".

💰 SPONSORS 💰
AWS Bites is brought to you by fourTheorem, an AWS Partner that does CLOUD stuff well, including helping you set up your AWS organisation! If that’s something you are looking for, go to fourtheorem.com to read more about us and to get in touch!
In this episode, we mentioned the following resources:
AWS Definition of Landing Zone: https://docs.aws.amazon.com/prescriptive-guidance/latest/strategy-migration/aws-landing-zone.html

Series of articles "Managing AWS accounts like a PRO": https://fourtheorem.com/managing-aws-accounts-part-1/

AWS Organizations service: https://docs.aws.amazon.com/organizations/latest/userguide/orgs_introduction.html

IAM Identity Center service: https://docs.aws.amazon.com/singlesignon/latest/userguide/what-is.html

Control Tower: https://docs.aws.amazon.com/controltower/latest/userguide/what-is-control-tower.html

org-formation: https://github.com/org-formation/org-formation-cli

Our previous episode "AWS Governance and landing zone with Control Tower, OrgFormation and Terraform": https://awsbites.com/96-aws-governance-and-landing-zone-with-control-tower-org-formation-and-terraform

granted.dev: https://granted.dev

AWS SSO util: https://github.com/benkehoe/aws-sso-util

Leapp: https://www.leapp.cloud/

Cloud Glance: https://cloudglance.dev/

Do you have any AWS questions you would like us to address?
Leave a comment here or connect with us on X, formerly Twitter:
- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/eoins⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠
- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/loige⁠⁠
26 min
March 08, 2024117. What do EBS and a jellyfish have in common?
In this episode, we provide an overview of Amazon EBS, which stands for Elastic Block Storage. We explain what block storage is and how EBS provides highly available and high-performance storage volumes that can be attached to EC2 instances. We discuss the various EBS volume types, including GP3, GP2, provisioned IOPS, and HDD volumes, and explain how they differ in performance characteristics like IOPS and throughput. We go over important concepts like IOPS, throughput, and volume types so listeners can make informed decisions when provisioning EBS. We also cover EBS features like snapshots, encryption, direct API access, and ECS integration. Overall, this is a comprehensive guide to understanding EBS and choosing the right options based on your workload needs.

💰 SPONSORS 💰
AWS Bites is brought to you by fourTheorem, an AWS Partner that does CLOUD stuff really well. Go to fourtheorem.com to read about our case studies!
In this episode, we mentioned the following resources:
EBS Official Documentation: https://docs.aws.amazon.com/ebs/latest/userguide/what-is-ebs.html

EBS Direct Access API: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-accessing-snapshot.html

EBS internal configuration is implemented as “millions of tiny databases” (paper): https://www.amazon.science/publications/millions-of-tiny-databases

EBS Pricing examples: https://aws.amazon.com/ebs/pricing/#Pricing_examples

Do you have any AWS questions you would like us to address?
Leave a comment here or connect with us on X, formerly Twitter:
- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/eoins⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠
- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/loige⁠⁠
22 min
March 01, 2024116. What is RAM (Resource Access Manager)?
In this episode, we discuss AWS Resource Access Manager (RAM) and how it can be used to securely share AWS resources like VPC subnets, databases, and SSM parameters across accounts. We explain the benefits of using RAM over other options like resource policies and assumed roles. Some key topics covered include how to get started with RAM, how it works from the resource owner and resource participant side, and common use cases like sharing VPC subnets, Aurora databases, and SSM parameters.

💰 SPONSORS 💰
AWS Bites is brought to you by fourTheorem, the AWS consulting partner with lots of experience with AWS, Serverless, and Lambda. If you are looking for a partner that can help you deliver your next Serverless workload successfully, look no further and reach out to us at ⁠⁠⁠⁠⁠https://fourTheorem.com⁠⁠⁠⁠⁠
In this episode, we mentioned the following resources:
Sharing Aurora Databases with RAM (Conor Maher's article): https://fourtheorem.com/using-aws-resource-access-manager-for-development/

Blog post "VPC Lattice: The Future of AWS Networking Explained": https://fourtheorem.com/vpc-lattice/

Our previous episode dedicated to VPC Lattice: https://awsbites.com/88-what-is-vpc-lattice/

VPC Lattice sample code base: https://github.com/fourTheorem/vpc-lattice-demo

Sharing AWS Systems Manager Parameters official announcement: https://aws.amazon.com/about-aws/whats-new/2024/02/aws-systems-manager-parameter-store-cross-account-sharing/

Official documentation for what can be shared with RAM: https://docs.aws.amazon.com/ram/latest/userguide/shareable.html

Do you have any AWS questions you would like us to address?
Leave a comment here or connect with us on X, formerly Twitter:
- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/eoins⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠
- ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/loige⁠⁠
14 min
February 23, 2024115. What can you do with Permissions Boundaries?
In this episode, we discuss Permission Boundary policies in AWS IAM. A permissions boundary is an advanced feature in which you set the maximum permissions that an identity-based policy can grant to an IAM entity. When you set a permissions boundary for an entity, the entity can perform only the actions allowed by its identity-based policies and its permissions boundaries.
In this episode, we discuss this concept a bit more in detail and we show how it can be used to give freedom to development teams while preventing privilege escalation. We also cover some of the disadvantages that come with using permission boundaries and other things to be aware of. Finally, we will give some practical advice on how to get the best out of Permissions Boundary Policy and get the best out of them.

💰 SPONSORS 💰
AWS Bites is brought to you by fourTheorem, the AWS consulting partner with lots of experience with AWS, Serverless, and Lambda. If you are looking for a partner that can help you deliver your next Serverless workload successfully, look no further and reach out to us at ⁠⁠⁠⁠https://fourTheorem.com⁠⁠⁠⁠
In this episode, we mentioned the following resources:
Episode 112. "What is a Service Control Policy (SCP)?": https://awsbites.com/112-what-is-a-service-control-policy-scp/

IAM Policy Simulator: https://policysim.aws.amazon.com/home/index.jsp?#roles

The famous RSA paper that introduces Alice and Bob in the world of cryptography: https://web.williams.edu/Mathematics/lg5/302/RSA.pdf

A biographical backstory on Alice and Bob: https://urbigenous.net/library/alicebob.html

Do you have any AWS questions you would like us to address?
Leave a comment here or connect with us on X, formerly Twitter:
- ⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/eoins⁠⁠⁠⁠⁠⁠⁠⁠⁠
- ⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/loige⁠⁠
14 min