AWS Bites

65. Solving SQS and Lambda concurrency problems


Listen Later

In this episode of the AWS Bites Podcast, we dive into the serverless pattern of using AWS Lambda together with SQS. We explain the basics of both Lambda and SQS for those who may not be familiar with them. We talk about how we use Lambda, a Function as a Service offering in AWS, to write our own functions and have AWS run them in response to certain events. And we also discuss SQS, a scalable and managed queuing system available on AWS, which we use to offload work to background workers.

We delve into how the two services work together through the use of "Event Source Mapping" in Lambda, which polls our SQS queue and makes synchronous Lambda invocation requests when messages are available. We also mention how this feature provides us with the ability to control batch size and window, as well as specify filters to save execution time and cost. But we also share one of the limitations we faced when using SQS and Lambda together which was the lack of control over concurrency and the potential for excessive throttling.

But recently, AWS has released a new feature called "SQS maximum concurrency support" which allows us to specify a maximum number of invocations for an Event Source Mapping. This solves the problem of excessive throttling and eliminates the need to use reserved concurrency. It also allows for more control over concurrency when using multiple Event Source Mappings with the same function. We explain how this new feature has improved our workflow and made it much more efficient.

💰 SPONSORS 💰

AWS Bites is sponsored by fourTheorem, an AWS Consulting Partner offering training, cloud migration, and modern application architecture.

In this episode, we mentioned the following resources:

  • AWS Lambda
  • Amazon SQS
  • Series of blog posts by Zac Charles covering the original problem and the solution in detail
  • Official AWS blog post with the announcement of the maximum concurrency feature
  • Our previous episode on SQS
  • Our video-series on AWS event services
  • You can listen to AWS Bites wherever you get your podcasts:

    • Apple Podcasts: https://podcasts.apple.com/us/podcast/aws-bites/id1585489017
    • Spotify: https://open.spotify.com/show/3Lh7PzqBFV6yt5WsTAmO5q
    • Google: https://podcasts.google.com/feed/aHR0cHM6Ly9hbmNob3IuZm0vcy82YTMzMTJhMC9wb2RjYXN0L3Jzcw==
    • Breaker: https://www.breaker.audio/aws-bites
    • RSS:  https://anchor.fm/s/6a3312a0/podcast/rss
    • Do you have any AWS questions you would like us to address? Leave a comment here or connect with us on Twitter:

      • https://twitter.com/eoins
      • https://twitter.com/loige
      • #AWS #rust #lambda

        ...more
        View all episodesView all episodes
        Download on the App Store

        AWS BitesBy AWS Bites

        • 4.6
        • 4.6
        • 4.6
        • 4.6
        • 4.6

        4.6

        11 ratings


        More shows like AWS Bites

        View all
        Software Engineering Radio - the podcast for professional software developers by se-radio@computer.org

        Software Engineering Radio - the podcast for professional software developers

        272 Listeners

        The Cloudcast by Massive Studios

        The Cloudcast

        154 Listeners

        a16z Podcast by Andreessen Horowitz

        a16z Podcast

        1,040 Listeners

        Thoughtworks Technology Podcast by Thoughtworks

        Thoughtworks Technology Podcast

        41 Listeners

        Talk Python To Me by Michael Kennedy

        Talk Python To Me

        585 Listeners

        Software Engineering Daily by Software Engineering Daily

        Software Engineering Daily

        624 Listeners

        AWS Podcast by Amazon Web Services

        AWS Podcast

        202 Listeners

        Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

        Super Data Science: ML & AI Podcast with Jon Krohn

        298 Listeners

        Data Engineering Podcast by Tobias Macey

        Data Engineering Podcast

        141 Listeners

        Syntax - Tasty Web Development Treats by Wes Bos & Scott Tolinski - Full Stack JavaScript Web Developers

        Syntax - Tasty Web Development Treats

        990 Listeners

        Kubernetes Podcast from Google by Abdel Sghiouar, Kaslin Fields

        Kubernetes Podcast from Google

        184 Listeners

        The Real Python Podcast by Real Python

        The Real Python Podcast

        138 Listeners

        Last Week in AI by Skynet Today

        Last Week in AI

        298 Listeners

        The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis by Nathaniel Whittemore

        The AI Daily Brief (Formerly The AI Breakdown): Artificial Intelligence News and Analysis

        496 Listeners

        The Pragmatic Engineer by Gergely Orosz

        The Pragmatic Engineer

        62 Listeners