From Spikes to Steady Throughput Using SQS and Lambda
Nov 16, 2025 · 20 min read
Abstract
Waiting a full day for a nightly batch job is no longer practical. In modern data systems, data often needs to be processed as it arrives. When it stays idle, it quickly loses value. Handling a sudden surge of thousands of events or requests per second is not just a coding problem; it's an architectural one. The choices you make around execution, concurrency, and failure handling matter just as much as the code itself. In this blog, I'll walk through how I scaled a Python ETL pipeline on AWS to handle thousands of concurrent executions. Everything in this blog is from production systems I was working on, including the tradeoffs, the mistakes, and the changes that helped to improve uptime and bring latency down. > [!IMPORTANT] I will walk through how to connect Amazon SQS to AWS Lambda and how that connection behaves under heavy load. We will dig into why maximum concurrency matters, how it affects throughput and failure modes, and what happens when you push it too far.
The Architecture: How It All Fits Together
Before diving into the code, let's look at the system architecture to get a better overview. At a bigger production scale, architecture cannot be fixed later. It determines whether the system can handle traffic spikes or starts failing under load. A common starting point is to trigger a Lambda function directly when a lead data log file lands in S3. This can work at low volume, but it falls apart quickly as concurrency increases. Sudden spikes lead to throttling, retries stack up, and failures become harder to understand and control. To avoid this, the pipeline uses a queue-based load leveling pattern, with Amazon SQS sitting between ingestion and processing. SQS acts as a buffer. It absorbs bursts of incoming work and smooths them out so the compute layer does not get overwhelmed. --- The High-Level Flow: 1. Ingestion: Events from your APIs and CRM systems are pushed into an Amazon SQS queue. 2. Buffering: SQS absorbs the burst; whether you send 10 or 10,000 messages per second, it just stores them reliably. 3. Polling: An AWS managed Lambda poller fleet continuously checks the queue for work. 4. Scaling: As the backlog grows, AWS automatically spins out more Lambda workers, scaling up to the configured cap (e.g., 5,000 concurrent executions for this case). 5. Processing: Each Lambda invocation grabs a small batch (typically 10 messages), processes them, and writes the output to your database. > [!TIP] Producers can push messages into the queue as fast as needed, without worrying about how quickly those messages are processed. This separation lets ingestion scale independently from processing.
The Concurrency Equation: Why 'Fast' Isn't Enough
To hit 5,000 concurrent executions, you need to understand the math behind the madness. It's governed by Little's Law: If your ETL process takes 2 seconds to run, and you want to process 2,500 events per second, you need 5,000 concurrent Lambdas. --- Why I Put SQS in the Middle At first, it is tempting to trigger Lambda directly for every event. But when a spike hits, you get the classic thundering herd problem. Thousands of invocations land at once, AWS allows some immediately, and the rest get throttled. That creates delays and retries right when you are already under pressure. SQS fixed that for me. Instead of forcing Lambda to take the spike, SQS holds the extra work and lets Lambda process it as capacity becomes available. The queue smooths out bursts and keeps the system stable. --- The Setting That Matters Most To actually reach 5,000 concurrency, I had to raise the limit in my account. > [!WARNING] AWS often starts you at around 1,000 concurrent executions, which is not enough for this setup. I requested a Service Quota increase for Lambda concurrent executions, and that removed the ceiling that was holding the system back. That one change turned a fragile system into one I could actually control under load.
SQS & Lambda: The Integration Details
This is where most people get it wrong. Connecting SQS to Lambda is easy; tuning it for speed and reliability is hard. --- Maximum Concurrency (The Safety Valve) There is a critical difference between Reserved Concurrency and Maximum Concurrency. | Type | Behavior | |------|----------| | Reserved Concurrency | Locks a chunk of your account's capacity for a specific function. It guarantees resources but can block other functions. | | Maximum Concurrency | Tells the SQS poller: "Do not ask for more than 5,000 Lambdas at once." If the queue fills up, the poller simply waits. It stops the "Throttle → Retry → Dead Letter Queue" loop that causes data loss. | --- Batch Size & Windows (The Efficiency Knob) For ETL, you rarely want to process one record at a time. - Batch Size = 10: Processing 10 records in one function invocation is vastly more efficient than spinning up 10 separate functions. It amortizes the "Cold Start" time across 10 items. - Batch Window = 1-2 Seconds: This tells Lambda, "Wait up to 2 seconds to see if we can fill a full batch of 10." It adds a tiny bit of latency but massively increases throughput. --- Handling Failures (Partial Batch Response) Imagine a batch of 10 messages. 9 are good, 1 is bad (malformed JSON). If your function crashes, all 10 items are returned to the queue. The 9 good ones get processed again (duplicates), and the bad ones crash it again. The Fix: Use ReportBatchItemFailures. Your Python code catches the error for the bad record, processes the 9 good ones, and tells SQS: "I finished these 9, but please retry just this 1 specific message ID."
Security: The IAM Backbone
I learned pretty quickly that I cannot just deploy this and hope it behaves as expected. In AWS, the default security roles are simple—if I do not explicitly allow an action, it is denied. Security isn't just a formality here; it's an essential component to make sure the Lambda functions only have permission to do the exact work intended, and nothing more. > [!CAUTION] The worst nightmare for a developer is a bug or bad input turning into a function that suddenly has the power to wipe out queues, tables, or anything else in my account. --- The Execution Role: Lambda's ID Card Think of the Execution Role as the ID badge your Lambda wears. When it tries to grab a message from SQS or write a log, AWS checks the badge. --- The "Least Privilege" Rule It is tempting to give your function AdministratorAccess just to get it working. Don't. If a bad actor injects code into your ETL pipeline, they now have admin keys to your kingdom. We use Least Privilege. We grant only what is needed: | Action | Permission | |--------|------------| | "Pick up the phone" | sqs:ReceiveMessage | | "Hang up" | sqs:DeleteMessage | | "Read the caller ID" | sqs:GetQueueAttributes | | "Write in the diary" | logs:PutLogEvents | --- The Terraform Setup for IAM Here is how you write a secure "handshake" in Terraform. Notice we limit the permission to only the specific queue we are using (resources = [awssqsqueue.etlqueue.arn]), not every queue in the account.
The Compute Layer: Docker & Python
Why use Docker for Lambda? Simple: Size. A standard Zip file on Lambda is limited to 250MB. Modern data libraries like pandas, NumPy, and scikit-learn are heavy. If you try to zip them, you'll hit the limit immediately. Lambda supports Container Images up to 10GB. This allows you to package complex ETL logic, machine learning models, and heavy drivers without breaking a sweat. --- Optimizing for Speed (The 47% Latency Reduction) To get the performance improvements, you need to optimize the Dockerfile using the Multi-Stage Build Pattern. This is like cooking in a messy kitchen but serving on a clean plate. You use a "Builder" stage to compile everything, and then copy only the necessary files to the final "Runtime" stage. > [!TIP] Copy your application code LAST in the Dockerfile. This maximizes Docker layer caching—your dependencies won't rebuild every time you change a line of Python.
Infrastructure as Code: The Full Terraform Blueprint
Here is the complete Terraform configuration that ties SQS, Lambda, and the scaling configurations together. This automates the deployment so you aren't clicking buttons in the console. > [!IMPORTANT] The scalingconfig.maximumconcurrency setting is the key to preventing throttle storms. Combined with ReportBatchItemFailures, you get a resilient, high-throughput pipeline.
Conclusion
Scaling to 5,000 concurrent executions isn't magic; it's engineering. By decoupling your ingestion with SQS, optimizing your runtime with Docker, securing your perimeter with granular IAM Roles, and managing it all with Terraform, you turn a chaotic data stream into a reliable, high-performance pipeline. --- Key Takeaways: | Component | Best Practice | |-----------|---------------| | Buffering | Use SQS as a shock absorber between data sources and Lambda | | Scaling | Set Maximum Concurrency to prevent thundering herd | | Efficiency | Batch Size of 10 + 2s Batch Window for optimal throughput | | Reliability | Enable ReportBatchItemFailures to prevent duplicate processing | | Security | Least Privilege IAM—only grant what's needed | | Packaging | Docker multi-stage builds for <10s cold starts | --- The results speak for themselves: 99.9% uptime and massive latency reductions. This is the blueprint for modern, high-scale data engineering. > [!NOTE] Works cited: > - Understanding AWS Lambda Concurrency > - Introducing Maximum Concurrency for SQS Event Sources > - The Hidden AWS Scaling Secret That Every Cloud Architect Should Know > - AWS Lambda Integration Patterns with SQS and SNS > - Benefits of Partial Batch Responses for SQS > - The Case for Containers on Lambda > - AWS Lambda Performance Optimization - Lumigo > - Amazon SQS FIFO Queue and Lambda Concurrency Behavior