AWS Lambda Concurrency Demystified: MicroVMs, Token Buckets, and Why 1,000 ≠ 1,000
Why This Post Exists
If you have ever opened the AWS Lambda Dashboard and seen this:
Full account concurrency 1,000
Unreserved account concurrency 971
…and wondered exactly what the difference is, why it is not 1,000 and 1,000, and what happens computationally when a function gets throttled — this post is the technical answer.
This is a companion reference to the incident post-mortem
How a Silent Data Product Build Starved 362 Lambda Functions.
That post explains what happened across three throttling incidents. This post explains why the maths made it possible.
Part I: Reading the Dashboard Numbers
The two numbers on the Lambda Dashboard mean different things and are linked by a simple formula.
AWS Lambda Dashboard — Resources for Europe (Ireland)
──────────────────────────────────────────────────────────────────────
Lambda function(s) Full account concurrency Unreserved account
377 1,000 concurrency: 971
──────────────────────────────────────────────────────────────────────
Relationship:
Unreserved = Full account concurrency − Σ(reserved_concurrent_executions)
971 = 1,000 − 29
Full account concurrency (1,000) is the regional soft limit AWS assigns to your account by default. It is a quota — not a physical ceiling on the number of MicroVMs that could exist, but a governance boundary that AWS enforces at the token level. It is raisable via a support ticket at no cost.
Unreserved account concurrency (971) is what remains after subtracting the total reserved concurrency explicitly allocated to individual functions. In this account, functions with reserved_concurrent_executions configured hold a combined total of 29 slots, leaving 971 for all unreserved functions to share.
The important implication: every function with no reserved_concurrent_executions setting competes in that shared 971-slot pool. Under burst load, a single function can — and will — consume all 971 slots if nothing stops it.
The 100-unit floor rule: AWS prevents you from reserving more than
(account limit − 100)in total. So in a 1,000-limit account you cannot reserve more than 900 across all functions. This guarantees at least 100 slots are always available for unreserved functions, preventing a total blackout. The 971 above means only 29 of the reservable 900 have been claimed.
Part II: Lambda Internal Architecture (HLD)
Understanding why throttling behaves the way it does requires a map of what happens inside Lambda when an event arrives from MSK.
Lambda is split into two independent planes: a Control Plane that handles configuration and a Data Plane that handles actual execution.
┌─────────────────────────────────────────────────────────────────────────────┐
│ AWS LAMBDA — TWO-PLANE ARCHITECTURE │
├──────────────────────────────────────┬──────────────────────────────────────┤
│ CONTROL PLANE │ DATA PLANE │
│ │ │
│ • AWS Lambda API │ • Event Source Mapping (ESM) │
│ • Function configuration storage │ • Frontend Service (Invoker) │
│ • IAM authorisation │ • Placement Service │
│ • Deployment management │ • Worker Manager │
│ • Quota management │ • Worker Nodes (Firecracker VMs) │
│ │ │
│ You interact with this when you │ This is what runs your code. │
│ deploy, update, or view config. │ You never touch it directly. │
└──────────────────────────────────────┴──────────────────────────────────────┘
For the throttling story, only the Data Plane matters. Here is the path a single MSK record takes:
MSK BROKER (Apache Kafka)
│
│ Broker exposes consumer group API
│
▼
┌──────────────────────────────────────────────────────────────────────────┐
│ EVENT SOURCE MAPPING (ESM) — AWS-Managed Internal EC2 Fleet │
│ │
│ • Runs a continuous polling loop against your MSK brokers │
│ • One ESM consumer thread per topic partition (3 partitions = 3 threads)│
│ • Collects records into a batch (up to batch_size) │
│ • Tracks consumer group offsets (committed AFTER successful invocation) │
│ • If invoke is throttled → does NOT commit offset → retries batch │
└──────────────────────────────────────────────────────────────────────────┘
│
│ HTTP POST (synchronous invoke, carries the record batch)
▼
┌──────────────────────────────────────────────────────────────────────────┐
│ FRONTEND SERVICE (Invoker) — The Concurrency Gate │
│ │
│ • Entry point to the Lambda Data Plane │
│ • Checks the token bucket for the calling function │
│ • If token available → acquire token, forward to Placement Service │
│ • If token unavailable → return HTTP 429 (TooManyRequests) to ESM │
│ │
│ ★ This is the exact point where throttles are counted and where │
│ "concurrency starvation" manifests. │
└──────────────────────────────────────────────────────────────────────────┘
│
│ Forward (only if token acquired)
▼
┌──────────────────────────────────────────────────────────────────────────┐
│ PLACEMENT SERVICE + WORKER MANAGER │
│ │
│ Worker Manager: Tracks which physical Worker Nodes have warm MicroVMs │
│ Placement Service: Assigns this invocation to a specific Worker Node │
│ │
│ Decision tree: │
│ 1. Is there a warm (idle) MicroVM for this function? │
│ → YES: route directly, near-zero cold start │
│ 2. Is there a warm Worker Node with capacity for a new MicroVM? │
│ → YES: spin up new MicroVM (~150ms cold start) │
│ 3. No capacity anywhere → cannot place │
│ → Placement fails → Frontend releases token → ESM gets 429 │
└──────────────────────────────────────────────────────────────────────────┘
│
│ Routed to a specific physical host
▼
┌──────────────────────────────────────────────────────────────────────────┐
│ WORKER NODE (Bare-Metal EC2 + Firecracker Hypervisor) │
│ │
│ Physical EC2 instance running the Firecracker hypervisor. │
│ Firecracker can launch extremely lightweight Virtual Machines │
│ (MicroVMs) in ~125ms with a 5 MB memory footprint. │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ MicroVM 1 │ │ MicroVM 2 │ │ MicroVM 3 │ ... │
│ │ Function A │ │ Function A │ │ Function B │ │
│ │ (busy) │ │ (warm/idle) │ │ (busy) │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │
│ CRITICAL RULE: One MicroVM processes exactly one invocation at a time. │
│ Concurrency = the number of MicroVMs currently executing your code. │
└──────────────────────────────────────────────────────────────────────────┘
Part III: The Concurrency Token Bucket (LLD)
What "concurrency" actually measures
When AWS says a function is running at "3 concurrent executions", that means 3 separate MicroVMs are simultaneously executing the function's handler code. Each MicroVM is a single-threaded executor. Concurrency is not a CPU or memory metric — it is a count of simultaneously active MicroVMs.
The token bucket mechanism
Concurrency in Lambda is enforced at the Frontend Service via a distributed lease system — conceptually a token bucket. Your AWS account in a region is assigned a total token count equal to the account limit.
TOKEN BUCKET — How concurrency is tracked
──────────────────────────────────────────────────────────────────────────
Account token bucket (eu-west-1): 1,000 tokens total
┌─────────────────────────────────────────────────────────────────┐
│ Reserved partition (29 tokens — explicitly allocated) │
│ ┌────────────┐ ┌────────────┐ ┌─────────────┐ │
│ │ Function A │ │ Function B │ │ Function C │ │
│ │ rsv = 10 │ │ rsv = 14 │ │ rsv = 5 │ │
│ └────────────┘ └────────────┘ └─────────────┘ │
├─────────────────────────────────────────────────────────────────┤
│ Unreserved partition (971 tokens — shared by all other fns) │
│ │
│ ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 971 available │
│ │
│ During burst (80+ functions all scaling simultaneously): │
│ ████████████████████████████████████████████ ~841 consumed │
│ ░░░░ ~130 remaining — severe contention │
└─────────────────────────────────────────────────────────────────┘
The exact formula
Unreserved pool = Account limit − Σ(reserved_concurrent_executions across all functions)
Example (this account):
971 = 1,000 − 29
Effective tokens available to any unreserved function during burst:
≤ 971 (the entire unreserved pool can be consumed by a single function
with no reserved ceiling — this is the noisy neighbour vulnerability)
What happens when the pool empties
When the Frontend Service receives an invoke request for an unreserved function and the pool is at zero:
THROTTLE SEQUENCE — Step by Step
──────────────────────────────────────────────────────────────────────────
[MSK batch ready]
│
▼
ESM poller → POST /invoke → Frontend Service
│
│ Token bucket check:
│ available = 0
│
▼
HTTP 429 TooManyRequests
│
▼
ESM catches 429
Does NOT commit Kafka offset (record is NOT lost — it stays in the topic)
Backs off (exponential, capped at ~5 min)
Retries the same batch
│
(waits for tokens to free up)
│
▼
Tokens released (other functions finish their executions)
│
▼
ESM retry succeeds → batch processed → offset committed → data delivered
──────────────────────────────────────────────────────────────────────────
Net result: zero data loss, but processing delay = duration of token starvation
This is why the three incidents in the companion post resulted in zero Lambda errors alongside thousands of throttles. Kafka's at-least-once delivery guarantee, combined with ESM's retry-on-throttle behaviour, ensured every message was eventually processed. The only cost was latency.
Part IV: The Multi-Environment Shared Account Problem
Why the default 1,000 limit fails in enterprise platforms
Most enterprise data platforms run DEV, QA, and UAT in a single AWS account for cost reasons. This creates a hidden coupling that only manifests under burst conditions.
SINGLE SHARED ACCOUNT — Three Environments, One Pool
──────────────────────────────────────────────────────────────────────────
Account token bucket: 1,000
DEV environment: ~120 Lambda functions (uses unreserved pool)
QA environment: ~80 Lambda functions (uses unreserved pool)
UAT environment: ~80 Lambda functions (uses unreserved pool)
─────────────────────────────────────────────────────────────────
Total unreserved demand at peak: 280 functions, each wanting 1–3 tokens
Normal operation:
280 functions × avg 1.5 concurrent = ~420 tokens used → 580 available ✓
Burst operation (new pipeline running historical data load in DEV):
1 function in DEV: 841 concurrent tokens consumed
280 other functions: competing for remaining 159 tokens
Result: throttle cascade across UAT pipelines ✗
Why increasing to 3,000 is necessary (the demand calculation)
The default 1,000 limit was designed for single-environment use. In a multi-environment shared account:
DEMAND ANALYSIS — Why 1,000 Is Insufficient
──────────────────────────────────────────────────────────────────────────
Environment demand at normal operating peak:
UAT functions: 30 pipelines × avg 3 concurrent = 90 tokens
QA functions: 30 pipelines × avg 3 concurrent = 90 tokens
DEV functions: 30 pipelines × avg 3 concurrent = 90 tokens
Burst headroom: single pipeline running data load = 500 tokens
────────────────────────────────────────────────────────────────
Total demand: 770 tokens
With 1,000 limit: 770 demand vs 971 available → fine under normal load
841 dev burst alone → leaves only 159 for everything else → cascade
With 3,000 limit: 841 dev burst → leaves 2,159 for everything else → no cascade
The quota increase from 1,000 to 3,000 is not about running more Lambda functions simultaneously — it is about providing burst headroom so a development activity cannot starve operational environments.
Part V: reserved_concurrent_executions as a Bulkhead
The noisy neighbour anti-pattern
Expanding the account limit from 1,000 to 3,000 expands the shared pool but does not eliminate the shared pool. Any unreserved function can still consume all 2,900 available unreserved tokens if it scales aggressively enough.
This is the noisy neighbour anti-pattern: a poorly behaved workload in one part of the system degrades completely unrelated workloads elsewhere because they share a single resource boundary.
How reserved concurrency breaks the coupling
When you set reserved_concurrent_executions = N on a function, the token bucket is repartitioned:
EFFECT OF reserved_concurrent_executions = 50 on a pipeline function
──────────────────────────────────────────────────────────────────────────
BEFORE (unreserved):
┌────────────────────────────────────────────────────────────────────┐
│ Unreserved pool: 971 tokens │
│ ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ All shared │
│ Function "ops-logistics-transformer" draws from this same pool │
│ → No guarantee it gets tokens when pool is depleted │
│ → No ceiling preventing it from consuming the entire pool │
└────────────────────────────────────────────────────────────────────┘
AFTER (reserved_concurrent_executions = 50):
┌────────────────────────────────────────────────────────────────────┐
│ Reserved partition (50 tokens — exclusively for this function): │
│ ██████████████████████████████████████████████████ 50 reserved │
│ → Guaranteed: always available regardless of account burst │
│ → Capped: function cannot consume more than 50 tokens ever │
├────────────────────────────────────────────────────────────────────┤
│ Unreserved pool: 921 tokens (971 − 50) │
│ ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ Remaining shared │
└────────────────────────────────────────────────────────────────────┘
Reserved concurrency does two jobs simultaneously:
| Role | Behaviour | Prevents |
|---|---|---|
| Guarantee (floor) | These N tokens are always reserved for this function — no other function can consume them | Starvation under shared-pool exhaustion |
| Ceiling (cap) | This function can never consume more than N tokens | The function itself becoming a noisy neighbour |
The bulkhead design pattern
In resilience engineering, a bulkhead is a structural partition that contains a failure — if one compartment floods, the others are isolated. Reserved concurrency is the Lambda implementation of this pattern.
BULKHEADED ARCHITECTURE — Each Critical Function Gets Its Own Partition
──────────────────────────────────────────────────────────────────────────
Account limit (after quota increase): 3,000 tokens
┌────────────────────────────────────────────────────────────────────────┐
│ ops-logistics-transformer │ 50 reserved │ Isolated from DEV burst│
│ ground-asset-raw-processed │ 30 reserved │ Isolated from DEV burst│
│ ground-asset-proc-curated │ 30 reserved │ Isolated from DEV burst│
│ ground-asset-curated-sns │ 30 reserved │ Isolated from DEV burst│
│ airside-raw-processed │ 20 reserved │ Isolated from DEV burst│
│ schedule-raw-processed │ 15 reserved │ Isolated from DEV burst│
│ crew-roster-batch-processor │ 250 reserved │ Capped — cannot starve │
│ │ │ the rest of the account│
├───────────────────────────────┴───────────────┴────────────────────────┤
│ Unreserved pool: 3,000 − 425 reserved = 2,575 shared │
│ Available for all other DEV / QA / UAT functions │
└────────────────────────────────────────────────────────────────────────┘
Result: crew-roster-batch-processor running a 500-concurrent historical
load in DEV cannot consume more than 250 tokens. The remaining 2,750+
are available for UAT operational pipelines regardless.
Part VI: MSK Event Source Mapping and the Partition Ceiling
One final constraint that interacts with concurrency and surprises most teams when first encountered.
MSK triggers have a hard concurrency ceiling per function
Lambda's Event Source Mapping for MSK creates exactly one consumer thread per topic partition. This is a hard architectural constraint — there is no parallelization_factor parameter for MSK (unlike Kinesis Data Streams or DynamoDB Streams, which support up to 10×).
MSK TOPIC PARTITION MODEL — Why 3 Partitions = Max 3 Concurrent Executions
──────────────────────────────────────────────────────────────────────────
Topic: raw.ground-asset.telemetry.v2 (3 partitions)
├── Partition 0 → ESM poller thread 0 → max 1 concurrent Lambda execution
├── Partition 1 → ESM poller thread 1 → max 1 concurrent Lambda execution
└── Partition 2 → ESM poller thread 2 → max 1 concurrent Lambda execution
─────────────────────────────────────
Hard ceiling: 3 concurrent, always
Setting reserved_concurrent_executions = 30 does nothing to increase throughput
beyond 3 for MSK-triggered functions. It only guarantees availability.
┌──────────────────────────────────────────────────────────────────┐
│ Trigger type │ parallelization_factor │ Max concurr │
│──────────────────────────────────────────────────────────────────│
│ MSK (Kafka) │ NOT supported │ = partitions│
│ Kinesis Data Streams │ Up to 10× │ shards × 10 │
│ DynamoDB Streams │ Up to 10× │ shards × 10 │
│ SQS │ N/A (batch-based) │ ~1000 max │
└──────────────────────────────────────────────────────────────────┘
Throughput sizing formula for MSK + Lambda
Max throughput (messages/sec) = partition_count × (1,000 / avg_lambda_duration_ms)
Example — ground asset telemetry:
3 partitions × (1,000 / 35ms) = 3 × 28.6 = ~86 msg/sec ceiling
To handle 200 msg/sec peak:
required_partitions = ceil(200 × 35 / 1,000) = ceil(7) = 7 partitions
Safety margin (2×):
recommended_partitions = 14
Warning: Kafka partition count is irreversible — partitions can be
added but never removed. Consumer groups rebalance on partition change
(brief processing pause). This is a maintenance-window operation.
Summary: The Full System in One Diagram
┌─────────────────────────────────────────────────────────────────────────────────┐
│ END-TO-END LAMBDA CONCURRENCY LIFECYCLE │
├─────────────────────────────────────────────────────────────────────────────────┤
│ │
│ MSK Topic (3 partitions) │
│ │ ├── P0 ──► ESM Poller 0 ──┐ │
│ │ ├── P1 ──► ESM Poller 1 ──┤ HTTP POST /invoke (batch) │
│ │ └── P2 ──► ESM Poller 2 ──┘ │ │
│ ▼ │
│ ┌───────────────────────┐ │
│ │ Frontend Service │ │
│ │ Token bucket check │ │
│ │ │ │
│ reserved_concurrent = 30 │ Reserved pool? ─────┤──► acquire reserved │
│ (floor + ceiling) │ │ │ │ token → invoke │
│ └────┤ No → Unreserved pool│ │
│ │ │ │ │
│ │ tokens > 0? ────────┤──► acquire unreserved │
│ │ │ │ token → invoke │
│ │ tokens = 0? ────────┤──► HTTP 429 │
│ └───────────────────────┘ ▼ │
│ ESM backs off │
│ Kafka offset NOT committed│
│ Batch retained in topic │
│ Retry after backoff │
│ → Data preserved, delayed │
│ │
│ On successful invoke: │
│ Placement Service ──► Worker Manager ──► Firecracker Worker Node │
│ │ │
│ ┌─────────────────┐ │
│ │ MicroVM │ │
│ │ (one per │ │
│ │ concurrent │ │
│ │ execution) │ │
│ │ executes code │ │
│ │ returns result│ │
│ └─────────────────┘ │
│ │ │
│ Token released back to pool │
│ Kafka offset committed │
└─────────────────────────────────────────────────────────────────────────────────┘
Quick Reference
The dashboard numbers:
Full account concurrency (1,000) = your account's regional quota
Unreserved concurrency (971) = 1,000 − Σ(reserved allocations) = 1,000 − 29
100-unit floor = AWS prevents total reserved from exceeding (limit − 100)
ensuring at least 100 tokens always exist for unreserved functions
The three levers:
| Lever | What it changes | When to use |
|---|---|---|
| AWS Service Quota increase | Raises account limit | Multi-env shared account; baseline demand exceeds 800+ |
reserved_concurrent_executions |
Isolates a function from the shared pool | Critical production pipelines; batch functions prone to burst |
| MSK partition increase | Raises per-function concurrency ceiling | When throughput capacity (msg/sec) is the bottleneck |
The rule: Quota increase expands the pool. Reserved concurrency isolates within the pool. MSK partition count controls the inlet rate. All three are independent levers — and all three may be needed simultaneously.
This post is a companion to How a Silent Data Product Build Starved 362 Lambda Functions. That post covers the forensic investigation; this one covers the underlying mechanism.
You've reached the end. Explore more engineering notes.
More Engineering Notes
ref›aws-lambda-concurrency-demystified