AppSync Events, part 4b: load, contention, and the aggregate-mode payoff

Part 4b of a five-part series on AWS AppSync Events, and the last. Part 1 mapped the service; part 2 designed Bid-Stream; part 3 built the instrument; part 4a measured the publisher-visible failure surface. This part puts the same instrument under load.

Part 4a settled the failure half of the trade: the ack carries per-event error detail the docs never mention, whole-request failures collapse into an indistinguishable 502, and EVENT-mode’s retry machinery only sees failures your resolver doesn’t swallow. All of that is about what happens when something breaks.

This part is about what happens when nothing breaks and the rate keeps climbing. Three questions carry it: where the latency knee sits and what the front door does past it, whether the ceiling on a hot auction is the DynamoDB item or the table, and whether part 2’s aggregate-mode short-circuit actually buys what its design argued it would.

Smoke scale first

What follows is deliberately labeled smoke-scale: small verification runs proving each load tool works against the deployed stack. They’re real measurements with honest error bars (“one run, one day, one region”); the calibrated experiments are queued behind them.

Measured 2026-07-23, us-east-1:

• Front door, EVENT-mode path (autocannon, 30 s, 50 connections): 11,210 requests at ~374 req/s sustained, zero errors, zero non-2xx, p50 101 ms / p99 467 ms. No throttling anywhere near this rate, which is unsurprising against part 1’s request-token quota (5,000/s by default, 10,000/s in us-east-1 where this ran) with that much headroom.
• Hot-auction contention (artillery, 20 requests × 5-bid batches against one auction): 25 accepted / 75 STALE_BID out of 100 bids, near-adjacent amounts racing on one item, exactly the per-item-mode contention part 2 predicted and its aggregate mode exists to reduce.
• Delivery latency, cold sync path (subscriber fleet, 4 samples): ~1.5 s publish-to-delivery including a cold start on the synchronous path, the REQUEST_RESPONSE tax, previewed.
• Retry drill (15 bids through the part-2 retry client): 5 delivered, 10 undelivered, 0 aborted batches. The 10 are intra-batch race losers, terminal STALE_BIDs that blind retry correctly declines to resurrect.

Four numbers, four working tools, and not one of them answers a question worth asking. That’s the honest status of smoke tests: they prove the harness, not the system. A 30-second run at 374 req/s against a quota five to ten times higher tells you the front door is open, not where it closes.

The experiment programme

Seven calibrated runs, each designed around a single question with a falsifiable answer, and each with its own slot below. The runbook lives at docs/experiments/runbook.md in the repo; the whole programme is budgeted at roughly $5 to $10, which is its own small finding about the cost of this kind of work.

[E-SLOT E1 - pending] Latency knee and error-code distribution as publish rate ramps 10→200 req/s. Run per docs/experiments/runbook.md, then fill.

[E-SLOT E2 - pending] Per-item vs aggregate on a hot auction: acceptance rate, Duration p95, BatchShortCircuits vs transactions - the number that settles part 1’s aggregate-mode argument.

[E-SLOT E3 - pending] One hot auction vs 50-way spread - confirming the ceiling is the item, not the table.

[E-SLOT E4 - pending] Fan-out: delivery latency across two 1,000-subscription topologies (1,000×1 vs 100×10 connections×subscriptions).

[E-SLOT E5 - pending] Throttle injection at reserved concurrency 5: the 502 surface at volume, retry-drill recovery, duplicate count (must be 0).

[E-SLOT E6 - pending] Provisioned concurrency on/off → p99 delta, attributing the 1.5 s smoke number.

[E-SLOT E7 - pending] Reactions storm with a failing handler: publisher latency vs E1, DLQ depth × batch-size accounting.

Two of these are load-bearing for claims made earlier in the series, and it’s worth saying which, because a measurement that can only confirm what you already wrote isn’t worth running. E2 settles part 1’s aggregate-mode argument. Part 1 claimed aggregate mode is a contention lever rather than a throughput lever, and part 2 built a short-circuit that should cut transactions on the hottest item by up to 5×. If BatchShortCircuits climbs and Duration p95 doesn’t improve, the argument was wrong about what it buys. E5 tests part 2’s retry client against the surface part 4a measured, and its duplicate count must be zero. If it isn’t, the client-generated-bidId idempotency story has a hole in it, and that would be the most important negative result in the series.

The trade, stated with data

Synchronous REQUESTRESPONSE buys per-event certainty, measurably including the _why (part 4a), and the bill is that the publisher inherits every downstream limit: Lambda concurrency, the hot item behind it, cold starts in the critical path, and an opaque 502 when any of it saturates. Asynchronous EVENT buys flat publisher latency and broadcast that nothing can gate, and converts failure into an invocation-level, eventually-visible, DLQ-shaped problem, with Lambda’s documented at-least-once caveats riding along.

The smoke numbers preview the shape of the cost: ~1.5 s publish-to-delivery on the cold synchronous path against a front door that shrugs at 374 req/s asynchronously. E6 attributes how much of that 1.5 s is cold start rather than architecture, and E1 finds the rate where the synchronous path stops being comfortable. Until those run, the honest statement is the qualitative one, which is why this section carries a comment telling me to come back to it.

Part 4a re-ranked one item on part 2’s “what we deliberately didn’t build” list and left three standing. E3 and E5 decide those: write-sharding the hot auction item is triggered by the item, not the table, being the ceiling, which is precisely E3’s question; load shedding ahead of DynamoDB is triggered by measured unhealthy saturation, which is E1 and E5. Identity-based subscribe auth is triggered by product requirements rather than numbers, so it will still be standing when this is done.

Cost postmortem

Part 4a reported that everything up to the load work cost cents, with the free tier absorbing most of it. Load testing is where a fan-out-priced service starts charging for what it’s good at: every subscriber in E4’s 1,000-subscription topologies meters its own deliveries, and part 1’s arithmetic (one publish to 10,000 subscribers is 10,001 operations, not 1) is exactly the mechanism. The billing alarm stays at $20.

Where this leaves the series

Five parts, and the shape of the argument was the same at every step: the documentation covers what the service does, and stops at the edges where you actually have to make decisions. Part 1 assembled a pricing model and a quota table from pages that don’t cite each other, and ended with two questions nobody had written down. Part 2 chose a domain where those questions are the main path rather than the edge case. Part 3 built an instrument and pinned every contract the build depended on, including two places where the docs and the source disagree. Part 4a fired it and got answers with dates on them. This part puts numbers on the half that only load can settle.

The findings will age. The request-token quota already moved once between AWS’s announcement and this writing, the Powertools discrepancy will presumably resolve in one direction or the other, and any measured behavior here can change without a changelog entry. That’s not a caveat on the work; it’s the reason the work is shaped the way it is. Every claim carries a date, every probe is a script rather than a memory, and the behaviors this series depends on are pinned by tests that fail when the answer changes.

Which is the part worth stealing, whatever service you’re up against: when the docs go quiet, build the smallest thing that forces an answer, measure it, date it, and pin it. Then publish the null results too.