AppSync Events, part 4a: measuring the undocumented

Part 4a of a five-part series on AWS AppSync Events. Part 1 flagged what’s undocumented; part 2 designed for it; part 3 built the instrument. This part fires it at the failure surface; part 4b fires it under load. Everything below is measured on 2026-07-23 in us-east-1 unless labeled otherwise, and every probe is a script in the repo.

We poisoned the reactions handler, published an event, and watched. The subscriber received it instantly: async mode really is un-gated, exactly as documented. Then we waited for the failure to reach our dead-letter queue: past the first retry window, past the second, five minutes and more. Nothing ever arrived.

That empty queue turned out to be the most instructive measurement in the project, and its explanation is waiting two sections down. First, the method, and the finding that changes how you’d write a publisher.

Methodology, briefly

Every probe in this post except one manual WebSocket session is a script in the repo: scripts/spike.sh runs the fixture capture (S1), the failure-surface matrix (S2), the DLQ probe (S3), and the metrics-dimension check (S4); raw transcripts are checked in alongside the curated findings. Reproducibility is the credibility claim. If you deploy the stack, the same commands should produce the same tables, and I’d genuinely like to hear about it if they don’t.

The discipline from part 1 holds throughout: every claim is documented, measured (with a date, and these are 2026-07-23, us-east-1), or inferred and labeled. AWS can change any measured behavior without notice, which is precisely why the probes are scripts rather than memories.

The publisher-visible failure surface

Part 1’s biggest open question: AWS documents the publish request and not the response body, so what does a publisher actually see when things go wrong? The matrix, measured:

Probe	How	HTTP status	What the publisher sees
Accepted bid	normal publish	200	successful: [{identifier, index}]
Handler rejection (stale)	second, lower bid	200	failed entry with code + message (below)
Handler rejection (invalid)	malformed payload	200	same shape; full Zod issue list in message
Invocation crash	env parse fails at init	502	DependencyFailedException body (below)
Lambda throttled	reserved concurrency 0	502	identical body, indistinguishable from a crash
Handler timeout	fn timeout 1 s	200 (no repro)	see below

The two response bodies worth reading verbatim, first a per-event rejection, then the whole-request failure:

"failed": [{ "identifier": "...", "index": 1, "code": "CustomError",
             "message": "STALE_BID - high bid is 10, bid of 5 loses" }]

{
  "errors": [
    {
      "errorType": "DependencyFailedException",
      "message": "Unable to process request"
    }
  ]
}

Two findings here, one happy and one not.

The happy one is the headline: the handler’s error string reaches the publisher. Every rejected event comes back in failed with a code and message, and the message is byte-for-byte the CODE - message string part 2’s taxonomy was designed to emit. This is documented nowhere. It’s richer than the Amplify client’s own PublishedEvent type, which admits only {identifier, index}. (The code field is always the literal "CustomError"; the taxonomy travels in the message prefix, not the code.) Part 2’s blind retry policy was designed assuming the publisher learns which events failed but not why; measurement says it can learn why, straight from the HTTP response. Blind retry turns out to be stricter than necessary, since a production client can parse the prefix and skip terminal codes entirely. Designed for ignorance, upgraded by measurement: a policy that assumes nothing keeps working when you learn more. This behavior is now regression-pinned by an e2e scenario, so if AWS quietly changes it, a test fails before a reader emails me.

The unhappy one: whole-request failures are opaque. A crashed invocation and a throttled Lambda produce the same HTTP 502 with the same DependencyFailedException body: no per-event detail, no Retry-After, no way to distinguish “your code is broken” from “you’re going too fast.” The practical classification rule that falls out of the matrix is what a publisher should actually implement: 2xx with failed entries → per-event and inspectable; 5xx → whole-batch and blind-retriable with backoff; 400/401/403 → abort. Part 2’s retry client already implements exactly that shape.

And one null result, published on purpose: the timeout row wouldn’t reproduce. With the function’s timeout cut to 1 second, init time doesn’t count against the cap and the warm handler finishes in well under it. A true timeout probe needs a sleep hook the prototype deliberately doesn’t ship. A blank cell honestly labeled beats a plausible guess.

The DLQ that stayed empty

Now the opening mystery. The poison pill’s first version threw inside the Powertools onPublish handler, the obvious place. Watch what each layer does with that: the resolver catches the throw and formats it as a per-item error response, exactly as designed and exactly as part 3 documented. Which means the Lambda invocation returns normally. Which means Lambda records a successful invocation. No function error → no async retries → no on-failure destination → an empty DLQ, forever.

In EVENT mode, a resolver-level throw is invisible to Lambda’s failure machinery. The failure semantics you bought with configureAsyncInvoke, retries and dead-lettering, operate at the invocation level, and a resolver whose job is turning throws into per-item errors is, from that machinery’s point of view, a machine for hiding failures. It’s not a Powertools bug; each layer is doing its documented job. It’s a semantics interaction that nobody documents because it lives between two systems’ documentation.

The rule to carry away: anything you want retried or dead-lettered must fail the invocation, so throw before or outside the resolver. The fixed handler moves the pill into the wrapper, with the rationale as a comment that now reads like a war memorial:

export const handler = async (
  event: unknown,
  context: Context,
): Promise<unknown> => {
  // The poison pill must fail the INVOCATION, not an item: a throw inside the
  // resolver is caught by Powertools and returned as a per-item error, which
  // Lambda counts as SUCCESS — no async retries, no on-failure destination.
  // Spike S3 (2026-07-23) measured exactly that: poisoned in-resolver, DLQ stayed empty.
  if (env.FAIL_MODE === "throw") {
    throw new Error("poison pill: FAIL_MODE=throw");
  }
  return app.resolve(event, context);
};

With the pill relocated, the full path proved out cleanly: the subscriber still receives the event instantly (broadcast un-gated, EVENT mode’s defining sentence, now measured as well as documented), and the DLQ record lands roughly three minutes later, initial invocation plus Lambda’s async retries at ~1 and ~2 minutes, approximateInvokeCount: 3. The e2e scenario measured 171 seconds end to end on its own run.

Reviewing that test surfaced a bonus accounting finding: the DLQ receives one invocation record per failed publish, not per failed event, and the whole batch rides inside the record’s requestPayload. Any alarm or drain job that treats DLQ depth as a failed-event count is wrong by up to 5×. Count the events inside the message bodies, not the messages.

And the sidebar promised at the end of part 3: our first run of the DLQ scenario passed in six seconds, which is physically impossible, since the retry schedule alone takes three minutes. It had consumed a stale DLQ message left by an earlier interrupted run. The mechanism it “verified” was real (that stale record itself showed approximateInvokeCount: 3 and ~3.5 minutes of retry timing), but the test proved nothing about the current run. The fixes: purge the queue, and make the assertion run-scoped by matching the invocation record’s channel to this run’s unique auction id. File under e2e hygiene: a test that can pass on another run’s evidence eventually will.

Small print that took an afternoon

Three findings too small for their own sections and too costly to rediscover:

• The CloudWatch dimension for Event API metrics is EventAPIId, not ApiId, not GraphQL’s GraphQLAPIId. The metrics page documents the metric names and not their dimensions; we got it from list-metrics and the dashboard construct carries the verified constant. Metrics arrive at three granularities: per API, per API + channel namespace, and a region aggregate.
• The fixture capture (S1) confirmed the documented Lambda event shape is accurate, with richer request.headers than documented (the full CloudFront set) and otherwise faithful. Confirming the docs is also a result; now it’s known rather than hoped, and pinned by every fixture-driven unit test.
• The WebSocket handshake behaved exactly as documented: connection_ack with connectionTimeoutMs: 300000 (measured), ka keepalives observed (the 60 s cadence is the documented value). Part 3’s two-subprotocol client needed no fallback.

What the failure surface settles about the trade

Part 2 framed synchronous versus asynchronous as a trade to be measured rather than argued. Half of it is now measured, and it’s the half about failure.

Synchronous REQUESTRESPONSE buys per-event certainty, and it buys more of it than the design assumed: the ack tells you, per bid, accepted or rejected and now measurably _why. The bill is that the publisher inherits every downstream limit, and that when the chain saturates, the signal collapses to an opaque 502 that can’t be told apart from a crash. Asynchronous EVENT buys flat publisher latency and broadcast that nothing can gate, and converts failure into an invocation-level, eventually-visible, DLQ-shaped problem, with Lambda’s documented at-least-once caveats (duplicates possible, drops possible under sustained overload) riding along. The catch is the one this post opened with: that invocation-level machinery only sees failures your resolver lets through.

Measurement also re-ranks part 2’s “what we deliberately didn’t build” list, at least in part. The receipts side-channel, the production pattern for getting rejection detail to publishers, may simply be unnecessary: its trigger was “the ack turns out to be opaque,” and the matrix above measured the opposite. Message-prefix parsing is proven viable today. The remaining three — identity-based auth, write-sharding the hot item, load shedding — have triggers that are all about saturation, so they keep their triggers until there are numbers to test them against. That’s part 4b.

Cost, so far

Everything to this point — all deploys, the spike matrix, six e2e scenarios with reruns — cost cents, and the free tier absorbed most of it. The billing alarm, set at $20 before any load work, never fired. For a service whose pricing model punishes fan-out at scale (part 1’s 10,001-operation example), the failure-surface research was almost free. The expensive thing about AppSync Events is success, not investigation. Load testing is where that stops being true, which is one reason it gets its own part and its own budget.

What this settles

Five answers the documentation doesn’t give, each with a date and a script behind it:

• The publish ack carries your handler’s error string, per event, in failed[].message (measured, undocumented).
• Crashes and throttles are indistinguishable at the publisher: same 502, same body, no retry hint (measured, undocumented).
• EVENT-mode failure semantics live at the invocation level, and a resolver can accidentally hide your failures from them (measured, and the sharpest lesson here).
• The DLQ counts publishes, not events, up to 5× off as a failed-event proxy (measured).
• The CloudWatch dimension is EventAPIId (measured).

None of this required privileged access, just a deployed stack, a shell script, and the discipline to write down what actually happened. That’s the method the whole series is really selling: when the docs go quiet, build the smallest thing that forces an answer, measure it, date it, and pin it with a test so you notice when the answer changes.

What’s left is the other half of the trade, the half about speed and contention: where the latency knee sits, what saturation actually looks like from the outside, and whether part 2’s aggregate-mode short-circuit pays off the way the design argued it would. Part 4b puts the instrument under load.