AppSync Events, part 3: the parts the tutorials skip

Part 3 of a five-part series on AWS AppSync Events. Part 1 mapped the service; part 2 designed Bid-Stream, an auction that makes the undocumented parts unavoidable. This part builds it. Part 4a measures the failure surface; part 4b measures it under load.

The Powertools docs say a per-item UnauthorizedException stops the whole batch. The Powertools source says it becomes an item error like any other. Those can’t both be true, and my handler’s behavior under an attempted-auth-rejection depends on which one wins. So we wrote a test that pins whichever one is true, and it’s the source.

That’s this post in miniature. The distance between “the demo works” and “I trust this” is a set of contracts nobody writes down: the handler’s exact return shapes, the places docs and source disagree, the wire protocol’s undocumented corners, the race conditions in your own test harness. This part is a tour of every contract Bid-Stream had to pin, and the harness engineering that made a live WebSocket service testable, from CDK to Powertools to a hand-rolled client.

The CDK layer: the L2 is ready

The good news first: EventApi in aws-cdk-lib (verified on 2.262.0) covers everything this design needs, including the parts most demos skip. The namespaces from part 2 wire up exactly as the design table promised:

this.api.addChannelNamespace("auction", {
  publishHandlerConfig: {
    dataSource: auctionDs,
    direct: true,
    lambdaInvokeType: appsync.LambdaInvokeType.REQUEST_RESPONSE,
  },
  subscribeHandlerConfig: {
    dataSource: auctionDs,
    direct: true,
    lambdaInvokeType: appsync.LambdaInvokeType.REQUEST_RESPONSE,
  },
});

this.api.addChannelNamespace("reactions", {
  publishHandlerConfig: {
    dataSource: reactionsDs,
    direct: true,
    lambdaInvokeType: appsync.LambdaInvokeType.EVENT,
  },
});

And the async plumbing that most EVENT-mode discussions never reach is two method calls on the reactions function:

this.reactionsFn.configureAsyncInvoke({
  retryAttempts: ASYNC_RETRY_ATTEMPTS, // 2
  onFailure: new destinations.SqsDestination(this.reactionsDlq),
});

Part 1’s source article wished aloud for a native SQS failure target on the service. This is effectively that wish, granted by composition: EVENT mode hands the invocation to Lambda’s standard async machinery, and EventInvokeConfig gives that machinery retries and an on-failure destination. One caveat the docs leave implied rather than stated: the Events guide says its invoke types are “synonymous with” Lambda’s, but never explicitly names destinations as applicable, so an e2e scenario proves the destination actually fires (it does; part 4a has the timings, and a trap).

Three gotchas, one bullet each, all learned by hitting them:

• The L2’s grant* helpers throw unless IAM is among the API’s auth providers. Irrelevant for an API-key-only prototype; surprising if you reach for grantPublish reflexively.
• The auto-created default API key surfaces as Object.values(api.apiKeys)[0], not a named property.
• The auction function’s timeout is 10 s, deliberately far below AppSync’s 30 s request-execution cap, so the function’s own timeout fires first and the failure is attributable.

The stack’s unit tests are CDK-assertions checks on precisely the properties the system’s behavior depends on: each namespace’s Behavior: DIRECT and invoke type, and the seven stack outputs the test tooling consumes. Infra tests as contract tests, pinning what matters, rather than snapshot noise that churns on every synth.

The Powertools contracts, precisely

The resolver’s routing model is small and worth stating exactly, because the handler’s correctness hangs on it: onPublish('/auction/*', handler) registers by channel pattern; wildcards are terminal-only (/ns/* and /*, no mid-path wildcards); most-specific wins; and a publish that matches no registration passes through unchanged, documented, and pinned by a test that feeds the auction handler a /reactions/... event and asserts it comes back untouched. Silent pass-through is exactly the kind of behavior you want a test to scream about if it ever changes.

Per-item mode hands your handler (payload, event, context), and note: no event id. Whatever you return becomes that item’s {id, payload}; whatever you throw becomes {id, error: "Name - message"} while the siblings continue; and the items run concurrently (part 1’s correction, now with consequences, as the e2e section shows). Aggregate mode hands you the whole events array and expects Array<{id, payload?} | {id, error?}> back, verbatim: omit an item to drop it silently, return [] to drop the batch. Subscribe handlers are simpler. The return value is ignored entirely; the only signal is throwing, and throwing UnauthorizedException fails the invocation so AppSync rejects the subscription.

Which brings us to the discrepancy from the opening. The docs say a per-item UnauthorizedException stops processing messages. The source re-throws it only in aggregate mode and onSubscribe. The pin test constructs a resolver inline and asserts what actually happens:

it("formats per-item UnauthorizedException as an item error, not an invocation failure", async () => {
  const { AppSyncEventsResolver, UnauthorizedException } =
    await import("@aws-lambda-powertools/event-handler/appsync-events");
  const app = new AppSyncEventsResolver();
  app.onPublish("/auction/*", () => {
    throw new UnauthorizedException("nope");
  });
  const response = (await app.resolve(
    publishEvent([{ id: "1", payload: {} }]),
    context,
  )) as EventsResponse;
  expect(response.events[0]?.error).toMatch(/^UnauthorizedException - nope/);
});

Measured true on 2.34.0: item error, siblings continue. The docs and the source disagree today, possibly because the docs describe intended future behavior, and the test records which one the installed version follows. Bid-Stream doesn’t rely on per-item UnauthorizedException, so why keep the test? Because that’s the pattern this whole post is selling: when docs and source disagree, encode the winner in a test. If a future Powertools release changes the behavior to match the docs, this test fails loudly and the upgrade gets reviewed instead of trusted.

Making handlers unit-testable without AWS

Three harness decisions did most of the work here.

Fixtures are captured, not invented. The unit tests run against real AppSync event payloads, captured from live Lambda logs and scrubbed. The capture mechanism is one permanent line in the handler, logger.debug("raw event", { event }), plus a spike script that publishes against the deployed stack and harvests the logs. The documented payload shape turned out to be accurate (the only delta: far richer request headers than documented), but “matched, verified” beats “matched, hoped,” and if AWS ever changes the shape, re-running the capture updates every fixture at once.

The env-at-module-load pattern. Lambda handlers that read configuration at module scope, and this one parses process.env through a zod schema and registers its routes at import time, are notoriously awkward to test. The pattern that works: set process.env at the top of the test file, then dynamic-import() the handler in beforeAll:

const ddbMock = mockClient(DynamoDBDocumentClient);
process.env.TABLE_NAME = "TestTable";
process.env.HANDLER_MODE = "aggregate";

let handler: (event: unknown, context: Context) => Promise<unknown>;
beforeAll(async () => {
  ({ handler } = await import("../../../src/handlers/auction"));
});
beforeEach(() => ddbMock.reset());

Two non-obvious rules make it hold together. Don’t reach for jest.resetModules() to get a fresh module per test; it breaks aws-sdk-client-mock, whose mocking is prototype-level and doesn’t survive module-registry resets. Instead, one test file per configuration (HANDLER_MODE=per-item and =aggregate get separate files), each with its own fresh registry. This pattern travels to any module-scope-configured Lambda, which in my experience is most of them.

Mock at the SDK boundary, discriminate in the fake. mockClient(DynamoDBDocumentClient) with a callsFake that inspects each transaction’s bid amount lets a single captured fixture exercise a mixed batch (accepted, stale, malformed) in one test. The unmarshall gotcha from part 2 reappears here from the other side: to test the cancellation mapping you must construct TransactionCanceledException with CancellationReasons[].Item as raw AttributeValue maps, because that’s what the SDK actually delivers. The test that gets this wrong passes against a fake that never occurs in production.

A WebSocket client in ~250 lines

The e2e suite and the load fleet both need a subscriber client with timing control, which the official options don’t offer (the Amplify events client exists as a fallback, still marked @experimental). Hand-rolling one also forces you to learn the protocol, and the protocol is where two of this project’s findings live.

First, the handshake nobody documents loudly. You connect to the realtime endpoint with two WebSocket subprotocols: the literal aws-appsync-event-ws, plus your auth encoded into a subprotocol name:

export const encodeAuthProtocol = (auth: Record<string, string>): string =>
  `header-${Buffer.from(JSON.stringify(auth)).toString("base64url")}`;

const auth = { host: config.httpDns, "x-api-key": config.apiKey };
const ws = new WebSocket(`wss://${config.realtimeDns}/event/realtime`, [
  AWS_APPSYNC_EVENTS_SUBPROTOCOL,
  encodeAuthProtocol(auth),
]);

Read that auth object again: the host is the HTTP endpoint’s hostname, sent while connecting to the realtime endpoint. That’s the documented contract, it’s what the official clients do, and it’s the kind of detail that costs an hour when you guess wrong.

After the socket opens the frame vocabulary is small: connection_init → connection_ack (carrying connectionTimeoutMs: 300000, measured), periodic ka keepalives you answer with nothing, subscribe → subscribe_success or subscribe_error, then data frames. And here’s docs-vs-reality discrepancy number two: the protocol page’s example shows a data frame’s event field as an array of JSON strings. In practice it’s a single JSON-encoded string per frame, one event per frame, and the official wscat tutorial and the Amplify source both agree with reality. The client’s parser handles every shape defensively:

// In practice `event` is a single JSON-encoded string; the protocol doc's example shows an array.
export const parseDataEvents = (event: unknown): unknown[] => {
  if (typeof event === "string") return [JSON.parse(event)];
  if (Array.isArray(event))
    return event.map((entry) =>
      typeof entry === "string" ? JSON.parse(entry) : entry,
    );
  return [event];
};

One more contract came from an unexpected direction: our own harness. The first sketch of the subscription’s next() awaited future frames only. An event that arrived in the gap between publishing and calling next() sat in the buffer while the test timed out, a heisen-failure that would have surfaced as flaky e2e runs and eaten days. The fix is a cursor-based queue where the backlog drains before anyone waits:

next: (waitMs, label) =>
  new Promise((resolveNext, rejectNext) => {
    if (cursor < received.length) {
      const value = received[cursor];
      cursor += 1;
      resolveNext(value);
      return;
    }
    const waiter: QueueWaiter = {
      deliver: (value) => { clearTimeout(timer); cursor += 1; resolveNext(value); },
      abort: (error) => { clearTimeout(timer); rejectNext(error); },
    };
    const timer = setTimeout(() => {
      const index = waiters.indexOf(waiter);
      if (index !== -1) waiters.splice(index, 1); // purge the dead waiter
      rejectNext(new Error(`no event on ${label} within ${waitMs}ms`));
    }, waitMs);
    waiters.push(waiter);
  }),

Timed-out waiters remove themselves so a late frame can’t resolve a promise the test already gave up on, and shutdown() fail-fasts every pending waiter when the socket closes. The queue is extracted as a pure function precisely so those race semantics are unit-testable: six tests cover interleavings, including backlog-then-wait ordering and shutdown behavior. The honest framing is that an async test harness has the same bug classes as production code, and deserves the same tests.

E2E design under concurrency

Six scenarios run against the live stack, each on a fresh Date.now()-prefixed auction so runs can’t contaminate each other. Two of them taught lessons worth the retelling.

The partial-batch scenario has to assert a deterministic successful/failed split, but part 1 established that per-item handlers run concurrently, so five valid bids in one batch race one another and the winner is nondeterministic. The fix is in the test data: exactly one valid bid, two stale, two malformed, making successful=[0], failed=[1,2,3,4] assertable every run. The race itself isn’t a flake to retry away. It’s a documented property of per-item mode, part 4b measures its contention cost, and the test design simply routes around it.

Scenario 5 pins part 2’s replay story end to end: publish the same bidId twice, assert both acks report success, assert the subscriber’s raw frame count is 2 (the replay re-broadcasts) and the monotonic filter’s output is 1 (the duplicate drops). Scenario 6, the DLQ proof, toggles the reactions handler’s FAIL_MODE env var through the Lambda SDK with a finally that restores it, then waits out Lambda’s async retry schedule on a 330-second budget. It also needed one assertion-hygiene fix that I’m saving, because the story around it deserves its own telling.

Tooling, briefly

Jest over Vitest, decided on CDK-integration grounds alone: cdk init scaffolds Jest, the aws-cdk-lib/assertions docs are Jest-shaped, and aws-sdk-client-mock-jest’s matchers are first-class. Biome for lint/format, tsx for scripts, esbuild via NodejsFunction for bundling. The unit suite is 79 tests across 13 suites in about 14 seconds with no AWS credentials; the e2e suite’s wall-clock is dominated by scenario 6’s 330-second async-retry budget.

Everything above is green against a live stack. Our first run of that DLQ test passed in six seconds, which was exactly the problem. Part 4a.