AppSync Events, part 2: an auction where losing is the product

Part 2 of a five-part series on AWS AppSync Events. Part 1 mapped the service and ended with two open questions: partial batch handling and the publisher-visible throttle surface. This part designs a system where both are unavoidable. Part 3 builds it; part 4a measures the failure surface and part 4b the load behavior.

Realtime demos gravitate toward chat rooms and live dashboards, and those domains share a property that keeps the hard parts out of view: every event succeeds. There’s no business rule that rejects a chat message, so partial batch handling never comes up, and nothing in the domain forces a decision about what a failed event even means.

An auction inverts that. When two bids race for the same item, one of them must lose. A failed event is someone getting outbid, which is the product working: ConditionalCheckFailed means “outbid,” not “outage.” Choose this domain and the two open questions from part 1 stop being edge cases you construct artificially; they’re the main path.

That’s the design bet of Bid-Stream, and this post walks every architectural choice it forced, each one traceable to a verified constraint from part 1, a library behavior, or a domain rule. (The alternatives got rejected for specific reasons: dashboards and chat have no rejection pressure, presence is lossy-only with no sync path, sports scores have a single publisher and no contention. The auction was the only candidate that exercises everything.)

The shape of the thing

Two namespaces, two failure philosophies, deliberately one of each invoke mode from part 1:

Namespace	Channel	Integration	Failure semantics
auction	/auction/{auctionId}	Direct Lambda, REQUEST_RESPONSE	Per-event accept/reject; rejected events never broadcast
reactions	/reactions/{auctionId}	Direct Lambda, EVENT	Broadcast proceeds regardless; failures retry, then DLQ

Bids must be validated and individually rejectable before broadcast (you cannot fan out a bid that lost), so the auction namespace runs synchronous, with the Lambda in the critical path on purpose. Viewer reactions are fire-and-forget, so they run async, where the handler is an observer and failures land in Lambda’s retry machinery and an SQS on-failure destination. Both modes in one app means the sync/async trade stops being a slogan and becomes measurable (parts 4a and 4b pay that off).

Storage is one on-demand DynamoDB table: pk=AUCTION#{id} with sk=META holding the auction’s status and current high bid, and sk=BID#{bidId} holding each accepted bid, a record that doubles as the idempotency ledger and turns out to carry half the design.

Part 1’s channel-name rules show up here as the first “constraint becomes a type” moment. An auctionId rides inside a channel path, so it must be a valid channel segment: alphanumeric plus dashes, at most 50 characters. Rather than remembering that everywhere, the format is the type:

// AppSync Events channel rules (verified): 1-5 segments, each [A-Za-z0-9] with inner dashes, <=50 chars.
export const CHANNEL_SEGMENT_PATTERN =
  /^[A-Za-z0-9](?:[A-Za-z0-9-]{0,48}[A-Za-z0-9])?$/;

export const AuctionIdSchema = z
  .string()
  .regex(CHANNEL_SEGMENT_PATTERN)
  .brand<"AuctionId">();
export type AuctionId = z.infer<typeof AuctionIdSchema>;

A zod brand means no unvalidated string can flow into a function expecting an AuctionId. The service quota is now enforced by the compiler.

One transaction, four outcomes

The core domain operation is placeBid, and the design goal was one DynamoDB round trip, no follow-up reads, regardless of outcome. It’s a single TransactWriteItems with two operations:

new TransactWriteCommand({
  TransactItems: [
    {
      Update: {
        Key: { pk: auctionPk(auctionId), sk: AUCTION_META_SK },
        UpdateExpression: "SET highBid = :amount, highBidder = :bidder",
        ConditionExpression:
          "attribute_exists(pk) AND #status = :open AND highBid < :amount",
        ReturnValuesOnConditionCheckFailure: "ALL_OLD",
        // ...names/values elided
      },
    },
    {
      Put: {
        Item: {
          pk: auctionPk(auctionId),
          sk: bidSk(bid.bidId),
          amount,
          bidder,
          acceptedAt,
        },
        ConditionExpression: "attribute_not_exists(pk)",
        ReturnValuesOnConditionCheckFailure: "ALL_OLD",
      },
    },
  ],
});

The Update is the auction logic: the item must exist, be OPEN, and hold a lower high bid. The Put is the idempotency record: this bidId must never have been written before. If both conditions pass, the bid is accepted and the response was already in memory. If either fails, DynamoDB throws TransactionCanceledException, and here’s the reframe the whole engine is built on: the cancellation reasons array is a rich result, not an error. Both operations set ReturnValuesOnConditionCheckFailure: ALL_OLD, so the exception arrives carrying the conflicting items, and the mapping function reads outcomes straight out of it:

• The Put condition failed → this bidId already exists → idempotent replay of a bid that already won. Map to success, returning the stored record’s original acceptedAt.
• The Update condition failed and the auction is missing or not OPEN → terminal AUCTION_CLOSED.
• The Update condition failed otherwise → terminal STALE_BID, with the current high bid (from the returned item) in the message: high bid is 750, bid of 500 loses.
• TransactionConflict on either op → retriable THROTTLED.

Two details here will save you an afternoon each. First, precedence: the Put check comes before the Update check. A replayed winning bid also looks stale (its amount no longer beats the high bid, because it set the high bid), so testing the stale condition first would misreport a replay as a loss. The bid already won once; re-reporting success is the idempotent answer. Second, a gotcha you will hit: CancellationReasons[].Item arrives as raw AttributeValue maps even through the document client, { status: { S: "OPEN" } } rather than { status: "OPEN" }. You need unmarshall() from @aws-sdk/util-dynamodb, and if you don’t know that, your first cancellation-mapping test will tell you in the most confusing way available. (Part 3 shows the aws-sdk-client-mock test that pins all four outcomes.)

The error taxonomy, and the library that dictated its wire format

Clients need to know one thing about a rejection: retry or don’t. So rejections are a discriminated union, terminal (STALE_BID, INVALID_PAYLOAD, AUCTION_CLOSED) versus retriable (THROTTLED, INTERNAL), and the plan was to serialize that as a small JSON envelope in each per-event error string.

A library detail killed the envelope. Powertools’ per-item mode formats a thrown error as `${error.name} - ${error.message}`: you don’t control the frame, only the name and the message. JSON inside that frame would arrive wrapped in Error - {...} noise. So the wire format became the frame itself, CODE - message, achieved by an Error subclass whose name is the taxonomy code:

// Error subclass so Powertools' per-item `${error.name} - ${error.message}` formatting
// emits exactly formatRejection(rejection).
export class BidRejectionError extends Error {
  readonly rejection: BidRejection;

  constructor(rejection: BidRejection) {
    super(rejection.message);
    this.name = rejection.code;
    this.rejection = rejection;
  }
}

Now a thrown BidRejectionError in per-item mode and a returned formatRejection(...) string in aggregate mode emit byte-identical error strings, STALE_BID - high bid is 750, bid of 500 loses, and a unit test pins the equivalence:

const error = new BidRejectionError(
  rejectBid("AUCTION_CLOSED", "auction x is not open"),
);
// Powertools per-item mode emits `${error.name} - ${error.message}` — must equal our format
expect(`${error.name} - ${error.message}`).toBe(
  formatRejection(error.rejection),
);

The generalizable lesson, and the one worth stealing even if you never touch this service: when a middleware owns the envelope, design your taxonomy to survive its formatter. Encode meaning in the parts you control, here the error’s name, and pin the round-trip with a test so a library upgrade that changes the frame fails loudly instead of silently garbling your protocol.

One question this section can’t answer: does that carefully formatted string ever reach the publisher? Part 1 established that the ack’s failed entries are documented nowhere and the Amplify client’s types say {identifier, index} only. The taxonomy lands in the handler logs at minimum. Whether it travels further is part 4a’s first measurement.

The real case for aggregate mode: the batch-max short-circuit

Part 1 established that per-item handlers already run concurrently, so throughput isn’t what aggregate mode offers, and promised a domain where what it does offer wins outright. Here it is.

Every event in a publish batch arrives on one channel, which in this design means one auction. Bids within a batch are therefore totally ordered by amount, and only the batch’s highest bid can possibly win: if the max loses against the table’s high bid, its siblings lose a fortiori; if the max wins, monotonicity makes every sibling stale the instant it lands. So the aggregate handler sorts the batch, runs one conditional transaction for the max, and resolves the other four locally without touching DynamoDB:

// All events in one publish share one channel — one auction — so only the batch's
// highest bid can possibly win. One transaction per batch; siblings resolve locally.
const siblingResult = (topResult: PlaceBidResult): PlaceBidResult => {
  if (topResult.outcome === "accepted") {
    return rejected(
      rejectBid("STALE_BID", `outbid within batch by ${topResult.bid.amount}`),
    );
  }
  if (topResult.rejection.code === "STALE_BID") {
    return rejected(
      rejectBid(
        "STALE_BID",
        "a higher bid in the same batch was already stale",
      ),
    );
  }
  return rejected(topResult.rejection); // AUCTION_CLOSED applies to all; THROTTLED/INTERNAL stay retriable
};

const placeBidBatch = async (auctionId, entries) => {
  if (entries.length === 0) return [];
  const top = [...entries].sort((a, b) => b.bid.amount - a.bid.amount)[0];
  const topResult = await placeBid(auctionId, top.bid);
  return entries.map((entry) =>
    entry.id === top.id
      ? { id: entry.id, ...topResult }
      : { id: entry.id, ...siblingResult(topResult) },
  );
};

Note the sibling mapping preserves the taxonomy’s retry semantics: if the max was THROTTLED, the siblings report retriable too. They were never attempted, and telling the client otherwise would lie about their fate.

Per-item mode cannot express this. Five concurrent handlers each hold one bid and no view of the others; they all race the same conditional update, up to five transactions contending on one hot item where one would do. Worst case, aggregate mode cuts write attempts on the hottest item in the system by 5×. That’s why part 1 called aggregate mode a contention lever: it’s invisible in an echo demo and decisive in a domain with cross-event structure. The handler emits a BatchShortCircuits metric (avoided round trips) precisely so part 4b can measure the claim instead of asserting it.

Two pieces of semantic fine print, both decided deliberately. Within a single batch, an intermediate bid that would have been momentarily winning is reported as already-outbid; defensible, since the batch is one publisher’s submission anyway. And a sibling that is an idempotent replay of a previously accepted bid is reported STALE_BID rather than the success per-item mode would return for it, because the short-circuit never queries its record. That one is narrower than it sounds: it needs a whole-batch retry that re-sends an accepted bid alongside a higher new one, the code is terminal either way so the retry client stops either way, and “outbid” is a true statement about the final state. It is still a case where the same bidId gets two different answers depending on its company, which is worth knowing before you copy this pattern into a domain where the losing record has to be durable.

Ordering without sequence numbers

AppSync Events documents no delivery-order guarantee (verified absence as of July 2026; the docs simply don’t address it). The instinct is to add sequencing infrastructure: counters, buffers, reordering windows. This design gets ordering for free instead, by noticing the payload already carries a monotonic value: accepted bid amounts are strictly increasing per auction, because the conditional write guarantees it. So subscribers reconcile with an eight-line filter:

// Spec §6.2: accepted amounts are strictly increasing per auction, so dropping any
// non-increasing amount reconciles ordering AND deduplicates idempotent re-broadcasts.
export const createMonotonicFilter = (): ((amount: number) => boolean) => {
  let high = Number.NEGATIVE_INFINITY;
  return (amount: number) => {
    if (amount <= high) return false;
    high = amount;
    return true;
  };
};

The comment’s second clause is a freebie discovered during test design. An idempotent replay re-broadcasts (the handler re-reports the stored bid as success, and suppressing it server-side would mean failing the replay toward the publisher, the worse trade), so subscribers can see the same accepted bid twice. The filter already drops it: the amount isn’t greater than the known high. One mechanism, two guarantees, ordering reconciliation and deduplication. The generalizable version: when your payload carries a naturally monotonic value, ordering costs nothing. Look for one before you build sequence numbers.

A retry policy designed for ignorance

The publisher-side design had to assume the worst answer to the open question above: the ack names which events failed and says nothing about why. The policy that works under total ignorance:

• Retry only the failed ids, re-batched, with full-jitter exponential backoff and a bounded attempt budget.
• Abort on 400/401/403, mirroring the Amplify client’s non-retryable set, since auth doesn’t heal with backoff.
• No parseable ack at all (transport failure, 5xx) → retry the whole pending set.

This is safe for exactly one reason: client-generated bidIds make every retry idempotent. A replayed accepted bid maps to success server-side (the Put-condition path from earlier); a retried stale bid fails again harmlessly; a throttled bid gets another chance. Correctness comes from idempotency, not from error visibility, which means the policy survives the ack being maximally opaque, and anything part 4 learns about the ack can only make it smarter, never necessary.

What we deliberately didn’t build

Four production concerns, each left out with a named trigger rather than built speculatively: identity-based subscribe auth (the prototype’s API-key subscribe handler checks auction-exists-and-open; swap in Cognito/OIDC and the hook is identical, the inputs get richer), write-sharding the hot auction item (trigger: a single auction’s bid rate approaching the per-item write ceiling), load shedding ahead of DynamoDB (trigger: measured unhealthy saturation), and a receipts side-channel giving publishers per-rejection detail (trigger: the ack turning out to be opaque). That last trigger should sound familiar. It’s the open question again: part 4a re-ranks it with the failure-surface measurements, and part 4b finishes the job with load numbers.

What we couldn’t know at design time

Two assumptions in this design are unverifiable by reading. Does the CODE - message string ever reach the publisher, or does it die in the logs? And when the system genuinely pushes back, Lambda throttled or handler crashed, what does the publisher’s response actually look like? The docs don’t say. The retry policy assumes nothing; the taxonomy hopes for more.

So we built the instrument: the handlers, the CDK stack, a hand-rolled WebSocket client, and a test harness that can catch the service in the act, including the places where the docs and the libraries’ source disagree with each other. That’s part 3.