Event sourcing on AWS serverless, part 2: CQRS and the read model that lags on purpose

Part 2 of a three-part series on event sourcing and CQRS with AWS serverless, grounded in appsync-es. Part 1 built the write model - the event log, the fold, optimistic concurrency, snapshots. This part builds the read side. Part 3 covers integration events and the trade-offs.

Deploy this system and try a specific sequence: openAccount, then depositFunds, then immediately getAccount with the returned id. There’s a decent chance the query shows the old balance - or no account at all. Run it again a second later and it’s correct.

A stale read like that usually gets filed as a bug. Here it’s documented behavior - the README includes a walkthrough called “watch eventual consistency happen.” By the end of this post you’ll know exactly which asynchronous hop the lag comes from, why the caller who just wrote never experiences it, and what your options are when a read genuinely can’t tolerate it.

The problem with reading an event log

Part 1 ended with a working write model and an awkward read story. Current state is a fold over events, so the honest way to serve getAccount is to call loadAccount - snapshot, range query, replay - on every read. That’s correct, always fresh, and it means every read pays for reconstruction work whose cost depends on write history. Snapshots bound it, but bounded isn’t free, and in most systems reads outnumber writes by orders of magnitude. You’d be doing the expensive thing often to support the cheap thing rarely.

listAccounts kills the idea outright. There is no fold-on-read version of “show me all accounts” that doesn’t rehydrate every account just to print ids and owners. Some queries simply cannot be served from an event log directly at any acceptable cost.

So we don’t serve them from the event log. We maintain a second representation - denormalized, query-shaped, one item per account holding its current state - and keep it up to date in the background. The event log stays the source of truth; the second representation is derived and disposable, like a snapshot but shaped for readers instead of for the fold.

CQRS, in one paragraph

Command Query Responsibility Segregation means the code path that writes and the code path that reads are allowed to be different shapes, hit different storage layouts, and optimize for different things. The write side cares about correctness: invariants, ordering, surviving concurrent writers. The read side cares about being cheap and fast: denormalized records, indexes, pagination. Once state lives in an event log, some split like this stops being optional - the log’s shape is right for appending facts and wrong for listing accounts, and no single layout is right for both.

In this repo the segregation is physical. Mutations route to command-resolver.ts, which appends events and never writes a view. Queries route to query-resolver.ts, which reads views and never touches an event:

case 'getAccount': {
  const { id } = parse(getAccountSchema, args);
  return getView(id); // one GetItem - no events, no fold
}

Nothing in the query path imports rehydrate. Reads are O(1) whether the account has six events or sixty thousand. That’s the entire point of the read model, and everything else in this post is about keeping it honest.

The view

The read model is one more item shape in the same single table from part 1, distinguished by its sort key:

item	pk	sk	GSI (gsi_pk / gsi_sk)
view	ACCOUNT#<id>	VIEW	ACCOUNT / <owner>#<id>

The VIEW item holds the account’s current state plus an updatedAt, and it projects itself into a ByOwner GSI so listAccounts can paginate accounts in owner order with a query instead of a scan. Events and snapshots never write GSI attributes, so the index contains nothing but views - listAccounts physically cannot trip over an event.

Who writes the view? A Lambda called the projector.

The projector

The table streams every write (NEW_IMAGE) to the projector, which turns new events into fresh views. The core of lib/bank-account/handlers/projector.ts:

export const handler = async (
  streamEvent: DynamoDBStreamEvent,
): Promise<void> => {
  const events = streamEvent.Records.filter(isNewEventRecord).map(toEvent);
  if (events.length === 0) return;

  const affectedIds = [...new Set(events.map((e) => e.id))];
  await Promise.all(
    affectedIds.map(async (id) => {
      const { state } = await loadAccount(id);
      if (state) await upsertView(state, new Date().toISOString());
    }),
  );
  // it also publishes each event to EventBridge - that's part 3
};

One design choice worth pausing on: the projector does not incrementally apply the one event it received to the existing view. It reloads the account’s full authoritative state from the event store and overwrites the view with that. This costs an extra read but buys a lot of simplicity - a batch containing three deposits for the same account collapses into one load and one upsert, records arriving out of order can’t assemble a wrong balance, and a missed or duplicated stream record can’t leave the view subtly drifted. The view is always a copy of what the event store says right now, never a running total maintained by hand.

Idempotency, because the stream will repeat itself

DynamoDB Streams delivers records at least once. Lambda retries failed batches (this repo configures 3 attempts, batch size 10). And the projector’s reload-everything strategy means it recomputes views it may have just written. upsertView will be called redundantly in perfectly normal operation - so it has to be harmless to call twice. The guard, in lib/bank-account/adapters/read-model.ts:

ConditionExpression: 'attribute_not_exists(#v) OR #v < :version',

Write the view only if none exists, or if the existing view’s version is strictly older than the one we’re writing. A ConditionalCheckFailedException here is caught and treated as a no-op, because it means the view is already at or ahead of this update. Duplicates become no-ops; a stale update racing a fresher one can’t drag the view backwards. The version - the same per-aggregate counter doing concurrency duty on the write side - is what makes projection idempotent on the read side.

If you take one operational rule away from this series: every projection must be safe to run more than once. At-least-once delivery is the contract, and the contract will be exercised.

The self-loop trap

Here’s the bug this repo exists to teach. The projector writes VIEW items into the very table whose stream triggers the projector. Follow the arrows: event insert → stream → projector → view write → stream → projector → view write → stream… If the projector processed every record it received, it would invoke itself forever.

The fix is the filter you already saw in the handler:

const isNewEventRecord = (record: DynamoDBRecord): boolean =>
  record.eventName === "INSERT" &&
  (record.dynamodb?.Keys?.sk?.S ?? "").startsWith("EVENT#");

Only INSERTs, and only items whose sort key marks them as events - never VIEW writes, never SNAPSHOT writes, never modifications. What makes this trap nasty is that removing the filter doesn’t break correctness. The version guard means each self-triggered upsert lands as a no-op, balances stay right, every test passes. What you get is a Lambda that never stops invoking itself: a runaway cost bug that looks like a healthy, busy system. If you use single-table design with streams, know which of your writers feed the stream that triggers them.

(The alternative that makes this bug impossible - a separate table for views, whose stream nobody consumes - is a real option with real trade-offs, covered in part 3.)

Where the lag comes from

Now the moment from the opening. The full path from mutation to queryable state:

Everything left of the dotted arrow happens before the mutation returns. Everything right of it happens after - stream delivery, projector invocation, view upsert - and no code path anywhere makes a query wait for it. So there is a real window, usually around a second, where the event store and the view disagree. That window is eventual consistency: a promise that the read model will catch up, with no promise about when.

You can watch it. Open an account, deposit, and note the balance and version in the mutation’s response. Immediately getAccount: depending on timing you’ll see the previous version, or null right after an open. Wait a beat, query again, and it matches. The README walks through this because seeing the window once teaches more than any diagram of it.

Why the writer never notices

Look back at the diagram: the mutation’s response arrow comes straight off the command Lambda, and part 1 ended by pointing out why. The command resolver returns apply(state, event) - the new state computed in memory from the fold plus the event it just appended. It never reads the view to answer its own caller. So the client that just deposited sees the post-deposit balance in the mutation response, guaranteed, even though a getAccount fired in the same instant might not show it yet.

This is “read your own writes” for the one caller who wrote, and it costs nothing - the data was already in memory. A large share of real-world consistency complaints reduce to exactly this case (“I just saved and the screen shows old data”), and here it’s handled without touching the read path at all: the client keeps the mutation response instead of immediately re-querying.

When that isn’t enough - some other read genuinely can’t tolerate the lag - you have options, in roughly increasing order of cost:

1. Reconcile client-side. Merge the mutation response into local state instead of re-fetching. Often this is the whole fix.
2. Version-aware polling. The mutation returned version: 7, so retry getAccount (bounded, short backoff) until the view’s version is ≥ 7. One client trades a little latency for a guarantee.
3. Read from the write path. Add a query that calls loadAccount directly, paying the fold cost, for the rare read that must be current. Casual reads keep using the cheap view.
4. Project synchronously. Have the command resolver upsert the view itself before returning. The lag disappears for that path - and now two places write views, both must stay correct, and the write Lambda’s latency and failure modes are coupled to the read model.

This repo implements none of them, on purpose. The pedagogical point is to make the lag visible and then show the cheapest mitigation (the in-memory return value) rather than engineer the window away. In production you’d reach for these one at a time, driven by an actual requirement, because each rung costs more than the one before.

Where we are

The system now has both halves: a write model that records facts correctly under concurrency, and a read model that serves them cheaply, connected by a stream and an idempotent projector, with a consistency window we can explain instead of apologize for.

There’s one arrow left in the architecture. The projector does a second job on every invocation: it publishes each event to an EventBridge bus, where systems that aren’t this service at all - audit, notifications, analytics - can react without the write path knowing they exist. What an event means once it leaves home, why you close an account instead of deleting it, and an honest accounting of what this architecture costs: part 3.