September 6, 2026

The semantic medallion in TypeScript: a developer's walkthrough

A from-zero guide to the semantic medallion prototype - Parquet, DuckDB, RDF and SPARQL, entity resolution, and the measurement that mattered most.

Data Engineering Knowledge Graphs TypeScript

This guide is for a developer who wants to understand and run the semantic medallion prototype without first mastering the usual data-engineering and knowledge-graph background. It explains the context (what the tools are and why they were chosen), the patterns the code follows, and how to use the application step by step.

Two notes before you start:

Where this guide shows a specific command, file path, or tag name, it comes from the project’s own CLAUDE.md and the build records. Where it shows an example query, identifier, or vocabulary term, treat it as illustrative — the exact names live in the repository, and the guide tells you where to look. If the repository and this guide ever disagree, the repository wins.
The best way to read this guide is with the repository open beside it. Every section ends with “where to look” so you can move from the explanation to the real code.

1. The big picture

The problem

Imagine a small company with three software systems: a customer system (contacts and companies), a billing system (accounts and invoices), and a support desk (tickets). The same real customer exists in all three — but each system spells the name a little differently, uses its own internal record number, and stores dates in its own format. Nobody can answer a simple question like “show me everything we know about this customer” without writing a script that joins the three exports by hand, and that script breaks every time a system changes.

The prototype solves this with two ideas from the data world, combined:

The medallion architecture — a way of organizing data processing into three layers named Bronze, Silver, and Gold.
A knowledge graph — a way of storing data as a web of facts and relationships rather than as separate tables.

The prototype’s argument is that if the Gold layer is a knowledge graph instead of a set of tables, the relationships between records live in the data itself, and the “show me everything” question becomes a single query.

The three layers

Bronze is the raw material: the exported files from the three systems, stored exactly as they arrived. Nothing is fixed or changed here. If something goes wrong downstream, you can always come back to Bronze.
Silver is the cleaned material: the same data with consistent types, trimmed whitespace, normalized dates — and, most importantly, a stable identifier attached to every record. Silver is stored as Parquet files (explained below).
Gold is the graph: every Silver row is converted into small facts (“customer 123 has email address X”, “invoice 88 belongs to customer 123”), those facts are loaded into a graph database, and the catalog questions are answered from there.

The end-to-end flow

three raw exports  →  DuckDB cleaning  →  Silver Parquet files
       (Bronze)                                (Silver)
                                                  ↓
                                the mapper turns rows into facts
                                                  ↓
                                    Oxigraph graph store (Gold)
                                      ← matcher adds "same-as" links
                                      ← catalog adds metadata and lineage
                                                  ↓
                    command-line queries, tests, and the browser front end

Everything to the right of Silver can be thrown away and rebuilt from the Parquet files at any time. That is a deliberate design choice, and it comes up again and again.

Where to look: the opening comment of src/mapper and src/store describe the flow in the project’s own words. docs/verdict.md is the build’s final summary.

2. Concepts you need

Each concept below has three parts: what it is, why this project uses it, and where to see it in the repository.

2.1 Parquet — what it is and why it matters

Most developers first meet data as CSV files (comma-separated values) or JSON. Both are row-oriented: each line holds one complete record. That is convenient for humans and for streaming, but it has real costs when data gets large:

Everything is text. A number, a date, and a piece of text all look the same until you parse them, so every tool has to re-guess the types.
To read one column, you still have to read every byte of every row.
Compression is weak, because neighboring bytes are unrelated (a name, then a number, then a date).

Parquet is a file format that stores data by column instead of by row. All the customer names sit together, all the invoice amounts sit together, and so on. This has three consequences:

Types are part of the file. A Parquet file knows that invoice_total is a decimal and created_at is a timestamp. Every tool that reads it gets the right types with no guessing.
Reading is selective. If a query only needs two columns, only those two columns are read from disk.
Compression is excellent. Similar values sit next to each other, so the file is often five to twenty times smaller than the equivalent CSV.

Parquet is also a standard: the same file opens in Python’s pandas, in Apache Spark, in AWS Athena, in DuckDB, and in dozens of other tools. Files are written once and not edited in place — you produce a new file rather than changing an old one — which makes Parquet a good “system of record” that other things are built from.

In this project, Parquet is the source of truth. The Silver layer is Parquet, and the whole graph can be regenerated from it. That is why the graph can be treated as disposable.

Where to look: the output of npm run pipeline includes the Silver Parquet files (their location is documented in the src/silver prologue). You can inspect them with DuckDB, described next.

2.2 DuckDB — a database that runs inside your program

Most databases are servers: you install them, run them as a separate process, and connect over the network. DuckDB is different. It runs inside your Node process, the way SQLite does — there is nothing to install or start. But unlike SQLite, it is built for analytics: it reads CSV, JSON, and Parquet files directly, runs SQL over them at high speed, and writes Parquet out.

That makes it ideal for Bronze-to-Silver work. Cleaning data is naturally expressed in SQL (TRIM, LOWER, CAST, CASE), and DuckDB can read the raw export and write the clean Parquet file in one statement. The project uses the @duckdb/node-api package.

An illustrative shape of what the Silver code does (the real SQL is in src/silver):

-- Illustrative. Read a raw export, clean it, and write Parquet.
COPY (
  SELECT
    TRIM(customer_name)            AS name,
    LOWER(TRIM(email))             AS email,
    CAST(created AS DATE)          AS created_on
  FROM read_csv('fixtures/customer-system.csv')
) TO 'silver/customers.parquet' (FORMAT PARQUET);

Where to look: src/silver — read the prologue comment first, then the SQL.

2.3 Stable identifiers — the most important design decision

Every system gives its records an internal number, but those numbers mean nothing outside that system. Customer 42 in billing has no relation to customer 42 in support. To connect records across systems, every record needs an identifier that:

names the thing rather than its position in a table (never an auto-increment number),
is deterministic — computed from the record’s own natural key (an email address, an organization number, an order number), so running the pipeline twice produces the same identifier twice (never a random value),
is scoped to its source, so the customer system’s record and the billing system’s record get different identifiers even when they describe the same person. Whether they are the same person is a separate decision (see entity resolution, 2.9).

In the graph world these identifiers are web-style addresses. The technical term is IRI (Internationalized Resource Identifier — a web address that may contain characters beyond plain ASCII). You will see that word in library documentation; this guide mostly just says “identifier.” An illustrative shape:

https://example.invalid/customer-system/customer/jane.doe@example.com
https://example.invalid/billing/account/ACC-00913

The real namespace and the exact rules are documented in one place only: the prologue of src/identifiers. Read it before anything else in the codebase, because every later layer depends on it.

2.4 Graphs: triples, quads, and named graphs

A knowledge graph stores facts as triples: subject, predicate, object. Read them as tiny sentences.

<customer/jane>   <hasEmail>      "jane.doe@example.com"
<customer/jane>   <worksFor>      <organization/acme>
<invoice/88>      <billedTo>      <customer/jane>

The subject and predicate are always identifiers. The object is either an identifier (linking to another node) or a literal — a typed value like a string, number, or date.

This project stores quads, not triples: a fourth element names the graph the fact belongs to. Think of a named graph as a folder. This project puts each entity’s facts from each source into its own folder:

graph: <customer-system/customer/jane>   contains all facts the customer system says about Jane
graph: <billing/account/ACC-00913>       contains all facts billing says about that account
graph: <links>                           contains the "these two are the same" facts
graph: <catalog>                         contains the dataset descriptions
graph: <provenance>                      contains the load history

Why folders per entity? Because replacing a folder is trivial and atomic: delete it, write the new one. That is how incremental updates work (2.11). The project’s rule is “a fact lives in the graph of its subject,” so each entity’s folder holds that entity’s outgoing facts.

One more rule that is easy to miss but that everything below depends on: the project never uses blank nodes. A blank node is a node with no identifier — many RDF tools create them for nested structures like addresses. They cannot be compared between two runs of the pipeline, which would break every determinism guarantee below. Instead, every nested thing gets its own minted identifier (something like …/customer/jane/address/billing).

The standard behind all this is RDF (Resource Description Framework), a web standard for expressing facts this way. The file format the project writes is N-Quads: one quad per line, plain text, easy to sort and compare.

Where to look: src/store (the graph layout), src/mapper (where quads are produced), and any .nq file produced by the pipeline.

2.5 Vocabulary and ontology

If one source says hasEmail and another says email_address, the graph can’t connect them. So the project defines a small vocabulary: a fixed set of classes (Customer, Organization, Invoice, Ticket) and properties (has email, billed to, opened by), and the mapper translates every source into that one vocabulary. Where a well-known public vocabulary already has the right term, the project borrows it — schema.org is the usual source.

An ontology is a vocabulary plus relationships between its terms — most importantly subclass relationships (“a ComplianceOfficer is a kind of Employee”). This matters because a query can then say “find everything related to Employee, including subtypes” and pick up ComplianceOfficers automatically. That is what the project calls semantic search.

Where to look: src/vocabulary.

2.6 SPARQL — the query language

SPARQL is to graphs what SQL is to tables. You describe a pattern of triples with variables (marked with ?), and the store returns every combination that matches. The essentials:

# Everything the store knows about one subject.
SELECT ?predicate ?object
WHERE { <PUT-AN-IDENTIFIER-HERE> ?predicate ?object }

# The same, but also tell me which graph (folder) each fact came from.
SELECT ?graph ?predicate ?object
WHERE { GRAPH ?graph { <PUT-AN-IDENTIFIER-HERE> ?predicate ?object } }

# Follow subclass relationships any number of steps. The * is a "property path".
SELECT ?thing
WHERE {
  ?thing a ?type .
  ?type rdfs:subClassOf* <SOME-CLASS> .
}

Two SPARQL features the project leans on:

GRAPH ?g { … } lets a query ask where a fact came from, which is how “which sources contribute to this customer” is answered.
Property paths (*, +, |) let a query walk chains of relationships — subclass chains for semantic search, and chains of same-as links for entity resolution.

SPARQL also has an update dialect. The two statements that power incremental updates:

DROP SILENT GRAPH <the-entity-graph> ;
INSERT DATA { GRAPH <the-entity-graph> { … the new facts … } }

Where to look: the real queries for the four catalog questions are in tests/catalog-questions.spec.ts. Copy them from there — they are the authoritative versions.

2.7 Oxigraph and WebAssembly

Oxigraph is a graph database written in Rust that supports SPARQL. The project uses it through its npm package, which is the Rust code compiled to WebAssembly — a binary format that runs inside JavaScript engines at near-native speed. Two things follow:

Like DuckDB, it runs inside your process. No server, nothing to install.
Because it is WebAssembly, the exact same store runs in a browser. That is how the front end (part 8) can load the whole graph client-side and answer queries without a backend.

The trade-off is memory: the store keeps everything in RAM. The build’s measurements show where that becomes the limit (section 6).

Where to look: src/store, and the browser playground added in part 4.

2.8 DCAT and PROV-O — the catalog that describes itself

A data catalog is normally a separate system that keeps a list of datasets and points at where they live. This project does something more interesting: it describes the datasets inside the same graph as the data, using two web standards.

DCAT (Data Catalog Vocabulary) defines terms for describing datasets: title, description, the files that distribute them, their format. Each of the three sources and each Silver table gets a DCAT description.
PROV-O (the Provenance Ontology) defines terms for describing how things came to be: an activity (a pipeline run) used some entity (a raw export) and generated another (a Silver file), at a time, by an agent.

Together they make lineage queryable: “which raw file did this fact come from, and when was it loaded?” is an ordinary SPARQL query, not a separate logging system.

Where to look: src/catalog.

2.9 Entity resolution — deciding two records are the same thing

This is the hard part that most demos skip. Given a customer-system record and a billing record, are they the same person? The project’s matcher answers with rules in tiers:

High confidence rules use shared natural keys — an identical email address or organization number is very strong evidence.
Medium confidence rules use weaker evidence — a matching name and address, for instance.

When the matcher decides two records match, it does not merge them or change their identifiers. It writes a same-as link (owl:sameAs, from the Web Ontology Language) into the links graph, with the confidence tier and the rule that fired recorded alongside. Queries that want the whole person then walk across the links with a property path.

Keeping links in their own graph is a deliberate isolation: the matcher is the least trustworthy component, and if it turns out to be wrong, its output can be thrown away and regenerated by dropping one graph without touching any entity data. Section 6 shows why that mattered.

Where to look: src/matcher, and the conflict manifest in src/generator — the list of deliberately planted duplicates and conflicts the matcher is tested against.

2.10 Determinism — same input, same bytes

The mapper (Silver rows in, quads out) is a pure function: no randomness, no clock, no reading the environment. Run it twice on the same input and you get byte-identical output. This one property buys three things:

Content hashing. Every entity’s Silver content is hashed, and the hash is stored in a metadata graph. On the next run, entities whose hash hasn’t changed are skipped entirely.
Golden-file tests. The expected output of the mapper is checked into the repository; the test compares the real output byte for byte.
The convergence test (2.11) — which is only possible because two graphs with no blank nodes can be compared by sorting their N-Quads and comparing the text.

The same discipline applies to the synthetic data: the generator takes a fixed seed, so it produces exactly the same records every run, and the committed copy in fixtures/ is the reference.

Where to look: src/mapper, src/format (the single module allowed to format dates and numbers — so there is exactly one canonical text form for each value), and fixtures/.

2.11 Incremental updates and the convergence property

Rebuilding the entire graph on every change is simple but slow. The project’s incremental path works like this:

Compute each entity’s content hash from Silver; compare with the stored hash.
For changed entities only, run the mapper and swap the entity’s graph (drop the old folder, insert the new one).
For entities that disappeared from the source, drop their graphs.
Emit a patch: the list of quads added and removed.

The question is whether this incremental path is trustworthy. The project answers with the convergence property test: take a store built by applying patches, take a store built fresh from scratch, sort both as N-Quads, and require them to be identical text. The build verified this at every scale it ran. When you read “convergence held byte-for-byte,” this is what it means.

Where to look: src/incremental, and the convergence test in the test suite (part 7).

2.12 The live layer (part 8)

The patch from step 4 above is also an event. The front end subscribes to a channel over AWS AppSync Events (a managed publish-subscribe service using WebSockets), receives patches, and applies them to its in-browser Oxigraph store — so an entity page you have open updates when the underlying data changes. You do not need this layer to learn the system; everything through part 7 runs locally.

Where to look: the web workspace added in part 8, and its own prologue and package scripts.

3. The patterns, and why they were chosen

These are the recurring design ideas. Recognizing them makes the code much easier to read.

Source of truth versus projection. Parquet is the truth; the graph is a projection of it — a derived view, like a search index. Anything derived can be deleted and rebuilt. This is why the project never worries about the graph store being corrupted or lost: npm run pipeline recreates it.

Pure functions between layers. Each pipeline stage is a function of its input and nothing else. Purity is what makes hashing, golden tests, and convergence possible. When you add code, ask “did I just make this stage depend on the clock, on randomness, or on the environment?” If yes, it belongs in the provenance layer, injected explicitly.

Named graph as the unit of replacement. Never update individual facts. Regenerate the entity’s whole graph and swap it. Updates and deletes become trivial, and nothing needs a diff at the fact level.

A fact lives in the graph of its subject. One simple rule that removes all ambiguity about where a fact goes.

Isolate the volatile part. The matcher’s output — the most likely thing to be wrong — lives in its own graph so that retracting it never touches entity data.

One module owns each canonical form. All date and number formatting goes through src/format. All identifier construction goes through src/identifiers. If two places could format the same value, they would eventually disagree.

Contracts are tests, not comments. Part 2 “reserved” a range of organization numbers for large-scale data in a comment; nothing enforced it, and four committed records were already inside the range. Part 9’s first scaled build added an assertion and caught it immediately. The project’s stated lesson: a written rule without a test does not stay true.

Immutable milestones. Each of the nine parts is a git tag (part-01 through part-09). Tags are never moved. The decision logs in docs/decisions/ record what was tried and rejected in each part — read them when a design choice looks strange; the reason is usually there.

4. Reading the repository

The layout follows CLAUDE.md:

Path	What it holds
`src/generator`	Synthetic exports from the three sources, fixed seed, plus the conflict manifest
`src/silver`	DuckDB landing and cleaning; writes Silver Parquet
`src/identifiers`	Identifier minting — read its prologue first
`src/vocabulary`	Classes, properties, namespaces
`src/format`	Canonical formatting for dates, decimals, booleans
`src/mapper`	Pure Silver-to-Gold mapping, rows to N-Quads
`src/store`	Oxigraph loading, named-graph layout, queries
`src/matcher`	Entity resolution; writes the links graph
`src/catalog`	DCAT and PROV-O emission
`src/incremental`	Content hashes, graph swaps, patch emission
`scripts/bench.ts`	Benchmark; appends to `docs/bench.csv`
`tests/catalog-questions.spec.ts`	The four catalog questions as tests
`docs/decisions/part-NN.md`	Decision log per milestone
`docs/verdict.md`	The build’s conclusions and measurements
`fixtures/`	Committed generator output and golden files

How to read a module: every module opens with a prologue comment — what it is, the one idea behind it, how it fits. Read the prologue, then the exported functions, then the comments at decision points (they explain why, never what). Skim the module’s decision log entries last.

Suggested reading order: CLAUDE.md → src/identifiers prologue → src/mapper prologue → src/store prologue → tests/catalog-questions.spec.ts → docs/verdict.md.

5. Step by step: running the application

5.0 Prerequisites

Node 22 (check with node --version).
git.
At least 8 GB of free memory if you intend to run the largest scale (5.3 and 6).

5.1 Install

git clone <repository-url>
cd <repository>
npm install

5.2 Generate the small world

npm run generate

This writes the three synthetic exports into fixtures/. Open them. Notice that they are in different file formats with different column conventions and date formats — that is deliberate, because real systems disagree exactly like this. The small world has about thirty customers, which is small enough to read by eye.

Then open the conflict manifest in src/generator. It lists every deliberately planted problem: the same customer under three spellings, records that share an email address, a billing account with no matching customer, a duplicate inside one source, casing and whitespace traps, a name with accented characters. These are the matcher’s exam questions.

Run npm run generate a second time and diff fixtures/ against git: nothing changes. That is the fixed seed at work.

5.3 Run the pipeline

npm run pipeline

This runs Bronze → Silver → Gold in full: DuckDB cleans the exports into Parquet, the mapper turns Parquet rows into quads, the store loads them, the matcher writes links, and the catalog writes DCAT and PROV-O. The prologue of src/silver documents where the Parquet files land; the prologue of src/store documents where the store and the N-Quads output land.

If you have DuckDB’s command-line tool installed, you can look at Silver directly:

SELECT * FROM read_parquet('<path-to-silver>/customers.parquet') LIMIT 10;

(Any Parquet-reading tool works; the point is that Silver is ordinary, typed, tabular data.)

5.4 Run the tests

npm test

Now open tests/catalog-questions.spec.ts. The four questions are:

Q1 — Everything about one customer, across every source, through same-as links.
Q2 — Which sources contribute to that customer, and when was each contribution loaded.
Q3 — Every entity related to a concept, including entities typed by subtypes of that concept.
Q4 — If one source dataset changes, which datasets and entities downstream are affected.

At the finished state, all four pass. Each test contains the real SPARQL for its question — these are the queries to learn from and to copy.

5.5 Ask the graph questions yourself

npm run query -- "<a SPARQL query>"

Start with queries that work regardless of the exact vocabulary, so you can discover the real names:

# 1. What graphs (folders) exist? Shows you the naming scheme.
SELECT DISTINCT ?g WHERE { GRAPH ?g { ?s ?p ?o } } LIMIT 50

# 2. What predicates are used? Shows you the vocabulary.
SELECT DISTINCT ?p WHERE { GRAPH ?g { ?s ?p ?o } }

# 3. Pick a subject identifier from query 1 and describe it.
SELECT ?g ?p ?o WHERE { GRAPH ?g { <PASTE-AN-IDENTIFIER> ?p ?o } }

# 4. Look at the links graph (use the real graph name from query 1).
SELECT ?a ?b WHERE { GRAPH <LINKS-GRAPH-NAME> { ?a <http://www.w3.org/2002/07/owl#sameAs> ?b } } LIMIT 20

Then copy Q1 from the test file and run it for a customer you saw in the manifest. Compare the result with the raw exports: you should see facts from all three sources gathered under one person, each fact still labeled with the graph it came from.

5.6 Walk the history — the best way to learn the system

Because every milestone is a tag, you can replay the build. This is the single most useful exercise in this guide:

git checkout part-01 && npm install && npm test
git checkout part-02 && npm install && npm test
# … and so on through part-09

At each tag, run the tests and read that part’s decision log. Here is what to look for (which questions pass at which tag is what the build planned; confirm it yourself by running the tests):

Tag	What exists	What to notice
part-01	Generator, fixtures, conflict manifest, all four questions skipped	The seeded conflicts; the empty scoreboard
part-02	Silver cleaning, identifier minting	The identifier rules in `src/identifiers`
part-03	Vocabulary, mapper, golden files	Zero blank nodes; the golden test
part-04	Store, graph layout, query tool, browser playground — Q3 passes	Subclass property paths doing semantic search
part-05	Matcher, links graph — Q1 passes	Cross-source questions answered through links
part-06	DCAT, PROV-O — Q2 and Q4 pass	Lineage as ordinary queries
part-07	Hashes, swaps, patches, convergence test	The byte-for-byte comparison
part-08	React front end, live patches	The store running in the browser
part-09	Scaled generator, measurements, verdict	Everything in section 6

Return to the final state with git checkout main (or whatever the default branch is called), then npm install again.

5.7 Experiments that teach

These are safe to try on a scratch branch; each one demonstrates a pattern from section 3.

Change one record. Edit one customer’s email in a raw export, run the pipeline again, and look at which entity graphs were regenerated. Only the changed entity’s graph should have been touched — that is content hashing plus graph swapping.
Delete one record. Remove a row from an export and run the pipeline. The entity’s graph should disappear; facts in other entities’ graphs that pointed at it remain (the project’s stated rule: the reference existed, its target is gone).
Break determinism on purpose. In a scratch copy of the mapper, add new Date() to a literal. The golden test fails; the convergence test fails. Revert. This is why the invariants exist.
Try to introduce a blank node. The mapper tests will refuse it. Read the test to see how it is enforced.
Retract the matcher. Drop the links graph (a single SPARQL DROP GRAPH), run Q1, and watch it collapse to single-source answers. Re-run the matcher and it comes back. Nothing else changed.

5.8 Scale up

npm run generate -- --scale 10
npm run pipeline
npm run bench

The scaled world contains every record of the small world unchanged, plus new records in reserved identifier ranges — so everything you learned at small scale still holds. npm run bench appends a row to docs/bench.csv with quad count, pipeline time, per-question latency, and memory.

For scale 50, the default Node heap is not enough. Raise it:

NODE_OPTIONS="--max-old-space-size=8192" npm run pipeline

Expect the pipeline to take around three minutes at scale 50 and to use about 5 GB of memory. Then read section 6 before drawing conclusions, because the most interesting result at scale is not about speed.

5.9 The front end

The web workspace added in part 8 has its own package scripts and its own prologue; start there for the run command. It loads the graph into an in-browser Oxigraph store and provides an entity page (one query per page), a neighborhood view, and a browse screen driven by the DCAT metadata. The live-update layer needs AWS credentials and an AppSync Events endpoint; the static catalog does not.

6. What the measurements showed, in plain terms

The build’s numbers (from docs/bench.csv):

World	Quads	Full pipeline	Slowest question	Memory
small (30 customers)	2,264	0.2 s	10 ms	232 MB
scale 10	98,440	7.6 s	172 ms	1.2 GB
scale 25	469,152	43.6 s	416 ms	2.8 GB
scale 50	1,693,064	171.5 s	816 ms	5.0 GB

What worked. The convergence test passed at every scale — the incremental path produced exactly the same store as a full rebuild. Queries stayed fast enough for a person at a keyboard even with 1.7 million quads.

The matcher’s precision collapsed at scale — and the data was not wrong. At scale 50, the matcher produced 3,481 correct links and 163,427 incorrect ones. Not one matching rule changed. What changed was the population. The synthetic generator draws names from a small list; in a world of thirty customers, two records sharing a name are almost certainly the same person, but in a world of fifteen hundred customers drawn from the same list, many different people share a name. The evidence lost its power because the population saturated it.

The everyday version of this is the birthday paradox: in a room of 23 people there is a fifty-percent chance two share a birthday, not because anyone’s birthday changed, but because 23 people is 253 pairs. Matching works on pairs, and pairs grow with the square of the population. The lesson generalizes far beyond this project: a rule validated on a small dataset carries no guarantee at a larger one, because precision is a property of the rule and the population together. The project’s verdict document calls this “population-relative discriminating power.”

Two practical notes. First, the magnitude here (about 98% of medium-confidence links wrong) depends on how small the synthetic name list is — real populations saturate more slowly, though they do saturate (common surnames, shared corporate email domains). Second, this is exactly why the links live in their own graph: retracting 163,427 bad links meant dropping one graph.

The matcher was slow at scale. It compared every person with every other person, so doubling the population quadrupled the work. This is a property of the simple implementation, not of the problem — the standard fix is “blocking”: only compare records that share a cheap key such as postal code or the first letters of a name.

Memory was the ceiling. The whole graph lives in RAM. With default settings the limit sits somewhere between half a million and a million quads. A production system would keep only the frequently used part of the graph in memory and generate the rest on demand from Parquet — the source-of-truth design makes that possible. Check docs/verdict.md for the breakdown of which graphs account for the quad count at scale; a large share of the scale-50 store may be matcher links rather than data.

The reservation lesson. Part 2 reserved an identifier range in a comment. Part 9 found four records already inside it, on the first run with an actual assertion. Written rules drift; tested rules hold.

7. Glossary

Blank node — a graph node with no identifier. Banned in this project because it cannot be compared across runs.
Blocking — in entity resolution, comparing only records that share a cheap key, to avoid comparing every pair.
Convergence property — the requirement that an incrementally updated store equals a fresh rebuild, byte for byte.
DCAT — a web standard vocabulary for describing datasets.
DuckDB — an in-process analytical SQL database that reads and writes Parquet directly.
Entity resolution — deciding whether two records refer to the same real-world thing.
Golden file — a checked-in copy of expected output that a test compares against exactly.
Identifier (IRI) — a web-style address that names a thing in the graph.
Literal — a typed value in a triple’s object position (string, number, date).
Medallion architecture — Bronze (raw), Silver (clean), Gold (serving) data layers.
N-Quads — a plain-text format with one quad per line.
Named graph — a labeled subset of a graph store; this project uses one per entity per source.
Natural key — a value from the real world that identifies a record (email, organization number), as opposed to a database-assigned number.
Ontology — a vocabulary plus relationships between its terms, such as subclass.
Oxigraph — the Rust graph database, used here as a WebAssembly package.
Parquet — a column-oriented, typed, compressed file format for tabular data.
PROV-O — a web standard vocabulary for describing how data was produced.
Property path — SPARQL syntax for following a chain of relationships (*, +).
Provenance — the record of where data came from and how it was processed.
Quad — a triple plus the name of the graph it belongs to.
RDF — the web standard for representing facts as triples.
Same-as link — a fact stating that two identifiers refer to the same thing.
SPARQL — the query language for RDF graphs.
Triple — a single fact: subject, predicate, object.
WebAssembly — a binary format that lets code written in languages like Rust run inside JavaScript engines, including browsers.

8. Where to go next

Run the tag walk in 5.6 before reading anything else. Seeing the four questions turn green one by one teaches the architecture faster than any explanation.
Read the decision logs in docs/decisions/ in order. The rejected alternatives explain the design better than the chosen ones.
For SPARQL, the W3C “SPARQL 1.1 Query Language” specification is dry but definitive; the property-path section is short and worth reading in full.
For the matcher result, look up the birthday paradox and the “base rate fallacy” — both are the same phenomenon the project measured.
When something in this guide and the repository disagree, open the relevant module’s prologue: the repository is the authority, and the prologues were written to be the first thing a newcomer reads.

← All articles