How we made every record carry its own justification

Most systems can name the file a record came from. They can't tell you if that file has since changed, or been deleted. We built a document model that can, down to the exact sentence.

Ask almost any system where a record came from and it hands you a filename: source: "policy.md". A filename isn't a source. The W3C's definition of provenance requires "information which can be used to form assessments about quality, reliability or trustworthiness." A filename has no text, no version, and no index behind it, so it can't support one. When the file changes, nothing downstream finds out.

Hyperstruck replaces the filename with an address into an immutable, versioned document, indexed in both directions. Three stores are involved, and each one only holds what it's actually good at holding:

Three stores: a relational database holds documents, versions, units, citations, the reverse index, and the lineage graph as rows and a recursive query, all in one engine. A blob store holds the encrypted original bytes and text, one wrapped key per document. A vector database holds one embedding per unit for semantic recall. An erasure touches all three, with the database and blob store cleared by destroying a key and the vector database cleared by deleting and then counting.

The rest of this post is about the top two rows: how a citation addresses into them, and what erasing one actually does.

The object model there is borrowed straight from git's own: a raw upload is a commit, its extracted text is a tree, and each unit of that text is a blob. All three are content-addressed and immutable, so nothing about an existing address can ever move. And just as two commits can point at the same tree, two uploads of byte-identical text share one set of units rather than duplicating them.

A Citation struct resolved through four layers: doc_id to the document, text_sha256 to the version, segmenter_version and unit_index to a specific unit, and start to a character offset inside it.

The address, not the filename

Every record Hyperstruck forms (a learning, a claim, an obligation) carries the same Citation:

doc_id: str
text_sha256: str
segmenter_version: str
unit_index: int
start: int          # character offset the passage starts at

Four coordinates: which document, which version of its text, which unit inside that version, and the character offset where the passage starts inside that unit. Not "this came from the policy," but this exact sentence, this exact version, resolvable back to the source at any time.

Why the version has to declare itself

A diff between two document versions can tell you what text changed. It can't tell you what the change means, and that's the part a downstream record needs. Take a payments policy: someone uploads a full rewrite that drops the clause about a surcharge, without saying so anywhere. Is the surcharge gone, or did the rewrite just not bring it up? So a version states its intent explicitly, at upload:

amends: changes what it mentions, leaves the rest standing. The default.
replaces: restates its subject as a full substitute for the ground it covers.

A version also declares which subjects it covers completely, via complete_scopes. That's what makes an omission decisive. If the rewrite declares replaces and lists payments as a complete scope, then leaving out the surcharge means the surcharge is gone. If it declares amends, or never claims complete coverage of payments, leaving out the surcharge means nothing: it never promised to mention everything.

Both fields are just two more headers on the same upload call:

PUT /v1/agents/{agent_id}/documents/finance-policy
x-document-filename: payments-policy-v2.pdf
x-document-type: markdown
x-document-relation: replaces
x-document-complete-scopes: payments

<the document's bytes>

Two rows comparing an identical omission: a version declaring replaces and complete coverage of payments retires the missing clause, while a version declaring amends with no complete scope leaves the same omission meaning nothing.

We could infer amends vs. replaces from a diff and be right most of the time. Two fields at upload beat a model's best guess on which of a customer's policies still bind.

The query that has to be fast

Given a citation, resolving the passage it points to is cheap: one lookup by primary key. The query that matters is the reverse one: given a document version, name every record that cites it. That's what a withdrawal needs to run, and it needs to run over a corpus, not a sample.

We keep a dedicated reverse-index table for this, record_support_edges, keyed on the unit rather than scanned out of a general-purpose index over the citation column:

Forward resolution of a citation to its unit is a cheap primary-key lookup. The reverse query, unit to every citing record over 25,000, costs 41 milliseconds scanning a general-purpose index over the citation column versus 5 milliseconds against a dedicated reverse-index table.

An 8x difference on one lookup sounds like a micro-benchmark. It's the difference between a check that runs automatically before every withdrawal and one that gets skipped because it's slow enough to notice.

What an upgrade to our own segmenter can't break

We improve the segmenter that splits documents into units. Each improvement redraws unit boundaries. If a citation's identity were just "unit 14," an upgrade would silently move the text behind every existing citation, same id, different sentence.

segmenter_version is inside the identity, not beside it, so an upgrade mints a new sibling set of units rather than mutating the old ones:

The same citation on the day the segmenter improves. Without the segmenter version in the unit id, the id survives and the words behind it change silently. With it, the upgrade mints a sibling set and the original citation still resolves to the words it was written against.

Old citations keep resolving to exactly what they were written against. We treat this as a constraint on ourselves: no improvement we ship is allowed to move a customer's existing record.

What deletion actually deletes

A citation never carries a copy of the text, only the address. So deletion is one act, not a cleanup job: destroy the key. That's cryptographic erasure, and once the key is gone the ciphertext sitting in the blob store is unreadable, whether or not the object itself is deleted a moment later. Every citation into that document now resolves to "withdrawn" instead of the sentence.

The vector database is different: it holds a derived copy, not ciphertext, so there's no key to destroy. Its points get deleted directly and then counted to confirm it. The receipt reports "key destroyed" for the text and "deleted, verified" for the copy, because those are two different guarantees, and claiming the stronger one for both would be a lie.

One key, one act, no copies left behind to outlive the document they came from.

In Conclusion

Every record Hyperstruck holds points at one exact sentence, in one exact version. That address survives our own upgrades, gets checked in milliseconds instead of a corpus walk, and goes stale the moment its source does.

Check the address, see which version it points to, see what that version declared. No guessing, no corpus walk, and no agent quietly repeating a rule six months after the company dropped it.