Backend systems
Ledgerline
An HTTP API on Cloudflare Workers for append-only event streams: exactly-once appends, gap-free order, and a SHA-256 hash chain that makes rewriting history detectable.
- Role
- Timeline
- Stack
- Links
By the numbers
99
tests in 15 files, run in the Workers runtime against real Durable Objects and a local D1
12
failure modes written down, each with what happens and the test that pins it
8
bugs found in review, each fixed and pinned by a regression test
1
runtime dependency (Hono)
- Client
- Worker
- Durable Object
- D1
- Client to Worker POST /v1
/streams with an Idempotency-Key/:id /events - Worker Key, rate and owner checks The API key is looked up by its SHA-256 hash, its token bucket spends one token (429 with Retry-After when empty), and the stream must belong to the key.
- Worker to Durable Object append(payload, key) routed by idFromName(streamId)
- Durable Object block
Concurrency A known key with the same body returns the original {seq, hash}; a different body is a conflict. Otherwise seq + 1 and the next hash.While - Durable Object One atomic write The event, the key record, the minute rollup, the stream head and the outbox alarm. All or nothing.
- Durable Object to Worker {seq, hash}
- Worker to D1 INSERT OR IGNORE The fast path, so a read right after a write usually sees it. If it fails, the client still gets its answer.
- Worker to Client 201 {seq, hash} A replay gets 200 with Idempotent-Replay: true; a different body gets 422
- Durable Object to D1 alarm(): outbox sweep About 5 s later it inserts only the rows D1 is missing, and retries every 30 s while D1 fails.
Hash chain
- genesis SHA-256("ledgerline:v2:" + stream
Id) - h1 event 1
- h2 event 2
- h3 event 3
hn = SHA-256(hn-1 | RFC 8785 canonical JSON | n)
/verify recomputes the chain from genesis, checking every seq, prevHash and hash, then compares the result with the committed head, so a deleted tail fails too.
The idea
Ledgerline is an HTTP API for append-only event streams: ledgers, audit trails and other records where history must not change after the fact. A retried append takes effect once, every stream keeps a gap-free order, and a hash chain makes a later edit, deletion or cut tail detectable.
What I built
Each stream is one Durable Object on Cloudflare Workers. Inside blockConcurrencyWhile it checks the idempotency key, assigns the next sequence number, links the event into the hash chain, and commits the event, the key record, the minute rollup, the stream head and an outbox alarm in one atomic write, so an append lands completely or not at all. D1 holds a read model for paginated history: the Worker mirrors each event right away, and the outbox alarm delivers anything that insert missed, so the copy can lag but cannot lose an event.
Each event is hashed over RFC 8785 canonical JSON together with the previous hash and its sequence number, and that rule is versioned per stream. API keys are stored only as SHA-256 hashes, a request for another key’s stream gets 404, and each key has its own token bucket that returns 429 with Retry-After. The 99 tests run in the Workers runtime against real Durable Objects and a local D1, with D1 failures injected on purpose, and CI runs them on every push to main and every pull request.
The hard parts
-
Safe retries, honest conflicts
An idempotency key is kept for the life of its stream. A retry with the same body, compared in canonical form, returns the original {seq, hash} with Idempotent-Replay: true, even long after the first request. A retry with a different body gets 422 and writes nothing, so a client bug is never reported as a success. A test sends 5 requests with one key at the same time and expects one event.
-
Ordering across awaits
A Durable Object runs one request at a time, but an await, such as the SHA-256 call, can let a second append start before the first one finishes. Every append runs inside blockConcurrencyWhile. The test makes the digest yield so 20 appends really interleave: they get seq 1 to 20, and with the guard removed the test fails.
-
Fixing the hash rule without breaking history
Conformance tests against RFC 8785’s own examples showed that the first canonicalizer put integer-like keys such as "9" and "10" out of order. Fixing it in place would have made untouched streams with such keys fail /verify. The fix shipped as chain version 2: each stream keeps the version it was created under, the version is part of the genesis hash, and version 1 streams keep appending and verifying.
Before and after
| Before | After | Source | |
|---|---|---|---|
| Reusing an Idempotency-Key with a different body | Answered as a successful replay of the original event. | 422 idempotency_key_reused, and nothing is written. | Commit ff5270e, October 9, 2026 |
| Delivery to the D1 read model | One insert after the commit; a failed insert could leave a permanent hole. | An outbox alarm, armed in the same atomic write, inserts only the rows D1 is missing and retries every 30 s until they land. | Commits d2404dc and 4650d2a, October 9, 2026 |
| Canonical JSON for the hash | Keys re-sorted, then JSON.stringify, which puts integer-like keys out of RFC 8785 order. | RFC 8785, shipped as chain version 2; streams created under version 1 still verify. | Commits e4db446 and a6923e9, October 9, 2026 |
| Tests in the Workers runtime | 69 in 14 files. | 99 in 15 files, and CI fails if the README or the live page quotes another count. | test/*.test.ts on August 2 and October 9, 2026 |
What’s next
Signed checkpoints of the stream head, published outside the server, so /verify can start from the last one and a consistent rewrite of the whole log is caught too.
Try it
Related by idea
Tests against the real runtime or the real data
Next project
