# torly architecture torly is the relay core of orly, rebuilt for TinyGo around the storage idea used by morly: immutable signed messages in an append-only log, with sorted flat-file indexes beside it. This document describes the pieces and the constraints that shaped them. ## 1. Why the shape changed orly stores events in Badger (memory-mapped LSM with a value log) and offers Neo4j and gRPC backends. None of those build under TinyGo: Badger depends on `mmap`, `unix` syscalls and reflection-heavy generic code, SQLite needs cgo, and gRPC/Neo4j pull in large runtimes. Porting "the whole of orly" would have meant reimplementing each subsystem before the first event could be stored. Events are immutable and signed. That makes a value log of encoded events the natural primary store, and it makes indexes the only mutable state. morly's pogreb-inspired engine already has that split, so torly reuses the design: - **WAL**: append-only, segmented, length- and CRC-framed. - **Indexes**: fixed-width sorted records. New writes become immutable sorted runs; a merge folds runs into one `.dat` file. - **Checkpoint**: the WAL position whose records are durable in the indexes. Nothing is memory-mapped and no record is updated in place. ## 2. Domain tree and run model The relay is a fixed tree of domains built on tiny-spawn, which models Moxie's `spawn(fn, data, chans)` on TinyGo. The tree is created once at startup; no domain is spawned per connection or per job. | Domain | Owns | Channels | |---|---|---| | root (`main`) | configuration value, breakers, lifecycle channels | wiring only | | acceptor | listening fd, poll state, connection IDs | `[]Assign` out | | reader[N] | read side, HTTP and WebSocket parsing, per-connection limits | assign in, `Inbound` out, `WriterCtl` out, ack in | | writer[N] | write side, frame/response serialization, socket close | `WriterCtl` in, `Outbound` in, ack out | | net | connection table, subscriptions, auth, pending query/verify/blob work | workers, DB, verifiers, Blossom | | db | store engine, WAL, indexes, expiry ticker | request/response duplex | | verify[V] | no state; one signature check at a time | request/response duplex | | blossom | blob index, blob files, BUD authorization | request/response duplex | `N = TORLY_CONN_WORKERS` and `V = TORLY_VERIFY_WORKERS`. Every channel is created by the root before the domain starts; no channel ever travels in a message. Each direction has one writer and one reader. A reader and its paired writer split one socket capability: the reader only reads and sets read deadlines, the writer only writes, frames, and closes. That is the only shared capability, and it is an fd-level one, not application state. Messages are values (`pkg/wire`). Slices, maps and the socket are handed over with ownership: the sender does not touch them after the send. Events and filters crossing to the database are cloned. No domain holds a lock that another domain takes; the tree contains no `sync.Mutex` over shared state. Shutdown is a breaker. Closing it wakes every domain: the acceptor's poll and the reader/writer deadlines return within 200 ms, the network loop within its 2 ms tick, and the database and Blossom domains close their files. The root waits for every lifecycle channel. ## 3. Serial numbers A serial is a physical address in the log: ``` serial = segment << 32 | offset ``` It is stored in index keys as a uint40 (5 bytes, big-endian). The top byte of a serial therefore limits the log to 256 segments, and segments are capped at 1 GiB, giving a 256 GiB ceiling. That is far above what a single-process relay needs and keeps every index record small. ## 4. Write-ahead log `wal/seg-NNN.dat` holds records: ``` uint32 length | uint32 crc32(payload) | payload ``` The payload is the event in a compact binary form: ``` version(1) kind(2) created_at(8) pubkey(32) id(32) sig(64) tag count(2), per tag: field count(2), per field: length(4) + bytes content length(4) + bytes ``` Reading an event is one `ReadAt` for the header and one for the payload. Appends are synced every 100 records and at every checkpoint. On open, the last segment is scanned and a torn tail (a crash in the middle of a write) is truncated. ## 5. Indexes Every index is a file of fixed-width records, sorted by their key bytes. A record is a key (a three-byte prefix plus fixed fields) optionally followed by a value. The serial is the last five bytes of the key for every scan index. | Index | Key | Purpose | |---|---|---| | `eid` | id hash (8) + serial | id → serial, duplicate detection | | `sei` | serial | serial → full 32-byte id | | `fpc` | serial | serial → id + pubkey hash + timestamp | | `ca` | created_at + serial | unfiltered, time-ordered scans | | `kc` | kind + created_at + serial | kind scans | | `pc` | pubkey hash + created_at + serial | author scans | | `kpc` | kind + pubkey hash + created_at + serial | kind + author | | `tc` | tag letter + value hash + created_at + serial | single-letter tag | | `tkc` | kind + tag letter + value hash + created_at + serial | + kind | | `tpc` | pubkey hash + tag letter + value hash + created_at + serial | + author | | `tkp` | kind + pubkey hash + tag letter + value hash + created_at + serial | all three | | `wrd` | word hash + serial | NIP-50 search | | `exp` | expiration + serial | NIP-40 purge | | `rep` | kind + pubkey hash + d hash + created_at + serial | replacement slot | | `del` | full 32-byte id | NIP-09 tombstone | | `pks` | pubkey hash + pubkey serial | intern a pubkey for graph edges | | `spk` | pubkey serial | serial → full 32-byte pubkey | | `epg` | event serial + pubkey serial + kind + direction | event → pubkey | | `peg` | pubkey serial + kind + direction + event serial | pubkey → event | | `eeg` | source serial + target serial + kind + direction | event → event (e-tag out) | | `gee` | target serial + kind + direction + source serial | event → event (e-tag in) | | `ppg` | source pk + target pk + kind + direction + event serial | pubkey → pubkey | | `gpp` | target pk + kind + direction + source pk + event serial | pubkey → pubkey (in) | `eid` and `pks`-style hashed indexes compare only the hash part: a point lookup returns the first record for the hash and the caller verifies the full id, so a hash collision cannot produce a false positive. ### Runs `QuickFlush` writes the in-memory records as a new, immutable, individually sorted file `name.dat.buf.NNNNNN`. Appending batches to one shared sidecar would not keep it sorted — two sorted runs concatenated are not a sorted array — and binary search over that file would silently miss records. Separate run files keep every source sorted. Reads consult, newest first: the memory buffer, the runs (newest run first), and the merged main file. A scan collects matching records from every source, sorts them by key, and drops exact duplicates, so replayed or overlapping runs are harmless. `Flush` merges the main file and all runs into a new `.dat` (written to `.tmp`, fsynced, renamed). Records whose key is marked deleted are dropped. The run files are removed after the rename. The runs' total size is bounded by `max(1 MiB, main/10)`, so the in-memory merge is bounded too. ### Deletion `Delete` records the key in an in-memory set consulted by `Get` and `Scan`, and is applied physically by the next merge. Physical immediacy is not needed: visibility is what matters, and the merge cannot resurrect a key that is still marked. NIP-09 goes one step further: the event id is written to the `del` index, which is never compacted away. A rebuild from the WAL checks the tombstone before indexing, so a deleted event cannot come back after a restart. ## 6. Recovery `checkpoint.dat` stores the next WAL position and the next pubkey serial, protected by a CRC: ``` "TCK1" | position(8) | next_pubkey_serial(8) | crc32(first 20) ``` Recovery at open: 1. Repair the WAL tail. 2. Read the checkpoint. A position beyond the WAL's write position is stale, so the index is rebuilt from offset zero. 3. Replay records after the position. Replay is not a blind re-index: it applies replacement and deletion semantics in serial order, skips tombstoned ids, and re-derives the word, tag and replacement indexes. 4. Make the result durable (sync WAL and runs, then advance the checkpoint). A checkpoint is only advanced after the WAL and every index run have been fsynced. That ordering is the whole durability contract: losing a checkpoint costs a replay, but it cannot lose an event that `OK` acknowledged. A store written before the graph indexes existed has a valid checkpoint but no `pks`/`spk` rows. Open detects that and rebuilds from offset zero, so an upgrade does not leave old events outside the graph. ## 7. Query planning `Engine.candidates` picks the most selective index available: ``` tags + kinds + authors -> tkp kinds + authors -> kpc tags + kinds -> tkc kinds -> kc tags + authors -> tpc authors -> pc tags -> tc none -> ca ids -> eid point lookups search -> wrd intersection ``` Each index family is scanned over `[since, until+1)` ranges. When `until` is unbounded the range end is the successor of the fixed key prefix rather than a fabricated maximum timestamp, which avoids overflow and off-by-one errors with short bounds. Candidates are deduplicated by serial, decoded from the WAL, filtered for tombstones, expiration and the full filter semantics, then ordered newest first (ties broken by lowest id, as NIP-01 specifies). `limit` and the engine's result cap are applied last. `COUNT` walks the same candidate set without materializing events. The planner scans one index family and then filters, so a filter mixing several tag values unions their ranges and relies on `Matches` for the conjunction. It is exact, not approximate, but it is not a cost-based optimizer. ### Bulk import `torly import` opens the engine with `Options.BulkImport`. In that mode, `SaveEvent` writes the WAL and all ordinary indexes but defers graph edges, and the sorted files keep every buffer as an immutable run instead of merging as the main file grows. Repeated merge-rewrites are what make a naive import superlinear, so deferring them matters more than any per-record optimization. Replacement slots are tracked in memory (`bulkRep`), seeded from the persisted `rep` index when the import starts against an existing store. At the end, `RebuildGraph` clears `pks`/`spk` and the six edge indexes, resets the pubkey serial counter, and walks every live `sei` record once. All e-tag targets are present by then, so graph edges are created without the per-event backfill scan. The final `Flush` merges the accumulated runs into each `.dat` file. An interrupted import is safe: the WAL and checkpoint recovery still own durability, and a repeated import skips ids that are already stored. ## 8. Canonical JSON An event id is SHA-256 over: ``` [0,"",,,,""] ``` NIP-01 defines this in terms of JavaScript's `JSON.stringify`. Go's `encoding/json` escapes `<`, `>` and `&` as `\u003c` etc. by default and would produce a different hash for content containing them. `nostr.AppendJSONString` implements the JS rules directly: short escapes for quotes, backslash and `\b\f\n\r\t`, `\u00XX` for other C0 controls, and raw UTF-8 for everything else, including U+2028/U+2029. The test suite pins the result against a string and hash produced by Node. ## 9. Wire JSON codec The event, filter and relay-message parser is hand-written and copied into `pkg/nostr/jsoncodec.go` from the Moxie/Nostr decoder. It is a single pass over the bytes with no `encoding/json`: object keys are compared as strings, `SplitArray` returns the raw elements of a relay message, and typed readers consume event and filter fields. Two edge cases are explicit: - **Empty object keys.** `{"":1}` is legal JSON. The reference decoder did `switch key[0]` after reading a key, which panicked with an index-out-of-range when the key was empty. Here `len(key) == 0` is checked before any indexing and returns `nostr: empty JSON object key`. The lenient `ExtractID` path skips the value instead, so a malformed event can still receive an `OK`. - **Surrogate bounds.** A `\uD800` high surrogate must be followed by `\uDC00..\uDFFF`. The lookahead requires six bytes with `p.i+6 > len` rejecting, rather than the reference's `i+10 <= len` which permitted `s[i+10]` on a truncated input. `encoding/json` is still used to *encode* NIP-11 and Blossom responses; it is no longer used to parse anything that comes from the network. ## 10. Transport The acceptor owns a non-blocking IPv4 socket created with `x/sys/unix`. It polls with a 200 ms timeout, accepts with `accept4`, assigns the fd to a reader worker in round-robin order, and backs off only when every worker already has a queued connection. The worker wraps the fd in `fdConn`, a `net.Conn` whose reads and writes use `poll` rather than blocking syscalls. A fixed pool of reader/writer pairs follows. A reader owns the read side and the parsing state for exactly one connection at a time: it parses HTTP with a transport-neutral version of `httpsrv.ReadRequest`, serves NIP-11 and static files locally, forwards health and Blossom to the network domain, and after an upgrade switches to the RFC 6455 reader. The WebSocket reader returns control frames to the reader (it never writes), while the paired writer serializes text, pong and close frames. The writer is the only domain that writes or closes the socket. Responses are one request per connection with `Connection: close`, except for an upgraded WebSocket, which stays open until either side closes. Idle WebSockets are closed after 150 s. A reader waits for the writer's bind/upgrade acknowledgement before handing the connection to the network domain, so a frame can never overtake the handshake. This is hand-written because TinyGo's `net/http` implements `Hijack` but closes the hijacked connection as soon as the handler returns, and writes to it never reach the peer. ## 11. Subscriptions and authentication The network domain is the single owner of connection state and subscriptions. It polls the reader inboxes, the database responses, the verifier responses and the Blossom responses on a 2 ms tick, so no lock is needed: only one goroutine mutates the tables. `REQ` validates the filters, registers them on the connection, and sends a database query. The database answers with owned event values; the network domain frames them, sends `EOSE`, and keeps the filters for live delivery. Broadcasts walk the subscription table and match filters (including NIP-50 token sets) before sending frames to the owning writer. NIP-42 sends a per-connection random challenge in an `AUTH` message after the upgrade. An AUTH event must be kind 22242, close to now, carry the challenge and a relay URL matching `TORLY_RELAY_URL`, and pass the same verifier pool as events. `TORLY_AUTH_REQUIRED` gates writes; NIP-70 events are restricted to their author. Test failures and protocol errors are returned as `NOTICE` or `CLOSED` values. Outbound sends to a writer are non-blocking. A writer whose queue is full is treated as a slow consumer: the network domain stops addressing that connection, and its reader observes the eventual close. ## 12. Blossom Blobs are content-addressed at `blossom/blobs//`. An append-only binary index log records `(op, hash, pubkey, size, created, type)` so listing by pubkey and serving `Content-Type` do not need to open the blob. The log is compacted on open when it exceeds 4 MiB. The Blossom domain owns the server value and answers `BlobRequest` with a `BlobResponse`; the reader forwards blob paths through the network domain, so no HTTP handler shares the index. Authorization is a signed kind 24242 event in the `Authorization: Nostr ` header: an unexpired `expiration`, a `t` tag for the action (`upload`, `delete`, `list`) and, for blob actions, an `x` tag matching the hash. Uploads verify that the declared `x` equals the SHA-256 of the body; deletes require the requester to own the blob. ## 13. TinyGo notes Several TinyGo runtime behaviours needed workarounds, all documented in the code: - **Signals.** The runtime records a signal in a mask and hands it to `os/signal` only when the scheduler reaches an idle wait. A relay keeps a socket in netpoll, so that path is never reached and `SIGTERM` would be ignored. `pkg/sigtrap` linknames the runtime's `checkSignals` and calls it every 200 ms; the non-TinyGo build makes `Trap` a no-op. - **Listener deadlines are a no-op.** TinyGo's `TCPListener.SetDeadline` does nothing and `Accept` is not woken by `Close`, so a listener domain would never observe shutdown. The acceptor therefore uses a non-blocking `x/sys/unix` socket and `poll` with a 200 ms timeout, and the workers use `fdConn`, which also polls rather than blocking in `read(2)`. - **Port 0.** The raw listener reports its bound port through `getsockname`, so it works with an ephemeral port. Production configuration uses explicit ports; the tests still pick and retry a port. - **Close is not idempotent.** TinyGo's `TCPConn.Close` calls `close(2)` unconditionally; a double close can close an unrelated descriptor that reused the number. `fdConn` closes exactly once (`sync.Once`) and shuts the socket down first so a polling reader wakes. The transport tests are deliberately timing-sensitive in one direction: they assert framing, shutdown, and error behaviour on real loopback sockets, and they run under both toolchains. The last two TinyGo issues above were found by those tests rather than by reading the runtime. **Schnorr performance.** orly's `pkg/p256k1` has fast paths, but its amd64 assembly/4×64 implementations are excluded under TinyGo, and the portable 32-bit branch it leaves for TinyGo rejects valid BIP-340 vectors on linux/amd64 (tested: vector 0 returns false). torly therefore verifies 32-byte messages through btcec's pure-Go BIP-340 implementation, which carries the same 5×52/10×26 field arithmetic and passes the full vector suite. Under TinyGo on this machine a verification takes about 0.3 ms, down from 6.4 ms for the `math/big` reference. The reference implementation is kept for BIP-340's variable-length messages and as an independent cross-check: the tests sign with both implementations and verify each other's output. ## 14. Social graph indexes and `_graph` The graph indexes are materialized from tags at write time. Every edge is written twice, once in each direction, so a traversal is one prefix scan and a deletion is also one prefix scan: ``` epg event -> pubkey peg pubkey -> event eeg event -> event gee event -> event ppg pubkey -> pubkey gpp pubkey -> pubkey ``` A pubkey is interned in `pks`/`spk` first, so an edge carries two five-byte serials rather than two 32-byte keys. The author edge has direction 0, a p-tag outbound edge 1, and its reverse inbound edge 2. E-tag edges use 0/1 and pubkey-pubkey edges use 0/1. These direction bytes are on-disk values. `epg`/`peg` are written for the author and every valid `p` tag. `ppg`/`gpp` are written for every valid `p` tag except the author's own pubkey. `eeg`/`gee` are written for every valid `e` tag whose target is already in the store. When the target arrives later, `backfillInboundETags` scans the `tc` index for `#e` references to the new id and creates the missing edge, so publishing a reply before its root does not leave a permanently incomplete thread. The backfill also means the tag index, not the WAL, is the source of truth for the reference direction. The `_graph` filter extension is parsed in `pkg/nostr/jsoncodec.go` and executed by `Engine.TraverseGraph`: | Edge | Seed | Direction | Result | |---|---|---|---| | `pp` | pubkey | `out` follows, `in` followers, `both` | pubkeys by depth | | `pe` | pubkey | `out` authored, `in` p-tag mentions, `both` | events at depth 1 | | `ee` | pubkey | `out` e-tag targets, `in` referencing events, `both` | events by depth | `pp` walks only kind-3 contact-list edges, matching orly's follows semantics. `ee` starts from every event authored by the seed pubkey; those roots are not part of the result. `pe` is deliberately one hop: the pubkey↔event graph is bipartite, so a second hop would return a different node type. Traversal is breadth-first, deduplicates by serial, stops when a frontier is empty, and caps the result at the engine's `MaxResults`. Results are sorted at every depth so the signed response is deterministic. The response is a relay-signed kind-39000 event whose content is `pubkeys_by_depth` or `events_by_depth`; the relay key comes from `TORLY_SECRET_KEY` or, when that is unset, `DATA_DIR/relay.key`, created on first start. ## 15. What was dropped, and why | orly subsystem | torly | |---|---| | Badger, Neo4j, SQLite, gRPC backends | one WAL + sorted indexes | | ACME/Let's Encrypt | TLS terminated by a reverse proxy | | Tor, WireGuard | not ported | | NIP-46 bunker, NIP-86 admin, payments, bridge, blossom media proxy | not ported | | SQL/Badger/Neo4j graph backends | materialized `epg`/`peg`/`eeg`/`gee`/`ppg`/`gpp` indexes and `_graph` | | Svelte web dashboard | deferred; `TORLY_STATIC_DIR` serves a WASM build later | The relay protocol, canonical encoding, replacement/deletion/expiry semantics and Blossom endpoints are the parts that make a deployment usable, and those are implemented and exercised end to end.