torly is the relay core of orly, rebuilt for TinyGo around the storage idea used by morly: immutable signed messages in an append-only log, with sorted flat-file indexes beside it. This document describes the pieces and the constraints that shaped them.
orly stores events in Badger (memory-mapped LSM with a value log) and offers
Neo4j and gRPC backends. None of those build under TinyGo: Badger depends on
mmap, unix syscalls and reflection-heavy generic code, SQLite needs cgo,
and gRPC/Neo4j pull in large runtimes. Porting "the whole of orly" would have
meant reimplementing each subsystem before the first event could be stored.
Events are immutable and signed. That makes a value log of encoded events the natural primary store, and it makes indexes the only mutable state. morly's pogreb-inspired engine already has that split, so torly reuses the design:
runs; a merge folds runs into one .dat file.
Nothing is memory-mapped and no record is updated in place.
The relay is a fixed tree of domains built on tiny-spawn, which models Moxie's
spawn(fn, data, chans) on TinyGo. The tree is created once at startup; no
domain is spawned per connection or per job.
| Domain | Owns | Channels |
|---|---|---|
root (main) | configuration value, breakers, lifecycle channels | wiring only |
| acceptor | listening fd, poll state, connection IDs | []Assign out |
| reader[N] | read side, HTTP and WebSocket parsing, per-connection limits | assign in, Inbound out, WriterCtl out, ack in |
| writer[N] | write side, frame/response serialization, socket close | WriterCtl in, Outbound in, ack out |
| net | connection table, subscriptions, auth, pending query/verify/blob work | workers, DB, verifiers, Blossom |
| db | store engine, WAL, indexes, expiry ticker | request/response duplex |
| verify[V] | no state; one signature check at a time | request/response duplex |
| blossom | blob index, blob files, BUD authorization | request/response duplex |
N = TORLY_CONN_WORKERS and V = TORLY_VERIFY_WORKERS. Every channel is
created by the root before the domain starts; no channel ever travels in a
message. Each direction has one writer and one reader. A reader and its paired
writer split one socket capability: the reader only reads and sets read
deadlines, the writer only writes, frames, and closes. That is the only shared
capability, and it is an fd-level one, not application state.
Messages are values (pkg/wire). Slices, maps and the socket are handed
over with ownership: the sender does not touch them after the send. Events and
filters crossing to the database are cloned. No domain holds a lock that another
domain takes; the tree contains no sync.Mutex over shared state.
Shutdown is a breaker. Closing it wakes every domain: the acceptor's poll and the reader/writer deadlines return within 200 ms, the network loop within its 2 ms tick, and the database and Blossom domains close their files. The root waits for every lifecycle channel.
A serial is a physical address in the log:
serial = segment << 32 | offset
It is stored in index keys as a uint40 (5 bytes, big-endian). The top byte of a serial therefore limits the log to 256 segments, and segments are capped at 1 GiB, giving a 256 GiB ceiling. That is far above what a single-process relay needs and keeps every index record small.
wal/seg-NNN.dat holds records:
uint32 length | uint32 crc32(payload) | payload
The payload is the event in a compact binary form:
version(1) kind(2) created_at(8) pubkey(32) id(32) sig(64)
tag count(2), per tag: field count(2), per field: length(4) + bytes
content length(4) + bytes
Reading an event is one ReadAt for the header and one for the payload.
Appends are synced every 100 records and at every checkpoint. On open, the
last segment is scanned and a torn tail (a crash in the middle of a write) is
truncated.
Every index is a file of fixed-width records, sorted by their key bytes. A record is a key (a three-byte prefix plus fixed fields) optionally followed by a value. The serial is the last five bytes of the key for every scan index.
| Index | Key | Purpose |
|---|---|---|
eid | id hash (8) + serial | id → serial, duplicate detection |
sei | serial | serial → full 32-byte id |
fpc | serial | serial → id + pubkey hash + timestamp |
ca | created_at + serial | unfiltered, time-ordered scans |
kc | kind + created_at + serial | kind scans |
pc | pubkey hash + created_at + serial | author scans |
kpc | kind + pubkey hash + created_at + serial | kind + author |
tc | tag letter + value hash + created_at + serial | single-letter tag |
tkc | kind + tag letter + value hash + created_at + serial | + kind |
tpc | pubkey hash + tag letter + value hash + created_at + serial | + author |
tkp | kind + pubkey hash + tag letter + value hash + created_at + serial | all three |
wrd | word hash + serial | NIP-50 search |
exp | expiration + serial | NIP-40 purge |
rep | kind + pubkey hash + d hash + created_at + serial | replacement slot |
del | full 32-byte id | NIP-09 tombstone |
pks | pubkey hash + pubkey serial | intern a pubkey for graph edges |
spk | pubkey serial | serial → full 32-byte pubkey |
epg | event serial + pubkey serial + kind + direction | event → pubkey |
peg | pubkey serial + kind + direction + event serial | pubkey → event |
eeg | source serial + target serial + kind + direction | event → event (e-tag out) |
gee | target serial + kind + direction + source serial | event → event (e-tag in) |
ppg | source pk + target pk + kind + direction + event serial | pubkey → pubkey |
gpp | target pk + kind + direction + source pk + event serial | pubkey → pubkey (in) |
eid and pks-style hashed indexes compare only the hash part: a point lookup
returns the first record for the hash and the caller verifies the full id, so a
hash collision cannot produce a false positive.
QuickFlush writes the in-memory records as a new, immutable, individually
sorted file name.dat.buf.NNNNNN. Appending batches to one shared sidecar
would not keep it sorted — two sorted runs concatenated are not a sorted
array — and binary search over that file would silently miss records. Separate
run files keep every source sorted.
Reads consult, newest first: the memory buffer, the runs (newest run first), and the merged main file. A scan collects matching records from every source, sorts them by key, and drops exact duplicates, so replayed or overlapping runs are harmless.
Flush merges the main file and all runs into a new .dat (written to
.tmp, fsynced, renamed). Records whose key is marked deleted are dropped.
The run files are removed after the rename. The runs' total size is bounded by
max(1 MiB, main/10), so the in-memory merge is bounded too.
Delete records the key in an in-memory set consulted by Get and Scan, and
is applied physically by the next merge. Physical immediacy is not needed:
visibility is what matters, and the merge cannot resurrect a key that is still
marked.
NIP-09 goes one step further: the event id is written to the del index, which
is never compacted away. A rebuild from the WAL checks the tombstone before
indexing, so a deleted event cannot come back after a restart.
checkpoint.dat stores the next WAL position and the next pubkey serial,
protected by a CRC:
"TCK1" | position(8) | next_pubkey_serial(8) | crc32(first 20)
Recovery at open:
the index is rebuilt from offset zero.
replacement and deletion semantics in serial order, skips tombstoned ids, and re-derives the word, tag and replacement indexes.
A checkpoint is only advanced after the WAL and every index run have been
fsynced. That ordering is the whole durability contract: losing a checkpoint
costs a replay, but it cannot lose an event that OK acknowledged.
A store written before the graph indexes existed has a valid checkpoint but no
pks/spk rows. Open detects that and rebuilds from offset zero, so an
upgrade does not leave old events outside the graph.
Engine.candidates picks the most selective index available:
tags + kinds + authors -> tkp kinds + authors -> kpc
tags + kinds -> tkc kinds -> kc
tags + authors -> tpc authors -> pc
tags -> tc none -> ca
ids -> eid point lookups
search -> wrd intersection
Each index family is scanned over [since, until+1) ranges. When until is
unbounded the range end is the successor of the fixed key prefix rather than a
fabricated maximum timestamp, which avoids overflow and off-by-one errors with
short bounds.
Candidates are deduplicated by serial, decoded from the WAL, filtered for
tombstones, expiration and the full filter semantics, then ordered newest
first (ties broken by lowest id, as NIP-01 specifies). limit and the engine's
result cap are applied last. COUNT walks the same candidate set without
materializing events.
The planner scans one index family and then filters, so a filter mixing several
tag values unions their ranges and relies on Matches for the conjunction. It
is exact, not approximate, but it is not a cost-based optimizer.
torly import opens the engine with Options.BulkImport. In that mode,
SaveEvent writes the WAL and all ordinary indexes but defers graph edges, and
the sorted files keep every buffer as an immutable run instead of merging as
the main file grows. Repeated merge-rewrites are what make a naive import
superlinear, so deferring them matters more than any per-record optimization.
Replacement slots are tracked in memory (bulkRep), seeded from the persisted
rep index when the import starts against an existing store. At the end,
RebuildGraph clears pks/spk and the six edge indexes, resets the pubkey
serial counter, and walks every live sei record once. All e-tag targets are
present by then, so graph edges are created without the per-event backfill
scan. The final Flush merges the accumulated runs into each .dat file.
An interrupted import is safe: the WAL and checkpoint recovery still own durability, and a repeated import skips ids that are already stored.
An event id is SHA-256 over:
[0,"<pubkey>",<created_at>,<kind>,<tags>,"<content>"]
NIP-01 defines this in terms of JavaScript's JSON.stringify. Go's
encoding/json escapes <, > and & as \u003c etc. by default and would
produce a different hash for content containing them. nostr.AppendJSONString
implements the JS rules directly: short escapes for quotes, backslash and
\b\f\n\r\t, \u00XX for other C0 controls, and raw UTF-8 for everything
else, including U+2028/U+2029. The test suite pins the result against a string
and hash produced by Node.
The event, filter and relay-message parser is hand-written and copied into
pkg/nostr/jsoncodec.go from the Moxie/Nostr decoder. It is a single pass over
the bytes with no encoding/json: object keys are compared as strings,
SplitArray returns the raw elements of a relay message, and typed readers
consume event and filter fields.
Two edge cases are explicit:
{"":1} is legal JSON. The reference decoder did switch key[0] after reading a key, which panicked with an index-out-of-range
when the key was empty. Here len(key) == 0 is checked before any indexing
and returns nostr: empty JSON object key. The lenient ExtractID path
skips the value instead, so a malformed event can still receive an OK.
\uD800 high surrogate must be followed by \uDC00..\uDFFF. The lookahead requires six bytes with p.i+6 > len
rejecting, rather than the reference's i+10 <= len which permitted
s[i+10] on a truncated input.
encoding/json is still used to encode NIP-11 and Blossom responses; it is no
longer used to parse anything that comes from the network.
The acceptor owns a non-blocking IPv4 socket created with x/sys/unix. It
polls with a 200 ms timeout, accepts with accept4, assigns the fd to a reader
worker in round-robin order, and backs off only when every worker already has a
queued connection. The worker wraps the fd in fdConn, a net.Conn whose reads
and writes use poll rather than blocking syscalls.
A fixed pool of reader/writer pairs follows. A reader owns the read side and
the parsing state for exactly one connection at a time: it parses HTTP with a
transport-neutral version of httpsrv.ReadRequest, serves NIP-11 and static
files locally, forwards health and Blossom to the network domain, and after an
upgrade switches to the RFC 6455 reader. The WebSocket reader returns control
frames to the reader (it never writes), while the paired writer serializes text,
pong and close frames. The writer is the only domain that writes or closes the
socket.
Responses are one request per connection with Connection: close, except for
an upgraded WebSocket, which stays open until either side closes. Idle
WebSockets are closed after 150 s. A reader waits for the writer's bind/upgrade
acknowledgement before handing the connection to the network domain, so a frame
can never overtake the handshake.
This is hand-written because TinyGo's net/http implements Hijack but closes
the hijacked connection as soon as the handler returns, and writes to it never
reach the peer.
The network domain is the single owner of connection state and subscriptions. It polls the reader inboxes, the database responses, the verifier responses and the Blossom responses on a 2 ms tick, so no lock is needed: only one goroutine mutates the tables.
REQ validates the filters, registers them on the connection, and sends a
database query. The database answers with owned event values; the network
domain frames them, sends EOSE, and keeps the filters for live delivery.
Broadcasts walk the subscription table and match filters (including NIP-50
token sets) before sending frames to the owning writer.
NIP-42 sends a per-connection random challenge in an AUTH message after the
upgrade. An AUTH event must be kind 22242, close to now, carry the challenge and
a relay URL matching TORLY_RELAY_URL, and pass the same verifier pool as
events. TORLY_AUTH_REQUIRED gates writes; NIP-70 events are restricted to
their author. Test failures and protocol errors are returned as NOTICE or
CLOSED values.
Outbound sends to a writer are non-blocking. A writer whose queue is full is treated as a slow consumer: the network domain stops addressing that connection, and its reader observes the eventual close.
Blobs are content-addressed at blossom/blobs/<first byte>/<sha256>.
An append-only binary index log records `(op, hash, pubkey, size, created,
type) so listing by pubkey and serving Content-Type` do not need to open the
blob. The log is compacted on open when it exceeds 4 MiB.
The Blossom domain owns the server value and answers BlobRequest with a
BlobResponse; the reader forwards blob paths through the network domain, so
no HTTP handler shares the index. Authorization is a signed kind 24242 event in
the Authorization: Nostr <base64> header: an unexpired expiration, a t tag for the action
(upload, delete, list) and, for blob actions, an x tag matching the
hash. Uploads verify that the declared x equals the SHA-256 of the body;
deletes require the requester to own the blob.
Several TinyGo runtime behaviours needed workarounds, all documented in the code:
os/signal only when the scheduler reaches an idle wait. A relay keeps a
socket in netpoll, so that path is never reached and SIGTERM would be
ignored. pkg/sigtrap linknames the runtime's checkSignals and calls
it every 200 ms; the non-TinyGo build makes Trap a no-op.
TCPListener.SetDeadline does nothing and Accept is not woken by Close, so a listener domain would never
observe shutdown. The acceptor therefore uses a non-blocking x/sys/unix
socket and poll with a 200 ms timeout, and the workers use fdConn, which
also polls rather than blocking in read(2).
getsockname, soit works with an ephemeral port. Production configuration uses explicit ports; the tests still pick and retry a port.
TCPConn.Close calls close(2) unconditionally; a double close can close an unrelated descriptor that reused
the number. fdConn closes exactly once (sync.Once) and shuts the socket
down first so a polling reader wakes.
The transport tests are deliberately timing-sensitive in one direction: they assert framing, shutdown, and error behaviour on real loopback sockets, and they run under both toolchains. The last two TinyGo issues above were found by those tests rather than by reading the runtime.
Schnorr performance. orly's pkg/p256k1 has fast paths, but its amd64
assembly/4×64 implementations are excluded under TinyGo, and the portable
32-bit branch it leaves for TinyGo rejects valid BIP-340 vectors on
linux/amd64 (tested: vector 0 returns false). torly therefore verifies
32-byte messages through btcec's pure-Go BIP-340 implementation, which carries
the same 5×52/10×26 field arithmetic and passes the full vector suite. Under
TinyGo on this machine a verification takes about 0.3 ms, down from 6.4 ms for
the math/big reference. The reference implementation is kept for BIP-340's
variable-length messages and as an independent cross-check: the tests sign with
both implementations and verify each other's output.
_graphThe graph indexes are materialized from tags at write time. Every edge is written twice, once in each direction, so a traversal is one prefix scan and a deletion is also one prefix scan:
epg event -> pubkey peg pubkey -> event
eeg event -> event gee event -> event
ppg pubkey -> pubkey gpp pubkey -> pubkey
A pubkey is interned in pks/spk first, so an edge carries two five-byte
serials rather than two 32-byte keys. The author edge has direction 0, a p-tag
outbound edge 1, and its reverse inbound edge 2. E-tag edges use 0/1 and
pubkey-pubkey edges use 0/1. These direction bytes are on-disk values.
epg/peg are written for the author and every valid p tag. ppg/gpp
are written for every valid p tag except the author's own pubkey.
eeg/gee are written for every valid e tag whose target is already in the
store. When the target arrives later, backfillInboundETags scans the tc
index for #e references to the new id and creates the missing edge, so
publishing a reply before its root does not leave a permanently incomplete
thread. The backfill also means the tag index, not the WAL, is the source of
truth for the reference direction.
The _graph filter extension is parsed in pkg/nostr/jsoncodec.go and executed
by Engine.TraverseGraph:
| Edge | Seed | Direction | Result |
|---|---|---|---|
pp | pubkey | out follows, in followers, both | pubkeys by depth |
pe | pubkey | out authored, in p-tag mentions, both | events at depth 1 |
ee | pubkey | out e-tag targets, in referencing events, both | events by depth |
pp walks only kind-3 contact-list edges, matching orly's follows semantics.
ee starts from every event authored by the seed pubkey; those roots are not
part of the result. pe is deliberately one hop: the pubkey↔event graph is
bipartite, so a second hop would return a different node type.
Traversal is breadth-first, deduplicates by serial, stops when a frontier is
empty, and caps the result at the engine's MaxResults. Results are sorted at
every depth so the signed response is deterministic. The response is a
relay-signed kind-39000 event whose content is pubkeys_by_depth or
events_by_depth; the relay key comes from TORLY_SECRET_KEY or, when that is
unset, DATA_DIR/relay.key, created on first start.
| orly subsystem | torly |
|---|---|
| Badger, Neo4j, SQLite, gRPC backends | one WAL + sorted indexes |
| ACME/Let's Encrypt | TLS terminated by a reverse proxy |
| Tor, WireGuard | not ported |
| NIP-46 bunker, NIP-86 admin, payments, bridge, blossom media proxy | not ported |
| SQL/Badger/Neo4j graph backends | materialized epg/peg/eeg/gee/ppg/gpp indexes and _graph |
| Svelte web dashboard | deferred; TORLY_STATIC_DIR serves a WASM build later |
The relay protocol, canonical encoding, replacement/deletion/expiry semantics and Blossom endpoints are the parts that make a deployment usable, and those are implemented and exercised end to end.