barrelfish proved that treating a multicore machine as a network of independent cores with message passing is both correct and performant. their SOSP 2009 paper showed that message-passing OS structure matches or beats shared-memory kernels even on cache-coherent hardware, and scales better as core counts increase.
barrelfish still carries legacy from the unix/seL4 tradition: dispatchers, monitors, a central capability system, and RPC semantics inherited from the client-server model. moxie's spawn/channel/domain model eliminates all of this and arrives at a cleaner architecture by different means - from the amiga's explicit multi-agent model and distributed systems theory rather than from academic OS research.
this document maps barrelfish's components onto what a moxie OS would look like, identifying what to keep, what to discard, and what emerges from the moxie model that barrelfish never had.
barrelfish runs a "CPU driver" per core: single-threaded, non-preemptible, interrupts disabled, shares no state with other cores. it schedules dispatchers, handles syscalls, manages capabilities, and processes interrupts. it uses a single statically allocated stack that resets after each handler.
moxie's version is simpler. each core runs a single executive that:
the executive is the only code that runs in privileged mode. it is not a kernel - it provides no services, no system calls in the unix sense. it provides three primitives: spawn a domain, switch to a domain, and forward an interrupt.
barrelfish's CPU driver maintains a capability database and processes capability invocations as system calls. the moxie executive does not do this. capabilities are replaced by the simpler mechanism of memory ownership - each domain owns its memory regions, and the executive enforces isolation through page tables. there is no capability transfer protocol because domains don't share memory.
barrelfish's dispatcher is a scheduler activation - the kernel upcalls into it, and it manages its own threads. multiple dispatchers can form a "domain" that spans cores. dispatchers don't migrate between cores.
moxie domains are simpler and more constrained:
the "scheduling" is implicit: a domain runs until it blocks on a channel receive. the executive then checks which other domains on the same core have pending messages and activates one. if no domain has pending messages, the core idles (or migrates a domain from a busy core - see layer 4).
this eliminates barrelfish's entire dispatcher machinery: upcall entry points (run, lrpc, pagefault, trap), enabled/disabled modes, save areas, thread control blocks, the libbarrelfish thread scheduler. none of it exists. a domain is a single execution context with a single stack.
barrelfish has multiple inter-domain communication mechanisms:
moxie replaces all of this with one mechanism: the channel pair.
a channel is a unidirectional buffer. every inter-domain connection consists of two channels - one in each direction. each channel is a fixed-size ring buffer in memory owned by the receiving domain. the sender writes to it (the executive maps a write-only view into the sender's address space). the receiver reads from it.
same-core channels: the buffer is in the receiver's memory. the sender writes, then signals the executive (a single instruction - a "notify" trap). the executive marks the receiver as runnable. no cache coherence traffic beyond the write itself.
cross-core channels: the buffer is still in the receiver's memory on the receiver's core. the sender writes via the executive, which performs the cross-core write (or uses a per-core notification FIFO like barrelfish's PCN pages). this is equivalent to barrelfish's UMP but without the polling overhead - the receiving executive gets an IPI or checks its notification slots on domain switch.
there is no IDL compiler. channel messages are typed at the language level - moxie's type system enforces the message format at compile time. the channel declaration in source code specifies the message type, and the compiler generates the serialization inline. no stubs, no runtime dispatch tables.
there is no binding/export protocol. a domain that spawns another domain provides the channel endpoints at spawn time. there is no name service, no iref lookup. you know who you're talking to because you created the connection.
backpressure: the channel buffer has a fixed size. if it's full, the sender's write blocks (yields to the executive). the receiver will eventually consume messages and free space. this is the water element - backpressure by buffer state, neither side can freeze the other, because blocking the sender just means the executive runs other domains until the receiver drains.
barrelfish has an elaborate seL4-derived capability system: typed capabilities, a derivation hierarchy (RAM -> Frame -> CNode -> VNode -> Dispatcher -> Endpoint), a distributed mapping database for tracking capabilities across cores, retype/copy/mint/delete operations, and a two-level CNode addressing scheme.
moxie discards all of this. the memory model is:
this eliminates:
the executive maintains a simple list of (physical base, size, owning domain) tuples. page table construction is done by the executive on behalf of the domain - the domain requests virtual mappings of its own physical memory, and the executive constructs the page tables. domains cannot manipulate page tables directly (unlike barrelfish's self-paging model) because that would require exposing page table memory as a typed capability, which we don't have.
for bulk data transfer (network packets, disk blocks, framebuffer regions), a domain can transfer ownership of a memory region to another domain through the executive. the region is unmapped from the sender and mapped into the receiver. zero copy, no sharing, clean ownership transfer. this replaces barrelfish's bulk transfer mechanism (which relied on shared memory pools with a master-slave protocol).
barrelfish dispatchers don't migrate between cores. moxie domains can.
when a core is idle and another core is overloaded (multiple domains with pending messages), the idle core's executive can request a domain migration. the migrating domain's memory is either:
migration is the executive's decision, not the domain's. the domain doesn't know it moved. its channels continue to work - the channel buffers get remapped, and the notification path updates to the new core.
this is simpler than barrelfish's approach because there's no dispatcher state to synchronize, no cspace to transfer, no monitor involvement. a domain is just registers + page tables + channel endpoints. all of those can be serialized and moved.
barrelfish puts device drivers in user-space domains that communicate with the hardware via IO capabilities and with other domains via IDC. the driver model involves PCI enumeration, interrupt routing through the kernel, DMA buffer management, and device register access via mackerel-generated accessor code.
moxie's driver model:
the driver sees a flat memory region (the device registers) and a channel (interrupts + requests from other domains). it doesn't know or care about PCI, ACPI, or bus topology. the executive abstracted all of that at spawn time.
barrelfish has a "system knowledge base" (SKB) that stores hardware topology information using ECLiPSe CLP (constraint logic programming). it answers queries about NUMA distances, cache hierarchies, interrupt routing, and device capabilities.
moxie replaces this with iskra.
the iskra lattice stores the hardware topology as a graph: cores, caches, memory controllers, interconnects, devices. the lattice relationships encode the physical distances and capabilities. when the executive needs to make a placement decision (which core to spawn a domain on, whether to migrate, which memory region to allocate for a device), it queries the iskra lattice.
this is more powerful than barrelfish's SKB because:
barrelfish has a complex network stack built on top of their IDC and bulk transfer mechanisms. moxie's approach:
there is no kernel network stack. the executive has zero knowledge of networking. the protocol stack is a domain like any other, subject to the same isolation and communication rules.
step 8 is the key divergence from barrelfish, which maintains a privileged monitor on each core throughout runtime. in moxie, after boot, all cores are peers. the BSP executive has no special authority.
phase 1 - executive on linux: the executive runs as a linux process, "cores" are threads, domains are green threads within the process. channels are in-process ring buffers. this validates the channel semantics, domain lifecycle, and scheduling logic without touching hardware.
phase 2 - executive on bare metal (single core): boot on RISC-V or x86-64 via UEFI. single core, multiple domains, real page tables, real interrupt handling. validates memory ownership model and driver architecture.
phase 3 - multicore: boot AP cores, implement cross-core channels via shared cache-line notification pages (similar to barrelfish's PCN). implement domain migration.
phase 4 - iskra integration: replace static topology tables with iskra lattice queries for placement and migration decisions.
phase 5 - self-hosting: the moxie compiler runs as a domain on the moxie OS, compiling moxie code into moxie executables that run as domains. the bootstrap loop closes at the OS level.