Causal system handbook / current edition

Causality is an interface.

SlOS records execution as a directed graph of typed causes. The operator can ask what happened, why it happened, what it caused, whether the record is intact, what the machine looked like at an earlier tick, and what would disappear if a selected event had not occurred.

Lamport time Parent edges SlFS journal Epoch proofs Replay /causal
A SlOS causal demonstration frame
Plate: the operating system narrates its own activity.

Seven instruments, one event record.

Instrument 01

Event

A typed record with ID, Lamport time, wall tick, subsystem, description, payload material, parent IDs, and child count.

Instrument 02

Hot graph

The 16,384-event live ring gives recent operations immediate query speed.

Instrument 03

Journal

cjournal spills aged events to SlFS segments and preserves searchable, verifiable history across reboot.

Instrument 04

Replay

The replay engine reconstructs system state at a tick; snapshots accelerate longer histories.

Instrument 05

Inspector

scrub is the full-screen and one-shot time-travel operator interface.

Instrument 06

Explorer

/causal exposes the same live DAG and replay state in a browser.

Instrument 07

Operator toolkit

Seven commands browse, explain, profile, time-travel, debug, triage, and prove against the same event record.

What a causal event is

A causal event is a named fact about the machine. The core types are SCHED, MEM, FS, NET, SYSCALL, SHELL, ACTOR, MESH, IO, and USER. Subsystems refine the type with labels such as sched, blockfs, tcp, notes, or httpd.

Each event increments Lamport time and stores up to four parent IDs. Those parent edges turn a log stream into a directed acyclic graph. A filesystem write can name the syscall that requested it; a note event can name the prior timeline head; a remote actor receive can name the sender's causal context.

Listing A include/causal.h
typedef struct {
    uint32_t       id;
    uint32_t       node_hash;
    causal_time_t  time;                         /* Lamport + wall + TSC */
    causal_type_t  type;                         /* SCHED..USER */
    char           subsystem[16];                /* "sched", "fs", "tcp" */
    char           desc[CAUSAL_DESC_LEN];
    uint8_t        payload[CAUSAL_PAYLOAD_LEN];
    uint32_t       payload_len;
    uint8_t        payload_hash[32];
    bool           payload_hash_valid;
    uint16_t       schema_id;
    uint8_t        schema_version;
    uint32_t       parents[CAUSAL_MAX_PARENTS];
    uint8_t        num_parents;
    uint16_t       child_count;
} causal_event_t;
Listing B kernel/graph/causal.c
static uint32_t emit_event(causal_type_t type,
                           const char *subsystem,
                           const char *desc,
                           const uint32_t *parent_ids,
                           int num_parents,
                           const void *payload,
                           uint32_t payload_len,
                           uint32_t origin_node)
{
    lamport_clock++;
    uint32_t id  = next_id++;
    uint32_t idx = ring_index(id);

    causal_event_t *ev = &events[idx];
    memset(ev, 0, sizeof(*ev));
    ev->id        = id;
    ev->node_hash = origin_node;
    ev->time      = causal_now();
    ev->type      = type;
    /* parents, payload, schema, child counts, task context follow */
}

DAG notation used by this manual

#4216 shell
->
#4217 syscall
->
#4218 fs
#4218 fs
->
#4219 notes
->
#4220 syscall

Querying the live graph

The hot ring contains the most recent 16,384 events. It is queried by the field commands events, why, trace, graph, blast, cpath, and cdot. These commands are intended for immediate operating diagnosis: identify the event, walk backwards to its causes, then walk forwards to its effects.

slos:/$ events 8
  ID    TICK    TYPE        DESCRIPTION
  4215  18394   sched       switch pid=12 -> pid=31
  4216  18396   shell       command: note add field report
  4217  18397   syscall     write pid=31
  4218  18397   fs          write /notes.db
  4219  18398   user        notes event appended
  4220  18399   syscall     yield pid=31

slos:/$ why 4219
Causal chain for event 4219:

[4219] T+183.98s notes: notes event appended
  └─ [4218] T+183.97s fs: write /notes.db
     └─ [4217] T+183.97s syscall: write pid=31
        └─ [4216] T+183.96s shell: command: note add field report
slos:/$ trace 4219
  [4219] t=18398 notes: notes event appended
    └─ caused by:
  [4218] t=18397 fs: write /notes.db
    └─ caused by:
  [4217] t=18397 syscall: write pid=31
    └─ caused by:
  [4216] t=18396 shell: command: note add field report

slos:/$ blast 4216
Forward trace from event #4216:
  [4217] T+18397 syscall: write pid=31
  [4218] T+18397 fs: write /notes.db
  [4219] T+18398 notes: notes event appended

slos:/$ cpath 4216 4219
Causal path #4216 -> #4219:
  [4216] T+18396 shell: command: note add field report
  [4217] T+18397 syscall: write pid=31
  [4218] T+18397 fs: write /notes.db
  [4219] T+18398 notes: notes event appended
cpath: 3 hops
slos:/$ cpath common 4219 4220
Nearest common ancestor of #4219 and #4220: #4216
  [4216] T+18396 shell: command: note add field report
cpath: dist 3 from #4219, 1 from #4220

Live tail

cwatch and cwatch tail turn the graph into an append-only operating stream: recent context first, then new events until any key is pressed. It is the quickest way to watch scheduler offloads, shell commands, and IO become causal evidence.

slos:/$ cwatch tail
recent context:
#4216 SHELL shell command: note add field report
#4217 SYSCALL syscall write pid=31

streaming new causal events; press any key to stop
#4218 FS blockfs write /notes.db
#4219 USER notes event appended
#4220 SCHED sched offload pid=47 to=cpu1

Graph export

cdot <id> emits Graphviz DOT for an event's ancestry. cdot <id> effects emits the downstream blast radius instead. This is deliberately plain text so it can be captured from serial output.

slos:/$ cdot 4219

digraph causal {
  rankdir=LR;
  "4216" [label="#4216
shell
command: note add field report"];
  "4217" [label="#4217
syscall
write pid=31"];
  "4218" [label="#4218
fs
write /notes.db"];
  "4219" [label="#4219
notes
notes event appended"];
  "4216" -> "4217";
  "4217" -> "4218";
  "4218" -> "4219";
}

slos:/$ cdot 4216 effects 32
# same format, but walks downstream blast radius

Durable searchable history

The original causal graph was a live ring. Current SlOS adds cjournal: a persistent SlFS journal under /causal. When the ring grows past its spill threshold, compact event records are written into 512-event segment files. Index, footer, hash-sidecar, and epoch-root records make the archive searchable and checkable after reboot.

Listing C kernel/graph/cjournal.c + include/cjournal.h
#define CJOURNAL_MAX_FILES         32
#define CJOURNAL_EVENTS_PER_FILE  512
#define CJOURNAL_DIR              "/causal"
#define CJOURNAL_SPILL_THRESHOLD 12288

typedef struct __attribute__((packed)) {
    uint32_t id;
    uint32_t node_hash;
    uint64_t lamport;
    uint32_t wall_ticks;
    uint8_t  type;
    char     subsystem[16];
    char     desc[80];
    uint32_t parents[4];
    uint8_t  num_parents;
    uint16_t child_count;
    uint32_t payload_len;
    uint8_t  payload_hash[32];
    uint16_t schema_id;
    uint8_t  schema_version;
} cjournal_event_t;
slos:/$ journal
Causal Journal:
  Status:     active
  Segments:   24 / 32 max
  Persisted:  12288 events
  ID range:   1 -- 12288 (on disk) | 12289 -- 16384 (in ring)
  Ring:       4096 / 16384

slos:/$ journal find notes 5
Journal find "notes": 5 events
  #3182  T+14211 hash=0x8b7a12f0 hashv=4 source=journal transport=syscall fs blockfs write /notes.db

slos:/$ events -a subsystem fs 5
Journal subsystem fs: 5 events

cq is the selector language

cq queries the hot ring by default. cq -a queries the archive with selectors such as type=, subsystem=, desc~, since=, until=, id=, and limit=. Use -l for long records or --json-ish for machine-readable rows.

slos:/$ cq -a type=fs subsystem=blockfs desc~notes since=12000 limit=3 -l
Event #3182
  Tick:      T+14211
  Type:      fs
  Subsystem: blockfs
  Hash:      0x8b7a12f0 (v4)
  Hash proof: cq -a id=3182 hash full
  Segment:   00000006
  Segment proof: cq -a segment=6 epoch-proof --json-ish
  Source:    journal

slos:/$ cq -a proof-health --json-ish
proof_health{source=journal,transport=syscall,status=PASS,segments=24,hash_pass=24,hash_total=24,root_pass=24,root_total=24,retry_pass=0,retry_total=0}

Integrity is an operating surface

Every event carries hash material. Journal segments carry Merkle sidecars and epoch proofs. journal verify, journal json selfcheck, journal proof-health, prove, cproof, and verify expose the trust checks.

Beyond self-checks, a per-node Ed25519 key signs each verified epoch root. Over the Noise-encrypted mesh, peers exchange signing keys and signed epoch roots; the receiver verifies the signature against the peer key, so shared causal history is attributable across machines. journal sign, journal verify-sig, journal trust-health, and federation share-root drive it; federation identity shows the bound signing key.

slos:/$ journal proof-health
journal proof-health: PASS segments=24 hash=24/24 root=24/24 retry=0/0

slos:/$ cq -a id=3182 epoch-proof --json-ish
cproof{kind=epoch-proof,segment=6,event=3182,source=journal,transport=syscall}
proof{kind=epoch-proof,status=PASS,segment=6,event=3182,leaf=109,depth=9,root=0xa104d9ce,format=v4,leaf_hash=0x8b7a12f0,source=sidecar}

slos:/$ journal json selfcheck
journal_selfcheck{source=journal,transport=syscall,segments=24,verified=24,failed=0,status=PASS}

Read-through rule: events -a reads archived events directly; why -a and archived cq recover causes after an event has aged out of the live ring.

Time travel, replay, and counterfactuals

The replay engine reconstructs a summary of kernel state at any wall tick by walking causal events up to the cursor. Snapshots speed long histories. The operator surface is rewind, replay, and especially scrub, the Causal Inspector.

Listing D kernel/graph/replay.c
int replay_at(uint32_t tick, replay_snapshot_t *snap)
{
    if (!snap) return -1;
    memset(snap, 0, sizeof(*snap));
    snap->tick = tick;

    uint32_t newest = causal_total_events();
    uint32_t oldest = causal_oldest_event_id();
    for (uint32_t id = oldest; newest != 0 && id <= newest; id++) {
        causal_event_t *ev = causal_get(id);
        if (!ev) continue;
        if (ev->time.wall_ticks > tick)
            continue;
        /* apply event to the reconstructed snapshot */
    }
    return 0;
}
slos:/$ scrub 18400
SlOS Causal Time-Travel  tick 18400 / 18402
[=======================================-]
Reconstructed state (replay of 4218 events):
  tasks: alive 9  created 31  exited 22
  io:    syscalls 701  fs 211  net tx/rx 44/88  actor msgs 37
Active tasks (9):
  [pid 31] shell
Recent events (8): #4215 #4216 #4217 #4218 #4219 #4220 #4221 #4222

slos:/$ scrub event 4219
Event #4219 focus
  type=USER subsystem=notes tick=T+18398
  origin: node=0x4f2a19d0 (local)
  desc: notes event appended
Caused by (parents):
  #4218 fs: write /notes.db
  #4217 syscall: write pid=31
  #4216 shell: command: note add field report
Led to (effects):
  #4220 syscall: yield pid=31

slos:/$ scrub diff 18000 18400
Causal diff: tick 18000 -> tick 18400
  events +41   syscalls +2   fs +3
  net tx +0   net rx +0   actor +1
  tasks alive: 8 -> 9

Counterfactual mode

whatif forks a shadow analysis, removes a selected event, and reports the downstream effects that would be eliminated. scrub whatif makes that analysis part of the time-travel inspector.

slos:/$ whatif summary 4218
whatif: drop #4218 (fs) -> 2 events eliminated, subsystems: notes(1) syscall(1), verdict: LOW IMPACT

slos:/$ scrub whatif 4218
whatif: drop #4218 (fs) -> 2 events eliminated, subsystems: notes(1) syscall(1), verdict: LOW IMPACT

The Web Causal Explorer

The userspace httpd serves a self-contained page at /causal. It fetches the live causal graph as JSON, renders nodes as an SVG layered by causal depth, colours them by event type, and polls for updates. The time-travel slider calls /api/causal/state?tick=N. Clicking a node focuses the DAG: causes are blue, effects are yellow, unrelated nodes dim.

slos:/$ httpd
[httpd] Started (pid 18) — http://10.0.2.15/ (host: http://localhost:8080/)
[httpd] Listening on port 80 — dashboard at http://<ip>/

host:$ curl http://10.0.2.15/api/causal/state
{"tick":18402,"total_events":4222,"tasks":{"alive":9,"created":31,"exited":22},"io":{"syscalls":703,"fs":214,"net_tx":44,"net_rx":88,"actor_msgs":37},"active_tasks":[{"pid":31,"name":"shell"}],"recent_events":[4217,4218,4219,4220,4221,4222]}

host:$ curl http://10.0.2.15/api/causal/state?tick=18000
{"tick":18000,"total_events":4181,"tasks":{"alive":8,"created":30,"exited":22},"io":{"syscalls":690,"fs":208,"net_tx":44,"net_rx":88,"actor_msgs":36},"active_tasks":[{"pid":31,"name":"shell"}],"recent_events":[4176,4177,4178,4179,4180,4181]}

host:$ curl http://10.0.2.15/api/causal/dot
digraph causal {
  "4216" -> "4217";
}

host:$ curl http://10.0.2.15/api/causal/graph
{"nodes":[{"id":4216,"tick":18396,"type":"shell","subsystem":"shell","desc":"command: note add field report","remote":false}],"edges":[{"from":4216,"to":4217}]}
Listing E programs/httpd.c
GET /causal
GET /api/causal/state
GET /api/causal/state?tick=N
GET /api/causal/recent
GET /api/causal/graph
GET /api/causal/dot

/* /causal fetches graph JSON, layers nodes by causal depth,
 * renders SVG, polls live data, and uses state?tick=N for
 * the time-travel scrubber. */

Browser view, described without a screenshot

/api/causal/graph
->
SVG DAG
->
click focus
/api/causal/state?tick=N
->
replay snapshot
->
slider panel
/api/causal/dot
->
Graphviz
->
external plate

This is the page's central claim: a bare-metal OS serves a live, navigable, time-travelling picture of its own causality over HTTP.

New causal field tools

The shell now includes a practical layer around the graph. ctop shows recent activity by subsystem. lens saves named causal queries to SlFS so they survive reboot. tally aggregates query output by frequency. why --md exports a shareable Markdown chain. causal is a glossary command, and learn is a guided tour that runs real commands.

slos:/$ ctop 128
ctop - causal activity by subsystem (last 128 events)
  SUBSYSTEM     EVENTS  SHARE
  syscall          42    32% ########
  fs               31    24% ######
  sched            25    19% ####

slos:/$ lens save fsnotes cq -a type=fs desc~notes limit=8
lens: saved 'fsnotes' = cq -a type=fs desc~notes limit=8
slos:/$ lens fsnotes
Journal filter: 8 events

slos:/$ journal type sched 50 | tally
sched.switch  31
sched.wake    12
sched.exit     7
slos:/$ why --md last
# Why event 4222

Causal chain (most recent first):

- **[4222]** T+184.02s `syscall`: write pid=31
  - **[4219]** T+183.98s `notes`: notes event appended
    - **[4218]** T+183.97s `fs`: write /notes.db

_Root cause reached._

Continue the route

Read /source for file listings, /subsystems for dependency order, /showcase for demonstration plates, and /learn for the guided operator path.

Operator commands over the one graph

The new operator toolkit does not add a second incident model. It reads the same causal record and gives the route names an operator rhythm: browse, explain, profile, time-travel, debug, triage, and prove. triage composes entropy, blame, critical-path, journal, and contract checks into one verdict. cbisect binary-searches replayable ticks to find when a cumulative metric crossed a threshold.

slos:/$ triage
SYSTEM TRIAGE  (tick 281)
====================================
STATUS: HEALTHY

Findings:
  [INFO] journal: durable; 0 event(s) in 0 segment(s) (verify journal for deep check)
  [ OK ] no anomalies, faults, or contract violations

Next action: (system healthy - no action needed)
slos:/$ cbisect events 100
CAUSAL BISECT: earliest tick where events >= 100
  range [0 .. 282], reconstructing state per probe via replay
  probe tick 141 -> events=0  miss
  probe tick 276 -> events=2049  hit
  probe tick 275 -> events=0  miss
FOUND: tick 276 (events=2049, crossed from 0) after 8 probes
  Next: replay at 276   (full reconstructed state)   |   events   (browse history)

Debug causation

cdb is a stateful debugger for causes: breakpoints, cursor, and the current event persist across shell calls.

slos:/$ cdb break type sched
cdb: breakpoint set (type=sched)
slos:/$ cdb run
Breakpoint hit at event #520144
#520144 [sched] sched: switch fbstream->telnetd (tick 289)
slos:/$ cdb bt
cause chain for #520144:
  (no causes -- root event)

Profile causal flow

cprof profiles cause-to-effect flow, marking cross-subsystem boundaries with * instead of sampling isolated events.

slos:/$ cprof
CAUSAL FLOW PROFILE (last 2048 events, 2046 causal edges)
Cause -> effect boundaries (by edge volume):
  sched -> sched  edges 1508  span 0t (avg 0t, max 0t)
  syscall -> syscall  edges 537  span 1t (avg 0t, max 1t)
  syscall -> sched *  edges 1  span 0t (avg 0t, max 0t)
Summary: 2046 edges, 1 cross-subsystem crossings, 1 total span ticks; dominant sched -> sched

Prove one outcome

attest turns one outcome into a signed receipt that can be verified against the node key.

slos:/$ attest first
CAUSAL RECEIPT for event #528880
  identity:  [sched] sched: switch federation->fbstream (tick 277, lamport 528880)
  hash:      d0e45a97d63ca691c9d14480dc7063fd96bb058849348af4f74b1f9a8b3ad88e
  source:    journal
  node:      f2971a2c86d8c402...   (Ed25519 node identity)
  segment:   1
  signature: f50029f30fe5a51f...  (Ed25519, 64 bytes)
  VERDICT:   SIGNED & VERIFIABLE   (run: attest verify 528880)

Carry the incident

capsule writes a portable incident slice to SlFS with the event, nearby causes and effects, hash, and proof status.

slos:/$ capsule last
CAUSAL CAPSULE for event #534692  (tick 284, node 7d165788)
  identity: [syscall] syscall: syscall 51 pid=30
  hash:     7f75abcb42d8264056553cdc9aa863e72fd8cace53bd2cfb1a267b956c561f8c
== Causes (ancestors, up to 8) ==
  #534691 [syscall] syscall: syscall 1 pid=30
== Effects (children, up to 8) ==
  (none)
== Proof ==
  not in a signed durable segment (run journal spill/sign to seal)
saved to /capsule/534692.txt (829 bytes)

Explain across federation boundaries

why -r keeps the ordinary cause walk, but tags each displayed hop with its origin node. It is deliberately honest about federation: imported events may arrive without their full remote ancestry, so the local walk says where the evidence stops.

slos:/$ why -r last
Causal chain for event 540525 (with origin nodes):
  local node: spiffe://slos.local/node/e55691c210199253
  [540525] T+282 syscall: syscall 51 pid=30   origin=local
  ... (max depth reached)
  all displayed hops originated on the local node
note: federation imports events individually, so remote ancestors beyond an imported event may not be present locally.

Operating rule: the hot ring is the immediate cockpit; the journal is durable history; replay and scrub reconstruct state; the web explorer makes the current DAG visible outside the guest without inventing a second model.