Instrument 01
Event
A typed record with ID, Lamport time, wall tick, subsystem, description, payload material, parent IDs, and child count.
Causal system handbook / current edition
SlOS records execution as a directed graph of typed causes. The operator can ask what happened, why it happened, what it caused, whether the record is intact, what the machine looked like at an earlier tick, and what would disappear if a selected event had not occurred.
Reader route
Instrument 01
A typed record with ID, Lamport time, wall tick, subsystem, description, payload material, parent IDs, and child count.
Instrument 02
The 16,384-event live ring gives recent operations immediate query speed.
Instrument 03
cjournal spills aged events to SlFS segments and preserves searchable, verifiable history across reboot.
Instrument 04
The replay engine reconstructs system state at a tick; snapshots accelerate longer histories.
Instrument 05
scrub is the full-screen and one-shot time-travel operator interface.
Instrument 06
/causal exposes the same live DAG and replay state in a browser.
Instrument 07
Seven commands browse, explain, profile, time-travel, debug, triage, and prove against the same event record.
Chapter 1
A causal event is a named fact about the machine. The core types are
SCHED, MEM, FS, NET,
SYSCALL, SHELL, ACTOR, MESH,
IO, and USER. Subsystems refine the type with labels
such as sched, blockfs, tcp,
notes, or httpd.
Each event increments Lamport time and stores up to four parent IDs. Those parent edges turn a log stream into a directed acyclic graph. A filesystem write can name the syscall that requested it; a note event can name the prior timeline head; a remote actor receive can name the sender's causal context.
typedef struct {
uint32_t id;
uint32_t node_hash;
causal_time_t time; /* Lamport + wall + TSC */
causal_type_t type; /* SCHED..USER */
char subsystem[16]; /* "sched", "fs", "tcp" */
char desc[CAUSAL_DESC_LEN];
uint8_t payload[CAUSAL_PAYLOAD_LEN];
uint32_t payload_len;
uint8_t payload_hash[32];
bool payload_hash_valid;
uint16_t schema_id;
uint8_t schema_version;
uint32_t parents[CAUSAL_MAX_PARENTS];
uint8_t num_parents;
uint16_t child_count;
} causal_event_t; static uint32_t emit_event(causal_type_t type,
const char *subsystem,
const char *desc,
const uint32_t *parent_ids,
int num_parents,
const void *payload,
uint32_t payload_len,
uint32_t origin_node)
{
lamport_clock++;
uint32_t id = next_id++;
uint32_t idx = ring_index(id);
causal_event_t *ev = &events[idx];
memset(ev, 0, sizeof(*ev));
ev->id = id;
ev->node_hash = origin_node;
ev->time = causal_now();
ev->type = type;
/* parents, payload, schema, child counts, task context follow */
} Chapter 2
The hot ring contains the most recent 16,384 events. It is queried by the field
commands events, why, trace, graph,
blast, cpath, and cdot. These commands are
intended for immediate operating diagnosis: identify the event, walk backwards
to its causes, then walk forwards to its effects.
slos:/$ events 8
ID TICK TYPE DESCRIPTION
4215 18394 sched switch pid=12 -> pid=31
4216 18396 shell command: note add field report
4217 18397 syscall write pid=31
4218 18397 fs write /notes.db
4219 18398 user notes event appended
4220 18399 syscall yield pid=31
slos:/$ why 4219
Causal chain for event 4219:
[4219] T+183.98s notes: notes event appended
└─ [4218] T+183.97s fs: write /notes.db
└─ [4217] T+183.97s syscall: write pid=31
└─ [4216] T+183.96s shell: command: note add field report slos:/$ trace 4219
[4219] t=18398 notes: notes event appended
└─ caused by:
[4218] t=18397 fs: write /notes.db
└─ caused by:
[4217] t=18397 syscall: write pid=31
└─ caused by:
[4216] t=18396 shell: command: note add field report
slos:/$ blast 4216
Forward trace from event #4216:
[4217] T+18397 syscall: write pid=31
[4218] T+18397 fs: write /notes.db
[4219] T+18398 notes: notes event appended
slos:/$ cpath 4216 4219
Causal path #4216 -> #4219:
[4216] T+18396 shell: command: note add field report
[4217] T+18397 syscall: write pid=31
[4218] T+18397 fs: write /notes.db
[4219] T+18398 notes: notes event appended
cpath: 3 hops
slos:/$ cpath common 4219 4220
Nearest common ancestor of #4219 and #4220: #4216
[4216] T+18396 shell: command: note add field report
cpath: dist 3 from #4219, 1 from #4220 cwatch and cwatch tail turn the graph into an
append-only operating stream: recent context first, then new events until
any key is pressed. It is the quickest way to watch scheduler offloads,
shell commands, and IO become causal evidence.
slos:/$ cwatch tail
recent context:
#4216 SHELL shell command: note add field report
#4217 SYSCALL syscall write pid=31
streaming new causal events; press any key to stop
#4218 FS blockfs write /notes.db
#4219 USER notes event appended
#4220 SCHED sched offload pid=47 to=cpu1 cdot <id> emits Graphviz DOT for an event's ancestry.
cdot <id> effects emits the downstream blast radius instead.
This is deliberately plain text so it can be captured from serial output.
slos:/$ cdot 4219
digraph causal {
rankdir=LR;
"4216" [label="#4216
shell
command: note add field report"];
"4217" [label="#4217
syscall
write pid=31"];
"4218" [label="#4218
fs
write /notes.db"];
"4219" [label="#4219
notes
notes event appended"];
"4216" -> "4217";
"4217" -> "4218";
"4218" -> "4219";
}
slos:/$ cdot 4216 effects 32
# same format, but walks downstream blast radius Chapter 3
The original causal graph was a live ring. Current SlOS adds cjournal:
a persistent SlFS journal under /causal. When the ring grows past
its spill threshold, compact event records are written into 512-event segment
files. Index, footer, hash-sidecar, and epoch-root records make the archive
searchable and checkable after reboot.
#define CJOURNAL_MAX_FILES 32
#define CJOURNAL_EVENTS_PER_FILE 512
#define CJOURNAL_DIR "/causal"
#define CJOURNAL_SPILL_THRESHOLD 12288
typedef struct __attribute__((packed)) {
uint32_t id;
uint32_t node_hash;
uint64_t lamport;
uint32_t wall_ticks;
uint8_t type;
char subsystem[16];
char desc[80];
uint32_t parents[4];
uint8_t num_parents;
uint16_t child_count;
uint32_t payload_len;
uint8_t payload_hash[32];
uint16_t schema_id;
uint8_t schema_version;
} cjournal_event_t; slos:/$ journal
Causal Journal:
Status: active
Segments: 24 / 32 max
Persisted: 12288 events
ID range: 1 -- 12288 (on disk) | 12289 -- 16384 (in ring)
Ring: 4096 / 16384
slos:/$ journal find notes 5
Journal find "notes": 5 events
#3182 T+14211 hash=0x8b7a12f0 hashv=4 source=journal transport=syscall fs blockfs write /notes.db
slos:/$ events -a subsystem fs 5
Journal subsystem fs: 5 events cq queries the hot ring by default. cq -a queries the
archive with selectors such as type=, subsystem=,
desc~, since=, until=, id=,
and limit=. Use -l for long records or
--json-ish for machine-readable rows.
slos:/$ cq -a type=fs subsystem=blockfs desc~notes since=12000 limit=3 -l
Event #3182
Tick: T+14211
Type: fs
Subsystem: blockfs
Hash: 0x8b7a12f0 (v4)
Hash proof: cq -a id=3182 hash full
Segment: 00000006
Segment proof: cq -a segment=6 epoch-proof --json-ish
Source: journal
slos:/$ cq -a proof-health --json-ish
proof_health{source=journal,transport=syscall,status=PASS,segments=24,hash_pass=24,hash_total=24,root_pass=24,root_total=24,retry_pass=0,retry_total=0}
Every event carries hash material. Journal segments carry Merkle sidecars and
epoch proofs. journal verify, journal json selfcheck,
journal proof-health, prove, cproof,
and verify expose the trust checks.
Beyond self-checks, a per-node Ed25519 key signs each verified epoch root.
Over the Noise-encrypted mesh, peers exchange signing keys and signed
epoch roots; the receiver verifies the signature against the peer key,
so shared causal history is attributable across machines.
journal sign, journal verify-sig,
journal trust-health, and federation share-root
drive it; federation identity shows the bound signing key.
slos:/$ journal proof-health
journal proof-health: PASS segments=24 hash=24/24 root=24/24 retry=0/0
slos:/$ cq -a id=3182 epoch-proof --json-ish
cproof{kind=epoch-proof,segment=6,event=3182,source=journal,transport=syscall}
proof{kind=epoch-proof,status=PASS,segment=6,event=3182,leaf=109,depth=9,root=0xa104d9ce,format=v4,leaf_hash=0x8b7a12f0,source=sidecar}
slos:/$ journal json selfcheck
journal_selfcheck{source=journal,transport=syscall,segments=24,verified=24,failed=0,status=PASS}
Read-through rule: events -a reads archived events directly;
why -a and archived cq recover causes after an event has
aged out of the live ring.
Chapter 4
The replay engine reconstructs a summary of kernel state at any wall tick by
walking causal events up to the cursor. Snapshots speed long histories. The
operator surface is rewind, replay, and especially
scrub, the Causal Inspector.
int replay_at(uint32_t tick, replay_snapshot_t *snap)
{
if (!snap) return -1;
memset(snap, 0, sizeof(*snap));
snap->tick = tick;
uint32_t newest = causal_total_events();
uint32_t oldest = causal_oldest_event_id();
for (uint32_t id = oldest; newest != 0 && id <= newest; id++) {
causal_event_t *ev = causal_get(id);
if (!ev) continue;
if (ev->time.wall_ticks > tick)
continue;
/* apply event to the reconstructed snapshot */
}
return 0;
} slos:/$ scrub 18400
SlOS Causal Time-Travel tick 18400 / 18402
[=======================================-]
Reconstructed state (replay of 4218 events):
tasks: alive 9 created 31 exited 22
io: syscalls 701 fs 211 net tx/rx 44/88 actor msgs 37
Active tasks (9):
[pid 31] shell
Recent events (8): #4215 #4216 #4217 #4218 #4219 #4220 #4221 #4222
slos:/$ scrub event 4219
Event #4219 focus
type=USER subsystem=notes tick=T+18398
origin: node=0x4f2a19d0 (local)
desc: notes event appended
Caused by (parents):
#4218 fs: write /notes.db
#4217 syscall: write pid=31
#4216 shell: command: note add field report
Led to (effects):
#4220 syscall: yield pid=31
slos:/$ scrub diff 18000 18400
Causal diff: tick 18000 -> tick 18400
events +41 syscalls +2 fs +3
net tx +0 net rx +0 actor +1
tasks alive: 8 -> 9 whatif forks a shadow analysis, removes a selected event, and reports
the downstream effects that would be eliminated. scrub whatif
makes that analysis part of the time-travel inspector.
slos:/$ whatif summary 4218
whatif: drop #4218 (fs) -> 2 events eliminated, subsystems: notes(1) syscall(1), verdict: LOW IMPACT
slos:/$ scrub whatif 4218
whatif: drop #4218 (fs) -> 2 events eliminated, subsystems: notes(1) syscall(1), verdict: LOW IMPACT Chapter 5
The userspace httpd serves a self-contained page at /causal.
It fetches the live causal graph as JSON, renders nodes as an SVG layered by
causal depth, colours them by event type, and polls for updates. The time-travel
slider calls /api/causal/state?tick=N. Clicking a node focuses the
DAG: causes are blue, effects are yellow, unrelated nodes dim.
slos:/$ httpd
[httpd] Started (pid 18) — http://10.0.2.15/ (host: http://localhost:8080/)
[httpd] Listening on port 80 — dashboard at http://<ip>/
host:$ curl http://10.0.2.15/api/causal/state
{"tick":18402,"total_events":4222,"tasks":{"alive":9,"created":31,"exited":22},"io":{"syscalls":703,"fs":214,"net_tx":44,"net_rx":88,"actor_msgs":37},"active_tasks":[{"pid":31,"name":"shell"}],"recent_events":[4217,4218,4219,4220,4221,4222]}
host:$ curl http://10.0.2.15/api/causal/state?tick=18000
{"tick":18000,"total_events":4181,"tasks":{"alive":8,"created":30,"exited":22},"io":{"syscalls":690,"fs":208,"net_tx":44,"net_rx":88,"actor_msgs":36},"active_tasks":[{"pid":31,"name":"shell"}],"recent_events":[4176,4177,4178,4179,4180,4181]}
host:$ curl http://10.0.2.15/api/causal/dot
digraph causal {
"4216" -> "4217";
}
host:$ curl http://10.0.2.15/api/causal/graph
{"nodes":[{"id":4216,"tick":18396,"type":"shell","subsystem":"shell","desc":"command: note add field report","remote":false}],"edges":[{"from":4216,"to":4217}]} GET /causal
GET /api/causal/state
GET /api/causal/state?tick=N
GET /api/causal/recent
GET /api/causal/graph
GET /api/causal/dot
/* /causal fetches graph JSON, layers nodes by causal depth,
* renders SVG, polls live data, and uses state?tick=N for
* the time-travel scrubber. */ This is the page's central claim: a bare-metal OS serves a live, navigable, time-travelling picture of its own causality over HTTP.
Chapter 6
The shell now includes a practical layer around the graph. ctop shows
recent activity by subsystem. lens saves named causal queries to SlFS
so they survive reboot. tally aggregates query output by frequency.
why --md exports a shareable Markdown chain. causal is a
glossary command, and learn is a guided tour that runs real commands.
slos:/$ ctop 128
ctop - causal activity by subsystem (last 128 events)
SUBSYSTEM EVENTS SHARE
syscall 42 32% ########
fs 31 24% ######
sched 25 19% ####
slos:/$ lens save fsnotes cq -a type=fs desc~notes limit=8
lens: saved 'fsnotes' = cq -a type=fs desc~notes limit=8
slos:/$ lens fsnotes
Journal filter: 8 events
slos:/$ journal type sched 50 | tally
sched.switch 31
sched.wake 12
sched.exit 7 slos:/$ why --md last
# Why event 4222
Causal chain (most recent first):
- **[4222]** T+184.02s `syscall`: write pid=31
- **[4219]** T+183.98s `notes`: notes event appended
- **[4218]** T+183.97s `fs`: write /notes.db
_Root cause reached._ Read /source for file listings, /subsystems for dependency order, /showcase for demonstration plates, and /learn for the guided operator path.
Chapter 7
The new operator toolkit does not add a second incident model. It reads the
same causal record and gives the route names an operator rhythm: browse,
explain, profile, time-travel, debug, triage, and prove. triage
composes entropy, blame, critical-path, journal, and contract checks into one
verdict. cbisect binary-searches replayable ticks to find when a
cumulative metric crossed a threshold.
slos:/$ triage
SYSTEM TRIAGE (tick 281)
====================================
STATUS: HEALTHY
Findings:
[INFO] journal: durable; 0 event(s) in 0 segment(s) (verify journal for deep check)
[ OK ] no anomalies, faults, or contract violations
Next action: (system healthy - no action needed) slos:/$ cbisect events 100
CAUSAL BISECT: earliest tick where events >= 100
range [0 .. 282], reconstructing state per probe via replay
probe tick 141 -> events=0 miss
probe tick 276 -> events=2049 hit
probe tick 275 -> events=0 miss
FOUND: tick 276 (events=2049, crossed from 0) after 8 probes
Next: replay at 276 (full reconstructed state) | events (browse history) cdb is a stateful debugger for causes: breakpoints, cursor,
and the current event persist across shell calls.
slos:/$ cdb break type sched
cdb: breakpoint set (type=sched)
slos:/$ cdb run
Breakpoint hit at event #520144
#520144 [sched] sched: switch fbstream->telnetd (tick 289)
slos:/$ cdb bt
cause chain for #520144:
(no causes -- root event) cprof profiles cause-to-effect flow, marking cross-subsystem
boundaries with * instead of sampling isolated events.
slos:/$ cprof
CAUSAL FLOW PROFILE (last 2048 events, 2046 causal edges)
Cause -> effect boundaries (by edge volume):
sched -> sched edges 1508 span 0t (avg 0t, max 0t)
syscall -> syscall edges 537 span 1t (avg 0t, max 1t)
syscall -> sched * edges 1 span 0t (avg 0t, max 0t)
Summary: 2046 edges, 1 cross-subsystem crossings, 1 total span ticks; dominant sched -> sched attest turns one outcome into a signed receipt that can be
verified against the node key.
slos:/$ attest first
CAUSAL RECEIPT for event #528880
identity: [sched] sched: switch federation->fbstream (tick 277, lamport 528880)
hash: d0e45a97d63ca691c9d14480dc7063fd96bb058849348af4f74b1f9a8b3ad88e
source: journal
node: f2971a2c86d8c402... (Ed25519 node identity)
segment: 1
signature: f50029f30fe5a51f... (Ed25519, 64 bytes)
VERDICT: SIGNED & VERIFIABLE (run: attest verify 528880) capsule writes a portable incident slice to SlFS with the
event, nearby causes and effects, hash, and proof status.
slos:/$ capsule last
CAUSAL CAPSULE for event #534692 (tick 284, node 7d165788)
identity: [syscall] syscall: syscall 51 pid=30
hash: 7f75abcb42d8264056553cdc9aa863e72fd8cace53bd2cfb1a267b956c561f8c
== Causes (ancestors, up to 8) ==
#534691 [syscall] syscall: syscall 1 pid=30
== Effects (children, up to 8) ==
(none)
== Proof ==
not in a signed durable segment (run journal spill/sign to seal)
saved to /capsule/534692.txt (829 bytes) why -r keeps the ordinary cause walk, but tags each displayed
hop with its origin node. It is deliberately honest about federation:
imported events may arrive without their full remote ancestry, so the local
walk says where the evidence stops.
slos:/$ why -r last
Causal chain for event 540525 (with origin nodes):
local node: spiffe://slos.local/node/e55691c210199253
[540525] T+282 syscall: syscall 51 pid=30 origin=local
... (max depth reached)
all displayed hops originated on the local node
note: federation imports events individually, so remote ancestors beyond an imported event may not be present locally. Operating rule: the hot ring is the immediate cockpit; the journal is durable history; replay and scrub reconstruct state; the web explorer makes the current DAG visible outside the guest without inventing a second model.