Operator labs

QEMU field exercises.

These labs are written for a reader who has cloned SlOS and wants a practical route through boot, causal history, durable journals, time travel, remote shells, the web explorer, SlFS, user programs, actors, mesh identity, and the showcase references.

Build once, then operate from the shell.

git clone https://github.com/slepp/slos.git
cd slos
make
make run-nographic

Keep the session observable.

Run the commands exactly at first. Record event IDs as they appear on your machine. When a lab asks for <event-id>, choose a recent event from the preceding events output.

SlFS state lives in disk.img. Keep it between reboots for persistence labs, and reset it when you intentionally want a fresh filesystem.

Boot and identify the machine

Purpose

Establish a clean QEMU session, confirm the kernel reached the shell, and record the machine identity before changing state.

Commands

# host terminal
make run-nographic

# SlOS shell
help
uname
status
neofetch
ps

Expected observation

  • Boot output reports memory, interrupts, SlFS, networking, timelines, actors, and mesh initialization.
  • The prompt reaches slos:/$, and uname/status identify the x86 QEMU target, uptime, memory, tasks, and devices.
  • neofetch runs as an embedded user ELF and returns to the shell when it exits.

Source files to read next

  • arch/x86/boot.asm
  • kernel/core/main.c
  • kernel/apps/shell.c
  • programs/shell.c
  • kernel/apps/cmd_system.c
  • programs/neofetch.c

Cleanup / reset

Exit nographic QEMU with Ctrl-A then X, or run halt from SlOS. make clean removes build products. Keep disk.img if later labs need persistence.

Observe recent causal events

Purpose

Learn the event surface before relying on it. The goal is to see event IDs, timestamps, subsystem labels, parent links, effects, and graph health.

Commands

events 12
events fs_op
events syscall
why last
trace last
cpath selftest
cdot last
graph 12
graph-health

Expected observation

  • events prints records with IDs, ticks, type/subsystem labels, and parent markers when present.
  • why, trace, cpath, and cdot give different views of causal ancestry, paths, and DOT output.
  • graph summarizes recent parent edges. graph-health reports event rate, orphan count, depth, and multi-parent count.

Source files to read next

  • include/causal.h
  • kernel/graph/causal.c
  • kernel/apps/cmd_graph.c
  • kernel/apps/shell.c
  • programs/shell.c

Cleanup / reset

No reset is needed. The causal ring is recent history; new activity naturally advances it.

Create timeline-backed notes and inspect why/trace

Purpose

Create application state through the notes timeline, then inspect the causal path that explains the resulting note and tag.

Commands

note add operator lab note one
note tag last lab
note search operator
note show last
tl show notes
tl tail notes -n 5
events user
why <event-id-from-notes>
trace <event-id-from-notes>

Expected observation

  • note add returns a note number. note tag records metadata. search and show reconstruct the current notes view.
  • tl show notes and tl tail notes show the timeline entries behind that view.
  • events user should include notes activity. Use a note-added or tag-attached event ID with why and trace to walk parent edges.

Source files to read next

  • kernel/apps/notes.c
  • include/notes.h
  • kernel/graph/timeline.c
  • include/timeline.h
  • kernel/apps/cmd_graph.c

Cleanup / reset

note delete last removes the note from the current reconstructed view and records a deletion event. The timeline remains useful for inspection.

Write/read files through SlFS and inspect persistence commands

Purpose

Exercise the persistent VirtIO-backed filesystem and read the metadata commands used to confirm what changed on disk.

Commands

diskmkdir -p /labs
diskwrite /labs/operator.txt SlFS lab line one
diskappend /labs/operator.txt SlFS lab line two
diskcat /labs/operator.txt
diskstat /labs/operator.txt
diskdf
fsck
diskcache
halt

# host terminal after QEMU exits
make run-nographic

# SlOS shell after reboot
diskcat /labs/operator.txt
disktree /labs 2
fsck

Expected observation

  • diskcat prints both lines. diskstat reports inode, file type, size, block count, times, and flags.
  • diskdf reports SlFS block and inode usage. fsck should report a clean filesystem for the normal path.
  • After reboot, the file remains readable because disk.img backs the VirtIO block device.

Source files to read next

  • kernel/fs/blockfs.c
  • include/blockfs.h
  • kernel/drivers/virtio_blk.c
  • kernel/apps/cmd_files.c
  • Makefile

Cleanup / reset

Use diskrm -r /labs to remove lab files. To reset all persistent SlFS state, intentionally replace or remove disk.img before the next boot.

Run a user ELF or built-in program

Purpose

Confirm that embedded user programs run through the ELF loader, then install one to SlFS and run it from a persistent path.

Commands

programs
elfinstall -l
elfinfo hello
exec hello
exec argdemo alpha beta
diskmkdir -p /labs
elfinstall hello /labs/hello.elf
elfinfo /labs/hello.elf
exec /labs/hello.elf
neofetch

# optional in a graphical build with DOOM assets installed
doom

Expected observation

  • programs and elfinstall -l list embedded installable programs. elfinfo reports ELF headers and compatibility.
  • exec hello and exec argdemo start user tasks, print output, and return to the shell after task exit.
  • The SlFS copy at /labs/hello.elf can be inspected and executed by path. neofetch is a convenient embedded ELF wrapper.

Source files to read next

  • kernel/core/elf.c
  • kernel/core/main.c
  • kernel/apps/cmd_system.c
  • programs/hello.c
  • programs/argdemo.c

Cleanup / reset

If the sample ELF was installed for this lab, run diskrm /labs/hello.elf. Embedded programs require no cleanup.

Inspect actors and mesh state

Purpose

Inspect local actors, deliver one mailbox message, and read mesh identity from a single QEMU node.

Commands

actors
send shell lab-message-from-operator
recv shell
mesh
routes
federation
events actor
events mesh

Expected observation

  • actors prints active local and remote actors, message counters, and mailbox depth. The shell, kernel, fs, net, mail, and dead-letter actors are useful landmarks.
  • send shell queues a message. recv shell consumes it and prints sender, timestamp, and payload details.
  • mesh prints the local SPIFFE ID, public key, and peer table. A single-node run may report no discovered peers; a multi-node run lists peer state and transport.

Source files to read next

  • kernel/graph/actor.c
  • include/actor.h
  • kernel/graph/mesh.c
  • include/mesh.h
  • kernel/graph/federation.c

Cleanup / reset

Run recv shell until the lab message is gone if you repeat the test. No persistent file state is created.

Watch causality in a browser

Purpose

Start the HTTP server, open the live Web Causal Explorer, scrub reconstructed state, and focus a node to inspect causes and effects.

Commands

# SlOS shell
ifconfig
httpd
httpd dump graph
httpd dump state 0

# host browser
open http://<slos-ip>/causal

# in the web page
# 1. watch the SVG DAG refresh
# 2. drag the time-travel slider
# 3. click a node to focus causes/effects
# 4. compare with /api/causal/graph and /api/causal/state

Expected observation

  • httpd starts the userspace server; httpd dump graph and state validate the endpoint builders from inside SlOS.
  • /causal renders a live SVG DAG from /api/causal/graph and shows state from /api/causal/state.
  • The slider requests /api/causal/state?tick=N. Clicking a node highlights its local causes and effects.

Source files to read next

  • programs/httpd.c
  • kernel/apps/cmd_net.c
  • scripts/web-explorer-check.mjs
  • docs/causal-inspector.md
  • Makefile

Cleanup / reset

Stop QEMU when finished. If the server task remains active during the session, reboot or halt for a clean shell.

Durable history across reboot

Purpose

Emit events, spill or read them through the durable journal, reboot on the same disk, and prove archived history still explains activity.

Commands

events 8
journal
journal spill-through last
journal stats
events -a sched 4
why -a last
journal proof-health
journal json selfcheck
halt

# host terminal after QEMU exits
make run-nographic

# SlOS shell after reboot
journal
events -a sched 4
cq -a id=last epoch-proof --json-ish
why -a last

Expected observation

  • journal shows active CJournal status and records persisted to SlFS.
  • events -a reads archived records, not only the current in-memory ring. why -a can explain using durable read-through history.
  • After reboot on the same disk.img, journal and cq -a still find persisted events and proof metadata.

Source files to read next

  • programs/shell.c
  • kernel/graph/causal.c
  • docs/manuals/VALIDATION.md
  • docs/JOURNAL.md
  • Makefile

Cleanup / reset

Keep disk.img for follow-up persistence tests. To reset durable history, intentionally reset disk.img before booting again.

Counterfactual: scrub whatif / whatif

Purpose

Ask which downstream events would be affected if a selected event were removed from the causal graph.

Commands

events 12
scrub event last
scrub whatif last
whatif <event-id-from-events>
scrub diff 100 99999
scrub json 99999

Expected observation

  • scrub event last prints event details, parents, and effects.
  • scrub whatif last and whatif <id> report the downstream impact and verdict for removing an event.
  • scrub diff and scrub json expose scriptable reconstructed state for comparing ticks.

Source files to read next

  • programs/shell.c
  • kernel/apps/cmd_graph.c
  • kernel/graph/replay.c
  • docs/causal-inspector.md

Cleanup / reset

No state is changed by the counterfactual query. New events produced by the commands simply advance the ring.

Save a lens and reuse it after reboot

Purpose

Persist a named causal query to SlFS, reboot on the same disk, and run the saved query again.

Commands

lens save faults events 5
lens list
lens run faults
halt

# host terminal after QEMU exits
make run-nographic

# SlOS shell after reboot
lens list
lens run faults

Expected observation

  • lens save stores the command under the chosen name. lens list shows the saved query.
  • After reboot, lens list still shows the entry loaded from /lens.db on SlFS.
  • lens run faults dispatches the stored command through the shell command table.

Source files to read next

  • programs/shell.c
  • docs/implementation/2026-06-12-lens.md
  • docs/manuals/VALIDATION.md
  • Makefile

Cleanup / reset

Run lens rm faults if you want to remove the saved query. It is intentionally persistent.

Compose shell filters with sequential pipelines

Purpose

Use cmd1 | cmd2 with stdin-aware filters while remembering the implementation is sequential capture, not POSIX pipes.

Commands

ps | grep shell
events | tail -n 5
help | grep Network
journal spill 600
journal type sched 50 | tally
journal type sched 50 | tally -f 4
cdot selftest | wc

Expected observation

  • grep, head, tail, and wc read pipeline stdin when no file argument is supplied.
  • tally counts repeated lines or fields and prints the most frequent keys first.
  • Each intermediate stage is capped at a 4096-byte buffer; large output is truncated with a notice rather than streamed concurrently.

Source files to read next

  • programs/shell.c
  • docs/shell.md
  • docs/JOURNAL.md
  • Makefile

Cleanup / reset

No cleanup is required. The journal spill intentionally adds durable causal history.

Use showcase videos as expected-observation references

Purpose

Use the recorded reels as a visual reference while keeping the local QEMU run as the primary observation.

Commands

# browser route
/showcase

# useful SlOS references while comparing
showcase
demo showcase
demo tour
learn

# match reels to labs
# 01 boot identity      -> lab 01
# 02 SlFS persistence  -> lab 04
# 04 external ELF/HEW  -> lab 05
# 05 timeline notes    -> lab 03
# 06 actors and mesh   -> lab 06

Expected observation

  • The video frames give approximate screen shape, command order, and output density for each topic.
  • Local event IDs, ticks, IP addresses, peer IDs, file sizes, and task numbers may differ from the recordings.
  • When a local result differs, prefer current source and command output, then use the reel to orient what to inspect next.

Source files to read next

  • programs/showcase.c
  • kernel/apps/demo.c
  • kernel/apps/cmd_graph.c
  • programs/shell.c
  • src/pages/showcase.astro

Cleanup / reset

No cleanup is needed. If a demo command leaves files or actors behind, use the relevant lab cleanup above.

Operate the causal toolkit

Purpose

Use the operator toolkit end to end: render a verdict, bisect replayable history, debug causes, profile flow, and seal one outcome with a verifiable receipt.

Commands

triage
cbisect events 100
cdb break type sched
cdb run
cdb bt
cprof
autopsy last
journal spill 64
journal sign 1
attest first
attest verify first
capsule last
why -r last

Expected observation

  • triage prints one STATUS verdict (HEALTHY/DEGRADED/CRITICAL) and a single next-action command.
  • cbisect binary-searches the tick timeline, reconstructing state per probe via replay, and reports the earliest tick the metric crossed the threshold.
  • cdb keeps a cursor and breakpoint across calls; bt walks causes and step walks effects of the current event.
  • cprof ranks cause-to-effect subsystem boundaries by edge volume and marks cross-subsystem crossings with *.
  • After spill+sign, attest first prints a SIGNED & VERIFIABLE receipt and attest verify first returns RECEIPT VALID; capsule writes /capsule/<id>.txt; why -r tags each hop origin=local.

Source files to read next

  • kernel/graph/triage.c
  • kernel/graph/cbisect.c
  • kernel/graph/cdb.c
  • kernel/graph/cprof.c
  • kernel/graph/attest.c
  • kernel/graph/capsule.c
  • kernel/apps/cmd_graph.c
  • docs/causal-toolkit.md

Cleanup / reset

cdb reset clears the debugger cursor. The /capsule/<id>.txt files persist on SlFS; remove them with rm if desired.

Use SlOS as a remote workstation

Purpose

Connect from the host, edit and run a persistent slosh script, start a detached service, disconnect, and verify the service continues to answer.

Commands

# host terminal A
make run

# host terminal B, default QEMU forwarding
telnet localhost 2323

# SlOS shell inside telnet
termsize
mkdir /home/remote-lab
edit /home/remote-lab/hello.sl
# in editor: write a small script, Ctrl-S save, Ctrl-Q quit
cat /home/remote-lab/hello.sl
run /home/remote-lab/hello.sl
httpd
exit

# host terminal B after disconnect
curl http://localhost:8080/

# optional encrypted shell, from the SlOS repo
python3 scripts/noise-shell-client.py localhost 5200

Expected observation

  • telnet reaches the slos:/$ prompt through host port 2323. Multiple remote sessions can be opened in parallel.
  • termsize reports the negotiated size when the telnet client supplies NAWS, otherwise 80x25. The full-screen editor uses ANSI output and cooked-key input over the connection.
  • The /home/remote-lab script remains in SlFS-backed persistent storage; /tmp and relative scratch paths would be RAM-backed instead.
  • After exit closes the remote shell, curl still reaches httpd because service daemons detach from the session. The optional noiseshd path is encrypted with ephemeral X25519 and ChaCha20-Poly1305, but not server-authenticated.

Source files to read next

  • docs/remote-access.md
  • kernel/apps/telnetd.c
  • kernel/core/syscall.c
  • kernel/drivers/console.c
  • programs/editor.c
  • kernel/apps/script.c
  • kernel/apps/cmd_files.c
  • scripts/noise-shell-client.py

Cleanup / reset

Reconnect and run rm /home/remote-lab/hello.sl or rm /home/remote-lab if you want to remove the persistent lab file. Reboot or halt to stop service tasks.

Validation evidence exists in the SlOS Makefile: smoke-boot, manual-smoke, causal-journal-check, web-explorer-check, and lens-persist-check.