Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Operations

This page covers running a mado server, reclaiming disk with garbage collection, recovering from a lost op-heads database, and benchmarking.

Running a server

mado serve exposes this repo’s content-addressed blob store (and a SQLite op-heads store) over gRPC, so other machines can use it as a networked mado backend. It serves the repo’s own .jj/repo/store — the same store local mado commands write through — so a remote reader sees exactly what was committed. The call blocks until the process is killed.

# Production shape: seed a fjall store, then serve it.
mkdir server && cd server && mado init --store-format fjall
mado import-git --git-repo ~/src/nixpkgs --ref master \
    --shallow-since 2024-01-01 --op-heads-db ../op_heads.sqlite
mado serve --addr 0.0.0.0:50051 --op-heads-db ../op_heads.sqlite

Key points:

  • --op-heads-db <file> persists the op-heads set to SQLite. Without it the op-heads set is ephemeral (in-memory) — fine for a throwaway server, but an ephemeral op-heads set is not a safe retention root, so scheduled GC and online git interop both require this flag.
  • A fjall store is held exclusively by one process. Bulk-import before starting the server, and stop the server for offline mado gc / mado export-git. The loose-file (fs) format is shareable but is the local default, not the production server format.
  • --auth-token <token> requires every RPC to present authorization: Bearer <token>. Without it the server is open — suitable only for a trusted network. Clients set MADO_TOKEN to match.
  • --prod-blobstore / --blob-prefix <p> select the production at-rest composition (Prefix → Pack(zstd) → Multiplexed → Fs), giving at-rest compression and per-repo namespacing. This changes the at-rest format: a store created with these flags must always be served (and GC’d, exported, recovered) with the same flags, and vice-versa. A --store-format fjall store implies the composition via its format marker, so the flags are unnecessary there.
  • --stats-interval-secs (default 60) logs a periodic blob-traffic summary (ops, hit rate, bytes — deltas since the previous line) when there was activity; 0 disables it.

The server also derives each commit’s changed paths (so lazy clients’ path-scoped revsets need not fetch whole trees) and, when given an op-heads db, maintains a segmented graph store for the Index.GraphDelta RPC.

Online git interop (optional)

mado serve can periodically git fetch a source remote into a server-side mirror, convert it, and publish the new tips (--git-ingest-url …), and/or export the current view’s bookmarks and git push them to a target remote (--git-export-url …). A named-remotes config file (--git-interop-config) scales this to N sources and M targets. All of it requires --op-heads-db. See the CLI reference for the full flag set and docs/design/online-git-interop.md for the design.

mado sync-now (against MADO_REMOTE_ADDR) wakes a running server’s interop tasks immediately instead of waiting for their poll interval:

mado sync-now                       # wake every interop task
mado sync-now --remote git-ingest[foo]

Garbage collection

mado gc collects blobs unreachable from any live op log. It reads the live op-head set from the same SQLite database mado serve writes and this workspace’s own local op log, walks both op DAGs to every commit any operation can still see, expands to the full ancestor closure, marks the blobs that closure needs, and reports — or, with --sweep, deletes — the rest.

mado gc --op-heads-db ../op_heads.sqlite            # dry run (reports only)
mado gc --op-heads-db ../op_heads.sqlite --sweep    # actually delete
  • --op-heads-db is required. GC derives the retained set from these heads. If the head set is empty, mado gc refuses to run — collecting against zero heads would mark the entire store garbage (almost always a wrong-database / wrong-scope mistake).
  • --min-age-secs (default 3600) is an age grace period: an unreachable blob younger than this (or of unknown age) is deferred, never collected. This protects in-flight commits whose blobs land before their op-head publishes. 0 disables the window — only safe against a quiesced store.
  • --scope reads live heads for a specific op-heads scope (defaults to the unscoped set).
  • Prod stores: pass the same --prod-blobstore / --blob-prefix the store is served with. GC decodes the object graph through the prod stack while sweeping the raw physical keys; a prod store read without these flags cannot decode a single commit.

Locality packs and other derived data are reclaimable cache, never GC roots — sweeping them only makes reads fall back to the per-file path.

Scheduled GC on the server

Instead of a cron’d mado gc --sweep, mado serve can sweep in-process:

mado serve --op-heads-db ../op_heads.sqlite \
    --gc-interval-secs 3600 --gc-min-age-secs 3600

The scheduled sweep retains the ancestor closure of every live op-head across all scopes (collecting against one scope would treat every other scope’s history as garbage), plus the host workspace’s local op log. The first sweep runs one full interval after startup; a GC failure is logged and retried next interval — it never takes the serving path down. --gc-interval-secs 0 (the default) disables it. Requires --op-heads-db.

Recovering op-heads

The op-heads database is the only pointer into the history of every scope: lose it and the blobs survive but are anonymous. Every op-heads mutation is journaled into the blobstore (the opjournal namespace, MDOJ records) before it commits, so mado recover-op-heads can rebuild the database:

mado recover-op-heads --out-db ./recovered.sqlite

It scans the journal, verifies each scope’s hash chain, and replays the verifiable prefix into a fresh database at --out-db (refusing to overwrite an existing file — so you can diff the recovered database against a suspect original before trusting it). It reports scopes recovered, the last sequence per scope, and any chain breaks (truncation / corruption). Like mado gc, pass matching --prod-blobstore / --blob-prefix if the store was served with them.

Benchmarking

scripts/bench.sh runs the whole performance suite as one command and archives the results:

nix develop --command scripts/bench.sh          # every bench row
nix develop --command scripts/bench.sh load_smoke   # a named subset

It executes the bench-corpus store-layer benches, load_smoke, and the FUSE macro_bench (via the scripts/test.sh userns wrapper), archiving each bench’s full output plus a summary.md (provenance: rustc, commit, CPU; per-bench pass/fail + wall-clock) under bench-results/<UTC stamp>/. That directory is git-ignored — curated numbers are hand-copied into docs/design/performance-testing.md, and the raw dated archives stay untracked. Positional names run a subset; --criterion opts into the micro-bench crates. Exit is nonzero if any bench’s assertions fail, while a FUSE-mount environment failure records the macro bench as SKIPPED rather than failing the run.

As measured (performance-testing.md §“Operationalization”, 2026-07-04, Intel N100): the first full run was 9/9 passed, ~26 min end-to-end — a viable nightly cadence.

The nightly timer

scripts/bench-nightly.sh wraps the runner with the trend policy — flag large regressions for a human, never fail on jitter: after the run it compares each bench’s wall-clock against the previous archived summary and flags any that is both > 30 % and > 10 s slower (appended to the new summary.md, printed to stderr, nonzero exit).

It is scheduled on the dev box as a systemd user timer:

systemctl --user status mado-bench      # last run's status
systemctl --user list-timers            # see the schedule

The unit (mado-bench.timer in ~/.config/systemd/user/) runs at 03:30 UTC nightly with Persistent=true and linger enabled; archives land in bench-results/ as usual.