Build from source
Operations
Running oceancode for a long time — days, not minutes — is a first-class case. This page
covers the operational surface that keeps a long run healthy: a metrics scrape, database and
ledger retention, the storage doctor, crash recovery, the session line, the event-bus drop counter, and log
levels. Everything here is conservative by default: session retention keeps a week
(retention.days: 7) and deletes nothing for the first 24 hours after an upgrade.
The backup, restore and crash-recovery procedures live in the
storage-and-recovery runbook (ocean-code/agent/docs/RUNBOOK-storage-and-recovery.md in the repository).
Metrics (OC_METRICS_ADDR)
Set OC_METRICS_ADDR=host:port to serve runtime gauges for a Prometheus scrape (uptime,
worker and live-task counts, injection-queue depth, and — on Linux — oceancode_rss_kb, the
process resident set from /proc/self/statm) plus three SQLite-writer gauges:
oceancode_writer_pending_ops (batched writes not yet committed), oceancode_writer_lag_ms
(age of the oldest of them) and oceancode_persist_failures (batched writes dropped because
they failed). Nothing listens by default. The endpoint is a
profiling aid, not a control surface: it exposes gauges, never accepts commands.
The HTTP server also answers GET /global/health, which carries dropped_events — see
Event-bus drops below.
A fresh headless server's resident set is held under the 45 MB SLO by the PERF=1 lane:
scripts/gates/rss-fresh.sh boots one release-binary serve on an ephemeral loopback port and
samples its RSS (RSS_FRESH_KB), which assert-budgets.mjs gates against the ceiling.
Retention and vacuum
One setting, retention.days, governs both stores that grow over a long run — the session
database oceancode.db and the Session Ledger directory. A session idle longer than that many
days is deleted from the database and its ledger directory goes with it, in one sweep, as does
the folder of images pasted into it in the TUI (tui/sessions/<project>/<session id>/ under
the data dir).
| Key | Default | Effect |
|---|---|---|
retention.days | 7 | A session tree (a top-level session and every sub-agent under it) whose members have all been idle longer than this many days is deleted, with each member's ledger directory. A tree with any member running or open is kept. 0 turns the sweep off. One setting for the database and the ledger. |
database.retention_days | — | Legacy. Read only when the config has no retention section in any layer, and only when > 0; then it is the retention. |
ledger.retention_days | — | Ignored since retention.days. A config that still sets it loads, with one warning at boot. |
database.vacuum | — | No longer read: every sweep that deleted something hands the freed pages back to the file itself (see below). |
{ "retention": { "days": 14 } } // or 0 to keep everythingThe sweep. A ticker runs it every 10 minutes; the first tick, at boot, only counts. One
session tree is deleted per writer job with a short pause between trees, so a step waiting
on the database runs in between; at most 500 sessions go per sweep, the rest at the next
tick. A ledger directory whose session is not in this database (another project's — the
ledger root is shared, each project has its own oceancode.db) is removed only on its own
age, past max(retention.days, 30) days. After a sweep that deleted something the freed pages
are returned to the file in small incremental_vacuum chunks; an older database is switched
to incremental auto-vacuum once, with one full VACUUM that runs only while no session is
running.
The grace window. The first boot that arms the sweep writes retention-armed-at in the
data dir. For 24 hours after it nothing is deleted — a sweep only counts what it would remove,
and the TUI shows retention N pending on its context row with a one-time notice — so an
upgrade with old sessions on disk never loses them unannounced. /retention off keeps them.
/retention in the TUI (or _ocean/retention from any ACP client) shows and sets it:
| Form | Effect |
|---|---|
/retention | retention 7 days · 12 sessions pending · first sweep after 2026-10-21 18:04 (grace), or after the window … · last sweep 5m ago, 340 swept. The pending counts are recounted for this request. |
/retention <days> | Sets retention.days (0..=3650) in your global config file — the user tier boot reads: $OC_CONFIG, else $XDG_CONFIG_HOME/oceancode/oceancode.json (~/.config/oceancode/oceancode.json) — and re-arms the sweep at once, without a restart. A project config cannot override it (the user tier wins over the project's); a managed config can. A number out of range is refused with the engine's message. |
/retention off | The same as /retention 0: the sweep stops; nothing is deleted. |
_ocean/retention takes {} (read) or {"days": n} (set) and answers
{days, pendingSessions, pendingLedgerDirs, armedAt, graceUntil, lastSweep, sweptTotal}
(instants in unix ms; lastSweep is {at, removedSessions, removedDirs} or null). Every
session_status frame carries the same object under ocean.retention, from the last count —
a frame never walks the disk.
The metrics endpoint carries oceancode_retention_swept_sessions_total (sessions deleted since
boot) and oceancode_retention_pending (sessions the last count found still to delete).
Free-space floor
The engine refuses every write to oceancode.db, and so every prompt, while the volume that holds
the database has less than 2 GB free. The TUI shows Out of disk space. Free some space and
try again.; free space, then send the prompt again. Nothing is lost: the refused prompt was never
recorded, and sessions already open keep their state in memory. The floor exists because SQLite
writing into a full disk corrupted a 3.9 GB store on 2026-11-03, after which no new session could
be created until the file was rebuilt. A volume whose free space cannot be read (an unsupported
filesystem) is never refused.
oceancode doctor --storage
oceancode doctor reports orphaned sessions and captured panic reports (read-only; it boots
nothing). Add --storage for a storage report:
$ oceancode doctor --storage
…orphan / panic report…
storage:
oceancode.db 4194304 bytes (wal 32768 bytes)
sessions=128
ledger 2097152 bytes, oldest 12 days
restore is offline-only: stop oceancode before restoring a backup--storage is a doctor flag, not a separate subcommand. The report is best-effort: it
renders zeros for a path that does not exist yet.
Backup and restore
Backups are consistent even under concurrent writes (VACUUM INTO, not a raw file copy), and
restore verifies the backup's PRAGMA integrity_check before it swaps anything into place.
Restore is offline-only — it replaces the live database file, so stop oceancode first.
The step-by-step commands are in the runbook (ocean-code/agent/docs/RUNBOOK-storage-and-recovery.md in the repository).
Crash recovery
A panic writes a timestamped report under the data dir's crash/, and an active-sessions
registry records a session whose process died without a clean exit. The next boot prints a
one-line stderr notice when it finds an orphan; oceancode doctor lists both. A crashed
session resumes with oceancode run --continue — the Session Ledger is intact byte-for-byte
across a kill -9. Full drill in the runbook (ocean-code/agent/docs/RUNBOOK-storage-and-recovery.md in the repository).
The session line
Every engine on the machine (TUI windows, oceancode acp, oceancode serve, the short commands peers, status
and resume, the Ocean app) joins one switchboard, the session line (2026-11-18). There is no extra process: the
first engine started hosts it, on a loopback port, and the others connect to it. Its files are in the data dir's
line/ folder:
| File | What it is |
|---|---|
line/messaging.db | The record: sessions, claims, requests between sessions, the log. Only the host writes it (SQLite, WAL); about once a second the host moves the write-ahead log into the file. No call waits for the disk. |
line/addr | The host's port and a random token, one line, mode 0600. An engine without the token is refused. |
line/lock | Held by the host for its whole life; the engine that takes it after the host is gone hosts next. |
- Liveness is the connection. An engine whose connection closes is gone at once: its claims are free and its
sessions' cards read
crashed. There is no heartbeat. - Takeover. When the host closes or dies, the next engine takes over, and the others join it again and claim again what they held; a call under way is sent once more. An engine that has not come back within 3 s is cleared. A Mac that loses power loses at most the last moment of the record; the sessions and their claims are gone with the engines anyway.
- The old file. Before the line, every engine opened
<data dir>/peers.db. A new engine reads it once, read-only, to carry over the session cards, the log, Slack message rows and the machine id, and never writes it. Older engines still use it among themselves and do not see new ones: restart the open windows after the update. - Speed. 200 messages between two engines, measured: 0.5 ms per message at the median, under 2 ms at p99 (before, about 1 s typical and 2 s at worst).
What the line decides is in 12 · Session peers.
Event-bus drops (dropped_events)
Each event subscriber has a bounded queue. When a subscriber falls behind, the bus's default
behaviour is to evict the oldest queued event and count the drop rather than block the
producer — so a slow consumer never stalls the streaming hot path. The running total is
visible as the dropped_events field on GET /global/health.
Two events config keys tune this:
| Key | Default | Effect |
|---|---|---|
events.sub_capacity | 1024 | Per-subscriber event-queue depth. A deeper queue tolerates a slower subscriber before it drops. 0 falls back to the built-in default. |
events.mode | drop_oldest | Overflow behaviour. drop_oldest evicts the oldest event and counts the drop (the streaming default). block_until briefly parks the publisher (up to one second) for a free slot before falling back to drop-oldest — bounded back-pressure for a deployment that would rather slow a producer than lose events. |
The bus is sized from the project-tier events config at boot.
Log levels (OC_LOG)
The engine logs through a tracing subscriber that redacts secret-shaped values before any
write (Anthropic / OpenAI / AWS / PEM / bearer / assignment shapes), so a token never lands
in a log file. Control it with OC_LOG or the obs config block.
| Setting | Default | Effect |
|---|---|---|
OC_LOG (env) | unset → obs.log_level | Level: trace, debug, info, warn, error. Overrides obs.log_level. An unrecognized value logs a one-line notice and falls back to info. |
obs.log_level | info | The configured level when OC_LOG is unset. |
obs.log_rotate_bytes | 0 → 10 MiB | Byte ceiling before the live log file rotates to <file>.1, so logs never grow unbounded on a multi-day run. |
obs.log_file | unset → stderr | Log sink path for the headless modes (oceancode serve / run). Absent logs to stderr. See the sink-precedence note below. |
The level (OC_LOG → obs.log_level → info) applies to every mode. The obs config is read
from the run's project directory (the --dir a serve/run was given, else the current
directory), the same way boot reads project config (ocean_config::load, honouring
$OC_CONFIG_DIR) — so oceancode run --dir /path/to/project uses that project's obs settings,
not the shell's cwd. A malformed or absent config falls back to defaults and never blocks startup.
Sink precedence. obs.log_file (bounded by obs.log_rotate_bytes) redirects the default
stderr diagnostic sink of the headless oceancode serve / run modes to a size-bounded
rotating file, so headless logs never grow unbounded on a multi-day run. The interactive shape is
Ocean TUI, which owns the terminal and keeps its own state under its seeded $OC_TUI_HOME (see
19 · Ocean TUI); tui.log is no longer written. So the jsonc example below
applies to serve / run:
{ "obs": { "log_level": "debug", "log_file": "/var/log/oceancode.log", "log_rotate_bytes": 10485760 } }Endurance soak (operators)
A deterministic soak drives 1000 metering-fixture steps (no live model) and grades per-step
p99 latency and RSS growth against pinned budgets. Write the soak record to its own path so a
short marathon-fixture run cannot clobber it (soak-metrics.mjs grades any record under 1000
steps as SOAK_RED reason=short):
STEPS=1000 TURNS_OUT=docs/results/soak/turns.jsonl bash scripts/bench/marathon-fixture.sh --steps 1000
node scripts/bench/soak-metrics.mjs docs/results/soak/turns.jsonl # → SOAK_OK / SOAK_REDOC_SQLITE_CACHE_KB and OC_READ_POOL_SIZE bound per-connection SQLite memory and the read
pool for a measurement run; leave both unset in normal operation. See
02 · Configuration.