18 · Operations

Running oceancode for a long time — days, not minutes — is a first-class case. This page covers the operational surface that keeps a long run healthy: a metrics scrape, database and ledger retention, the storage doctor, crash recovery, the event-bus drop counter, and log levels. Everything here is conservative by default: session retention keeps a week (retention.days: 7) and deletes nothing for the first 24 hours after an upgrade.

The backup, restore and crash-recovery procedures live in the storage-and-recovery runbook (apps/ocean-coding-agent-rust/docs/RUNBOOK-storage-and-recovery.md in the repository).

Metrics (OC_METRICS_ADDR)

Set OC_METRICS_ADDR=host:port to serve runtime gauges for a Prometheus scrape (uptime, worker and live-task counts, injection-queue depth, and — on Linux — oceancode_rss_kb, the process resident set from /proc/self/statm) plus three SQLite-writer gauges: oceancode_writer_pending_ops (batched writes not yet committed), oceancode_writer_lag_ms (age of the oldest of them) and oceancode_persist_failures (batched writes dropped because they failed). Nothing listens by default. The endpoint is a profiling aid, not a control surface: it exposes gauges, never accepts commands.

The HTTP server also answers GET /global/health, which carries dropped_events — see Event-bus drops below.

A fresh headless server's resident set is held under the 45 MB SLO by the PERF=1 lane: scripts/gates/rss-fresh.sh boots one release-binary serve on an ephemeral loopback port and samples its RSS (RSS_FRESH_KB), which assert-budgets.mjs gates against the ceiling.

Retention and vacuum

One setting, retention.days, governs both stores that grow over a long run — the session database oceancode.db and the Session Ledger directory. A session idle longer than that many days is deleted from the database and its ledger directory goes with it, in one sweep.

{ "retention": { "days": 14 } }   // or 0 to keep everything

The sweep. A ticker runs it every 10 minutes; the first tick, at boot, only counts. One session tree is deleted per writer job with a short pause between trees, so a step waiting on the database runs in between; at most 500 sessions go per sweep, the rest at the next tick. A ledger directory whose session is not in this database (another project's — the ledger root is shared, each project has its own oceancode.db) is removed only on its own age, past max(retention.days, 30) days. After a sweep that deleted something the freed pages are returned to the file in small incremental_vacuum chunks; an older database is switched to incremental auto-vacuum once, with one full VACUUM that runs only while no session is running.

The grace window. The first boot that arms the sweep writes retention-armed-at in the data dir. For 24 hours after it nothing is deleted — a sweep only counts what it would remove, and the pager shows retention N pending on its context row with a one-time notice — so an upgrade with old sessions on disk never loses them unannounced. /retention off keeps them.

/retention in the pager (or _ocean/retention from any ACP client) shows and sets it:

_ocean/retention takes {} (read) or {"days": n} (set) and answers {days, pendingSessions, pendingLedgerDirs, armedAt, graceUntil, lastSweep, sweptTotal} (instants in unix ms; lastSweep is {at, removedSessions, removedDirs} or null). Every session_status frame carries the same object under ocean.retention, from the last count — a frame never walks the disk.

The metrics endpoint carries oceancode_retention_swept_sessions_total (sessions deleted since boot) and oceancode_retention_pending (sessions the last count found still to delete).

oceancode doctor --storage

oceancode doctor reports orphaned sessions and captured panic reports (read-only; it boots nothing). Add --storage for a storage report:

$ oceancode doctor --storage
…orphan / panic report…
storage:
  oceancode.db 4194304 bytes (wal 32768 bytes)
  sessions=128
  ledger 2097152 bytes, oldest 12 days
  restore is offline-only: stop oceancode before restoring a backup

--storage is a doctor flag, not a separate subcommand. The report is best-effort: it renders zeros for a path that does not exist yet.

Backup and restore

Backups are consistent even under concurrent writes (VACUUM INTO, not a raw file copy), and restore verifies the backup's PRAGMA integrity_check before it swaps anything into place. Restore is offline-only — it replaces the live database file, so stop oceancode first. The step-by-step commands are in the runbook (apps/ocean-coding-agent-rust/docs/RUNBOOK-storage-and-recovery.md in the repository).

Crash recovery

A panic writes a timestamped report under the data dir's crash/, and an active-sessions registry records a session whose process died without a clean exit. The next boot prints a one-line stderr notice when it finds an orphan; oceancode doctor lists both. A crashed session resumes with oceancode run --continue — the Session Ledger is intact byte-for-byte across a kill -9. Full drill in the runbook (apps/ocean-coding-agent-rust/docs/RUNBOOK-storage-and-recovery.md in the repository).

Event-bus drops (dropped_events)

Each event subscriber has a bounded queue. When a subscriber falls behind, the bus's default behaviour is to evict the oldest queued event and count the drop rather than block the producer — so a slow consumer never stalls the streaming hot path. The running total is visible as the dropped_events field on GET /global/health.

Two events config keys tune this:

The bus is sized from the project-tier events config at boot.

Log levels (OC_LOG)

The engine logs through a tracing subscriber that redacts secret-shaped values before any write (Anthropic / OpenAI / AWS / PEM / bearer / assignment shapes), so a token never lands in a log file. Control it with OC_LOG or the obs config block.

The level (OC_LOG → obs.log_level → info) applies to every mode. The obs config is read from the run's project directory (the --dir a serve/run was given, else the current directory), the same way boot reads project config (ocean_config::load, honouring $OC_CONFIG_DIR) — so oceancode run --dir /path/to/project uses that project's obs settings, not the shell's cwd. A malformed or absent config falls back to defaults and never blocks startup.

Sink precedence. obs.log_file (bounded by obs.log_rotate_bytes) redirects the default stderr diagnostic sink of the headless oceancode serve / run modes to a size-bounded rotating file, so headless logs never grow unbounded on a multi-day run. The interactive shape is Ocean pager, which owns the terminal and keeps its own state under its seeded $OC_PAGER_HOME (see 19 · Ocean pager); tui.log is no longer written. So the jsonc example below applies to serve / run:

{ "obs": { "log_level": "debug", "log_file": "/var/log/oceancode.log", "log_rotate_bytes": 10485760 } }

Endurance soak (operators)

A deterministic soak drives 1000 metering-fixture steps (no live model) and grades per-step p99 latency and RSS growth against pinned budgets. Write the soak record to its own path so a short marathon-fixture run cannot clobber it (soak-metrics.mjs grades any record under 1000 steps as SOAK_RED reason=short):

STEPS=1000 TURNS_OUT=docs/results/soak/turns.jsonl bash scripts/bench/marathon-fixture.sh --steps 1000
node scripts/bench/soak-metrics.mjs docs/results/soak/turns.jsonl        # → SOAK_OK / SOAK_RED

OC_SQLITE_CACHE_KB and OC_READ_POOL_SIZE bound per-connection SQLite memory and the read pool for a measurement run; leave both unset in normal operation. See 02 · Configuration.