> ## Documentation Index
> Fetch the complete documentation index at: https://www.plungeai.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Operations

> Running oceancode for a long time — days, not minutes — is a first-class case. This page covers the operational surface that keeps a long run healthy: a metrics scrape, database and ledger retention, the storage doctor, crash recovery, the event-bus drop counter, and log levels. Everything here …


Running `oceancode` for a long time — days, not minutes — is a first-class case. This page
covers the operational surface that keeps a long run healthy: a metrics scrape, database and
ledger retention, the storage doctor, crash recovery, the event-bus drop counter, and log
levels. Everything here is conservative by default: session retention keeps a week
(`retention.days: 7`) and deletes nothing for the first 24 hours after an upgrade.

The backup, restore and crash-recovery *procedures* live in the
storage-and-recovery runbook (`apps/ocean-coding-agent-rust/docs/RUNBOOK-storage-and-recovery.md` in the repository).

## Metrics (`OC_METRICS_ADDR`)

Set `OC_METRICS_ADDR=host:port` to serve runtime gauges for a Prometheus scrape (uptime,
worker and live-task counts, injection-queue depth, and — on Linux — `oceancode_rss_kb`, the
process resident set from `/proc/self/statm`) plus three SQLite-writer gauges:
`oceancode_writer_pending_ops` (batched writes not yet committed), `oceancode_writer_lag_ms`
(age of the oldest of them) and `oceancode_persist_failures` (batched writes dropped because
they failed). Nothing listens by default. The endpoint is a
profiling aid, not a control surface: it exposes gauges, never accepts commands.

The HTTP server also answers `GET /global/health`, which carries `dropped_events` — see
[Event-bus drops](#event-bus-drops-dropped_events) below.

A fresh headless server's resident set is held under the 45 MB SLO by the `PERF=1` lane:
`scripts/gates/rss-fresh.sh` boots one release-binary `serve` on an ephemeral loopback port and
samples its RSS (`RSS_FRESH_KB`), which `assert-budgets.mjs` gates against the ceiling.

## Retention and vacuum

One setting, `retention.days`, governs both stores that grow over a long run — the session
database `oceancode.db` and the Session Ledger directory. A session idle longer than that many
days is deleted from the database and its ledger directory goes with it, in one sweep.

| Key | Default | Effect |
|---|---|---|
| `retention.days` | `7` | A session tree (a top-level session and every sub-agent under it) whose members have **all** been idle longer than this many days is deleted, with each member's ledger directory. A tree with any member running or open is kept. `0` turns the sweep off. One setting for the database and the ledger. |
| `database.retention_days` | — | Legacy. Read only when the config has no `retention` section in any layer, and only when > 0; then it is the retention. |
| `ledger.retention_days` | — | Ignored since `retention.days`. A config that still sets it loads, with one warning at boot. |
| `database.vacuum` | — | No longer read: every sweep that deleted something hands the freed pages back to the file itself (see below). |

```jsonc
{ "retention": { "days": 14 } }   // or 0 to keep everything
```

**The sweep.** A ticker runs it every 10 minutes; the first tick, at boot, only counts. One
session tree is deleted per writer job with a short pause between trees, so a step waiting
on the database runs in between; at most 500 sessions go per sweep, the rest at the next
tick. A ledger directory whose session is not in this database (another project's — the
ledger root is shared, each project has its own `oceancode.db`) is removed only on its own
age, past `max(retention.days, 30)` days. After a sweep that deleted something the freed pages
are returned to the file in small `incremental_vacuum` chunks; an older database is switched
to incremental auto-vacuum once, with one full `VACUUM` that runs only while no session is
running.

**The grace window.** The first boot that arms the sweep writes `retention-armed-at` in the
data dir. For 24 hours after it nothing is deleted — a sweep only counts what it would remove,
and the pager shows `retention N pending` on its context row with a one-time notice — so an
upgrade with old sessions on disk never loses them unannounced. `/retention off` keeps them.

**`/retention`** in the pager (or `_ocean/retention` from any ACP client) shows and sets it:

| Form | Effect |
|---|---|
| `/retention` | `retention 7 days · 12 sessions pending · first sweep after 2026-10-21 18:04 (grace)`, or after the window `… · last sweep 5m ago, 340 swept`. The pending counts are recounted for this request. |
| `/retention <days>` | Sets `retention.days` (`0..=3650`) in your **global** config file — the user tier boot reads: `$OC_CONFIG`, else `$XDG_CONFIG_HOME/oceancode/oceancode.json` (`~/.config/oceancode/oceancode.json`) — and re-arms the sweep at once, without a restart. A project config cannot override it (the user tier wins over the project's); a managed config can. A number out of range is refused with the engine's message. |
| `/retention off` | The same as `/retention 0`: the sweep stops; nothing is deleted. |

`_ocean/retention` takes `{}` (read) or `{"days": n}` (set) and answers
`{days, pendingSessions, pendingLedgerDirs, armedAt, graceUntil, lastSweep, sweptTotal}`
(instants in unix ms; `lastSweep` is `{at, removedSessions, removedDirs}` or `null`). Every
`session_status` frame carries the same object under `ocean.retention`, from the last count —
a frame never walks the disk.

The metrics endpoint carries `oceancode_retention_swept_sessions_total` (sessions deleted since
boot) and `oceancode_retention_pending` (sessions the last count found still to delete).

## `oceancode doctor --storage`

`oceancode doctor` reports orphaned sessions and captured panic reports (read-only; it boots
nothing). Add `--storage` for a storage report:

```
$ oceancode doctor --storage
…orphan / panic report…
storage:
  oceancode.db 4194304 bytes (wal 32768 bytes)
  sessions=128
  ledger 2097152 bytes, oldest 12 days
  restore is offline-only: stop oceancode before restoring a backup
```

`--storage` is a `doctor` flag, not a separate subcommand. The report is best-effort: it
renders zeros for a path that does not exist yet.

## Backup and restore

Backups are consistent even under concurrent writes (`VACUUM INTO`, not a raw file copy), and
restore verifies the backup's `PRAGMA integrity_check` before it swaps anything into place.
Restore is **offline-only** — it replaces the live database file, so stop `oceancode` first.
The step-by-step commands are in the runbook (`apps/ocean-coding-agent-rust/docs/RUNBOOK-storage-and-recovery.md` in the repository).

## Crash recovery

A panic writes a timestamped report under the data dir's `crash/`, and an active-sessions
registry records a session whose process died without a clean exit. The next boot prints a
one-line stderr notice when it finds an orphan; `oceancode doctor` lists both. A crashed
session resumes with `oceancode run --continue` — the Session Ledger is intact byte-for-byte
across a `kill -9`. Full drill in the runbook (`apps/ocean-coding-agent-rust/docs/RUNBOOK-storage-and-recovery.md` in the repository).

## Event-bus drops (`dropped_events`)

Each event subscriber has a bounded queue. When a subscriber falls behind, the bus's default
behaviour is to evict the oldest queued event and count the drop rather than block the
producer — so a slow consumer never stalls the streaming hot path. The running total is
visible as the `dropped_events` field on `GET /global/health`.

Two `events` config keys tune this:

| Key | Default | Effect |
|---|---|---|
| `events.sub_capacity` | `1024` | Per-subscriber event-queue depth. A deeper queue tolerates a slower subscriber before it drops. `0` falls back to the built-in default. |
| `events.mode` | `drop_oldest` | Overflow behaviour. `drop_oldest` evicts the oldest event and counts the drop (the streaming default). `block_until` briefly parks the publisher (up to one second) for a free slot before falling back to drop-oldest — bounded back-pressure for a deployment that would rather slow a producer than lose events. |

The bus is sized from the project-tier `events` config at boot.

## Log levels (`OC_LOG`)

The engine logs through a tracing subscriber that **redacts secret-shaped values before any
write** (Anthropic / OpenAI / AWS / PEM / bearer / assignment shapes), so a token never lands
in a log file. Control it with `OC_LOG` or the `obs` config block.

| Setting | Default | Effect |
|---|---|---|
| `OC_LOG` (env) | unset → `obs.log_level` | Level: `trace`, `debug`, `info`, `warn`, `error`. Overrides `obs.log_level`. An unrecognized value logs a one-line notice and falls back to `info`. |
| `obs.log_level` | `info` | The configured level when `OC_LOG` is unset. |
| `obs.log_rotate_bytes` | `0` → 10 MiB | Byte ceiling before the live log file rotates to `<file>.1`, so logs never grow unbounded on a multi-day run. |
| `obs.log_file` | unset → stderr | Log sink path for the **headless** modes (`oceancode serve` / `run`). Absent logs to stderr. See the sink-precedence note below. |

The level (`OC_LOG` → `obs.log_level` → `info`) applies to every mode. The `obs` config is read
from the **run's project directory** (the `--dir` a `serve`/`run` was given, else the current
directory), the same way boot reads project config (`ocean_config::load`, honouring
`$OC_CONFIG_DIR`) — so `oceancode run --dir /path/to/project` uses that project's `obs` settings,
not the shell's cwd. A malformed or absent config falls back to defaults and never blocks startup.

**Sink precedence.** `obs.log_file` (bounded by `obs.log_rotate_bytes`) redirects the default
**stderr** diagnostic sink of the headless `oceancode serve` / `run` modes to a size-bounded
rotating file, so headless logs never grow unbounded on a multi-day run. The interactive shape is
Ocean pager, which owns the terminal and keeps its own state under its seeded `$OC_PAGER_HOME` (see
[19 · Ocean pager](/oceancode/pager)); `tui.log` is no longer written. So the jsonc example below
applies to `serve` / `run`:

```jsonc
{ "obs": { "log_level": "debug", "log_file": "/var/log/oceancode.log", "log_rotate_bytes": 10485760 } }
```

## Endurance soak (operators)

A deterministic soak drives 1000 metering-fixture steps (no live model) and grades per-step
p99 latency and RSS growth against pinned budgets. Write the soak record to its own path so a
short marathon-fixture run cannot clobber it (`soak-metrics.mjs` grades any record under 1000
steps as `SOAK_RED reason=short`):

```bash
STEPS=1000 TURNS_OUT=docs/results/soak/turns.jsonl bash scripts/bench/marathon-fixture.sh --steps 1000
node scripts/bench/soak-metrics.mjs docs/results/soak/turns.jsonl        # → SOAK_OK / SOAK_RED
```

`OC_SQLITE_CACHE_KB` and `OC_READ_POOL_SIZE` bound per-connection SQLite memory and the read
pool for a measurement run; leave both unset in normal operation. See
[02 · Configuration](/oceancode/configuration#test-and-bench-knobs).