16 · Testing and gates

Everything in oceancode is verified by one script — ./check.sh — and a handful of opt-in lanes on it. This page covers what those lanes are, and then the one test surface a terminal UI needs that unit tests cannot give it: the interface probe, which drives the real TUI over a pseudo-terminal and proves the interface never freezes.

The lanes

export PATH="$HOME/.cargo/bin:$PATH"
./check.sh                 # fmt · clippy -D warnings · build · nextest · goldens · doc + gate checks
E2E=1     ./check.sh       # + real-engine PTY tests, chaos, crash-recovery drills, the interface probe
PERF=1    ./check.sh       # + divan benches and the budget gates
COVERAGE=1 ./check.sh      # + llvm-cov with the zero-covered-handler tripwire
MUTANTS=1 ./check.sh       # + cargo-mutants over the risk-ranked modules

The default lane dials no model provider and touches no network. Every checker prints a sentinel as its last line and exits non-zero on failure; the full roster is in scripts/README.md (apps/ocean-coding-agent-rust/scripts/README.md in the repository). No lane spends money — the only live-provider scripts (scripts/probes/live-*.sh) are run by hand, never by check.sh.

The interface probe

A TUI can pass every unit test and still lock up the instant a slash command awaits the whole turn on the event loop — the exact freeze that shipped and was fixed on 2026-09-15. Unit tests run the update function in-process; they cannot catch a real event loop that has stopped turning keystrokes into frames. The interface probe can, because it is black-box: it spawns the real oceancode binary behind a pseudo-terminal, renders its screen with pyte, feeds it real keys, and asserts on what a person would see.

It costs nothing to run. The binary is wired to a fake Anthropic provider with a fake key, an isolated HOME, and a loopback --listen port; the owner's real provider keys never enter a probe run.

bash scripts/probe/run-probe.sh                 # every scenario
bash scripts/probe/run-probe.sh -k slash        # only scenarios whose id (or file) contains "slash"
bash scripts/probe/run-probe.sh -k baseline     # one file's scenarios (-k matches the module name too)
bash scripts/probe/run-probe.sh -j 4            # run 4 at a time, each on its own ports
bash scripts/probe/run-probe.sh --list          # module:id of every discovered scenario
bash scripts/probe/run-probe.sh --keep          # keep each scenario's temp HOME to debug a failure

The last line is the sentinel:

INTERFACE_PROBE_OK scenarios=10 passed=10 xfail=0        # exit 0
INTERFACE_PROBE_FAIL scenarios=10 failed=1 mcp_roundtrip  # exit 1

run-probe.sh bootstraps its own Python venv at target/probe-venv (pexpect + pyte, idempotent and offline-tolerant), resolves the binary from $OCEANCODE_BIN — building target/debug/oceancode with cargo build -p ocean-cli when it is unset — and runs the harness. It is wired into the E2E=1 lane after the crash-recovery drills, so a freeze is a red build.

The liveness oracle

Every scenario ends on the same universal check, t.assert_live(), which the harness calls for you after the scenario body returns:

type a unique nonce into the composer, assert the composer echoes it within 1.5 s, then erase it.

A frozen event loop cannot echo a keystroke, so that one assertion is the proof the interface is still alive — no matter what the scenario did before it. Because of this, a scenario only has to assert its own behavior and leave the composer on screen; the liveness check is free.

Driving the TUI interactively (for agents)

The same probe also runs as a persistent session, so an agent can drive the app step by step from separate Bash calls — open it, look, type, look again — instead of only inside a batch scenario. scripts/probe/session.py holds one PTY and pyte screen in a daemon; a client speaks to it over a unix socket:

session.py open <name> --app oceancode|pager|cli|cmd [--cols 120 --rows 40] [--keep] [-- cmd args…]
session.py <name> screen | styled <row> | history | send <text> | key <k> [k…] | paste <text>
session.py <name> wait <needle> [timeout] | wait_re <regex> [timeout] | resize <c> <r> | alive | requests | evidence <file> | close

oceancode / pager boot the real binary on the fake provider in a fresh HOME (the scenarios' isolation, but colours kept so styled can read them); cli runs the cli/ REPL against deployed Studio. Wait for the app's ready marker before driving — ❯ for the pager (oceancode boots Ocean pager too since the native TUI was removed on 2026-10-05), Ask anything for the REPL — and wait for a marker, never sleep (wait polls the screen every 150 ms).

S="target/probe-venv/bin/python scripts/probe/session.py"
$S open pager --app pager            # fresh HOME, fake provider, PTY 120×40
$S pager wait '❯' 40                 # block on the ready marker
$S pager send 'hello'; $S pager key enter
$S pager wait 'ok: hello' 40
$S pager styled 38                   # one row as (fg, bg, bold) runs — colour/bold asserts
$S pager close                       # kill app + fixture, remove the temp HOME

End every interactive check the way a scenario does: type a unique nonce and confirm it echoes in the composer — the liveness oracle, by hand. bash scripts/probe/session-selftest.sh drives all three surfaces from separate processes and prints PROBE_SESSION_OK apps=<n> as its last line; --no-cli runs the hermetic surfaces only (the pager), which is how the E2E=1 lane wires it in.

Adding a scenario

Drop a function named scenario_<name>(t, fx) into any file under scripts/probe/scenarios/. New files and functions are discovered automatically — you never register anything. t is a live Tui, fx a Fixture on the fake provider. Raise AssertionError (or call t.fail(msg)) to fail; a screen excerpt is attached automatically.

# scripts/probe/scenarios/my_feature.py
def scenario_models_open_and_esc(t, fx):
    """/models opens the model picker and Esc returns to a live composer."""
    t.send("/models")
    t.key("enter", 0.6)
    if "MODEL" not in t.text():
        t.fail("/models did not open the model picker")
    t.key("esc", 0.4)
    if t.PLACEHOLDER not in t.text():
        t.fail("Esc did not return to the composer")
    # the harness calls t.assert_live() after this returns

Driving the screen (t):

Call	What it does
`t.send(text)`	type text into whatever has focus
`t.key(name)`	one key: `enter`, `esc`, `tab`, `backtab`, `backspace`, `up`/`down`/`left`/`right`, `home`/`end`, `pgup`/`pgdn`, `ctrl-c`, `ctrl-d`, `ctrl-u`, `shift-enter`, `f1`–`f12`
`t.paste(text)`	bracketed paste
`t.resize(cols, rows)`	resize the terminal (delivers `SIGWINCH`)
`t.wait_for(needle, timeout)`	wait until `needle` (a substring or a compiled regex) is on screen; returns `True`/`False`
`t.text()` / `t.line(n)`	the whole screen / one row, as text
`t.alive()` / `t.pid`	is the process still running / its pid
`t.snapshot(label)`	record a labelled screen for later
`t.fail(msg)`	raise with a screen excerpt attached
`t.assert_live()`	the liveness oracle (also usable mid-scenario)
`t.assert_version_bar()`	assert a `vX.Y.Z` build stamp is in the status bar
`t.PLACEHOLDER`	the composer's empty-prompt placeholder text

Steering the provider (fx): the fake provider picks its reply from MAGIC WORDS in the last user message — put one in the prompt text your scenario sends:

Word	Reply
(none)	one text block `ok: <first 40 chars of the prompt>`
`SLOW`	the same text paced over 30 deltas (a visibly streaming turn)
`ERR500`	HTTP 500 with an Anthropic error body
`HANG`	an open stream that never answers (abort it from the TUI)
`TOOLUSE`	a `tool_use` for the `bash` tool on the first turn (the permission dialog appears), text on the follow-up
`WORKFLOW`	a `tool_use` for the `workflow` tool launching the built-in two-step `pipeline` (its sub-agents get default replies and the run completes) — the pager's Workflows-dock regression guard
`GOAL`	a `tool_use` launching the built-in `goal` supervisor; its planner/skeptic `output_schema` requests are answered with valid JSON so the run converges (Plan → Work → Verify)
`MD`	a markdown-heavy reply (heading, table, fenced code, wide CJK + emoji)
`TASKAGENT`	a `tool_use` for the `task` tool on the first turn: one sub-agent whose prompt carries `DOCKCHILD`; text on the follow-up
`DOCKCHILD`	the sub-agent's reply, 30 words paced over ~12 s, so a probe reads the dock while it runs (`pager_dock_agents`)

Read what the binary actually sent with fx.requests(), fx.last_request(), fx.user_texts() (the last-user text of every request), and fx.base_url / fx.port.

Each scenario is fully isolated: its own HOME, its own fixture project directory carrying its own .claude/commands/{long,short}.md, its own two free ports, and its own fresh fixture process. Scenarios never share state, so -j N runs them in parallel safely.

xfail policy

A scenario for a known, tracked defect is decorated @xfail("REASON-ID"):

from scenarios.xfail import xfail

@xfail("OCA-1234-slash-refusal-toast")
def scenario_unknown_slash_is_refused_visibly(t, fx):
    ...

It is reported as XFAIL <REASON-ID> and does not fail the run; a scenario that is xfail-marked but unexpectedly passes is reported XPASS and counts as a pass. Use @xfail only for a defect that has a tracking id — never to silence a flaky or half-finished scenario. The moment the defect is fixed, remove the decorator so a regression turns the scenario red again.