ollie/doc/resources/remote-execution.md

# Remote Execution (Split-Brain)

Local agent loop + remote tool execution over SSH.

## Motivation

The default ollie deployment runs everything on one machine: the agent loop (LLM calls, prompt assembly, event bus), tool execution (sandbox, bash, file I/O), and the interaction layer (D-Bus signals, GUI). This works for local development but breaks down when:

- The codebase lives on a remote server (SSH host, cloud VM, container)
- The build environment requires specific hardware or OS
- You want local GUI responsiveness while operating on remote filesystems
- Network latency makes full-remote agents sluggish (per-token streaming over WAN)

The split-brain model keeps the **brain local** (agent loop, LLM, D-Bus, GUI) and sends the **hands remote** (tool execution, sandbox, file access). This is the inverse of 9P remote mount: instead of bringing remote state to your local filesystem, you send local decisions to a remote executor.

## Architecture

```mermaid
flowchart TB
    subgraph Local["Local Machine"]
        SRV["olliesrv"]
        AG["agent.Agent\n- LLM streaming\n- prompt assembly\n- event bus\n- compaction"]
        DBUS["D-Bus adapter\n- SessionCreated\n- StateChanged\n- ChatUpdated"]
        GUI["KDE GUI"]
    end

    subgraph Remote["Remote Host (via SSH)"]
        REMOTE_BIN["ollie-remote"]
        EXEC["toolsrv.Server\n- landrun sandbox\n- shell execution\n- tool scripts\n- file I/O"]
        CWD["Working directory:\n/home/user/project/"]
        SANDCFG["Sandbox config:\nremote host's landrun"]
    end

    AG -- "SSH / JSON-RPC" --> REMOTE_BIN
    REMOTE_BIN --> EXEC
    EXEC --> CWD
    EXEC --> SANDCFG
    DBUS -.->|"D-Bus signals"| GUI
```

## Split Point

The natural split is at `toolsrv.Runner`. Today, `agent.Agent` calls `toolsrv.Server` in-process. The remote mode replaces this with a `RemoteServer` that forwards calls over an SSH-backed RPC channel:

```go
// Local side
type RemoteServer struct {
    conn *rpc.Conn  // JSON-RPC over SSH stdin/stdout
}

func (r *RemoteServer) CallTool(ctx context.Context, tool string, args json.RawMessage) (json.RawMessage, error) {
    return r.conn.Call(ctx, "execute", tool, args)
}
```

The agent loop doesn't know or care whether execution is local or remote. The `NewToolServer` factory in session creation just returns a `RemoteServer` instead of a local `toolsrv.Server` when the session is configured for remote execution.

## SSH Bootstrap

Inspired by goq's approach (`toolserv/ssh.go`):

1. Open SSH connection to target host
2. Send a bootstrap shell script over stdin
3. Bootstrap checks for cached `ollie-remote` binary (by SHA-256 hash)
4. If not cached: receive binary over stdin (gzipped + base64), install to `~/.cache/ollie/bin/`
5. Launch `ollie-remote` with the working directory as argument
6. Signal readiness over stderr
7. Switch to JSON-RPC over stdin/stdout

```bash
# Bootstrap (simplified)
HASH="@@HASH@@"
BIN="$HOME/.cache/ollie/bin/ollie-remote-$HASH"
if [ -x "$BIN" ]; then
    echo "GOQ_LOADER_START {\"need_download\":false}" >&2
else
    echo "GOQ_LOADER_START {\"need_download\":true}" >&2
    # ... receive and install binary ...
    echo "GOQ_LOADER_READY" >&2
fi
exec "$BIN" serve --cwd "@@CWD@@"
```

## ollie-remote Binary

A minimal binary containing:

- `toolsrv.Server` (the same one used locally)
- Sandbox enforcement (landrun)
- Tool script discovery from a bundled or remote-local `OLLIE_TOOLS_PATH`
- JSON-RPC server over stdin/stdout

It does NOT contain:
- Agent loop
- LLM backends
- D-Bus anything
- Session management
- Prompt resolution

```go
// cmd/ollie-remote/main.go
func main() {
    cwd := flag.Arg(0)
    srv := toolsrv.NewServer(cwd, toolsPath, sandboxConfig)
    rpc.ServeStdio(srv)
}
```

## RPC Protocol

JSON-RPC 2.0 over stdin/stdout (newline-delimited):

```json
// Request (local → remote)
{"jsonrpc":"2.0","id":1,"method":"execute","params":{
  "steps":[{"code":"go build ./..."}],
  "timeout":60,
  "sandbox":"default"
}}

// Response (remote → local)
{"jsonrpc":"2.0","id":1,"result":{
  "output":"...",
  "exit_code":0
}}

// Streaming output (remote → local, notification)
{"jsonrpc":"2.0","method":"output","params":{
  "step":0,"data":"compiling..."
}}
```

Streaming output notifications allow the local agent to see partial results in real-time (for the event bus / ChatUpdated signals).

## Session Configuration

A session targeting a remote host:

```bash
# Via 9P
printf 'cwd=/home/user/project\nremote=devbox\n' > $OLLIE/session/new

# Via D-Bus
dbus-send ... org.ollie.SessionManager.CreateSession \
  string:"/home/user/project" string:"" string:"" string:"default" \
  string:"" string:"devbox"  # remote host
```

Or as a session config key:
```
remote=user@devbox.internal
```

The `remote` field is an SSH target. When set, the session manager:
1. Bootstraps `ollie-remote` on the target
2. Creates a `RemoteServer` connected over the SSH channel
3. Passes it to `agent.Agent` as the tool runner

## Tool Scripts

Two strategies for making tool scripts available remotely:

1. **Embedded**: `ollie-remote` bundles the standard tool scripts (file_read, file_grep, etc.) at compile time. Custom tools would need to be synced.

2. **Sync-on-connect**: During bootstrap, rsync `OLLIE_TOOLS_PATH` to the remote `~/.cache/ollie/tools/`. This handles custom tools transparently.

3. **Hybrid**: Standard tools embedded, custom tools synced on first use (detected by hash mismatch).

Option 2 is simplest and matches goq's philosophy of "the remote is a mirror of local capabilities."

## What Stays Local

| Component | Location | Why |
|---|---|---|
| Agent loop | Local | Latency: LLM streaming is per-token |
| LLM backends | Local | API keys stay local, streaming stays low-latency |
| D-Bus signals | Local | GUI responsiveness |
| Memory | Local | Personal knowledge, not project-specific |
| Skills | Local | Read during prompt assembly, not execution |
| Prompts | Local | Assembled before the loop starts |
| Session persistence | Local | State management is a local concern |

## What Goes Remote

| Component | Location | Why |
|---|---|---|
| shell | Remote | Needs remote filesystem, build tools |
| Named tool scripts | Remote | File I/O targets remote tree |
| Sandbox | Remote | Landrun enforces on the execution host |
| Working directory | Remote | The project lives there |

## Comparison with 9P Remote Mount

| | 9P Mount | Split-Brain |
|---|---|---|
| Agent runs on | Remote | Local |
| Tools execute on | Remote | Remote |
| GUI connects to | Remote (via 9P) | Local (D-Bus) |
| LLM calls from | Remote | Local |
| Latency profile | All remote | Only tool calls remote |
| Network requirement | Persistent 9P mount | SSH session per tool call batch |
| API keys | On remote host | On local machine |
| Use case | Observe/control remote agents | Develop on remote, interact locally |

They complement each other. You might use split-brain for active development (local GUI, remote execution) and 9P mount for monitoring a fleet of headless agents.

## Implementation Status

All five phases are implemented:

### Phase 1: ollie-remote binary ✓
- `cmd/ollie-remote/` with `main.go`
- Wraps `toolsrv.Server` with JSON-RPC stdio interface
- Embeds `sandbox-remote.yaml` and tool scripts via `go:embed`
- RPC methods: `shell`, `list_tools`, `host_info`, `ping`

### Phase 2: SSH bootstrap ✓
- `toolsrv/bootstrap.sh` embedded template
- SHA-256 hash-based caching on the remote
- Binary transfer: gzip + base64 over stdin
- `Dial()` handles full lifecycle: bootstrap → transfer → exec → JSON-RPC

### Phase 3: RemoteServer adapter ✓
- `toolsrv/remote.go` implements `toolsrv.Runner`
- `ListTools()` calls remote `list_tools` RPC
- `CallTool()` passes tool name as RPC method (routes shell)
- `HostInfo` fetched after connect (platform, arch, is_git_repo)

### Phase 4: Session integration ✓
- `remote=user@host:port` config key in session creation
- Eager dial during session creation (gets HostInfo for prompts)
- `PromptEnvExtra` stored on agent for /agent reloads
- FUSE mount started directly for remote sessions (prompt resolution)

### Phase 5: Tool scripts ✓
- Embedded in ollie-remote binary at compile time
- `justfile` copies from `tools/` submodule before building
- Extracted to tmpdir on startup, cleaned up on exit
- `file_grep`/`file_glob` fall back to GNU grep/find when rg unavailable
- Hot-reload: detect local tool changes, re-sync

## Sandbox on Remote

The remote sandbox is simpler than local — `ollie-remote` only does execution, not session management, prompts, or D-Bus.

### Configuration layering

1. **Embedded default** — ships with `ollie-remote`, covers 90% of Linux hosts
2. **`.ollie-sandbox.yaml` in project root** — per-project overrides, version-controlled
3. **Auto-detection** — bootstrap detects NixOS, adjusts paths accordingly

Convention over configuration: the common case requires zero setup.

### Embedded default config

```yaml
filesystem:
  ro:
    - "/etc"
    - "/proc"
    - "/sys"
    - "/dev"
    - "{HOME}/.ssh/known_hosts"
    - "{HOME}/.gitconfig"
    - "{HOME}/.netrc"
  rox:
    - "/usr"
    - "/lib"
    - "/lib64"
    - "/bin"
    - "/sbin"
    - "{HOME}/go/bin"
    - "{HOME}/.cargo/bin"
    - "{HOME}/.local/bin"
    - "{HOME}/.nvm"
    - "{HOME}/.pyenv"
    - "{OLLIE_TOOLS_PATH}"
  rwx:
    - "{CWD}"
    - "{TMPDIR}"
    - "{HOME}/.cache"
    - "{HOME}/go"
    - "{HOME}/.cargo"
    - "{HOME}/.npm"
    - "{HOME}/.cache/go-build"
env:
  - PATH
  - HOME
  - TMPDIR
  - GOPATH
  - GOBIN
  - SSH_AUTH_SOCK
network:
  unrestricted: true
general:
  best_effort: true
```

### Per-project override (`.ollie-sandbox.yaml`)

Dropped in the project root, merged on top of the default:

```yaml
# .ollie-sandbox.yaml — this project needs Docker
filesystem:
  rw:
    - "/var/run/docker.sock"
    - "{HOME}/.docker"
env:
    - DOCKER_HOST
```

### Why it's simpler than local

| Local sandbox needs | Remote sandbox needs |
|---|---|
| OLLIE_* paths (10+) | Just OLLIE_TOOLS_PATH |
| D-Bus socket | No |
| Bypass socket | No |
| 9P mount paths | No |
| Session/transcript dirs | No |
| Editor integration | No |

The remote config is ~20 lines vs ~170 locally.

### Kernel compatibility

- **Linux 5.13+**: Full Landlock enforcement
- **Older kernels**: `best_effort: true` degrades gracefully (unenforced rules skipped)
- **No Landlock at all**: `--yolo` flag skips sandboxing (explicit opt-in, logged as warning)

`ollie-remote` ships `landrun` alongside itself in `~/.cache/ollie/bin/`. No system-wide install required.

## Open Questions

- **Bypass on remote**: `bypass: true` escapes the sandbox. On remote, this means escaping the remote sandbox. The bypass socket doesn't exist remotely. Options: skip bypass, or run a remote bypass adapter.
- **Detached processes**: Currently tracked locally. Remote detached processes need their own lifecycle.
- **Multiple remotes per session**: Useful? Or one remote per session is sufficient?
- **Fallback**: If SSH drops mid-execution, retry? Fail the tool call? The agent can handle tool errors gracefully already.
- **Session restore**: `remote=` is not persisted in PersistedSession. Restored sessions always come back as local.
- **Eager dial**: Session creation blocks on SSH connect. Should be async with lazy tool list.
- **Streaming output**: ✓ Resolved. See Streaming section below.

## Lessons Learned

1. **Prompt resolution is always local.** The prime script runs on the local machine. It must be a pure template renderer — no local detection of platform/git/cwd.
2. **PRIME_* env vars are the contract.** Callers populate PRIME_CWD, PRIME_PLATFORM, PRIME_IS_GIT_REPO and store them in `PromptEnvExtra` for /agent reloads.
3. **Sandbox config must overwrite on deploy.** Skip-if-exists means binary updates don't propagate config changes. Always write.
4. **Tool scripts must be self-contained.** They need fallbacks when rg/ripgrep isn't available (GNU grep, find).
5. **Embed over sync.** Embedding tools in the binary (go:embed) is simpler than rsync — single artifact, atomic deploy, no state management.
6. **Build from source, not installed copies.** The justfile must copy from the repo's tools/ submodule, not ~/.config/ollie/tools/.
7. **No new event roles.** Streaming uses the same `"tool"` role as final results. Empty content signals stream start; frontends need no changes.

## Streaming

Tool output streams in real-time over the SSH channel, visible to the user as it arrives.

### Protocol

Remote → local streaming uses JSON-RPC notifications (no `id` field):

```json
{"jsonrpc":"2.0","result":{"data":"compiling...\n"}}
```

`remote.Server.CallTool` reads these in a loop, calling `toolsrv.StreamOutput(ctx, data)` for each, until the actual response (with matching `id`) arrives.

### Agent Loop Integration

The agent loop wraps the execution context with a streaming callback:

```go
streamCtx := toolsrv.WithOutputStream(ctx, func(data string) {
    emit(cfg, Event{Role: "tool", Name: tc.Name, Content: data})
})
```

On first chunk, an empty-content tool event is emitted to signal stream start. If streaming occurred, the final result emit is suppressed (the content was already delivered chunk-by-chunk).

### Execute Server

`limitedWriter` has a `stream func(string)` field, wired to `toolsrv.StreamFunc(ctx)`. Every `Write()` call flushes the chunk both to the output buffer (for the full result) and to the stream callback (for real-time display).

### Chat Log

`startEventLog` treats tool streaming like assistant streaming:
- Empty-content tool event → writes `[tool:name]\n` header, enters streaming mode
- Subsequent tool events → content appended directly (no repeated header)
- Non-tool event → closes the stream with a newline

For non-streamed tools (fast results), the single tool event renders as before.