14 KiB

Raw Blame History

Remote Execution (Split-Brain)

Local agent loop + remote tool execution over SSH.

Motivation

The default ollie deployment runs everything on one machine: the agent loop (LLM calls, prompt assembly, event bus), tool execution (sandbox, bash, file I/O), and the interaction layer (D-Bus signals, GUI). This works for local development but breaks down when:

The codebase lives on a remote server (SSH host, cloud VM, container)
The build environment requires specific hardware or OS
You want local GUI responsiveness while operating on remote filesystems
Network latency makes full-remote agents sluggish (per-token streaming over WAN)

The split-brain model keeps the brain local (agent loop, LLM, D-Bus, GUI) and sends the hands remote (tool execution, sandbox, file access). This is the inverse of 9P remote mount: instead of bringing remote state to your local filesystem, you send local decisions to a remote executor.

Architecture

flowchart TB
    subgraph Local["Local Machine"]
        SRV["olliesrv"]
        AG["agent.Agent\n- LLM streaming\n- prompt assembly\n- event bus\n- compaction"]
        DBUS["D-Bus adapter\n- SessionCreated\n- StateChanged\n- ChatUpdated"]
        GUI["KDE GUI"]
    end

    subgraph Remote["Remote Host (via SSH)"]
        REMOTE_BIN["ollie-remote"]
        EXEC["toolsrv.Server\n- landrun sandbox\n- shell execution\n- tool scripts\n- file I/O"]
        CWD["Working directory:\n/home/user/project/"]
        SANDCFG["Sandbox config:\nremote host's landrun"]
    end

    AG -- "SSH / JSON-RPC" --> REMOTE_BIN
    REMOTE_BIN --> EXEC
    EXEC --> CWD
    EXEC --> SANDCFG
    DBUS -.->|"D-Bus signals"| GUI

Split Point

The natural split is at toolsrv.Runner. Today, agent.Agent calls toolsrv.Server in-process. The remote mode replaces this with a RemoteServer that forwards calls over an SSH-backed RPC channel:

// Local side
type RemoteServer struct {
    conn *rpc.Conn  // JSON-RPC over SSH stdin/stdout
}

func (r *RemoteServer) CallTool(ctx context.Context, tool string, args json.RawMessage) (json.RawMessage, error) {
    return r.conn.Call(ctx, "execute", tool, args)
}

The agent loop doesn't know or care whether execution is local or remote. The NewToolServer factory in session creation just returns a RemoteServer instead of a local toolsrv.Server when the session is configured for remote execution.

SSH Bootstrap

Inspired by goq's approach (toolserv/ssh.go):

Open SSH connection to target host
Send a bootstrap shell script over stdin
Bootstrap checks for cached ollie-remote binary (by SHA-256 hash)
If not cached: receive binary over stdin (gzipped + base64), install to ~/.cache/ollie/bin/
Launch ollie-remote with the working directory as argument
Signal readiness over stderr
Switch to JSON-RPC over stdin/stdout

# Bootstrap (simplified)
HASH="@@HASH@@"
BIN="$HOME/.cache/ollie/bin/ollie-remote-$HASH"
if [ -x "$BIN" ]; then
    echo "GOQ_LOADER_START {\"need_download\":false}" >&2
else
    echo "GOQ_LOADER_START {\"need_download\":true}" >&2
    # ... receive and install binary ...
    echo "GOQ_LOADER_READY" >&2
fi
exec "$BIN" serve --cwd "@@CWD@@"

ollie-remote Binary

A minimal binary containing:

toolsrv.Server (the same one used locally)
Sandbox enforcement (landrun)
Tool script discovery from a bundled or remote-local OLLIE_TOOLS_PATH
JSON-RPC server over stdin/stdout

It does NOT contain:

Agent loop
LLM backends
D-Bus anything
Session management
Prompt resolution

// cmd/ollie-remote/main.go
func main() {
    cwd := flag.Arg(0)
    srv := toolsrv.NewServer(cwd, toolsPath, sandboxConfig)
    rpc.ServeStdio(srv)
}

RPC Protocol

JSON-RPC 2.0 over stdin/stdout (newline-delimited):

// Request (local → remote)
{"jsonrpc":"2.0","id":1,"method":"execute","params":{
  "steps":[{"code":"go build ./..."}],
  "timeout":60,
  "sandbox":"default"
}}

// Response (remote → local)
{"jsonrpc":"2.0","id":1,"result":{
  "output":"...",
  "exit_code":0
}}

// Streaming output (remote → local, notification)
{"jsonrpc":"2.0","method":"output","params":{
  "step":0,"data":"compiling..."
}}

Streaming output notifications allow the local agent to see partial results in real-time (for the event bus / ChatUpdated signals).

Session Configuration

A session targeting a remote host:

# Via 9P
printf 'cwd=/home/user/project\nremote=devbox\n' > $OLLIE/session/new

# Via D-Bus
dbus-send ... org.ollie.SessionManager.CreateSession \
  string:"/home/user/project" string:"" string:"" string:"default" \
  string:"" string:"devbox"  # remote host

Or as a session config key:

remote=user@devbox.internal

The remote field is an SSH target. When set, the session manager:

Bootstraps ollie-remote on the target
Creates a RemoteServer connected over the SSH channel
Passes it to agent.Agent as the tool runner

Tool Scripts

Two strategies for making tool scripts available remotely:

Embedded: ollie-remote bundles the standard tool scripts (file_read, file_grep, etc.) at compile time. Custom tools would need to be synced.
Sync-on-connect: During bootstrap, rsync OLLIE_TOOLS_PATH to the remote ~/.cache/ollie/tools/. This handles custom tools transparently.
Hybrid: Standard tools embedded, custom tools synced on first use (detected by hash mismatch).

Option 2 is simplest and matches goq's philosophy of "the remote is a mirror of local capabilities."

What Stays Local

Component	Location	Why
Agent loop	Local	Latency: LLM streaming is per-token
LLM backends	Local	API keys stay local, streaming stays low-latency
D-Bus signals	Local	GUI responsiveness
Memory	Local	Personal knowledge, not project-specific
Skills	Local	Read during prompt assembly, not execution
Prompts	Local	Assembled before the loop starts
Session persistence	Local	State management is a local concern

What Goes Remote

Component	Location	Why
shell	Remote	Needs remote filesystem, build tools
Named tool scripts	Remote	File I/O targets remote tree
Sandbox	Remote	Landrun enforces on the execution host
Working directory	Remote	The project lives there

Comparison with 9P Remote Mount

	9P Mount	Split-Brain
Agent runs on	Remote	Local
Tools execute on	Remote	Remote
GUI connects to	Remote (via 9P)	Local (D-Bus)
LLM calls from	Remote	Local
Latency profile	All remote	Only tool calls remote
Network requirement	Persistent 9P mount	SSH session per tool call batch
API keys	On remote host	On local machine
Use case	Observe/control remote agents	Develop on remote, interact locally

They complement each other. You might use split-brain for active development (local GUI, remote execution) and 9P mount for monitoring a fleet of headless agents.

Implementation Status

All five phases are implemented:

Phase 1: ollie-remote binary ✓

cmd/ollie-remote/ with main.go
Wraps toolsrv.Server with JSON-RPC stdio interface
Embeds sandbox-remote.yaml and tool scripts via go:embed
RPC methods: shell, list_tools, host_info, ping

Phase 2: SSH bootstrap ✓

toolsrv/bootstrap.sh embedded template
SHA-256 hash-based caching on the remote
Binary transfer: gzip + base64 over stdin
Dial() handles full lifecycle: bootstrap → transfer → exec → JSON-RPC

Phase 3: RemoteServer adapter ✓

toolsrv/remote.go implements toolsrv.Runner
ListTools() calls remote list_tools RPC
CallTool() passes tool name as RPC method (routes shell)
HostInfo fetched after connect (platform, arch, is_git_repo)

Phase 4: Session integration ✓

remote=user@host:port config key in session creation
Eager dial during session creation (gets HostInfo for prompts)
PromptEnvExtra stored on agent for /agent reloads
FUSE mount started directly for remote sessions (prompt resolution)

Phase 5: Tool scripts ✓

Embedded in ollie-remote binary at compile time
justfile copies from tools/ submodule before building
Extracted to tmpdir on startup, cleaned up on exit
file_grep/file_glob fall back to GNU grep/find when rg unavailable
Hot-reload: detect local tool changes, re-sync

Sandbox on Remote

The remote sandbox is simpler than local — ollie-remote only does execution, not session management, prompts, or D-Bus.

Configuration layering

Embedded default — ships with ollie-remote, covers 90% of Linux hosts
.ollie-sandbox.yaml in project root — per-project overrides, version-controlled
Auto-detection — bootstrap detects NixOS, adjusts paths accordingly

Convention over configuration: the common case requires zero setup.

Embedded default config

filesystem:
  ro:
    - "/etc"
    - "/proc"
    - "/sys"
    - "/dev"
    - "{HOME}/.ssh/known_hosts"
    - "{HOME}/.gitconfig"
    - "{HOME}/.netrc"
  rox:
    - "/usr"
    - "/lib"
    - "/lib64"
    - "/bin"
    - "/sbin"
    - "{HOME}/go/bin"
    - "{HOME}/.cargo/bin"
    - "{HOME}/.local/bin"
    - "{HOME}/.nvm"
    - "{HOME}/.pyenv"
    - "{OLLIE_TOOLS_PATH}"
  rwx:
    - "{CWD}"
    - "{TMPDIR}"
    - "{HOME}/.cache"
    - "{HOME}/go"
    - "{HOME}/.cargo"
    - "{HOME}/.npm"
    - "{HOME}/.cache/go-build"
env:
  - PATH
  - HOME
  - TMPDIR
  - GOPATH
  - GOBIN
  - SSH_AUTH_SOCK
network:
  unrestricted: true
general:
  best_effort: true

Per-project override (`.ollie-sandbox.yaml`)

Dropped in the project root, merged on top of the default:

# .ollie-sandbox.yaml — this project needs Docker
filesystem:
  rw:
    - "/var/run/docker.sock"
    - "{HOME}/.docker"
env:
    - DOCKER_HOST

Why it's simpler than local

Local sandbox needs	Remote sandbox needs
OLLIE_* paths (10+)	Just OLLIE_TOOLS_PATH
D-Bus socket	No
Elevation socket	No
9P mount paths	No
Session/transcript dirs	No
Editor integration	No

The remote config is ~20 lines vs ~170 locally.

Kernel compatibility

Linux 5.13+: Full Landlock enforcement
Older kernels: best_effort: true degrades gracefully (unenforced rules skipped)
No Landlock at all: --yolo flag skips sandboxing (explicit opt-in, logged as warning)

ollie-remote ships landrun alongside itself in ~/.cache/ollie/bin/. No system-wide install required.

Open Questions

Elevation on remote: elevated: true escapes the sandbox. On remote, this means escaping the remote sandbox. The elevation socket doesn't exist remotely. Options: skip elevation, or run a remote elevation adapter.
Detached processes: Currently tracked locally. Remote detached processes need their own lifecycle.
Multiple remotes per session: Useful? Or one remote per session is sufficient?
Fallback: If SSH drops mid-execution, retry? Fail the tool call? The agent can handle tool errors gracefully already.
Session restore: remote= is not persisted in PersistedSession. Restored sessions always come back as local.
Eager dial: Session creation blocks on SSH connect. Should be async with lazy tool list.
Streaming output: ✓ Resolved. See Streaming section below.

Lessons Learned

Prompt resolution is always local. The prime script runs on the local machine. It must be a pure template renderer — no local detection of platform/git/cwd.
PRIME_ env vars are the contract.* Callers populate PRIME_CWD, PRIME_PLATFORM, PRIME_IS_GIT_REPO and store them in PromptEnvExtra for /agent reloads.
Sandbox config must overwrite on deploy. Skip-if-exists means binary updates don't propagate config changes. Always write.
Tool scripts must be self-contained. They need fallbacks when rg/ripgrep isn't available (GNU grep, find).
Embed over sync. Embedding tools in the binary (go:embed) is simpler than rsync — single artifact, atomic deploy, no state management.
Build from source, not installed copies. The justfile must copy from the repo's tools/ submodule, not ~/.config/ollie/tools/.
No new event roles. Streaming uses the same "tool" role as final results. Empty content signals stream start; frontends need no changes.

Streaming

Tool output streams in real-time over the SSH channel, visible to the user as it arrives.

Protocol

Remote → local streaming uses JSON-RPC notifications (no id field):

{"jsonrpc":"2.0","result":{"data":"compiling...\n"}}

remote.Server.CallTool reads these in a loop, calling toolsrv.StreamOutput(ctx, data) for each, until the actual response (with matching id) arrives.

Agent Loop Integration

The agent loop wraps the execution context with a streaming callback:

streamCtx := toolsrv.WithOutputStream(ctx, func(data string) {
    emit(cfg, Event{Role: "tool", Name: tc.Name, Content: data})
})

On first chunk, an empty-content tool event is emitted to signal stream start. If streaming occurred, the final result emit is suppressed (the content was already delivered chunk-by-chunk).

Execute Server

limitedWriter has a stream func(string) field, wired to toolsrv.StreamFunc(ctx). Every Write() call flushes the chunk both to the output buffer (for the full result) and to the stream callback (for real-time display).

Chat Log

startEventLog treats tool streaming like assistant streaming:

Empty-content tool event → writes [tool:name]\n header, enters streaming mode
Subsequent tool events → content appended directly (no repeated header)
Non-tool event → closes the stream with a newline

For non-streamed tools (fast results), the single tool event renders as before.

14 KiB Raw Blame History