From f63cb3bb377b03021d640628260c1939a419ceb5 Mon Sep 17 00:00:00 2001 From: Ollie Agent Date: Thu, 30 Jul 2026 19:58:53 +0200 Subject: [PATCH] doc: update architecture diagrams to Mermaid, reflect current codebase MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Convert all ASCII diagrams to Mermaid (gitea-rendered) - ARCHITECTURE.md: System Overview, Data Flow, Frontends → Mermaid - CORE.md: Package Map, Per-Iteration Flow → Mermaid - EVOLUTION.md: Timeline → Gantt, coordination/security/frontends → Mermaid - MULTI_AGENT.md: coordination patterns → Mermaid - REMOTE_EXECUTION.md: split-brain architecture → Mermaid - Update all references: agent.Core → *agent.Agent, execute.Server → toolsrv.Server, tools.Dispatcher → toolsrv.Runner, s/ → session/ namespace - Fix namespace layout to show agent/{aid}/ subdirectory hierarchy - Update package layout to reflect flat Go module (no core/ or 9p/ submodules) --- doc/ARCHITECTURE.md | 416 ++++++++++++++++++++++++---------------- doc/CORE.md | 416 ++++++++++++++++++++++------------------ doc/EVOLUTION.md | 310 ++++++++++++++++-------------- doc/MULTI_AGENT.md | 56 ++++-- doc/REMOTE_EXECUTION.md | 79 ++++---- 5 files changed, 730 insertions(+), 547 deletions(-) diff --git a/doc/ARCHITECTURE.md b/doc/ARCHITECTURE.md index 7f58e6e..d065cd8 100644 --- a/doc/ARCHITECTURE.md +++ b/doc/ARCHITECTURE.md @@ -2,7 +2,7 @@ ## Philosophy -ollie's design philosophy is Emacs: a small, extensible core. The Go runtime (`agent.Core`) defines what an agent *is* — an event loop, a backend connection, a message history, and seven built-in primitives (`shell`, `tool_*`, `skill_*`). Everything else — orchestration, scheduling, workflows, UIs — is pushed out to the surrounding environment. +ollie's design philosophy is Emacs: a small, extensible core. The Go runtime (`agent.Agent`) defines what an agent *is* — an event loop, a backend connection, a message history, and three built-in primitives (`shell`, tool registry, skill registry). Everything else — orchestration, scheduling, workflows, UIs — is pushed out to the surrounding environment. The primary integration philosophy is Plan 9's "everything is a file." `olliesrv` exposes agent state and behaviors as files in a 9P namespace. Any program that can read and write files can drive an agent: shell scripts, editors, web apps, cron, containers. @@ -10,117 +10,138 @@ For desktop-native integration, `olliesrv` embeds a D-Bus adapter (`org.ollie.Se Design principles: - **Small extensible core.** The agent runtime is minimal; capabilities come from composing external scripts. -- **Seven built-in primitives.** `shell`, `tool_load`/`tool_list`/`tool_active`, `skill_load`/`skill_list`/`skill_active` are the only tools compiled into the core. Everything else is a script. +- **Three built-in primitives.** `shell`, tool registry (`tool_load`/`tool_list`/`tool_active`), and skill registry (`skill_load`/`skill_list`/`skill_active`) are the only tools compiled into the core. Everything else is a script. - **Two integration paths.** 9P filesystem (canonical) and/or D-Bus (conventional). Same core, different surfaces, selectable via flags. - **No framework lock-in.** Frontends are decoupled via whichever interface they prefer; the core doesn't know or care. ## Repository Structure -The project is a monorepo with Git submodules: +Single Go module (`ollie`) with two Git submodules for decoupled frontends: ``` ollie/ -├── core/ → ollie-core Go library: agent loop, backends, tools, config -├── 9p/ → ollie-9p 9P filesystem server (olliesrv) -├── kde/ → ollie-kde KDE GUI, plasmoid, Kate plugin, KRunner, tray -├── httpgw/ → ollie-httpgw HTTP-to-D-Bus gateway -├── webui/ → ollie-webui Preact SPA (browser frontend) -├── acme/ → ollie-acme Plan 9 acme editor frontend -├── el/ → ollie-el Emacs frontend (ellie.el) -├── skills/ → agent-skills Domain knowledge files -├── agents/ Agent config JSONs (default, coding, orchestrator, worker, ...) -├── prompts/ Prompt templates (markdown) -├── tools/ Tool scripts (file_read, lsp_*, memory_*, subagent_*, ...) -├── scripts/ User-facing scripts organized by namespace (s/, u/, x/) -├── sandbox/ Sandbox config YAML -└── doc/ Documentation +├── agent/ Agent loop, history, hooks, prompt resolution, commands +├── backend/ LLM providers (Anthropic, OpenAI, Ollama, Gemini, Copilot, CodeWhisperer) +├── toolsrv/ Tool server: sandboxed execution, tool registry, skill management +├── session/ Session lifecycle (config, creation, persistence) +├── fs/session/ 9P filesystem tree for sessions +├── dbus/ D-Bus adapter (org.ollie.SessionManager) +├── detach/ Background process management (ring buffer, signal) +├── elevate/ Elevation broker (privilege escalation daemon) +├── sandbox/ Landlock sandbox config YAML +├── env/ Environment variable loading +├── log/ Structured logging +├── paths/ XDG path resolution +├── mount/ 9P FUSE mount client +├── cmd/ Binaries: +│ ├── olliesrv/ 9P server +│ ├── ollie-9p/ 9P client +│ ├── ollie-httpgw/ HTTP-to-9P gateway +│ └── ollie-remote/ Remote execution server +├── kde/ KDE integration (submodule) — plasmoid, GUI, Kate plugin, KRunner, tray +├── webui/ Preact SPA browser frontend (submodule) +├── agents/ Agent config JSONs (default, coding, orchestrator, worker, ...) +├── prompts/ Prompt templates (markdown) +├── tools/ Tool scripts (file_read, lsp_*, memory_*, subagent_*, ...) +├── skills/ Domain knowledge files (markdown) +├── scripts/ User-facing scripts organized by namespace (s/, u/, x/) +├── sandbox/ Sandbox config YAML files +└── doc/ Documentation ``` ## System Overview -``` -┌───────────────────────────────────────────────────────────────────────────┐ -│ Frontends │ -│ s/sh (terminal) │ acme │ ellie (Emacs) │ KDE GUI/plasmoid │ Web UI │ -└───────┬──────────┴───┬──┴───────┬───────┴────────┬─────────┴──────┬──────┘ - │ 9P │ 9P │ 9P │ D-Bus │ D-Bus - │ │ │ ▼ ▼ - │ │ │ ┌──────────────────────────────────┐ - │ │ │ │ org.ollie.SessionManager (D-Bus)│ - │ │ │ └──────────────┬───────────────────┘ - ▼ ▼ ▼ │ embedded adapter -┌───────────────────────────────────────────────────────▼───────────────────┐ -│ olliesrv (9P server + D-Bus adapter) │ -│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ -│ │ Sessions │ │ backends │ │ models │ │ -│ │ s/ │ │ (file) │ │ (file) │ │ -│ └────┬─────┘ └──────────┘ └──────────┘ │ -└───────┼───────────────────────────────────────────────────────────────────┘ - │ per-session - ▼ +```mermaid +flowchart TB + subgraph Frontends["Frontends"] + TUI["s/sh (terminal)"] + ACME["acme (Plan 9)"] + ELLIE["ellie (Emacs)"] + KDE["KDE GUI / Plasmoid"] + WEB["Web UI"] + end -D-Bus-only mode: olliesrv -no9p (same interface, no filesystem listener) -┌────────────────────────────────────────────────────────────────┐ -│ agent.Core (per session) │ -│ ┌─────────────┐ ┌──────────────┐ ┌────────────────────────┐ │ -│ │ Agent Loop │ │ Session │ │ tools.Dispatcher │ │ -│ │ (loop.go) │ │ (messages, │ │ ┌──────────────────┐ │ │ -│ │ │ │ task state, │ │ │ execute.Server │ │ │ -│ │ stream ←────┼──┤ compaction) │ │ │ (shell, │ │ │ -│ │ dispatch ───┼──┼───────────────┼──┤ │ tool_*, skill_*)│ │ │ -│ └─────────────┘ └──────────────┘ │ └────────┬─────────┘ │ │ -│ └───────────┼────────────┘ │ -└──────────────────────────────────────────────────┼──────────────┘ - │ - ┌──────────────────────────────────────────┼──────┐ - │ Tool Execution │ │ - │ ┌─────────────┐ ┌───────────────────┐ │ │ - │ │ landrun │ │ Tool Scripts │ │ │ - │ │ (sandbox) │ │ (OLLIE_TOOLS_PATH)│ │ │ - │ └─────────────┘ └───────────────────┘ │ │ - └──────────────────────────────────────────────────┘ - │ - ▼ - ┌─────────────────┐ - │ LLM Backends │ - │ Ollama│OpenAI│ │ - │ Anthropic│... │ - └─────────────────┘ + subgraph Integration["Integration Layer"] + direction LR + P9["9P Filesystem"] + DBUS["org.ollie.SessionManager (D-Bus)"] + end + + subgraph Server["olliesrv"] + direction TB + NS["9P Namespace\nsession/ · backends · models"] + DBA["Embedded D-Bus Adapter"] + end + + subgraph Core["Agent Engine (per session)"] + LOOP["Agent Loop\n(agent/loop.go)"] + SESS["Session State\n(agent/history.go)"] + TRV["toolsrv.Server\n· shell execution\n· tool registry\n· landlock sandbox"] + end + + subgraph Backends["LLM Backends"] + OLLAMA["Ollama"] + OPENAI["OpenAI / OpenRouter"] + ANTHROPIC["Anthropic"] + GEMINI["Gemini"] + CW["CodeWhisperer"] + end + + TUI --> P9 + ACME --> P9 + ELLIE --> P9 + KDE --> DBUS + WEB --> DBUS + + P9 --> NS + DBUS --> DBA + + NS --> LOOP + DBA --> LOOP + + LOOP <--> SESS + LOOP --> TRV + LOOP <--> Backends + + TRV --> Shell["sandboxed shell\n(landrun)"] + TRV --> Tools["tool scripts\n(OLLIE_TOOLS_PATH)"] ``` -## Core Library (`core/`) +## Core Library -The core is a Go library (`ollie/`) with no binary. Frontends import it as a dependency. +The core is a single Go module (`ollie`) with no binary. Binaries live in `cmd/`. ### Package Layout | Package | Purpose | |---|---| -| `agent/` | Core interface, agent loop, compaction, hooks, session persistence, config loading | +| `agent/` | Agent struct, loop, history, compaction, hooks, commands, prompt resolution, state | | `backend/` | Backend interface + LLM providers (Anthropic, OpenAI, Ollama, Gemini, Copilot, CodeWhisperer) | -| `tools/` | Server/Dispatcher interfaces, tool registry, discovery, schema parsing | -| `execute/` | Sandboxed shell execution, promoted tool dispatch, skill dispatch, remote SSH | +| `toolsrv/` | Tool server: sandboxed shell execution, named tool dispatch, skill management, result tiering, remote execution | +| `session/` | Session lifecycle, config, persistence | | `detach/` | Background process management (ring buffer, signal) | -| `skills/` | Skill metadata parsing and per-session registry | | `elevate/` | Elevation broker (privilege escalation daemon, policy) | | `sandbox/` | Landrun sandbox configuration and command wrapping | | `env/` | Session environment helpers | | `log/` | Structured logging | | `paths/` | XDG path resolution | -### Key Interfaces +### Key Types -**`agent.Core`** — the public API frontends drive: +**`agent.Agent`** — the concrete struct frontends drive: ```go -type Core interface { - Submit(ctx context.Context, input string) - Interrupt(cause error) bool - Queue(prompt string) - State() string // "idle" | "thinking" | "calling: " - WaitChange(ctx context.Context, field, current string) (string, bool) - Close() - // ... model/backend info, CWD, usage, cost, etc. +type Agent struct { + // unexported fields — no public interface } + +func New(cfg AgentConfig) *Agent +func (a *Agent) Submit(ctx context.Context, input string) +func (a *Agent) Interrupt(cause error) bool +func (a *Agent) Queue(prompt string) +func (a *Agent) State() string +func (a *Agent) WaitChange(ctx context.Context, field, current string) (string, bool) +func (a *Agent) Close() +// ... model/backend info, CWD, usage, cost, etc. ``` **`backend.Backend`** — LLM provider contract: @@ -135,21 +156,17 @@ type Backend interface { } ``` -**`tools.Server`** — any tool provider: +**`toolsrv.Runner`** — minimal tool execution interface: ```go -type Server interface { +type Runner interface { ListTools() ([]ToolInfo, error) CallTool(ctx context.Context, tool string, args json.RawMessage) (json.RawMessage, error) } ``` -**`tools.Dispatcher`** — routes tool calls to servers: +**`toolsrv.Server`** — the concrete tool server (implements Runner): ```go -type Dispatcher interface { - AddServer(name string, s Server) - ListTools() ([]ToolInfo, error) - Dispatch(ctx context.Context, server, tool string, args json.RawMessage) (json.RawMessage, error) -} +type Server struct { /* ... */ } ``` ### Agent Loop (`agent/loop.go`) @@ -157,7 +174,7 @@ type Dispatcher interface { The loop is the heart of the system. On each turn: 1. **Stream** — call `backend.ChatStream()` with messages + tool definitions -2. **Dispatch** — if the model returns tool calls, execute them via `tools.Dispatcher` +2. **Dispatch** — if the model returns tool calls, execute them via `toolsrv.Server` 3. **Update** — append assistant message + tool results to session history 4. **Repeat** — loop until the model produces a final text response (no tool calls) @@ -169,9 +186,9 @@ Safety mechanisms: - **Transient retry**: up to 3 retries with exponential backoff for rate limits and 5xx - **Context overflow**: triggers automatic compaction and retry -### Session & Context Management (`agent/session.go`) +### Session & Context Management (`agent/history.go`) -`Session` owns the message history as a flat `[]backend.Message` slice. Key behaviors: +`History` owns the message history as a flat `[]backend.Message` slice. Key behaviors: - **TaskState**: structured JSON overlay (objective, plan_step, constraints, last_action, next_decision) injected at the top of every turn - **Compaction**: three-zone strategy when context reaches 50% of model limit: @@ -200,50 +217,78 @@ Lifecycle callbacks executed as shell commands with JSON payload on stdin: Agent configs declare prompts as a JSON array of shell commands. Each command is executed with `$OLLIE` and `$PWD` available; stdout is concatenated to form the system prompt. This enables dynamic, composable prompts assembled from template fragments. -## 9P Server (`9p/`) +## 9P Server (`cmd/olliesrv/`) `olliesrv` is the central daemon. It implements a 9P2000 file server that exposes the entire agent namespace: ### Namespace Layout -| Path | Contents | -|---|---| -| `s/` | Sessions — each session is a directory | -| `s/{id}/chat` | Full chat history (streamable) | -| `s/{id}/prompt` | Write to submit a prompt | -| `s/{id}/ctl` | Control commands (stop, kill, compact, model, ...) | -| `s/{id}/cfg` | Session config (key=value) | -| `s/{id}/state` | Current state (idle/thinking/calling) | -| `s/{id}/statewait` | Blocking read until state changes | -| `s/{id}/plan` | Session-scoped markdown checklist | - +``` +session/ +├── new write key=value to create session +├── idx session index (tab-separated) +├── ls exec to list sessions +├── kill exec to kill a session +├── sh interactive shell +├── b batch one-shot (create, submit, wait, print, kill) +├── bfg batch foreground (submit to existing, wait, print) +├── bbg batch background (submit, return path) +├── cleanup kill idle sessions +├── {sid}/ +│ ├── plan session-scoped markdown checklist +│ ├── env session environment +│ └── agent/ +│ └── {aid}/ +│ ├── cfg session config (key=value) +│ ├── ctl control commands (stop, kill, compact, model, ...) +│ ├── state current state (idle/thinking/calling) +│ ├── statewait blocking read until state changes +│ ├── chat full chat history (streamable) +│ ├── offset byte offset after last user prompt +│ ├── prompt write to submit a prompt +│ ├── prompt.prev last submitted prompt +│ ├── fifo.in queue a prompt (FIFO input) +│ ├── fifo.out pop queued prompt (FIFO output) +│ ├── context rendered context window +│ ├── systemprompt rendered system prompt +│ ├── usage token usage stats +│ ├── cost estimated cost +│ ├── ctxsz context size +│ ├── models available models +│ ├── tools tool management (read all, write name to load) +│ ├── toolsrv.loaded currently loaded tools +│ ├── toolsrv.rev revision counter +│ ├── tail exec helper +│ └── proc/ detached background processes +``` ### 9P Interaction Model All agent interaction uses the `ollie-9p` client, which auto-discovers the server via `$NAMESPACE` and identifies the caller via `$OLLIE_UNAME`. The 9P filesystem exposes session state, control, and inter-agent communication as files: -- Sessions are directories under `s/` +- Sessions are directories under `session/` - Prompts, state, chat history, and config are readable/writable files - Permission enforcement uses 9P user principals -### Permission Model (`9p/fs/perm.go`) +### Permission Model (`fs/session/perm.go`) -All permissions are declared in a single registry (`fs.Perms`). Key design: -- `prompt` is mode `0466` — group+other can write, owner (the agent) cannot +All permissions are declared in a single registry (`Perms`). Key design: +- `prompt` is mode `0666` — group+other can write, owner (the agent) cannot - `chat` is mode `0444` — read-only for everyone - `ctl` is mode `0666` — anyone can send control commands - `plan` is mode `0666` — readable by peer agents -### Session Manager (`9p/session/`) +### Session Management (`fs/session/`) -The `Manager` handles session lifecycle: -- Creating sessions (write to `s/new`) -- Killing sessions (delete `s/{id}` or write "kill" to `s/{id}/ctl`) -- Renaming sessions -- Assigning unique numeric UIDs for 9P principals -- Wiring up `agent.Core` with backends and tool servers -- Persisting/restoring sessions across restarts +The `*fs.Tree` IS the session collection. Package functions manage lifecycle: +- `NewRoot(cfg)` — create the root tree +- `Lookup(tree, id)` — find a session by ID +- `All(tree)` — list all sessions +- `CreateFromRoot(tree, args)` — create a new session +- `KillFromRoot(tree, id)` — kill a session +- `RenameFromRoot(tree, old, new)` — rename a session +- `Shutdown(tree)` — clean shutdown -### Embedded D-Bus Adapter (`9p/dbus.go`) +### Embedded D-Bus Adapter `olliesrv` includes an embedded D-Bus adapter that claims `org.ollie.SessionManager` on the session bus. This provides: - The same method/signal interface used by KDE frontends @@ -253,9 +298,9 @@ The `Manager` handles session lifecycle: The adapter is best-effort: if no session bus is available (e.g., headless/container), `olliesrv` continues functioning as a pure 9P server. -## D-Bus Daemon (`dbus/`) +## D-Bus Adapter (`dbus/`) -For D-Bus-only deployments (no 9P), run `olliesrv -no9p`. This disables the filesystem listener and per-session FUSE mounts while keeping the full D-Bus interface. Session persistence uses the same `~/.local/share/ollie/sessions/active/` directory. +The D-Bus adapter is embedded in `olliesrv` (no separate daemon). For D-Bus-only deployments, run `olliesrv -no9p`. This disables the filesystem listener while keeping the full D-Bus interface. Session persistence uses the same `~/.local/share/ollie/sessions/active/` directory. ## KDE Integration (`kde/`) @@ -275,13 +320,15 @@ All KDE components use `QDBusServiceWatcher` to handle daemon restarts gracefull ### Architecture -The tool system has exactly one built-in server (`execute.Server`) that exposes three executors: +The tool system has exactly one built-in server (`toolsrv.Server`) that exposes three primitives: | Primitive | Purpose | |---|---| | `shell` | Run a bash command in a sandbox | +| `tool_load`/`tool_list`/`tool_active` | Dynamic tool registry | +| `skill_load`/`skill_list`/`skill_active` | Dynamic skill registry | -All other capabilities are **tool scripts** — executable files discovered from `OLLIE_TOOLS_PATH` (default: `~/.config/ollie/tools`). +All other capabilities are **tool scripts** — executable files discovered from `OLLIE_TOOLS_PATH` (default: `~/.config/ollie/tools/`). ### Tool Scripts @@ -293,10 +340,11 @@ Tool scripts are plain executables with a structured header comment declaring th - `reasoning_think` — scratchpad for externalized reasoning - `subagent_spawn`, `subagent_generate` — sub-agent lifecycle - `route` — orchestrator dispatch +- `gui_*` — KDE desktop automation (screenshot, windows, clipboard, input, ...) Scripts annotated `ollie:parallel read` can be fanned out concurrently by the agent loop. -### Sandboxing (`internal/sandbox/`) +### Sandboxing Every code execution step is wrapped with [landrun](https://github.com/landlock-lsm/landrun) (Landlock LSM). Configuration is YAML-based: @@ -323,6 +371,9 @@ All backends implement the streaming-only `Backend` interface: | `OllamaBackend` | `/api/chat` | Local models, context length from `/api/show` | | `OpenAIBackend` | `/v1/chat/completions` | OpenAI, OpenRouter, any compatible API | | `AnthropicBackend` | `/v1/messages` | Extended thinking, prompt caching | +| `GeminiBackend` | Gemini API | Google Gemini models | +| `CopilotBackend` | GitHub Copilot API | Copilot Chat | +| `CodeWhispererBackend` | CodeWhisperer API | AWS CodeWhisperer | Shared infrastructure: - `streamRequest()` — common HTTP streaming with error classification @@ -335,10 +386,10 @@ Multi-agent coordination is built entirely on the filesystem primitive: ### Mechanism -1. **Spawn**: write session spec to `s/new` → new session directory appears -2. **Prompt**: write task to `s/{id}/prompt` -3. **Observe**: read `s/{id}/state` or block on `s/{id}/statewait` -4. **Result**: child writes to `s/{parent_id}/prompt` when done +1. **Spawn**: write session spec to `session/new` → new session directory appears +2. **Prompt**: write task to `session/{id}/prompt` +3. **Observe**: read `session/{id}/state` or block on `session/{id}/statewait` +4. **Result**: child writes to `session/{parent_id}/prompt` when done ### Delegation Tools @@ -359,26 +410,53 @@ ARTIFACTS: The parent never actively waits — results arrive as queued prompts. -## HTTP Gateway (`httpgw/`) +## HTTP Gateway (`cmd/ollie-httpgw/`) -`ollie-httpgw` bridges HTTP clients (like the web UI) to the D-Bus session manager. It translates REST-style requests into `org.ollie.SessionManager` method calls and streams `ChatUpdated` signals back as Server-Sent Events. +`ollie-httpgw` bridges HTTP clients (like the web UI) to the 9P filesystem. It translates REST-style requests into 9P filesystem operations and streams events back as Server-Sent Events. -This is the secondary integration path for non-LLM consumers (web UIs, dashboards, external tools) that want a conventional HTTP API without speaking 9P or D-Bus directly. +This is the secondary integration path for non-LLM consumers (web UIs, dashboards, external tools) that want a conventional HTTP API without speaking 9P directly. ## Frontends Frontends connect via either 9P (filesystem) or D-Bus, depending on preference: -| Frontend | Technology | Connection | -|---|---|---| -| `s/sh` | Terminal (readline) | 9P | -| `ollie-acme` | Plan 9 acme editor | 9P | -| `ellie` | Emacs Lisp | 9P | -| KDE GUI | Qt/QML standalone app | D-Bus | -| Plasmoid | Plasma widget | D-Bus | -| Kate plugin | KTextEditor plugin | D-Bus | -| KRunner | Runner plugin | D-Bus | -| Web UI | Preact SPA | D-Bus (via `httpgw`) | +```mermaid +flowchart LR + subgraph Core["agent.Agent"] + AG["Agent Engine"] + end + + subgraph Surfaces["Integration Surfaces"] + P9["9P Filesystem\n(session/ namespace)"] + DB["D-Bus\n(org.ollie.SessionManager)"] + end + + subgraph Frontends + SH["s/sh (terminal)"] + ACME["acme (Plan 9)"] + EL["ellie (Emacs)"] + KG["KDE GUI"] + KP["KDE Plasmoid"] + KK["Kate Plugin"] + KR["KRunner"] + WEB["Web UI"] + HTTP["curl / scripts"] + end + + AG --> P9 + AG --> DB + + SH --> P9 + ACME --> P9 + EL --> P9 + HTTP --> P9 + + KG --> DB + KP --> DB + KK --> DB + KR --> DB + WEB --> DB +``` Frontends are fully decoupled — the core doesn't know which interface they use. @@ -419,28 +497,42 @@ The `prompt` array is the key innovation: each element is a shell command whose ## Data Flow -``` -User input (via any frontend) - │ - ▼ write to s/{id}/prompt -olliesrv - │ - ▼ agent.Core.Submit() -agent loop (loop.go) - │ - ├── backend.ChatStream() → LLM API call (streaming) - │ - ├── tools.Dispatcher.Dispatch() → execute.Server - │ │ - │ ├── {code} steps → landrun sandbox → subprocess - │ └── {tool} steps → landrun sandbox → tool script - │ - ├── session.update() → append to message history - │ - └── loop (until no tool calls) - │ - ▼ event bus → session log -Frontend reads s/{id}/chat (streaming) +```mermaid +sequenceDiagram + participant F as Frontend + participant S as olliesrv + participant A as agent.Agent + participant B as LLM Backend + participant T as toolsrv.Server + participant SH as sandbox (landrun) + + F->>S: write to session/{id}/prompt + S->>A: Submit() + activate A + + loop Agent Loop + A->>B: ChatStream(messages, tools) + activate B + B-->>A: stream events + deactivate B + + alt has tool calls + A->>T: Dispatch(tool, args) + activate T + T->>SH: sandboxed execution + SH-->>T: result + T-->>A: ToolResult + deactivate T + A->>A: append to history + else no tool calls + A->>A: turn complete + break + end + end + + A-->>S: turn complete + deactivate A + S-->>F: readable via session/{id}/chat ``` ## Extension Points @@ -449,7 +541,7 @@ Frontend reads s/{id}/chat (streaming) |---|---| | Add an LLM backend | Implement `backend.Backend` | | Add a tool | Drop an executable into `OLLIE_TOOLS_PATH` | -| Add a tool server | Implement `tools.Server`, register in `NewDispatcherFunc` | +| Add a tool server | Implement `toolsrv.Runner`, register in `NewToolServer` | | Add a frontend | Connect via 9P (filesystem) or D-Bus (method calls) | | Add a skill | Drop a markdown file into `skills/` | | Add an agent persona | Create a JSON config in `agents/` | @@ -458,7 +550,7 @@ Frontend reads s/{id}/chat (streaming) ## Key Design Decisions 1. **Two integration surfaces** — 9P filesystem (canonical, Unix philosophy) and D-Bus (conventional, for GUI/web consumers). Both are thin adapters over the same core. -2. **Seven built-in primitives** — `shell`, `tool_*`, `skill_*` are the only compiled-in capabilities. This keeps the core small and makes all other tools equal citizens. +2. **Three built-in primitives** — `shell`, tool registry, skill registry. Everything else is a script. This keeps the core small and makes all other tools equal citizens. 3. **Mandatory sandboxing** — no unsandboxed fallback. Security is not optional. 4. **Streaming-only backends** — simplifies the interface; blocking APIs wrap as single-event streams. 5. **Fire-and-forget sub-agents** — no active waiting, no polling. Results arrive as filesystem writes. diff --git a/doc/CORE.md b/doc/CORE.md index 99e8aec..e3e7e32 100644 --- a/doc/CORE.md +++ b/doc/CORE.md @@ -1,115 +1,139 @@ # ollie-core: Detailed Architecture -This document covers the internal design of the `core/` library — the Go package that implements the agent engine. It is the dependency imported by all consumers (olliesrv, frontends, tests). +This document covers the internal design of the Go library — the packages that implement the agent engine. It is the dependency imported by all consumers (olliesrv, frontends, tests). ## Package Map -``` -core/ +```mermaid +flowchart TB + subgraph Agent["agent/ package"] + AGENT["agent.go\nAgent struct, New()"] + LOOP["loop.go\nrun() — agent loop"] + HISTORY["history.go\nHistory, TaskState, compaction"] + TURN["turn.go\nper-turn state machine"] + HOOKS["hooks.go\nlifecycle callbacks"] + CMDS["commands.go\n/command dispatch"] + BUILD["build_runtime.go\nBuildRuntime()"] + PROMPT["prompt_resolver.go\nsystem prompt assembly"] + STATE["state.go\nResultTier, toolResult"] + COST["cost.go\npricing table"] + FIFO["fifo.go\nprompt queue"] + COMPLETE["complete.go\ncode completion"] + CONFIG["agent_config.go\nConfig schema"] + CONFIG_PATHS["config_paths.go\nXDG config discovery"] + RUNTIME["runtime.go\nRuntime struct"] + NEW["new.go\nNew() factory"] + USAGE["usage_log.go\nusage tracking"] + end -│ ├── agent/ The agent engine -│ │ ├── types.go Core interface, Event, PromptFIFO -│ │ ├── runtime.go Runtime struct (swappable per-agent config) -│ │ ├── core.go AgentCoreConfig, agent struct, NewAgentCore, BuildRuntime -│ │ ├── loop.go The agent loop (run function) -│ │ ├── session.go Session, TaskState, compaction, persistence -│ │ ├── state.go state interface, ResultTier, toolResult -│ │ ├── hooks.go Hooks map, HookResult, hook execution -│ │ ├── commands.go Slash-command dispatch (/model, /agent, /compact, ...) -│ │ ├── cost.go Model pricing table, cost computation -│ │ └── prompt_resolver.go System prompt assembly from shell commands -│ ├── backend/ LLM provider implementations -│ │ ├── backend.go Backend interface, shared types, streamRequest -│ │ ├── openai.go OpenAI/OpenRouter/compatible APIs -│ │ ├── anthropic.go Anthropic Messages API -│ │ └── ollama.go Ollama local inference -│ ├── config/ Agent config JSON schema and loader -│ ├── env/ Environment variable loading -│ ├── log/ Structured logger -│ ├── paths/ XDG directory resolution -│ └── tools/ Tool system -│ ├── tools.go Server, Dispatcher, classifier interfaces -│ └── execute/ The execute.Server -│ ├── server.go Server struct, Decl factory, ListTools, options -│ ├── code.go Step resolution, validation, dispatch routing -│ ├── tool.go Named tool script discovery and execution -│ ├── flock.go Advisory locking, parallelism classification -│ └── tier.go Result tier classification (hot/warm/cold) -└── internal/ - └── sandbox/ Landlock sandbox wrapper - ├── config.go YAML config, path expansion, access checking - └── wrapper.go landrun command construction + subgraph Backend["backend/ package"] + BACKEND["backend.go\nBackend interface"] + OPENAI["openai.go"] + ANTHROPIC["anthropic.go"] + OLLAMA["ollama.go"] + GEMINI["gemini.go"] + COPILOT["copilot.go"] + CW["codewhisperer.go"] + POOL["pool.go\nbackend pool"] + end + + subgraph Toolsrv["toolsrv/ package"] + SERVER["server.go\nServer struct"] + SHELL["shell.go\nshell execution"] + TOOLS["tools.go\nRunner interface"] + REGISTRY["registry.go\ntool discovery"] + DISCOVER["discover.go\ntool script scanning"] + SCHEMA["schema.go\nJSON Schema parsing"] + TIER["tier.go\nresult tiering"] + SKILLS["skills.go\nskill management"] + STREAM["stream.go\noutput streaming"] + REMOTE["remote.go\nRemoteServer"] + DIAL["dial.go\nSSH bootstrap"] + ACCESSORS["accessors.go\nIsParallelRead, ResultTier"] + BOOTSTRAP["bootstrap.sh\nSSH bootstrap script"] + end + + subgraph Session["session/ package"] + SESS["session.go\nSession, Config"] + end + + subgraph FsSession["fs/session/ package"] + TREE["tree.go\n*fs.Tree"] + ROOT["root.go\nNewRoot()"] + CREATE["create.go\nCreateFromRoot()"] + FILES["files.go\nfile operations"] + PERM["perm.go\npermission registry"] + PERSIST["persist.go\nsession persistence"] + FSGO["fs.go\n9P filesystem"] + FORMAT["format.go\nevent formatting"] + SYNTH["synth.go\nsynthetic files"] + TYPES["types.go\nSession struct"] + end + + subgraph Support["Supporting Packages"] + DETACH["detach/\nprocess management"] + ELEVATE["elevate/\nelevation broker"] + SANDBOX["sandbox/\nLandlock config"] + ENV["env/\nenvironment"] + LOG["log/\nstructured logging"] + PATHS["paths/\nXDG resolution"] + end + + Agent --> Backend + Agent --> Toolsrv + Agent --> Session + FsSession --> Session + Toolsrv --> SANDBOX + Toolsrv --> DETACH + Toolsrv --> ELEVATE ``` -## The Core Interface (`agent/types.go`) +## The Agent Struct (`agent/agent.go`) -`Core` is the boundary between the engine and any consumer. It is deliberately narrow: +`Agent` is the boundary between the engine and any consumer. It is a concrete struct, not an interface — the interface was removed during the Great Flattening because no consumer ever implemented it independently. ```go -type Core interface { - // Turn lifecycle - Submit(ctx context.Context, input string) // process user input (commands or prompts) - Interrupt(cause error) bool // cancel current turn - Inject(prompt string) // append to next tool result - Queue(prompt string) // FIFO for deferred prompts - PopQueue() (string, bool) // drain one queued prompt - - // Observation - IsRunning() bool - State() string // "idle" | "thinking" | "calling: " - Reply() string // last assistant text - Bus() *pubsub.Bus // event stream - WaitChange(ctx context.Context, field, current string) (string, bool) - - // Configuration - AgentName() string - BackendName() string - ModelName() string - CWD() string - SetCWD(dir string) error - SetSessionID(newID string) error - SetEnv(key, value string) - SetGenerationParams(params backend.GenerationParams) error - - // Introspection - Context() []backend.Message - SystemPrompt() string - GenerationParams() backend.GenerationParams - CtxSz() string - Usage() string - Cost() string - ListModels() string - ToolCallCount() int64 - - // Cleanup - Close() +type Agent struct { + history *History + runtime *Runtime + cfg agentConfig + agentName string + agentsDir string + baseLayers []string + promptEnvExtra []string + newToolServer func() toolsrv.Runner + newBackend func(string) (backend.Backend, error) + currentAction atomic.Pointer[actionHandle] + // ... execution state, sync primitives, session metadata } + +func New(cfg AgentConfig) *Agent ``` All output flows through the event bus as `Event{Role, Name, Content}` values. Roles include: `assistant`, `tool`, `call`, `info`, `exec`, `error`, `usage`, `reasoning`, `limitretry`. -## Agent Construction (`agent/core.go`) +## Agent Construction (`agent/new.go`) -### AgentCoreConfig +### AgentConfig The full construction config: | Field | Purpose | |---|---| -| `Backend` | Active LLM backend instance | -| `ModelName` | Override backend's default model | +| `History` | Pre-existing session history to restore (nil = new) | +| `Runtime` | Pre-built *Runtime (tools, hooks, prompt, params) | | `AgentName` | Name of the active agent config | | `AgentsDir` | Path to agent JSON directory | | `SessionsDir` | Path to persisted session files | | `SessionID` | Unique session identifier | -| `Uname` | Immutable 9P user principal | +| `AgentID` | Immutable agent principal | | `CWD` | Working directory for tools and prompt | -| `Session` | Pre-existing session to restore (nil = new) | -| `Runtime` | Pre-built *Runtime (tools, hooks, prompt, params) | -| `NewDispatcher` | Factory for fresh Dispatcher on agent switch | +| `NewToolServer` | Factory for fresh toolsrv.Runner on agent switch | | `NewBackend` | Factory for backend by name | | `Log` | Logger instance | | `MaxSteps` | Override agent config's maxSteps | +| `ReadPlanStep` | Function to read next unchecked plan step | +| `ListHandlers` | Additional list handlers | ### Runtime @@ -118,67 +142,68 @@ The `Runtime` struct holds all per-agent configuration that changes on an `/agen ```go type Runtime struct { Backend backend.Backend - Dispatcher tools.Dispatcher + Runner toolsrv.Runner Hooks Hooks Preamble string // compiled system prompt Tools []backend.Tool - Exec toolExecutor - ClassifyTool toolClassifier - ClassifyTier func(string) ResultTier + ClassifyTool func(string) bool // parallelism check + ClassifyTier func(string, json.RawMessage) string GenParams backend.GenerationParams MaxSteps int - ToolResultMaxBytes int // 0 means use default (128KB) - CfgBackend string // backend override from agent JSON - CfgModel string // model override from agent JSON - Messages []string // non-fatal warnings from construction + ToolResultMaxBytes int + CfgBackend string + CfgModel string + Messages []string // non-fatal warnings } ``` ### BuildRuntime -`BuildRuntime(cfg, dispatcher, cwd, env)` wires everything together: +`BuildRuntime(cfg, runner, cwd, env)` wires everything together: -1. Lists all tools from the dispatcher +1. Lists all tools from the runner 2. Resolves the system prompt by executing the config's `prompt` commands 3. Extracts generation parameters from the config -4. Builds the tool executor closure (routes calls through the dispatcher) -5. Extracts parallelism and tier classifiers from the execute server +4. Builds the tool executor closure (routes calls through the runner) +5. Extracts parallelism and tier classifiers 6. Applies tool/executor restrictions if configured -7. Returns a `*Runtime` struct ready for `NewAgentCore` +7. Returns a `*Runtime` struct ready for `New` -### NewAgentCore +### New -Creates the `agent` struct from `AgentCoreConfig`: +Creates the `Agent` struct from `AgentConfig`: 1. Sweeps stale tmpdirs from previous crashes 2. Stores the provided `Runtime` (or creates an empty one) 3. Sets the backend model if `ModelName` is specified 4. Creates `/tmp/ollie/{sessionID}` for this session -5. Sets up the plan file reader (reads `s/{id}/plan` for the next unchecked step) +5. Sets up the plan file reader 6. Wires the `turnError` hook with a detached context -7. Pushes `OLLIE_SESSION_ID` and `OLLIE_UNAME` into the execute server env +7. Pushes `OLLIE_SESSION_ID` and `OLLIE_AGENT_ID` into the runner env 8. Sets the flock directory for parallel coordination ## The Agent Loop (`agent/loop.go`) The `run()` function is the core execution engine. It takes an `agentConfig` and a `state` interface, and loops until the model stops calling tools. -The `agentConfig` struct bundles per-turn dependencies: `Backend`, `Tools`, `Exec` (tool executor closure), `ClassifyTool`/`ClassifyTier` (parallelism and retention classifiers), `ToolResultMaxBytes`, `Output` (event handler), `preamble`, `GenerationParams`, and lifecycle callbacks: `PopInject`, `AutoCompact`, `SaveSession`, `PreTool`, `PostTool`, `TurnError`, `MaxSteps`, `ReadPlanStep`, and `IncrToolCallCount`. - ### Per-Iteration Flow -``` -1. Context gate: strip cold material if over budget -2. Build message history: - - If TaskState exists: system prompt + task state + hot tail only - - Otherwise: system prompt + full history -3. Stream from backend (with retry logic) -4. If no tool calls: break (turn complete) -5. Execute tool calls (serial or parallel) -6. Update state (append messages + results) -7. Infer TaskState updates from what happened -8. Safety checks (error limits, stall, replan, max steps) -9. Auto-compact if needed -10. Loop +```mermaid +flowchart TB + START["Start turn"] + GATE["Context gate:\nstrip cold material\nif over budget"] + BUILD["Build message history:\nsystem prompt + task state + hot tail"] + STREAM["Stream from backend\n(with retry logic)"] + CHECK{"Tool calls\nin response?"} + EXEC["Execute tool calls\n(serial or parallel)"] + UPDATE["Update state:\nappend messages + results"] + INFER["Infer TaskState\nupdates"] + SAFETY["Safety checks:\nerror limits, stall,\nreplan, max steps"] + COMPACT["Auto-compact\nif needed"] + DONE["Turn complete"] + + START --> GATE --> BUILD --> STREAM --> CHECK + CHECK -- yes --> EXEC --> UPDATE --> INFER --> SAFETY --> COMPACT --> BUILD + CHECK -- no --> DONE ``` ### Streaming & Retry @@ -187,9 +212,9 @@ The streaming loop handles three failure modes: | Failure | Detection | Recovery | |---|---|---| -| Rate limit (429) | `*RateLimitError` | Exponential backoff (5×2^attempt seconds), up to 3 retries | +| Rate limit (429) | `*RateLimitError` | Exponential backoff (5x2^attempt seconds), up to 3 retries | | Transient (5xx, network) | `*TransientError` | Same retry logic | -| Stream drop | `Done` never received, channel closes | Retry with 2×2^attempt second delay | +| Stream drop | `Done` never received, channel closes | Retry with 2x2^attempt second delay | | Context overflow | `*ContextOverflowError` | Trigger compaction, retry | | Tool unsupported | `*ToolUnsupportedError` | Propagate to turnError hook | @@ -231,13 +256,13 @@ After each tool round, `inferTaskStateUpdate` updates the structured overlay: - `LastAction` ← most recent tool call + truncated result - `PlanStep` ← next unchecked item from the plan file (or heuristic from assistant text) -## Session & Context (`agent/session.go`) +## Session & Context (`agent/history.go`) ### Message History -`Session` holds a flat `[]backend.Message` slice. No rolling window — grows until compaction. +`History` holds a flat `[]backend.Message` slice. No rolling window — grows until compaction. -**Checkpoint**: `Session.Checkpoint(ts TaskState)` forks the session — returns a new Session that inherits the given TaskState but starts with a clean message history. Used for narrow-context sub-agents that know what to do without inheriting parent message noise. +**Checkpoint**: `History.Checkpoint(ts TaskState)` forks the session — returns a new History that inherits the given TaskState but starts with a clean message history. Used for narrow-context sub-agents that know what to do without inheriting parent message noise. Token tracking per session: - `TotalInputTokens`, `TotalOutputTokens`, `TotalRequests` @@ -275,7 +300,7 @@ Tool results are classified at execution time: | Cold | Immediately collapsed to one-line summary | `file_read`, `memory_recall`, `lsp_*` | Classification sources (in priority order): -1. Built-in table in `execute/tier.go` (`defaultColdTools`) +1. Built-in table in `toolsrv/tier.go` (`defaultColdTools`) 2. `ollie:tier` annotation in the tool script header 3. Default: hot @@ -291,7 +316,7 @@ Sessions are saved as `PersistedSession` JSON: } ``` -Saved after every tool round (`SaveSession` callback) and on clean exit. +Saved after every tool round (`saveSession` callback) and on clean exit. ## Hooks (`agent/hooks.go`) @@ -341,52 +366,38 @@ Multiple commands per hook run sequentially. If any exits 2, execution stops imm | `/sp` | Show rendered system prompt | | `/i ` | Inject into running turn | | `/irw ` | Inject-rewrite | -| `/queued [pop\|clear]` | Manage queued prompts | +| `/queued [pop|clear]` | Manage queued prompts | | `/help` | Show command help | ### Agent Switching When `/agent ` is issued: 1. Load the new agent config JSON -2. Create a fresh dispatcher via `newDispatcher()` +2. Create a fresh runner via `newToolServer()` 3. Call `BuildRuntime` with the new config 4. If the config specifies a backend/model, switch those too 5. Replace the entire `Runtime` pointer on the agent struct 6. Clear the session (fresh context for the new persona) -## Tool System (`tools/`) +## Tool System (`toolsrv/`) -### Dispatcher +### Runner Interface -The `Dispatcher` is a simple name→Server router: +The `Runner` is the minimal interface satisfied by any tool server (local or remote): ```go -type Dispatcher interface { - AddServer(name string, s Server) - GetServer(name string) (Server, bool) +type Runner interface { ListTools() ([]ToolInfo, error) - Dispatch(ctx context.Context, server, tool string, args json.RawMessage) (json.RawMessage, error) + CallTool(ctx context.Context, tool string, args json.RawMessage) (json.RawMessage, error) } ``` -`NewDispatcherFunc` returns a factory that creates a fresh dispatcher with all servers instantiated. Called on agent switches to get clean server state. +### Server -### Optional Server Interfaces - -Servers may implement additional interfaces for runtime configuration: - -| Interface | Purpose | -|---|---| -| `CWDSetter` | Update working directory | -| `EnvSetter` | Inject session env vars | -| `LockDirSetter` | Set flock directory for parallel coordination | -| `ToolRestrictionSetter` | Restrict available executors/tools | -| `ParallelClassifier` | Report if a tool is read-safe | -| `TierClassifier` | Report result retention tier | - -### execute.Server (`execute/`) - -The built-in server exposes seven primitives: `shell`, `tool_load`, `tool_list`, `tool_active`, `skill_load`, `skill_list`, `skill_active`. +The concrete `Server` struct implements `Runner` and provides: +- **shell**: sandboxed bash command execution +- **tool registry**: `tool_load`, `tool_list`, `tool_active` — lazy tool discovery +- **skill registry**: `skill_load`, `skill_list`, `skill_active` — skill management #### shell @@ -414,7 +425,7 @@ Before execution, inline code is checked against dangerous patterns: Validation failures are rate-limited: 5 failures within 1 minute triggers a 5-minute block. -### Parallelism & Locking (`execute/flock.go`) +### Parallelism & Locking Tool scripts declare their concurrency class via header annotations: @@ -426,7 +437,7 @@ Tool scripts declare their concurrency class via header annotations: Lock files live in `/tmp/ollie/{sessionID}/` and are acquired via `flock(2)`. -### Result Tier Classification (`execute/tier.go`) +### Result Tier Classification (`toolsrv/tier.go`) Built-in cold tools (results consumed immediately, don't need verbatim retention): - `file_read`, `memory_recall`, `web_fetch`, `web_search` @@ -438,9 +449,29 @@ Custom tools declare their tier via `ollie:tier cold|warm|hot` in the script hea Steps marked `elevated: true` bypass the sandbox entirely. They are dispatched to the elevation backend (`x/elevate`) which runs outside landrun. Only bash is supported for elevated steps. -### FUSE Mount +### Remote Execution -When `WithMount()` is set, the execute server spawns `ollie-9p-mount -u {uname}` to create a per-session FUSE mount. This gives the agent its own 9P identity for permission enforcement. The mount path is exposed as `$OLLIE` in subprocess environments. +When a session is configured with `remote=user@host`, the runner is a `RemoteServer` that forwards tool calls over SSH to an `ollie-remote` binary: + +```go +type RemoteServer struct { + conn *rpc.Conn // JSON-RPC over SSH stdin/stdout +} + +func (r *RemoteServer) CallTool(ctx context.Context, tool string, args json.RawMessage) (json.RawMessage, error) { + return r.conn.Call(ctx, "execute", tool, args) +} +``` + +The agent loop doesn't know or care whether execution is local or remote. The `NewToolServer` factory returns a `RemoteServer` instead of a local `Server` when the session is configured for remote execution. + +SSH bootstrap: +1. Open SSH connection to target host +2. Send bootstrap script over stdin +3. Bootstrap checks for cached `ollie-remote` binary (by SHA-256 hash) +4. If not cached: receive binary over stdin (gzipped + base64) +5. Launch `ollie-remote` with the working directory +6. Switch to JSON-RPC over stdin/stdout ## Backend System (`backend/`) @@ -493,33 +524,42 @@ Shared HTTP streaming infrastructure used by all backends: - 200k context window (hardcoded) - SSE streaming with content_block_delta events +**GeminiBackend** (Gemini API): +- Google Gemini models + +**CopilotBackend** (GitHub Copilot API): +- Copilot Chat integration + +**CodeWhispererBackend** (AWS CodeWhisperer API): +- AWS CodeWhisperer integration + ## Config System (`agent/`) ### Agent Config Schema ```go type Config struct { - Hooks map[string]HookCmds // lifecycle callbacks - Prompt Prompt // system prompt (string or command array) - Backend string // backend override - Model string // model override - Tools *bool // enable/disable tools (default: true) - AllowExecutors []string // restrict executors - AllowTools []string // restrict tool scripts - MaxTokens int // generation limit - MaxCompletionTokens int // OpenAI o-series - MaxSteps int // tool rounds per turn + Hooks map[string]HookCmds + Prompt Prompt + Backend string + Model string + Tools *bool + AllowExecutors []string + AllowTools []string + MaxTokens int + MaxCompletionTokens int + MaxSteps int Temperature *float64 TopP, TopK, MinP, TopA *float64/*int FrequencyPenalty *float64 PresencePenalty *float64 RepetitionPenalty *float64 - Reasoning int // Anthropic thinking budget - ReasoningEffort string // OpenAI: low/medium/high - IncludeReasoning *bool // OpenRouter - ResponseFormat string // json_object, text - Stop []string // stop sequences - Verbosity string // output detail level + Reasoning int + ReasoningEffort string + IncludeReasoning *bool + ResponseFormat string + Stop []string + Verbosity string } ``` @@ -563,32 +603,32 @@ When the backend doesn't report cost directly (only OpenRouter does), cost is co - Cache creation tokens charged at 125% of input rate - Unknown/local models return zero cost -## Sandbox (`internal/sandbox/`) +## Sandbox (`sandbox/`) ### Config Format ```yaml general: - best_effort: true # don't fail if kernel doesn't support all features - log_level: "" # landrun log level + best_effort: true + log_level: "" filesystem: - ro: ["/usr", "/lib", "/etc"] # read-only - rox: ["/usr/bin", "/usr/local/bin"] # read + execute - rw: ["{CWD}", "{TMPDIR}"] # read + write - rwx: [] # read + write + execute + ro: ["/usr", "/lib", "/etc"] + rox: ["/usr/bin", "/usr/local/bin"] + rw: ["{CWD}", "{TMPDIR}"] + rwx: [] network: enabled: true - unrestricted: true # all TCP allowed - bind_tcp: [] # specific bind ports - connect_tcp: [] # specific connect targets + unrestricted: true + bind_tcp: [] + connect_tcp: [] -env: [] # extra environment variables +env: [] advanced: - ldd: false # resolve shared library paths - add_exec: false # add execute to all read paths + ldd: false + add_exec: false ``` ### Template Variables @@ -606,32 +646,28 @@ advanced: | `{OLLIE_CFG_PATH}` | ollie config directory | | `{OLLIE_DATA_PATH}` | ollie data directory | -### Path Validation - -`checkPath(cfg, path, write, cwd, getenv)` validates that a given absolute path falls within the sandbox's allowed paths. Symlinks are resolved before checking. Used by file tools to pre-validate paths before execution. - ## Event System The agent communicates exclusively through events published on a `pubsub.Bus`: ```go type Event struct { - Role string // event type - Name string // tool name (for call/tool events) - Content string // payload + Role string + Name string + Content string } ``` | Role | When | |---|---| | `assistant` | Streaming text delta from the model | -| `reasoning` | Thinking/reasoning delta (wrapped in `` tags) | +| `reasoning` | Thinking/reasoning delta | | `call` | Tool call initiated (Name=tool, Content=args JSON) | | `tool` | Tool result returned (Name=tool, Content=result) | | `info` | Informational message (command output, status) | | `exec` | Shell command output (`!cmd`) | | `error` | Error message | -| `usage` | Token usage report (space-separated numbers) | +| `usage` | Token usage report | | `limitretry` | Rate limit retry initiated | Consumers subscribe to the `"event"` topic on the bus. The 9P server's session manager subscribes to build the chat log; frontends subscribe to render output. @@ -650,4 +686,4 @@ Consumers subscribe to the `"event"` topic on the bus. The 9P server's session m -<6-hex-chars> ``` -Lexicographically sortable by creation time. Unique even within the same nanosecond due to the random suffix. +Lexicographically sortable by creation time. Unique even within the same nanosecond due to the random suffix. \ No newline at end of file diff --git a/doc/EVOLUTION.md b/doc/EVOLUTION.md index 25792cf..f3da745 100644 --- a/doc/EVOLUTION.md +++ b/doc/EVOLUTION.md @@ -5,44 +5,48 @@ How the Ollie system-of-systems emerged and evolved. ## Timeline -``` - Apr May Jun Jul - |----|--------|--------|--------|--------| -Foundation - Monorepo + submodules === - 9P filesystem . ============ +```mermaid +gantt + title Ollie Evolution + dateFormat YYYY-MM-DD + axisFormat %b %d -Core Primitives - Session lifecycle (s/) ================== - Tool scripts ============ - Skills system ======== + section Foundation + Monorepo + submodules :done, 2026-04-11, 14d + 9P filesystem :done, 2026-04-14, 28d -Execution - execute_code ========== - Batch jobs (b/→s/bfg) ================== - Sandbox (landrun) ==================================== + section Core Primitives + Session lifecycle (session/) :done, 2026-04-20, 30d + Tool scripts :done, 2026-05-01, 20d + Skills system :done, 2026-05-10, 14d -Multi-Agent - Subagent spawn ========== - Cascade orchestrator ========== - JIT agent generation ==== + section Execution + execute_code / shell :done, 2026-05-08, 20d + Batch jobs (b/ → session/bfg) :done, 2026-05-15, 25d + Sandbox (landrun) :done, 2026-05-20, 40d -Frontends - TUI + Emacs (ellie) ===================================== - Acme (Plan 9) ============================ - Web UI + httpgw ==================== - KDE/Plasma ======================== + section Multi-Agent + Subagent spawn :done, 2026-05-25, 15d + Cascade orchestrator :done, 2026-06-05, 20d + JIT agent generation :done, 2026-07-10, 10d -Control Planes - D-Bus adapter (ollied) =============== - D-Bus embedded in srv ===== + section Frontends + TUI + Emacs (ellie) :done, 2026-04-15, 45d + Acme (Plan 9) :done, 2026-05-20, 30d + Web UI + httpgw :done, 2026-06-01, 25d + KDE/Plasma :done, 2026-06-10, 40d -Late Evolution - Remote execution ========== - Elevation broker ========== - Tool registry (lazy) ====== - Reactions / feedback ===== - FUSE → 9P client ==== + section Control Planes + D-Bus adapter (ollied) :done, 2026-06-25, 20d + D-Bus embedded in srv :done, 2026-07-15, 10d + + section Late Evolution + Remote execution :done, 2026-07-05, 15d + Elevation broker :done, 2026-07-10, 15d + Tool registry (lazy) :done, 2026-07-20, 10d + Reactions / feedback :done, 2026-07-22, 8d + FUSE → 9P client :done, 2026-07-25, 5d + The Great Flattening :done, 2026-07-29, 2d ``` ## Phase 1: Monorepo Bootstrap (Apr 11) @@ -50,8 +54,8 @@ Late Evolution Started as independent git repos unified under a monorepo with submodules. Initial components: -- **core** — agent loop (Go): backend dispatch, tool execution, session state -- **9p** — 9P file server exposing agent sessions as a synthetic filesystem +- **agent** — agent loop (Go): backend dispatch, tool execution, session state +- **fs/session** — 9P file server exposing agent sessions as a synthetic filesystem - **tui** — terminal UI (later removed) - **el** — Emacs integration (ellie.el) @@ -61,13 +65,13 @@ Build system was `mkfile` (Plan 9 make), later `Makefile`, finally `justfile`. The defining architectural choice: **every agent primitive is a file**. -Sessions are directories. Sending a prompt is writing to `s/{id}/prompt`. -Reading state is `cat s/{id}/state`. This made the control plane +Sessions are directories. Sending a prompt is writing to `session/{id}/prompt`. +Reading state is `cat session/{id}/state`. This made the control plane protocol-agnostic — any program that can read/write files can be a frontend. Key files crystallized: `prompt`, `chat`, `state`, `ctl`, `cfg`, `statewait`. -The `s/new` file (write key=value pairs, read back session ID) became the +The `session/new` file (write key=value pairs, read back session ID) became the session factory. ## Phase 3: Tool System (Apr 28 – May 10) @@ -95,7 +99,7 @@ Planning went through the most iterations of any subsystem: 3. **Beads** integration (external issue tracker) — made optional, then dropped 4. `task_add` / `task_check` tools 5. Inline markdown plans forbidden -6. **Final form**: a single `s/{id}/plan` file (markdown checklist), written by +6. **Final form**: a single `session/{id}/plan` file (markdown checklist), written by the agent, persisted across context compaction The lesson: planning needed to be simple, agent-controlled, and not bureaucratic. @@ -109,91 +113,100 @@ Three patterns emerged: - **cascade** — script-driven fan-out with `-max-workers`, `-retries`, optional synthesis step -``` - ┌──────────────────┐ - │ User prompt │ - └────────┬─────────┘ - │ - ┌────────▼─────────┐ - │ Parent Session │ - └──┬──────┬──────┬─┘ - subagent_ │ │ │ subagent_ - spawn │ │ │ spawn - ┌────────▼┐ ┌──▼───┐ ┌▼────────┐ - │Worker 1 │ │Wkr 2 │ │Worker 3 │ - └────┬────┘ └──┬───┘ └────┬────┘ - │ │ │ - └──────────┼───────────┘ - write to parent's prompt +```mermaid +flowchart TB + UP["User prompt"] + PARENT["Parent Session"] + W1["Worker 1"] + W2["Worker 2"] + W3["Worker 3"] + CASCADE["u/cascade\n(spawn + throttle)"] + CW1["Cascade Worker"] + CW2["Cascade Worker"] - ┌────────▼─────────┐ - │ Parent Session │──── cascade ───┐ - └──────────────────┘ │ - ┌─────────▼──────────┐ - │ u/cascade │ - │ (spawn + throttle) │ - └───┬──────────┬─────┘ - │ │ - ┌────▼───┐ ┌───▼────┐ - │Cascade │ │Cascade │ - │Worker │ │Worker │ - └────────┘ └────────┘ + UP --> PARENT + PARENT -- subagent_spawn --> W1 + PARENT -- subagent_spawn --> W2 + PARENT -- subagent_spawn --> W3 + W1 --> PARENT + W2 --> PARENT + W3 --> PARENT + PARENT -- cascade --> CASCADE + CASCADE --> CW1 + CASCADE --> CW2 ``` Inter-agent communication uses the filesystem: agents write to each other's -`prompt` file or use bidirectional `peer/` links. +`prompt` file. ## Phase 6: Frontend Proliferation -``` - ┌─────────────────┐ - │ agent.Core │ - └───┬─────────┬───┘ - │ │ - ┌──────▼───┐ ┌───▼──────┐ - │ 9P Server│ │D-Bus Svc │ - └──┬──┬──┬┘ └──┬─────┬──┘ - │ │ │ │ │ - ┌─────────┘ │ └──┐ │ └──────────┐ - │ │ │ │ │ - ┌─────▼───┐ ┌────▼──┐ │ ├────────┐ ┌────▼────┐ - │ s/sh │ │ acme │ │ │ KDE │ │ Web UI │ - └─────────┘ └───────┘ │ │Plasmoid│ │(httpgw) │ - ┌────▼──┤KRunner │ └─────────┘ - │ellie │Kate │ - └───────┤GUI │ - └────────┘ +```mermaid +flowchart TB + subgraph Core["agent.Agent"] + AG["Agent Engine"] + end + + subgraph Surfaces["Integration Surfaces"] + P9["9P Filesystem\n(session/ namespace)"] + DB["D-Bus\n(org.ollie.SessionManager)"] + end + + subgraph Frontends + SH["s/sh (terminal)"] + ACME["acme (Plan 9)"] + EL["ellie (Emacs)"] + KG["KDE GUI"] + KP["KDE Plasmoid"] + KK["Kate Plugin"] + KR["KRunner"] + WEB["Web UI"] + HTTP["curl / scripts"] + end + + AG --> P9 + AG --> DB + + SH --> P9 + ACME --> P9 + EL --> P9 + HTTP --> P9 + + KG --> DB + KP --> DB + KK --> DB + KR --> DB + WEB --> DB ``` - **s/sh** — shell script frontend (bash, briefly rc, back to bash) - **acme** — Plan 9 editor integration with `Oi` (quick query), `Kmpl` (completion) - **ellie.el** — Emacs with ghost-text completion -- **httpgw** — HTTP→D-Bus gateway +- **httpgw** — HTTP→9P gateway - **webui** — browser frontend via httpgw - **ollie-kde** — full Plasma integration: plasmoid, KRunner, Kate plugin, standalone GUI, GUI automation tools - **TUI** — removed (Jul) in favor of s/sh and richer GUIs ## Phase 7: Security Model (May – Jul) -``` - 9P Permissions Execution Sandbox - ────────────── ───────────────── - Owner → rw Agent - Agent → restricted │ - Peers → read-only ▼ - landrun sandbox (restricted fs) - │ - ▼ - Tool scripts - │ needs escape? - ▼ - x/elevate (socket broker) - │ - ▼ - Integrated Broker - │ user confirms - ▼ - Privileged Action +```mermaid +flowchart LR + subgraph Permissions["9P Permissions"] + OWNER["Owner → rw"] + AGENT["Agent → restricted"] + PEERS["Peers → read-only"] + end + + subgraph Execution["Execution Sandbox"] + AGT["Agent"] + SANDBOX["landrun sandbox\n(restricted fs)"] + TOOLS["Tool scripts"] + ELEV["x/elevate\n(socket broker)"] + PRIV["Privileged Action"] + end + + AGT --> SANDBOX --> TOOLS + TOOLS -- needs escape --> ELEV --> PRIV ``` Evolution: @@ -210,7 +223,7 @@ Evolution: Originally everything was 9P-only. D-Bus was added for desktop integration: 1. Separate `ollie-dbus` daemon (ollied) -2. Embedded directly into `olliesrv` (9P server got `-nodbus` flag) +2. Embedded directly into `olliesrv` (9P server got `-no9p` flag) 3. KDE uses D-Bus exclusively; acme/sh/el use 9P 4. **9P and D-Bus do not share sessions** — independent stores, same core @@ -242,25 +255,24 @@ This solved system prompt bloat — only load what's needed per task. Largest single-day structural change: **−4,573 lines net** across the codebase. Eliminated accumulated abstractions that no longer served a purpose. -### Core repo (−4,400 lines) +### Core (−4,400 lines) - **Killed `agent.Core` interface** — concrete `*agent.Agent` used directly. Nobody else implemented Core; the interface just added indirection. -- **Killed `session.Core` interface** — same pattern, same outcome. -- **Killed `Dispatcher` indirection** in toolsrv — tools.Server called directly. +- **Killed `Dispatcher` indirection** in toolsrv — `toolsrv.Server` called directly. - **Merged `execute/` into `tools/`** — separate package for 3 functions was pointless overhead. -- **Renamed packages for clarity**: `agent.Session` → `agent.History`, +- **Renamed for clarity**: `agent.Session` → `agent.History`, `session.Core` → `session.Session`, `NewAgentCore` → `New`. - **Extracted `agent/` package** from the monolithic session package. - **Threaded context.Context** properly through daemon → session → agent (replaced ad-hoc interrupt mechanisms with cancellation). - Dead code removal: `GlobalToolNames`, `ExtractReturnSchema`. -### 9P server repo (−214 lines) +### 9P server (−214 lines) - **Killed `mgr/` package entirely** — replaced `Manager` struct with package - functions in `session/`. The `fs.Tree` IS the session collection; `rootState` + functions in `fs/session/`. The `*fs.Tree` IS the session collection; `rootState` (unexported) lives in `tree.Data`. CRUD via `NewRoot`, `Lookup`, `Create`, `Kill`, `Rename`, `Shutdown`. - **Fixed infinite recursion** in `session/{sid}/agent/` directory — `pathType()` @@ -291,23 +303,30 @@ session/ │ ├── env session environment │ └── agent/ │ └── {aid}/ -│ ├── chat conversation log -│ ├── prompt submit prompts -│ ├── cfg agent configuration -│ ├── ctl control commands -│ ├── state current state -│ ├── statewait blocks until state changes -│ ├── usage token stats -│ ├── cost cost estimate -│ ├── ctxsz context size +│ ├── cfg session configuration (key=value) +│ ├── ctl control commands +│ ├── state current state +│ ├── statewait blocks until state changes +│ ├── chat conversation log +│ ├── prompt submit prompts +│ ├── prompt.prev last submitted prompt +│ ├── offset byte offset after last user prompt +│ ├── fifo.in queue a prompt (FIFO input) +│ ├── fifo.out pop queued prompt (FIFO output) +│ ├── context rendered context window │ ├── systemprompt rendered system prompt -│ ├── tools tool management -│ ├── tools.loaded currently loaded tools -│ └── proc/ -│ └── {pid} detached process output +│ ├── usage token stats +│ ├── cost cost estimate +│ ├── ctxsz context size +│ ├── models available models +│ ├── tools tool management +│ ├── toolsrv.loaded currently loaded tools +│ ├── toolsrv.rev revision counter +│ ├── tail exec helper +│ └── proc/ detached process output ``` -### Architecture after +### Architecture after (single Go module, no submodules for core/9p) ``` *fs.Tree (root) ← IS the session collection @@ -328,8 +347,8 @@ Package functions (no Manager): | Component | Lines | |-----------|-------| -| core (minus backend) | 10,437 | -| 9p server | 7,925 | +| agent + backend + toolsrv + session | 10,437 | +| fs/session + cmd/olliesrv | 7,925 | | kde gui | 12,331 | | el + acme + httpgw | 1,845 | | **Total** | **32,538** | @@ -339,30 +358,37 @@ Zero dead exported functions remain (verified via LSP + grep across all repos). ## Current Topology ``` -ollie/ ← monorepo root -├── core/ ← agent loop, backends, tool dispatch (Go) -├── 9p/ ← 9P server + embedded D-Bus adapter (Go) -├── acme/ ← Plan 9 acme frontend (Go) -├── el/ ← Emacs integration (elisp) -├── kde/ ← KDE Plasma: plasmoid, GUI, Kate, KRunner (C++/QML) -├── httpgw/ ← HTTP→D-Bus gateway (Go) -├── webui/ ← browser frontend (JS) -├── remote/ ← split-brain execution server (Go) +ollie/ ← single Go module +├── agent/ ← agent loop, history, hooks, commands +├── backend/ ← LLM providers (6 backends) +├── toolsrv/ ← tool server, registry, remote execution +├── session/ ← session lifecycle, config +├── fs/session/ ← 9P filesystem tree +├── dbus/ ← D-Bus adapter +├── detach/ ← background process management +├── elevate/ ← elevation broker +├── sandbox/ ← landrun sandbox config +├── env/ ← environment helpers +├── log/ ← structured logging +├── paths/ ← XDG path resolution +├── mount/ ← 9P FUSE mount client +├── cmd/ ← binaries (olliesrv, ollie-9p, ollie-httpgw, ollie-remote) +├── kde/ ← KDE Plasma (submodule) +├── webui/ ← browser frontend (submodule) ├── tools/ ← 37 tool scripts (Python) -├── scripts/{s,u,x}/ ← filesystem-hosted scripts +├── scripts/{s,u,x}/ ← filesystem-hosted scripts ├── agents/ ← agent configs (JSON) ├── prompts/ ← system prompt fragments (markdown) ├── skills/ ← domain knowledge modules (markdown) ├── sandbox/ ← landrun sandbox profiles (YAML) -├── services/ ← init/service files -└── experiments/ ← multi-agent trial docs +└── doc/ ← documentation ``` ## Principles That Emerged 1. **Filesystem-as-API** — everything is read/write on synthetic files. No custom protocols needed. -2. **Submodule independence** — each component has its own repo, tests, build. +2. **Submodule independence** — KDE and webui have their own repos, tests, build. Monorepo coordinates versions. 3. **Scripts over servers** — MCP servers removed early. Simple scripts won. 4. **Progressive disclosure** — lazy tool/skill loading keeps system prompts @@ -388,4 +414,4 @@ ollie/ ← monorepo root | Tier routing (FAST/POWER) | Replaced by /route endpoint with real model discovery | | call_tool / pipe | Replaced by native tool registry with tool_load | | execute_code (multi-lang) | Simplified to shell-only; tool scripts handle language choice | -| chatwait | Reverted; acme tails chat directly | +| chatwait | Reverted; acme tails chat directly | \ No newline at end of file diff --git a/doc/MULTI_AGENT.md b/doc/MULTI_AGENT.md index 8d20c0e..a9b41c0 100644 --- a/doc/MULTI_AGENT.md +++ b/doc/MULTI_AGENT.md @@ -11,11 +11,13 @@ Every session lives under `/session/{id}/`: | Path | Purpose | |------|---------| | `/session/new` | Create a session (write `name`, `backend`, `model`, `agent`, `cwd`) | -| `/session/{id}/prompt` | Submit a prompt (dispatched async on close) | -| `/session/{id}/chat` | Cumulative conversation history (append-only read) | -| `/session/{id}/cfg` | Session config (KV): state, backend, model, agent, cwd, params | -| `/session/{id}/statewait` | **Blocking read** — returns when state changes | -| `/session/{id}/ctl` | Control commands: `stop`, `kill`, `compact`, `rn ` | +| `/session/{id}/plan` | Session-scoped markdown checklist | +| `/session/{id}/env` | Session environment variables | +| `/session/{id}/agent/{aid}/prompt` | Submit a prompt (dispatched async on close) | +| `/session/{id}/agent/{aid}/chat` | Cumulative conversation history (append-only read) | +| `/session/{id}/agent/{aid}/cfg` | Agent config (KV): state, backend, model, agent, cwd, params | +| `/session/{id}/agent/{aid}/statewait` | **Blocking read** — returns when state changes | +| `/session/{id}/agent/{aid}/ctl` | Control commands: `stop`, `kill`, `compact`, `rn ` | Utility files at the session root (`/session/`) provide higher-level operations: @@ -31,9 +33,8 @@ Global namespaces are shared across all sessions: | Path | Purpose | |------|---------| | `/session/idx` | Session index (id, state, cwd, backend, model) | -| `/session/{id}/prompt` | Submit a prompt to a peer session | | `/session/{id}/plan` | Shared session-scoped checklist | -| `/session/{id}/chat` | Full chat history (searchable) | +| `/session/{id}/agent/{aid}/chat` | Full chat history (searchable) | ## Spawning Agents @@ -44,15 +45,27 @@ subagent_spawn [-name NAME] [-agent AGENT] [-backend BACKEND] [-model MODEL] [-c ``` This creates a child session, writes the prompt, and returns the session path -(e.g. `s/my-session`). Child sessions carry the parent session ID in their name, +(e.g. `session/my-session`). Child sessions carry the parent session ID in their name, so the spawning tree is recoverable from session IDs alone. ## Coordination Patterns ### Hub-and-Spoke (Conductor) +```mermaid +flowchart TB + ROOT["Root Agent\n(reasoning)"] + W1["Worker A\n(cheap model)"] + W2["Worker B\n(cheap model)"] + + ROOT -- "subagent_spawn" --> W1 + ROOT -- "subagent_spawn" --> W2 + W1 -- "statewait → read chat" --> ROOT + W2 -- "statewait → read chat" --> ROOT +``` + A root agent reasons about a task and delegates subtasks to workers. Workers -report back via their chat history; the conductor reads `/session/{id}/chat` after +report back via their chat history; the conductor reads `/session/{id}/agent/{aid}/chat` after `statewait` unblocks. ```sh @@ -73,6 +86,13 @@ to cheaper models; reasoning stays on the expensive one. ### Pipeline +```mermaid +flowchart LR + A["Step 1\nExtract TODOs"] + B["Step 2\nPrioritise"] + A -->|"/tmp/todos.txt"| B +``` + Agent A produces output; agent B consumes it. Composition via `/tmp/` or direct path passing. @@ -84,6 +104,20 @@ subagent_spawn "prioritise the TODOs in /tmp/todos.txt and produce a plan" ### Scatter-Gather (Swarm) +```mermaid +flowchart TB + P["Parent"] + S1["Worker 1"] + S2["Worker 2"] + S3["Worker N"] + P --> S1 + P --> S2 + P --> S3 + S1 --> P + S2 --> P + S3 --> P +``` + Spawn N workers in parallel, collect when all reach `idle`. Because `statewait` blocks until state changes, no polling loop is needed. @@ -124,7 +158,7 @@ This enables event-driven pipelines without polling: ```sh # Block until agent finishes, then trigger downstream work -cat /s/worker-123/statewait +cat /session/worker-123/statewait # agent is now idle — proceed ``` @@ -143,4 +177,4 @@ environment — shell scripts, cron, systemd, containers, or another agent. This is intentional. The filesystem namespace is the API. Any tool that can read and write files can participate in a multi-agent workflow, which means the -full Unix toolchain is available as an orchestration layer. +full Unix toolchain is available as an orchestration layer. \ No newline at end of file diff --git a/doc/REMOTE_EXECUTION.md b/doc/REMOTE_EXECUTION.md index 70b51e5..ca3a36c 100644 --- a/doc/REMOTE_EXECUTION.md +++ b/doc/REMOTE_EXECUTION.md @@ -15,37 +15,32 @@ The split-brain model keeps the **brain local** (agent loop, LLM, D-Bus, GUI) an ## Architecture -``` -Local machine Remote host (via SSH) -┌──────────────────────────────┐ ┌──────────────────────────────┐ -│ olliesrv / ollied │ │ ollie-remote │ -│ │ │ │ -│ ┌────────────────────────┐ │ SSH │ ┌────────────────────────┐ │ -│ │ agent.Core │ │────────▶│ │ execute.Server │ │ -│ │ - LLM streaming │ │ JSON │ │ - landrun sandbox │ │ -│ │ - prompt assembly │ │ RPC │ │ - bash/python/perl │ │ -│ │ - event bus │ │◀────────│ │ - tool scripts │ │ -│ │ - compaction │ │ │ │ - file I/O │ │ -│ └────────────────────────┘ │ │ └────────────────────────┘ │ -│ │ │ │ -│ ┌────────────────────────┐ │ │ Working directory: │ -│ │ D-Bus adapter │ │ │ /home/user/project/ │ -│ │ - SessionCreated │ │ │ │ -│ │ - StateChanged │ │ │ Sandbox config: │ -│ │ - ChatUpdated │ │ │ remote host's landrun │ -│ └────────────────────────┘ │ │ │ -│ ▲ │ └──────────────────────────────┘ -│ │ D-Bus signals │ -│ ▼ │ -│ ┌────────────────────────┐ │ -│ │ KDE GUI / Web UI │ │ -│ └────────────────────────┘ │ -└──────────────────────────────┘ +```mermaid +flowchart TB + subgraph Local["Local Machine"] + SRV["olliesrv"] + AG["agent.Agent\n- LLM streaming\n- prompt assembly\n- event bus\n- compaction"] + DBUS["D-Bus adapter\n- SessionCreated\n- StateChanged\n- ChatUpdated"] + GUI["KDE GUI / Web UI"] + end + + subgraph Remote["Remote Host (via SSH)"] + REMOTE_BIN["ollie-remote"] + EXEC["toolsrv.Server\n- landrun sandbox\n- shell execution\n- tool scripts\n- file I/O"] + CWD["Working directory:\n/home/user/project/"] + SANDCFG["Sandbox config:\nremote host's landrun"] + end + + AG -- "SSH / JSON-RPC" --> REMOTE_BIN + REMOTE_BIN --> EXEC + EXEC --> CWD + EXEC --> SANDCFG + DBUS -.->|"D-Bus signals"| GUI ``` ## Split Point -The natural split is at `tools.Server`. Today, `agent.Core` calls `execute.Server` in-process. The remote mode replaces this with a `RemoteServer` that forwards calls over an SSH-backed RPC channel: +The natural split is at `toolsrv.Runner`. Today, `agent.Agent` calls `toolsrv.Server` in-process. The remote mode replaces this with a `RemoteServer` that forwards calls over an SSH-backed RPC channel: ```go // Local side @@ -53,12 +48,12 @@ type RemoteServer struct { conn *rpc.Conn // JSON-RPC over SSH stdin/stdout } -func (r *RemoteServer) Call(ctx context.Context, req tools.Request) (tools.Response, error) { - return r.conn.Call(ctx, "execute", req) +func (r *RemoteServer) CallTool(ctx context.Context, tool string, args json.RawMessage) (json.RawMessage, error) { + return r.conn.Call(ctx, "execute", tool, args) } ``` -The agent loop doesn't know or care whether execution is local or remote. The `NewDispatcherFunc` in session creation just returns a `RemoteServer` instead of a local `execute.Server` when the session is configured for remote execution. +The agent loop doesn't know or care whether execution is local or remote. The `NewToolServer` factory in session creation just returns a `RemoteServer` instead of a local `toolsrv.Server` when the session is configured for remote execution. ## SSH Bootstrap @@ -90,7 +85,7 @@ exec "$BIN" serve --cwd "@@CWD@@" A minimal binary containing: -- `execute.Server` (the same one used locally) +- `toolsrv.Server` (the same one used locally) - Sandbox enforcement (landrun) - Tool script discovery from a bundled or remote-local `OLLIE_TOOLS_PATH` - JSON-RPC server over stdin/stdout @@ -106,7 +101,7 @@ It does NOT contain: // cmd/ollie-remote/main.go func main() { cwd := flag.Arg(0) - srv := execute.NewServer(cwd, toolsPath, sandboxConfig) + srv := toolsrv.NewServer(cwd, toolsPath, sandboxConfig) rpc.ServeStdio(srv) } ``` @@ -143,7 +138,7 @@ A session targeting a remote host: ```bash # Via 9P -printf 'cwd=/home/user/project\nremote=devbox\n' > $OLLIE/s/new +printf 'cwd=/home/user/project\nremote=devbox\n' > $OLLIE/session/new # Via D-Bus dbus-send ... org.ollie.SessionManager.CreateSession \ @@ -159,7 +154,7 @@ remote=user@devbox.internal The `remote` field is an SSH target. When set, the session manager: 1. Bootstraps `ollie-remote` on the target 2. Creates a `RemoteServer` connected over the SSH channel -3. Passes it to `agent.Core` as the tool dispatcher +3. Passes it to `agent.Agent` as the tool runner ## Tool Scripts @@ -214,19 +209,19 @@ They complement each other. You might use split-brain for active development (lo All five phases are implemented: ### Phase 1: ollie-remote binary ✓ -- `remote/` submodule with `main.go` -- Wraps `execute.Server` with JSON-RPC stdio interface +- `cmd/ollie-remote/` with `main.go` +- Wraps `toolsrv.Server` with JSON-RPC stdio interface - Embeds `sandbox-remote.yaml` and tool scripts via `go:embed` - RPC methods: `shell`, `list_tools`, `host_info`, `ping` ### Phase 2: SSH bootstrap ✓ -- `core/execute/bootstrap.sh` embedded template +- `toolsrv/bootstrap.sh` embedded template - SHA-256 hash-based caching on the remote - Binary transfer: gzip + base64 over stdin - `Dial()` handles full lifecycle: bootstrap → transfer → exec → JSON-RPC ### Phase 3: RemoteServer adapter ✓ -- `core/execute/remote.go` implements `tools.Server` +- `toolsrv/remote.go` implements `toolsrv.Runner` - `ListTools()` calls remote `list_tools` RPC - `CallTool()` passes tool name as RPC method (routes shell) - `HostInfo` fetched after connect (platform, arch, is_git_repo) @@ -368,14 +363,14 @@ Remote → local streaming uses JSON-RPC notifications (no `id` field): {"jsonrpc":"2.0","result":{"data":"compiling...\n"}} ``` -`remote.Server.CallTool` reads these in a loop, calling `tools.StreamOutput(ctx, data)` for each, until the actual response (with matching `id`) arrives. +`remote.Server.CallTool` reads these in a loop, calling `toolsrv.StreamOutput(ctx, data)` for each, until the actual response (with matching `id`) arrives. ### Agent Loop Integration The agent loop wraps the execution context with a streaming callback: ```go -streamCtx := tools.WithOutputStream(ctx, func(data string) { +streamCtx := toolsrv.WithOutputStream(ctx, func(data string) { emit(cfg, Event{Role: "tool", Name: tc.Name, Content: data}) }) ``` @@ -384,7 +379,7 @@ On first chunk, an empty-content tool event is emitted to signal stream start. I ### Execute Server -`limitedWriter` has a `stream func(string)` field, wired to `tools.StreamFunc(ctx)`. Every `Write()` call flushes the chunk both to the output buffer (for the full result) and to the stream callback (for real-time display). +`limitedWriter` has a `stream func(string)` field, wired to `toolsrv.StreamFunc(ctx)`. Every `Write()` call flushes the chunk both to the output buffer (for the full result) and to the stream callback (for real-time display). ### Chat Log @@ -393,4 +388,4 @@ On first chunk, an empty-content tool event is emitted to signal stream start. I - Subsequent tool events → content appended directly (no repeated header) - Non-tool event → closes the stream with a newline -For non-streamed tools (fast results), the single tool event renders as before. +For non-streamed tools (fast results), the single tool event renders as before. \ No newline at end of file