From 4b7d5cdd7902df4634b6903bc7fca94040f3f102 Mon Sep 17 00:00:00 2001 From: Ollie Agent Date: Tue, 11 Aug 2026 07:24:53 +0200 Subject: [PATCH] doc: update architecture, evolution, README for toolsrv 9P migration - README: toolsrv shown as separate child process in architecture diagram, cmd/toolsrv added to repository layout table - architecture.md: repository structure reflects cmd/{olliesrv,toolsrv} internal packages, package table updated for two-server model - evolution.md: add Phase 20 (toolsrv 9P migration & process isolation), update topology diagram, add to gantt chart --- README.md | 18 +++--- doc/architecture.md | 53 +++++++--------- doc/resources/evolution.md | 127 +++++++++++++++++++++++++++++-------- 3 files changed, 135 insertions(+), 63 deletions(-) diff --git a/README.md b/README.md index 01a909d..a98408d 100644 --- a/README.md +++ b/README.md @@ -70,7 +70,8 @@ Single Go module with one Git submodule (kde). |-----------|----------|-------------| | `agent/` | Go | Agent loop, history, prompt resolution, context compaction | | `backend/` | Go | LLM providers (Anthropic, OpenAI, Ollama, Gemini, Copilot, CodeWhisperer) | -| `toolsrv/` | Go | Tool server, dynamic tool dispatch, sandboxed execution, remote execution | +| `toolsrv/` | Go | Tool server client library (9P client for toolsrv) | +| `cmd/toolsrv/` | Go | Tool execution server (separate 9P server, sandboxed execution) | | `tools/lsp/` | Go | LSP bridge daemon + client library (gopls, clangd, intelephense) | | `session/` | Go | Session lifecycle, config, persistence | | `fs/` | Go | 9P filesystem tree (EDSL-declared namespace, ctl dispatch) | @@ -78,7 +79,8 @@ Single Go module with one Git submodule (kde). | `bypass/` | Go | Bypass broker (sandbox escape approval) | | `sandbox/` | Go | Landlock sandbox config | | `lib9p/` | Go | 9P protocol library + native C client | -| `cmd/olliesrv/` | Go | The main binary | +| `cmd/olliesrv/` | Go | The main server binary (session/agent 9P namespace) | +| `cmd/toolsrv/` | Go | Tool execution server (spawned by olliesrv, separate 9P server) | | `cmd/ollie-9p/` | Go | 9P client CLI | | `cmd/ollie-remote/` | Go | Remote execution binary | | `kde/` | C++/Qt6 | KDE plasmoid, GUI, Kate plugin, KRunner *(submodule)* | @@ -109,7 +111,11 @@ graph TB P9[9P Filesystem
session/ namespace] SESS[session.Session
tool server conn · persistence] AG[agent.Agent
loop · compaction
prompt assembly] - TOOLS[toolsrv.Server
dynamic dispatch · sandbox] + end + + subgraph "toolsrv (child process, 9P)" + TOOLS[9P Filesystem
ctl · tools · proc/] + EXEC[Tool Scripts
· shell, reasoning_think
· file_*, lsp_*, memory_*
· gui_*, subagent_*] end subgraph "LLM Backends" @@ -120,16 +126,12 @@ graph TB KIRO[Kiro] end - subgraph Execution - EXEC[Tool Scripts
loaded via ctl
· shell, reasoning_think
· file_*, lsp_*, memory_*
· gui_*, subagent_*] - end - TUI & SCRIPTS & ACME --> O O --> P9 KDE & KATE & EMACS --> P9 P9 --> SESS SESS --> AG - SESS -->|shared RPC conn| TOOLS + SESS -->|9P over Unix socket| TOOLS AG -->|streaming| OLL & OAI & ANT & COP & KIRO diff --git a/doc/architecture.md b/doc/architecture.md index 8745631..461003a 100644 --- a/doc/architecture.md +++ b/doc/architecture.md @@ -15,31 +15,19 @@ Design principles: Single Go module (`ollie`) with one Git submodule (`kde/`) for the KDE frontend: ``` ollie/ -├── agent/ Agent loop, history, prompt resolution, context compaction -├── backend/ LLM providers (Anthropic, OpenAI, Ollama, Gemini, Copilot, CodeWhisperer) -├── toolsrv/ Tool server: sandboxed execution, tool registry, skill management -├── session/ Session lifecycle (config, creation, persistence) -├── fs/ 9P filesystem: EDSL spec + handlers (flat package) -│ ├── spec.go Namespace declaration (single source of truth) -│ ├── handlers.go All file handlers (root, session, agent) -│ ├── ctl.go rdwrHandler type + rdwrDispatch (shared by all ctl files) -│ ├── agent_log.go AgentLog (chat streaming, chatCond) -│ ├── session_node.go SessionNode wrapper -│ ├── lifecycle.go RootState, session node cache -│ ├── newroot.go NewRoot — tree construction config -│ └── cache.go ModelCache +├── cmd/ +│ ├── olliesrv/ 9P server (sessions, agents, backends) +│ │ └── internal/ agent/, backend/, bypass/, fs/, session/, prompts/, toolclient/ +│ ├── toolsrv/ 9P tool execution server (separate process) +│ │ └── internal/ fs/, exec/, registry/, sandbox/ +│ ├── ollie-9p/ 9P client CLI +│ └── ollie-remote/ Remote execution server +├── toolsrv/ Client library (9P client for toolsrv) ├── fsedsl/ Filesystem declaration EDSL (generic, reusable) -├── bypass/ Bypass broker (sandbox escape approval) -├── sandbox/ Landlock sandbox config YAML -├── lib9p/ 9P protocol library + native C client ├── env/ Environment variable loading ├── log/ Structured logging ├── paths/ XDG path resolution ├── format/ Chat log formatting constants -├── cmd/ Binaries: -│ ├── olliesrv/ 9P server -│ ├── ollie-9p/ 9P client CLI -│ └── ollie-remote/ Remote execution server ├── kde/ KDE integration (submodule) — standalone GUI, Kate plugin, KRunner ├── data/agents/ Agent config JSONs (default, coding, orchestrator, worker, ...) ├── data/prompts/ Prompt templates (markdown) @@ -71,7 +59,9 @@ flowchart TB subgraph Core["Agent Engine (per session)"] LOOP["Agent Loop\n(agent/loop.go)"] SESS["Session State\n(agent/history.go)"] - TRV["toolsrv.Server\n· dynamic tool dispatch\n· landlock sandboxed execution\n· remote execution via SSH"] + end + subgraph ToolSrv["toolsrv (child process)"] + TRV["9P Server\n· dynamic tool dispatch\n· landlock sandboxed execution\n· idle timeout + Pdeathsig lifecycle"] end subgraph Backends["LLM Backends"] OLLAMA["Ollama"] @@ -98,15 +88,20 @@ The core is a single Go module (`ollie`) with no binary. Binaries live in `cmd/` ### Package Layout | Package | Purpose | |---|---| -| `agent/` | Agent struct, loop, history, compaction, hooks, commands, prompt resolution, state | -| `backend/` | Backend interface + LLM providers (Anthropic, OpenAI, Ollama, Gemini, Copilot, CodeWhisperer) | -| `toolsrv/` | Tool server: dynamic tool dispatch, sandboxed execution, result tiering, remote execution (ollie-remote) | -| `session/` | Session lifecycle, config, persistence | -| `detach/` | Background process management (ring buffer, signal) | -| `bypass/` | Bypass broker (privilege escalation daemon, policy) | -| `sandbox/` | Landrun sandbox configuration and command wrapping | +| `cmd/olliesrv/internal/agent/` | Agent struct, loop, history, compaction, hooks, commands, prompt resolution, state | +| `cmd/olliesrv/internal/backend/` | Backend interface + LLM providers (Anthropic, OpenAI, Ollama, Gemini, Copilot, CodeWhisperer) | +| `cmd/olliesrv/internal/fs/` | 9P filesystem: EDSL spec + handlers, ctl dispatch, session nodes | +| `cmd/olliesrv/internal/session/` | Session lifecycle, persistence, tool server spawning | +| `cmd/olliesrv/internal/bypass/` | Bypass broker (privilege escalation daemon, policy) | +| `cmd/olliesrv/internal/toolclient/` | Spawning and managing toolsrv child processes | +| `cmd/toolsrv/internal/fs/` | toolsrv filesystem spec, state, process management | +| `cmd/toolsrv/internal/exec/` | Sandboxed tool execution (landrun, bypass) | +| `cmd/toolsrv/internal/registry/` | Session-scoped tool registry | +| `cmd/toolsrv/internal/sandbox/` | Landrun sandbox configuration and command wrapping | +| `toolsrv/` | 9P client library for connecting to toolsrv | +| `fsedsl/` | Generic filesystem declaration EDSL (used by both servers) | | `env/` | Session environment helpers | -| `log/` | Structured logging | +| `log/` | Structured logging (olliesrv) | | `paths/` | XDG path resolution | ### Key Types **`agent.Agent`** — the concrete struct frontends drive: diff --git a/doc/resources/evolution.md b/doc/resources/evolution.md index ea27a64..a8cf4e2 100644 --- a/doc/resources/evolution.md +++ b/doc/resources/evolution.md @@ -39,6 +39,7 @@ gantt 9P Declarative EDSL :done, 2026-08-02, 1d Zero built-in tools :done, 2026-08-02, 2d EDSL extraction + cleanup :done, 2026-08-03, 2d + toolsrv 9P + process isolation :done, 2026-08-10, 2d ``` ## Phase 1: Monorepo Bootstrap (Apr 11) Started as independent git repos unified under a monorepo with submodules. @@ -621,34 +622,18 @@ The KDE frontend received extensive updates across both days: ## Updated Current Topology ``` ollie/ ← single Go module -├── agent/ ← agent loop, history, hooks, commands -├── backend/ ← LLM providers (6 backends) -├── toolsrv/ ← tool server, registry, remote execution -├── session/ ← session lifecycle, config -├── fs/ ← 9P filesystem (EDSL-declared, flat package) -│ ├── spec.go Namespace declaration (single source of truth) -│ ├── fsnode.go FsNodeDecl type + Dir/Leaf/TemplateDir -│ ├── builder.go BuildTree — spec → wired *Tree -│ ├── rootfiles.go Root-level handlers -│ ├── sessionfiles.go Session-level handlers -│ ├── agentfiles.go Agent-level handlers -│ ├── bypassfiles.go Bypass handlers -│ ├── procfiles.go Process handlers -│ ├── lifecycle.go Create/kill/rename + event ring -│ ├── newroot.go NewRoot constructor -│ ├── persist.go Session persistence -│ ├── types.go Session/AgentLog types -│ ├── tree.go 9P *Tree -│ ├── fs.go 9P File/FileConfig -│ └── format.go Event formatting -├── detach/ ← background process management -├── bypass/ ← bypass broker -├── sandbox/ ← landrun sandbox config +├── cmd/ +│ ├── olliesrv/ ← 9P server (sessions, agents, backends) +│ │ └── internal/ agent/, backend/, bypass/, fs/, session/, prompts/, toolclient/ +│ ├── toolsrv/ ← 9P tool execution server (separate process) +│ │ └── internal/ fs/, exec/, registry/, sandbox/ +│ ├── ollie-9p/ ← 9P client CLI +│ └── ollie-remote/ ← remote execution binary +├── toolsrv/ ← 9P client library for toolsrv +├── fsedsl/ ← filesystem declaration EDSL (used by both servers) ├── env/ ← environment helpers ├── log/ ← structured logging ├── paths/ ← XDG path resolution -├── mount/ ← 9P FUSE mount client (network transparency) -├── cmd/ ← binaries (olliesrv, ollie-9p, ollie-remote) ├── kde/ ← KDE Plasma (submodule) ├── data/tools/ ← tool executables + .meta sidecar files ├── data/agents/ ← agent configs (JSON) @@ -657,7 +642,6 @@ ollie/ ← single Go module ├── data/scripts/ ← ollie-remount, o CLI ├── data/services/ ← systemd, xdg-autostart ├── prompts/ ← embedded prompt templates -├── sandbox/ ← landrun sandbox profiles (YAML) └── doc/ ← documentation ``` @@ -1085,3 +1069,94 @@ Systematic audit of the prompt construction system and codebase maintainability. | **Total (core)** | **12,793** | Excludes: backends (4,229), fsedsl (1,030), tests, KDE, tools, generated code. + +## Phase 20: toolsrv 9P Migration & Process Isolation (Aug 10–11) + +The tool server completed its evolution from an in-process library to a +**fully independent 9P server** running as a child process of olliesrv. + +### Before + +`toolsrv` was a Go package (`ollie/toolsrv`) with a `Server` struct that ran +in-process within olliesrv. Tool calls were Go function calls — no process +boundary, no separate namespace. The package mixed client code, server code, +registry, sandbox wrappers, and execution logic in a flat directory. + +### After + +Two distinct components: + +1. **`cmd/toolsrv/`** — a standalone binary that serves its own 9P2000 + filesystem over a Unix socket. Has its own `internal/` packages: + - `internal/fs/` — filesystem spec (fsedsl), server state, process management + - `internal/exec/` — sandboxed tool execution (landrun, bypass broker) + - `internal/registry/` — session-scoped tool registry + - `internal/sandbox/` — landrun configuration (unchanged) + +2. **`toolsrv/`** (root package) — a 9P client library. `Conn` dials toolsrv + over a Unix socket, authenticates via Tauth, and exposes methods like + `CallTool`, `LoadTool`, `ListTools`, `Ping`. + +### toolsrv 9P Namespace + +``` +/ +├── ctl write: load , unload , env K=V, cwd +├── tools read: list loaded tools (JSON), write: tool name to load +├── info read: platform, arch +└── proc/ + ├── new rdwr: write tool+args, blocks, read result + ├── new.bg write: tool+args, returns pid immediately + └── {pid}/ + ├── out read: output + ├── wait read: blocks until exit, returns exit code + ├── stat read: running/exited, runtime, tool + └── ctl write: signal , dismiss +``` + +Both `cmd/toolsrv` and `cmd/olliesrv` declare their namespaces using the +same `fsedsl` library. Both implement their own 9P protocol handlers (using +`9fans.net/go/plan9`) — they are intentionally separate servers that +communicate over a socket. + +### Process Lifecycle + +Three-layer defense against orphaned toolsrv processes: + +1. **Pdeathsig** — `SysProcAttr{Pdeathsig: SIGTERM}` on local spawn. When + olliesrv dies, the kernel terminates toolsrv immediately. + +2. **Idle timeout** — toolsrv tracks active 9P connections. After the first + client connects and then all connections drop, a 30s timer starts. + If no new connection arrives, toolsrv exits cleanly. 60s startup grace + period for the initial connection. + +3. **Kill-before-respawn** — `ProcessKeeper.Dial()` kills the old process + before spawning a replacement. Prevents accumulation during reconnect cycles. + +### Authentication + +toolsrv uses 9P Tauth for authentication. The first client sets the shared +secret; subsequent clients must provide the same secret. This replaces +the previous token-in-environment approach and is compatible with socket +permission security. + +### Structural Consistency + +Both servers now follow the same physical layout: + +``` +cmd/{server}/ +├── main.go entry point +├── server.go 9P protocol handler (top-level, like Plan 9 tradition) +└── internal/ + ├── fs/ filesystem spec + handlers + state + ├── ... domain-specific packages +``` + +### Logging + +toolsrv uses `log/slog` (Go stdlib) with a `"svc": "toolsrv"` attribute. +This is intentionally separate from olliesrv's `ollie/log` package — they +are different programs with different lifecycles. +