# Architectural Evolution How the Ollie system-of-systems emerged and evolved. ~1300 commits over ~3.5 months (Apr 11 – Jul 30, 2026). ## Timeline ```mermaid gantt title Ollie Evolution dateFormat YYYY-MM-DD axisFormat %b %d section Foundation Monorepo + submodules :done, 2026-04-11, 14d 9P filesystem :done, 2026-04-14, 28d section Core Primitives Session lifecycle (session/) :done, 2026-04-20, 30d Tool scripts :done, 2026-05-01, 20d Skills system :done, 2026-05-10, 14d section Execution execute_code / shell :done, 2026-05-08, 20d Batch jobs (b/ → session/bfg) :done, 2026-05-15, 25d Sandbox (landrun) :done, 2026-05-20, 40d section Multi-Agent Subagent spawn :done, 2026-05-25, 15d Cascade orchestrator :done, 2026-06-05, 20d JIT agent generation :done, 2026-07-10, 10d section Frontends TUI + Emacs (ellie) :done, 2026-04-15, 45d Acme (Plan 9) :done, 2026-05-20, 30d KDE/Plasma :done, 2026-06-10, 40d section Control Planes D-Bus adapter (ollied) :done, 2026-06-25, 20d D-Bus embedded in srv :done, 2026-07-15, 10d section Late Evolution Remote execution :done, 2026-07-05, 15d Elevation broker :done, 2026-07-10, 15d Tool registry (lazy) :done, 2026-07-20, 10d Reactions / feedback :done, 2026-07-22, 8d FUSE → 9P client :done, 2026-07-25, 5d The Great Flattening :done, 2026-07-29, 2d ``` ## Phase 1: Monorepo Bootstrap (Apr 11) Started as independent git repos unified under a monorepo with submodules. Initial components: - **agent** — agent loop (Go): backend dispatch, tool execution, session state - **fs/session** — 9P file server exposing agent sessions as a synthetic filesystem - **tui** — terminal UI (later removed) - **el** — Emacs integration (ellie.el) Build system was `mkfile` (Plan 9 make), later `Makefile`, finally `justfile`. ## Phase 2: The 9P Decision (Apr 14–28) The defining architectural choice: **every agent primitive is a file**. Sessions are directories. Sending a prompt is writing to `session/{id}/prompt`. Reading state is `cat session/{id}/state`. This made the control plane protocol-agnostic — any program that can read/write files can be a frontend. Key files crystallized: `prompt`, `chat`, `state`, `ctl`, `cfg`, `statewait`. The `session/new` file (write key=value pairs, read back session ID) became the session factory. ## Phase 3: Tool System (Apr 28 – May 10) Tools evolved through several incarnations: 1. **MCP servers** (denote-mcp, 9beads-mcp) — external processes, heavyweight 2. **Shell/Python scripts** in `~/.config/ollie/tools/` — lightweight, sandboxed 3. **execute_code** as the single built-in tool — all scripts invoked through it 4. **call_tool/pipe** — named tool dispatch and cross-tool pipelines 5. **Tool registry** (final form) — dynamic lazy-loading via `tool_list`/`tool_load`/`tool_active`; `execute_code` renamed to `shell`; tools hot-reloadable without server restart The file tools (`file_read`, `file_write`, `file_edit`, `file_grep`, `file_glob`) were extracted into standalone Python scripts. Memory (`memory_remember`, `memory_recall`) and reasoning (`reasoning_think`) followed the same pattern. MCP servers were removed early. Simple scripts won over complex daemons. ## Phase 4: Planning System Churn (May – Jul) Planning went through the most iterations of any subsystem: 1. `pl/` directory namespace with flat file-per-task 2. `plan_create` / `plan_complete` tools 3. **Beads** integration (external issue tracker) — made optional, then dropped 4. `task_add` / `task_check` tools 5. Inline markdown plans forbidden 6. **Final form**: a single `session/{id}/plan` file (markdown checklist), written by the agent, persisted across context compaction The lesson: planning needed to be simple, agent-controlled, and not bureaucratic. ## Phase 5: Multi-Agent Coordination (May 25 – Jun 15) Three patterns emerged: - **subagent_spawn** — fire-and-forget session creation, parent never blocks - **subagent_generate** — JIT agent config generation (role, constraints, tools) - **cascade** — script-driven fan-out with `-max-workers`, `-retries`, optional synthesis step ```mermaid flowchart TB UP["User prompt"] PARENT["Parent Session"] W1["Worker 1"] W2["Worker 2"] W3["Worker 3"] CASCADE["u/cascade\n(spawn + throttle)"] CW1["Cascade Worker"] CW2["Cascade Worker"] UP --> PARENT PARENT -- subagent_spawn --> W1 PARENT -- subagent_spawn --> W2 PARENT -- subagent_spawn --> W3 W1 --> PARENT W2 --> PARENT W3 --> PARENT PARENT -- cascade --> CASCADE CASCADE --> CW1 CASCADE --> CW2 ``` Inter-agent communication uses the filesystem: agents write to each other's `prompt` file. ## Phase 6: Frontend Proliferation ```mermaid flowchart TB subgraph Core["agent.Agent"] AG["Agent Engine"] end subgraph Surfaces["Integration Surfaces"] P9["9P Filesystem\n(session/ namespace)"] DB["D-Bus\n(org.ollie.SessionManager)"] end subgraph Frontends SH["s/sh (terminal)"] ACME["acme (Plan 9)"] EL["ellie (Emacs)"] KG["KDE GUI"] KP["KDE Plasmoid"] KK["Kate Plugin"] KR["KRunner"] HTTP["curl / scripts"] end AG --> P9 AG --> DB SH --> P9 ACME --> P9 EL --> P9 HTTP --> P9 KG --> DB KP --> DB KK --> DB KR --> DB WEB --> DB ``` - **s/sh** — shell script frontend (bash, briefly rc, back to bash) - **acme** — Plan 9 editor integration with `Oi` (quick query), `Kmpl` (completion) - **ellie.el** — Emacs with ghost-text completion - **ollie-kde** — full Plasma integration: plasmoid, KRunner, Kate plugin, standalone GUI, GUI automation tools - **TUI** — removed (Jul) in favor of s/sh and richer GUIs ## Phase 7: Security Model (May – Jul) ```mermaid flowchart LR subgraph Permissions["9P Permissions"] OWNER["Owner → rw"] AGENT["Agent → restricted"] PEERS["Peers → read-only"] end subgraph Execution["Execution Sandbox"] AGT["Agent"] SANDBOX["landrun sandbox\n(restricted fs)"] TOOLS["Tool scripts"] ELEV["x/elevate\n(socket broker)"] PRIV["Privileged Action"] end AGT --> SANDBOX --> TOOLS TOOLS -- needs escape --> ELEV --> PRIV ``` Evolution: 1. No sandboxing initially 2. YAML-based sandbox configs (landrun) for execute_code 3. Per-session 9P identity — agents can't read other sessions' tools 4. Elevation broker: socket-based, user-confirmed privilege escalation 5. SSH agent proxy added then removed (too much attack surface) 6. Final: integrated elevation broker replaces separate superpowerd adapter ## Phase 8: D-Bus as Second Control Plane (Jun 25 – Jul) Originally everything was 9P-only. D-Bus was added for desktop integration: 1. Separate `ollie-dbus` daemon (ollied) 2. Embedded directly into `olliesrv` (9P server got `-no9p` flag) 3. KDE uses D-Bus exclusively; acme/sh/el use 9P 4. **9P and D-Bus do not share sessions** — independent stores, same core ## Phase 9: Remote Execution (Jul 5–19) Split-brain architecture: orchestration stays local, code execution on a remote machine via SSH. - `ollie-remote` binary auto-deployed via SSH bootstrap - Tools embedded in remote binary - Streaming output notifications back to local session - Session persistence across remote restarts ## Phase 10: Tool Registry (Jul 20–29) The final major architectural shift — from "agent knows all tools upfront" to lazy discovery: 1. Tools embed their own `ollie:prompt` metadata blocks 2. `tool_list` — discover available tools and descriptions 3. `tool_load` — promote a tool to a native callable 4. `tool_active` — introspect loaded tools 5. Skills get the same treatment: `skill_list`, `skill_load`, `skill_active` This solved system prompt bloat — only load what's needed per task. ## Phase 11: The Great Flattening (Jul 29–30) Largest single-day structural change: **−4,573 lines net** across the codebase. Eliminated accumulated abstractions that no longer served a purpose. ### Core (−4,400 lines) - **Killed `agent.Core` interface** — concrete `*agent.Agent` used directly. Nobody else implemented Core; the interface just added indirection. - **Killed `Dispatcher` indirection** in toolsrv — `toolsrv.Server` called directly. - **Merged `execute/` into `tools/`** — separate package for 3 functions was pointless overhead. - **Renamed for clarity**: `agent.Session` → `agent.History`, `session.Core` → `session.Session`, `NewAgentCore` → `New`. - **Extracted `agent/` package** from the monolithic session package. - **Threaded context.Context** properly through daemon → session → agent (replaced ad-hoc interrupt mechanisms with cancellation). - Dead code removal: `GlobalToolNames`, `ExtractReturnSchema`. ### 9P server (−214 lines) - **Killed `mgr/` package entirely** — replaced `Manager` struct with package functions in `fs/session/`. The `*fs.Tree` IS the session collection; `rootState` (unexported) lives in `tree.Data`. CRUD via `NewRoot`, `Lookup`, `Create`, `Kill`, `Rename`, `Shutdown`. - **Fixed infinite recursion** in `session/{sid}/agent/` directory — `pathType()` didn't recognize agent subdirectories, causing walk to succeed at any depth. - **Fixed phantom session descent** — walk into non-existent sessions now fails immediately. - **Fixed `/agents` empty read** — `makeStat` wasn't computing Length for root files, FUSE kernel saw 0 bytes and never issued a read. - **Fixed `session.All()`** — was looking at `root.Children()` (empty) instead of `rootState.sessions`. Broke D-Bus ListSessions and GUI session restore. ### Frontends - **acme**: paths updated `s/` → `session/`, agent files route through `session/{id}/agent/{aid}/`. - **ellie.el**: same path migration, added `ellie--agent-dir` for agent ID discovery. - **KDE GUI**: no changes needed (D-Bus operates on Go objects, not paths). ### New filesystem layout ``` session/ ├── new write key=value to create session ├── idx session index (tab-separated) ├── {sid}/ │ ├── plan session-scoped markdown checklist │ ├── env session environment │ └── agent/ │ └── {aid}/ │ ├── cfg session configuration (key=value) │ ├── ctl control commands │ ├── state current state │ ├── statewait blocks until state changes │ ├── chat conversation log │ ├── prompt submit prompts │ ├── prompt.prev last submitted prompt │ ├── offset byte offset after last user prompt │ ├── fifo.in queue a prompt (FIFO input) │ ├── fifo.out pop queued prompt (FIFO output) │ ├── context rendered context window │ ├── systemprompt rendered system prompt │ ├── usage token stats │ ├── cost cost estimate │ ├── ctxsz context size │ ├── models available models │ ├── tools tool management │ ├── toolsrv.loaded currently loaded tools │ ├── toolsrv.rev revision counter │ ├── tail exec helper │ └── proc/ detached process output ``` ### Architecture after (single Go module, no submodules for core/9p) ``` *fs.Tree (root) ← IS the session collection └── .Data = *rootState ← sessions map, config, nextUID └── sessions[id] = *Session ← owns Agent, context, cancel Package functions (no Manager): session.NewRoot(cfg) → *fs.Tree session.Lookup(tree, id) → *Session session.All(tree) → []*Session session.CreateFromRoot(tree, args) → (id, error) session.KillFromRoot(tree, id) session.RenameFromRoot(tree, old, new) session.Shutdown(tree) ``` ### SLOC after flattening | Component | Lines | |-----------|-------| | agent + backend + toolsrv + session | 10,437 | | fs/session + cmd/olliesrv | 7,925 | | kde gui | 12,331 | | **Total** | **32,538** | Zero dead exported functions remain (verified via LSP + grep across all repos). ## Phase 12: Second Flattening & .meta Sidecar (Jul 30) Another aggressive structural pass. Eliminated the entire script namespace layer (s/, u/, x/), flattened `contrib/` into `data/`, and replaced header-comment-based tool metadata with JSON sidecar files. ### Scripts removed (−1,100 lines) - **s/{sh,b,bfg,bbg,ls,kill,cleanup}** — shell-based session frontends replaced by `ollie-9p rdwr generate/complete/route` and GUI frontends - **u/{optimize,complete,cascade,escalate}** — utility scripts replaced by `generate`, `complete`, `route` files in the 9P namespace - **x/{bd,prime,freeloader,task}** — internal plumbing scripts; PATH prepend (`prependOlliePath`) removed from toolsrv - 9P session root trimmed to just `new` and `idx`; script-serving code removed - `install-scripts` target now empty ### Tool metadata: .meta sidecar files Header comment parsing (`ollie:prompt`, `ollie:tier`, `ollie:parallel read`, `args_json:`) **eliminated entirely**. All tool metadata now lives in a JSON sidecar file alongside the executable: ``` tools/ file_edit ← executable (any language) file_edit.meta ← JSON metadata ``` ```json { "description": "Replace text in a file.", "prompt": "## file_edit\n\n...", "args": {"type":"object", ...}, "tier": "cold", "readOnly": true } ``` This decouples metadata from implementation language — compiled Go binaries, Python scripts, and bash tools all use the same discovery mechanism. ### LSP tools ported to Go The Python LSP bridge (`_lib/lsp/`) replaced with a pure Go implementation: ``` tools/ ├── builtin/ ← in-process handlers (shell, reasoning, tool/skill registry) └── lsp/ ← shared LSP client library └── cmd/ ← individual binaries (lsp_definition, lsp_hover, ...) ``` Architecture: each LSP tool binary embeds a bridge daemon (started on first invocation via `--bridge` flag, persists via Unix socket, auto-exits after 5min idle). No Python. No `_lib`. No external dependencies beyond the LSP servers themselves (gopls, clangd, intelephense). ### Repo reorganization - `contrib/{prompts,scripts,services,tools,skills,agents}` → `data/` - `contrib/elisp` stays (Emacs frontend contribution) - `tools/builtin/` — built-in tool handlers (moved from `tools/`) - `tools/lsp/` — Go LSP implementation (replaced `data/tools/_lib/lsp/`) ## Current Topology ``` ollie/ ← single Go module ├── agent/ ← agent loop, history, hooks, commands ├── backend/ ← LLM providers (6 backends) ├── toolsrv/ ← tool server, registry, remote execution ├── session/ ← session lifecycle, config ├── fs/session/ ← 9P filesystem tree ├── dbus/ ← D-Bus adapter ├── detach/ ← background process management ├── elevate/ ← elevation broker ├── sandbox/ ← landrun sandbox config ├── env/ ← environment helpers ├── log/ ← structured logging ├── paths/ ← XDG path resolution ├── mount/ ← 9P FUSE mount client (network transparency) ├── cmd/ ← binaries (olliesrv, ollie-9p, ollie-remote) ├── kde/ ← KDE Plasma (submodule) ├── data/tools/ ← tool executables + .meta sidecar files ├── data/agents/ ← agent configs (JSON) ├── data/prompts/ ← system prompt fragments (markdown) ├── data/skills/ ← domain knowledge modules (markdown) ├── data/scripts/ ← ollie-remount, ollie-watchdog ├── data/services/ ← systemd, xdg-autostart ├── prompts/ ← embedded prompt templates ├── sandbox/ ← landrun sandbox profiles (YAML) └── doc/ ← documentation ``` ## Principles That Emerged 1. **Filesystem-as-API** — everything is read/write on synthetic files. No custom protocols needed. Monorepo coordinates versions. 3. **Scripts over servers** — MCP servers removed early. Simple scripts won. 4. **Progressive disclosure** — lazy tool/skill loading keeps system prompts small until complexity is needed. 5. **Plan simplicity** — after 5 iterations, planning settled on one markdown checklist file per session. 6. **Dual control plane** — 9P for power users/editors, D-Bus for desktop. Same core, different surfaces. 7. **Security by default** — sandboxed execution with explicit elevation, per-session identity. ## Dead Ends and Reversals | What | Why it was removed | |------|-------------------| | MCP servers | Too heavyweight; scripts are simpler and faster | | TUI frontend | Replaced by KDE GUI and Emacs (ellie) | | s/sh, s/bfg, s/bbg | Replaced by `generate`/`complete`/`route` 9P files | | `pl/` planning namespace | Over-engineered; plan file is sufficient | | Beads integration | External dependency for something a checklist file handles | | SSH agent proxy | Attack surface not worth the convenience | | Separate ollie-dbus daemon | Consolidated into olliesrv | | FUSE mounts | Replaced by standalone 9P client binary | | Tier routing (FAST/POWER) | Replaced by /route endpoint with real model discovery | | call_tool / pipe | Replaced by native tool registry with tool_load | | execute_code (multi-lang) | Simplified to shell-only; tool scripts handle language choice | | chatwait | Reverted; acme tails chat directly | | Header comment metadata | Replaced by .meta sidecar JSON; decouples metadata from language | | `contrib/` directory | Renamed to `data/`; nothing was community-contributed | | Script namespaces (s/, u/, x/) | Replaced by 9P request-response files |