From c5b6e5b6941a33770d1dfc6fcb1ad2c3d78112c6 Mon Sep 17 00:00:00 2001 From: Levi Neely Date: Wed, 29 Jul 2026 15:40:21 +0200 Subject: [PATCH] =?UTF-8?q?doc:=20add=20EVOLUTION.md=20=E2=80=94=20archite?= =?UTF-8?q?ctural=20history=20of=20the=20system?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- doc/EVOLUTION.md | 282 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 282 insertions(+) create mode 100644 doc/EVOLUTION.md diff --git a/doc/EVOLUTION.md b/doc/EVOLUTION.md new file mode 100644 index 0000000..b0ddd70 --- /dev/null +++ b/doc/EVOLUTION.md @@ -0,0 +1,282 @@ +# Architectural Evolution + +How the Ollie system-of-systems emerged and evolved. +1261 commits over ~3.5 months (Apr 11 – Jul 29, 2026). + +## Timeline + +```mermaid +gantt + title Ollie Architecture Evolution + dateFormat YYYY-MM-DD + axisFormat %b %d + + section Foundation + Monorepo + submodules :2026-04-11, 3d + 9P filesystem interface :2026-04-14, 14d + + section Core Primitives + Session lifecycle (s/) :2026-04-20, 21d + Tool scripts (file/memory/reasoning) :2026-04-28, 14d + Skills system :2026-05-01, 10d + + section Execution + execute_code (multi-lang) :2026-05-10, 14d + Batch jobs (b/ → s/bfg,s/bbg) :2026-05-15, 21d + Sandbox (landrun) :2026-05-20, 60d + + section Multi-Agent + Subagent spawn :2026-05-25, 14d + Cascade orchestrator :2026-06-01, 14d + JIT agent generation :2026-06-15, 7d + + section Frontends + TUI + Emacs (ellie) :2026-04-11, 60d + Acme (Plan 9) :2026-06-01, 45d + Web UI + httpgw :2026-05-28, 30d + KDE/Plasma (plasmoid + GUI) :2026-06-20, 40d + + section Control Planes + D-Bus adapter (ollied) :2026-06-25, 21d + D-Bus embedded in olliesrv :2026-07-10, 7d + + section Late Evolution + Remote execution (split-brain) :2026-07-05, 14d + Elevation broker :2026-07-10, 14d + Tool registry (lazy-load) :2026-07-20, 9d + Reactions / feedback :2026-07-18, 7d + FUSE → 9P client migration :2026-07-20, 5d +``` + +## Phase 1: Monorepo Bootstrap (Apr 11) + +Started as independent git repos unified under a monorepo with submodules. +Initial components: + +- **core** — agent loop (Go): backend dispatch, tool execution, session state +- **9p** — 9P file server exposing agent sessions as a synthetic filesystem +- **tui** — terminal UI (later removed) +- **el** — Emacs integration (ellie.el) + +Build system was `mkfile` (Plan 9 make), later `Makefile`, finally `justfile`. + +## Phase 2: The 9P Decision (Apr 14–28) + +The defining architectural choice: **every agent primitive is a file**. + +Sessions are directories. Sending a prompt is writing to `s/{id}/prompt`. +Reading state is `cat s/{id}/state`. This made the control plane +protocol-agnostic — any program that can read/write files can be a frontend. + +Key files crystallized: `prompt`, `chat`, `state`, `ctl`, `cfg`, `statewait`. + +The `s/new` file (write key=value pairs, read back session ID) became the +session factory. + +## Phase 3: Tool System (Apr 28 – May 10) + +Tools evolved through several incarnations: + +1. **MCP servers** (denote-mcp, 9beads-mcp) — external processes, heavyweight +2. **Shell/Python scripts** in `~/.config/ollie/tools/` — lightweight, sandboxed +3. **execute_code** as the single built-in tool — all scripts invoked through it +4. **call_tool/pipe** — named tool dispatch and cross-tool pipelines +5. **Tool registry** (final form) — lazy-loading via `tool_list`/`tool_load`/`tool_active`; `execute_code` renamed to `shell` + +The file tools (`file_read`, `file_write`, `file_edit`, `file_grep`, `file_glob`) +were extracted into standalone Python scripts. Memory (`memory_remember`, +`memory_recall`) and reasoning (`reasoning_think`) followed the same pattern. + +MCP servers were removed early. Simple scripts won over complex daemons. + +## Phase 4: Planning System Churn (May – Jul) + +Planning went through the most iterations of any subsystem: + +1. `pl/` directory namespace with flat file-per-task +2. `plan_create` / `plan_complete` tools +3. **Beads** integration (external issue tracker) — made optional, then dropped +4. `task_add` / `task_check` tools +5. Inline markdown plans forbidden +6. **Final form**: a single `s/{id}/plan` file (markdown checklist), written by + the agent, persisted across context compaction + +The lesson: planning needed to be simple, agent-controlled, and not bureaucratic. + +## Phase 5: Multi-Agent Coordination (May 25 – Jun 15) + +Three patterns emerged: + +- **subagent_spawn** — fire-and-forget session creation, parent never blocks +- **subagent_generate** — JIT agent config generation (role, constraints, tools) +- **cascade** — script-driven fan-out with `-max-workers`, `-retries`, optional + synthesis step + +```mermaid +flowchart TD + User -->|prompt| Parent[Parent Session] + Parent -->|subagent_spawn| W1[Worker 1] + Parent -->|subagent_spawn| W2[Worker 2] + Parent -->|subagent_spawn| W3[Worker 3] + W1 -->|write to prompt| Parent + W2 -->|write to prompt| Parent + W3 -->|write to prompt| Parent + + Parent -->|cascade| Cascade[u/cascade script] + Cascade -->|spawn + throttle| CW1[Cascade Worker] + Cascade -->|spawn + throttle| CW2[Cascade Worker] + Cascade -->|synthesize| Parent +``` + +Inter-agent communication uses the filesystem: agents write to each other's +`prompt` file or use bidirectional `peer/` links. + +## Phase 6: Frontend Proliferation + +```mermaid +flowchart LR + subgraph Control Planes + 9P[9P File Server] + DBus[D-Bus Service] + end + + subgraph Frontends + SH[s/sh — shell] + Acme[ollie-acme] + El[ellie.el — Emacs] + KDE[ollie-kde — Plasma] + Web[Web UI] + end + + subgraph Server + Core[agent.Core] + end + + Core --- 9P + Core --- DBus + 9P --- SH + 9P --- Acme + 9P --- El + DBus --- KDE + DBus ---|httpgw| Web +``` + +- **s/sh** — shell script frontend (bash, briefly rc, back to bash) +- **acme** — Plan 9 editor integration with `Oi` (quick query), `Kmpl` (completion) +- **ellie.el** — Emacs with ghost-text completion +- **httpgw** — HTTP→D-Bus gateway +- **webui** — browser frontend via httpgw +- **ollie-kde** — full Plasma integration: plasmoid, KRunner, Kate plugin, standalone GUI, GUI automation tools +- **TUI** — removed (Jul) in favor of s/sh and richer GUIs + +## Phase 7: Security Model (May – Jul) + +```mermaid +flowchart TD + Agent -->|sandboxed| Landrun[landrun sandbox] + Landrun -->|restricted fs| Tools + Tools -->|needs escape| Elevate[x/elevate broker] + Elevate -->|socket| Broker[Integrated Broker] + Broker -->|user confirms| Action[Privileged Action] + + subgraph 9P Permissions + Owner[Session Owner — rw] + AgentID[Agent — restricted] + Peer[Peers — read-only] + end +``` + +Evolution: + +1. No sandboxing initially +2. YAML-based sandbox configs (landrun) for execute_code +3. Per-session 9P identity — agents can't read other sessions' tools +4. Elevation broker: socket-based, user-confirmed privilege escalation +5. SSH agent proxy added then removed (too much attack surface) +6. Final: integrated elevation broker replaces separate superpowerd adapter + +## Phase 8: D-Bus as Second Control Plane (Jun 25 – Jul) + +Originally everything was 9P-only. D-Bus was added for desktop integration: + +1. Separate `ollie-dbus` daemon (ollied) +2. Embedded directly into `olliesrv` (9P server got `-nodbus` flag) +3. KDE uses D-Bus exclusively; acme/sh/el use 9P +4. **9P and D-Bus do not share sessions** — independent stores, same core + +## Phase 9: Remote Execution (Jul 5–19) + +Split-brain architecture: orchestration stays local, code execution on a remote +machine via SSH. + +- `ollie-remote` binary auto-deployed via SSH bootstrap +- Tools embedded in remote binary +- Streaming output notifications back to local session +- Session persistence across remote restarts + +## Phase 10: Tool Registry (Jul 20–29) + +The final major architectural shift — from "agent knows all tools upfront" to +lazy discovery: + +1. Tools embed their own `ollie:prompt` metadata blocks +2. `tool_list` — discover available tools and descriptions +3. `tool_load` — promote a tool to a native callable +4. `tool_active` — introspect loaded tools +5. Skills get the same treatment: `skill_list`, `skill_load`, `skill_active` + +This solved system prompt bloat — only load what's needed per task. + +## Current Topology + +``` +ollie/ ← monorepo root +├── core/ ← agent loop, backends, tool dispatch (Go) +├── 9p/ ← 9P server + embedded D-Bus adapter (Go) +├── acme/ ← Plan 9 acme frontend (Go) +├── el/ ← Emacs integration (elisp) +├── kde/ ← KDE Plasma: plasmoid, GUI, Kate, KRunner (C++/QML) +├── httpgw/ ← HTTP→D-Bus gateway (Go) +├── webui/ ← browser frontend (JS) +├── remote/ ← split-brain execution server (Go) +├── tools/ ← 37 tool scripts (Python) +├── scripts/{s,u,x}/ ← filesystem-hosted scripts +├── agents/ ← agent configs (JSON) +├── prompts/ ← system prompt fragments (markdown) +├── skills/ ← domain knowledge modules (markdown) +├── sandbox/ ← landrun sandbox profiles (YAML) +├── services/ ← init/service files +└── experiments/ ← multi-agent trial docs +``` + +## Principles That Emerged + +1. **Filesystem-as-API** — everything is read/write on synthetic files. No + custom protocols needed. +2. **Submodule independence** — each component has its own repo, tests, build. + Monorepo coordinates versions. +3. **Scripts over servers** — MCP servers removed early. Simple scripts won. +4. **Progressive disclosure** — lazy tool/skill loading keeps system prompts + small until complexity is needed. +5. **Plan simplicity** — after 5 iterations, planning settled on one markdown + checklist file per session. +6. **Dual control plane** — 9P for power users/editors, D-Bus for desktop. + Same core, different surfaces. +7. **Security by default** — sandboxed execution with explicit elevation, + per-session identity. + +## Dead Ends and Reversals + +| What | Why it was removed | +|------|-------------------| +| MCP servers | Too heavyweight; scripts are simpler and faster | +| TUI frontend | Replaced by s/sh (lighter) and KDE GUI (richer) | +| `pl/` planning namespace | Over-engineered; plan file is sufficient | +| Beads integration | External dependency for something a checklist file handles | +| SSH agent proxy | Attack surface not worth the convenience | +| Separate ollie-dbus daemon | Consolidated into olliesrv | +| FUSE mounts | Replaced by standalone 9P client binary | +| Tier routing (FAST/POWER) | Replaced by /route endpoint with real model discovery | +| call_tool / pipe | Replaced by native tool registry with tool_load | +| execute_code (multi-lang) | Simplified to shell-only; tool scripts handle language choice | +| chatwait | Reverted; acme tails chat directly |