ollie/doc/EVOLUTION.md

Architectural Evolution

How the Ollie system-of-systems emerged and evolved. ~1300 commits over ~3.5 months (Apr 11 – Jul 30, 2026).

Timeline

gantt
    title Ollie Evolution
    dateFormat  YYYY-MM-DD
    axisFormat  %b %d
    section Foundation
    Monorepo + submodules          :done, 2026-04-11, 14d
    9P filesystem                  :done, 2026-04-14, 28d
    section Core Primitives
    Session lifecycle (session/)   :done, 2026-04-20, 30d
    Tool scripts                   :done, 2026-05-01, 20d
    Skills system                  :done, 2026-05-10, 14d
    section Execution
    execute_code / shell           :done, 2026-05-08, 20d
    Batch jobs (b/ → session/bfg)  :done, 2026-05-15, 25d
    Sandbox (landrun)              :done, 2026-05-20, 40d
    section Multi-Agent
    Subagent spawn                 :done, 2026-05-25, 15d
    Cascade orchestrator           :done, 2026-06-05, 20d
    JIT agent generation           :done, 2026-07-10, 10d
    section Frontends
    TUI + Emacs (ellie)            :done, 2026-04-15, 45d
    Acme (Plan 9)                  :done, 2026-05-20, 30d
    KDE/Plasma                     :done, 2026-06-10, 40d
    section Control Planes
    D-Bus adapter (ollied)         :done, 2026-06-25, 20d
    D-Bus embedded in srv          :done, 2026-07-15, 10d
    section Late Evolution
    Remote execution               :done, 2026-07-05, 15d
    Elevation broker               :done, 2026-07-10, 15d
    Tool registry (lazy)           :done, 2026-07-20, 10d
    Reactions / feedback           :done, 2026-07-22, 8d
    FUSE → 9P client               :done, 2026-07-25, 5d
    The Great Flattening           :done, 2026-07-29, 2d

Phase 1: Monorepo Bootstrap (Apr 11)

Started as independent git repos unified under a monorepo with submodules. Initial components:

• agent — agent loop (Go): backend dispatch, tool execution, session state
• fs/session — 9P file server exposing agent sessions as a synthetic filesystem
• tui — terminal UI (later removed)
• el — Emacs integration (ellie.el) Build system was mkfile (Plan 9 make), later Makefile, finally justfile.

Phase 2: The 9P Decision (Apr 14–28)

The defining architectural choice: every agent primitive is a file. Sessions are directories. Sending a prompt is writing to session/{id}/prompt. Reading state is cat session/{id}/state. This made the control plane protocol-agnostic — any program that can read/write files can be a frontend. Key files crystallized: prompt, chat, state, ctl, cfg, statewait. The session/new file (write key=value pairs, read back session ID) became the session factory.

Phase 3: Tool System (Apr 28 – May 10)

Tools evolved through several incarnations:

1. MCP servers (denote-mcp, 9beads-mcp) — external processes, heavyweight
2. Shell/Python scripts in ~/.config/ollie/tools/ — lightweight, sandboxed
3. execute_code as the single built-in tool — all scripts invoked through it
4. call_tool/pipe — named tool dispatch and cross-tool pipelines
5. Tool registry (final form) — dynamic lazy-loading via tool_list/tool_load/tool_active; execute_code renamed to shell; tools hot-reloadable without server restart The file tools (file_read, file_write, file_edit, file_grep, file_glob) were extracted into standalone Python scripts. Memory (memory_remember, memory_recall) and reasoning (reasoning_think) followed the same pattern. MCP servers were removed early. Simple scripts won over complex daemons.

Phase 4: Planning System Churn (May – Jul)

Planning went through the most iterations of any subsystem:

1. pl/ directory namespace with flat file-per-task
2. plan_create / plan_complete tools
3. Beads integration (external issue tracker) — made optional, then dropped
4. task_add / task_check tools
5. Inline markdown plans forbidden
6. Final form: a single session/{id}/plan file (markdown checklist), written by the agent, persisted across context compaction The lesson: planning needed to be simple, agent-controlled, and not bureaucratic.

Phase 5: Multi-Agent Coordination (May 25 – Jun 15)

Three patterns emerged:

• subagent_spawn — fire-and-forget session creation, parent never blocks
• subagent_generate — JIT agent config generation (role, constraints, tools)
• cascade — script-driven fan-out with -max-workers, -retries, optional synthesis step

flowchart TB
    UP["User prompt"]
    PARENT["Parent Session"]
    W1["Worker 1"]
    W2["Worker 2"]
    W3["Worker 3"]
    CASCADE["u/cascade\n(spawn + throttle)"]
    CW1["Cascade Worker"]
    CW2["Cascade Worker"]
    UP --> PARENT
    PARENT -- subagent_spawn --> W1
    PARENT -- subagent_spawn --> W2
    PARENT -- subagent_spawn --> W3
    W1 --> PARENT
    W2 --> PARENT
    W3 --> PARENT
    PARENT -- cascade --> CASCADE
    CASCADE --> CW1
    CASCADE --> CW2

Inter-agent communication uses the filesystem: agents write to each other's prompt file.

Phase 6: Frontend Proliferation

flowchart TB
    subgraph Core["agent.Agent"]
        AG["Agent Engine"]
    end
    subgraph Surfaces["Integration Surfaces"]
        P9["9P Filesystem\n(session/ namespace)"]
        DB["D-Bus\n(org.ollie.SessionManager)"]
    end
    subgraph Frontends
        SH["s/sh (terminal)"]
        ACME["acme (Plan 9)"]
        EL["ellie (Emacs)"]
        KG["KDE GUI"]
        KP["KDE Plasmoid"]
        KK["Kate Plugin"]
        KR["KRunner"]
        HTTP["curl / scripts"]
    end
    AG --> P9
    AG --> DB
    SH --> P9
    ACME --> P9
    EL --> P9
    HTTP --> P9
    KG --> DB
    KP --> DB
    KK --> DB
    KR --> DB
    WEB --> DB

• s/sh — shell script frontend (bash, briefly rc, back to bash)
• acme — Plan 9 editor integration with Oi (quick query), Kmpl (completion)
• ellie.el — Emacs with ghost-text completion
• ollie-kde — full Plasma integration: plasmoid, KRunner, Kate plugin, standalone GUI, GUI automation tools
• TUI — removed (Jul) in favor of s/sh and richer GUIs

Phase 7: Security Model (May – Jul)

flowchart LR
    subgraph Permissions["9P Permissions"]
        OWNER["Owner → rw"]
        AGENT["Agent → restricted"]
        PEERS["Peers → read-only"]
    end
    subgraph Execution["Execution Sandbox"]
        AGT["Agent"]
        SANDBOX["landrun sandbox\n(restricted fs)"]
        TOOLS["Tool scripts"]
        ELEV["x/elevate\n(socket broker)"]
        PRIV["Privileged Action"]
    end
    AGT --> SANDBOX --> TOOLS
    TOOLS -- needs escape --> ELEV --> PRIV

Evolution:

1. No sandboxing initially
2. YAML-based sandbox configs (landrun) for execute_code
3. Per-session 9P identity — agents can't read other sessions' tools
4. Elevation broker: socket-based, user-confirmed privilege escalation
5. SSH agent proxy added then removed (too much attack surface)
6. Final: integrated elevation broker replaces separate superpowerd adapter

Phase 8: D-Bus as Second Control Plane (Jun 25 – Jul)

Originally everything was 9P-only. D-Bus was added for desktop integration:

1. Separate ollie-dbus daemon (ollied)
2. Embedded directly into olliesrv (9P server got -no9p flag)
3. KDE uses D-Bus exclusively; acme/sh/el use 9P
4. 9P and D-Bus do not share sessions — independent stores, same core

Phase 9: Remote Execution (Jul 5–19)

Split-brain architecture: orchestration stays local, code execution on a remote machine via SSH.

• ollie-remote binary auto-deployed via SSH bootstrap
• Tools embedded in remote binary
• Streaming output notifications back to local session
• Session persistence across remote restarts

Phase 10: Tool Registry (Jul 20–29)

The final major architectural shift — from "agent knows all tools upfront" to lazy discovery:

1. Tools embed their own ollie:prompt metadata blocks
2. tool_list — discover available tools and descriptions
3. tool_load — promote a tool to a native callable
4. tool_active — introspect loaded tools
5. Skills get the same treatment: skill_list, skill_load, skill_active This solved system prompt bloat — only load what's needed per task.

Phase 11: The Great Flattening (Jul 29–30)

Largest single-day structural change: −4,573 lines net across the codebase. Eliminated accumulated abstractions that no longer served a purpose.

Core (−4,400 lines)

• Killed agent.Core interface — concrete *agent.Agent used directly. Nobody else implemented Core; the interface just added indirection.
• Killed Dispatcher indirection in toolsrv — toolsrv.Server called directly.
• Merged execute/ into tools/ — separate package for 3 functions was pointless overhead.
• Renamed for clarity: agent.Session → agent.History, session.Core → session.Session, NewAgentCore → New.
• Extracted agent/ package from the monolithic session package.
• Threaded context.Context properly through daemon → session → agent (replaced ad-hoc interrupt mechanisms with cancellation).
• Dead code removal: GlobalToolNames, ExtractReturnSchema.

9P server (−214 lines)

• Killed mgr/ package entirely — replaced Manager struct with package functions in fs/session/. The *fs.Tree IS the session collection; rootState (unexported) lives in tree.Data. CRUD via NewRoot, Lookup, Create, Kill, Rename, Shutdown.
• Fixed infinite recursion in session/{sid}/agent/ directory — pathType() didn't recognize agent subdirectories, causing walk to succeed at any depth.
• Fixed phantom session descent — walk into non-existent sessions now fails immediately.
• Fixed /agents empty read — makeStat wasn't computing Length for root files, FUSE kernel saw 0 bytes and never issued a read.
• Fixed session.All() — was looking at root.Children() (empty) instead of rootState.sessions. Broke D-Bus ListSessions and GUI session restore.

Frontends

• acme: paths updated s/ → session/, agent files route through session/{id}/agent/{aid}/.
• ellie.el: same path migration, added ellie--agent-dir for agent ID discovery.
• KDE GUI: no changes needed (D-Bus operates on Go objects, not paths).

New filesystem layout

session/
├── new              write key=value to create session
├── idx              session index (tab-separated)
├── {sid}/
│   ├── plan         session-scoped markdown checklist
│   ├── env          session environment
│   └── agent/
│       └── {aid}/
│           ├── cfg          session configuration (key=value)
│           ├── ctl          control commands
│           ├── state        current state
│           ├── statewait    blocks until state changes
│           ├── chat         conversation log
│           ├── prompt       submit prompts
│           ├── prompt.prev  last submitted prompt
│           ├── offset       byte offset after last user prompt
│           ├── fifo.in      queue a prompt (FIFO input)
│           ├── fifo.out     pop queued prompt (FIFO output)
│           ├── context      rendered context window
│           ├── systemprompt rendered system prompt
│           ├── usage        token stats
│           ├── cost         cost estimate
│           ├── ctxsz        context size
│           ├── models       available models
│           ├── tools        tool management
│           ├── toolsrv.loaded  currently loaded tools
│           ├── toolsrv.rev     revision counter
│           ├── tail         exec helper
│           └── proc/        detached process output

Architecture after (single Go module, no submodules for core/9p)

*fs.Tree (root)                     ← IS the session collection
  └── .Data = *rootState            ← sessions map, config, nextUID
       └── sessions[id] = *Session  ← owns Agent, context, cancel
Package functions (no Manager):
  session.NewRoot(cfg) → *fs.Tree
  session.Lookup(tree, id) → *Session
  session.All(tree) → []*Session
  session.CreateFromRoot(tree, args) → (id, error)
  session.KillFromRoot(tree, id)
  session.RenameFromRoot(tree, old, new)
  session.Shutdown(tree)

SLOC after flattening

Component	Lines
agent + backend + toolsrv + session	10,437
fs/session + cmd/olliesrv	7,925
kde gui	12,331
Total	32,538
Zero dead exported functions remain (verified via LSP + grep across all repos).

Phase 12: Second Flattening & .meta Sidecar (Jul 30)

Another aggressive structural pass. Eliminated the entire script namespace layer (s/, u/, x/), flattened contrib/ into data/, and replaced header-comment-based tool metadata with JSON sidecar files.

Scripts removed (−1,100 lines)

• s/{sh,b,bfg,bbg,ls,kill,cleanup} — shell-based session frontends replaced by ollie-9p rdwr generate/complete/route and GUI frontends
• u/{optimize,complete,cascade,escalate} — utility scripts replaced by generate, complete, route files in the 9P namespace
• x/{bd,prime,freeloader,task} — internal plumbing scripts; PATH prepend (prependOlliePath) removed from toolsrv
• 9P session root trimmed to just new and idx; script-serving code removed
• install-scripts target now empty

Tool metadata: .meta sidecar files

Header comment parsing (ollie:prompt, ollie:tier, ollie:parallel read, args_json:) eliminated entirely. All tool metadata now lives in a JSON sidecar file alongside the executable:

tools/
  file_edit       ← executable (any language)
  file_edit.meta  ← JSON metadata

{
  "description": "Replace text in a file.",
  "prompt": "## file_edit\n\n...",
  "args": {"type":"object", ...},
  "tier": "cold",
  "readOnly": true
}

This decouples metadata from implementation language — compiled Go binaries, Python scripts, and bash tools all use the same discovery mechanism.

LSP tools ported to Go

The Python LSP bridge (_lib/lsp/) replaced with a pure Go implementation:

tools/
├── builtin/          ← in-process handlers (shell, reasoning, tool/skill registry)
└── lsp/              ← shared LSP client library
    └── cmd/          ← individual binaries (lsp_definition, lsp_hover, ...)

Architecture: each LSP tool binary embeds a bridge daemon (started on first invocation via --bridge flag, persists via Unix socket, auto-exits after 5min idle). No Python. No _lib. No external dependencies beyond the LSP servers themselves (gopls, clangd, intelephense).

Repo reorganization

• contrib/{prompts,scripts,services,tools,skills,agents} → data/
• contrib/elisp stays (Emacs frontend contribution)
• tools/builtin/ — built-in tool handlers (moved from tools/)
• tools/lsp/ — Go LSP implementation (replaced data/tools/_lib/lsp/)

Current Topology

ollie/                      ← single Go module
├── agent/                  ← agent loop, history, hooks, commands
├── backend/                ← LLM providers (6 backends)
├── toolsrv/                ← tool server, registry, remote execution
├── session/                ← session lifecycle, config
├── fs/session/             ← 9P filesystem tree
├── dbus/                   ← D-Bus adapter
├── detach/                 ← background process management
├── elevate/                ← elevation broker
├── sandbox/                ← landrun sandbox config
├── env/                    ← environment helpers
├── log/                    ← structured logging
├── paths/                  ← XDG path resolution
├── mount/                  ← 9P FUSE mount client (network transparency)
├── cmd/                    ← binaries (olliesrv, ollie-9p, ollie-remote)
├── kde/                    ← KDE Plasma (submodule)
├── data/tools/             ← tool executables + .meta sidecar files
├── data/agents/            ← agent configs (JSON)
├── data/prompts/           ← system prompt fragments (markdown)
├── data/skills/            ← domain knowledge modules (markdown)
├── data/scripts/           ← ollie-remount, ollie-watchdog
├── data/services/          ← systemd, xdg-autostart
├── prompts/                ← embedded prompt templates
├── sandbox/                ← landrun sandbox profiles (YAML)
└── doc/                    ← documentation

Principles That Emerged

1. Filesystem-as-API — everything is read/write on synthetic files. No custom protocols needed. Monorepo coordinates versions.
2. Scripts over servers — MCP servers removed early. Simple scripts won.
3. Progressive disclosure — lazy tool/skill loading keeps system prompts small until complexity is needed.
4. Plan simplicity — after 5 iterations, planning settled on one markdown checklist file per session.
5. Dual control plane — 9P for power users/editors, D-Bus for desktop. Same core, different surfaces.
6. Security by default — sandboxed execution with explicit elevation, per-session identity.

Dead Ends and Reversals

What	Why it was removed
MCP servers	Too heavyweight; scripts are simpler and faster
TUI frontend	Replaced by KDE GUI and Emacs (ellie)
s/sh, s/bfg, s/bbg	Replaced by generate/complete/route 9P files
pl/ planning namespace	Over-engineered; plan file is sufficient
Beads integration	External dependency for something a checklist file handles
SSH agent proxy	Attack surface not worth the convenience
Separate ollie-dbus daemon	Consolidated into olliesrv
FUSE mounts	Replaced by standalone 9P client binary
Tier routing (FAST/POWER)	Replaced by /route endpoint with real model discovery
call_tool / pipe	Replaced by native tool registry with tool_load
execute_code (multi-lang)	Simplified to shell-only; tool scripts handle language choice
chatwait	Reverted; acme tails chat directly
Header comment metadata	Replaced by .meta sidecar JSON; decouples metadata from language
contrib/ directory	Renamed to data/; nothing was community-contributed
Script namespaces (s/, u/, x/)	Replaced by 9P request-response files