ollie/doc/EVOLUTION.md

Architectural Evolution

How the Ollie system-of-systems emerged and evolved. ~1300 commits over ~3.5 months (Apr 11 – Jul 30, 2026).

Timeline

gantt
    title Ollie Evolution
    dateFormat  YYYY-MM-DD
    axisFormat  %b %d
    section Foundation
    Monorepo + submodules          :done, 2026-04-11, 14d
    9P filesystem                  :done, 2026-04-14, 28d
    section Core Primitives
    Session lifecycle (session/)   :done, 2026-04-20, 30d
    Tool scripts                   :done, 2026-05-01, 20d
    Skills system                  :done, 2026-05-10, 14d
    section Execution
    execute_code / shell           :done, 2026-05-08, 20d
    Batch jobs (b/ → session/bfg)  :done, 2026-05-15, 25d
    Sandbox (landrun)              :done, 2026-05-20, 40d
    section Multi-Agent
    Subagent spawn                 :done, 2026-05-25, 15d
    Cascade orchestrator           :done, 2026-06-05, 20d
    JIT agent generation           :done, 2026-07-10, 10d
    section Frontends
    TUI + Emacs (ellie)            :done, 2026-04-15, 45d
    Acme (Plan 9)                  :done, 2026-05-20, 30d
    KDE/Plasma                     :done, 2026-06-10, 40d
    section Control Planes
    D-Bus adapter (ollied)         :done, 2026-06-25, 20d
    D-Bus embedded in srv          :done, 2026-07-15, 10d
    section Late Evolution
    Remote execution               :done, 2026-07-05, 15d
    Elevation broker               :done, 2026-07-10, 15d
    Tool registry (lazy)           :done, 2026-07-20, 10d
    Reactions / feedback           :done, 2026-07-22, 8d
    FUSE → 9P client               :done, 2026-07-25, 5d
    The Great Flattening           :done, 2026-07-29, 2d

Phase 1: Monorepo Bootstrap (Apr 11)

Started as independent git repos unified under a monorepo with submodules. Initial components:

• agent — agent loop (Go): backend dispatch, tool execution, session state
• fs/session — 9P file server exposing agent sessions as a synthetic filesystem
• tui — terminal UI (later removed)
• el — Emacs integration (ellie.el) Build system was mkfile (Plan 9 make), later Makefile, finally justfile.

Phase 2: The 9P Decision (Apr 14–28)

The defining architectural choice: every agent primitive is a file. Sessions are directories. Sending a prompt is writing to session/{id}/prompt. Reading state is cat session/{id}/state. This made the control plane protocol-agnostic — any program that can read/write files can be a frontend. Key files crystallized: prompt, chat, state, ctl, cfg, statewait. The session/new file (write key=value pairs, read back session ID) became the session factory.

Phase 3: Tool System (Apr 28 – May 10)

Tools evolved through several incarnations:

1. MCP servers (denote-mcp, 9beads-mcp) — external processes, heavyweight
2. Shell/Python scripts in ~/.config/ollie/tools/ — lightweight, sandboxed
3. execute_code as the single built-in tool — all scripts invoked through it
4. call_tool/pipe — named tool dispatch and cross-tool pipelines
5. Tool registry (final form) — dynamic lazy-loading via tool_list/tool_load/tool_active; execute_code renamed to shell; tools hot-reloadable without server restart The file tools (file_read, file_write, file_edit, file_grep, file_glob) were extracted into standalone Python scripts. Memory (memory_remember, memory_recall) and reasoning (reasoning_think) followed the same pattern. MCP servers were removed early. Simple scripts won over complex daemons.

Phase 4: Planning System Churn (May – Jul)

Planning went through the most iterations of any subsystem:

1. pl/ directory namespace with flat file-per-task
2. plan_create / plan_complete tools
3. Beads integration (external issue tracker) — made optional, then dropped
4. task_add / task_check tools
5. Inline markdown plans forbidden
6. Final form: a single session/{id}/plan file (markdown checklist), written by the agent, persisted across context compaction The lesson: planning needed to be simple, agent-controlled, and not bureaucratic.

Phase 5: Multi-Agent Coordination (May 25 – Jun 15)

Three patterns emerged:

• subagent_spawn — fire-and-forget session creation, parent never blocks
• subagent_generate — JIT agent config generation (role, constraints, tools)
• cascade — script-driven fan-out with -max-workers, -retries, optional synthesis step

flowchart TB
    UP["User prompt"]
    PARENT["Parent Session"]
    W1["Worker 1"]
    W2["Worker 2"]
    W3["Worker 3"]
    CASCADE["u/cascade\n(spawn + throttle)"]
    CW1["Cascade Worker"]
    CW2["Cascade Worker"]
    UP --> PARENT
    PARENT -- subagent_spawn --> W1
    PARENT -- subagent_spawn --> W2
    PARENT -- subagent_spawn --> W3
    W1 --> PARENT
    W2 --> PARENT
    W3 --> PARENT
    PARENT -- cascade --> CASCADE
    CASCADE --> CW1
    CASCADE --> CW2

Inter-agent communication uses the filesystem: agents write to each other's prompt file.

Phase 6: Frontend Proliferation

flowchart TB
    subgraph Core["agent.Agent"]
        AG["Agent Engine"]
    end
    subgraph Surfaces["Integration Surfaces"]
        P9["9P Filesystem\n(session/ namespace)"]
        DB["D-Bus\n(org.ollie.SessionManager)"]
    end
    subgraph Frontends
        SH["s/sh (terminal)"]
        ACME["acme (Plan 9)"]
        EL["ellie (Emacs)"]
        KG["KDE GUI"]
        KP["KDE Plasmoid"]
        KK["Kate Plugin"]
        KR["KRunner"]
        HTTP["curl / scripts"]
    end
    AG --> P9
    AG --> DB
    SH --> P9
    ACME --> P9
    EL --> P9
    HTTP --> P9
    KG --> DB
    KP --> DB
    KK --> DB
    KR --> DB
    WEB --> DB

• s/sh — shell script frontend (bash, briefly rc, back to bash)
• acme — Plan 9 editor integration with Oi (quick query), Kmpl (completion)
• ellie.el — Emacs with ghost-text completion
• ollie-kde — full Plasma integration: plasmoid, KRunner, Kate plugin, standalone GUI, GUI automation tools
• TUI — removed (Jul) in favor of s/sh and richer GUIs

Phase 7: Security Model (May – Jul)

flowchart LR
    subgraph Permissions["9P Permissions"]
        OWNER["Owner → rw"]
        AGENT["Agent → restricted"]
        PEERS["Peers → read-only"]
    end
    subgraph Execution["Execution Sandbox"]
        AGT["Agent"]
        SANDBOX["landrun sandbox\n(restricted fs)"]
        TOOLS["Tool scripts"]
        ELEV["x/elevate\n(socket broker)"]
        PRIV["Privileged Action"]
    end
    AGT --> SANDBOX --> TOOLS
    TOOLS -- needs escape --> ELEV --> PRIV

Evolution:

1. No sandboxing initially
2. YAML-based sandbox configs (landrun) for execute_code
3. Per-session 9P identity — agents can't read other sessions' tools
4. Elevation broker: socket-based, user-confirmed privilege escalation
5. SSH agent proxy added then removed (too much attack surface)
6. Final: integrated elevation broker replaces separate superpowerd adapter

Phase 8: D-Bus as Second Control Plane (Jun 25 – Jul)

Originally everything was 9P-only. D-Bus was added for desktop integration:

1. Separate ollie-dbus daemon (ollied)
2. Embedded directly into olliesrv (9P server got -no9p flag)
3. KDE uses D-Bus exclusively; acme/sh/el use 9P
4. 9P and D-Bus do not share sessions — independent stores, same core

Phase 9: Remote Execution (Jul 5–19)

Split-brain architecture: orchestration stays local, code execution on a remote machine via SSH.

• ollie-remote binary auto-deployed via SSH bootstrap
• Tools embedded in remote binary
• Streaming output notifications back to local session
• Session persistence across remote restarts

Phase 10: Tool Registry (Jul 20–29)

The final major architectural shift — from "agent knows all tools upfront" to lazy discovery:

1. Tools embed their own ollie:prompt metadata blocks
2. tool_list — discover available tools and descriptions
3. tool_load — promote a tool to a native callable
4. tool_active — introspect loaded tools
5. Skills get the same treatment: skill_list, skill_load, skill_active This solved system prompt bloat — only load what's needed per task.

Phase 11: The Great Flattening (Jul 29–30)

Largest single-day structural change: −4,573 lines net across the codebase. Eliminated accumulated abstractions that no longer served a purpose.

Core (−4,400 lines)

• Killed agent.Core interface — concrete *agent.Agent used directly. Nobody else implemented Core; the interface just added indirection.
• Killed Dispatcher indirection in toolsrv — toolsrv.Server called directly.
• Merged execute/ into tools/ — separate package for 3 functions was pointless overhead.
• Renamed for clarity: agent.Session → agent.History, session.Core → session.Session, NewAgentCore → New.
• Extracted agent/ package from the monolithic session package.
• Threaded context.Context properly through daemon → session → agent (replaced ad-hoc interrupt mechanisms with cancellation).
• Dead code removal: GlobalToolNames, ExtractReturnSchema.

9P server (−214 lines)

• Killed mgr/ package entirely — replaced Manager struct with package functions in fs/session/. The *fs.Tree IS the session collection; rootState (unexported) lives in tree.Data. CRUD via NewRoot, Lookup, Create, Kill, Rename, Shutdown.
• Fixed infinite recursion in session/{sid}/agent/ directory — pathType() didn't recognize agent subdirectories, causing walk to succeed at any depth.
• Fixed phantom session descent — walk into non-existent sessions now fails immediately.
• Fixed /agents empty read — makeStat wasn't computing Length for root files, FUSE kernel saw 0 bytes and never issued a read.
• Fixed session.All() — was looking at root.Children() (empty) instead of rootState.sessions. Broke D-Bus ListSessions and GUI session restore.

Frontends

• acme: paths updated s/ → session/, agent files route through session/{id}/agent/{aid}/.
• ellie.el: same path migration, added ellie--agent-dir for agent ID discovery.
• KDE GUI: no changes needed (D-Bus operates on Go objects, not paths).

New filesystem layout

session/
├── new              write key=value to create session
├── idx              session index (tab-separated)
├── {sid}/
│   ├── plan         session-scoped markdown checklist
│   ├── env          session environment
│   └── agent/
│       └── {aid}/
│           ├── cfg          session configuration (key=value)
│           ├── ctl          control commands
│           ├── state        current state
│           ├── statewait    blocks until state changes
│           ├── chat         conversation log
│           ├── prompt       submit prompts
│           ├── prompt.prev  last submitted prompt
│           ├── offset       byte offset after last user prompt
│           ├── fifo.in      queue a prompt (FIFO input)
│           ├── fifo.out     pop queued prompt (FIFO output)
│           ├── context      rendered context window
│           ├── systemprompt rendered system prompt
│           ├── usage        token stats
│           ├── cost         cost estimate
│           ├── ctxsz        context size
│           ├── models       available models
│           ├── tools        tool management
│           ├── toolsrv.loaded  currently loaded tools
│           ├── toolsrv.rev     revision counter
│           ├── tail         exec helper
│           └── proc/        detached process output

Architecture after (single Go module, no submodules for core/9p)

*fs.Tree (root)                     ← IS the session collection
  └── .Data = *rootState            ← sessions map, config, nextUID
       └── sessions[id] = *Session  ← owns Agent, context, cancel
Package functions (no Manager):
  session.NewRoot(cfg) → *fs.Tree
  session.Lookup(tree, id) → *Session
  session.All(tree) → []*Session
  session.CreateFromRoot(tree, args) → (id, error)
  session.KillFromRoot(tree, id)
  session.RenameFromRoot(tree, old, new)
  session.Shutdown(tree)

SLOC after flattening

Component	Lines
agent + backend + toolsrv + session	10,437
fs/session + cmd/olliesrv	7,925
kde gui	12,331
Total	32,538
Zero dead exported functions remain (verified via LSP + grep across all repos).

Phase 12: Second Flattening & .meta Sidecar (Jul 30)

Another aggressive structural pass. Eliminated the entire script namespace layer (s/, u/, x/), flattened contrib/ into data/, and replaced header-comment-based tool metadata with JSON sidecar files.

Scripts removed (−1,100 lines)

• s/{sh,b,bfg,bbg,ls,kill,cleanup} — shell-based session frontends replaced by ollie-9p rdwr generate/complete/route and GUI frontends
• u/{optimize,complete,cascade,escalate} — utility scripts replaced by generate, complete, route files in the 9P namespace
• x/{bd,prime,freeloader,task} — internal plumbing scripts; PATH prepend (prependOlliePath) removed from toolsrv
• 9P session root trimmed to just new and idx; script-serving code removed
• install-scripts target now empty

Tool metadata: .meta sidecar files

Header comment parsing (ollie:prompt, ollie:tier, ollie:parallel read, args_json:) eliminated entirely. All tool metadata now lives in a JSON sidecar file alongside the executable:

tools/
  file_edit       ← executable (any language)
  file_edit.meta  ← JSON metadata

{
  "description": "Replace text in a file.",
  "prompt": "## file_edit\n\n...",
  "args": {"type":"object", ...},
  "tier": "cold",
  "readOnly": true
}

This decouples metadata from implementation language — compiled Go binaries, Python scripts, and bash tools all use the same discovery mechanism.

LSP tools ported to Go

The Python LSP bridge (_lib/lsp/) replaced with a pure Go implementation:

tools/
├── builtin/          ← in-process handlers (shell, reasoning, tool/skill registry)
└── lsp/              ← shared LSP client library
    └── cmd/          ← individual binaries (lsp_definition, lsp_hover, ...)

Architecture: each LSP tool binary embeds a bridge daemon (started on first invocation via --bridge flag, persists via Unix socket, auto-exits after 5min idle). No Python. No _lib. No external dependencies beyond the LSP servers themselves (gopls, clangd, intelephense).

Repo reorganization

• contrib/{prompts,scripts,services,tools,skills,agents} → data/
• contrib/elisp stays (Emacs frontend contribution)
• tools/builtin/ — built-in tool handlers (moved from tools/)
• tools/lsp/ — Go LSP implementation (replaced data/tools/_lib/lsp/)

Conditional variants for network transparency

Tools can declare multiple variants gated by host conditions (binary, file, os, arch, env, nenv). First match determines the tool's schema and executable. Enables one .meta to work across heterogeneous hosts — the contract adapts to capabilities. Critical for ollie-remote deployments where the remote host may have different tools, package managers, or init systems.

Current Topology

ollie/                      ← single Go module
├── agent/                  ← agent loop, history, hooks, commands
├── backend/                ← LLM providers (6 backends)
├── toolsrv/                ← tool server, registry, remote execution
├── session/                ← session lifecycle, config
├── fs/session/             ← 9P filesystem tree
├── dbus/                   ← D-Bus adapter
├── detach/                 ← background process management
├── elevate/                ← elevation broker
├── sandbox/                ← landrun sandbox config
├── env/                    ← environment helpers
├── log/                    ← structured logging
├── paths/                  ← XDG path resolution
├── mount/                  ← 9P FUSE mount client (network transparency)
├── cmd/                    ← binaries (olliesrv, ollie-9p, ollie-remote)
├── kde/                    ← KDE Plasma (submodule)
├── data/tools/             ← tool executables + .meta sidecar files
├── data/agents/            ← agent configs (JSON)
├── data/prompts/           ← system prompt fragments (markdown)
├── data/skills/            ← domain knowledge modules (markdown)
├── data/scripts/           ← ollie-remount, ollie-watchdog
├── data/services/          ← systemd, xdg-autostart
├── prompts/                ← embedded prompt templates
├── sandbox/                ← landrun sandbox profiles (YAML)
└── doc/                    ← documentation

Principles That Emerged

1. Filesystem-as-API — everything is read/write on synthetic files. No custom protocols needed. Monorepo coordinates versions.
2. Scripts over servers — MCP servers removed early. Simple scripts won.
3. Progressive disclosure — lazy tool/skill loading keeps system prompts small until complexity is needed.
4. Plan simplicity — after 5 iterations, planning settled on one markdown checklist file per session.
5. Dual control plane — 9P for power users/editors, D-Bus for desktop. Same core, different surfaces.
6. Security by default — sandboxed execution with explicit elevation, per-session identity.

Dead Ends and Reversals

What	Why it was removed
MCP servers	Too heavyweight; scripts are simpler and faster
TUI frontend	Replaced by KDE GUI and Emacs (ellie)
s/sh, s/bfg, s/bbg	Replaced by generate/complete/route 9P files
pl/ planning namespace	Over-engineered; plan file is sufficient
Beads integration	External dependency for something a checklist file handles
SSH agent proxy	Attack surface not worth the convenience
Separate ollie-dbus daemon	Consolidated into olliesrv
FUSE mounts	Replaced by standalone 9P client binary
Tier routing (FAST/POWER)	Replaced by /route endpoint with real model discovery
call_tool / pipe	Replaced by native tool registry with tool_load
execute_code (multi-lang)	Simplified to shell-only; tool scripts handle language choice
chatwait	Reverted; acme tails chat directly
Header comment metadata	Replaced by .meta sidecar JSON; decouples metadata from language
contrib/ directory	Renamed to data/; nothing was community-contributed
Script namespaces (s/, u/, x/)	Replaced by 9P request-response files