397 lines
18 KiB
Markdown
397 lines
18 KiB
Markdown
# Architectural Evolution
|
||
How the Ollie system-of-systems emerged and evolved.
|
||
~1300 commits over ~3.5 months (Apr 11 – Jul 30, 2026).
|
||
## Timeline
|
||
```mermaid
|
||
gantt
|
||
title Ollie Evolution
|
||
dateFormat YYYY-MM-DD
|
||
axisFormat %b %d
|
||
section Foundation
|
||
Monorepo + submodules :done, 2026-04-11, 14d
|
||
9P filesystem :done, 2026-04-14, 28d
|
||
section Core Primitives
|
||
Session lifecycle (session/) :done, 2026-04-20, 30d
|
||
Tool scripts :done, 2026-05-01, 20d
|
||
Skills system :done, 2026-05-10, 14d
|
||
section Execution
|
||
execute_code / shell :done, 2026-05-08, 20d
|
||
Batch jobs (b/ → session/bfg) :done, 2026-05-15, 25d
|
||
Sandbox (landrun) :done, 2026-05-20, 40d
|
||
section Multi-Agent
|
||
Subagent spawn :done, 2026-05-25, 15d
|
||
Cascade orchestrator :done, 2026-06-05, 20d
|
||
JIT agent generation :done, 2026-07-10, 10d
|
||
section Frontends
|
||
TUI + Emacs (ellie) :done, 2026-04-15, 45d
|
||
Acme (Plan 9) :done, 2026-05-20, 30d
|
||
KDE/Plasma :done, 2026-06-10, 40d
|
||
section Control Planes
|
||
D-Bus adapter (ollied) :done, 2026-06-25, 20d
|
||
D-Bus embedded in srv :done, 2026-07-15, 10d
|
||
section Late Evolution
|
||
Remote execution :done, 2026-07-05, 15d
|
||
Elevation broker :done, 2026-07-10, 15d
|
||
Tool registry (lazy) :done, 2026-07-20, 10d
|
||
Reactions / feedback :done, 2026-07-22, 8d
|
||
FUSE → 9P client :done, 2026-07-25, 5d
|
||
The Great Flattening :done, 2026-07-29, 2d
|
||
```
|
||
## Phase 1: Monorepo Bootstrap (Apr 11)
|
||
Started as independent git repos unified under a monorepo with submodules.
|
||
Initial components:
|
||
- **agent** — agent loop (Go): backend dispatch, tool execution, session state
|
||
- **fs/session** — 9P file server exposing agent sessions as a synthetic filesystem
|
||
- **tui** — terminal UI (later removed)
|
||
- **el** — Emacs integration (ellie.el)
|
||
Build system was `mkfile` (Plan 9 make), later `Makefile`, finally `justfile`.
|
||
## Phase 2: The 9P Decision (Apr 14–28)
|
||
The defining architectural choice: **every agent primitive is a file**.
|
||
Sessions are directories. Sending a prompt is writing to `session/{id}/prompt`.
|
||
Reading state is `cat session/{id}/state`. This made the control plane
|
||
protocol-agnostic — any program that can read/write files can be a frontend.
|
||
Key files crystallized: `prompt`, `chat`, `state`, `ctl`, `cfg`, `statewait`.
|
||
The `session/new` file (write key=value pairs, read back session ID) became the
|
||
session factory.
|
||
## Phase 3: Tool System (Apr 28 – May 10)
|
||
Tools evolved through several incarnations:
|
||
1. **MCP servers** (denote-mcp, 9beads-mcp) — external processes, heavyweight
|
||
2. **Shell/Python scripts** in `~/.config/ollie/tools/` — lightweight, sandboxed
|
||
3. **execute_code** as the single built-in tool — all scripts invoked through it
|
||
4. **call_tool/pipe** — named tool dispatch and cross-tool pipelines
|
||
5. **Tool registry** (final form) — dynamic lazy-loading via `tool_list`/`tool_load`/`tool_active`; `execute_code` renamed to `shell`; tools hot-reloadable without server restart
|
||
The file tools (`file_read`, `file_write`, `file_edit`, `file_grep`, `file_glob`)
|
||
were extracted into standalone Python scripts. Memory (`memory_remember`,
|
||
`memory_recall`) and reasoning (`reasoning_think`) followed the same pattern.
|
||
MCP servers were removed early. Simple scripts won over complex daemons.
|
||
## Phase 4: Planning System Churn (May – Jul)
|
||
Planning went through the most iterations of any subsystem:
|
||
1. `pl/` directory namespace with flat file-per-task
|
||
2. `plan_create` / `plan_complete` tools
|
||
3. **Beads** integration (external issue tracker) — made optional, then dropped
|
||
4. `task_add` / `task_check` tools
|
||
5. Inline markdown plans forbidden
|
||
6. **Final form**: a single `session/{id}/plan` file (markdown checklist), written by
|
||
the agent, persisted across context compaction
|
||
The lesson: planning needed to be simple, agent-controlled, and not bureaucratic.
|
||
## Phase 5: Multi-Agent Coordination (May 25 – Jun 15)
|
||
Three patterns emerged:
|
||
- **subagent_spawn** — fire-and-forget session creation, parent never blocks
|
||
- **subagent_generate** — JIT agent config generation (role, constraints, tools)
|
||
- **cascade** — script-driven fan-out with `-max-workers`, `-retries`, optional
|
||
synthesis step
|
||
```mermaid
|
||
flowchart TB
|
||
UP["User prompt"]
|
||
PARENT["Parent Session"]
|
||
W1["Worker 1"]
|
||
W2["Worker 2"]
|
||
W3["Worker 3"]
|
||
CASCADE["u/cascade\n(spawn + throttle)"]
|
||
CW1["Cascade Worker"]
|
||
CW2["Cascade Worker"]
|
||
UP --> PARENT
|
||
PARENT -- subagent_spawn --> W1
|
||
PARENT -- subagent_spawn --> W2
|
||
PARENT -- subagent_spawn --> W3
|
||
W1 --> PARENT
|
||
W2 --> PARENT
|
||
W3 --> PARENT
|
||
PARENT -- cascade --> CASCADE
|
||
CASCADE --> CW1
|
||
CASCADE --> CW2
|
||
```
|
||
Inter-agent communication uses the filesystem: agents write to each other's
|
||
`prompt` file.
|
||
## Phase 6: Frontend Proliferation
|
||
```mermaid
|
||
flowchart TB
|
||
subgraph Core["agent.Agent"]
|
||
AG["Agent Engine"]
|
||
end
|
||
subgraph Surfaces["Integration Surfaces"]
|
||
P9["9P Filesystem\n(session/ namespace)"]
|
||
DB["D-Bus\n(org.ollie.SessionManager)"]
|
||
end
|
||
subgraph Frontends
|
||
SH["s/sh (terminal)"]
|
||
ACME["acme (Plan 9)"]
|
||
EL["ellie (Emacs)"]
|
||
KG["KDE GUI"]
|
||
KP["KDE Plasmoid"]
|
||
KK["Kate Plugin"]
|
||
KR["KRunner"]
|
||
HTTP["curl / scripts"]
|
||
end
|
||
AG --> P9
|
||
AG --> DB
|
||
SH --> P9
|
||
ACME --> P9
|
||
EL --> P9
|
||
HTTP --> P9
|
||
KG --> DB
|
||
KP --> DB
|
||
KK --> DB
|
||
KR --> DB
|
||
WEB --> DB
|
||
```
|
||
- **s/sh** — shell script frontend (bash, briefly rc, back to bash)
|
||
- **acme** — Plan 9 editor integration with `Oi` (quick query), `Kmpl` (completion)
|
||
- **ellie.el** — Emacs with ghost-text completion
|
||
- **ollie-kde** — full Plasma integration: plasmoid, KRunner, Kate plugin, standalone GUI, GUI automation tools
|
||
- **TUI** — removed (Jul) in favor of s/sh and richer GUIs
|
||
## Phase 7: Security Model (May – Jul)
|
||
```mermaid
|
||
flowchart LR
|
||
subgraph Permissions["9P Permissions"]
|
||
OWNER["Owner → rw"]
|
||
AGENT["Agent → restricted"]
|
||
PEERS["Peers → read-only"]
|
||
end
|
||
subgraph Execution["Execution Sandbox"]
|
||
AGT["Agent"]
|
||
SANDBOX["landrun sandbox\n(restricted fs)"]
|
||
TOOLS["Tool scripts"]
|
||
ELEV["x/elevate\n(socket broker)"]
|
||
PRIV["Privileged Action"]
|
||
end
|
||
AGT --> SANDBOX --> TOOLS
|
||
TOOLS -- needs escape --> ELEV --> PRIV
|
||
```
|
||
Evolution:
|
||
1. No sandboxing initially
|
||
2. YAML-based sandbox configs (landrun) for execute_code
|
||
3. Per-session 9P identity — agents can't read other sessions' tools
|
||
4. Elevation broker: socket-based, user-confirmed privilege escalation
|
||
5. SSH agent proxy added then removed (too much attack surface)
|
||
6. Final: integrated elevation broker replaces separate superpowerd adapter
|
||
## Phase 8: D-Bus as Second Control Plane (Jun 25 – Jul)
|
||
Originally everything was 9P-only. D-Bus was added for desktop integration:
|
||
1. Separate `ollie-dbus` daemon (ollied)
|
||
2. Embedded directly into `olliesrv` (9P server got `-no9p` flag)
|
||
3. KDE uses D-Bus exclusively; acme/sh/el use 9P
|
||
4. **9P and D-Bus do not share sessions** — independent stores, same core
|
||
## Phase 9: Remote Execution (Jul 5–19)
|
||
Split-brain architecture: orchestration stays local, code execution on a remote
|
||
machine via SSH.
|
||
- `ollie-remote` binary auto-deployed via SSH bootstrap
|
||
- Tools embedded in remote binary
|
||
- Streaming output notifications back to local session
|
||
- Session persistence across remote restarts
|
||
## Phase 10: Tool Registry (Jul 20–29)
|
||
The final major architectural shift — from "agent knows all tools upfront" to
|
||
lazy discovery:
|
||
1. Tools embed their own `ollie:prompt` metadata blocks
|
||
2. `tool_list` — discover available tools and descriptions
|
||
3. `tool_load` — promote a tool to a native callable
|
||
4. `tool_active` — introspect loaded tools
|
||
5. Skills get the same treatment: `skill_list`, `skill_load`, `skill_active`
|
||
This solved system prompt bloat — only load what's needed per task.
|
||
## Phase 11: The Great Flattening (Jul 29–30)
|
||
Largest single-day structural change: **−4,573 lines net** across the codebase.
|
||
Eliminated accumulated abstractions that no longer served a purpose.
|
||
### Core (−4,400 lines)
|
||
- **Killed `agent.Core` interface** — concrete `*agent.Agent` used directly.
|
||
Nobody else implemented Core; the interface just added indirection.
|
||
- **Killed `Dispatcher` indirection** in toolsrv — `toolsrv.Server` called directly.
|
||
- **Merged `execute/` into `tools/`** — separate package for 3 functions was
|
||
pointless overhead.
|
||
- **Renamed for clarity**: `agent.Session` → `agent.History`,
|
||
`session.Core` → `session.Session`, `NewAgentCore` → `New`.
|
||
- **Extracted `agent/` package** from the monolithic session package.
|
||
- **Threaded context.Context** properly through daemon → session → agent
|
||
(replaced ad-hoc interrupt mechanisms with cancellation).
|
||
- Dead code removal: `GlobalToolNames`, `ExtractReturnSchema`.
|
||
### 9P server (−214 lines)
|
||
- **Killed `mgr/` package entirely** — replaced `Manager` struct with package
|
||
functions in `fs/session/`. The `*fs.Tree` IS the session collection; `rootState`
|
||
(unexported) lives in `tree.Data`. CRUD via `NewRoot`, `Lookup`, `Create`,
|
||
`Kill`, `Rename`, `Shutdown`.
|
||
- **Fixed infinite recursion** in `session/{sid}/agent/` directory — `pathType()`
|
||
didn't recognize agent subdirectories, causing walk to succeed at any depth.
|
||
- **Fixed phantom session descent** — walk into non-existent sessions now fails
|
||
immediately.
|
||
- **Fixed `/agents` empty read** — `makeStat` wasn't computing Length for root
|
||
files, FUSE kernel saw 0 bytes and never issued a read.
|
||
- **Fixed `session.All()`** — was looking at `root.Children()` (empty) instead
|
||
of `rootState.sessions`. Broke D-Bus ListSessions and GUI session restore.
|
||
### Frontends
|
||
- **acme**: paths updated `s/` → `session/`, agent files route through
|
||
`session/{id}/agent/{aid}/`.
|
||
- **ellie.el**: same path migration, added `ellie--agent-dir` for agent ID
|
||
discovery.
|
||
- **KDE GUI**: no changes needed (D-Bus operates on Go objects, not paths).
|
||
### New filesystem layout
|
||
```
|
||
session/
|
||
├── new write key=value to create session
|
||
├── idx session index (tab-separated)
|
||
├── {sid}/
|
||
│ ├── plan session-scoped markdown checklist
|
||
│ ├── env session environment
|
||
│ └── agent/
|
||
│ └── {aid}/
|
||
│ ├── cfg session configuration (key=value)
|
||
│ ├── ctl control commands
|
||
│ ├── state current state
|
||
│ ├── statewait blocks until state changes
|
||
│ ├── chat conversation log
|
||
│ ├── prompt submit prompts
|
||
│ ├── prompt.prev last submitted prompt
|
||
│ ├── offset byte offset after last user prompt
|
||
│ ├── fifo.in queue a prompt (FIFO input)
|
||
│ ├── fifo.out pop queued prompt (FIFO output)
|
||
│ ├── context rendered context window
|
||
│ ├── systemprompt rendered system prompt
|
||
│ ├── usage token stats
|
||
│ ├── cost cost estimate
|
||
│ ├── ctxsz context size
|
||
│ ├── models available models
|
||
│ ├── tools tool management
|
||
│ ├── toolsrv.loaded currently loaded tools
|
||
│ ├── toolsrv.rev revision counter
|
||
│ ├── tail exec helper
|
||
│ └── proc/ detached process output
|
||
```
|
||
### Architecture after (single Go module, no submodules for core/9p)
|
||
```
|
||
*fs.Tree (root) ← IS the session collection
|
||
└── .Data = *rootState ← sessions map, config, nextUID
|
||
└── sessions[id] = *Session ← owns Agent, context, cancel
|
||
Package functions (no Manager):
|
||
session.NewRoot(cfg) → *fs.Tree
|
||
session.Lookup(tree, id) → *Session
|
||
session.All(tree) → []*Session
|
||
session.CreateFromRoot(tree, args) → (id, error)
|
||
session.KillFromRoot(tree, id)
|
||
session.RenameFromRoot(tree, old, new)
|
||
session.Shutdown(tree)
|
||
```
|
||
### SLOC after flattening
|
||
| Component | Lines |
|
||
|-----------|-------|
|
||
| agent + backend + toolsrv + session | 10,437 |
|
||
| fs/session + cmd/olliesrv | 7,925 |
|
||
| kde gui | 12,331 |
|
||
| **Total** | **32,538** |
|
||
Zero dead exported functions remain (verified via LSP + grep across all repos).
|
||
## Phase 12: Second Flattening & .meta Sidecar (Jul 30)
|
||
|
||
Another aggressive structural pass. Eliminated the entire script namespace
|
||
layer (s/, u/, x/), flattened `contrib/` into `data/`, and replaced
|
||
header-comment-based tool metadata with JSON sidecar files.
|
||
|
||
### Scripts removed (−1,100 lines)
|
||
- **s/{sh,b,bfg,bbg,ls,kill,cleanup}** — shell-based session frontends replaced
|
||
by `ollie-9p rdwr generate/complete/route` and GUI frontends
|
||
- **u/{optimize,complete,cascade,escalate}** — utility scripts replaced by
|
||
`generate`, `complete`, `route` files in the 9P namespace
|
||
- **x/{bd,prime,freeloader,task}** — internal plumbing scripts; PATH prepend
|
||
(`prependOlliePath`) removed from toolsrv
|
||
- 9P session root trimmed to just `new` and `idx`; script-serving code removed
|
||
- `install-scripts` target now empty
|
||
|
||
### Tool metadata: .meta sidecar files
|
||
Header comment parsing (`ollie:prompt`, `ollie:tier`, `ollie:parallel read`,
|
||
`args_json:`) **eliminated entirely**. All tool metadata now lives in a JSON
|
||
sidecar file alongside the executable:
|
||
|
||
```
|
||
tools/
|
||
file_edit ← executable (any language)
|
||
file_edit.meta ← JSON metadata
|
||
```
|
||
|
||
```json
|
||
{
|
||
"description": "Replace text in a file.",
|
||
"prompt": "## file_edit\n\n...",
|
||
"args": {"type":"object", ...},
|
||
"tier": "cold",
|
||
"readOnly": true
|
||
}
|
||
```
|
||
|
||
This decouples metadata from implementation language — compiled Go binaries,
|
||
Python scripts, and bash tools all use the same discovery mechanism.
|
||
|
||
### LSP tools ported to Go
|
||
The Python LSP bridge (`_lib/lsp/`) replaced with a pure Go implementation:
|
||
|
||
```
|
||
tools/
|
||
├── builtin/ ← in-process handlers (shell, reasoning, tool/skill registry)
|
||
└── lsp/ ← shared LSP client library
|
||
└── cmd/ ← individual binaries (lsp_definition, lsp_hover, ...)
|
||
```
|
||
|
||
Architecture: each LSP tool binary embeds a bridge daemon (started on first
|
||
invocation via `--bridge` flag, persists via Unix socket, auto-exits after 5min
|
||
idle). No Python. No `_lib`. No external dependencies beyond the LSP servers
|
||
themselves (gopls, clangd, intelephense).
|
||
|
||
### Repo reorganization
|
||
- `contrib/{prompts,scripts,services,tools,skills,agents}` → `data/`
|
||
- `contrib/elisp` stays (Emacs frontend contribution)
|
||
- `tools/builtin/` — built-in tool handlers (moved from `tools/`)
|
||
- `tools/lsp/` — Go LSP implementation (replaced `data/tools/_lib/lsp/`)
|
||
|
||
## Current Topology
|
||
```
|
||
ollie/ ← single Go module
|
||
├── agent/ ← agent loop, history, hooks, commands
|
||
├── backend/ ← LLM providers (6 backends)
|
||
├── toolsrv/ ← tool server, registry, remote execution
|
||
├── session/ ← session lifecycle, config
|
||
├── fs/session/ ← 9P filesystem tree
|
||
├── dbus/ ← D-Bus adapter
|
||
├── detach/ ← background process management
|
||
├── elevate/ ← elevation broker
|
||
├── sandbox/ ← landrun sandbox config
|
||
├── env/ ← environment helpers
|
||
├── log/ ← structured logging
|
||
├── paths/ ← XDG path resolution
|
||
├── mount/ ← 9P FUSE mount client (network transparency)
|
||
├── cmd/ ← binaries (olliesrv, ollie-9p, ollie-remote)
|
||
├── kde/ ← KDE Plasma (submodule)
|
||
├── data/tools/ ← tool executables + .meta sidecar files
|
||
├── data/agents/ ← agent configs (JSON)
|
||
├── data/prompts/ ← system prompt fragments (markdown)
|
||
├── data/skills/ ← domain knowledge modules (markdown)
|
||
├── data/scripts/ ← ollie-remount, ollie-watchdog
|
||
├── data/services/ ← systemd, xdg-autostart
|
||
├── prompts/ ← embedded prompt templates
|
||
├── sandbox/ ← landrun sandbox profiles (YAML)
|
||
└── doc/ ← documentation
|
||
```
|
||
## Principles That Emerged
|
||
1. **Filesystem-as-API** — everything is read/write on synthetic files. No
|
||
custom protocols needed.
|
||
Monorepo coordinates versions.
|
||
3. **Scripts over servers** — MCP servers removed early. Simple scripts won.
|
||
4. **Progressive disclosure** — lazy tool/skill loading keeps system prompts
|
||
small until complexity is needed.
|
||
5. **Plan simplicity** — after 5 iterations, planning settled on one markdown
|
||
checklist file per session.
|
||
6. **Dual control plane** — 9P for power users/editors, D-Bus for desktop.
|
||
Same core, different surfaces.
|
||
7. **Security by default** — sandboxed execution with explicit elevation,
|
||
per-session identity.
|
||
## Dead Ends and Reversals
|
||
| What | Why it was removed |
|
||
|------|-------------------|
|
||
| MCP servers | Too heavyweight; scripts are simpler and faster |
|
||
| TUI frontend | Replaced by KDE GUI and Emacs (ellie) |
|
||
| s/sh, s/bfg, s/bbg | Replaced by `generate`/`complete`/`route` 9P files |
|
||
| `pl/` planning namespace | Over-engineered; plan file is sufficient |
|
||
| Beads integration | External dependency for something a checklist file handles |
|
||
| SSH agent proxy | Attack surface not worth the convenience |
|
||
| Separate ollie-dbus daemon | Consolidated into olliesrv |
|
||
| FUSE mounts | Replaced by standalone 9P client binary |
|
||
| Tier routing (FAST/POWER) | Replaced by /route endpoint with real model discovery |
|
||
| call_tool / pipe | Replaced by native tool registry with tool_load |
|
||
| execute_code (multi-lang) | Simplified to shell-only; tool scripts handle language choice |
|
||
| chatwait | Reverted; acme tails chat directly |
|
||
| Header comment metadata | Replaced by .meta sidecar JSON; decouples metadata from language |
|
||
| `contrib/` directory | Renamed to `data/`; nothing was community-contributed |
|
||
| Script namespaces (s/, u/, x/) | Replaced by 9P request-response files | |