doc: add EVOLUTION.md — architectural history of the system
This commit is contained in:
Levi Neely
parent
4ac381dff0
commit
c5b6e5b694
|
|
@ -0,0 +1,282 @@
|
|||
# Architectural Evolution
|
||||
|
||||
How the Ollie system-of-systems emerged and evolved.
|
||||
1261 commits over ~3.5 months (Apr 11 – Jul 29, 2026).
|
||||
|
||||
## Timeline
|
||||
|
||||
```mermaid
|
||||
gantt
|
||||
title Ollie Architecture Evolution
|
||||
dateFormat YYYY-MM-DD
|
||||
axisFormat %b %d
|
||||
|
||||
section Foundation
|
||||
Monorepo + submodules :2026-04-11, 3d
|
||||
9P filesystem interface :2026-04-14, 14d
|
||||
|
||||
section Core Primitives
|
||||
Session lifecycle (s/) :2026-04-20, 21d
|
||||
Tool scripts (file/memory/reasoning) :2026-04-28, 14d
|
||||
Skills system :2026-05-01, 10d
|
||||
|
||||
section Execution
|
||||
execute_code (multi-lang) :2026-05-10, 14d
|
||||
Batch jobs (b/ → s/bfg,s/bbg) :2026-05-15, 21d
|
||||
Sandbox (landrun) :2026-05-20, 60d
|
||||
|
||||
section Multi-Agent
|
||||
Subagent spawn :2026-05-25, 14d
|
||||
Cascade orchestrator :2026-06-01, 14d
|
||||
JIT agent generation :2026-06-15, 7d
|
||||
|
||||
section Frontends
|
||||
TUI + Emacs (ellie) :2026-04-11, 60d
|
||||
Acme (Plan 9) :2026-06-01, 45d
|
||||
Web UI + httpgw :2026-05-28, 30d
|
||||
KDE/Plasma (plasmoid + GUI) :2026-06-20, 40d
|
||||
|
||||
section Control Planes
|
||||
D-Bus adapter (ollied) :2026-06-25, 21d
|
||||
D-Bus embedded in olliesrv :2026-07-10, 7d
|
||||
|
||||
section Late Evolution
|
||||
Remote execution (split-brain) :2026-07-05, 14d
|
||||
Elevation broker :2026-07-10, 14d
|
||||
Tool registry (lazy-load) :2026-07-20, 9d
|
||||
Reactions / feedback :2026-07-18, 7d
|
||||
FUSE → 9P client migration :2026-07-20, 5d
|
||||
```
|
||||
|
||||
## Phase 1: Monorepo Bootstrap (Apr 11)
|
||||
|
||||
Started as independent git repos unified under a monorepo with submodules.
|
||||
Initial components:
|
||||
|
||||
- **core** — agent loop (Go): backend dispatch, tool execution, session state
|
||||
- **9p** — 9P file server exposing agent sessions as a synthetic filesystem
|
||||
- **tui** — terminal UI (later removed)
|
||||
- **el** — Emacs integration (ellie.el)
|
||||
|
||||
Build system was `mkfile` (Plan 9 make), later `Makefile`, finally `justfile`.
|
||||
|
||||
## Phase 2: The 9P Decision (Apr 14–28)
|
||||
|
||||
The defining architectural choice: **every agent primitive is a file**.
|
||||
|
||||
Sessions are directories. Sending a prompt is writing to `s/{id}/prompt`.
|
||||
Reading state is `cat s/{id}/state`. This made the control plane
|
||||
protocol-agnostic — any program that can read/write files can be a frontend.
|
||||
|
||||
Key files crystallized: `prompt`, `chat`, `state`, `ctl`, `cfg`, `statewait`.
|
||||
|
||||
The `s/new` file (write key=value pairs, read back session ID) became the
|
||||
session factory.
|
||||
|
||||
## Phase 3: Tool System (Apr 28 – May 10)
|
||||
|
||||
Tools evolved through several incarnations:
|
||||
|
||||
1. **MCP servers** (denote-mcp, 9beads-mcp) — external processes, heavyweight
|
||||
2. **Shell/Python scripts** in `~/.config/ollie/tools/` — lightweight, sandboxed
|
||||
3. **execute_code** as the single built-in tool — all scripts invoked through it
|
||||
4. **call_tool/pipe** — named tool dispatch and cross-tool pipelines
|
||||
5. **Tool registry** (final form) — lazy-loading via `tool_list`/`tool_load`/`tool_active`; `execute_code` renamed to `shell`
|
||||
|
||||
The file tools (`file_read`, `file_write`, `file_edit`, `file_grep`, `file_glob`)
|
||||
were extracted into standalone Python scripts. Memory (`memory_remember`,
|
||||
`memory_recall`) and reasoning (`reasoning_think`) followed the same pattern.
|
||||
|
||||
MCP servers were removed early. Simple scripts won over complex daemons.
|
||||
|
||||
## Phase 4: Planning System Churn (May – Jul)
|
||||
|
||||
Planning went through the most iterations of any subsystem:
|
||||
|
||||
1. `pl/` directory namespace with flat file-per-task
|
||||
2. `plan_create` / `plan_complete` tools
|
||||
3. **Beads** integration (external issue tracker) — made optional, then dropped
|
||||
4. `task_add` / `task_check` tools
|
||||
5. Inline markdown plans forbidden
|
||||
6. **Final form**: a single `s/{id}/plan` file (markdown checklist), written by
|
||||
the agent, persisted across context compaction
|
||||
|
||||
The lesson: planning needed to be simple, agent-controlled, and not bureaucratic.
|
||||
|
||||
## Phase 5: Multi-Agent Coordination (May 25 – Jun 15)
|
||||
|
||||
Three patterns emerged:
|
||||
|
||||
- **subagent_spawn** — fire-and-forget session creation, parent never blocks
|
||||
- **subagent_generate** — JIT agent config generation (role, constraints, tools)
|
||||
- **cascade** — script-driven fan-out with `-max-workers`, `-retries`, optional
|
||||
synthesis step
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
User -->|prompt| Parent[Parent Session]
|
||||
Parent -->|subagent_spawn| W1[Worker 1]
|
||||
Parent -->|subagent_spawn| W2[Worker 2]
|
||||
Parent -->|subagent_spawn| W3[Worker 3]
|
||||
W1 -->|write to prompt| Parent
|
||||
W2 -->|write to prompt| Parent
|
||||
W3 -->|write to prompt| Parent
|
||||
|
||||
Parent -->|cascade| Cascade[u/cascade script]
|
||||
Cascade -->|spawn + throttle| CW1[Cascade Worker]
|
||||
Cascade -->|spawn + throttle| CW2[Cascade Worker]
|
||||
Cascade -->|synthesize| Parent
|
||||
```
|
||||
|
||||
Inter-agent communication uses the filesystem: agents write to each other's
|
||||
`prompt` file or use bidirectional `peer/` links.
|
||||
|
||||
## Phase 6: Frontend Proliferation
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph Control Planes
|
||||
9P[9P File Server]
|
||||
DBus[D-Bus Service]
|
||||
end
|
||||
|
||||
subgraph Frontends
|
||||
SH[s/sh — shell]
|
||||
Acme[ollie-acme]
|
||||
El[ellie.el — Emacs]
|
||||
KDE[ollie-kde — Plasma]
|
||||
Web[Web UI]
|
||||
end
|
||||
|
||||
subgraph Server
|
||||
Core[agent.Core]
|
||||
end
|
||||
|
||||
Core --- 9P
|
||||
Core --- DBus
|
||||
9P --- SH
|
||||
9P --- Acme
|
||||
9P --- El
|
||||
DBus --- KDE
|
||||
DBus ---|httpgw| Web
|
||||
```
|
||||
|
||||
- **s/sh** — shell script frontend (bash, briefly rc, back to bash)
|
||||
- **acme** — Plan 9 editor integration with `Oi` (quick query), `Kmpl` (completion)
|
||||
- **ellie.el** — Emacs with ghost-text completion
|
||||
- **httpgw** — HTTP→D-Bus gateway
|
||||
- **webui** — browser frontend via httpgw
|
||||
- **ollie-kde** — full Plasma integration: plasmoid, KRunner, Kate plugin, standalone GUI, GUI automation tools
|
||||
- **TUI** — removed (Jul) in favor of s/sh and richer GUIs
|
||||
|
||||
## Phase 7: Security Model (May – Jul)
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Agent -->|sandboxed| Landrun[landrun sandbox]
|
||||
Landrun -->|restricted fs| Tools
|
||||
Tools -->|needs escape| Elevate[x/elevate broker]
|
||||
Elevate -->|socket| Broker[Integrated Broker]
|
||||
Broker -->|user confirms| Action[Privileged Action]
|
||||
|
||||
subgraph 9P Permissions
|
||||
Owner[Session Owner — rw]
|
||||
AgentID[Agent — restricted]
|
||||
Peer[Peers — read-only]
|
||||
end
|
||||
```
|
||||
|
||||
Evolution:
|
||||
|
||||
1. No sandboxing initially
|
||||
2. YAML-based sandbox configs (landrun) for execute_code
|
||||
3. Per-session 9P identity — agents can't read other sessions' tools
|
||||
4. Elevation broker: socket-based, user-confirmed privilege escalation
|
||||
5. SSH agent proxy added then removed (too much attack surface)
|
||||
6. Final: integrated elevation broker replaces separate superpowerd adapter
|
||||
|
||||
## Phase 8: D-Bus as Second Control Plane (Jun 25 – Jul)
|
||||
|
||||
Originally everything was 9P-only. D-Bus was added for desktop integration:
|
||||
|
||||
1. Separate `ollie-dbus` daemon (ollied)
|
||||
2. Embedded directly into `olliesrv` (9P server got `-nodbus` flag)
|
||||
3. KDE uses D-Bus exclusively; acme/sh/el use 9P
|
||||
4. **9P and D-Bus do not share sessions** — independent stores, same core
|
||||
|
||||
## Phase 9: Remote Execution (Jul 5–19)
|
||||
|
||||
Split-brain architecture: orchestration stays local, code execution on a remote
|
||||
machine via SSH.
|
||||
|
||||
- `ollie-remote` binary auto-deployed via SSH bootstrap
|
||||
- Tools embedded in remote binary
|
||||
- Streaming output notifications back to local session
|
||||
- Session persistence across remote restarts
|
||||
|
||||
## Phase 10: Tool Registry (Jul 20–29)
|
||||
|
||||
The final major architectural shift — from "agent knows all tools upfront" to
|
||||
lazy discovery:
|
||||
|
||||
1. Tools embed their own `ollie:prompt` metadata blocks
|
||||
2. `tool_list` — discover available tools and descriptions
|
||||
3. `tool_load` — promote a tool to a native callable
|
||||
4. `tool_active` — introspect loaded tools
|
||||
5. Skills get the same treatment: `skill_list`, `skill_load`, `skill_active`
|
||||
|
||||
This solved system prompt bloat — only load what's needed per task.
|
||||
|
||||
## Current Topology
|
||||
|
||||
```
|
||||
ollie/ ← monorepo root
|
||||
├── core/ ← agent loop, backends, tool dispatch (Go)
|
||||
├── 9p/ ← 9P server + embedded D-Bus adapter (Go)
|
||||
├── acme/ ← Plan 9 acme frontend (Go)
|
||||
├── el/ ← Emacs integration (elisp)
|
||||
├── kde/ ← KDE Plasma: plasmoid, GUI, Kate, KRunner (C++/QML)
|
||||
├── httpgw/ ← HTTP→D-Bus gateway (Go)
|
||||
├── webui/ ← browser frontend (JS)
|
||||
├── remote/ ← split-brain execution server (Go)
|
||||
├── tools/ ← 37 tool scripts (Python)
|
||||
├── scripts/{s,u,x}/ ← filesystem-hosted scripts
|
||||
├── agents/ ← agent configs (JSON)
|
||||
├── prompts/ ← system prompt fragments (markdown)
|
||||
├── skills/ ← domain knowledge modules (markdown)
|
||||
├── sandbox/ ← landrun sandbox profiles (YAML)
|
||||
├── services/ ← init/service files
|
||||
└── experiments/ ← multi-agent trial docs
|
||||
```
|
||||
|
||||
## Principles That Emerged
|
||||
|
||||
1. **Filesystem-as-API** — everything is read/write on synthetic files. No
|
||||
custom protocols needed.
|
||||
2. **Submodule independence** — each component has its own repo, tests, build.
|
||||
Monorepo coordinates versions.
|
||||
3. **Scripts over servers** — MCP servers removed early. Simple scripts won.
|
||||
4. **Progressive disclosure** — lazy tool/skill loading keeps system prompts
|
||||
small until complexity is needed.
|
||||
5. **Plan simplicity** — after 5 iterations, planning settled on one markdown
|
||||
checklist file per session.
|
||||
6. **Dual control plane** — 9P for power users/editors, D-Bus for desktop.
|
||||
Same core, different surfaces.
|
||||
7. **Security by default** — sandboxed execution with explicit elevation,
|
||||
per-session identity.
|
||||
|
||||
## Dead Ends and Reversals
|
||||
|
||||
| What | Why it was removed |
|
||||
|------|-------------------|
|
||||
| MCP servers | Too heavyweight; scripts are simpler and faster |
|
||||
| TUI frontend | Replaced by s/sh (lighter) and KDE GUI (richer) |
|
||||
| `pl/` planning namespace | Over-engineered; plan file is sufficient |
|
||||
| Beads integration | External dependency for something a checklist file handles |
|
||||
| SSH agent proxy | Attack surface not worth the convenience |
|
||||
| Separate ollie-dbus daemon | Consolidated into olliesrv |
|
||||
| FUSE mounts | Replaced by standalone 9P client binary |
|
||||
| Tier routing (FAST/POWER) | Replaced by /route endpoint with real model discovery |
|
||||
| call_tool / pipe | Replaced by native tool registry with tool_load |
|
||||
| execute_code (multi-lang) | Simplified to shell-only; tool scripts handle language choice |
|
||||
| chatwait | Reverted; acme tails chat directly |
|
||||
Loading…
Reference in New Issue