# Architectural Evolution
How the Ollie system-of-systems emerged and evolved.
~1400 commits over ~3.75 months (Apr 11 – Aug 4, 2026).
## Timeline
```mermaid
gantt
    title Ollie Evolution
    dateFormat  YYYY-MM-DD
    axisFormat  %b %d
    section Foundation
    Monorepo + submodules          :done, 2026-04-11, 14d
    9P filesystem                  :done, 2026-04-14, 28d
    section Core Primitives
    Session lifecycle (session/)   :done, 2026-04-20, 30d
    Tool scripts                   :done, 2026-05-01, 20d
    Skills system                  :done, 2026-05-10, 14d
    section Execution
    execute_code / shell           :done, 2026-05-08, 20d
    Batch jobs (b/ → session/bfg)  :done, 2026-05-15, 25d
    Sandbox (landrun)              :done, 2026-05-20, 40d
    section Multi-Agent
    Subagent spawn                 :done, 2026-05-25, 15d
    Cascade orchestrator           :done, 2026-06-05, 20d
    JIT agent generation           :done, 2026-07-10, 10d
    section Frontends
    TUI + Emacs (ellie)            :done, 2026-04-15, 45d
    Acme (Plan 9)                  :done, 2026-05-20, 30d
    KDE/Plasma                     :done, 2026-06-10, 40d
    section Control Planes
    D-Bus adapter (ollied)         :done, 2026-06-25, 20d
    D-Bus embedded in srv          :done, 2026-07-15, 10d
    D-Bus removed                  :done, 2026-07-31, 1d
    section Late Evolution
    Remote execution               :done, 2026-07-05, 15d
    Elevation broker               :done, 2026-07-10, 15d
    Tool registry (lazy)           :done, 2026-07-20, 10d
    FUSE → 9P client               :done, 2026-07-25, 5d
    The Great Flattening           :done, 2026-07-29, 2d
    9P Declarative EDSL            :done, 2026-08-02, 1d
    Zero built-in tools            :done, 2026-08-02, 2d
    EDSL extraction + cleanup      :done, 2026-08-03, 2d
```
## Phase 1: Monorepo Bootstrap (Apr 11)
Started as independent git repos unified under a monorepo with submodules.
Initial components:
- **agent** — agent loop (Go): backend dispatch, tool execution, session state
- **fs**** — 9P file server exposing agent sessions as a synthetic filesystem
- **tui** — terminal UI (later removed)
- **el** — Emacs integration (ellie.el)
Build system was `mkfile` (Plan 9 make), later `Makefile`, finally `justfile`.
## Phase 2: The 9P Decision (Apr 14–28)
The defining architectural choice: **every agent primitive is a file**.
Sessions are directories. Sending a prompt is writing to `session/{sname}/prompt`.
Reading state is `cat session/{sname}/state`. This made the control plane
protocol-agnostic — any program that can read/write files can be a frontend.
Key files crystallized: `prompt`, `chat`, `state`, `ctl`, `cfg`, `statewait`.
The `session/new` file (write key=value pairs, read back session ID) became the
session factory.
## Phase 3: Tool System (Apr 28 – May 10)
Tools evolved through several incarnations:
1. **MCP servers** (denote-mcp, 9beads-mcp) — external processes, heavyweight
2. **Shell/Python scripts** in `~/.config/ollie/tools/` — lightweight, sandboxed
3. **execute_code** as the single built-in tool — all scripts invoked through it
4. **call_tool/pipe** — named tool dispatch and cross-tool pipelines
5. **Tool registry** (final form) — dynamic lazy-loading via `tool_list`/`tool_load`/`tool_active`; `execute_code` renamed to `shell`; tools hot-reloadable without server restart
The file tools (`file_read`, `file_write`, `file_edit`, `file_grep`, `file_glob`)
were extracted into standalone Python scripts. Memory (`memory_remember`,
`memory_recall`) and reasoning (`reasoning_think`) followed the same pattern.
MCP servers were removed early. Simple scripts won over complex daemons.
## Phase 4: Planning System Churn (May – Jul)
Planning went through the most iterations of any subsystem:
1. `pl/` directory namespace with flat file-per-task
2. `plan_create` / `plan_complete` tools
3. **Beads** integration (external issue tracker) — made optional, then dropped
4. `task_add` / `task_check` tools
5. Inline markdown plans forbidden
6. **Final form**: a single `session/{sname}/plan` file (markdown checklist), written by
   the agent, persisted across context compaction
The lesson: planning needed to be simple, agent-controlled, and not bureaucratic.
## Phase 5: Multi-Agent Coordination (May 25 – Jun 15)
Three patterns emerged:
- **subagent_spawn** — fire-and-forget session creation, parent never blocks
- **subagent_generate** — JIT agent config generation (role, constraints, tools)
- **cascade** — script-driven fan-out with `-max-workers`, `-retries`, optional
  synthesis step
```mermaid
flowchart TB
    UP["User prompt"]
    PARENT["Parent Session"]
    W1["Worker 1"]
    W2["Worker 2"]
    W3["Worker 3"]
    CASCADE["u/cascade\n(spawn + throttle)"]
    CW1["Cascade Worker"]
    CW2["Cascade Worker"]
    UP --> PARENT
    PARENT -- subagent_spawn --> W1
    PARENT -- subagent_spawn --> W2
    PARENT -- subagent_spawn --> W3
    W1 --> PARENT
    W2 --> PARENT
    W3 --> PARENT
    PARENT -- cascade --> CASCADE
    CASCADE --> CW1
    CASCADE --> CW2
```
Inter-agent communication uses the filesystem: agents write to each other's
`prompt` file.
## Phase 6: Frontend Proliferation
```mermaid
flowchart TB
    subgraph Core["agent.Agent"]
        AG["Agent Engine"]
    end
    subgraph Surfaces["Integration Surfaces"]
        P9["9P Filesystem\n(session/ namespace)"]
        DB["D-Bus\n(org.ollie.SessionManager)"]
    end
    subgraph Frontends
        SH["s/sh (terminal)"]
        ACME["acme (Plan 9)"]
        EL["ellie (Emacs)"]
        KG["KDE GUI"]
        KP["KDE Plasmoid"]
        KK["Kate Plugin"]
        KR["KRunner"]
        HTTP["curl / scripts"]
    end
    AG --> P9
    AG --> DB
    SH --> P9
    ACME --> P9
    EL --> P9
    HTTP --> P9
    KG --> DB
    KP --> DB
    KK --> DB
    KR --> DB
    WEB --> DB
```
- **s/sh** — shell script frontend (bash, briefly rc, back to bash)
- **acme** — Plan 9 editor integration with workspace-scoped navigator sessions
- **ellie.el** — Emacs with ghost-text completion
- **ollie-kde** — full Plasma integration: plasmoid, KRunner, Kate plugin, standalone GUI, GUI automation tools
- **TUI** — removed (Jul) in favor of s/sh and richer GUIs
## Phase 7: Security Model (May – Jul)
```mermaid
flowchart LR
    subgraph Permissions["9P Permissions"]
        OWNER["Owner → rw"]
        AGENT["Agent → restricted"]
        PEERS["Peers → read-only"]
    end
    subgraph Execution["Execution Sandbox"]
        AGT["Agent"]
        SANDBOX["landrun sandbox\n(restricted fs)"]
        TOOLS["Tool scripts"]
        ELEV["x/elevate\n(socket broker)"]
        PRIV["Privileged Action"]
    end
    AGT --> SANDBOX --> TOOLS
    TOOLS -- needs escape --> ELEV --> PRIV
```
Evolution:
1. No sandboxing initially
2. YAML-based sandbox configs (landrun) for execute_code
3. Per-session 9P identity — agents can't read other sessions' tools
4. Elevation broker: socket-based, user-confirmed privilege escalation
5. SSH agent proxy added then removed (too much attack surface)
6. Final: integrated elevation broker replaces separate superpowerd adapter
## Phase 8: D-Bus as Second Control Plane (Jun 25 – Jul)
Originally everything was 9P-only. D-Bus was added for desktop integration:
1. Separate `ollie-dbus` daemon (ollied)
2. Embedded directly into `olliesrv` (9P server got `-no9p` flag)
3. KDE uses D-Bus exclusively; acme/sh/el use 9P
4. **9P and D-Bus do not share sessions** — independent stores, same core
## Phase 9: Remote Execution (Jul 5–19)
Split-brain architecture: orchestration stays local, code execution on a remote
machine via SSH.
- `ollie-remote` binary auto-deployed via SSH bootstrap
- Tools embedded in remote binary
- Streaming output notifications back to local session
- Session persistence across remote restarts
## Phase 10: Tool Registry (Jul 20–29)
The final major architectural shift — from "agent knows all tools upfront" to
lazy discovery:
1. Tools embed their own `ollie:prompt` metadata blocks
2. `tool_list` — discover available tools and descriptions
3. `tool_load` — promote a tool to a native callable
4. `tool_active` — introspect loaded tools
5. Skills get the same treatment: `skill_list`, `skill_load`, `skill_active`
This solved system prompt bloat — only load what's needed per task.
## Phase 11: The Great Flattening (Jul 29–30)
Largest single-day structural change: **−4,573 lines net** across the codebase.
Eliminated accumulated abstractions that no longer served a purpose.
### Core (−4,400 lines)
- **Killed `agent.Core` interface** — concrete `*agent.Agent` used directly.
  Nobody else implemented Core; the interface just added indirection.
- **Killed `Dispatcher` indirection** in toolsrv — `toolsrv.Server` called directly.
- **Merged `execute/` into `tools/`** — separate package for 3 functions was
  pointless overhead.
- **Renamed for clarity**: `agent.Session` → `agent.History`,
  `session.Core` → `session.Session`, `NewAgentCore` → `New`.
- **Extracted `agent/` package** from the monolithic session package.
- **Threaded context.Context** properly through daemon → session → agent
  (replaced ad-hoc interrupt mechanisms with cancellation).
- Dead code removal: `GlobalToolNames`, `ExtractReturnSchema`.
### 9P server (−214 lines)
- **Killed `mgr/` package entirely** — replaced `Manager` struct with package
  functions in `**fs**/`. The `*fs.Tree` IS the session collection; `rootState`
  (unexported) lives in `tree.Data`. CRUD via `NewRoot`, `Lookup`, `Create`,
  `Kill`, `Rename`, `Shutdown`.
- **Fixed infinite recursion** in `session/{sname}/agent/` directory — `pathType()`
  didn't recognize agent subdirectories, causing walk to succeed at any depth.
- **Fixed phantom session descent** — walk into non-existent sessions now fails
  immediately.
- **Fixed `/agents` empty read** — `makeStat` wasn't computing Length for root
  files, FUSE kernel saw 0 bytes and never issued a read.
- **Fixed `session.All()`** — was looking at `root.Children()` (empty) instead
  of `rootState.sessions`. Broke D-Bus ListSessions and GUI session restore.
### Frontends
- **acme**: paths updated `s/` → `session/`, agent files route through
  `session/{sname}/agent/{aname}/`.
- **ellie.el**: same path migration, added `ellie--agent-dir` for agent ID
  discovery.
- **KDE GUI**: no changes needed (D-Bus operates on Go objects, not paths).
### New filesystem layout
```
session/
├── new              write key=value to create session
├── idx              session index (tab-separated)
├── {sname}/
│   ├── plan         session-scoped markdown checklist
│   ├── env          session environment
│   └── agent/
│       └── {aname}/
│           ├── cfg          session configuration (key=value)
│           ├── ctl          control commands
│           ├── state        current state
│           ├── statewait    blocks until state changes
│           ├── chat         conversation log
│           ├── prompt       submit prompts
│           ├── prompt.prev  last submitted prompt
│           ├── offset       byte offset after last user prompt
│           ├── fifo.in      queue a prompt (FIFO input)
│           ├── fifo.out     pop queued prompt (FIFO output)
│           ├── context      rendered context window
│           ├── systemprompt rendered system prompt
│           ├── usage        token stats
│           ├── cost         cost estimate
│           ├── ctxsz        context size
│           ├── models       available models
│           ├── tools        tool management
│           ├── toolsrv.loaded  currently loaded tools
│           ├── toolsrv.rev     revision counter
│           ├── tail         exec helper
│           └── proc/        detached process output
```
### Architecture after (single Go module, no submodules for core/9p)
```
*fs.Tree (root)                     ← IS the session collection
  └── .Data = *rootState            ← sessions map, config, nextUID
       └── sessions[id] = *Session  ← owns Agent, context, cancel
Package functions (no Manager):
  session.NewRoot(cfg) → *fs.Tree
  session.Lookup(tree, id) → *Session
  session.All(tree) → []*Session
  session.CreateFromRoot(tree, args) → (id, error)
  session.KillFromRoot(tree, id)
  session.RenameFromRoot(tree, old, new)
  session.Shutdown(tree)
```
### SLOC after flattening
| Component | Lines |
|-----------|-------|
| agent + backend + toolsrv + session | 10,437 |
| **fs** + cmd/olliesrv | 7,925 |
| kde gui | 12,331 |
| **Total** | **32,538** |
Zero dead exported functions remain (verified via LSP + grep across all repos).
## Phase 12: Second Flattening & .meta Sidecar (Jul 30)

Another aggressive structural pass. Eliminated the entire script namespace
layer (s/, u/, x/), flattened `contrib/` into `data/`, and replaced
header-comment-based tool metadata with JSON sidecar files.

### Scripts removed (−1,100 lines)
- **s/{sh,b,bfg,bbg,ls,kill,cleanup}** — shell-based session frontends replaced
  by `ollie-9p rdwr generate/complete/route` and GUI frontends
- **u/{optimize,complete,cascade,escalate}** — utility scripts replaced by
  `generate`, `complete`, `route` files in the 9P namespace
- **x/{bd,prime,freeloader,task}** — internal plumbing scripts; PATH prepend
  (`prependOlliePath`) removed from toolsrv
- 9P session root trimmed to just `new` and `idx`; script-serving code removed
- `install-scripts` target now empty

### Tool metadata: .meta sidecar files
Header comment parsing (`ollie:prompt`, `ollie:tier`, `ollie:parallel read`,
`args_json:`) **eliminated entirely**. All tool metadata now lives in a JSON
sidecar file alongside the executable:

```
tools/
  file_edit       ← executable (any language)
  file_edit.meta  ← JSON metadata
```

```json
{
  "description": "Replace text in a file.",
  "prompt": "## file_edit\n\n...",
  "args": {"type":"object", ...},
  "tier": "cold",
  "readOnly": true
}
```

This decouples metadata from implementation language — compiled Go binaries,
Python scripts, and bash tools all use the same discovery mechanism.

### LSP tools ported to Go
The Python LSP bridge (`_lib/lsp/`) replaced with a pure Go implementation:

```
tools/
├── builtin/          ← in-process handlers (shell, reasoning, tool/skill registry)
└── lsp/              ← shared LSP client library
    └── cmd/          ← individual binaries (lsp_definition, lsp_hover, ...)
```

Architecture: each LSP tool binary embeds a bridge daemon (started on first
invocation via `--bridge` flag, persists via Unix socket, auto-exits after 5min
idle). No Python. No `_lib`. No external dependencies beyond the LSP servers
themselves (gopls, clangd, intelephense).

### Repo reorganization
- `contrib/{prompts,scripts,services,tools,skills,agents}` → `data/`
- `contrib/elisp` stays (Emacs frontend contribution)
- `tools/builtin/` — built-in tool handlers (moved from `tools/`, removed in Phase 18)
- `tools/lsp/` — Go LSP implementation (replaced `data/tools/_lib/lsp/`)

### Conditional variants for network transparency
Tools can declare multiple variants gated by host conditions (`binary`, `file`,
`os`, `arch`, `env`, `nenv`). First match determines the tool's schema and
executable. Enables one `.meta` to work across heterogeneous hosts — the
contract adapts to capabilities. Critical for `ollie-remote` deployments where
the remote host may have different tools, package managers, or init systems.

### Sudo credential broker
Tools declare `"sudo": true` in their `.meta` to require root privileges.
Dispatch becomes a two-gate chain: elevation approval → credential prompt → `sudo -S`.
Credentials are requested via `kdialog`/`zenity` on the local desktop, or forwarded
over SSH for remote execution. The tool itself has no knowledge of sudo — the
privilege wrapping is entirely in the dispatch layer.

## Principles That Emerged
1. **Filesystem-as-API** — everything is read/write on synthetic files. No
   custom protocols needed.
   Monorepo coordinates versions.
3. **Scripts over servers** — MCP servers removed early. Simple scripts won.
4. **Progressive disclosure** — lazy tool/skill loading keeps system prompts
   small until complexity is needed.
5. **Plan simplicity** — after 5 iterations, planning settled on one markdown
   checklist file per session.
6. **Single control plane — 9P for everything.
   Streaming via blocking reads. No polling.
7. **Security by default** — sandboxed execution with explicit elevation,
   per-session identity.
## Dead Ends and Reversals
| What | Why it was removed |
|------|-------------------|
| MCP servers | Too heavyweight; scripts are simpler and faster |
| TUI frontend | Replaced by KDE GUI and Emacs (ellie) |
| s/sh, s/bfg, s/bbg | Replaced by `generate`/`complete`/`route` 9P files |
| `pl/` planning namespace | Over-engineered; plan file is sufficient |
| Beads integration | External dependency for something a checklist file handles |
| SSH agent proxy | Attack surface not worth the convenience |
| Separate ollie-dbus daemon | Consolidated into olliesrv |
| FUSE mounts | Replaced by standalone 9P client binary |
| Tier routing (FAST/POWER) | Replaced by /route endpoint with real model discovery |
| call_tool / pipe | Replaced by native tool registry with tool_load |
| execute_code (multi-lang) | Simplified to shell-only; tool scripts handle language choice |
| chatwait | Reverted; acme tails chat directly |
| Header comment metadata | Replaced by .meta sidecar JSON; decouples metadata from language |
| `contrib/` directory | Renamed to `data/`; nothing was community-contributed |
| Script namespaces (s/, u/, x/) | Replaced by 9P request-response files |
| D-Bus adapter | 9P streaming is superior; no polling, no offset tracking, no frozen GUIs |
## Phase 13: 9P Streaming & D-Bus Removal (Jul 31)

The D-Bus adapter (`org.ollie.SessionManager`) is deleted. All clients now
use 9P exclusively via blocking reads for natural streaming.

Key changes:
- **`chat` file**: blocking read that delivers tokens as the agent produces
  them. Per-fid offset. Never EOF (blocks between turns). EOF only on kill.
- **`log` file**: replaces old `chat`. Returns last 64KB (sliding window).
  Non-blocking, tail-able via Qid.Vers.
- **`kill`/`.` ctl command**: kills session → EOF on chat readers.
- **KDE GUI**: rewritten from scratch. Uses `9p` (plan9port) via QProcess.
  Streaming via `readyReadStandardOutput`. Plain text tail (8KB).
  No ChatBlockModel, no D-Bus, no ThemeManager. ~250 lines total.
- **Kate plugin**: all D-Bus calls replaced with `9p` subprocess calls.
  Streaming chat + statewait via persistent QProcess.
- **KRunner**: uses `9p` for session listing and one-shot generation.
- **Acme frontend**: uses 9fans.net/go plan9/client directly. Blocking
  read loop on `chat`. No FUSE mount.
- **Plasmoid, tray**: deleted (not useful enough to maintain).
- **dbus/ package**: deleted (-771 lines from server).

The only remaining godbus usage: `org.freedesktop.Notifications` for
elevation prompts (desktop integration, not ollie protocol).

Lessons:
- 9P's request-response model gives natural streaming for free.
  Server delays Rread until data arrives. No polling needed.
- FUSE mounts do NOT support blocking reads (return EOF immediately).
  Clients must use the 9P protocol directly.
- Never bind a growing string to a QTextArea with word-wrap.
  QTextDocument relayout is O(n) on the full content.
- The `loadEarlier` scroll-triggered cascade was creating 500 delegates
  on startup. Bounded initial window + explicit scroll-back is correct.

## Phase 14: Multi-Agent Deepening — ID/Name Split & Empty Sessions (Aug 1)

The multi-agent data structures introduced during the Great Flattening
required further refinement to support true multi-agent sessions.

### ID/Name Split
Sessions and agents now have **two identities**:
- **`id`** (immutable UUIDv4, set at creation, never changes)
- **`name`** (mutable human-readable string, defaults to first UUID segment)

The sessions map is keyed by Name, not ID. Renaming a session or agent moves
the directory in the 9P namespace — `mv` on the session directory calls
`RenameFromRoot`, and `write` to the agent's `name` file calls `SetName`.

Both entities expose `id` and `name` files in their 9P directories.

### Two-Step Session Creation
Creation split into two phases:
1. `session/new` — write `name=<name>` to create an empty session (no agent yet)
2. `session/{name}/agent/new` — write `cwd=<dir> backend=<backend> model=<model> agent=<agent>` to create the agent

This enables creating sessions in advance and attaching agents later. Empty
sessions (`Core=nil`) are valid — they appear in `session/idx` with empty
fields and are fully killable/renameable.

### Multi-Agent Data Structures
- `session.Session` supports multiple agents (`Agents()` returns `[]*agent.Agent`)
- `Session.AgentLogs` maps agent IDs to `*AgentLog` instances
- Active agent is selectable, each agent has its own prompt/chat/state/log
- `session/idx` emits one line per agent, not one line per session

Key commits: 92 commits across Aug 1 touching session/, agent/, fs/, and kde/.

## Phase 15: 9P Declarative EDSL (Aug 2 — morning)

The most significant structural change to the 9P server since its inception:
**replaced the imperative filesystem with a declarative EDSL**.

### Before: Imperative
Every 9P operation (stat, walk, read, write, open, create, remove) was
hand-coded per node. The `fs/session/` sub-package contained ~3,100 lines
of manual tree-building code split across 6 files (`files.go`, `root.go`,
`perm.go`, `synth.go`, `create.go`, `persist.go`). Adding a new file meant
updating stat, readdir, open, and permission code paths.

### After: Declarative EDSL
The entire namespace is declared in a single `FsNodeDecl` tree in `fs/spec.go`:
```go
var treeSpec = Dir("/",
    Leaf("backends", 0444, Read(readBackends), GID("agent")),
    Leaf("eventwait", 0444, Read(readEventwait), Stream(blockEventwait), GID("agent")),
    Leaf("complete", 0666, Request(requestComplete), GID("agent")),
    // ...
    Dir("session", GID("agent"),
        Leaf("new", 0666, Request(requestSessionNew)),
        Leaf("idx", 0444, Read(readSessionIdx)),
        TemplateDir("{id}", listSessions),
    ),
)
```

`BuildTree()` in `fs/builder.go` walks the spec, validates invariants
(directories can't have leaf handlers, at most one blocking variant,
template nodes must have `List`, ownership inheritance), and produces
a fully-wired `*fs.Tree`.

### Package consolidation
- Entire `fs/session/` sub-package flattened into `fs/` — 6 files deleted
- `cmd/olliesrv/elevate_tree.go` — elevation tree moved to `fs/elevatefiles.go`
- `cmd/olliesrv/server.go` simplified from ~820+ lines to ~200 lines of thin
  9P protocol handling; all filesystem logic is now in `fs/`
- Handler files organized by scope: `rootfiles.go`, `sessionfiles.go`,
  `agentfiles.go`, `elevatefiles.go`, `procfiles.go`

### Net result
- **−3,152 lines** deleted, **+2,086 lines** added across 24 files
- Adding a file = adding one `Leaf()` call. No manual stat/readdir/write.
- Validation at `BuildTree` time catches structural errors before serving.
- Permission model inlined in the spec via `mode` and `GID()` — no separate `perm.go`.

## Phase 16: 9P Server Streamlining (Aug 2 — afternoon)

Following the EDSL conversion, the 9P protocol server itself
(`cmd/olliesrv/server.go`) received a focused cleanup over 13 commits:

### Fid management
- **Server owns the fid map**: validates newfids (not already in use, not stale),
  garbage-collects orphaned fids. No more leaking fids on client disconnect.
- **Store opened file on fid**: read/write use `f.entry` directly instead of
  re-resolving the path on every operation. Eliminates a class of TOCTOU bugs.

### Method extraction
- `attach` and `open` extracted from `*Server` — pass groups and log explicitly
- `GroupTable` extracted from `*Server` — permission checking uses bitmask OR
  instead of a boolean dance (`checkPermBits` → `hasPermBits`)
- `checkPerm` extracted to package-level, permissions resolved in `open`
- 9 `rootTree`-only methods extracted to package-level functions; switch
  statement replaced with handler map
- Three thin wrapper methods inlined (`readFile`, `writeFile`, `FileTree` alias)
- `handle` inlined into `Start`; renamed `Serve` → `Start`, `Shutdown` → `Kill`

### Result
- `server.go` reduced from ~820 to ~200 lines
- `main.go` simplified as elevation tree wiring moved to `fs/`
- Server is now a thin 9P2000 protocol translator, not a filesystem

## Phase 17: Eventwait Redesign & Chat Log Format (Aug 2)

### Global /eventwait
Replaced per-session event polling with a global event ring buffer and a
single `/eventwait` file. Events carry structured prefixes and delta descriptions:
- `S new session/<name>` — session created
- `S kill session/<name>` — session destroyed
- `S rename session/<old> session/<new>` — session renamed
- `A kill session/<name>/agent/<aid>` — agent killed

The event ring is a fixed-size circular buffer (100 slots) with a `sync.Cond`
for blocking reads. Frontends block on `/eventwait` and receive deltas since
their last known offset. No polling, no timer, no D-Bus.

Fixes along the way:
- Dangling pointer in `eventRing` cond initialization (caused panics under load)
- Double-unlock panic in `WaitEvent`
- Spurious `[[[end]]]` markers on reasoning blocks (reasoning events were
  silently suppressed from chat but still advanced the role-transition state)

### Chat log: [[[type]]]/[[[end]]] block format
The chat log switched from plain text to a structured block format:
```
[[[assistant:resp-abc123]]]
Hello! How can I help?

[[[end]]]
[[[tool:file_read]]]
File contents...

[[[end]]]
```

This enables reliable parsing of multi-turn, multi-role conversations
from the flat log. Each event type (`assistant`, `tool`, `call`, `retry`,
`error`, `usage`, `info`, `exec`) opens a block and `[[[end]]]` closes it.
`reasoning` and `reasoning_think` events are suppressed from the log
entirely — they're included in the LLM context but invisible to the user.

### KDE submodule evolution
The KDE frontend received extensive updates across both days:
- **StreamFsm refactor** — chat block FSM with proper fence-post handling
- **Responsive session tree** — selection/statewait stream management
- **Explicit agent selection** — per-agent statewait/chat streams,
  only `switchAgent` starts streams; session click clears agent selection
- **Emoji-labeled states** — visual state indicators (❌ for errors, ⚠️ for warnings)
- **Double-click rename** — inline renaming of agent nodes
- **Robust session loading** — loads sessions on startup, handles empty state
- **Context menu** — kill session, per-session operations
- **Eventwait-driven updates** — replaced timer-based polling with blocking `/eventwait`
- **Fence fix** — fixed a bug where code fences could eat surrounding content
- Removed D-Bus entirely; all communication via `9p` plan9port binary

### Updated Filesystem Layout (post-refactor)
```
/                          ← root (owned by system user)
├── backends               backends list
├── help                   help text
├── models                 model list (cached)
├── agents                 agent config list
├── ctl                    root control (invalidate, kill)
├── eventwait              global event stream (blocking read)
├── complete               request-response: code completion
├── generate               request-response: one-shot LLM generation
├── route                  request-response: model routing
├── elevate/
│   ├── policy             global elevation policy
│   └── pending/{sname}       pending elevation requests
└── session/
    ├── new                write key=value to create session
    ├── idx                session index (one line per agent)
    ├── {name}/
    │   ├── env            session environment variables
    │   ├── ctl            session control (kill, save, invalidate)
    │   ├── plan           session-scoped markdown checklist
    │   ├── id             immutable session UUID
    │   ├── name           mutable session name (write to rename)
    │   ├── elevate        per-session elevation policy
    │   └── agent/
    │       ├── new        write config to create agent (rdwr)
    │       └── {aname}/
    │           ├── prompt, prompt.prev, fifo.in, fifo.out
    │           ├── chat (streaming), log (64KB window)
    │           ├── state, statewait (blocking)
    │           ├── cfg, ctl, cwd
    │           ├── id (immutable), name (mutable)
    │           ├── offset, usage, cost, ctxsz, models
    │           ├── systemprompt, context, tail
    │           ├── tools, toolsrv.loaded, toolsrv.rev
    │           └── proc/{pid}
```

## Updated Current Topology
```
ollie/                      ← single Go module
├── agent/                  ← agent loop, history, hooks, commands
├── backend/                ← LLM providers (6 backends)
├── toolsrv/                ← tool server, registry, remote execution
├── session/                ← session lifecycle, config
├── fs/                     ← 9P filesystem (EDSL-declared, flat package)
│   ├── spec.go                 Namespace declaration (single source of truth)
│   ├── fsnode.go               FsNodeDecl type + Dir/Leaf/TemplateDir
│   ├── builder.go              BuildTree — spec → wired *Tree
│   ├── rootfiles.go            Root-level handlers
│   ├── sessionfiles.go         Session-level handlers
│   ├── agentfiles.go           Agent-level handlers
│   ├── elevatefiles.go         Elevation handlers
│   ├── procfiles.go            Process handlers
│   ├── lifecycle.go            Create/kill/rename + event ring
│   ├── newroot.go              NewRoot constructor
│   ├── persist.go              Session persistence
│   ├── types.go                Session/AgentLog types
│   ├── tree.go                 9P *Tree
│   ├── fs.go                   9P File/FileConfig
│   └── format.go               Event formatting
├── detach/                 ← background process management
├── elevate/                ← elevation broker
├── sandbox/                ← landrun sandbox config
├── env/                    ← environment helpers
├── log/                    ← structured logging
├── paths/                  ← XDG path resolution
├── mount/                  ← 9P FUSE mount client (network transparency)
├── cmd/                    ← binaries (olliesrv, ollie-9p, ollie-remote)
├── kde/                    ← KDE Plasma (submodule)
├── data/tools/             ← tool executables + .meta sidecar files
├── data/agents/            ← agent configs (JSON)
├── data/prompts/           ← system prompt fragments (markdown)
├── data/skills/            ← domain knowledge modules (markdown)
├── data/scripts/           ← ollie-remount, o CLI
├── data/services/          ← systemd, xdg-autostart
├── prompts/                ← embedded prompt templates
├── sandbox/                ← landrun sandbox profiles (YAML)
└── doc/                    ← documentation
```

## Dead Ends and Reversals (updated)
| What | Why it was removed |
|------|-------------------|
| MCP servers | Too heavyweight; scripts are simpler and faster |
| TUI frontend | Replaced by KDE GUI and Emacs (ellie) |
| s/sh, s/bfg, s/bbg | Replaced by `generate`/`complete`/`route` 9P files |
| `pl/` planning namespace | Over-engineered; plan file is sufficient |
| Beads integration | External dependency for something a checklist file handles |
| SSH agent proxy | Attack surface not worth the convenience |
| Separate ollie-dbus daemon | Consolidated into olliesrv |
| FUSE mounts | Replaced by standalone 9P client binary |
| Tier routing (FAST/POWER) | Replaced by /route endpoint with real model discovery |
| call_tool / pipe | Replaced by native tool registry with tool_load |
| execute_code (multi-lang) | Simplified to shell-only; tool scripts handle language choice |
| chatwait | Reverted; acme tails chat directly |
| Header comment metadata | Replaced by .meta sidecar JSON; decouples metadata from language |
| `contrib/` directory | Renamed to `data/`; nothing was community-contributed |
| Script namespaces (s/, u/, x/) | Replaced by 9P request-response files |
| D-Bus adapter | 9P streaming is superior; no polling, no offset tracking, no frozen GUIs |
| `fs/session/` sub-package | Flattened into `fs/` — no more sub-package indirection |
| `cmd/olliesrv/elevate_tree.go` | Elevation tree handlers moved to `fs/elevatefiles.go` |
| Per-session/agent timestamp IDs | Replaced by UUIDv4 — immutable `id` + mutable `name` |
| Single-step session creation | Split into `session/new` + `session/{name}/agent/new` |
| D-Bus-driven KDE frontends | Rewritten on pure 9P via plan9port `9p` binary |
| Plasmoid/tray KDE components | Deleted — not useful enough to maintain |
| Built-in tool handlers (shell, reasoning_think, tool_list, tool_load, tool_active) | Replaced by external scripts loaded via 9P `agent/{id}/tools` write |
| Go-compiled tool registry in `tools/builtin/` | Zero built-in tools — all tools are external scripts with `.meta` sidecars |
| Embedded tool scripts in `ollie-remote` | Remote uses `$OLLIE_TOOLS_PATH` from environment; tools deployed via tarball over SSH |
| SSH binary transfer (gzip+base64+hash-verify) | Replaced by simple `tar | ssh` tarball of `~/.config/ollie/` |
| `toolsrv.loaded` / `toolsrv.rev` 9P files | Removed — `tools` file handles both list and load |

## Phase 18: Zero Built-in Tools (Aug 2–3)

The most radical simplification yet: **zero tools compiled into the Go binary.** Every tool — including `shell` and `reasoning_think` — is now an external script loaded dynamically through the 9P filesystem.

### What changed

**Before**: `tools/builtin/builtins.go` returned four Go-compiled handlers (`shell`, `reasoning_think`, `tool_list`, `tool_load`, `tool_active`). These were hard-linked into every agent's tool definition, consuming LLM context slots even when unused. The tool registry was a two-tier system: built-in handlers (Go code, always available) + script tools (loaded on demand via `tool_load`).

**After**: `tools/builtin/` was removed entirely — `builtins.go` last returned `nil`. All tools are loaded through a single convergent path: writing the tool name to `session/{sname}/agent/{aname}/tools`. The `shell` and `reasoning_think` tools are now external scripts (`data/tools/shell`, `data/tools/reasoning_think`) with `.meta` sidecars — identical to `file_read`, `file_grep`, or any other tool.

### Unified tool loading

Tool loading follows a single path shared by both 9P writes and autoLoad initialization:

```
9P write to agent/{id}/tools
         │
         ▼
    session.loadTool(name, agent)
         │
         ├── Check disallow list
         └── RPC "tool_load" → ollie-remote process
```

The `tool_load` tool itself is a bash script (`data/tools/tool_load`) that writes the tool name to the 9P filesystem — it's a meta-tool that uses the same interface as every other operation.

### Agent config: `autoLoad`

Agent configuration files now declare which tools to load at startup via an `autoLoad` array:

```json
{
  "autoLoad": [
    "file_read",
    "file_edit",
    "file_glob",
    "file_grep",
    "file_write",
    "tool_load"
  ]
}
```

Each agent profile (default, copilot, explorer, librarian, navigator, taskmanager, theo) has its own `autoLoad` list tailored to its domain. The `allowTools` field provides a secondary security gate.

### Tool output format classification

Each tool's `.meta` can declare an `outputFormat` field specifying the source-fence language for its output:

```json
{
  "description": "...",
  "outputFormat": "markdown"
}
```

The agent loop propagates this to the chat log as the fence language (`[[[tool:file_read]]]\`\`\`markdown`), enabling syntax-highlighted output in the KDE GUI. Tools without `outputFormat` default to plaintext fences.

### Remote execution simplified

`ollie-remote` no longer embeds tool scripts or sandbox config:
- Removed `//go:embed all:tools` — tools come from `$OLLIE_TOOLS_PATH` in the environment
- Removed embedded sandbox `default.yaml` — sandbox config is provisioned alongside tools
- Removed `--tools` flag — uses `~/.config/ollie/tools` by default

### SSH bootstrap simplified

The SSH bootstrap sequence was cut from a complex multi-stage protocol to a single tarball transfer:

| Before | After |
|--------|-------|
| SHA256 hash verification | No hashing |
| gzip + base64 binary encoding | Raw `tar czf` tarball |
| `OLLIE_LOADER_START` / `OLLIE_LOADER_READY` handshake | Single `OLLIE_LISTEN_READY` signal |
| Separate `bootstrap.sh` template (88 lines) | Inline 10-line script |
| Per-binary transfer (ollie-remote + landrun) | Single `tar` of `~/.config/ollie/` |

The simplified bootstrap:

```sh
CACHE_DIR="${XDG_CACHE_HOME:-$HOME/.cache}/ollie"
mkdir -p "$CACHE_DIR"
tar xzf - -C "$CACHE_DIR"                    # receives tarball from local
export OLLIE_TOOLS_PATH="$CACHE_DIR/tools"
export PATH="$CACHE_DIR/bin:$PATH"
exec "$CACHE_DIR/bin/ollie-remote" serve --cwd <dir> --listen <sock>
```

### 9P namespace changes

- `agent/{id}/tools` — **read** lists all currently loaded tools (via the session's `toolsConn`), **write** loads a tool by name (via `session.loadTool`)
- Removed `agent/{id}/toolsrv.loaded` — subsumed by `tools` read
- Removed `agent/{id}/toolsrv.rev` — no longer needed
- Added root-level `/tools` — lists all discoverable tools on disk (global catalog)
- Removed `HandlerCtx.ToolReg` — the in-memory Go-level registry is no longer passed through 9P handlers

### Files removed

| File | Status |
|------|--------|
| `tools/builtin/` | Package removed entirely — all tool logic lives in external scripts |
| `tools/builtin/shell.go` | Deleted (now `data/tools/shell` + `data/tools/shell.meta`) |
| `tools/builtin/reasoning.go` | Deleted (now `data/tools/reasoning_think` + `data/tools/reasoning_think.meta`) |
| `tools/builtin/tool.go` | Deleted (`tool_list`/`tool_load`/`tool_active` replaced by 9P filesystem mechanism) |
| `toolsrv/bootstrap.sh` | Deleted (inline bootstrap replaces multi-stage protocol) |
| `cmd/ollie-remote/tools/.gitkeep` | Deleted (no embedded tools) |

### Lines of code

- **−530 lines** from the Go core (removed built-in handlers, embedded tools, bootstrap script)
- **+82 lines** for external tool scripts + `.meta` sidecars (`shell`, `reasoning_think`, `tool_load`)
- **Net**: tools system is smaller, simpler, and more uniform

### Why it matters

The zero-tools transition is the culmination of the architectural trajectory that began with the tool registry in Phase 10: **every facility is a file; nothing is hard-coded.**

1. **Uniformity** — There is no distinction between "core" tools and "script" tools. `shell`, `reasoning_think`, `file_read`, `file_write` — all loaded the same way, all have `.meta` sidecars, all called via the same RPC mechanism.

2. **Minimal core** — The Go binary no longer contains any tool logic. The agent loop is pure coordination: stream LLM → parse tool calls → dispatch to remote process → loop. Tool definitions, schemas, and execution are entirely external.

3. **Agent-choosable tools** — Each agent profile declares exactly which tools it needs. No agent pays the context-window cost of `shell` and `reasoning_think` unless its config asks for them.

4. **9P as the sole control plane** — Tool loading is now a file write. The same mechanism (`ollie-9p write ...`) that submits prompts and reads state also manages tool loading. No special RPC, no separate tool-manager protocol.

5. **Simpler remote deployment** — No embedded files, no binary transfer protocol, no hash verification. Provisioning a remote host is `tar czf ~/.config/ollie | ssh host tar xzf -`.

The result: the Go core is a 9P file server and an agent loop, nothing more. All behavior — every tool, skill, prompt, and configuration — lives in data files on disk. This is the Plan 9 ideal: a small, fixed kernel; all policy and capability in the filesystem.

### SLOC current (Aug 3)
| Component | Lines (excl. tests) |
|-----------|--------------------|
| agent + backend + toolsrv + session | 11,828 |
| fs + cmd/olliesrv | 4,838 |
| kde gui | 6,491 |
| **Total** | **23,157** |
Zero dead exported functions remain (verified via LSP + grep across all repos).

## Phase 19: EDSL Extraction & Structural Cleanup (Aug 3–4)

The declarative EDSL created in Phase 15 was extracted into a standalone,
reusable library. This wasn't just moving code — the EDSL itself was
**created from scratch on Aug 2** to replace ~3,100 lines of imperative
filesystem code. Within 48 hours it went from concept to extracted library.

The trajectory:
1. **Aug 2 morning**: Created the EDSL (Phase 15) — replaced imperative tree-building with declarative spec
2. **Aug 2 afternoon**: Streamlined the 9P server (Phase 16) — reduced to thin protocol handler
3. **Aug 3**: Extracted EDSL to `fsedsl/` as a reusable library
4. **Aug 4**: Structural cleanup — removed dead packages, improved `o` CLI

This rapid iteration exemplifies the project's development style: build the
right abstraction, validate it works, then extract it for reuse.

### fsedsl: Standalone Library

The generic EDSL types and builder logic extracted from `fs/` into `fsedsl/`:

```
fsedsl/
├── fsnode.go     FsNodeDecl[C], NodeOption[C], Dir/Leaf/Each constructors
├── builder.go    BuildTree — walks spec, validates, wires handlers
├── tree.go       Tree, File, FileConfig, FileTree types
├── option.go     Doc, Read, Write, Stream, BlockOnce, Request, etc.
└── README.md     Integration guide
```

The library is **protocol-agnostic** — it knows nothing about 9P. It produces
a `*Tree` structure that any protocol (9P, FUSE, HTTP, gRPC) can walk.
The `fs/` package becomes a thin adapter: type aliases + ollie-specific handlers.

Key design:
- Generic over context type `C` — ollie uses `HandlerCtx`, others can use anything
- Single `BuildTree[C]()` call validates and wires the entire namespace
- Validation at build time: no leaf handlers on directories, at most one blocking
  variant per file, template nodes require `List`, ownership inheritance

### fs/ Package Consolidation

After EDSL extraction, the fs package was restructured:

**Before** (scattered):
```
fs/
├── spec.go           namespace declaration
├── builder.go        tree builder (moved to fsedsl)
├── adapter.go        type aliases
├── session/          sub-package with 6 files
│   ├── files.go
│   ├── root.go
│   ├── perm.go
│   └── ...
└── elevatefiles.go
```

**After** (flat, by scope):
```
fs/
├── spec.go           single source of truth — entire namespace
├── fsnode.go         type aliases from fsedsl
├── builder.go        BuildTree wrapper
├── rootfiles.go      root-level handlers (backends, models, help, ctl)
├── sessionfiles.go   session-level handlers (env, ctl, name)
├── agentfiles.go     agent-level handlers (prompt, chat, state, tools)
├── elevatefiles.go   elevation broker handlers
└── procfiles.go      detached process handlers
```

No sub-packages. Handler files are organized by 9P tree scope.

### Session Package Refactors

Major restructuring of session lifecycle:

- **Session/Agent split** — `session.Session` manages lifecycle, `agent.Agent` manages conversation. Clear ownership boundaries.
- **Persistence to session/** — `fs/persist.go` removed, persistence logic moved to `session/` package
- **Manager eliminated** — Package-level registry replaces `Manager` struct. CRUD via `session.Lookup()`, `session.All()`, `session.CreateFromRoot()`, etc.
- **Remote promoted to session level** — Remote execution managed at session, not agent level

### Dead Code Removal

| Removed | Reason |
|---------|--------|
| `cmd/Ollie` (Acme frontend) | Use `o` CLI to compose acme front-end |
| `mount/` package | FUSE mount replaced by direct 9P client (`ollie-9p`) |
| `ollie-watchdog` script | No longer needed — 9P client doesn't have FUSE stale mount issues |
| `doc/experiments/` | Historical experiments moved to git history |
| `doc/help.md` | Now generated dynamically from `Doc()` strings in spec |

### o CLI Improvements

The terminal CLI (`data/scripts/o`) was simplified:

- **Removed auto-session/agent** — Context must be set explicitly via `export` or `eval $(o env ...)`. Eliminates slow `ollie-9p read session/idx` calls.
- **Path classification** — Paths categorized as root/session/agent level. Clear error messages: "chat requires agent context".
- **TUI percentage layout** — Fixed-size splits replaced with percentages (`-l 25%`). Adapts to terminal size.
- **Block marker filtering** — Chat stream piped through `grep -v '\[\[\[.*\]\]\]'` to hide internal markers.

### Documentation

- Removed outdated docs (`doc/experiments/`, stale `help.md`)
- Renamed doc files to lowercase-kebab-case
- Updated usage.md for new `o` CLI behavior
- Removed mount/ and watchdog references from architecture docs

### Gantt Update

The timeline now extends to Aug 4, 2026 — ~3.75 months of development.

### SLOC current (Aug 4)
| Component | Lines (excl. tests) |
|-----------|--------------------|
| agent + toolsrv + session | 9,077 |
| fs + cmd/olliesrv | 2,919 |
| fsedsl (extracted library) | 1,220 |
| kde gui | 4,162 |
| **Total (core)** | **17,378** |

Note: Aug 3 count included `backend/` (4,366 lines) and had different kde scope.
Comparable core (fs + agent + toolsrv + session + fsedsl + kde gui) shrank from ~23k to ~17k.