diff --git a/AGENTS.md b/AGENTS.md index 6c263ce..2cc0818 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -9,7 +9,10 @@ ollie/ ← you are here ├── agent/ (Go) Agent loop, history, hooks, commands ├── toolsrv/ (Go) Tool server, registry, sandboxed execution ├── session/ (Go) Session lifecycle, config, persistence -├── fs/session/ (Go) 9P filesystem tree +├── fs/ (Go) 9P filesystem tree (EDSL-declared, flat package) +│ ├── spec.go Single source of truth — entire namespace declared here +│ ├── fsnode.go Core FsNodeDecl type + Dir/Leaf/TemplateDir constructors +│ └── builder.go BuildTree — walks spec, validates, wires handlers ├── cmd/ (Go) Binaries (olliesrv, ollie-9p, ollie-remote) ├── tools/ (Go) Tool implementations: │ ├── builtin/ Built-in handlers (shell, reasoning, tool/skill registry) @@ -87,6 +90,7 @@ just test-remote 4. **Sandbox** (`sandbox/`): Landlock-based. Config in `sandbox/*.yaml` defines filesystem access per profile. 5. **Backends** (`backend/`): Ollama, OpenAI-compatible, Anthropic, Copilot, Kiro, Gemini, CodeWhisperer. Selectable per-session. 6. **Prompts assembled at runtime**: Agent JSON `prompt` array specifies which prompt files to concatenate. Static prompt files can be included directly; the base system prompt is embedded in the binary and always prepended. +7. **9P namespace declared via EDSL**: The entire filesystem is a single recursive `FsNodeDecl` tree in `fs/spec.go`, built by `BuildTree()` in `fs/builder.go`. The `fs/` package is flat — no sub-package — handler files (`rootfiles.go`, `sessionfiles.go`, `agentfiles.go`) are organized by scope. See `doc/edsl.md` for the full reference. ## Key Files | What | Where | |------|-------| @@ -94,7 +98,8 @@ just test-remote | Tool server | `toolsrv/server.go` | | Remote execution | `toolsrv/remote.go`, `cmd/ollie-remote/` | | Sandbox enforcement | `sandbox/` | -| 9P filesystem | `fs/` | +| 9P filesystem (EDSL spec) | `fs/spec.go` | +| 9P filesystem (builder) | `fs/builder.go` | | Session management | `session/session.go` | | System prompt template | Embedded in binary | | Agent configs | `agents/*.json` | diff --git a/doc/ARCHITECTURE.md b/doc/ARCHITECTURE.md index 88c741f..be2cf35 100644 --- a/doc/ARCHITECTURE.md +++ b/doc/ARCHITECTURE.md @@ -2,21 +2,37 @@ ## Philosophy ollie's design philosophy is Emacs: a small, extensible core. The Go runtime (`agent.Agent`) defines what an agent *is* — an event loop, a backend connection, a message history, and three built-in primitives (`shell`, tool registry, skill registry). Everything else — orchestration, scheduling, workflows, UIs — is pushed out to the surrounding environment. The primary integration philosophy is Plan 9's "everything is a file." `olliesrv` exposes agent state and behaviors as files in a 9P namespace. Any program that can read and write files can drive an agent: shell scripts, editors, web apps, cron, containers. -All clients communicate via 9P. Desktop notifications use D-Bus directly (org.freedesktop.Notifications) for elevation prompts only. +All clients communicate via 9P exclusively. Desktop notifications use D-Bus directly (org.freedesktop.Notifications) for elevation prompts only — this is the last remaining D-Bus dependency. Design principles: - **Small extensible core.** The agent runtime is minimal; capabilities come from composing external scripts. - **Three built-in primitives.** `shell`, tool registry (`tool_load`/`tool_list`/`tool_active`), and skill registry (`skill_load`/`skill_list`/`skill_active`) are the only tools compiled into the core. Everything else is a script. - **One integration path: 9P filesystem. Streaming via blocking reads. No polling. +- **9P namespace declared via EDSL.** The entire filesystem is a single recursive `FsNodeDecl` tree in `fs/spec.go`, validated and built by `BuildTree()` in `fs/builder.go`. Adding a file means adding a `Leaf()` call — no manual stat/readdir/write wiring. - **No framework lock-in.** Frontends are decoupled via whichever interface they prefer; the core doesn't know or care. ## Repository Structure -Single Go module (`ollie`) with two Git submodules for decoupled frontends: +Single Go module (`ollie`) with one Git submodule (`kde/`) for the KDE frontend: ``` ollie/ ├── agent/ Agent loop, history, hooks, prompt resolution, commands ├── backend/ LLM providers (Anthropic, OpenAI, Ollama, Gemini, Copilot, CodeWhisperer) ├── toolsrv/ Tool server: sandboxed execution, tool registry, skill management ├── session/ Session lifecycle (config, creation, persistence) -├── fs/session/ 9P filesystem tree for sessions +├── fs/ 9P filesystem: EDSL spec + handlers (flat package) +│ ├── spec.go Namespace declaration (single source of truth) +│ ├── fsnode.go FsNodeDecl type + Dir/Leaf/TemplateDir constructors +│ ├── builder.go BuildTree — spec -> *Tree wiring +│ ├── rootfiles.go Root-level handlers (backends, models, eventwait, complete, generate, route) +│ ├── sessionfiles.go Session-level handlers (env, ctl, plan, agent/, ...) +│ ├── agentfiles.go Agent-level handlers (prompt, chat, state, cfg, ...) +│ ├── elevatefiles.go Elevation handlers (policy, pending) +│ ├── procfiles.go Process handlers +│ ├── lifecycle.go Session create/kill/rename/shutdown + event ring +│ ├── newroot.go NewRoot — tree construction + persistence restore +│ ├── persist.go Session persistence to disk +│ ├── types.go Session/AgentLog types +│ ├── tree.go 9P *Tree (from fs package) +│ ├── fs.go 9P File/FileConfig implementations +│ └── format.go Event formatting helpers ├── detach/ Background process management (ring buffer, signal) ├── elevate/ Elevation broker (privilege escalation daemon) ├── sandbox/ Landlock sandbox config YAML @@ -28,13 +44,14 @@ ollie/ │ ├── olliesrv/ 9P server │ ├── ollie-9p/ 9P client │ └── ollie-remote/ Remote execution server -├── kde/ KDE integration (submodule) — plasmoid, GUI, Kate plugin, KRunner, tray -├── agents/ Agent config JSONs (default, coding, orchestrator, worker, ...) -├── prompts/ Prompt templates (markdown) -├── tools/ Tool scripts (file_read, lsp_*, memory_*, subagent_*, ...) -├── skills/ Domain knowledge files (markdown) -├── scripts/ User-facing scripts organized by namespace (s/, u/, x/) -├── sandbox/ Sandbox config YAML files +├── kde/ KDE integration (submodule) — standalone GUI, Kate plugin, KRunner +├── data/agents/ Agent config JSONs (default, coding, orchestrator, worker, ...) +├── data/prompts/ Prompt templates (markdown) +├── data/tools/ Tool scripts + .meta sidecar files +├── data/skills/ Domain knowledge modules (markdown) +├── data/scripts/ Helper scripts (ollie-remount, ollie-watchdog) +├── data/services/ Systemd/xdg-autostart service files +├── prompts/ Embedded prompt templates (compiled into binary) └── doc/ Documentation ``` ## System Overview @@ -43,15 +60,17 @@ flowchart TB subgraph Frontends["Frontends"] ACME["acme (Plan 9)"] ELLIE["ellie (Emacs)"] - KDE["KDE GUI / Plasmoid"] + KDE["KDE GUI / Kate / KRunner"] + SH["s/sh (terminal)"] + HTTP["curl / scripts"] end subgraph Integration["Integration Layer"] direction LR - P9["9P Filesystem"] + P9["9P Filesystem\n(ollie-9p client)"] end subgraph Server["olliesrv"] direction TB - NS["9P Namespace\nsession/ · backends · models"] + NS["9P Namespace\nEDSL-declared in fs/spec.go"] end subgraph Core["Agent Engine (per session)"] LOOP["Agent Loop\n(agent/loop.go)"] @@ -65,15 +84,13 @@ flowchart TB GEMINI["Gemini"] CW["CodeWhisperer"] end - TUI --> P9 + SH --> P9 ACME --> P9 ELLIE --> P9 - KDE --> DBUS - WEB --> DBUS + KDE --> P9 + HTTP --> P9 P9 --> NS - DBUS --> DBA NS --> LOOP - DBA --> LOOP LOOP <--> SESS LOOP --> TRV LOOP <--> Backends @@ -172,61 +189,84 @@ Agent configs declare prompts as a JSON array of shell commands. Each command is `olliesrv` is the central daemon. It implements a 9P2000 file server that exposes the entire agent namespace: ### Namespace Layout ``` -session/ -├── new write key=value to create session -├── idx session index (tab-separated) -├── ls exec to list sessions -├── kill exec to kill a session -├── sh interactive shell -├── b batch one-shot (create, submit, wait, print, kill) -├── bfg batch foreground (submit to existing, wait, print) -├── bbg batch background (submit, return path) -├── cleanup kill idle sessions -├── {sid}/ -│ ├── plan session-scoped markdown checklist -│ ├── env session environment -│ └── agent/ -│ └── {aid}/ -│ ├── cfg session config (key=value) -│ ├── ctl control commands (stop, kill, compact, model, ...) -│ ├── state current state (idle/thinking/calling) -│ ├── statewait blocking read until state changes -│ ├── chat full chat history (streamable) -│ ├── offset byte offset after last user prompt -│ ├── prompt write to submit a prompt -│ ├── prompt.prev last submitted prompt -│ ├── fifo.in queue a prompt (FIFO input) -│ ├── fifo.out pop queued prompt (FIFO output) -│ ├── context rendered context window -│ ├── systemprompt rendered system prompt -│ ├── usage token usage stats -│ ├── cost estimated cost -│ ├── ctxsz context size -│ ├── models available models -│ ├── tools tool management (read all, write name to load) -│ ├── toolsrv.loaded currently loaded tools -│ ├── toolsrv.rev revision counter -│ ├── tail exec helper -│ └── proc/ detached background processes +/ ← root (owned by system user) +├── backends backends list +├── help help text +├── models model list (cached) +├── agents agent config list +├── ctl root control (invalidate, kill) +├── eventwait global event stream (blocking read) +├── complete request-response: code completion +├── generate request-response: one-shot LLM generation +├── route request-response: model routing +├── elevate/ +│ ├── policy global elevation policy +│ └── pending/{id} pending elevation requests (approve/deny/persist) +└── session/ + ├── new write key=value to create session + ├── idx session index (one line per agent) + ├── {name}/ + │ ├── env session environment variables + │ ├── ctl session control (kill, save, invalidate) + │ ├── plan session-scoped markdown checklist + │ ├── id immutable session UUID + │ ├── name mutable session name (write to rename) + │ ├── elevate per-session elevation policy + │ └── agent/ + │ ├── new write config to create agent (rdwr) + │ └── {aid}/ + │ ├── prompt submit a prompt + │ ├── prompt.prev last submitted prompt + │ ├── fifo.in queue a prompt + │ ├── fifo.out pop queued prompt + │ ├── chat streaming chat log (blocking read) + │ ├── log last 64KB of chat (non-blocking) + │ ├── state current state (idle/thinking/calling) + │ ├── statewait blocking read until state changes + │ ├── cfg agent config (key=value) + │ ├── ctl agent control (stop, compact, ...) + │ ├── cwd working directory + │ ├── id immutable agent UUID + │ ├── name mutable agent name + │ ├── offset byte offset after last user prompt + │ ├── usage token usage stats + │ ├── cost estimated cost + │ ├── ctxsz context size + │ ├── models available models + │ ├── systemprompt rendered system prompt + │ ├── context rendered context window + │ ├── tail exec helper for tailing chat + │ ├── tools tool management (read all, write name to load) + │ ├── toolsrv.loaded currently loaded tools + │ ├── toolsrv.rev revision counter + │ └── proc/{pid} detached process output ``` ### 9P Interaction Model All agent interaction uses the `ollie-9p` client, which auto-discovers the server via `$NAMESPACE` and identifies the caller via `$OLLIE_UNAME`. The 9P filesystem exposes session state, control, and inter-agent communication as files: - Sessions are directories under `session/` +- Session creation is a two-step process: `session/new` (name only) then `session/{name}/agent/new` (full config) +- Sessions and agents have **immutable UUIDs** (`id` file) and **mutable human-readable names** (`name` file); writing to `name` renames the entry - Prompts, state, chat history, and config are readable/writable files +- Streaming via blocking reads: `chat`, `statewait`, `eventwait` block the 9P read until data arrives - Permission enforcement uses 9P user principals -### Permission Model (`fs/perm.go`) -All permissions are declared in a single registry (`Perms`). Key design: +- The entire namespace is declared as a single EDSL tree in `fs/spec.go` — see `doc/edsl.md` +### Permission Model (`fs/spec.go`) +Permissions are declared inline in the EDSL spec via `mode` and `GID()` options. There is no separate permission registry — the spec is the source of truth: - `prompt` is mode `0666` — group+other can write, owner (the agent) cannot - `chat` is mode `0444` — read-only for everyone - `ctl` is mode `0666` — anyone can send control commands - `plan` is mode `0666` — readable by peer agents -### Session Management (`fs/session/`) +- `eventwait` is `0444` — blocking read for global event stream +- `session/new` is `0666` with `Request` handler — anyone can create sessions +- Ownership inherits: empty `UID`/`GID` values inherit from parent node, all the way up to root +### Session Management (`fs/`) The `*fs.Tree` IS the session collection. Package functions manage lifecycle: -- `NewRoot(cfg)` — create the root tree +- `NewRoot(cfg)` — create the root tree from the EDSL spec (`fs/spec.go` → `BuildTree` → wired `*Tree`) - `Lookup(tree, id)` — find a session by ID - `All(tree)` — list all sessions -- `CreateFromRoot(tree, args)` — create a new session +- `CreateFromRoot(tree, args)` — create a new session (two-step: session name, then agent creation) - `KillFromRoot(tree, id)` — kill a session - `RenameFromRoot(tree, old, new)` — rename a session -- `Shutdown(tree)` — clean shutdown -The adapter is best-effort: if no session bus is available (e.g., headless/container), `olliesrv` continues functioning as a pure 9P server. +- `Shutdown(tree)` — clean shutdown (interrupt all, wait for idle, persist, close) + +Event streaming uses a global ring buffer (`eventRing`) with a global `/eventwait` file. Events carry prefixes (`S` for session, `A` for agent) with structured descriptions (`new`, `kill`, `rename`). Frontends block on `/eventwait` and receive delta events without polling. diff --git a/doc/CORE.md b/doc/CORE.md index 5b3f54e..c918bb8 100644 --- a/doc/CORE.md +++ b/doc/CORE.md @@ -57,17 +57,22 @@ flowchart TB SESS["session.go\nSession, Config"] end - subgraph FsSession["fs/session/ package"] - TREE["tree.go\n*fs.Tree"] - ROOT["root.go\nNewRoot()"] - CREATE["create.go\nCreateFromRoot()"] - FILES["files.go\nfile operations"] - PERM["perm.go\npermission registry"] + subgraph Fs["fs/ package"] + TREE["tree.go\n*Tree"] + NEWROOT["newroot.go\nNewRoot()"] + LIFECYCLE["lifecycle.go\nCreate/Kill/Rename"] + SESSFILES["sessionfiles.go\nsession file handlers"] + AGENTFILES["agentfiles.go\nagent file handlers"] + ROOTFILES["rootfiles.go\nroot file handlers"] + ELEVATEFILES["elevatefiles.go\nelevation handlers"] + PROCFILES["procfiles.go\nprocess handlers"] PERSIST["persist.go\nsession persistence"] - FSGO["fs.go\n9P filesystem"] + FS_GO["fs.go\n9P File/FileConfig"] FORMAT["format.go\nevent formatting"] - SYNTH["synth.go\nsynthetic files"] TYPES["types.go\nSession struct"] + SPEC["spec.go\nEDSL namespace declaration"] + BUILDER["builder.go\nBuildTree"] + FSNODE["fsnode.go\nFsNodeDecl"] end subgraph Support["Supporting Packages"] @@ -82,7 +87,7 @@ flowchart TB Agent --> Backend Agent --> Toolsrv Agent --> Session - FsSession --> Session + Fs --> Session Toolsrv --> SANDBOX Toolsrv --> DETACH Toolsrv --> ELEVATE diff --git a/doc/EVOLUTION.md b/doc/EVOLUTION.md index 4206905..8217690 100644 --- a/doc/EVOLUTION.md +++ b/doc/EVOLUTION.md @@ -350,34 +350,6 @@ Credentials are requested via `kdialog`/`zenity` on the local desktop, or forwar over SSH for remote execution. The tool itself has no knowledge of sudo — the privilege wrapping is entirely in the dispatch layer. -## Current Topology -``` -ollie/ ← single Go module -├── agent/ ← agent loop, history, hooks, commands -├── backend/ ← LLM providers (6 backends) -├── toolsrv/ ← tool server, registry, remote execution -├── session/ ← session lifecycle, config -├── **fs**/ ← 9P filesystem tree -├── dbus/ ← D-Bus adapter -├── detach/ ← background process management -├── elevate/ ← elevation broker -├── sandbox/ ← landrun sandbox config -├── env/ ← environment helpers -├── log/ ← structured logging -├── paths/ ← XDG path resolution -├── mount/ ← 9P FUSE mount client (network transparency) -├── cmd/ ← binaries (olliesrv, ollie-9p, ollie-remote) -├── kde/ ← KDE Plasma (submodule) -├── data/tools/ ← tool executables + .meta sidecar files -├── data/agents/ ← agent configs (JSON) -├── data/prompts/ ← system prompt fragments (markdown) -├── data/skills/ ← domain knowledge modules (markdown) -├── data/scripts/ ← ollie-remount, ollie-watchdog -├── data/services/ ← systemd, xdg-autostart -├── prompts/ ← embedded prompt templates -├── sandbox/ ← landrun sandbox profiles (YAML) -└── doc/ ← documentation -``` ## Principles That Emerged 1. **Filesystem-as-API** — everything is read/write on synthetic files. No custom protocols needed. @@ -444,3 +416,270 @@ Lessons: QTextDocument relayout is O(n) on the full content. - The `loadEarlier` scroll-triggered cascade was creating 500 delegates on startup. Bounded initial window + explicit scroll-back is correct. + +## Phase 14: Multi-Agent Deepening — ID/Name Split & Empty Sessions (Aug 1) + +The multi-agent data structures introduced during the Great Flattening +required further refinement to support true multi-agent sessions. + +### ID/Name Split +Sessions and agents now have **two identities**: +- **`id`** (immutable UUIDv4, set at creation, never changes) +- **`name`** (mutable human-readable string, defaults to first UUID segment) + +The sessions map is keyed by Name, not ID. Renaming a session or agent moves +the directory in the 9P namespace — `mv` on the session directory calls +`RenameFromRoot`, and `write` to the agent's `name` file calls `SetName`. + +Both entities expose `id` and `name` files in their 9P directories. + +### Two-Step Session Creation +Creation split into two phases: +1. `session/new` — write `name=` to create an empty session (no agent yet) +2. `session/{name}/agent/new` — write `cwd= backend= model= agent=` to create the agent + +This enables creating sessions in advance and attaching agents later. Empty +sessions (`Core=nil`) are valid — they appear in `session/idx` with empty +fields and are fully killable/renameable. + +### Multi-Agent Data Structures +- `session.Session` supports multiple agents (`Agents()` returns `[]*agent.Agent`) +- `Session.AgentLogs` maps agent IDs to `*AgentLog` instances +- Active agent is selectable, each agent has its own prompt/chat/state/log +- `session/idx` emits one line per agent, not one line per session + +Key commits: 92 commits across Aug 1 touching session/, agent/, fs/, and kde/. + +## Phase 15: 9P Declarative EDSL (Aug 2 — morning) + +The most significant structural change to the 9P server since its inception: +**replaced the imperative filesystem with a declarative EDSL**. + +### Before: Imperative +Every 9P operation (stat, walk, read, write, open, create, remove) was +hand-coded per node. The `fs/session/` sub-package contained ~3,100 lines +of manual tree-building code split across 6 files (`files.go`, `root.go`, +`perm.go`, `synth.go`, `create.go`, `persist.go`). Adding a new file meant +updating stat, readdir, open, and permission code paths. + +### After: Declarative EDSL +The entire namespace is declared in a single `FsNodeDecl` tree in `fs/spec.go`: +```go +var treeSpec = Dir("/", + Leaf("backends", 0444, Read(readBackends), GID("agent")), + Leaf("eventwait", 0444, Read(readEventwait), Stream(blockEventwait), GID("agent")), + Leaf("complete", 0666, Request(requestComplete), GID("agent")), + // ... + Dir("session", GID("agent"), + Leaf("new", 0666, Request(requestSessionNew)), + Leaf("idx", 0444, Read(readSessionIdx)), + TemplateDir("{id}", listSessions), + ), +) +``` + +`BuildTree()` in `fs/builder.go` walks the spec, validates invariants +(directories can't have leaf handlers, at most one blocking variant, +template nodes must have `List`, ownership inheritance), and produces +a fully-wired `*fs.Tree`. + +### Package consolidation +- Entire `fs/session/` sub-package flattened into `fs/` — 6 files deleted +- `cmd/olliesrv/elevate_tree.go` — elevation tree moved to `fs/elevatefiles.go` +- `cmd/olliesrv/server.go` simplified from ~820+ lines to ~200 lines of thin + 9P protocol handling; all filesystem logic is now in `fs/` +- Handler files organized by scope: `rootfiles.go`, `sessionfiles.go`, + `agentfiles.go`, `elevatefiles.go`, `procfiles.go` + +### Net result +- **−3,152 lines** deleted, **+2,086 lines** added across 24 files +- Adding a file = adding one `Leaf()` call. No manual stat/readdir/write. +- Validation at `BuildTree` time catches structural errors before serving. +- Permission model inlined in the spec via `mode` and `GID()` — no separate `perm.go`. + +## Phase 16: 9P Server Streamlining (Aug 2 — afternoon) + +Following the EDSL conversion, the 9P protocol server itself +(`cmd/olliesrv/server.go`) received a focused cleanup over 13 commits: + +### Fid management +- **Server owns the fid map**: validates newfids (not already in use, not stale), + garbage-collects orphaned fids. No more leaking fids on client disconnect. +- **Store opened file on fid**: read/write use `f.entry` directly instead of + re-resolving the path on every operation. Eliminates a class of TOCTOU bugs. + +### Method extraction +- `attach` and `open` extracted from `*Server` — pass groups and log explicitly +- `GroupTable` extracted from `*Server` — permission checking uses bitmask OR + instead of a boolean dance (`checkPermBits` → `hasPermBits`) +- `checkPerm` extracted to package-level, permissions resolved in `open` +- 9 `rootTree`-only methods extracted to package-level functions; switch + statement replaced with handler map +- Three thin wrapper methods inlined (`readFile`, `writeFile`, `FileTree` alias) +- `handle` inlined into `Start`; renamed `Serve` → `Start`, `Shutdown` → `Kill` + +### Result +- `server.go` reduced from ~820 to ~200 lines +- `main.go` simplified as elevation tree wiring moved to `fs/` +- Server is now a thin 9P2000 protocol translator, not a filesystem + +## Phase 17: Eventwait Redesign & Chat Log Format (Aug 2) + +### Global /eventwait +Replaced per-session event polling with a global event ring buffer and a +single `/eventwait` file. Events carry structured prefixes and delta descriptions: +- `S new session/` — session created +- `S kill session/` — session destroyed +- `S rename session/ session/` — session renamed +- `A kill session//agent/` — agent killed + +The event ring is a fixed-size circular buffer (100 slots) with a `sync.Cond` +for blocking reads. Frontends block on `/eventwait` and receive deltas since +their last known offset. No polling, no timer, no D-Bus. + +Fixes along the way: +- Dangling pointer in `eventRing` cond initialization (caused panics under load) +- Double-unlock panic in `WaitEvent` +- Spurious `[[[end]]]` markers on reasoning blocks (reasoning events were + silently suppressed from chat but still advanced the role-transition state) + +### Chat log: [[[type]]]/[[[end]]] block format +The chat log switched from plain text to a structured block format: +``` +[[[assistant:resp-abc123]]] +Hello! How can I help? + +[[[end]]] +[[[tool:file_read]]] +File contents... + +[[[end]]] +``` + +This enables reliable parsing of multi-turn, multi-role conversations +from the flat log. Each event type (`assistant`, `tool`, `call`, `retry`, +`error`, `usage`, `info`, `exec`) opens a block and `[[[end]]]` closes it. +`reasoning` and `reasoning_think` events are suppressed from the log +entirely — they're included in the LLM context but invisible to the user. + +### KDE submodule evolution +The KDE frontend received extensive updates across both days: +- **StreamFsm refactor** — chat block FSM with proper fence-post handling +- **Responsive session tree** — selection/statewait stream management +- **Explicit agent selection** — per-agent statewait/chat streams, + only `switchAgent` starts streams; session click clears agent selection +- **Emoji-labeled states** — visual state indicators (❌ for errors, ⚠️ for warnings) +- **Double-click rename** — inline renaming of agent nodes +- **Robust session loading** — loads sessions on startup, handles empty state +- **Context menu** — kill session, per-session operations +- **Eventwait-driven updates** — replaced timer-based polling with blocking `/eventwait` +- **Fence fix** — fixed a bug where code fences could eat surrounding content +- Removed D-Bus entirely; all communication via `9p` plan9port binary + +### Updated Filesystem Layout (post-refactor) +``` +/ ← root (owned by system user) +├── backends backends list +├── help help text +├── models model list (cached) +├── agents agent config list +├── ctl root control (invalidate, kill) +├── eventwait global event stream (blocking read) +├── complete request-response: code completion +├── generate request-response: one-shot LLM generation +├── route request-response: model routing +├── elevate/ +│ ├── policy global elevation policy +│ └── pending/{id} pending elevation requests +└── session/ + ├── new write key=value to create session + ├── idx session index (one line per agent) + ├── {name}/ + │ ├── env session environment variables + │ ├── ctl session control (kill, save, invalidate) + │ ├── plan session-scoped markdown checklist + │ ├── id immutable session UUID + │ ├── name mutable session name (write to rename) + │ ├── elevate per-session elevation policy + │ └── agent/ + │ ├── new write config to create agent (rdwr) + │ └── {aid}/ + │ ├── prompt, prompt.prev, fifo.in, fifo.out + │ ├── chat (streaming), log (64KB window) + │ ├── state, statewait (blocking) + │ ├── cfg, ctl, cwd + │ ├── id (immutable), name (mutable) + │ ├── offset, usage, cost, ctxsz, models + │ ├── systemprompt, context, tail + │ ├── tools, toolsrv.loaded, toolsrv.rev + │ └── proc/{pid} +``` + +## Updated Current Topology +``` +ollie/ ← single Go module +├── agent/ ← agent loop, history, hooks, commands +├── backend/ ← LLM providers (6 backends) +├── toolsrv/ ← tool server, registry, remote execution +├── session/ ← session lifecycle, config +├── fs/ ← 9P filesystem (EDSL-declared, flat package) +│ ├── spec.go Namespace declaration (single source of truth) +│ ├── fsnode.go FsNodeDecl type + Dir/Leaf/TemplateDir +│ ├── builder.go BuildTree — spec → wired *Tree +│ ├── rootfiles.go Root-level handlers +│ ├── sessionfiles.go Session-level handlers +│ ├── agentfiles.go Agent-level handlers +│ ├── elevatefiles.go Elevation handlers +│ ├── procfiles.go Process handlers +│ ├── lifecycle.go Create/kill/rename + event ring +│ ├── newroot.go NewRoot constructor +│ ├── persist.go Session persistence +│ ├── types.go Session/AgentLog types +│ ├── tree.go 9P *Tree +│ ├── fs.go 9P File/FileConfig +│ └── format.go Event formatting +├── detach/ ← background process management +├── elevate/ ← elevation broker +├── sandbox/ ← landrun sandbox config +├── env/ ← environment helpers +├── log/ ← structured logging +├── paths/ ← XDG path resolution +├── mount/ ← 9P FUSE mount client (network transparency) +├── cmd/ ← binaries (olliesrv, ollie-9p, ollie-remote) +├── kde/ ← KDE Plasma (submodule) +├── data/tools/ ← tool executables + .meta sidecar files +├── data/agents/ ← agent configs (JSON) +├── data/prompts/ ← system prompt fragments (markdown) +├── data/skills/ ← domain knowledge modules (markdown) +├── data/scripts/ ← ollie-remount, ollie-watchdog +├── data/services/ ← systemd, xdg-autostart +├── prompts/ ← embedded prompt templates +├── sandbox/ ← landrun sandbox profiles (YAML) +└── doc/ ← documentation +``` + +## Dead Ends and Reversals (updated) +| What | Why it was removed | +|------|-------------------| +| MCP servers | Too heavyweight; scripts are simpler and faster | +| TUI frontend | Replaced by KDE GUI and Emacs (ellie) | +| s/sh, s/bfg, s/bbg | Replaced by `generate`/`complete`/`route` 9P files | +| `pl/` planning namespace | Over-engineered; plan file is sufficient | +| Beads integration | External dependency for something a checklist file handles | +| SSH agent proxy | Attack surface not worth the convenience | +| Separate ollie-dbus daemon | Consolidated into olliesrv | +| FUSE mounts | Replaced by standalone 9P client binary | +| Tier routing (FAST/POWER) | Replaced by /route endpoint with real model discovery | +| call_tool / pipe | Replaced by native tool registry with tool_load | +| execute_code (multi-lang) | Simplified to shell-only; tool scripts handle language choice | +| chatwait | Reverted; acme tails chat directly | +| Header comment metadata | Replaced by .meta sidecar JSON; decouples metadata from language | +| `contrib/` directory | Renamed to `data/`; nothing was community-contributed | +| Script namespaces (s/, u/, x/) | Replaced by 9P request-response files | +| D-Bus adapter | 9P streaming is superior; no polling, no offset tracking, no frozen GUIs | +| `fs/session/` sub-package | Flattened into `fs/` — no more sub-package indirection | +| `cmd/olliesrv/elevate_tree.go` | Elevation tree handlers moved to `fs/elevatefiles.go` | +| Per-session/agent timestamp IDs | Replaced by UUIDv4 — immutable `id` + mutable `name` | +| Single-step session creation | Split into `session/new` + `session/{name}/agent/new` | +| D-Bus-driven KDE frontends | Rewritten on pure 9P via plan9port `9p` binary | +| Plasmoid/tray KDE components | Deleted — not useful enough to maintain | diff --git a/doc/MULTI_AGENT.md b/doc/MULTI_AGENT.md index 688e7cd..f4b2066 100644 --- a/doc/MULTI_AGENT.md +++ b/doc/MULTI_AGENT.md @@ -6,18 +6,26 @@ ordinary shell scripting against a mounted filesystem. ## The Namespace -Every session lives under `/session/{id}/`: +Every session lives under `/session/{name}/` where `{name}` is the session's +**mutable human-readable name** (not the immutable UUID — see `id` and `name` +files). Session directories are keyed by Name; renaming a session via writing +to its `name` file moves the entire directory subtree. | Path | Purpose | |------|---------| -| `/session/new` | Create a session (write `name`, `backend`, `model`, `agent`, `cwd`) | -| `/session/{id}/plan` | Session-scoped markdown checklist | -| `/session/{id}/env` | Session environment variables | -| `/session/{id}/agent/{aid}/prompt` | Submit a prompt (dispatched async on close) | -| `/session/{id}/agent/{aid}/chat` | Cumulative conversation history (append-only read) | -| `/session/{id}/agent/{aid}/cfg` | Agent config (KV): state, backend, model, agent, cwd, params | -| `/session/{id}/agent/{aid}/statewait` | **Blocking read** — returns when state changes | -| `/session/{id}/agent/{aid}/ctl` | Control commands: `stop`, `kill`, `compact`, `rn ` | +| `/session/new` | Create an empty session (write `name=...`, read back name) | +| `/session/{name}/agent/new` | Create an agent within a session (write `cwd=... backend=...`) | +| `/session/{name}/plan` | Session-scoped markdown checklist | +| `/session/{name}/env` | Session environment variables | +| `/session/{name}/id` | Immutable session UUID | +| `/session/{name}/name` | Mutable session name (write to rename) | +| `/session/{name}/agent/{aid}/prompt` | Submit a prompt | +| `/session/{name}/agent/{aid}/chat` | Streaming chat log (blocking read) | +| `/session/{name}/agent/{aid}/cfg` | Agent config KV: backend, model, agent, cwd, params | +| `/session/{name}/agent/{aid}/statewait` | **Blocking read** — returns when state changes | +| `/session/{name}/agent/{aid}/ctl` | Control commands: `stop`, `kill`, `compact`, `rn ` | +| `/session/{name}/agent/{aid}/id` | Immutable agent UUID | +| `/session/{name}/agent/{aid}/name` | Mutable agent name (write to rename) | Global request-response files provide stateless operations without sessions: @@ -32,8 +40,8 @@ Global namespaces are shared across all sessions: | Path | Purpose | |------|---------| | `/session/idx` | Session index (id, state, cwd, backend, model) | -| `/session/{id}/plan` | Shared session-scoped checklist | -| `/session/{id}/agent/{aid}/chat` | Full chat history (searchable) | +| `/session/{name}/plan` | Shared session-scoped checklist | +| `/session/{name}/agent/{aid}/chat` | Full chat history (searchable) | ## Spawning Agents @@ -43,9 +51,21 @@ From within an agent session, use `subagent_spawn`: subagent_spawn [-name NAME] [-agent AGENT] [-backend BACKEND] [-model MODEL] [-cwd DIR] ``` -This creates a child session, writes the prompt, and returns the session path -(e.g. `session/my-session`). Child sessions carry the parent session ID in their name, -so the spawning tree is recoverable from session IDs alone. +This creates a child session + agent atomically, writes the prompt, and returns +the session path (e.g. `session/my-session`). Child sessions carry the parent +session ID in their name, so the spawning tree is recoverable from session IDs alone. + +### Explicit Two-Step Creation + +Session creation can be split into two phases for advanced orchestration +scenarios: + +1. **Create empty session**: `echo "name=my-session" | ollie-9p rdwr session/new` +2. **Add agent**: `echo "cwd=/home/user agent=default backend=openai model=gpt-4o" | ollie-9p rdwr session/my-session/agent/new` + +Empty sessions (`Core=nil`) are valid — they appear in `session/idx`, have +no running agent, and can be killed or renamed. This enables pre-provisioning +session directories before the agent environment is ready. ## Coordination Patterns @@ -64,7 +84,7 @@ flowchart TB ``` A root agent reasons about a task and delegates subtasks to workers. Workers -report back via their chat history; the conductor reads `/session/{id}/agent/{aid}/chat` after +report back via their chat history; the conductor reads `/session/{name}/agent/{aid}/chat` after `statewait` unblocks. ```sh diff --git a/doc/edsl.md b/doc/edsl.md index 8dbd9c3..ff1d955 100644 --- a/doc/edsl.md +++ b/doc/edsl.md @@ -76,7 +76,7 @@ Leaf("chat", 0444, Read(readAgentChat), Stream(blockAgentChat)) | `Stream(fn)` | `func(HandlerCtx, ctx, base) (data, base, err)` | `d.Stream` | | `BlockOnce(fn)` | `func(HandlerCtx, ctx, base) (data, base, err)` | `d.BlockOnce` | | `Request(fn)` | `func(HandlerCtx, []byte) ([]byte, error)` | `d.Request` | -| `Create(fn)` | `func(HandlerCtx, string) error` | `d.Create` | +| `CreateFile(fn)` | `func(HandlerCtx, string) error` | `d.Create` | | `Remove(fn)` | `func(HandlerCtx) error` | `d.Remove` | | `Stat(fn)` | `func() os.FileInfo` | `d.Stat` | | `UID(s)` | string | `d.UID` |