diff --git a/AGENTS.md b/AGENTS.md index 87513f0..8c28876 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -9,86 +9,85 @@ Project-level context for AI agents working in this repository. ## Project Overview Ollie is an AI agent runtime inspired by Plan 9: agent state and behaviors are exposed as files in a 9P namespace. Orchestration, scheduling, and UIs are external — shell scripts, editors, web apps. The core is minimal; capabilities come from composing scripts. ## Repository Layout -Monorepo with Git submodules. Each submodule has its own Go module (except `el` which is Elisp and `kde` which is C++/Qt). +Single Go module with one Git submodule (`kde/`). ``` ollie/ ← you are here -├── virtfs/ (Go module) Virtual filesystem EDSL (FsNodeDecl, BuildTree, Tree) -├── toolsrv/ (Go) Tool server client library (Conn, Dial) ├── cmd/ -│ ├── olliesrv/ (Go) 9P server binary +│ ├── olliesrv/ 9P server: sessions, agents, backends │ │ └── internal/ -│ │ ├── fs/ 9P namespace (everything in spec.go) -│ │ │ ├── spec.go Single source of truth — entire namespace + handlers -│ │ │ └── support.go Shared utilities (streamChat, stripMarkers, dispatch) -│ │ ├── agent/ Agent loop, history, hooks -│ │ ├── session/ Session lifecycle, persistence -│ │ ├── backend/ LLM backends (ollama, openai, anthropic, etc.) -│ │ └── bypass/ Sandbox bypass broker -│ ├── toolsrv/ (Go) Tool server binary (sandboxed execution) +│ │ ├── agent/ Agent loop, history, compaction, prompts +│ │ ├── backend/ LLM provider implementations +│ │ ├── bypass/ Sandbox bypass broker +│ │ ├── fs/ 9P namespace and handlers +│ │ ├── prompts/ System-prompt resolution +│ │ ├── session/ Session lifecycle and persistence +│ │ └── toolclient/ Local/remote toolsrv process management +│ ├── toolsrv/ Sandboxed tool execution server │ │ └── internal/ -│ │ ├── fs/ toolsrv 9P namespace (spec.go) -│ │ └── sandbox/ Landlock enforcement -│ ├── ollie-9p/ (Go) CLI client for the 9P namespace -│ └── ollie-remote/ (Go) Remote execution agent -├── tools/ (Go) Tool implementations: -│ └── lsp/ LSP bridge + cmd/ binaries -├── kde/ (C++/Qt6) KDE integration: GUI, Kate plugin, KRunner, tray -├── contrib/elisp (Elisp) Emacs frontend (ellie.el) -├── data/agents/ Agent config JSONs (loaded at runtime) -├── data/prompts/ System prompt templates (markdown) -├── data/tools/ Tool executables + .meta sidecar files -├── data/skills/ Domain knowledge modules (markdown) -├── sandbox/ Landlock sandbox config YAML -├── doc/ Architecture docs, usage guide -├── contrib/ Community scripts -└── experiments/ Trial notes +│ │ ├── exec/ Tool execution +│ │ ├── fs/ toolsrv 9P namespace +│ │ ├── registry/ Per-agent tool registry +│ │ └── sandbox/ Landlock configuration and enforcement +│ └── ollie-9p/ 9P client CLI +├── tools/ Compiled tool implementations +│ ├── codeintel/ Tree-sitter code-intelligence tools +│ ├── filetools/ Go file tools +│ ├── lsp/ LSP bridge and client tools +│ └── web/ Web-fetch tool +├── toolsrv/ toolsrv 9P client library and registry types +├── virtfs/ Virtual filesystem declaration EDSL +├── lib9p/ 9P protocol library and native client +├── env/, format/, log/, paths/ Shared Go packages +├── kde/ KDE GUI, Kate, KRunner, and KIO integration +├── contrib/elisp/ Emacs frontend (`ellie.el`) +├── data/agents/ Agent configuration JSON +├── data/prompts/ Prompt templates +├── data/tools/ Script tools and `.meta` files +├── data/skills/ Domain knowledge modules +├── data/scripts/ CLI and integration scripts +├── data/services/ User service files +├── cmd/toolsrv/internal/sandbox/ Installed sandbox configuration source +├── doc/ Architecture and usage documentation +└── experiments/ Experimental code ``` -### Canonical source for prompts/tools/skills -> **⛔ DO NOT edit `~/.config/ollie/` directly — it is an install target. All changes go in this repo.** +### Canonical source for prompts, tools, and skills -Prompt and tool files live in `data/`. The `just install-data` target copies them to `~/.config/ollie/`. -| Location | Purpose | Deployed by | -|----------|---------|-------------| -| `data/prompts/`, `data/tools/` | Canonical source for all prompts, tools, and skills | `just install-data` | -| `data/skills/` | Domain knowledge modules (markdown) | `just install-data` | -| `kde/` | KDE-specific tool scripts (`gui_*`) | `just install-kde` | +Do not edit `~/.config/ollie/` directly. It is an install target. Runtime data is copied from `data/` by `just install-data`; compiled tools are built into the same runtime tools directory by their build targets. The installed configuration also contains `backends.conf`, `sandbox.yaml`, agents, prompts, skills, scripts, and tools. ## Build System +The default `just` target builds and installs the KF6 variant. Build and install are separate phases. ```bash -# Build and install everything: -just -# Individual build targets: -just ninep # olliesrv + ollie-9p -just toolsrv # tool server binary -just core # core Go packages -just kde # KDE integration (cmake with ~/.local prefix) -just lsp-tools # LSP tool binaries -# Install targets: -just install-data # agents, prompts, tools, skills → ~/.config/ollie/ -just install-el # ellie.el → ~/.config/emacs/ellie/ -# Test: -just test # run all tests -just test-core # core Go tests -just test-9p # 9p integration tests -# Lifecycle: -just uninstall # remove all installed files -just clean # remove build artifacts -``` -Requires [just](https://github.com/casey/just): `cargo install just` -## Testing -```bash -# All Go tests (excludes cmd/ollie-remote which needs just to build): -go test $(go list ./... | grep -v cmd/ollie-remote) -# KDE has integration test scripts: -cd kde && ./test-e2e.sh -# ollie-remote requires the just build pipeline to resolve embedded deps: -just ollie-remote -just test -# or individually: -just test-remote +just # build + test + install (KF6) +just kf5 # build + install the KF5/Qt5 variant +just build # build core, 9P, lib9p, tools, and KDE +just core # go build ./... +just ninep # olliesrv and ollie-9p +just lib9p # native lib9p shared library and header +just toolsrv # toolsrv binary +just code-tools # Tree-sitter code-intelligence tools +just file-tools # Go file tools +just lsp-tools # LSP tools +just tools-web # web_fetch +just kde # KDE KF6 integration +just kde-kf5 # KDE KF5 integration +just install-data # runtime configuration, prompts, skills, scripts, and tools +just install-el # Emacs frontend +just test # core and lib9p tests +just test-core # cmd/olliesrv, cmd/toolsrv, shared packages, and toolsrv tests +just test-9p # lib9p tests +just test-remote # build-check toolsrv remote package +just clean # remove build artifacts ``` +Requires [just](https://github.com/casey/just): `cargo install just`. -**Note on `cmd/ollie-remote`**: This package uses `//go:embed sandbox/default.yaml` and `//go:embed all:tools`. The embedded files are not in the source tree — `just ollie-remote` copies them in before building then cleans up. Running `go test ./...` will fail on this package. Use `just test-remote` or `go test $(go list ./... | grep -v cmd/ollie-remote)` instead. +## Testing +The supported test entry points are: +```bash +just test +just test-core +just test-9p +``` +For direct Go testing, use the packages covered by `just test-core` and `just test-9p`. The current repository does not contain the former `cmd/ollie-remote` package; remote tool execution is implemented by `toolclient.SpawnRemote` and the remote toolsrv deployment path. ## Language & Conventions - **Go** (root module): Go 1.25+, standard library preferred, minimal dependencies. @@ -100,36 +99,46 @@ just test-remote - Tool scripts: emit structured output (`STATUS=ok`, `STATUS=error`). Image/LSP tools return JSON content blocks. - Prompts: markdown, concise, example-driven. Follow the pattern in existing `tools-*.md` files. ## Architecture (key concepts) -1. **One integration surface**: 9P filesystem. Sessions at `session/{sname}/agent/{aname}/`. All control via `ctl` (rdwr: write command, read response). Tools, skills, memory on physical filesystem via env vars. -2. **Two-process model**: Each session runs an `olliesrv` (the 9P namespace, agent loop, backends) and a per-session `toolsrv` (sandboxed tool execution). They communicate over a Unix socket using 9P. The toolsrv is spawned by olliesrv at session creation and has its own namespace (`/proc/new`, `/ctl`, `/tools`, etc.). -3. **Agent loop** (`cmd/olliesrv/internal/agent/loop.go`): Streaming LLM call → parse tool calls → dispatch to toolsrv → loop until no more tool calls or max steps. -4. **Tool dispatch**: ALL tools (including `reasoning_think`, `file_read`, `shell`) execute through the per-session toolsrv. The agent holds a `*toolsrv.Conn` (9P client) and calls `CallTool(name, args)` which writes to `proc/new` and reads the result. Tool registry is per-agent, keyed by agent ID in the protocol. -5. **Sandbox** (`cmd/toolsrv/internal/sandbox/`): Landlock-based. Config in `sandbox/*.yaml` defines filesystem access per profile. Escape via bypass broker (`cmd/olliesrv/internal/bypass/`). -6. **Backends** (`cmd/olliesrv/internal/backend/`): Ollama, OpenAI-compatible, Anthropic, Copilot, Kiro, Gemini. Selectable per-session. -7. **Prompts assembled at runtime**: Agent JSON `prompt` array specifies which prompt files to concatenate. The base system prompt is embedded in the binary and always prepended. -8. **9P namespace declared via virtfs EDSL**: The entire filesystem is a single recursive `FsNodeDecl` tree in `cmd/olliesrv/internal/fs/spec.go`, built by `virtfs.BuildTree()`. Every handler is an inline closure — no indirection. Same pattern in `cmd/toolsrv/internal/fs/spec.go`. +1. **One integration surface**: `olliesrv` exposes sessions and agents through a 9P2000 filesystem. Frontends include `o`, `ollie-9p`, KDE, Kate, Emacs, and scripts. Reads from `chat`, `statewait`, `eventwait`, and `feed` provide blocking/event-driven synchronization. +2. **Session and tool processes**: Each session owns an agent runtime in `olliesrv` and a separate `toolsrv` process. They communicate over an authenticated Unix socket using 9P. `toolclient` can respawn local toolsrv processes and can deploy/start toolsrv remotely over SSH with socket forwarding. +3. **Agent loop** (`cmd/olliesrv/internal/agent/loop.go` and `turn.go`): Stream an LLM response, parse native or text tool calls, execute tools, update history, and repeat until a final response, cancellation, or a configured limit. It includes transient retries, context-overflow compaction, error/stall/replan controls, and tool-result caching. +4. **Dynamic tools**: Tools are external executables described by `.meta` files. `toolsrv` owns discovery and per-agent registries. `olliesrv` refreshes the registry, injects common dispatch flags (`bypass`, `timeout`, `sandbox`, `background`), and calls tools through `toolsrv.Conn`. +5. **Parallel and background dispatch**: Non-conflicting tool calls in one turn run concurrently. Metadata scopes schedule `read`, `write`, and global operations; shell-like global operations serialize. Any tool may run in the background, producing a process ID whose output is injected when the process changes or exits. +6. **Sandbox** (`cmd/toolsrv/internal/sandbox/`): Landlock/landrun policies control filesystem and network access. The installed `sandbox.yaml` is sourced from `cmd/toolsrv/internal/sandbox/sandbox.yaml`. The bypass broker provides policy-controlled escape requests, approval, persistence, and rate limiting. +7. **Backends** (`cmd/olliesrv/internal/backend/`): Supported names are `ollama`, `openai`, `openrouter`, `anthropic`, `copilot`, `kiro`, and `gemini`. Configuration is read from `~/.config/ollie/backends.conf`; environment variables are fallback inputs. +8. **Prompt assembly**: `cmd/olliesrv/internal/prompts/system_prompt.md` is embedded as the default system prompt. An agent's `systemPrompt` can override it with a filesystem path. `prompt` and `userPrompts` entries resolve files, expand environment variables, and support legacy `!command` entries. The runtime combines system, environment, agent, and tool sections. +9. **Context management**: History tracks messages, usage, costs, cache statistics, and structured task state. Cold/warm/hot result tiers and automatic compaction preserve recent context while summarizing older material. +10. **Sub-agents**: `subagent_spawn` creates a transient child session with an independent runtime and context. The child receives a one-time parent-history snapshot and returns only its final reply. Parent/child IDs are retained for tracing; concurrent children are supported. +11. **9P namespace declaration**: `cmd/olliesrv/internal/fs/spec.go` and `cmd/toolsrv/internal/fs/spec.go` declare the namespaces with the `virtfs` EDSL. `virtfs.BuildTree()` validates and constructs the trees; handlers are defined inline in the specs. ## Key Files | What | Where | |------|-------| | 9P namespace (olliesrv) | `cmd/olliesrv/internal/fs/spec.go` | | 9P namespace (toolsrv) | `cmd/toolsrv/internal/fs/spec.go` | | virtfs EDSL | `virtfs/decl.go`, `virtfs/builder.go` | -| Agent loop | `cmd/olliesrv/internal/agent/loop.go` | +| Agent loop and dispatch | `cmd/olliesrv/internal/agent/loop.go`, `turn.go` | +| Runtime and prompt assembly | `cmd/olliesrv/internal/agent/runtime.go`, `prompt_resolver.go` | | Tool server binary | `cmd/toolsrv/` | -| Tool server client | `toolsrv/client9p.go` | -| Remote execution | `cmd/ollie-remote/` | +| Tool server client and process lifecycle | `toolsrv/client9p.go`, `cmd/olliesrv/internal/toolclient/` | +| Remote tool execution | `cmd/olliesrv/internal/toolclient/spawn.go` | | Sandbox enforcement | `cmd/toolsrv/internal/sandbox/` | -| Session management | `cmd/olliesrv/internal/session/` | -| System prompt template | Embedded in binary (`cmd/olliesrv/internal/prompts/`) | +| Bypass broker | `cmd/olliesrv/internal/bypass/` | +| Session management and persistence | `cmd/olliesrv/internal/session/` | +| Embedded default system prompt | `cmd/olliesrv/internal/prompts/system_prompt.md` | | Agent configs | `data/agents/*.json` | +| Backend configuration | `data/backends.conf`, `~/.config/ollie/backends.conf` | +| Compiled tools | `tools/codeintel/`, `tools/filetools/`, `tools/lsp/`, `tools/web/` | | KDE GUI | `kde/gui/` | | Kate plugin | `kde/kate/` | ## Environment -Config lives in `~/.config/ollie/env`. Key variables: -- `OLLIE_BACKEND` — default backend (ollama, openai, anthropic, copilot, kiro) -- `OLLIE_MODEL` — default model -- Tools live at `$XDG_CONFIG_HOME/ollie/tools` (default: `~/.config/ollie/tools`) -- Memory lives at `$XDG_DATA_HOME/ollie/optmem` (default: `~/.local/share/ollie/optmem`) +Runtime configuration is read from `~/.config/ollie/` (or `$XDG_CONFIG_HOME/ollie/`). The backend configuration file is `backends.conf`, not `env`. +- `backend = ...` in `backends.conf` selects the default backend; a session `backend=...` can override it. +- `OLLIE_BACKEND` is a fallback when no configured backend or session backend is selected. +- `OLLIE_MODEL` is a legacy/environment model fallback; configured backend sections can set `model = ...`. +- The runtime path helpers honor `XDG_CONFIG_HOME`, `XDG_DATA_HOME`, and `XDG_RUNTIME_DIR`. +- The current `justfile` install targets use `~/.config/ollie`, `~/.local/share/ollie`, and `~/.local/bin` directly; non-default XDG locations therefore require adjusting the install targets or copying the installed files manually. +- Runtime tools are discovered from `$XDG_CONFIG_HOME/ollie/tools` (default: `~/.config/ollie/tools`). +- OptMem runtime data is under `$XDG_DATA_HOME/ollie/optmem` (default: `~/.local/share/ollie/optmem`). ## Adding a new tool ### Script-based tool (Python/Bash)