lkn
/
ollie
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
ollie/AGENTS.md

257 lines
19 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# AGENTS.md
Project-level context for AI agents working in this repository.
> **⚠️ IMPORTANT: All work must be done in THIS source directory (`~/src/ollie/`).**
> Never edit files under `~/.config/ollie/` — that is an install target.
> Changes made there are overwritten on the next `make install-data`.
> Edit source files here, then run `make` to build and install.
## Project Overview
Ollie is an AI agent runtime inspired by Plan 9: agent state and behaviors are exposed as files in a 9P namespace. Orchestration, scheduling, and UIs are external — shell scripts, editors, web apps. The core is minimal; capabilities come from composing scripts.
## Repository Layout
Single Go module. KDE integration is part of the repository under `kde/`.
```
ollie/ ← you are here
├── cmd/
│ ├── olliesrv/ 9P server: sessions, agents, backends
│ │ └── internal/
│ │ ├── agent/ Agent loop, history, compaction, prompts
│ │ ├── backend/ LLM provider implementations
│ │ ├── bypass/ Sandbox bypass broker
│ │ ├── fs/ 9P namespace and handlers
│ │ ├── prompts/ System-prompt resolution
│ │ ├── session/ Session lifecycle and persistence
│ │ └── toolclient/ Local/remote toolsrv process management
│ ├── toolsrv/ Sandboxed tool execution server
│ │ └── internal/
│ │ ├── exec/ Tool execution
│ │ ├── registry/ Per-agent tool registry
│ │ ├── sandbox/ Landlock configuration and enforcement
│ │ └── server/ Namespace specification and process state
│ └── ollie-9p/ 9P client CLI
├── tools/ Compiled tool implementations
│ ├── codeintel/ Tree-sitter code-intelligence tools
│ ├── filetools/ Go file tools
│ ├── lsp/ LSP bridge and client tools
│ └── web/ Web-fetch tool
├── toolsrv/ toolsrv 9P client library and registry types
├── virtfs/ Virtual filesystem declaration EDSL
├── lib9p/ 9P protocol library and native client
├── env/, format/, log/, paths/ Shared Go packages
├── kde/ KDE GUI, Kate, KRunner, and KIO integration
├── contrib/elisp/ Emacs frontend (`ellie.el`)
├── data/agents/ Agent configuration JSON
├── data/prompts/ Prompt templates
├── data/tools/ Script tools and `.meta` files
├── data/skills/ Domain knowledge modules
├── data/scripts/ CLI and integration scripts
├── data/services/ User service files
├── cmd/toolsrv/internal/sandbox/ Installed sandbox configuration source
├── doc/ Architecture and usage documentation
└── experiments/ Experimental code
```
### Canonical source for prompts, tools, and skills
Do not edit `~/.config/ollie/` directly. It is an install target. Runtime data is copied from `data/` by `make install-data`; compiled tools are built into the same runtime tools directory by their build targets. The installed configuration also contains `backends.conf`, `sandbox.yaml`, agents, prompts, skills, scripts, and tools.
## Build System
The default `make` target builds, tests, and installs. Build and install are separate phases.
```bash
make # build + test + install
make build # build core, 9P, client, tools, and KDE
make core # go build ./...
make ninep # olliesrv and ollie-9p
make client # native lib9p shared library and header
make tools # compiled tools (code-intel, file, LSP, web)
make kde # KDE KF6 integration
make install-data # runtime configuration, prompts, skills, scripts, and tools
make test # core and lib9p tests
make test-core # cmd/olliesrv, cmd/toolsrv, shared packages
make test-9p # lib9p tests
make clean # remove build artifacts
```
Requires GNU Make.
## Testing
The supported test entry points are:
```bash
make test
make test-core
make test-9p
```
For direct Go testing, use the packages covered by `make test-core` and `make test-9p`.
## Language & Conventions
- **Go** (root module): Go 1.25+, standard library preferred, minimal dependencies.
- **C++20/Qt6/KF6** (kde): CMake build.
- **Elisp** (el): single file `ellie.el`.
- **Tool scripts**: Python 3, Bash, or compiled binaries. Must be executable. Metadata lives in a `.meta` sidecar JSON file (see `data/tools/*.meta`).
### Code style
- Go: `gofmt`, short variable names, error returns (no panics), table-driven tests.
- Tool scripts: emit structured output (`STATUS=ok`, `STATUS=error`). Image/LSP tools return JSON content blocks.
- Prompts: markdown, concise, example-driven. Follow the pattern in existing `tools-*.md` files.
## Stability & compatibility
Ollie is **experimental and unstable** software. The features and API are approaching stability but are not there yet. Optimize for a clean, minimal codebase over preserving existing behavior.
- **Do not add backward-compatibility code.** No shims, deprecation aliases, legacy fallbacks, dual-format parsers, or "keep the old path working" branches. When something changes, change it fully and delete the old form.
- **Prefer subtraction.** Removing code is a feature. If a rename, refactor, or new design lets you delete the old thing, delete it — don't leave both.
- **Break callers freely.** Renaming a field, changing a wire format, or altering a `ctl` verb is fine; update all call sites in the same change. There are no external consumers to protect.
- **Extreme minimalism.** No pointless indirection, no defensive code for cases that can't happen, no configuration knobs "just in case."
This is a standing preference, not a per-task instruction. Apply it without asking.
## Architecture (key concepts)
1. **One integration surface**: `olliesrv` exposes sessions and agents through a 9P2000 filesystem. Frontends include `o`, `ollie-9p`, KDE, Kate, Emacs, and scripts. Reads from `chat`, `statewait`, `eventwait`, and `feed` provide blocking/event-driven synchronization.
2. **Session and tool processes**: Each session owns an agent runtime in `olliesrv` and a separate `toolsrv` process. They communicate over an authenticated Unix socket using 9P. `toolclient` can respawn local toolsrv processes and can deploy/start toolsrv remotely over SSH with socket forwarding.
3. **Agent loop** (`cmd/olliesrv/internal/agent/`): The agent package is organized by concern:
- `loop.go`: Main loop — stream LLM, execute tools, update history
- `turn.go`: Submit entry point, turn orchestration
- `dispatch.go`: Tool execution, batching, conflict detection
- `history.go`: Message history, usage tracking
- `compact.go`: Context compaction, cold summarization
- `cache.go`: Tool result caching with file staleness detection
- `retry.go`: Error tracking, transient retry logic
- `state.go`, `chat.go`, `peer.go`, `subagent.go`: Agent state and coordination
4. **Dynamic tools**: Tools are external executables described by `.meta` files. `toolsrv` owns discovery and per-agent registries. `olliesrv` refreshes the registry, injects common dispatch flags (`bypass`, `timeout`, `sandbox`, `background`), and calls tools through `toolsrv.Conn`.
5. **Parallel and background dispatch**: Non-conflicting tool calls in one turn run concurrently. Metadata scopes schedule `read`, `write`, and global operations; shell-like global operations serialize. Any tool may run in the background, producing a process ID whose output is injected when the process changes or exits.
6. **Sandbox** (`cmd/toolsrv/internal/sandbox/`): Native Landlock policies control filesystem access in a short-lived child helper. The installed `sandbox.yaml` is sourced from `cmd/toolsrv/internal/sandbox/sandbox.yaml`. The toolsrv namespace and process state live under `cmd/toolsrv/p9.go` and `cmd/toolsrv/internal/server/`. The bypass broker provides policy-controlled escape requests, approval, persistence, and rate limiting.
7. **Backends** (`cmd/olliesrv/internal/backend/`): Supported names are `ollama`, `openai`, `openrouter`, `anthropic`, `copilot`, `kiro`, and `gemini`. Configuration is read from `~/.config/ollie/backends.conf`; environment variables are fallback inputs.
8. **Prompt assembly**: `cmd/olliesrv/internal/prompts/system_prompt.md` is embedded as the default system prompt. An agent's `systemPrompt` can override it with a filesystem path. `prompt` and `userPrompts` entries resolve files, expand environment variables, and support legacy `!command` entries. The runtime combines system, environment, agent, and tool sections.
9. **Context management**: History tracks messages, usage, costs, cache statistics, and structured task state. Cold/warm/hot result tiers and automatic compaction preserve recent context while summarizing older material.
10. **Sub-agents**: `subagent_spawn` creates a transient child session with an independent runtime and context. The child receives a one-time parent-history snapshot and returns only its final reply. Parent/child IDs are retained for tracing; concurrent children are supported.
11. **Peers**: Persistent agents in the same session can be linked via `peeradd`. Links are bidirectional. Agents communicate by writing to `peer/{name}`, which delivers to the target's prompt handler. Only declared peers can be messaged — the `peer/` directory is the access control surface.
12. **9P namespace declaration**: `cmd/olliesrv/internal/fs/spec.go` declares the olliesrv namespace. The toolsrv namespace is declared by `cmd/toolsrv/p9.go` using `cmd/toolsrv/internal/server.Spec`; process state and handlers are in `cmd/toolsrv/internal/server/`. Both use the `virtfs` EDSL and `virtfs.BuildTree()`.
## Where to Start
Entry points for understanding different parts of the codebase:
**Agent execution**
1. `cmd/olliesrv/internal/agent/loop.go` — Main loop: stream LLM, execute tools, update history
2. `cmd/olliesrv/internal/agent/turn.go` — Turn orchestration; `Submit` is the entry point
3. `cmd/olliesrv/internal/agent/dispatch.go` — Tool batching and conflict detection
**9P namespace**
1. `cmd/olliesrv/internal/fs/spec.go` — All handlers inline, no chasing
2. `cmd/toolsrv/p9.go` — toolsrv namespace; `internal/server/proc.go` owns process state
**Backend integration**
1. `cmd/olliesrv/internal/backend/backend.go` — Interface and shared types
2. Pick a concrete backend (e.g., `anthropic.go`, `openai.go`) to see implementation
**Context management**
1. `cmd/olliesrv/internal/agent/history.go` — Message storage, token tracking
2. `cmd/olliesrv/internal/agent/compact.go` — Compaction logic, cold summarization
**Tool discovery**
1. `cmd/olliesrv/internal/agent/tool_match.go` — Semantic matching
2. `cmd/olliesrv/internal/agent/runtime.go` — Preamble and tool schema assembly
**Sandbox and execution**
1. `cmd/toolsrv/internal/sandbox/` — Landlock policy configuration
2. `cmd/toolsrv/internal/exec/exec.go` — Tool execution wrapper
3. `cmd/toolsrv/internal/server/proc.go` — Process lifecycle
## Key Files
| What | Where |
|------|-------|
| 9P namespace (olliesrv) | `cmd/olliesrv/internal/fs/spec.go` |
| 9P namespace (toolsrv) | `cmd/toolsrv/p9.go`, `cmd/toolsrv/internal/server/server.go` |
| virtfs EDSL | `virtfs/decl.go`, `virtfs/builder.go` |
| Agent core and identity | `cmd/olliesrv/internal/agent/agent.go` |
| Agent loop | `cmd/olliesrv/internal/agent/loop.go` |
| Turn orchestration | `cmd/olliesrv/internal/agent/turn.go` |
| Tool dispatch and batching | `cmd/olliesrv/internal/agent/dispatch.go` |
| Message history | `cmd/olliesrv/internal/agent/history.go` |
| Context compaction | `cmd/olliesrv/internal/agent/compact.go` |
| Tool result caching | `cmd/olliesrv/internal/agent/cache.go` |
| Error retry logic | `cmd/olliesrv/internal/agent/retry.go` |
| Runtime and prompt assembly | `cmd/olliesrv/internal/agent/runtime.go`, `prompt_resolver.go` |
| Text-based tool parsing | `cmd/olliesrv/internal/agent/text_parse.go` |
| Semantic tool/skill matching | `cmd/olliesrv/internal/agent/tool_match.go`, `skill_match.go` |
| Chat output formatting | `cmd/olliesrv/internal/agent/chatlog.go` |
| Local summarization | `cmd/olliesrv/internal/agent/local_summary.go` |
| Workflow classification | `cmd/olliesrv/internal/agent/workflow.go` |
| Tool server binary | `cmd/toolsrv/` |
| Tool server client | `cmd/olliesrv/internal/toolclient/` |
| Process lifecycle | `cmd/toolsrv/internal/server/proc.go` |
| Sandbox enforcement | `cmd/toolsrv/internal/sandbox/` |
| Bypass broker | `cmd/olliesrv/internal/bypass/` |
| Session management | `cmd/olliesrv/internal/session/` |
| Embedded system prompt | `cmd/olliesrv/internal/prompts/system_prompt.md` |
| Agent configs | `data/agents/*.json` |
| Backend configuration | `data/backends.conf`, `~/.config/ollie/backends.conf` |
| Compiled tools | `tools/codeintel/`, `tools/filetools/`, `tools/lsp/`, `tools/web/` |
| KDE GUI | `kde/gui/` |
| Kate plugin | `kde/kate/` |
## Environment
Runtime configuration is read from `~/.config/ollie/` (or `$XDG_CONFIG_HOME/ollie/`). The backend configuration file is `backends.conf`, not `env`.
- `backend = ...` in `backends.conf` selects the default backend; a session `backend=...` can override it.
- `OLLIE_BACKEND` is a fallback when no configured backend or session backend is selected.
- `OLLIE_MODEL` is a legacy/environment model fallback; configured backend sections can set `model = ...`.
- The runtime path helpers honor `XDG_CONFIG_HOME`, `XDG_DATA_HOME`, and `XDG_RUNTIME_DIR`.
- The Makefile install targets use `~/.config/ollie`, `~/.local/share/ollie`, and `~/.local/bin` directly; non-default XDG locations require adjusting the install targets or copying the installed files manually.
- Runtime tools are discovered from `$XDG_CONFIG_HOME/ollie/tools` (default: `~/.config/ollie/tools`).
- OptMem runtime data is under `$XDG_DATA_HOME/ollie/optmem` (default: `~/.local/share/ollie/optmem`).
## Adding a new tool
### Script-based tool (Python/Bash)
1. Create an executable script in `data/tools/<name>`
2. Create `data/tools/<name>.meta` with JSON metadata:
```json
{"description":"...","prompt":"...","args":{...},"tier":"hot","readOnly":false}
```
3. Run `make install-data` to install
### Compiled tool (Go)
1. Create a package under `tools/<family>/cmd/<name>/main.go`
- Read JSON args from stdin, write result to stdout, exit 0/1
- Share library code in `tools/<family>/` (e.g. `tools/lsp/`)
2. Create `<name>.meta` alongside `main.go` in the same `cmd/<name>/` directory (same format as above)
3. Add a build target in the Makefile that compiles to `$(CFG)/tools/<name>` **and** installs the `.meta` file alongside it
4. Run `make` to build and install
The `.meta` file lives with the code that produces the tool, not in `data/tools/`. See the `tools` target in the Makefile for the canonical pattern.
Both paths produce the same result: an executable + `.meta` in `$XDG_CONFIG_HOME/ollie/tools`.
The registry doesn't distinguish between scripts and binaries.
## Adding a new prompt
1. Write the markdown file in `data/prompts/`
2. If it should be loaded by default, reference it in `data/agents/default.json`
3. Run `make install-data` to install
## KDE development
KDE integration is part of this repository under `kde/`. Build and install it through the root Makefile target (`make kde`).
## Key Lessons (Aug 14–17 session)
1. **Unix permissions ARE the enforcement mechanism.** Sub-agents get GID "subagent"; top-level agents get GID "agent". File modes control access. Don't invent authorization layers when `chmod` works.
2. **Context cancellation propagates automatically.** Interrupting a parent kills all sub-agents at arbitrary depth through Go's `context.Context` chain. No explicit cleanup code needed.
3. **New features should be wiring, not construction.** If a feature requires more than ~50 lines, you're probably building infrastructure that already exists. Sub-agents: 43 lines. Goals: ~40 lines of handler. The rest is prompt.
4. **No pointless indirection.** Thin wrappers, thin delegations, adapter functions that just call another function — these are banned. Call the real thing directly. Move the code, don't wrap it.
5. **Shared code goes in shared packages.** `ollie/toolsrv` is importable by both `olliesrv` and `toolsrv` binaries. Don't duplicate functions across internal packages.
6. **The plan file is per-agent, NOT per-session.** Each agent owns its own plan at `session/{s}/agent/{a}/plan`.
7. **The goal file is per-session.** Writing to `session/{s}/goal` triggers a workflow (default: conductor). The goal text is NEVER overwritten by status changes.
8. **Subtraction > addition.** Removing `maxSteps` was -52 lines. The timeout on sub-agents replaced it with 4 lines. Always look for what to remove first.
9. **The 9P namespace is the API.** Every capability is a file. Read, Write, or Rdwr. BlockOnce and Stream are special cases of Read. That's the entire interface.
10. **`Rdwr` is an atomic operation** — not a variant of read or write. It's write-then-read as one unit. Sub-agents, session creation, tool execution, and generation all use this primitive.
11. **Persisted state can contain garbage.** When debugging impossible errors, check if the data itself is corrupted. A failed command's stderr captured and stored as state will return that error on every subsequent read — the bug isn't in the code, it's in the data.
12. **Environment variables don't always propagate.** Tools run through toolsrv, which sets specific env vars. If a tool calls another binary that expects `$USER` or `$OLLIE_UNAME`, verify those are actually set in the execution context. Provide explicit fallbacks.
13. **Don't blame the build system.** When a "fixed" bug keeps appearing, the code is probably fine. Check if you're reading stale data, hitting a different code path, or misunderstanding the actual error source. The build cache, the compiler, and the linker are rarely at fault.
14. **Separate index files for separate concerns.** Don't cram session and agent data into one line. `session/idx` lists sessions; `session/{s}/agent/idx` lists agents per session. Simpler parsing, fewer race conditions, cleaner code.
15. **Split large files by concern, not by size.** A 900-line file with mixed responsibilities is worse than three 300-line files with clear boundaries. Name files by what they do (`dispatch.go`, `compact.go`, `retry.go`), not by their parent type (`agent_dispatch.go`).
16. **Never shell out to git in agent plumbing.** A repo's own `.git/config` can name commands (`core.fsmonitor`, hooks, filters) that run as the user on any git invocation — silently, before any prompt (GitSpawn class, Sep 2026). Repo detection stays `os.Stat(.git)` (`util/paths.go`, `cmd/toolsrv/internal/server/proc.go`). If a feature needs git context, run `git -c core.fsmonitor=false ...` inside the sandbox, or read the files directly. Sandboxed tool env already forces `core.fsmonitor=false` via `GIT_CONFIG_*` (cmd/toolsrv/internal/exec/exec.go).
Reference in New Issue View Git Blame Copy Permalink