From 1159bd700a49441b012eb4652e0273f556a21456 Mon Sep 17 00:00:00 2001 From: Levi Neely Date: Thu, 27 Aug 2026 10:15:31 +0200 Subject: [PATCH] update AGENTS.md and architecture-core.md for agent package reorganization AGENTS.md: - Update Architecture section with detailed agent package file list - Update Key Files table with new agent package files - Add lesson 15: split by concern, not size architecture-core.md: - Update Package Map diagram with new files - Add file responsibility table in Agent Loop section - Reference compact.go in Session & Context section --- AGENTS.md | 26 ++++++++++++++++++++------ doc/architecture-core.md | 38 +++++++++++++++++++++++++++++--------- 2 files changed, 49 insertions(+), 15 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 257d2eb..4ea8e58 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -94,7 +94,15 @@ For direct Go testing, use the packages covered by `make test-core` and `make te ## Architecture (key concepts) 1. **One integration surface**: `olliesrv` exposes sessions and agents through a 9P2000 filesystem. Frontends include `o`, `ollie-9p`, KDE, Kate, Emacs, and scripts. Reads from `chat`, `statewait`, `eventwait`, and `feed` provide blocking/event-driven synchronization. 2. **Session and tool processes**: Each session owns an agent runtime in `olliesrv` and a separate `toolsrv` process. They communicate over an authenticated Unix socket using 9P. `toolclient` can respawn local toolsrv processes and can deploy/start toolsrv remotely over SSH with socket forwarding. -3. **Agent loop** (`cmd/olliesrv/internal/agent/loop.go` and `turn.go`): Stream an LLM response, parse native or text tool calls, execute tools, update history, and repeat until a final response, cancellation, or a configured limit. It includes transient retries, context-overflow compaction, error/stall/replan controls, and tool-result caching. +3. **Agent loop** (`cmd/olliesrv/internal/agent/`): The agent package is organized by concern: + - `loop.go`: Main loop — stream LLM, execute tools, update history + - `turn.go`: Submit entry point, turn orchestration + - `dispatch.go`: Tool execution, batching, conflict detection + - `history.go`: Message history, usage tracking + - `compact.go`: Context compaction, cold summarization + - `cache.go`: Tool result caching with file staleness detection + - `retry.go`: Error tracking, transient retry logic + - `state.go`, `chat.go`, `peer.go`, `subagent.go`: Agent state and coordination 4. **Dynamic tools**: Tools are external executables described by `.meta` files. `toolsrv` owns discovery and per-agent registries. `olliesrv` refreshes the registry, injects common dispatch flags (`bypass`, `timeout`, `sandbox`, `background`), and calls tools through `toolsrv.Conn`. 5. **Parallel and background dispatch**: Non-conflicting tool calls in one turn run concurrently. Metadata scopes schedule `read`, `write`, and global operations; shell-like global operations serialize. Any tool may run in the background, producing a process ID whose output is injected when the process changes or exits. 6. **Sandbox** (`cmd/toolsrv/internal/sandbox/`): Native Landlock policies control filesystem access in a short-lived child helper. The installed `sandbox.yaml` is sourced from `cmd/toolsrv/internal/sandbox/sandbox.yaml`. The toolsrv namespace and process state live under `cmd/toolsrv/p9.go` and `cmd/toolsrv/internal/server/`. The bypass broker provides policy-controlled escape requests, approval, persistence, and rate limiting. @@ -110,15 +118,19 @@ For direct Go testing, use the packages covered by `make test-core` and `make te | 9P namespace (olliesrv) | `cmd/olliesrv/internal/fs/spec.go` | | 9P namespace (toolsrv) | `cmd/toolsrv/p9.go`, `cmd/toolsrv/internal/server/server.go` | | virtfs EDSL | `virtfs/decl.go`, `virtfs/builder.go` | -| Agent loop and dispatch | `cmd/olliesrv/internal/agent/loop.go`, `turn.go` | +| Agent core and identity | `cmd/olliesrv/internal/agent/agent.go` | +| Agent loop | `cmd/olliesrv/internal/agent/loop.go` | +| Turn orchestration | `cmd/olliesrv/internal/agent/turn.go` | +| Tool dispatch and batching | `cmd/olliesrv/internal/agent/dispatch.go` | +| Message history | `cmd/olliesrv/internal/agent/history.go` | +| Context compaction | `cmd/olliesrv/internal/agent/compact.go` | | Runtime and prompt assembly | `cmd/olliesrv/internal/agent/runtime.go`, `prompt_resolver.go` | | Tool server binary | `cmd/toolsrv/` | -| Tool server client and process lifecycle | `toolsrv/client9p.go`, `cmd/olliesrv/internal/toolclient/` | -| Remote tool execution | `cmd/olliesrv/internal/toolclient/spawn.go` | +| Tool server client | `cmd/olliesrv/internal/toolclient/` | | Sandbox enforcement | `cmd/toolsrv/internal/sandbox/` | | Bypass broker | `cmd/olliesrv/internal/bypass/` | -| Session management and persistence | `cmd/olliesrv/internal/session/` | -| Embedded default system prompt | `cmd/olliesrv/internal/prompts/system_prompt.md` | +| Session management | `cmd/olliesrv/internal/session/` | +| Embedded system prompt | `cmd/olliesrv/internal/prompts/system_prompt.md` | | Agent configs | `data/agents/*.json` | | Backend configuration | `data/backends.conf`, `~/.config/ollie/backends.conf` | | Compiled tools | `tools/codeintel/`, `tools/filetools/`, `tools/lsp/`, `tools/web/` | @@ -191,3 +203,5 @@ KDE integration is part of this repository under `kde/`. Build and install it th 13. **Don't blame the build system.** When a "fixed" bug keeps appearing, the code is probably fine. Check if you're reading stale data, hitting a different code path, or misunderstanding the actual error source. The build cache, the compiler, and the linker are rarely at fault. 14. **Separate index files for separate concerns.** Don't cram session and agent data into one line. `session/idx` lists sessions; `session/{s}/agent/idx` lists agents per session. Simpler parsing, fewer race conditions, cleaner code. + +15. **Split large files by concern, not by size.** A 900-line file with mixed responsibilities is worse than three 300-line files with clear boundaries. Name files by what they do (`dispatch.go`, `compact.go`, `retry.go`), not by their parent type (`agent_dispatch.go`). diff --git a/doc/architecture-core.md b/doc/architecture-core.md index c21223c..d8827f3 100644 --- a/doc/architecture-core.md +++ b/doc/architecture-core.md @@ -9,11 +9,15 @@ This document covers the internal Go packages that implement Ollie's agent runti ```mermaid flowchart TB subgraph Agent["cmd/olliesrv/internal/agent/"] - AGENT["agent.go — Agent, events, interrupt"] - LOOP["loop.go — agent loop"] - HISTORY["history.go — history, task state, compaction"] - TURN["turn.go — per-turn state and cancellation"] - CMDS["commands.go — command dispatch"] + AGENT["agent.go — Agent struct, identity, events"] + LOOP["loop.go — main agent loop, streaming"] + TURN["turn.go — turn orchestration, Submit"] + DISPATCH["dispatch.go — tool execution, batching"] + HISTORY["history.go — message history, usage"] + COMPACT["compact.go — compaction, cold summarization"] + CACHE["cache.go — tool result caching"] + RETRY["retry.go — error tracking, retry logic"] + STATE["state.go, chat.go, peer.go, subagent.go"] RUNTIME["runtime.go — Runtime, preamble, tool dispatch"] CONFIG["agent_config.go — profile configuration"] PROMPT["prompt_resolver.go — prompt resolution"] @@ -21,7 +25,7 @@ flowchart TB subgraph Backend["cmd/olliesrv/internal/backend/"] BACKEND["backend.go — Backend interface and streaming"] - PROVIDERS["openai, anthropic, ollama, gemini, copilot, kiro, codewhisperer"] + PROVIDERS["openai, anthropic, ollama, gemini, copilot, kiro"] POOL["pool.go — backend pool"] end @@ -148,9 +152,25 @@ type Runtime struct { `NewAgent(AgentParams)` creates an agent with identity, session, profile, history, runtime, working directory, persistence callbacks, event handler state, chat-log state, and session environment. The configuration type on disk is `AgentConfig`; construction dependencies are carried separately by `AgentParams`. -## The Agent Loop (`cmd/olliesrv/internal/agent/loop.go`, `turn.go`) +## The Agent Loop (`cmd/olliesrv/internal/agent/`) -The `run()` function is the core execution engine. It takes an `agentConfig` and a `state` interface, and loops until the model stops calling tools. +The agent package is organized by concern: + +| File | Responsibility | +|------|---------------| +| `loop.go` | Main loop: stream LLM, execute tools, update history | +| `turn.go` | Submit entry point, turn orchestration | +| `dispatch.go` | Tool execution, batching, conflict detection | +| `history.go` | Message history, usage tracking | +| `compact.go` | Context compaction, cold summarization | +| `cache.go` | Tool result caching with file staleness | +| `retry.go` | Error tracking, transient retry logic | +| `state.go` | State management, signals, WaitChange | +| `chat.go` | Chat log, streaming output | +| `peer.go` | Peer agent management | +| `subagent.go` | Sub-agent depth tracking | + +The `run()` function in `loop.go` is the core execution engine. It loops until the model stops calling tools. ### Per-Iteration Flow @@ -224,7 +244,7 @@ After each tool round, `inferTaskStateUpdate` updates the structured overlay: - `LastAction` ← most recent tool call + truncated result - `PlanStep` ← next unchecked item from the plan file (or heuristic from assistant text) -## Session & Context (`cmd/olliesrv/internal/agent/history.go`) +## Session & Context (`cmd/olliesrv/internal/agent/history.go`, `compact.go`) ### Message History