Update prompting architecture documentation
This commit is contained in:
Ollie Agent
parent
0bab759f56
commit
3b587431a4
|
|
@ -0,0 +1,92 @@
|
|||
# Prompting Architecture
|
||||
|
||||
Ollie builds one system-prompt preamble for each active agent runtime. Prompting is assembled at agent selection or reload time, then reused across turns while the runtime remains active.
|
||||
|
||||
## Runtime flow
|
||||
|
||||
```text
|
||||
agent config + embedded system prompt + environment
|
||||
│
|
||||
├─ resolve prompt files
|
||||
├─ list tools and convert schemas
|
||||
├─ render named preamble sections
|
||||
└─ create Runtime
|
||||
│
|
||||
├─ system message: Preamble
|
||||
└─ each user turn: UserPrompt + input
|
||||
```
|
||||
|
||||
`agent.BuildRuntime` is the assembly point. It creates four ordered preamble sections:
|
||||
|
||||
1. **system** — the embedded default prompt, or the configured `systemPrompt` override.
|
||||
2. **env** — the current working directory, platform, Git-repository flag, and sandbox.
|
||||
3. **agent** — the resolved contents of the configured `prompt` entries.
|
||||
4. **tools** — descriptions and `.meta` documentation for the tools returned by the tool server.
|
||||
|
||||
Empty sections are skipped. Non-empty sections are joined with one newline. The resulting string is sent as the provider’s system message before conversation history.
|
||||
|
||||
## Agent configuration
|
||||
|
||||
Agent JSON is loaded from the installed agent configuration directory. The relevant fields are:
|
||||
|
||||
| Field | Behavior |
|
||||
|---|---|
|
||||
| `prompt` | A string or array of file paths concatenated into the agent section. |
|
||||
| `userPrompts` | A string or array of file paths concatenated and prepended to every user turn. |
|
||||
| `systemPrompt` | Replaces the embedded base system prompt with a resolved file. |
|
||||
| `autoLoad` | Selects tools loaded by the tool server before runtime construction. |
|
||||
| `tools` | Disables tool discovery and execution when explicitly set to `false`. |
|
||||
| `maxSteps` | Limits tool-call rounds in one turn; zero means unlimited. |
|
||||
| `compactionModel` | Selects the model used for history compaction. |
|
||||
| generation fields | Configure backend/model sampling and token limits. |
|
||||
|
||||
`prompt` and `userPrompts` accept either a single path or an array. Environment variables are expanded in paths and file contents. Missing files are retried with a `.md` suffix and then skipped. A leading `!` remains supported for legacy command entries; the command runs with the agent working directory and its stdout becomes prompt text.
|
||||
|
||||
The default configuration uses installed prompt files under `$XDG_CONFIG_HOME/ollie/prompts`. Runtime data is installed by `just install-data`; source prompt templates live under `data/prompts`.
|
||||
|
||||
## Tool rendering
|
||||
|
||||
When tools are enabled, `BuildRuntime` calls the configured `ToolsrvConn` once to obtain tool metadata. The metadata becomes both:
|
||||
|
||||
- the backend function/tool definitions, with dispatch flags added to each schema; and
|
||||
- the rendered `# Tools` preamble section.
|
||||
|
||||
`RenderTools` sorts tools by name, omits remote-server entries, and renders each local tool’s `.meta` prompt when present. It falls back to the description when no prompt is available. The current implementation combines the tool listing and documentation instead of maintaining separate listing and schema-dump sections.
|
||||
|
||||
Tool restrictions are not implemented in the prompt builder. The tool server determines which tools are available, and the runtime uses that same returned set for both execution schemas and prompt text.
|
||||
|
||||
## Per-turn prompting
|
||||
|
||||
`Runtime.UserPrompt` is not part of the system preamble. Before a turn is submitted, `executeTurn` prepends the resolved `userPrompts` content to the user input:
|
||||
|
||||
```text
|
||||
resolved user prompts
|
||||
|
||||
user input
|
||||
```
|
||||
|
||||
Queued prompts, injected prompts, and tool results then participate in the normal agent history and turn loop. The system preamble remains stable until the runtime is rebuilt.
|
||||
|
||||
## System prompt resolution
|
||||
|
||||
`ResolveSystemPrompt` selects the configured `systemPrompt` override when present. Otherwise it uses the embedded `prompts.DefaultSystemPrompt`, generated from `cmd/olliesrv/internal/prompts/system_prompt.md`.
|
||||
|
||||
The base prompt is intentionally generic. Agent-specific behavior belongs in the agent’s `prompt` files; active per-user rules belong in `userPrompts`. The system prompt also documents the common 9P runtime and tool-loading interface shared by agents.
|
||||
|
||||
## Context and compaction
|
||||
|
||||
The runtime passes the preamble and message history to the backend. The agent maintains history separately from prompt assembly. When history reaches configured limits, compaction produces a summary and the runtime continues with the compacted history plus the same system preamble.
|
||||
|
||||
The `systemprompt` 9P file exposes the current rendered preamble for inspection. It is the assembled runtime value, not a separate prompt source.
|
||||
|
||||
## Source locations
|
||||
|
||||
| Responsibility | Source |
|
||||
|---|---|
|
||||
| Agent configuration types | `cmd/olliesrv/internal/agent/agent_config.go` |
|
||||
| Prompt file and environment expansion | `cmd/olliesrv/internal/agent/prompt_resolver.go` |
|
||||
| Preamble and runtime construction | `cmd/olliesrv/internal/agent/runtime.go` |
|
||||
| Per-turn user-prompt injection | `cmd/olliesrv/internal/agent/turn.go` |
|
||||
| Embedded default system prompt | `cmd/olliesrv/internal/prompts/system_prompt.md` |
|
||||
| Installed agent configuration | `data/agents/*.json` |
|
||||
| Installed prompt templates | `data/prompts/*.md` |
|
||||
|
|
@ -1,150 +0,0 @@
|
|||
# Prompt Construction Audit
|
||||
|
||||
Date: 2026-08-09
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────┐
|
||||
│ Final system message (one blob) │
|
||||
├────────────┬──────────┬────────┬─────────┬───────────────┤
|
||||
│ system │ env │ agent │ tools │ tool_docs │
|
||||
│ (embedded) │(runtime) │(.json) │(listing)│(.meta prompts)│
|
||||
└────────────┴──────────┴────────┴─────────┴───────────────┘
|
||||
```
|
||||
|
||||
- **system**: `prompts/system_prompt.md` — embedded in binary, always prepended
|
||||
- **env**: `EnvironmentBlock()` — cwd, platform, date, git status, sandbox
|
||||
- **agent**: resolved from `prompt` array in agent JSON config (file paths → concatenated)
|
||||
- **tools**: `"- **name** — description"` per loaded tool
|
||||
- **tool_docs**: full `.meta` prompt text + JSON schema per tool (where `Server == ""`)
|
||||
|
||||
All sections join with single `\n`. Sent as a single system message before conversation history on every turn.
|
||||
|
||||
---
|
||||
|
||||
## Findings
|
||||
|
||||
### 1. System prompt is agent-unaware
|
||||
|
||||
The embedded `system_prompt.md` is identical for every agent profile. It contains:
|
||||
|
||||
- ~100 lines of **9P namespace reference** (session management, ctl commands, examples)
|
||||
- Large **tool-first table** mapping tasks→tools (lsp_*, gui_*, memory_*)
|
||||
- **Sandbox & Bypass** section
|
||||
- **API documentation fetching** section (`web_fetch` examples)
|
||||
|
||||
Agents that can't run `ollie-9p` (copilot, librarian, explorer) pay the token cost for all of it. The model sees instructions about tools it doesn't have.
|
||||
|
||||
**Affected agents**: all (but most wasteful for restricted agents)
|
||||
|
||||
---
|
||||
|
||||
### 2. Tool visibility vs. executability mismatch
|
||||
|
||||
`ListTools()` returns all loaded tools regardless of `allowTools`. The restriction only fires at execution time.
|
||||
|
||||
**Example**: `copilot.json` autoLoads `shell` + `reasoning_think`, then sets `allowTools: [file_read, file_glob, file_grep]`. The model sees shell in the preamble, the tool listing, tool_docs, AND the function-calling schema — tries to call it — gets rejected at execution.
|
||||
|
||||
**Code path**: `BuildRuntime()` calls `srv.ListTools()` first, then `srv.SetAllowTools()` after. The listing is never filtered.
|
||||
|
||||
**Affected agents**: copilot, librarian, taskmanager (any with `allowTools`)
|
||||
|
||||
---
|
||||
|
||||
### 3. Prompt references tools that don't exist or aren't loaded
|
||||
|
||||
| Reference | Where | Problem |
|
||||
|-----------|-------|---------|
|
||||
| `route` | driver.json autoLoad | No tool or .meta exists — it's a 9P file |
|
||||
| `reasoning_think` | agent-coding.md line 35 | default.json doesn't load it |
|
||||
| `lsp_*` | system_prompt.md table | Only loaded in driver.json |
|
||||
| `gui_*` | system_prompt.md table | Only in navigator/explorer/driver |
|
||||
| `memory_*` | system_prompt.md table | Only in librarian/taskmanager/driver |
|
||||
|
||||
The model either ignores phantom tool references (best case) or tries to use them (worst case, wastes a step).
|
||||
|
||||
---
|
||||
|
||||
### 4. Redundant/conflicting content between layers
|
||||
|
||||
- System prompt: "Format output as markdown" — copilot prompt has strict code-fence format
|
||||
- System prompt: "Never agree with something incorrect" — user-preferences.md says the same differently
|
||||
- System prompt: tool-first principle — coding prompt says "run the build, run tests" but default.json has no shell
|
||||
- user-preferences.md included in every agent — but system prompt already covers accuracy/honesty
|
||||
|
||||
Repeated instructions dilute signal. Conflicting format guidance forces reconciliation.
|
||||
|
||||
---
|
||||
|
||||
### 5. 9P reference is documentation, not behavioral instruction
|
||||
|
||||
The 9P section reads like API docs for a frontend developer. Lists every file, mode, operation. The agent actually needs:
|
||||
|
||||
1. How to read/write plan: `ollie-9p write session/$OLLIE_SESSION_ID/agent/$OLLIE_UNAME/plan`
|
||||
2. How to load a tool: `tool_load` tool handles this now
|
||||
|
||||
Everything else (session/new, session/idx, generate, complete, route, etc.) is irrelevant to agent behavior.
|
||||
|
||||
~100 lines of prompt budget on reference material the model rarely uses.
|
||||
|
||||
---
|
||||
|
||||
### 6. Tool docs section is bloated with schemas
|
||||
|
||||
`renderToolDocs()` includes full `.meta` prompt AND JSON schema for every loaded tool with `Server == ""` and a non-empty prompt.
|
||||
|
||||
The function-calling API already provides the schema to the model natively. The tool_docs section is a second copy.
|
||||
|
||||
For a fully-loaded driver agent (30+ tools): ~500+ lines of preamble on duplicated schema information.
|
||||
|
||||
---
|
||||
|
||||
### 7. `PRIME_*` env vars — legacy naming
|
||||
|
||||
`PromptEnv()` produces `PRIME_CWD`, `PRIME_PLATFORM`, `PRIME_DATE`, `PRIME_IS_GIT_REPO`. Inherited from upstream fork. Inconsistent with `OLLIE_SESSION_ID`, `OLLIE_UNAME`.
|
||||
|
||||
Minor impact but confusing for anyone reading configs.
|
||||
|
||||
---
|
||||
|
||||
### 8. `agents/README.md` is stale
|
||||
|
||||
References things that don't exist:
|
||||
- A `coding` agent JSON (no file — the prompt is used by default.json and driver.json)
|
||||
- `execute_bypass_bash` trusted tool
|
||||
- Hooks and MCP servers (no current config uses them)
|
||||
- "Shell command prompt entries" (prompts are file paths now, not commands)
|
||||
|
||||
---
|
||||
|
||||
## Priority ranking
|
||||
|
||||
| # | Finding | Impact | Effort | Status |
|
||||
|---|---------|--------|--------|--------|
|
||||
| 1 | System prompt agent-unaware | High | Medium | RESOLVED — trimmed content, not architecture |
|
||||
| 2 | Tool visibility/executability mismatch | High | Low | RESOLVED — removed AllowTools entirely |
|
||||
| 3 | Phantom tool references | Medium | Low | RESOLVED — removed route, added reasoning_think |
|
||||
| 4 | Redundant/conflicting content | Medium | Medium | RESOLVED — moved output protocol to agents, added shell to default |
|
||||
| 5 | 9P reference bloat | High | Medium | RESOLVED — trimmed to essentials |
|
||||
| 6 | Tool docs schema duplication | High | Medium | RESOLVED — merged listing+docs, removed schema dump |
|
||||
| 7 | PRIME_* legacy naming | Low | Low | RESOLVED — killed entirely, direct struct fields |
|
||||
| 8 | Stale README | Low | Low | RESOLVED — rewritten |
|
||||
|
||||
## Decisions made
|
||||
|
||||
### Finding 1: System prompt is agent-unaware
|
||||
**Decision**: Not an architecture problem. The system prompt IS generic and should remain so — all agents live in 9P, all agents can use ollie-9p, all agents have sandbox. The fix is content quality, not composability.
|
||||
|
||||
Changes applied to `prompts/system_prompt.md`:
|
||||
- Trimmed Core Identity (removed filler paragraph)
|
||||
- Removed shell heredoc reference from Output protocol, removed size restriction
|
||||
- Moved API Documentation section to `data/prompts/agent-coding.md` (coding-specific)
|
||||
- Rewrote Tool & Skill Registry: clear tool/skill distinction, explicit `ollie-9p read tools`, no vague "write to ctl"
|
||||
- Removed entire Tool-First Principle section (16-row table, hard rule, rationale) — dynamic registry and tool_docs handle this now; will re-add if needed
|
||||
- Trimmed 9P section: removed dead files (complete, route), removed help/aliases, merged tables, replaced abstract {sname}/{aname} with $OLLIE_SESSION_ID/$OLLIE_UNAME for self-ops, cut examples from 15 to 6
|
||||
|
||||
Result: ~200 lines → ~90 lines. Higher signal density.
|
||||
|
||||
### Finding 5: 9P reference bloat
|
||||
**Decision**: Resolved as part of finding 1. Kept all namespace tables (agents need to know the API) but cut dead entries and reduced examples to what agents actually do.
|
||||
Loading…
Reference in New Issue