7.4 KiB

Raw Blame History

Prompt Construction Audit

Date: 2026-08-09

Architecture

┌──────────────────────────────────────────────────────────┐
│             Final system message (one blob)               │
├────────────┬──────────┬────────┬─────────┬───────────────┤
│   system   │   env    │ agent  │  tools  │   tool_docs   │
│ (embedded) │(runtime) │(.json) │(listing)│(.meta prompts)│
└────────────┴──────────┴────────┴─────────┴───────────────┘

system: prompts/system_prompt.md — embedded in binary, always prepended
env: EnvironmentBlock() — cwd, platform, date, git status, sandbox
agent: resolved from prompt array in agent JSON config (file paths → concatenated)
tools: "- **name** — description" per loaded tool
tool_docs: full .meta prompt text + JSON schema per tool (where Server == "")

All sections join with single \n. Sent as a single system message before conversation history on every turn.

Findings

1. System prompt is agent-unaware

The embedded system_prompt.md is identical for every agent profile. It contains:

~100 lines of 9P namespace reference (session management, ctl commands, examples)
Large tool-first table mapping tasks→tools (lsp_, gui_, memory_*)
Sandbox & Bypass section
API documentation fetching section (web_fetch examples)

Agents that can't run ollie-9p (copilot, librarian, explorer) pay the token cost for all of it. The model sees instructions about tools it doesn't have.

Affected agents: all (but most wasteful for restricted agents)

2. Tool visibility vs. executability mismatch

ListTools() returns all loaded tools regardless of allowTools. The restriction only fires at execution time.

Example: copilot.json autoLoads shell + reasoning_think, then sets allowTools: [file_read, file_glob, file_grep]. The model sees shell in the preamble, the tool listing, tool_docs, AND the function-calling schema — tries to call it — gets rejected at execution.

Code path: BuildRuntime() calls srv.ListTools() first, then srv.SetAllowTools() after. The listing is never filtered.

Affected agents: copilot, librarian, taskmanager (any with allowTools)

3. Prompt references tools that don't exist or aren't loaded

Reference	Where	Problem
`route`	driver.json autoLoad	No tool or .meta exists — it's a 9P file
`reasoning_think`	agent-coding.md line 35	default.json doesn't load it
`lsp_*`	system_prompt.md table	Only loaded in driver.json
`gui_*`	system_prompt.md table	Only in navigator/explorer/driver
`memory_*`	system_prompt.md table	Only in librarian/taskmanager/driver

The model either ignores phantom tool references (best case) or tries to use them (worst case, wastes a step).

4. Redundant/conflicting content between layers

System prompt: "Format output as markdown" — copilot prompt has strict code-fence format
System prompt: "Never agree with something incorrect" — user-preferences.md says the same differently
System prompt: tool-first principle — coding prompt says "run the build, run tests" but default.json has no shell
user-preferences.md included in every agent — but system prompt already covers accuracy/honesty

Repeated instructions dilute signal. Conflicting format guidance forces reconciliation.

5. 9P reference is documentation, not behavioral instruction

The 9P section reads like API docs for a frontend developer. Lists every file, mode, operation. The agent actually needs:

How to read/write plan: ollie-9p write session/$OLLIE_SESSION_ID/agent/$OLLIE_UNAME/plan
How to load a tool: tool_load tool handles this now

Everything else (session/new, session/idx, generate, complete, route, etc.) is irrelevant to agent behavior.

~100 lines of prompt budget on reference material the model rarely uses.

6. Tool docs section is bloated with schemas

renderToolDocs() includes full .meta prompt AND JSON schema for every loaded tool with Server == "" and a non-empty prompt.

The function-calling API already provides the schema to the model natively. The tool_docs section is a second copy.

For a fully-loaded driver agent (30+ tools): ~500+ lines of preamble on duplicated schema information.

7. `PRIME_*` env vars — legacy naming

PromptEnv() produces PRIME_CWD, PRIME_PLATFORM, PRIME_DATE, PRIME_IS_GIT_REPO. Inherited from upstream fork. Inconsistent with OLLIE_SESSION_ID, OLLIE_UNAME.

Minor impact but confusing for anyone reading configs.

8. `agents/README.md` is stale

References things that don't exist:

A coding agent JSON (no file — the prompt is used by default.json and driver.json)
execute_bypass_bash trusted tool
Hooks and MCP servers (no current config uses them)
"Shell command prompt entries" (prompts are file paths now, not commands)

Priority ranking

#	Finding	Impact	Effort	Status
1	System prompt agent-unaware	High	Medium	RESOLVED — trimmed content, not architecture
2	Tool visibility/executability mismatch	High	Low	RESOLVED — removed AllowTools entirely
3	Phantom tool references	Medium	Low	RESOLVED — removed route, added reasoning_think
4	Redundant/conflicting content	Medium	Medium	RESOLVED — moved output protocol to agents, added shell to default
5	9P reference bloat	High	Medium	RESOLVED — trimmed to essentials
6	Tool docs schema duplication	High	Medium	RESOLVED — merged listing+docs, removed schema dump
7	PRIME_* legacy naming	Low	Low	RESOLVED — killed entirely, direct struct fields
8	Stale README	Low	Low	RESOLVED — rewritten

Decisions made

Finding 1: System prompt is agent-unaware

Decision: Not an architecture problem. The system prompt IS generic and should remain so — all agents live in 9P, all agents can use ollie-9p, all agents have sandbox. The fix is content quality, not composability.

Changes applied to prompts/system_prompt.md:

Trimmed Core Identity (removed filler paragraph)
Removed shell heredoc reference from Output protocol, removed size restriction
Moved API Documentation section to data/prompts/agent-coding.md (coding-specific)
Rewrote Tool & Skill Registry: clear tool/skill distinction, explicit ollie-9p read tools, no vague "write to ctl"
Removed entire Tool-First Principle section (16-row table, hard rule, rationale) — dynamic registry and tool_docs handle this now; will re-add if needed
Trimmed 9P section: removed dead files (complete, route), removed help/aliases, merged tables, replaced abstract {sname}/{aname} with $OLLIE_SESSION_ID/$OLLIE_UNAME for self-ops, cut examples from 15 to 6

Result: ~200 lines → ~90 lines. Higher signal density.

Finding 5: 9P reference bloat

Decision: Resolved as part of finding 1. Kept all namespace tables (agents need to know the API) but cut dead entries and reduced examples to what agents actually do.

7.4 KiB Raw Blame History