doc: update architecture, evolution, README for toolsrv 9P migration
- README: toolsrv shown as separate child process in architecture diagram,
cmd/toolsrv added to repository layout table
- architecture.md: repository structure reflects cmd/{olliesrv,toolsrv}
internal packages, package table updated for two-server model
- evolution.md: add Phase 20 (toolsrv 9P migration & process isolation),
update topology diagram, add to gantt chart
This commit is contained in:
Ollie Agent
parent
df74c3fa60
commit
4b7d5cdd79
18
README.md
18
README.md
|
|
@ -70,7 +70,8 @@ Single Go module with one Git submodule (kde).
|
|||
|-----------|----------|-------------|
|
||||
| `agent/` | Go | Agent loop, history, prompt resolution, context compaction |
|
||||
| `backend/` | Go | LLM providers (Anthropic, OpenAI, Ollama, Gemini, Copilot, CodeWhisperer) |
|
||||
| `toolsrv/` | Go | Tool server, dynamic tool dispatch, sandboxed execution, remote execution |
|
||||
| `toolsrv/` | Go | Tool server client library (9P client for toolsrv) |
|
||||
| `cmd/toolsrv/` | Go | Tool execution server (separate 9P server, sandboxed execution) |
|
||||
| `tools/lsp/` | Go | LSP bridge daemon + client library (gopls, clangd, intelephense) |
|
||||
| `session/` | Go | Session lifecycle, config, persistence |
|
||||
| `fs/` | Go | 9P filesystem tree (EDSL-declared namespace, ctl dispatch) |
|
||||
|
|
@ -78,7 +79,8 @@ Single Go module with one Git submodule (kde).
|
|||
| `bypass/` | Go | Bypass broker (sandbox escape approval) |
|
||||
| `sandbox/` | Go | Landlock sandbox config |
|
||||
| `lib9p/` | Go | 9P protocol library + native C client |
|
||||
| `cmd/olliesrv/` | Go | The main binary |
|
||||
| `cmd/olliesrv/` | Go | The main server binary (session/agent 9P namespace) |
|
||||
| `cmd/toolsrv/` | Go | Tool execution server (spawned by olliesrv, separate 9P server) |
|
||||
| `cmd/ollie-9p/` | Go | 9P client CLI |
|
||||
| `cmd/ollie-remote/` | Go | Remote execution binary |
|
||||
| `kde/` | C++/Qt6 | KDE plasmoid, GUI, Kate plugin, KRunner *(submodule)* |
|
||||
|
|
@ -109,7 +111,11 @@ graph TB
|
|||
P9[9P Filesystem<br>session/ namespace]
|
||||
SESS[session.Session<br>tool server conn · persistence]
|
||||
AG[agent.Agent<br>loop · compaction<br>prompt assembly]
|
||||
TOOLS[toolsrv.Server<br>dynamic dispatch · sandbox]
|
||||
end
|
||||
|
||||
subgraph "toolsrv (child process, 9P)"
|
||||
TOOLS[9P Filesystem<br>ctl · tools · proc/]
|
||||
EXEC[Tool Scripts<br>· shell, reasoning_think<br>· file_*, lsp_*, memory_*<br>· gui_*, subagent_*]
|
||||
end
|
||||
|
||||
subgraph "LLM Backends"
|
||||
|
|
@ -120,16 +126,12 @@ graph TB
|
|||
KIRO[Kiro]
|
||||
end
|
||||
|
||||
subgraph Execution
|
||||
EXEC[Tool Scripts<br>loaded via ctl<br>· shell, reasoning_think<br>· file_*, lsp_*, memory_*<br>· gui_*, subagent_*]
|
||||
end
|
||||
|
||||
TUI & SCRIPTS & ACME --> O
|
||||
O --> P9
|
||||
KDE & KATE & EMACS --> P9
|
||||
P9 --> SESS
|
||||
SESS --> AG
|
||||
SESS -->|shared RPC conn| TOOLS
|
||||
SESS -->|9P over Unix socket| TOOLS
|
||||
|
||||
AG -->|streaming| OLL & OAI & ANT & COP & KIRO
|
||||
|
||||
|
|
|
|||
|
|
@ -15,31 +15,19 @@ Design principles:
|
|||
Single Go module (`ollie`) with one Git submodule (`kde/`) for the KDE frontend:
|
||||
```
|
||||
ollie/
|
||||
├── agent/ Agent loop, history, prompt resolution, context compaction
|
||||
├── backend/ LLM providers (Anthropic, OpenAI, Ollama, Gemini, Copilot, CodeWhisperer)
|
||||
├── toolsrv/ Tool server: sandboxed execution, tool registry, skill management
|
||||
├── session/ Session lifecycle (config, creation, persistence)
|
||||
├── fs/ 9P filesystem: EDSL spec + handlers (flat package)
|
||||
│ ├── spec.go Namespace declaration (single source of truth)
|
||||
│ ├── handlers.go All file handlers (root, session, agent)
|
||||
│ ├── ctl.go rdwrHandler type + rdwrDispatch (shared by all ctl files)
|
||||
│ ├── agent_log.go AgentLog (chat streaming, chatCond)
|
||||
│ ├── session_node.go SessionNode wrapper
|
||||
│ ├── lifecycle.go RootState, session node cache
|
||||
│ ├── newroot.go NewRoot — tree construction config
|
||||
│ └── cache.go ModelCache
|
||||
├── cmd/
|
||||
│ ├── olliesrv/ 9P server (sessions, agents, backends)
|
||||
│ │ └── internal/ agent/, backend/, bypass/, fs/, session/, prompts/, toolclient/
|
||||
│ ├── toolsrv/ 9P tool execution server (separate process)
|
||||
│ │ └── internal/ fs/, exec/, registry/, sandbox/
|
||||
│ ├── ollie-9p/ 9P client CLI
|
||||
│ └── ollie-remote/ Remote execution server
|
||||
├── toolsrv/ Client library (9P client for toolsrv)
|
||||
├── fsedsl/ Filesystem declaration EDSL (generic, reusable)
|
||||
├── bypass/ Bypass broker (sandbox escape approval)
|
||||
├── sandbox/ Landlock sandbox config YAML
|
||||
├── lib9p/ 9P protocol library + native C client
|
||||
├── env/ Environment variable loading
|
||||
├── log/ Structured logging
|
||||
├── paths/ XDG path resolution
|
||||
├── format/ Chat log formatting constants
|
||||
├── cmd/ Binaries:
|
||||
│ ├── olliesrv/ 9P server
|
||||
│ ├── ollie-9p/ 9P client CLI
|
||||
│ └── ollie-remote/ Remote execution server
|
||||
├── kde/ KDE integration (submodule) — standalone GUI, Kate plugin, KRunner
|
||||
├── data/agents/ Agent config JSONs (default, coding, orchestrator, worker, ...)
|
||||
├── data/prompts/ Prompt templates (markdown)
|
||||
|
|
@ -71,7 +59,9 @@ flowchart TB
|
|||
subgraph Core["Agent Engine (per session)"]
|
||||
LOOP["Agent Loop\n(agent/loop.go)"]
|
||||
SESS["Session State\n(agent/history.go)"]
|
||||
TRV["toolsrv.Server\n· dynamic tool dispatch\n· landlock sandboxed execution\n· remote execution via SSH"]
|
||||
end
|
||||
subgraph ToolSrv["toolsrv (child process)"]
|
||||
TRV["9P Server\n· dynamic tool dispatch\n· landlock sandboxed execution\n· idle timeout + Pdeathsig lifecycle"]
|
||||
end
|
||||
subgraph Backends["LLM Backends"]
|
||||
OLLAMA["Ollama"]
|
||||
|
|
@ -98,15 +88,20 @@ The core is a single Go module (`ollie`) with no binary. Binaries live in `cmd/`
|
|||
### Package Layout
|
||||
| Package | Purpose |
|
||||
|---|---|
|
||||
| `agent/` | Agent struct, loop, history, compaction, hooks, commands, prompt resolution, state |
|
||||
| `backend/` | Backend interface + LLM providers (Anthropic, OpenAI, Ollama, Gemini, Copilot, CodeWhisperer) |
|
||||
| `toolsrv/` | Tool server: dynamic tool dispatch, sandboxed execution, result tiering, remote execution (ollie-remote) |
|
||||
| `session/` | Session lifecycle, config, persistence |
|
||||
| `detach/` | Background process management (ring buffer, signal) |
|
||||
| `bypass/` | Bypass broker (privilege escalation daemon, policy) |
|
||||
| `sandbox/` | Landrun sandbox configuration and command wrapping |
|
||||
| `cmd/olliesrv/internal/agent/` | Agent struct, loop, history, compaction, hooks, commands, prompt resolution, state |
|
||||
| `cmd/olliesrv/internal/backend/` | Backend interface + LLM providers (Anthropic, OpenAI, Ollama, Gemini, Copilot, CodeWhisperer) |
|
||||
| `cmd/olliesrv/internal/fs/` | 9P filesystem: EDSL spec + handlers, ctl dispatch, session nodes |
|
||||
| `cmd/olliesrv/internal/session/` | Session lifecycle, persistence, tool server spawning |
|
||||
| `cmd/olliesrv/internal/bypass/` | Bypass broker (privilege escalation daemon, policy) |
|
||||
| `cmd/olliesrv/internal/toolclient/` | Spawning and managing toolsrv child processes |
|
||||
| `cmd/toolsrv/internal/fs/` | toolsrv filesystem spec, state, process management |
|
||||
| `cmd/toolsrv/internal/exec/` | Sandboxed tool execution (landrun, bypass) |
|
||||
| `cmd/toolsrv/internal/registry/` | Session-scoped tool registry |
|
||||
| `cmd/toolsrv/internal/sandbox/` | Landrun sandbox configuration and command wrapping |
|
||||
| `toolsrv/` | 9P client library for connecting to toolsrv |
|
||||
| `fsedsl/` | Generic filesystem declaration EDSL (used by both servers) |
|
||||
| `env/` | Session environment helpers |
|
||||
| `log/` | Structured logging |
|
||||
| `log/` | Structured logging (olliesrv) |
|
||||
| `paths/` | XDG path resolution |
|
||||
### Key Types
|
||||
**`agent.Agent`** — the concrete struct frontends drive:
|
||||
|
|
|
|||
|
|
@ -39,6 +39,7 @@ gantt
|
|||
9P Declarative EDSL :done, 2026-08-02, 1d
|
||||
Zero built-in tools :done, 2026-08-02, 2d
|
||||
EDSL extraction + cleanup :done, 2026-08-03, 2d
|
||||
toolsrv 9P + process isolation :done, 2026-08-10, 2d
|
||||
```
|
||||
## Phase 1: Monorepo Bootstrap (Apr 11)
|
||||
Started as independent git repos unified under a monorepo with submodules.
|
||||
|
|
@ -621,34 +622,18 @@ The KDE frontend received extensive updates across both days:
|
|||
## Updated Current Topology
|
||||
```
|
||||
ollie/ ← single Go module
|
||||
├── agent/ ← agent loop, history, hooks, commands
|
||||
├── backend/ ← LLM providers (6 backends)
|
||||
├── toolsrv/ ← tool server, registry, remote execution
|
||||
├── session/ ← session lifecycle, config
|
||||
├── fs/ ← 9P filesystem (EDSL-declared, flat package)
|
||||
│ ├── spec.go Namespace declaration (single source of truth)
|
||||
│ ├── fsnode.go FsNodeDecl type + Dir/Leaf/TemplateDir
|
||||
│ ├── builder.go BuildTree — spec → wired *Tree
|
||||
│ ├── rootfiles.go Root-level handlers
|
||||
│ ├── sessionfiles.go Session-level handlers
|
||||
│ ├── agentfiles.go Agent-level handlers
|
||||
│ ├── bypassfiles.go Bypass handlers
|
||||
│ ├── procfiles.go Process handlers
|
||||
│ ├── lifecycle.go Create/kill/rename + event ring
|
||||
│ ├── newroot.go NewRoot constructor
|
||||
│ ├── persist.go Session persistence
|
||||
│ ├── types.go Session/AgentLog types
|
||||
│ ├── tree.go 9P *Tree
|
||||
│ ├── fs.go 9P File/FileConfig
|
||||
│ └── format.go Event formatting
|
||||
├── detach/ ← background process management
|
||||
├── bypass/ ← bypass broker
|
||||
├── sandbox/ ← landrun sandbox config
|
||||
├── cmd/
|
||||
│ ├── olliesrv/ ← 9P server (sessions, agents, backends)
|
||||
│ │ └── internal/ agent/, backend/, bypass/, fs/, session/, prompts/, toolclient/
|
||||
│ ├── toolsrv/ ← 9P tool execution server (separate process)
|
||||
│ │ └── internal/ fs/, exec/, registry/, sandbox/
|
||||
│ ├── ollie-9p/ ← 9P client CLI
|
||||
│ └── ollie-remote/ ← remote execution binary
|
||||
├── toolsrv/ ← 9P client library for toolsrv
|
||||
├── fsedsl/ ← filesystem declaration EDSL (used by both servers)
|
||||
├── env/ ← environment helpers
|
||||
├── log/ ← structured logging
|
||||
├── paths/ ← XDG path resolution
|
||||
├── mount/ ← 9P FUSE mount client (network transparency)
|
||||
├── cmd/ ← binaries (olliesrv, ollie-9p, ollie-remote)
|
||||
├── kde/ ← KDE Plasma (submodule)
|
||||
├── data/tools/ ← tool executables + .meta sidecar files
|
||||
├── data/agents/ ← agent configs (JSON)
|
||||
|
|
@ -657,7 +642,6 @@ ollie/ ← single Go module
|
|||
├── data/scripts/ ← ollie-remount, o CLI
|
||||
├── data/services/ ← systemd, xdg-autostart
|
||||
├── prompts/ ← embedded prompt templates
|
||||
├── sandbox/ ← landrun sandbox profiles (YAML)
|
||||
└── doc/ ← documentation
|
||||
```
|
||||
|
||||
|
|
@ -1085,3 +1069,94 @@ Systematic audit of the prompt construction system and codebase maintainability.
|
|||
| **Total (core)** | **12,793** |
|
||||
|
||||
Excludes: backends (4,229), fsedsl (1,030), tests, KDE, tools, generated code.
|
||||
|
||||
## Phase 20: toolsrv 9P Migration & Process Isolation (Aug 10–11)
|
||||
|
||||
The tool server completed its evolution from an in-process library to a
|
||||
**fully independent 9P server** running as a child process of olliesrv.
|
||||
|
||||
### Before
|
||||
|
||||
`toolsrv` was a Go package (`ollie/toolsrv`) with a `Server` struct that ran
|
||||
in-process within olliesrv. Tool calls were Go function calls — no process
|
||||
boundary, no separate namespace. The package mixed client code, server code,
|
||||
registry, sandbox wrappers, and execution logic in a flat directory.
|
||||
|
||||
### After
|
||||
|
||||
Two distinct components:
|
||||
|
||||
1. **`cmd/toolsrv/`** — a standalone binary that serves its own 9P2000
|
||||
filesystem over a Unix socket. Has its own `internal/` packages:
|
||||
- `internal/fs/` — filesystem spec (fsedsl), server state, process management
|
||||
- `internal/exec/` — sandboxed tool execution (landrun, bypass broker)
|
||||
- `internal/registry/` — session-scoped tool registry
|
||||
- `internal/sandbox/` — landrun configuration (unchanged)
|
||||
|
||||
2. **`toolsrv/`** (root package) — a 9P client library. `Conn` dials toolsrv
|
||||
over a Unix socket, authenticates via Tauth, and exposes methods like
|
||||
`CallTool`, `LoadTool`, `ListTools`, `Ping`.
|
||||
|
||||
### toolsrv 9P Namespace
|
||||
|
||||
```
|
||||
/
|
||||
├── ctl write: load <tool>, unload <tool>, env K=V, cwd <path>
|
||||
├── tools read: list loaded tools (JSON), write: tool name to load
|
||||
├── info read: platform, arch
|
||||
└── proc/
|
||||
├── new rdwr: write tool+args, blocks, read result
|
||||
├── new.bg write: tool+args, returns pid immediately
|
||||
└── {pid}/
|
||||
├── out read: output
|
||||
├── wait read: blocks until exit, returns exit code
|
||||
├── stat read: running/exited, runtime, tool
|
||||
└── ctl write: signal <N>, dismiss
|
||||
```
|
||||
|
||||
Both `cmd/toolsrv` and `cmd/olliesrv` declare their namespaces using the
|
||||
same `fsedsl` library. Both implement their own 9P protocol handlers (using
|
||||
`9fans.net/go/plan9`) — they are intentionally separate servers that
|
||||
communicate over a socket.
|
||||
|
||||
### Process Lifecycle
|
||||
|
||||
Three-layer defense against orphaned toolsrv processes:
|
||||
|
||||
1. **Pdeathsig** — `SysProcAttr{Pdeathsig: SIGTERM}` on local spawn. When
|
||||
olliesrv dies, the kernel terminates toolsrv immediately.
|
||||
|
||||
2. **Idle timeout** — toolsrv tracks active 9P connections. After the first
|
||||
client connects and then all connections drop, a 30s timer starts.
|
||||
If no new connection arrives, toolsrv exits cleanly. 60s startup grace
|
||||
period for the initial connection.
|
||||
|
||||
3. **Kill-before-respawn** — `ProcessKeeper.Dial()` kills the old process
|
||||
before spawning a replacement. Prevents accumulation during reconnect cycles.
|
||||
|
||||
### Authentication
|
||||
|
||||
toolsrv uses 9P Tauth for authentication. The first client sets the shared
|
||||
secret; subsequent clients must provide the same secret. This replaces
|
||||
the previous token-in-environment approach and is compatible with socket
|
||||
permission security.
|
||||
|
||||
### Structural Consistency
|
||||
|
||||
Both servers now follow the same physical layout:
|
||||
|
||||
```
|
||||
cmd/{server}/
|
||||
├── main.go entry point
|
||||
├── server.go 9P protocol handler (top-level, like Plan 9 tradition)
|
||||
└── internal/
|
||||
├── fs/ filesystem spec + handlers + state
|
||||
├── ... domain-specific packages
|
||||
```
|
||||
|
||||
### Logging
|
||||
|
||||
toolsrv uses `log/slog` (Go stdlib) with a `"svc": "toolsrv"` attribute.
|
||||
This is intentionally separate from olliesrv's `ollie/log` package — they
|
||||
are different programs with different lifecycles.
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue