From 7bf9b6da9b804715a77ded72b9150a848dd15492 Mon Sep 17 00:00:00 2001 From: Ollie Agent Date: Thu, 20 Aug 2026 17:38:13 +0200 Subject: [PATCH] Document embedding-guided discovery architecture --- doc/architecture-tools.md | 25 +++++++++++++++++++++++++ doc/architecture.md | 4 ++++ doc/evolution.md | 30 ++++++++++++++++++++++++++++-- 3 files changed, 57 insertions(+), 2 deletions(-) diff --git a/doc/architecture-tools.md b/doc/architecture-tools.md index 7f491fd..2691390 100644 --- a/doc/architecture-tools.md +++ b/doc/architecture-tools.md @@ -68,6 +68,31 @@ The `.meta` file is the sole source of discovery and schema. The registry never | `outputFormat` | string | Output format metadata. | | `resetsCounter` | bool | Marks tools that reset the relevant counter. | +## Tool and skill discovery + +Tool discovery remains metadata-driven: `toolsrv` scans `.meta` files and manages +which tools are loaded for an agent. Ollie adds a separate semantic ranking +layer before prompt assembly: + +```text +user request + │ + ├── embedding.Model.Embed() + │ + ├── cosine similarity → skill descriptions → up to 3 injected skills + └── cosine similarity → tool descriptions → up to 5 injected hints +``` + +The shared `skills.Index` precomputes vectors for skill or tool descriptions and +returns results above a configured threshold in descending relevance order. The +agent keeps the full tool registry and callable tool loading semantics +unchanged; embeddings only improve which capabilities are disclosed first. + +The model is loaded lazily from `skills.DefaultModelDir()` and shared through +per-process cached indexes. It is not the conversational model and does not +send user requests or descriptions to a provider. See the `embedding/` and +`skills/` packages for the implementation. + ## Command resolution When the registry resolves a tool named `name`, it checks: diff --git a/doc/architecture.md b/doc/architecture.md index 3f92fec..9acf23d 100644 --- a/doc/architecture.md +++ b/doc/architecture.md @@ -12,6 +12,8 @@ flowchart TB A --> B[provider backend] A --> T[toolsrv] T --> R[tool registry] + R --> E[embedding index] + E --> M[local ONNX embedding model] T --> X[native Landlock sandbox helper] T --> P[bypass broker] ``` @@ -30,6 +32,8 @@ The loop resolves system and user prompts, renders context, calls the configured `toolsrv` is a separate process connected through 9P. It owns tool discovery, metadata, lazy registry loading, process lifecycle, output streaming, and sandbox invocation. Tool implementations are installed executables or scripts; they are not compiled into the agent loop. The same service can expose local and remotely deployed tools. +Agent-side semantic discovery uses the shared embedding and skills packages to index tool metadata and skill descriptions. A local `all-MiniLM-L6-v2` ONNX model ranks descriptions against the user request with cosine similarity. Only high-scoring, bounded results are injected: up to five tool hints and three skill definitions. This keeps prompts small while making discovery effective when a request does not use the exact tool or skill name. The embedding model is local and independent of the conversational backend. + The tool namespace also provides process and sandbox state. Normal execution is restricted by the configured native Landlock policy. The bypass broker is a policy-controlled path for explicitly approved operations outside that policy. ## Deliberately unimplemented agent patterns diff --git a/doc/evolution.md b/doc/evolution.md index 5ff3468..47d2d21 100644 --- a/doc/evolution.md +++ b/doc/evolution.md @@ -96,6 +96,7 @@ gantt Goals + conductor workflow :done, 2026-08-17, 1d Split session/agent indexes :done, 2026-08-17, 1d Agent peers + consensus :done, 2026-08-19, 1d + Embedding-guided discovery :done, 2026-08-20, 1d ``` ## Phase 1: Monorepo Bootstrap (Apr 11) Started as independent git repos unified under a monorepo with submodules. @@ -1440,11 +1441,36 @@ The maximum tool result included in model context was reduced from 128 KiB to 32 KiB. Large command or file results are therefore less likely to crowd out the conversation and instructions. +## Phase 31: Embedding-Guided Tool and Skill Discovery (Aug 20) + +Ollie added a local embedding subsystem for semantic discovery of tools and +skills. It loads the `all-MiniLM-L6-v2` sentence-transformer through ONNX +Runtime, tokenizes descriptions, produces 384-dimensional vectors, and ranks +matches with cosine similarity. + +The skill index scans configured `SKILL.md` directories, parses their +frontmatter, precomputes description embeddings, and matches each user request +against the indexed skills. The agent injects up to three relevant skills when +they exceed the configured similarity threshold. Tool metadata is indexed by +the same mechanism, allowing the agent to inject up to five relevant tool hints +without placing every tool in the model context. + +The embedding model and ONNX Runtime are installed under the XDG data model +directory by `make install-models` / `make install-data`. Skill directories may +be overridden through `embedding.conf`; the default is the installed skills +directory. Matching is lazy and cached per process, so the common path pays the +model-loading cost once. + +This is a significant architectural shift for small models: discovery becomes +semantic rather than dependent on exact tool or skill names, while progressive +disclosure keeps the prompt bounded. The embedding model is local and separate +from the conversational backend, so provider choice does not affect discovery. + --- Commit count: ~60 commits over 3 days. The recent work added native structural -and file tooling, persistent memory, and tighter context bounds while removing -obsolete tool implementations. +and file tooling, persistent memory, tighter context bounds, and embedding- +guided discovery while removing obsolete tool implementations. ---