docs: remove misplaced OBSERVATIONS.md (belongs in ollie/doc)

This commit is contained in:
Levi Neely 2026-05-15 18:59:26 +00:00
parent 21e6c0954b
commit 46940eef07
1 changed files with 0 additions and 46 deletions

View File

@ -1,46 +0,0 @@
# Observations
Running notes on design decisions, experiments, and lessons learned during
development. Entries are roughly chronological.
---
## Tool Schema Complexity and Model Capability
**Context.** The execute package originally exposed a single `execute_code`
tool with a flexible schema: callers could pass inline `steps`, a `pipe` array
for sequential stdin-chained execution, tool-name references, parallel fan-out
groups, elevated steps, and a sandbox selector — all through one JSON object.
**Observation.** Larger models with strong reasoning ability (e.g. frontier
chat models and extended-thinking variants) handled the unified schema well.
They correctly discriminated between the `steps` and `pipe` fields, chose
appropriate execution modes, and composed pipelines without confusion.
Smaller models struggled. Common failure modes:
- Emitting both `steps` and `pipe` in the same call, producing ambiguous or
undefined behaviour.
- Using `pipe` where `steps` was correct (or vice versa), because the
distinction between "parallel batch" and "stdin pipeline" was not obvious
from a single tool description.
- Ignoring the `tool` field inside a step and writing inline shell code that
reimplemented an existing named tool.
**Decision.** Split `execute_code` into three purpose-specific tools:
| Tool | Input key | Purpose |
|------|-----------|---------------------------|
| `execute_code` | `steps` | Arbitrary inline code. Each step is independent; read-safe steps may run in parallel automatically. |
| `call_tool` | `calls` | Named scripts from the tools directory. Inline `code` fields are rejected at dispatch. |
| `pipe` | `stages` | Sequential pipeline. stdout of each stage feeds stdin of the next. Stages may be inline code or named tools. |
The `parallel` option (a `parallel` array inside a step) is retained in
`execute_code` and `pipe` for fan-out within a stage. This gives smaller
models a clear, narrow contract per tool without removing any capability.
**Trade-off.** Sophisticated models now make three tool choices where they
previously made one. For those models this is a minor regression in ergonomics.
The bet is that the schema clarity gained at the smaller-model end outweighs
the slight overhead at the larger-model end, and that a well-prompted larger
model will use the right tool consistently regardless.