diff --git a/OBSERVATIONS.md b/OBSERVATIONS.md deleted file mode 100644 index 6ec1a9b..0000000 --- a/OBSERVATIONS.md +++ /dev/null @@ -1,46 +0,0 @@ -# Observations - -Running notes on design decisions, experiments, and lessons learned during -development. Entries are roughly chronological. - ---- - -## Tool Schema Complexity and Model Capability - -**Context.** The execute package originally exposed a single `execute_code` -tool with a flexible schema: callers could pass inline `steps`, a `pipe` array -for sequential stdin-chained execution, tool-name references, parallel fan-out -groups, elevated steps, and a sandbox selector — all through one JSON object. - -**Observation.** Larger models with strong reasoning ability (e.g. frontier -chat models and extended-thinking variants) handled the unified schema well. -They correctly discriminated between the `steps` and `pipe` fields, chose -appropriate execution modes, and composed pipelines without confusion. - -Smaller models struggled. Common failure modes: - -- Emitting both `steps` and `pipe` in the same call, producing ambiguous or - undefined behaviour. -- Using `pipe` where `steps` was correct (or vice versa), because the - distinction between "parallel batch" and "stdin pipeline" was not obvious - from a single tool description. -- Ignoring the `tool` field inside a step and writing inline shell code that - reimplemented an existing named tool. - -**Decision.** Split `execute_code` into three purpose-specific tools: - -| Tool | Input key | Purpose | -|------|-----------|---------------------------| -| `execute_code` | `steps` | Arbitrary inline code. Each step is independent; read-safe steps may run in parallel automatically. | -| `call_tool` | `calls` | Named scripts from the tools directory. Inline `code` fields are rejected at dispatch. | -| `pipe` | `stages` | Sequential pipeline. stdout of each stage feeds stdin of the next. Stages may be inline code or named tools. | - -The `parallel` option (a `parallel` array inside a step) is retained in -`execute_code` and `pipe` for fan-out within a stage. This gives smaller -models a clear, narrow contract per tool without removing any capability. - -**Trade-off.** Sophisticated models now make three tool choices where they -previously made one. For those models this is a minor regression in ergonomics. -The bet is that the schema clarity gained at the smaller-model end outweighs -the slight overhead at the larger-model end, and that a well-prompted larger -model will use the right tool consistently regardless.