docs: remove misplaced OBSERVATIONS.md (belongs in ollie/doc)
This commit is contained in:
parent
21e6c0954b
commit
46940eef07
|
|
@ -1,46 +0,0 @@
|
|||
# Observations
|
||||
|
||||
Running notes on design decisions, experiments, and lessons learned during
|
||||
development. Entries are roughly chronological.
|
||||
|
||||
---
|
||||
|
||||
## Tool Schema Complexity and Model Capability
|
||||
|
||||
**Context.** The execute package originally exposed a single `execute_code`
|
||||
tool with a flexible schema: callers could pass inline `steps`, a `pipe` array
|
||||
for sequential stdin-chained execution, tool-name references, parallel fan-out
|
||||
groups, elevated steps, and a sandbox selector — all through one JSON object.
|
||||
|
||||
**Observation.** Larger models with strong reasoning ability (e.g. frontier
|
||||
chat models and extended-thinking variants) handled the unified schema well.
|
||||
They correctly discriminated between the `steps` and `pipe` fields, chose
|
||||
appropriate execution modes, and composed pipelines without confusion.
|
||||
|
||||
Smaller models struggled. Common failure modes:
|
||||
|
||||
- Emitting both `steps` and `pipe` in the same call, producing ambiguous or
|
||||
undefined behaviour.
|
||||
- Using `pipe` where `steps` was correct (or vice versa), because the
|
||||
distinction between "parallel batch" and "stdin pipeline" was not obvious
|
||||
from a single tool description.
|
||||
- Ignoring the `tool` field inside a step and writing inline shell code that
|
||||
reimplemented an existing named tool.
|
||||
|
||||
**Decision.** Split `execute_code` into three purpose-specific tools:
|
||||
|
||||
| Tool | Input key | Purpose |
|
||||
|------|-----------|---------------------------|
|
||||
| `execute_code` | `steps` | Arbitrary inline code. Each step is independent; read-safe steps may run in parallel automatically. |
|
||||
| `call_tool` | `calls` | Named scripts from the tools directory. Inline `code` fields are rejected at dispatch. |
|
||||
| `pipe` | `stages` | Sequential pipeline. stdout of each stage feeds stdin of the next. Stages may be inline code or named tools. |
|
||||
|
||||
The `parallel` option (a `parallel` array inside a step) is retained in
|
||||
`execute_code` and `pipe` for fan-out within a stage. This gives smaller
|
||||
models a clear, narrow contract per tool without removing any capability.
|
||||
|
||||
**Trade-off.** Sophisticated models now make three tool choices where they
|
||||
previously made one. For those models this is a minor regression in ergonomics.
|
||||
The bet is that the schema clarity gained at the smaller-model end outweighs
|
||||
the slight overhead at the larger-model end, and that a well-prompted larger
|
||||
model will use the right tool consistently regardless.
|
||||
Reference in New Issue