Merge branch 'remove-execute-pipe'

This commit is contained in:
Levi Neely 2026-04-16 21:40:03 +02:00
commit f48f208682
1 changed files with 31 additions and 47 deletions

View File

@ -1,61 +1,47 @@
---
name: execute-pipe
intent: parallel, pipeline, pipe, fan-out, concurrent, batch, etl, transform, compose
description: Compose execute_code into sequential pipelines with optional parallel fan-out stages using execute_pipe. Use for multi-step data processing, parallel independent operations, and ETL workflows.
description: Use execute_code pipeline stages for sequential data processing, parallel fan-out, and ETL workflows.
---
# Execute Pipeline Patterns
## Principle: smart data, dumb code
The primitives stay simple. Complexity lives in the data schema and the skill (here), not in the tools themselves. Design pipelines around data shape, not around clever tool use.
The primitive stays simple. Complexity lives in the data schema and the skill (here), not in the tool itself. Design pipelines around data shape, not around clever tool use.
## Tools
## execute_code: the unified primitive
| Tool | What it does |
|---|---|
| `execute_code` | Run one or more steps. Steps may be inline `{code}` or named `{tool, args}` scripts. Multiple steps run in parallel; single step returns raw output. |
| `execute_pipe` | Chain stages sequentially, piping stdout → stdin. Each stage: `{code}`, `{tool, args}`, or `{parallel: [steps]}`. |
`execute_code` accepts a `steps` array of pipeline stages. Stages run sequentially, each stage's stdout fed to the next stage's stdin. A single stage returning raw output is the degenerate case.
## execute_code: parallel fan-out
Single step — raw output:
```json
{"steps": [{"code": "wc -l file.txt"}]}
```
Multiple steps — run concurrently, results under `=== step N ===` headers:
```json
{"steps": [
{"code": "wc -l a.txt"},
{"code": "wc -l b.txt"},
{"tool": "count-lines.sh", "args": ["c.txt"]}
]}
```
Each stage is one of:
- `{code, language}` — inline code (default: bash)
- `{tool, args}` — named script from `ollie/t`; language detected from shebang
- `{parallel: [steps]}` — concurrent fan-out; outputs concatenated in submission order, fed to next stage
Tool steps and inline code steps can be mixed freely. Tool steps are trusted (no pattern validation); inline code steps are validated before execution.
## Language selection
How the interpreter is chosen depends on step type:
| Step type | How language is determined |
|---|---|
| `{code}` | `language` field; defaults to `bash` if absent |
| `{tool}` | shebang line in the script (`#!/usr/bin/env python3`, etc.); `language` field is ignored |
| `{tool}` | shebang line in the script; `language` field is ignored |
Supported `language` values for inline code: `bash`, `python3`, `perl`, `lua`, `awk`, `sed`, `jq`, `ed`, `expect`, `bc`.
Do not set `language` on a tool step — it has no effect and will mislead readers of the call.
Results are always in submission order, regardless of completion order.
## execute_pipe: sequential pipeline
Each stage's stdout becomes the next stage's stdin.
## Single stage (degenerate case)
Raw output, no headers:
```json
{"pipe": [
{"steps": [{"code": "wc -l file.txt"}]}
```
## Sequential pipeline
Each stage's stdout becomes the next stage's stdin:
```json
{"steps": [
{"code": "grep ERROR app.log"},
{"code": "sort"},
{"code": "uniq -c"}
@ -64,18 +50,18 @@ Each stage's stdout becomes the next stage's stdin.
Tool stage:
```json
{"pipe": [
{"steps": [
{"tool": "fetch-metrics.sh", "args": ["--last=1h"]},
{"code": "jq '.[] | select(.latency > 500)'"}
]}
```
## execute_pipe: parallel stage
## Parallel stage
A `{parallel: [...]}` stage fans out concurrently. Each step is `{code}` or `{tool, args}`. Outputs are concatenated in submission order and fed as a single stream to the next stage.
A `{parallel: [...]}` stage fans out concurrently. Outputs are concatenated in submission order and fed as a single stream to the next stage:
```json
{"pipe": [
{"steps": [
{"parallel": [
{"tool": "fetch-app1.sh"},
{"tool": "fetch-app2.sh"},
@ -90,10 +76,10 @@ A `{parallel: [...]}` stage fans out concurrently. Each step is `{code}` or `{to
## ETL pattern: disparate schemas
When parallel steps produce different output schemas, normalize each before merging. Use an inner pipe as the transform stage.
When parallel steps produce different output schemas, normalize each before merging:
```json
{"pipe": [
{"steps": [
{"parallel": [
{"code": "jq -r '.name' users.json"},
{"code": "awk -F: '{print $1}' /etc/passwd"}
@ -103,9 +89,7 @@ When parallel steps produce different output schemas, normalize each before merg
]}
```
Here both parallel steps produce newline-delimited names (same schema). The `sort -u` stage then deduplicates the merged stream.
If normalization requires multiple steps, extract it as a named tool in `ollie/t` and reference it with `{tool}`.
If normalization requires multiple steps, extract it as a named tool in `ollie/t`.
## Timeout
@ -115,8 +99,8 @@ If normalization requires multiple steps, extract it as a named tool in `ollie/t
| Situation | Pattern |
|---|---|
| One independent operation | `execute_code` single step |
| N independent operations, same output schema | `execute_code` parallel steps, or `execute_pipe` parallel stage |
| N operations feeding one transform | `execute_pipe` with parallel first stage |
| Multi-step data transform | `execute_pipe` sequential stages |
| Reusable transform logic | `{tool}` step in `execute_code`, or named script in `ollie/t` as a pipe stage |
| One independent operation | single stage |
| N independent operations, same output schema | `{parallel: [...]}` stage |
| N operations feeding one transform | parallel first stage, then sequential |
| Multi-step data transform | sequential stages |
| Reusable transform logic | `{tool}` stage referencing a named script in `ollie/t` |