execute-pipe: new skill for pipeline and parallel execution patterns

Covers execute_code parallel fan-out, execute_pipe sequential and parallel
stages, ETL pattern for disparate schemas, language selection (inline code
vs shebang detection for tool steps), and when to load this skill.
This commit is contained in:
Levi Neely 2026-04-16 20:07:30 +02:00
parent 9e7f499088
commit bc1d8377c1
1 changed files with 123 additions and 0 deletions

123
execute-pipe/SKILL.md Normal file
View File

@ -0,0 +1,123 @@
---
name: execute-pipe
intent: parallel, pipeline, pipe, fan-out, concurrent, batch, etl, transform, compose
description: Compose execute_code and execute_tool into sequential pipelines with optional parallel fan-out stages. Use for multi-step data processing, parallel independent operations, and ETL workflows.
---
# Execute Pipeline Patterns
## Principle: smart data, dumb code
The primitives stay simple. Complexity lives in the data schema and the skill (here), not in the tools themselves. Design pipelines around data shape, not around clever tool use.
## Tools
| Tool | What it does |
|---|---|
| `execute_code` | Run one or more code snippets. Multiple steps run in parallel; single step returns raw output. |
| `execute_pipe` | Chain stages sequentially, piping stdout → stdin. Each stage: `{code}`, `{tool, args}`, or `{parallel: [steps]}`. |
| `execute_tool` | Run a named script from `ollie/t`. Used inside `execute_pipe` as a `{tool}` stage. |
## execute_code: parallel fan-out
Single step — raw output:
```json
{"steps": [{"code": "wc -l file.txt"}]}
```
Multiple steps — run concurrently, results under `=== step N ===` headers:
```json
{"steps": [
{"code": "wc -l a.txt"},
{"code": "wc -l b.txt"},
{"tool": "count-lines.sh", "args": ["c.txt"]}
]}
```
Tool steps and inline code steps can be mixed freely. Tool steps are trusted (no pattern validation); inline code steps are validated before execution.
## Language selection
How the interpreter is chosen depends on step type:
| Step type | How language is determined |
|---|---|
| `{code}` | `language` field; defaults to `bash` if absent |
| `{tool}` | shebang line in the script (`#!/usr/bin/env python3`, etc.); `language` field is ignored |
Supported `language` values for inline code: `bash`, `python3`, `perl`, `lua`, `awk`, `sed`, `jq`, `ed`, `expect`, `bc`.
Do not set `language` on a tool step — it has no effect and will mislead readers of the call.
Results are always in submission order, regardless of completion order.
## execute_pipe: sequential pipeline
Each stage's stdout becomes the next stage's stdin.
```json
{"pipe": [
{"code": "grep ERROR app.log"},
{"code": "sort"},
{"code": "uniq -c"}
]}
```
Tool stage:
```json
{"pipe": [
{"tool": "fetch-metrics.sh", "args": ["--last=1h"]},
{"code": "jq '.[] | select(.latency > 500)'"}
]}
```
## execute_pipe: parallel stage
A `{parallel: [...]}` stage fans out concurrently. Each step is `{code}` or `{tool, args}`. Outputs are concatenated in submission order and fed as a single stream to the next stage.
```json
{"pipe": [
{"parallel": [
{"tool": "fetch-app1.sh"},
{"tool": "fetch-app2.sh"},
{"code": "cat /var/log/app3.log"}
]},
{"code": "grep ERROR"},
{"code": "wc -l"}
]}
```
**Rule: use the same tool across all parallel steps in a stage.** Parallel steps that produce the same output schema concatenate cleanly. Parallel steps with disparate schemas produce an incoherent stream.
## ETL pattern: disparate schemas
When parallel steps produce different output schemas, normalize each before merging. Use an inner pipe as the transform stage.
```json
{"pipe": [
{"parallel": [
{"code": "jq -r '.name' users.json"},
{"code": "awk -F: '{print $1}' /etc/passwd"}
]},
{"code": "sort -u"},
{"tool": "dedup.py"}
]}
```
Here both parallel steps produce newline-delimited names (same schema). The `sort -u` stage then deduplicates the merged stream.
If normalization requires multiple steps, extract it as a named tool in `ollie/t` and reference it with `{tool}`.
## Timeout
`timeout` applies per stage (default: 30s). A pipeline of N stages can run up to N × timeout seconds total. Set explicitly for long-running stages.
## When to use which pattern
| Situation | Pattern |
|---|---|
| One independent operation | `execute_code` single step |
| N independent operations, same output schema | `execute_code` parallel steps, or `execute_pipe` parallel stage |
| N operations feeding one transform | `execute_pipe` with parallel first stage |
| Multi-step data transform | `execute_pipe` sequential stages |
| Reusable transform logic | `execute_tool` or named script in `ollie/t` |