execute: update skill for execute_pipe removal
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
611e3384c3
commit
f9aa756a9b
|
|
@ -1,61 +1,47 @@
|
||||||
---
|
---
|
||||||
name: execute-pipe
|
name: execute-pipe
|
||||||
intent: parallel, pipeline, pipe, fan-out, concurrent, batch, etl, transform, compose
|
intent: parallel, pipeline, pipe, fan-out, concurrent, batch, etl, transform, compose
|
||||||
description: Compose execute_code into sequential pipelines with optional parallel fan-out stages using execute_pipe. Use for multi-step data processing, parallel independent operations, and ETL workflows.
|
description: Use execute_code pipeline stages for sequential data processing, parallel fan-out, and ETL workflows.
|
||||||
---
|
---
|
||||||
|
|
||||||
# Execute Pipeline Patterns
|
# Execute Pipeline Patterns
|
||||||
|
|
||||||
## Principle: smart data, dumb code
|
## Principle: smart data, dumb code
|
||||||
|
|
||||||
The primitives stay simple. Complexity lives in the data schema and the skill (here), not in the tools themselves. Design pipelines around data shape, not around clever tool use.
|
The primitive stays simple. Complexity lives in the data schema and the skill (here), not in the tool itself. Design pipelines around data shape, not around clever tool use.
|
||||||
|
|
||||||
## Tools
|
## execute_code: the unified primitive
|
||||||
|
|
||||||
| Tool | What it does |
|
`execute_code` accepts a `steps` array of pipeline stages. Stages run sequentially, each stage's stdout fed to the next stage's stdin. A single stage returning raw output is the degenerate case.
|
||||||
|---|---|
|
|
||||||
| `execute_code` | Run one or more steps. Steps may be inline `{code}` or named `{tool, args}` scripts. Multiple steps run in parallel; single step returns raw output. |
|
|
||||||
| `execute_pipe` | Chain stages sequentially, piping stdout → stdin. Each stage: `{code}`, `{tool, args}`, or `{parallel: [steps]}`. |
|
|
||||||
|
|
||||||
## execute_code: parallel fan-out
|
Each stage is one of:
|
||||||
|
- `{code, language}` — inline code (default: bash)
|
||||||
Single step — raw output:
|
- `{tool, args}` — named script from `ollie/t`; language detected from shebang
|
||||||
```json
|
- `{parallel: [steps]}` — concurrent fan-out; outputs concatenated in submission order, fed to next stage
|
||||||
{"steps": [{"code": "wc -l file.txt"}]}
|
|
||||||
```
|
|
||||||
|
|
||||||
Multiple steps — run concurrently, results under `=== step N ===` headers:
|
|
||||||
```json
|
|
||||||
{"steps": [
|
|
||||||
{"code": "wc -l a.txt"},
|
|
||||||
{"code": "wc -l b.txt"},
|
|
||||||
{"tool": "count-lines.sh", "args": ["c.txt"]}
|
|
||||||
]}
|
|
||||||
```
|
|
||||||
|
|
||||||
Tool steps and inline code steps can be mixed freely. Tool steps are trusted (no pattern validation); inline code steps are validated before execution.
|
Tool steps and inline code steps can be mixed freely. Tool steps are trusted (no pattern validation); inline code steps are validated before execution.
|
||||||
|
|
||||||
## Language selection
|
## Language selection
|
||||||
|
|
||||||
How the interpreter is chosen depends on step type:
|
|
||||||
|
|
||||||
| Step type | How language is determined |
|
| Step type | How language is determined |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `{code}` | `language` field; defaults to `bash` if absent |
|
| `{code}` | `language` field; defaults to `bash` if absent |
|
||||||
| `{tool}` | shebang line in the script (`#!/usr/bin/env python3`, etc.); `language` field is ignored |
|
| `{tool}` | shebang line in the script; `language` field is ignored |
|
||||||
|
|
||||||
Supported `language` values for inline code: `bash`, `python3`, `perl`, `lua`, `awk`, `sed`, `jq`, `ed`, `expect`, `bc`.
|
Supported `language` values for inline code: `bash`, `python3`, `perl`, `lua`, `awk`, `sed`, `jq`, `ed`, `expect`, `bc`.
|
||||||
|
|
||||||
Do not set `language` on a tool step — it has no effect and will mislead readers of the call.
|
## Single stage (degenerate case)
|
||||||
|
|
||||||
Results are always in submission order, regardless of completion order.
|
|
||||||
|
|
||||||
## execute_pipe: sequential pipeline
|
|
||||||
|
|
||||||
Each stage's stdout becomes the next stage's stdin.
|
|
||||||
|
|
||||||
|
Raw output, no headers:
|
||||||
```json
|
```json
|
||||||
{"pipe": [
|
{"steps": [{"code": "wc -l file.txt"}]}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Sequential pipeline
|
||||||
|
|
||||||
|
Each stage's stdout becomes the next stage's stdin:
|
||||||
|
```json
|
||||||
|
{"steps": [
|
||||||
{"code": "grep ERROR app.log"},
|
{"code": "grep ERROR app.log"},
|
||||||
{"code": "sort"},
|
{"code": "sort"},
|
||||||
{"code": "uniq -c"}
|
{"code": "uniq -c"}
|
||||||
|
|
@ -64,18 +50,18 @@ Each stage's stdout becomes the next stage's stdin.
|
||||||
|
|
||||||
Tool stage:
|
Tool stage:
|
||||||
```json
|
```json
|
||||||
{"pipe": [
|
{"steps": [
|
||||||
{"tool": "fetch-metrics.sh", "args": ["--last=1h"]},
|
{"tool": "fetch-metrics.sh", "args": ["--last=1h"]},
|
||||||
{"code": "jq '.[] | select(.latency > 500)'"}
|
{"code": "jq '.[] | select(.latency > 500)'"}
|
||||||
]}
|
]}
|
||||||
```
|
```
|
||||||
|
|
||||||
## execute_pipe: parallel stage
|
## Parallel stage
|
||||||
|
|
||||||
A `{parallel: [...]}` stage fans out concurrently. Each step is `{code}` or `{tool, args}`. Outputs are concatenated in submission order and fed as a single stream to the next stage.
|
A `{parallel: [...]}` stage fans out concurrently. Outputs are concatenated in submission order and fed as a single stream to the next stage:
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{"pipe": [
|
{"steps": [
|
||||||
{"parallel": [
|
{"parallel": [
|
||||||
{"tool": "fetch-app1.sh"},
|
{"tool": "fetch-app1.sh"},
|
||||||
{"tool": "fetch-app2.sh"},
|
{"tool": "fetch-app2.sh"},
|
||||||
|
|
@ -90,10 +76,10 @@ A `{parallel: [...]}` stage fans out concurrently. Each step is `{code}` or `{to
|
||||||
|
|
||||||
## ETL pattern: disparate schemas
|
## ETL pattern: disparate schemas
|
||||||
|
|
||||||
When parallel steps produce different output schemas, normalize each before merging. Use an inner pipe as the transform stage.
|
When parallel steps produce different output schemas, normalize each before merging:
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{"pipe": [
|
{"steps": [
|
||||||
{"parallel": [
|
{"parallel": [
|
||||||
{"code": "jq -r '.name' users.json"},
|
{"code": "jq -r '.name' users.json"},
|
||||||
{"code": "awk -F: '{print $1}' /etc/passwd"}
|
{"code": "awk -F: '{print $1}' /etc/passwd"}
|
||||||
|
|
@ -103,9 +89,7 @@ When parallel steps produce different output schemas, normalize each before merg
|
||||||
]}
|
]}
|
||||||
```
|
```
|
||||||
|
|
||||||
Here both parallel steps produce newline-delimited names (same schema). The `sort -u` stage then deduplicates the merged stream.
|
If normalization requires multiple steps, extract it as a named tool in `ollie/t`.
|
||||||
|
|
||||||
If normalization requires multiple steps, extract it as a named tool in `ollie/t` and reference it with `{tool}`.
|
|
||||||
|
|
||||||
## Timeout
|
## Timeout
|
||||||
|
|
||||||
|
|
@ -115,8 +99,8 @@ If normalization requires multiple steps, extract it as a named tool in `ollie/t
|
||||||
|
|
||||||
| Situation | Pattern |
|
| Situation | Pattern |
|
||||||
|---|---|
|
|---|---|
|
||||||
| One independent operation | `execute_code` single step |
|
| One independent operation | single stage |
|
||||||
| N independent operations, same output schema | `execute_code` parallel steps, or `execute_pipe` parallel stage |
|
| N independent operations, same output schema | `{parallel: [...]}` stage |
|
||||||
| N operations feeding one transform | `execute_pipe` with parallel first stage |
|
| N operations feeding one transform | parallel first stage, then sequential |
|
||||||
| Multi-step data transform | `execute_pipe` sequential stages |
|
| Multi-step data transform | sequential stages |
|
||||||
| Reusable transform logic | `{tool}` step in `execute_code`, or named script in `ollie/t` as a pipe stage |
|
| Reusable transform logic | `{tool}` stage referencing a named script in `ollie/t` |
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue