Update docs: execute_code is the only MCP tool
- Rewrite code execution docs to reflect single-tool model - Remove obsolete migration guide and testing doc - Update README, kiro-cli README, BEADS.md MCP sections - Remove all 'direct tool calls' / 'legacy tools' framing - Update sandbox-config diagram: roles→sandbox, override→merge - Add 9P security mitigation: run locally on trusted network - Regenerate sandbox-config.svg
This commit is contained in:
Levi Neely
parent
4fa1e0cc7d
commit
c5b96bcac1
23
BEADS.md
23
BEADS.md
|
|
@ -204,19 +204,18 @@ echo 'claim bd-xyz' > anvillm/beads/ctl
|
|||
|
||||
## MCP Integration
|
||||
|
||||
anvilmcp exposes beads operations as MCP tools (calls `9p write anvillm/beads/ctl`):
|
||||
Beads are accessed via `execute_code` through 9P:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "create_bead",
|
||||
"inputSchema": {
|
||||
"properties": {
|
||||
"title": {"type": "string"},
|
||||
"role": {"type": "string"},
|
||||
"description": {"type": "string"}
|
||||
}
|
||||
}
|
||||
}
|
||||
```bash
|
||||
# Create a bead
|
||||
echo 'new "Implement login" "Add JWT auth"' | 9p write anvillm/beads/ctl
|
||||
|
||||
# List beads
|
||||
9p read anvillm/beads/list | jq '.[] | "\(.id): \(.title)"'
|
||||
|
||||
# Claim and complete
|
||||
echo 'claim bd-a1b2' | 9p write anvillm/beads/ctl
|
||||
echo 'complete bd-a1b2' | 9p write anvillm/beads/ctl
|
||||
```
|
||||
|
||||
## Benefits
|
||||
|
|
|
|||
14
README.md
14
README.md
|
|
@ -128,7 +128,7 @@ Templates use `{VARNAME}` syntax. Any environment variable can be referenced.
|
|||
|
||||
<p align="center"><img src="docs/diagrams/sandbox-config.svg?v=2" width="250"></p>
|
||||
|
||||
- Default role: `roles/default.yaml`
|
||||
- Default sandbox: `sandbox/default.yaml`
|
||||
|
||||
```yaml
|
||||
network: {enabled: true, unrestricted: true}
|
||||
|
|
@ -348,17 +348,15 @@ Conductor analyzes dependencies, spawns specialized bots, and delegates work in
|
|||
./kiro-cli/install-mcp.sh # Adds to ~/.kiro/settings/cli.json
|
||||
```
|
||||
|
||||
**Tool:** `execute_code` - Execute bash scripts in isolated subprocess
|
||||
**Tool:** `execute_code` — the only tool needed. Executes bash scripts in an isolated subprocess (sandboxed via landrun). Agents discover and invoke all AnviLLM functionality through 9P and helper scripts:
|
||||
|
||||
Enables code execution pattern for 80-99% token reduction:
|
||||
- Progressive tool discovery via 9P filesystem
|
||||
- Data processing in subprocess (not in context)
|
||||
- Loops, conditionals, and complex workflows
|
||||
```sh
|
||||
9p read anvillm/tools/anvilmcp/check_inbox.sh # discover scripts
|
||||
bash <(9p read tools/send_message.sh) ... # invoke them
|
||||
```
|
||||
|
||||
See [Code Execution User Guide](docs/code-execution-user-guide.md) for details.
|
||||
|
||||
**Legacy tools removed**: Direct tool calls (check_inbox, send_message, etc.) replaced by bash scripts accessible via `9p read anvillm/tools/anvilmcp/*.sh`
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Problem | Solution |
|
||||
|
|
|
|||
|
|
@ -1,379 +0,0 @@
|
|||
# Migration Guide: Direct Tools to Code Execution
|
||||
|
||||
## Overview
|
||||
|
||||
This guide shows how to migrate from direct MCP tool calls to the code execution pattern. The migration reduces token consumption by 80-99% while enabling more powerful workflows.
|
||||
|
||||
## Key Differences
|
||||
|
||||
| Aspect | Direct Tools | Code Execution |
|
||||
|--------|--------------|----------------|
|
||||
| Tool loading | All upfront | Progressive discovery |
|
||||
| Data flow | Through context | In subprocess |
|
||||
| Control flow | Alternating calls | Loops, conditionals |
|
||||
| Token cost | High (linear with data) | Low (constant) |
|
||||
| Privacy | Data in context | Data isolated |
|
||||
|
||||
## Migration Patterns
|
||||
|
||||
### Pattern 1: Single Tool Call
|
||||
|
||||
**Before**:
|
||||
```
|
||||
Agent: TOOL CALL check_inbox(agent_id: "82b93a8a")
|
||||
MCP: { "content": [{"type": "text", "text": "[Message 1]\n[Message 2]\n..."}] }
|
||||
```
|
||||
|
||||
**After**:
|
||||
```
|
||||
execute_code with code: "bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) 82b93a8a"
|
||||
```
|
||||
|
||||
**Token savings**: Minimal for single calls, but enables further optimization
|
||||
|
||||
### Pattern 2: Sequential Tool Calls
|
||||
|
||||
**Before**:
|
||||
```
|
||||
Agent: TOOL CALL check_inbox(agent_id: "agent-1")
|
||||
MCP: { "content": [...] } // 5,000 tokens
|
||||
Agent: TOOL CALL check_inbox(agent_id: "agent-2")
|
||||
MCP: { "content": [...] } // 5,000 tokens
|
||||
Agent: TOOL CALL check_inbox(agent_id: "agent-3")
|
||||
MCP: { "content": [...] } // 5,000 tokens
|
||||
```
|
||||
|
||||
**Total**: 15,000 tokens
|
||||
|
||||
**After**:
|
||||
```
|
||||
execute_code with code: "
|
||||
for agent_id in agent-1 agent-2 agent-3; do
|
||||
bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) \$agent_id | wc -l
|
||||
done | awk '{sum+=\$1} END {print \"Total messages:\", sum}'
|
||||
"
|
||||
```
|
||||
|
||||
**Output**: "Total messages: 47" (~10 tokens)
|
||||
|
||||
**Token savings**: 99.9% (10 tokens vs 15,000 tokens)
|
||||
|
||||
### Pattern 3: Filtering Large Results
|
||||
|
||||
**Before**:
|
||||
```
|
||||
Agent: TOOL CALL get_spreadsheet(sheet_id: "abc123")
|
||||
MCP: { "content": [...] } // 200,000 tokens (10,000 rows)
|
||||
Agent: [Processes in context, filters to 47 pending orders]
|
||||
```
|
||||
|
||||
**Total**: 200,000+ tokens
|
||||
|
||||
**After**:
|
||||
```
|
||||
execute_code with code: "
|
||||
rows=\$(9p read anvillm/sheets/abc123)
|
||||
pending=\$(echo \"\$rows\" | jq '[.[] | select(.Status == \"pending\")]')
|
||||
count=\$(echo \"\$pending\" | jq 'length')
|
||||
total=\$(echo \"\$rows\" | jq 'length')
|
||||
echo \"Found \$count pending orders out of \$total total\"
|
||||
"
|
||||
```
|
||||
|
||||
**Output**: "Found 47 pending orders out of 10000 total" (~15 tokens)
|
||||
|
||||
**Token savings**: 99.99% (15 tokens vs 200,000 tokens)
|
||||
|
||||
### Pattern 4: Polling Loops
|
||||
|
||||
**Before**:
|
||||
```
|
||||
Agent: TOOL CALL check_status(job_id: "job-123")
|
||||
MCP: { "status": "running" }
|
||||
Agent: [Waits, then calls again]
|
||||
Agent: TOOL CALL check_status(job_id: "job-123")
|
||||
MCP: { "status": "running" }
|
||||
[Repeats 10 times]
|
||||
```
|
||||
|
||||
**Total**: ~5,000 tokens (10 calls × 500 tokens)
|
||||
|
||||
**After**:
|
||||
```
|
||||
execute_code with code: "
|
||||
job_id=job-123
|
||||
status=running
|
||||
attempts=0
|
||||
|
||||
while [ \"\$status\" = \"running\" ] && [ \$attempts -lt 10 ]; do
|
||||
status=\$(9p read anvillm/jobs/\$job_id/status)
|
||||
if [ \"\$status\" = \"running\" ]; then
|
||||
sleep 5
|
||||
fi
|
||||
attempts=\$((attempts + 1))
|
||||
done
|
||||
|
||||
echo \"Job \$job_id status: \$status (checked \$attempts times)\"
|
||||
"
|
||||
```
|
||||
|
||||
**Output**: "Job job-123 status: completed (checked 7 times)" (~15 tokens)
|
||||
|
||||
**Token savings**: 99.7% (15 tokens vs 5,000 tokens)
|
||||
|
||||
### Pattern 5: Multi-Step Data Pipeline
|
||||
|
||||
**Before**:
|
||||
```
|
||||
Agent: TOOL CALL check_inbox(agent_id: "82b93a8a")
|
||||
MCP: { "content": [...] } // 10,000 tokens
|
||||
Agent: [Identifies 5 urgent messages]
|
||||
Agent: TOOL CALL send_message(to: "manager", body: "5 urgent messages")
|
||||
MCP: { "content": "Message sent" }
|
||||
Agent: TOOL CALL set_state(agent_id: "82b93a8a", state: "idle")
|
||||
MCP: { "content": "State updated" }
|
||||
```
|
||||
|
||||
**Total**: ~11,000 tokens
|
||||
|
||||
**After**:
|
||||
```
|
||||
execute_code with code: "
|
||||
inbox=\$(bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) 82b93a8a)
|
||||
urgent_count=\$(echo \"\$inbox\" | grep -c URGENT || echo 0)
|
||||
|
||||
if [ \$urgent_count -gt 0 ]; then
|
||||
bash <(9p read anvillm/tools/anvilmcp/send_message.sh) 82b93a8a manager NOTIFICATION 'Urgent messages' \"\$urgent_count urgent messages in inbox\"
|
||||
fi
|
||||
|
||||
bash <(9p read anvillm/tools/anvilmcp/set_state.sh) 82b93a8a idle
|
||||
echo \"Processed inbox: \$urgent_count urgent messages\"
|
||||
"
|
||||
```
|
||||
|
||||
**Output**: "Processed inbox: 5 urgent messages" (~10 tokens)
|
||||
|
||||
**Token savings**: 99.9% (10 tokens vs 11,000 tokens)
|
||||
|
||||
## Tool-Specific Migrations
|
||||
|
||||
### check_inbox
|
||||
|
||||
**Before**:
|
||||
```
|
||||
TOOL CALL check_inbox(agent_id: "82b93a8a")
|
||||
```
|
||||
|
||||
**After**:
|
||||
```
|
||||
execute_code with code: "bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) 82b93a8a"
|
||||
```
|
||||
|
||||
**Or use 9P directly**:
|
||||
```
|
||||
execute_code with code: "9p read anvillm/inbox/82b93a8a/json"
|
||||
```
|
||||
|
||||
### send_message
|
||||
|
||||
**Before**:
|
||||
```
|
||||
TOOL CALL send_message(from: "agent-1", to: "agent-2", type: "PROMPT_REQUEST", subject: "Task", body: "Do X")
|
||||
```
|
||||
|
||||
**After**:
|
||||
```bash
|
||||
bash <(9p read anvillm/tools/anvilmcp/send_message.sh) agent-1 agent-2 PROMPT_REQUEST 'Task' 'Do X'
|
||||
```
|
||||
|
||||
### list_sessions
|
||||
|
||||
**Before**:
|
||||
```
|
||||
TOOL CALL list_sessions()
|
||||
```
|
||||
|
||||
**After**:
|
||||
```
|
||||
execute_code with code: "bash <(9p read anvillm/tools/anvilmcp/list_sessions.sh)"
|
||||
```
|
||||
|
||||
### set_state
|
||||
|
||||
**Before**:
|
||||
```
|
||||
TOOL CALL set_state(agent_id: "82b93a8a", state: "running")
|
||||
```
|
||||
|
||||
**After**:
|
||||
```
|
||||
execute_code with code: "bash <(9p read anvillm/tools/anvilmcp/set_state.sh) 82b93a8a running"
|
||||
```
|
||||
|
||||
### list_skills
|
||||
|
||||
**Before**:
|
||||
```
|
||||
TOOL CALL list_skills()
|
||||
```
|
||||
|
||||
**After**:
|
||||
```
|
||||
execute_code with code: "bash <(9p read anvillm/tools/anvilmcp/list_skills.sh) | jq -r '.[].name' | paste -sd, -"
|
||||
```
|
||||
|
||||
## Compatibility Notes
|
||||
|
||||
### When to Use Direct Tools
|
||||
|
||||
Direct tools are better for:
|
||||
|
||||
1. **Single operations**: Simple one-off calls
|
||||
2. **Small data**: Results under 1,000 tokens
|
||||
3. **Simple logic**: No conditionals or loops needed
|
||||
4. **Debugging**: When you need to see intermediate results
|
||||
|
||||
### When to Use Code Execution
|
||||
|
||||
Code execution is better for:
|
||||
|
||||
1. **Multiple tool calls**: Any workflow with 2+ tool calls
|
||||
2. **Large data**: Any operation returning >1,000 tokens
|
||||
3. **Filtering/aggregation**: Processing data to extract summaries
|
||||
4. **Loops**: Polling, batch processing, retries
|
||||
5. **Privacy**: Operations involving PII or sensitive data
|
||||
|
||||
## Migration Checklist
|
||||
|
||||
- [ ] Identify workflows with multiple tool calls
|
||||
- [ ] Identify operations with large data returns
|
||||
- [ ] Convert sequential calls to loops
|
||||
- [ ] Add filtering/aggregation in subprocess
|
||||
- [ ] Return summaries instead of raw data
|
||||
- [ ] Add error handling (try/catch)
|
||||
- [ ] Test in isolated subprocess
|
||||
- [ ] Measure token savings
|
||||
- [ ] Update agent prompts with examples
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
### Pitfall 1: Returning Raw Data
|
||||
|
||||
**Bad**:
|
||||
```bash
|
||||
result=$(bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) 82b93a8a)
|
||||
echo "$result" # Could be 50,000 tokens
|
||||
```
|
||||
|
||||
**Good**:
|
||||
```bash
|
||||
result=$(bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) 82b93a8a)
|
||||
count=$(echo "$result" | wc -l)
|
||||
echo "Inbox has $count messages" # ~10 tokens
|
||||
```
|
||||
|
||||
### Pitfall 2: Not Using Loops
|
||||
|
||||
**Bad**:
|
||||
```bash
|
||||
result1=$(bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) agent-1)
|
||||
echo "$result1"
|
||||
result2=$(bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) agent-2)
|
||||
echo "$result2"
|
||||
result3=$(bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) agent-3)
|
||||
echo "$result3"
|
||||
```
|
||||
|
||||
**Good**:
|
||||
```bash
|
||||
for agent_id in agent-1 agent-2 agent-3; do
|
||||
result=$(bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) "$agent_id")
|
||||
count=$(echo "$result" | wc -l)
|
||||
echo "$agent_id: $count messages"
|
||||
done
|
||||
```
|
||||
|
||||
### Pitfall 3: Ignoring Errors
|
||||
|
||||
**Bad**:
|
||||
```bash
|
||||
result=$(bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) invalid-id)
|
||||
echo "$result" # May fail silently
|
||||
```
|
||||
|
||||
**Good**:
|
||||
```bash
|
||||
if result=$(bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) invalid-id 2>&1); then
|
||||
echo "$result"
|
||||
else
|
||||
echo "Failed to read inbox: $result" >&2
|
||||
fi
|
||||
```
|
||||
|
||||
### Pitfall 4: Re-reading Tool Definitions
|
||||
|
||||
**Bad**:
|
||||
```bash
|
||||
for i in {1..10}; do
|
||||
tool=$(9p read anvillm/tools/anvilmcp/check_inbox.sh)
|
||||
# Parse and use...
|
||||
done
|
||||
```
|
||||
|
||||
**Good**:
|
||||
```bash
|
||||
# Call tool directly each time
|
||||
for i in {1..10}; do
|
||||
bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) 82b93a8a
|
||||
done
|
||||
```
|
||||
|
||||
## Performance Comparison
|
||||
|
||||
| Workflow | Direct Tools | Code Execution | Savings |
|
||||
|----------|--------------|----------------|---------|
|
||||
| Single small call | 500 tokens | 400 tokens | 20% |
|
||||
| 3 sequential calls | 15,000 tokens | 150 tokens | 99% |
|
||||
| Filter 10k rows | 200,000 tokens | 100 tokens | 99.95% |
|
||||
| Poll 10 times | 5,000 tokens | 200 tokens | 96% |
|
||||
| Multi-step pipeline | 50,000 tokens | 500 tokens | 99% |
|
||||
|
||||
## Rollout Strategy
|
||||
|
||||
### Phase 1: Pilot (Week 1)
|
||||
- Migrate 1-2 simple workflows
|
||||
- Measure token savings
|
||||
- Identify issues
|
||||
|
||||
### Phase 2: Expand (Week 2-3)
|
||||
- Migrate high-volume workflows
|
||||
- Update agent prompts
|
||||
- Train team on patterns
|
||||
|
||||
### Phase 3: Optimize (Week 4+)
|
||||
- Refine implementations
|
||||
- Add more examples
|
||||
- Monitor metrics
|
||||
|
||||
## Support
|
||||
|
||||
For questions or issues:
|
||||
- Review [User Guide](./code-execution-user-guide.md)
|
||||
- Check [Security Documentation](./code-execution-security.md)
|
||||
- See [Example Workflows](./code-execution-examples.md)
|
||||
|
||||
## Token Savings Calculator
|
||||
|
||||
Estimate your savings:
|
||||
|
||||
```
|
||||
Current tokens = (tool_count × 90) + (data_size_kb × 250)
|
||||
Code execution tokens = 30 + (summary_size_kb × 250)
|
||||
Savings = (Current - Code execution) / Current × 100%
|
||||
```
|
||||
|
||||
Example:
|
||||
- 100 tools, 10 KB data
|
||||
- Current: (100 × 90) + (10 × 250) = 11,500 tokens
|
||||
- Code execution: 30 + (0.01 × 250) = 33 tokens
|
||||
- Savings: 99.7%
|
||||
|
|
@ -2,191 +2,59 @@
|
|||
|
||||
## Overview
|
||||
|
||||
The code execution pattern reduces token consumption by 80-99% by executing data transformations and tool interactions in a isolated subprocess instead of passing all data through the model's context window.
|
||||
`execute_code` is the single MCP tool exposed by `anvilmcp`. It executes bash scripts in an isolated subprocess sandboxed via landrun. All AnviLLM functionality — messaging, sessions, beads, state — is accessed through 9P and helper scripts within these subprocesses.
|
||||
|
||||
Reference: [Code execution with MCP: Building more efficient agents](https://www.anthropic.com/engineering/code-execution-with-mcp)
|
||||
## Tool Discovery
|
||||
|
||||
## Progressive Discovery Workflow
|
||||
|
||||
Instead of loading all tool definitions upfront, discover tools on-demand:
|
||||
|
||||
1. **List available tools**
|
||||
```bash
|
||||
9p ls anvillm/tools/anvilmcp/
|
||||
```
|
||||
|
||||
2. **Read specific tool definitions**
|
||||
```bash
|
||||
9p read anvillm/tools/anvilmcp/beads.sh
|
||||
```
|
||||
|
||||
3. **Import and use in bash**
|
||||
```bash
|
||||
# Call tools from anvillm/tools/anvilmcp/beads';
|
||||
|
||||
const allBeads = await beads.list();
|
||||
const open = allBeads.filter(b => b.status === 'open');
|
||||
console.log(`Found ${open.length} open beads`);
|
||||
```
|
||||
|
||||
## Data Transformation Examples
|
||||
|
||||
### Filtering Large Datasets
|
||||
|
||||
Instead of loading 10,000 rows into context:
|
||||
Discover available helper scripts on-demand via 9P:
|
||||
|
||||
```bash
|
||||
// Load data in subprocess
|
||||
const allRows = await beads.list();
|
||||
|
||||
// Filter in subprocess
|
||||
const pending = allRows.filter(row => row.status === 'open' && row.priority <= 2);
|
||||
|
||||
// Only return summary to context
|
||||
console.log(`Found ${pending.length} high-priority open beads`);
|
||||
console.log(pending.slice(0, 5)); // Show first 5 for review
|
||||
9p ls anvillm/tools/anvilmcp/
|
||||
```
|
||||
|
||||
### Polling Loops
|
||||
|
||||
Execute loops in subprocess instead of alternating tool calls:
|
||||
Read and invoke a specific script:
|
||||
|
||||
```bash
|
||||
let found = false;
|
||||
while (!found) {
|
||||
const beads = await beads.list();
|
||||
found = beads.some(b => b.id === 'bd-abc' && b.status === 'closed');
|
||||
if (!found) await new Promise(r => setTimeout(r, 5000));
|
||||
}
|
||||
console.log('Bead bd-abc completed');
|
||||
bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) <agent-id>
|
||||
```
|
||||
|
||||
### Aggregations
|
||||
## Examples
|
||||
|
||||
Compute statistics without context bloat:
|
||||
### Messaging
|
||||
|
||||
```bash
|
||||
const allBeads = await beads.list();
|
||||
|
||||
const stats = {
|
||||
total: allBeads.length,
|
||||
byStatus: {},
|
||||
byPriority: {}
|
||||
};
|
||||
|
||||
for (const bead of allBeads) {
|
||||
stats.byStatus[bead.status] = (stats.byStatus[bead.status] || 0) + 1;
|
||||
stats.byPriority[bead.priority] = (stats.byPriority[bead.priority] || 0) + 1;
|
||||
}
|
||||
|
||||
console.log(JSON.stringify(stats, null, 2));
|
||||
bash <(9p read anvillm/tools/anvilmcp/send_message.sh) \
|
||||
--to user --type PROMPT_RESPONSE --subject "Done" --body "Task complete"
|
||||
```
|
||||
|
||||
## Token Savings
|
||||
|
||||
- **Without code execution**: Load all tool definitions (150k tokens) + all intermediate results
|
||||
- **With code execution**: Load only needed tools (~2k tokens) + filtered results
|
||||
- **Savings**: 80-99% reduction in token consumption
|
||||
|
||||
## When to Use
|
||||
|
||||
Use code execution when:
|
||||
- Working with large datasets (>100 rows)
|
||||
- Performing multiple transformations on the same data
|
||||
- Implementing loops or polling
|
||||
- Computing aggregations or statistics
|
||||
- Filtering data before presenting to user
|
||||
|
||||
Use direct tool calls when:
|
||||
- Single, simple operations
|
||||
- Small result sets
|
||||
- No data transformation needed
|
||||
|
||||
## Extended Examples
|
||||
|
||||
### Multi-Source Data Joins
|
||||
|
||||
Join data from multiple sources without loading everything into context:
|
||||
### Beads
|
||||
|
||||
```bash
|
||||
# Call tools from anvillm/tools/anvilmcp/beads';
|
||||
# Call tools from anvillm/tools/anvilmcp/kb';
|
||||
|
||||
// Load data from both sources
|
||||
const allBeads = await beads.list();
|
||||
const kbEntries = await kb.search('implementation');
|
||||
|
||||
// Join in subprocess
|
||||
const enriched = allBeads
|
||||
.filter(b => b.status === 'open')
|
||||
.map(bead => {
|
||||
const relatedKB = kbEntries.filter(kb =>
|
||||
kb.content.includes(bead.id) ||
|
||||
bead.description.includes(kb.title)
|
||||
);
|
||||
return { ...bead, kbReferences: relatedKB.length };
|
||||
})
|
||||
.filter(b => b.kbReferences > 0);
|
||||
|
||||
console.log(`Found ${enriched.length} beads with KB references`);
|
||||
console.log(enriched.slice(0, 3));
|
||||
echo 'new "Implement login" "Add JWT auth"' | 9p write anvillm/beads/ctl
|
||||
9p read anvillm/beads/ready | jq -r '.[] | "\(.id): \(.title)"'
|
||||
```
|
||||
|
||||
### Batch Operations with Progress Tracking
|
||||
|
||||
Process many items without flooding context:
|
||||
### Polling
|
||||
|
||||
```bash
|
||||
const beads = await beads.query({ status: 'open', priority: 1 });
|
||||
|
||||
let processed = 0;
|
||||
let errors = [];
|
||||
|
||||
for (const bead of beads) {
|
||||
try {
|
||||
await beads.update(bead.id, { priority: 2 });
|
||||
processed++;
|
||||
} catch (e) {
|
||||
errors.push({ id: bead.id, error: e.message });
|
||||
}
|
||||
|
||||
if (processed % 10 === 0) {
|
||||
console.log(`Processed ${processed}/${beads.length}`);
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`Complete: ${processed} updated, ${errors.length} errors`);
|
||||
if (errors.length > 0) {
|
||||
console.log('Errors:', errors.slice(0, 5));
|
||||
}
|
||||
while true; do
|
||||
state=$(9p read anvillm/$ID/state)
|
||||
[ "$state" = "idle" ] && break
|
||||
sleep 5
|
||||
done
|
||||
echo "Session $ID is idle"
|
||||
```
|
||||
|
||||
### Complex Filtering with Multiple Criteria
|
||||
|
||||
Apply complex business logic in subprocess:
|
||||
### Filtering
|
||||
|
||||
```bash
|
||||
const allBeads = await beads.list();
|
||||
|
||||
const candidates = allBeads.filter(bead => {
|
||||
// Multiple conditions
|
||||
const isHighPriority = bead.priority <= 2;
|
||||
const isStale = new Date() - new Date(bead.updated_at) > 7 * 24 * 60 * 60 * 1000;
|
||||
const hasNoAssignee = !bead.assignee;
|
||||
const isNotBlocked = !bead.blockers || bead.blockers.length === 0;
|
||||
|
||||
return isHighPriority && isStale && hasNoAssignee && isNotBlocked;
|
||||
});
|
||||
|
||||
// Group by priority
|
||||
const grouped = candidates.reduce((acc, bead) => {
|
||||
acc[bead.priority] = acc[bead.priority] || [];
|
||||
acc[bead.priority].push(bead);
|
||||
return acc;
|
||||
}, {});
|
||||
|
||||
console.log('Stale unassigned beads by priority:');
|
||||
for (const [priority, beads] of Object.entries(grouped)) {
|
||||
console.log(` Priority ${priority}: ${beads.length} beads`);
|
||||
}
|
||||
9p read anvillm/beads/list | jq '[.[] | select(.status == "open" and .priority <= 2)]'
|
||||
```
|
||||
|
||||
## Why One Tool
|
||||
|
||||
A single `execute_code` tool means:
|
||||
- No upfront tool-definition loading (saves tokens)
|
||||
- Data stays in the subprocess (not in context)
|
||||
- Loops, conditionals, and pipes work naturally
|
||||
- Progressive discovery — only read what you need
|
||||
|
|
|
|||
|
|
@ -125,7 +125,7 @@ Validation is basic pattern matching. Not comprehensive - relies on sandbox for
|
|||
|
||||
**Scenario**: Sensitive data leaked from subprocess
|
||||
|
||||
**Risk**: LOW (Improved from direct tools)
|
||||
**Risk**: LOW
|
||||
|
||||
**Mitigations**:
|
||||
1. Data stays in subprocess (not in context)
|
||||
|
|
@ -133,20 +133,22 @@ Validation is basic pattern matching. Not comprehensive - relies on sandbox for
|
|||
3. No network access (unless explicitly granted)
|
||||
4. 9P socket access controlled by outer sandbox
|
||||
|
||||
**Residual risk**: Data could be written to workspace, but workspace is deleted. Data could be sent via 9P to other agents - same risk as direct tools.
|
||||
**Residual risk**: Data could be written to workspace, but workspace is deleted. Data could be sent via 9P to other agents.
|
||||
|
||||
### Threat 5: 9P Socket Access
|
||||
|
||||
**Scenario**: Subprocess abuses 9P access to read/write unauthorized data
|
||||
|
||||
**Risk**: MEDIUM (Same as direct tools)
|
||||
**Risk**: MEDIUM
|
||||
|
||||
**Mitigations**:
|
||||
1. 9P socket permissions (Unix socket, user-only)
|
||||
2. anvilsrv enforces access control
|
||||
3. Audit logging in anvilsrv
|
||||
|
||||
**Residual risk**: If subprocess has 9P access, it has same capabilities as direct tool calls. This is by design - code execution is meant to replace direct tools.
|
||||
**Residual risk**: If subprocess has 9P access, it has the same capabilities as any 9P client. This is by design.
|
||||
|
||||
**Mitigation**: Run locally on a trusted network. On a single-user system, the only processes with socket access are ones the user launched. Do not expose the 9P mount over the network, or at minimum, implement access control (e.g., firewall).
|
||||
|
||||
### Threat 6: Path Traversal
|
||||
|
||||
|
|
@ -161,25 +163,14 @@ Validation is basic pattern matching. Not comprehensive - relies on sandbox for
|
|||
|
||||
**Residual risk**: None - path traversal blocked at multiple layers.
|
||||
|
||||
## Security Benefits vs Direct Tools
|
||||
## Security Properties
|
||||
|
||||
**Improved**:
|
||||
- Data isolation (subprocess)
|
||||
- Data isolation (subprocess, not in model context)
|
||||
- Privacy preservation (PII doesn't enter context)
|
||||
- Explicit control over data flow
|
||||
- Workspace isolation
|
||||
|
||||
**Same**:
|
||||
- 9P socket access control
|
||||
- Process isolation
|
||||
- Agent trust model
|
||||
|
||||
**Additional Considerations**:
|
||||
- Code validation required
|
||||
- Sandbox configuration complexity
|
||||
- Resource limit tuning
|
||||
|
||||
**Conclusion**: Code execution improves security posture compared to direct tool calls, with acceptable additional complexity.
|
||||
- Process isolation via landrun (Landlock)
|
||||
|
||||
## Configuration
|
||||
|
||||
|
|
@ -389,4 +380,3 @@ All tests should fail safely without compromising the system.
|
|||
- landrun: https://github.com/zouuup/landrun
|
||||
- Anthropic code execution pattern: https://www.anthropic.com/engineering/code-execution-with-mcp
|
||||
- User guide: ./code-execution-user-guide.md
|
||||
- Migration guide: ./code-execution-migration-guide.md
|
||||
|
|
|
|||
|
|
@ -182,7 +182,7 @@ var executionSem = make(chan struct{}, 4) // Max 4 concurrent
|
|||
|
||||
**Description**: Code leaks sensitive data outside the system.
|
||||
|
||||
**Risk Level**: LOW (Improved from direct tools)
|
||||
**Risk Level**: LOW
|
||||
|
||||
**Attack Vectors**:
|
||||
- Network exfiltration (blocked)
|
||||
|
|
@ -219,11 +219,7 @@ for (const email of emails) {
|
|||
console.log(`Processed ${emails.length} emails`); // Summary only
|
||||
```
|
||||
|
||||
**Residual Risk**: LOW - Better isolation than direct tools
|
||||
|
||||
**Comparison to Direct Tools**:
|
||||
- **Direct tools**: All data flows through model context
|
||||
- **Code execution**: Data stays in subprocess, only summaries returned
|
||||
**Residual Risk**: LOW — data stays in subprocess, only summaries returned to the model
|
||||
|
||||
### Threat 5: 9P Socket Access
|
||||
|
||||
|
|
@ -263,9 +259,9 @@ log.Printf("9P: agent=%s op=%s path=%s", agentID, op, path)
|
|||
allowedOps := []string{"read", "write", "ls"}
|
||||
```
|
||||
|
||||
**Residual Risk**: MEDIUM - Same as current system
|
||||
**Residual Risk**: MEDIUM — inherent to any 9P client
|
||||
|
||||
**Note**: This is not a new risk introduced by code execution. The same 9P access exists with direct tool calls.
|
||||
**Mitigation**: Run locally on a trusted network. On a single-user system, the only processes with socket access are ones the user launched — self-inflicted abuse is not a meaningful threat. Do not expose the 9P mount over the network, or at minimum, implement access control (e.g., firewall).
|
||||
|
||||
### Threat 6: Supply Chain
|
||||
|
||||
|
|
@ -307,33 +303,14 @@ import { evil } from "/etc/passwd";
|
|||
|
||||
**Residual Risk**: LOW - No remote imports possible
|
||||
|
||||
## Security Benefits vs Direct Tools
|
||||
## Security Properties
|
||||
|
||||
### Improved Security
|
||||
|
||||
1. **Data Isolation**: Intermediate data stays in subprocess
|
||||
- **Direct tools**: All data flows through model context
|
||||
- **Code execution**: Data processed in subprocess, summaries only
|
||||
|
||||
2. **Privacy Preservation**: PII doesn't enter model context
|
||||
- **Direct tools**: PII visible to model
|
||||
- **Code execution**: PII stays in subprocess
|
||||
|
||||
3. **Explicit Control**: Agent explicitly logs what model sees
|
||||
- **Direct tools**: All tool output visible
|
||||
- **Code execution**: Only console.log output visible
|
||||
|
||||
### Same Security
|
||||
|
||||
1. **9P Access Control**: Unix permissions, audit logging
|
||||
2. **Process Isolation**: Separate processes per agent
|
||||
3. **Authentication**: Agent ID validation
|
||||
|
||||
### Additional Considerations
|
||||
|
||||
1. **Code Validation**: New attack surface (pattern matching)
|
||||
2. **Subprocess Configuration**: Complexity in setup
|
||||
3. **Resource Limits**: Requires tuning and monitoring
|
||||
1. **Data Isolation**: Intermediate data stays in subprocess — only stdout/stderr returned to the model
|
||||
2. **Privacy Preservation**: PII processed in subprocess, never enters model context
|
||||
3. **Explicit Control**: Agent explicitly chooses what to print (and thus what the model sees)
|
||||
4. **9P Access Control**: Unix socket permissions, audit logging
|
||||
5. **Process Isolation**: Separate landrun sandbox per execution
|
||||
6. **Code Validation**: Pattern matching rejects dangerous constructs
|
||||
|
||||
## Subprocess Configuration
|
||||
|
||||
|
|
@ -551,18 +528,6 @@ func TestSubprocessEscape(t *testing.T) {
|
|||
|
||||
## Comparison to Alternatives
|
||||
|
||||
### vs Direct Tool Calls
|
||||
|
||||
| Aspect | Direct Tools | Code Execution |
|
||||
|--------|--------------|----------------|
|
||||
| Data isolation | Poor (all in context) | Good (subprocess) |
|
||||
| Privacy | Poor (PII in context) | Good (PII isolated) |
|
||||
| Attack surface | 9P socket | 9P socket + subprocess |
|
||||
| Resource control | Limited | Good (timeouts, limits) |
|
||||
| Audit trail | 9P logs | 9P + execution logs |
|
||||
|
||||
**Conclusion**: Code execution improves security posture
|
||||
|
||||
### vs Unrestricted Execution
|
||||
|
||||
| Aspect | Unrestricted | Code Execution |
|
||||
|
|
@ -609,8 +574,8 @@ The code execution pattern provides strong security through multiple layers:
|
|||
4. **Resource Limits**: Prevent exhaustion
|
||||
5. **Audit Logging**: Comprehensive trail
|
||||
|
||||
**Security Posture**: Code execution improves security compared to direct tool calls by isolating data and providing explicit control over information flow.
|
||||
**Security Posture**: Data isolation and explicit control over information flow via sandboxed subprocesses.
|
||||
|
||||
**Residual Risks**: Primarily resource exhaustion (MEDIUM) and 9P access control (MEDIUM, same as current system).
|
||||
**Residual Risks**: Primarily resource exhaustion (MEDIUM) and 9P access control (MEDIUM, inherent to any 9P client). Mitigated by running locally on a trusted network — do not expose the 9P mount over the network, or at minimum, implement access control (e.g., firewall).
|
||||
|
||||
**Recommendation**: Proceed with implementation. Security benefits outweigh risks.
|
||||
|
|
|
|||
|
|
@ -2,328 +2,98 @@
|
|||
|
||||
## Overview
|
||||
|
||||
The `execute_code` tool enables agents to write and execute bash scripts in a isolated subprocess. This reduces token consumption by 80-99% compared to direct tool calls by enabling progressive tool discovery and in-subprocess data processing.
|
||||
`execute_code` is the only MCP tool provided by `anvilmcp`. It runs bash scripts in an isolated subprocess sandboxed via landrun. All AnviLLM functionality is accessed through 9P and helper scripts within these subprocesses.
|
||||
|
||||
## Why Use Code Execution?
|
||||
## Tool Discovery
|
||||
|
||||
### Problem 1: Tool Definitions Consume Context
|
||||
|
||||
Loading 100 tools upfront costs ~9,000 tokens (45% of a 200k context window). With code execution, you discover tools progressively:
|
||||
List available helper scripts:
|
||||
|
||||
```bash
|
||||
# List available tools (~30 tokens)
|
||||
9p ls anvillm/tools/anvilmcp
|
||||
|
||||
# Read only the tool you need (~90 tokens)
|
||||
9p read anvillm/tools/anvilmcp/check_inbox.sh
|
||||
9p ls anvillm/tools/anvilmcp/
|
||||
```
|
||||
|
||||
**Savings**: 97% (120 tokens vs 9,000 tokens)
|
||||
Read a script to see its usage:
|
||||
|
||||
### Problem 2: Large Data Flows Through Context
|
||||
|
||||
Processing a 10,000-row spreadsheet directly costs ~200,000 tokens. With code execution, filter in the subprocess:
|
||||
|
||||
```bash
|
||||
all_rows=$(get_sheet "abc123")
|
||||
pending=$(echo "$all_rows" | jq '[.[] | select(.Status == "pending")]')
|
||||
count=$(echo "$pending" | jq 'length')
|
||||
echo "Found $count pending orders"
|
||||
# Agent sees: "Found 47 pending orders" (~10 tokens)
|
||||
```
|
||||
|
||||
**Savings**: 99.995% (10 tokens vs 200,000 tokens)
|
||||
|
||||
## Basic Usage
|
||||
|
||||
### Tool Discovery Workflow
|
||||
|
||||
1. **List available tools**:
|
||||
```bash
|
||||
9p ls anvillm/tools/anvilmcp
|
||||
```
|
||||
|
||||
2. **Read tool documentation**:
|
||||
```bash
|
||||
9p read anvillm/tools/anvilmcp/check_inbox.sh
|
||||
```
|
||||
|
||||
3. **Call tool via MCP**:
|
||||
```bash
|
||||
echo '{"name":"check_inbox","arguments":{"agent_id":"82b93a8a"}}' | 9p write anvillm/mcp/call
|
||||
```
|
||||
|
||||
### Calling 9P Operations
|
||||
|
||||
You can also call 9P operations directly:
|
||||
Invoke a script:
|
||||
|
||||
```bash
|
||||
beads=$(9p read anvillm/beads/list)
|
||||
count=$(echo "$beads" | jq 'length')
|
||||
echo "Found $count beads"
|
||||
bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) <agent-id>
|
||||
```
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Pattern 1: Filtering Large Datasets
|
||||
### Messaging
|
||||
|
||||
```bash
|
||||
# Call check_inbox via MCP
|
||||
result=$(echo '{"name":"check_inbox","arguments":{"agent_id":"82b93a8a"}}' | 9p write anvillm/mcp/call)
|
||||
total=$(echo "$result" | jq -r '.content[0].text' | wc -l)
|
||||
urgent=$(echo "$result" | jq -r '.content[0].text' | grep -c "URGENT")
|
||||
# Check inbox
|
||||
bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) "$AGENT_ID"
|
||||
|
||||
echo "Found $urgent urgent messages out of $total total"
|
||||
```
|
||||
|
||||
### Pattern 2: Polling Loops
|
||||
|
||||
```bash
|
||||
found=false
|
||||
attempts=0
|
||||
|
||||
while [ $attempts -lt 10 ] && [ "$found" = false ]; do
|
||||
sessions=$(anvillm/tools/anvilmcp/list_sessions.sh)
|
||||
if echo "$sessions" | jq -e '.[] | select(.state == "completed")' > /dev/null; then
|
||||
found=true
|
||||
else
|
||||
echo "Attempt $((attempts + 1)): Not ready yet"
|
||||
sleep 5
|
||||
fi
|
||||
attempts=$((attempts + 1))
|
||||
done
|
||||
|
||||
if [ "$found" = true ]; then
|
||||
echo "Session completed"
|
||||
else
|
||||
echo "Timeout after 10 attempts"
|
||||
fi
|
||||
```
|
||||
|
||||
### Pattern 3: Multi-Step Aggregations
|
||||
|
||||
```bash
|
||||
agents=("agent-1" "agent-2" "agent-3")
|
||||
total_messages=0
|
||||
|
||||
for agent_id in "${agents[@]}"; do
|
||||
result=$(anvillm/tools/anvilmcp/check_inbox.sh "$agent_id")
|
||||
count=$(echo "$result" | wc -l)
|
||||
total_messages=$((total_messages + count))
|
||||
done
|
||||
|
||||
echo "Total messages across ${#agents[@]} agents: $total_messages"
|
||||
```
|
||||
|
||||
### Pattern 4: Privacy-Preserving Operations
|
||||
|
||||
Process sensitive data without exposing it to the model:
|
||||
|
||||
```bash
|
||||
# Read PII from one system
|
||||
emails=$(9p read anvillm/data/customer-emails)
|
||||
|
||||
# Write to another system without PII entering context
|
||||
echo "$emails" | jq -c '.[]' | while read -r email; do
|
||||
address=$(echo "$email" | jq -r '.address')
|
||||
name=$(echo "$email" | jq -r '.name')
|
||||
|
||||
echo "{\"email\":\"$address\",\"name\":\"$name\"}" | \
|
||||
9p write anvillm/crm/contacts
|
||||
done
|
||||
|
||||
count=$(echo "$emails" | jq 'length')
|
||||
echo "Processed $count contacts"
|
||||
```
|
||||
|
||||
## Available Tools
|
||||
|
||||
All tools are accessed via 9P commands. Read tool documentation first:
|
||||
|
||||
```bash
|
||||
9p read anvillm/tools/anvilmcp/check_inbox.sh
|
||||
```
|
||||
|
||||
### Communication
|
||||
|
||||
**check_inbox**: Read messages from agent's inbox
|
||||
```bash
|
||||
# List inbox messages
|
||||
9p ls anvillm/AGENT_ID/inbox
|
||||
|
||||
# Read specific message
|
||||
9p read anvillm/AGENT_ID/inbox/MESSAGE_ID.json
|
||||
```
|
||||
|
||||
**send_message**: Send message to another agent or user
|
||||
```bash
|
||||
echo '{"from":"agent-1","to":"agent-2","type":"PROMPT_REQUEST","subject":"Task","body":"Do work"}' | \
|
||||
9p write anvillm/agent-1/mail
|
||||
# Send message
|
||||
bash <(9p read anvillm/tools/anvilmcp/send_message.sh) \
|
||||
--to user --type PROMPT_RESPONSE --subject "Done" --body "Complete"
|
||||
```
|
||||
|
||||
### Session Management
|
||||
|
||||
**list_sessions**: List all active sessions
|
||||
```bash
|
||||
9p read anvillm/list
|
||||
9p read anvillm/$ID/state
|
||||
```
|
||||
|
||||
**set_state**: Set agent state
|
||||
### Beads
|
||||
|
||||
```bash
|
||||
echo "running" | 9p write anvillm/AGENT_ID/state
|
||||
9p read anvillm/beads/list | jq '[.[] | select(.status == "open")]'
|
||||
echo 'new "Task title" "Task description"' | 9p write anvillm/beads/ctl
|
||||
```
|
||||
|
||||
### Skills
|
||||
### Polling
|
||||
|
||||
**list_skills**: List available skills
|
||||
```bash
|
||||
anvillm-skills list
|
||||
while true; do
|
||||
state=$(9p read anvillm/$ID/state)
|
||||
[ "$state" = "idle" ] && break
|
||||
sleep 5
|
||||
done
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
### Filtering
|
||||
|
||||
Always handle errors in your scripts:
|
||||
Process data in the subprocess — only return summaries to the agent:
|
||||
|
||||
```bash
|
||||
if output=$(9p read anvillm/beads/list 2>&1); then
|
||||
echo "$output"
|
||||
else
|
||||
echo "9P command failed: $output" >&2
|
||||
exit 1
|
||||
fi
|
||||
beads=$(9p read anvillm/beads/list)
|
||||
count=$(echo "$beads" | jq '[.[] | select(.priority <= 2)] | length')
|
||||
echo "Found $count high-priority beads"
|
||||
```
|
||||
|
||||
## Subprocess Restrictions
|
||||
|
||||
Your code runs in a isolated subprocess with limited permissions:
|
||||
Scripts run in a landrun sandbox with limited permissions:
|
||||
|
||||
### Allowed
|
||||
- Read from `./anvillm/tools/` directory
|
||||
- Execute `/usr/bin/9p` command
|
||||
- Access environment variables: `NAMESPACE`, `AGENT_ID`
|
||||
- Standard bash utilities (jq, grep, sed, awk)
|
||||
- Read/write to the session's working directory
|
||||
- Read from system paths (`/usr`, `/lib`, `/bin`)
|
||||
- Access to 9P namespace
|
||||
- Standard utilities: `jq`, `grep`, `sed`, `awk`, `9p`
|
||||
- Network access per sandbox config
|
||||
|
||||
### Blocked
|
||||
- Network access
|
||||
- Filesystem writes outside workspace
|
||||
- Executing arbitrary commands beyond allowed list
|
||||
- Dangerous patterns: `rm -rf`, `:(){ :|:& };:`, eval with untrusted input
|
||||
|
||||
## Performance Tips
|
||||
|
||||
### 1. Minimize Tool Discovery
|
||||
|
||||
Cache tool paths instead of re-reading:
|
||||
|
||||
```bash
|
||||
# Bad: Re-read every time
|
||||
for i in {1..10}; do
|
||||
9p read anvillm/tools/anvilmcp/check_inbox.sh
|
||||
done
|
||||
|
||||
# Good: Call directly
|
||||
for i in {1..10}; do
|
||||
anvillm/tools/anvilmcp/check_inbox.sh "82b93a8a"
|
||||
done
|
||||
```
|
||||
|
||||
### 2. Process Data in Batches
|
||||
|
||||
```bash
|
||||
agents=("agent-1" "agent-2" "agent-3")
|
||||
for agent_id in "${agents[@]}"; do
|
||||
anvillm/tools/anvilmcp/check_inbox.sh "$agent_id" &
|
||||
done
|
||||
wait
|
||||
echo "Processed ${#agents[@]} inboxes"
|
||||
```
|
||||
|
||||
### 3. Return Summaries, Not Raw Data
|
||||
|
||||
```bash
|
||||
# Bad: Return all data
|
||||
result=$(anvillm/tools/anvilmcp/check_inbox.sh "82b93a8a")
|
||||
echo "$result" # Could be 50,000 tokens
|
||||
|
||||
# Good: Return summary
|
||||
result=$(anvillm/tools/anvilmcp/check_inbox.sh "82b93a8a")
|
||||
count=$(echo "$result" | wc -l)
|
||||
echo "Inbox has $count messages" # ~10 tokens
|
||||
```
|
||||
|
||||
## Debugging
|
||||
|
||||
### View Execution Output
|
||||
|
||||
The `execute_code` tool returns stdout and stderr:
|
||||
|
||||
```bash
|
||||
echo "Debug: Starting execution"
|
||||
echo "Warning: This is a warning" >&2
|
||||
```
|
||||
|
||||
### Check 9P Command Results
|
||||
## Error Handling
|
||||
|
||||
```bash
|
||||
if output=$(9p read anvillm/beads/list 2>&1); then
|
||||
count=$(echo "$output" | jq 'length')
|
||||
echo "Found $count beads"
|
||||
echo "$output" | jq 'length'
|
||||
else
|
||||
echo "9P command failed: $output" >&2
|
||||
echo "Failed: $output" >&2
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
|
||||
## Token Savings Examples
|
||||
## Further Reading
|
||||
|
||||
| Scenario | Direct Tools | Code Execution | Savings |
|
||||
|----------|--------------|----------------|---------|
|
||||
| List 5 tools | 452 tokens | 30 tokens | 93% |
|
||||
| List 100 tools | 9,000 tokens | 30 tokens | 99.7% |
|
||||
| Filter 10k rows | 200,000 tokens | 100 tokens | 99.95% |
|
||||
| Poll 10 times | 5,000 tokens | 200 tokens | 96% |
|
||||
| Process 3 inboxes | 15,000 tokens | 150 tokens | 99% |
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Discover tools progressively**: Only read tools you need
|
||||
2. **Filter data in subprocess**: Return summaries, not raw data
|
||||
3. **Use loops for repetitive tasks**: Avoid alternating between tool calls and reasoning
|
||||
4. **Handle errors gracefully**: Always check exit codes
|
||||
5. **Log meaningful output**: Help the agent understand what happened
|
||||
6. **Respect subprocess limits**: Don't try to bypass restrictions
|
||||
7. **Keep scripts simple**: Minimal, focused implementations
|
||||
|
||||
## Migration from Direct Tools
|
||||
|
||||
### Before (Direct Tool Calls)
|
||||
|
||||
```
|
||||
Agent: TOOL CALL check_inbox(agent_id: "82b93a8a")
|
||||
MCP: { "content": [{"type": "text", "text": "[50,000 tokens of messages]"}] }
|
||||
Agent: TOOL CALL send_message(...)
|
||||
MCP: { "content": [{"type": "text", "text": "Message sent"}] }
|
||||
```
|
||||
|
||||
**Cost**: 50,452 tokens
|
||||
|
||||
### After (Code Execution)
|
||||
|
||||
```
|
||||
Agent: TOOL CALL execute_code(code: "
|
||||
result=$(anvillm/tools/anvilmcp/check_inbox.sh '82b93a8a')
|
||||
urgent=$(echo \"$result\" | grep -c 'URGENT')
|
||||
echo \"Found $urgent urgent messages\"
|
||||
", language: "bash")
|
||||
MCP: { "content": [{"type": "text", "text": "Found 3 urgent messages"}] }
|
||||
```
|
||||
|
||||
**Cost**: 200 tokens (tool discovery) + 50 tokens (output) = 250 tokens
|
||||
|
||||
**Savings**: 99.5%
|
||||
|
||||
## Next Steps
|
||||
|
||||
- Read the [Migration Guide](./code-execution-migration-guide.md) for converting existing workflows
|
||||
- Review [Security Documentation](./code-execution-security.md) for threat model and subprocess configuration
|
||||
- Explore [Example Workflows](./code-execution-examples.md) for common patterns
|
||||
- [Code Execution Pattern](./code-execution-pattern.md) — design rationale
|
||||
- [Security Documentation](./code-execution-security.md) — threat model and sandbox config
|
||||
- [Example Workflows](./code-execution-examples.md) — common patterns
|
||||
|
|
|
|||
|
|
@ -1,11 +1,9 @@
|
|||
graph TB
|
||||
global["global.yaml<br/>(Base configuration)"]
|
||||
backends["backends/*.yaml<br/>(Backend-specific)"]
|
||||
roles["roles/*.yaml<br/>(Role-specific)<br/>default: roles/default.yaml"]
|
||||
tasks["tasks/*.yaml<br/>(Task-specific)"]
|
||||
sandbox["sandbox/*.yaml<br/>(Sandbox-specific)<br/>default: sandbox/default.yaml"]
|
||||
final["Final Config<br/>(most permissive wins)"]
|
||||
|
||||
global -->|override| backends
|
||||
backends -->|override| roles
|
||||
roles -->|override| tasks
|
||||
tasks --> final
|
||||
global -->|merge| backends
|
||||
backends -->|merge| sandbox
|
||||
sandbox --> final
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
|
Before Width: | Height: | Size: 12 KiB After Width: | Height: | Size: 10 KiB |
|
|
@ -1,142 +0,0 @@
|
|||
# Testing Code Execution Pattern Adoption
|
||||
|
||||
## Overview
|
||||
|
||||
This document describes how to verify that agents are using the code execution pattern instead of direct tool calls, and how to measure token reduction.
|
||||
|
||||
## Test Script
|
||||
|
||||
Run `./test-code-execution.sh` to automatically test agent adoption of the code execution pattern.
|
||||
|
||||
The script:
|
||||
1. Creates a test agent with Taskmaster template
|
||||
2. Sends queries that should trigger execute_code usage
|
||||
3. Verifies tool usage patterns
|
||||
4. Cleans up test agent
|
||||
|
||||
## Manual Testing
|
||||
|
||||
### Test Case 1: Data Filtering
|
||||
|
||||
**Query**: "Find all beads that haven't been updated in 30+ days and are still open. Show count by priority."
|
||||
|
||||
**Expected behavior**:
|
||||
- Agent uses `execute_code` with bash
|
||||
- Loads beads once with `beads.list()`
|
||||
- Filters in subprocess using `.filter()` and date comparison
|
||||
- Aggregates by priority in subprocess
|
||||
- Returns only summary statistics to context
|
||||
|
||||
**Anti-pattern** (what NOT to do):
|
||||
- Multiple `beads.list()` calls
|
||||
- Loading all bead data into context
|
||||
- Using separate tool calls for each filter condition
|
||||
|
||||
### Test Case 2: Batch Operations
|
||||
|
||||
**Query**: "Update all priority 1 beads to priority 2. Show progress."
|
||||
|
||||
**Expected behavior**:
|
||||
- Agent uses `execute_code` with bash
|
||||
- Queries beads once with `beads.query({ priority: 1 })`
|
||||
- Loops through results in subprocess
|
||||
- Updates each bead with `beads.update()`
|
||||
- Reports progress periodically (e.g., every 10 items)
|
||||
- Returns final summary to context
|
||||
|
||||
**Anti-pattern**:
|
||||
- Alternating between tool calls and model responses for each update
|
||||
- Loading full bead details into context for each item
|
||||
|
||||
### Test Case 3: Multi-Source Joins
|
||||
|
||||
**Query**: "Show all open beads that have related KB entries. Include KB reference count."
|
||||
|
||||
**Expected behavior**:
|
||||
- Agent uses `execute_code` with bash
|
||||
- Loads beads and KB entries in parallel
|
||||
- Joins data in subprocess
|
||||
- Filters to only beads with KB references
|
||||
- Returns enriched summary to context
|
||||
|
||||
**Anti-pattern**:
|
||||
- Sequential tool calls for each bead
|
||||
- Loading all KB content into context
|
||||
- Performing join logic through model reasoning
|
||||
|
||||
## Measuring Token Reduction
|
||||
|
||||
### Baseline (Direct Tool Calls)
|
||||
|
||||
For a query that processes 1000 beads:
|
||||
- Tool definitions: ~150k tokens (all tools loaded upfront)
|
||||
- Tool results: ~500k tokens (all bead data in context)
|
||||
- Total: ~650k tokens
|
||||
|
||||
### With Code Execution
|
||||
|
||||
For the same query:
|
||||
- Tool definitions: ~2k tokens (only beads.sh loaded)
|
||||
- Code execution: ~1k tokens (bash code)
|
||||
- Results: ~500 tokens (filtered summary)
|
||||
- Total: ~3.5k tokens
|
||||
|
||||
**Reduction**: 99.5% (650k → 3.5k tokens)
|
||||
|
||||
### Measuring in Practice
|
||||
|
||||
1. **Enable token tracking** in agent stats:
|
||||
```bash
|
||||
9p read anvillm/<agent-id>/stats
|
||||
```
|
||||
|
||||
2. **Compare before/after**:
|
||||
- Run same query with direct tool calls
|
||||
- Run same query with code execution
|
||||
- Calculate reduction percentage
|
||||
|
||||
3. **Look for indicators**:
|
||||
- Fewer tool calls in agent logs
|
||||
- Smaller context window usage
|
||||
- Faster response times
|
||||
- Lower API costs
|
||||
|
||||
## Verification Checklist
|
||||
|
||||
- [ ] Agent uses `execute_code` for data transformations
|
||||
- [ ] Agent loads tool definitions on-demand (progressive discovery)
|
||||
- [ ] Agent filters data in subprocess before returning to context
|
||||
- [ ] Agent uses loops in subprocess instead of alternating tool calls
|
||||
- [ ] Token usage reduced by 80-99% compared to direct tool calls
|
||||
- [ ] Response times improved
|
||||
- [ ] Agent behavior is correct (no regressions)
|
||||
|
||||
## Common Issues
|
||||
|
||||
### Agent still using direct tool calls
|
||||
|
||||
**Cause**: Agent prompt doesn't emphasize code execution pattern strongly enough.
|
||||
|
||||
**Fix**: Add more prominent code execution examples to agent context. Show specific use cases relevant to agent's role.
|
||||
|
||||
### Agent loads all tools upfront
|
||||
|
||||
**Cause**: Agent doesn't understand progressive discovery workflow.
|
||||
|
||||
**Fix**: Add explicit instructions to list `anvillm/tools/anvilmcp/` first, then read only needed tool files.
|
||||
|
||||
### Agent passes large datasets through context
|
||||
|
||||
**Cause**: Agent doesn't filter data in subprocess.
|
||||
|
||||
**Fix**: Add examples showing filtering, aggregation, and summarization in subprocess before returning results.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
An agent successfully adopts the code execution pattern when:
|
||||
|
||||
1. **Token usage** is reduced by 80-99% for data-heavy operations
|
||||
2. **Tool calls** are consolidated (1-2 execute_code calls instead of 10+ direct tool calls)
|
||||
3. **Context efficiency** is improved (only summaries and filtered results in context)
|
||||
4. **Correctness** is maintained (same or better results than direct tool calls)
|
||||
5. **Response time** is improved (fewer round-trips to model)
|
||||
|
|
@ -12,11 +12,10 @@ Install the AnviLLM MCP server to enable agent communication tools:
|
|||
./kiro-cli/install-mcp.sh
|
||||
```
|
||||
|
||||
This configures the `anvilmcp` MCP server in `~/.kiro/settings/cli.json`, providing tools:
|
||||
- `check_inbox` - Read messages from agent's inbox
|
||||
- `send_message` - Send messages to other agents or user
|
||||
- `list_sessions` - List all active sessions
|
||||
- `set_state` - Set agent state (idle, running, etc.)
|
||||
This configures the `anvilmcp` MCP server in `~/.kiro/settings/cli.json`, providing a single tool:
|
||||
- `execute_code` - Execute bash scripts in an isolated subprocess (sandboxed via landrun)
|
||||
|
||||
All AnviLLM functionality (messaging, sessions, beads, etc.) is accessed through 9P and helper scripts within `execute_code`.
|
||||
|
||||
### Agent Configuration
|
||||
|
||||
|
|
|
|||
Reference in New Issue