diff --git a/BEADS.md b/BEADS.md index 3415d49..ec6004a 100644 --- a/BEADS.md +++ b/BEADS.md @@ -204,19 +204,18 @@ echo 'claim bd-xyz' > anvillm/beads/ctl ## MCP Integration -anvilmcp exposes beads operations as MCP tools (calls `9p write anvillm/beads/ctl`): +Beads are accessed via `execute_code` through 9P: -```json -{ - "name": "create_bead", - "inputSchema": { - "properties": { - "title": {"type": "string"}, - "role": {"type": "string"}, - "description": {"type": "string"} - } - } -} +```bash +# Create a bead +echo 'new "Implement login" "Add JWT auth"' | 9p write anvillm/beads/ctl + +# List beads +9p read anvillm/beads/list | jq '.[] | "\(.id): \(.title)"' + +# Claim and complete +echo 'claim bd-a1b2' | 9p write anvillm/beads/ctl +echo 'complete bd-a1b2' | 9p write anvillm/beads/ctl ``` ## Benefits diff --git a/README.md b/README.md index 0faaf05..6e091e2 100644 --- a/README.md +++ b/README.md @@ -128,7 +128,7 @@ Templates use `{VARNAME}` syntax. Any environment variable can be referenced.

-- Default role: `roles/default.yaml` +- Default sandbox: `sandbox/default.yaml` ```yaml network: {enabled: true, unrestricted: true} @@ -348,17 +348,15 @@ Conductor analyzes dependencies, spawns specialized bots, and delegates work in ./kiro-cli/install-mcp.sh # Adds to ~/.kiro/settings/cli.json ``` -**Tool:** `execute_code` - Execute bash scripts in isolated subprocess +**Tool:** `execute_code` — the only tool needed. Executes bash scripts in an isolated subprocess (sandboxed via landrun). Agents discover and invoke all AnviLLM functionality through 9P and helper scripts: -Enables code execution pattern for 80-99% token reduction: -- Progressive tool discovery via 9P filesystem -- Data processing in subprocess (not in context) -- Loops, conditionals, and complex workflows +```sh +9p read anvillm/tools/anvilmcp/check_inbox.sh # discover scripts +bash <(9p read tools/send_message.sh) ... # invoke them +``` See [Code Execution User Guide](docs/code-execution-user-guide.md) for details. -**Legacy tools removed**: Direct tool calls (check_inbox, send_message, etc.) replaced by bash scripts accessible via `9p read anvillm/tools/anvilmcp/*.sh` - ## Troubleshooting | Problem | Solution | diff --git a/docs/code-execution-migration-guide.md b/docs/code-execution-migration-guide.md deleted file mode 100644 index 894d80a..0000000 --- a/docs/code-execution-migration-guide.md +++ /dev/null @@ -1,379 +0,0 @@ -# Migration Guide: Direct Tools to Code Execution - -## Overview - -This guide shows how to migrate from direct MCP tool calls to the code execution pattern. The migration reduces token consumption by 80-99% while enabling more powerful workflows. - -## Key Differences - -| Aspect | Direct Tools | Code Execution | -|--------|--------------|----------------| -| Tool loading | All upfront | Progressive discovery | -| Data flow | Through context | In subprocess | -| Control flow | Alternating calls | Loops, conditionals | -| Token cost | High (linear with data) | Low (constant) | -| Privacy | Data in context | Data isolated | - -## Migration Patterns - -### Pattern 1: Single Tool Call - -**Before**: -``` -Agent: TOOL CALL check_inbox(agent_id: "82b93a8a") -MCP: { "content": [{"type": "text", "text": "[Message 1]\n[Message 2]\n..."}] } -``` - -**After**: -``` -execute_code with code: "bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) 82b93a8a" -``` - -**Token savings**: Minimal for single calls, but enables further optimization - -### Pattern 2: Sequential Tool Calls - -**Before**: -``` -Agent: TOOL CALL check_inbox(agent_id: "agent-1") -MCP: { "content": [...] } // 5,000 tokens -Agent: TOOL CALL check_inbox(agent_id: "agent-2") -MCP: { "content": [...] } // 5,000 tokens -Agent: TOOL CALL check_inbox(agent_id: "agent-3") -MCP: { "content": [...] } // 5,000 tokens -``` - -**Total**: 15,000 tokens - -**After**: -``` -execute_code with code: " -for agent_id in agent-1 agent-2 agent-3; do - bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) \$agent_id | wc -l -done | awk '{sum+=\$1} END {print \"Total messages:\", sum}' -" -``` - -**Output**: "Total messages: 47" (~10 tokens) - -**Token savings**: 99.9% (10 tokens vs 15,000 tokens) - -### Pattern 3: Filtering Large Results - -**Before**: -``` -Agent: TOOL CALL get_spreadsheet(sheet_id: "abc123") -MCP: { "content": [...] } // 200,000 tokens (10,000 rows) -Agent: [Processes in context, filters to 47 pending orders] -``` - -**Total**: 200,000+ tokens - -**After**: -``` -execute_code with code: " -rows=\$(9p read anvillm/sheets/abc123) -pending=\$(echo \"\$rows\" | jq '[.[] | select(.Status == \"pending\")]') -count=\$(echo \"\$pending\" | jq 'length') -total=\$(echo \"\$rows\" | jq 'length') -echo \"Found \$count pending orders out of \$total total\" -" -``` - -**Output**: "Found 47 pending orders out of 10000 total" (~15 tokens) - -**Token savings**: 99.99% (15 tokens vs 200,000 tokens) - -### Pattern 4: Polling Loops - -**Before**: -``` -Agent: TOOL CALL check_status(job_id: "job-123") -MCP: { "status": "running" } -Agent: [Waits, then calls again] -Agent: TOOL CALL check_status(job_id: "job-123") -MCP: { "status": "running" } -[Repeats 10 times] -``` - -**Total**: ~5,000 tokens (10 calls × 500 tokens) - -**After**: -``` -execute_code with code: " -job_id=job-123 -status=running -attempts=0 - -while [ \"\$status\" = \"running\" ] && [ \$attempts -lt 10 ]; do - status=\$(9p read anvillm/jobs/\$job_id/status) - if [ \"\$status\" = \"running\" ]; then - sleep 5 - fi - attempts=\$((attempts + 1)) -done - -echo \"Job \$job_id status: \$status (checked \$attempts times)\" -" -``` - -**Output**: "Job job-123 status: completed (checked 7 times)" (~15 tokens) - -**Token savings**: 99.7% (15 tokens vs 5,000 tokens) - -### Pattern 5: Multi-Step Data Pipeline - -**Before**: -``` -Agent: TOOL CALL check_inbox(agent_id: "82b93a8a") -MCP: { "content": [...] } // 10,000 tokens -Agent: [Identifies 5 urgent messages] -Agent: TOOL CALL send_message(to: "manager", body: "5 urgent messages") -MCP: { "content": "Message sent" } -Agent: TOOL CALL set_state(agent_id: "82b93a8a", state: "idle") -MCP: { "content": "State updated" } -``` - -**Total**: ~11,000 tokens - -**After**: -``` -execute_code with code: " -inbox=\$(bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) 82b93a8a) -urgent_count=\$(echo \"\$inbox\" | grep -c URGENT || echo 0) - -if [ \$urgent_count -gt 0 ]; then - bash <(9p read anvillm/tools/anvilmcp/send_message.sh) 82b93a8a manager NOTIFICATION 'Urgent messages' \"\$urgent_count urgent messages in inbox\" -fi - -bash <(9p read anvillm/tools/anvilmcp/set_state.sh) 82b93a8a idle -echo \"Processed inbox: \$urgent_count urgent messages\" -" -``` - -**Output**: "Processed inbox: 5 urgent messages" (~10 tokens) - -**Token savings**: 99.9% (10 tokens vs 11,000 tokens) - -## Tool-Specific Migrations - -### check_inbox - -**Before**: -``` -TOOL CALL check_inbox(agent_id: "82b93a8a") -``` - -**After**: -``` -execute_code with code: "bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) 82b93a8a" -``` - -**Or use 9P directly**: -``` -execute_code with code: "9p read anvillm/inbox/82b93a8a/json" -``` - -### send_message - -**Before**: -``` -TOOL CALL send_message(from: "agent-1", to: "agent-2", type: "PROMPT_REQUEST", subject: "Task", body: "Do X") -``` - -**After**: -```bash -bash <(9p read anvillm/tools/anvilmcp/send_message.sh) agent-1 agent-2 PROMPT_REQUEST 'Task' 'Do X' -``` - -### list_sessions - -**Before**: -``` -TOOL CALL list_sessions() -``` - -**After**: -``` -execute_code with code: "bash <(9p read anvillm/tools/anvilmcp/list_sessions.sh)" -``` - -### set_state - -**Before**: -``` -TOOL CALL set_state(agent_id: "82b93a8a", state: "running") -``` - -**After**: -``` -execute_code with code: "bash <(9p read anvillm/tools/anvilmcp/set_state.sh) 82b93a8a running" -``` - -### list_skills - -**Before**: -``` -TOOL CALL list_skills() -``` - -**After**: -``` -execute_code with code: "bash <(9p read anvillm/tools/anvilmcp/list_skills.sh) | jq -r '.[].name' | paste -sd, -" -``` - -## Compatibility Notes - -### When to Use Direct Tools - -Direct tools are better for: - -1. **Single operations**: Simple one-off calls -2. **Small data**: Results under 1,000 tokens -3. **Simple logic**: No conditionals or loops needed -4. **Debugging**: When you need to see intermediate results - -### When to Use Code Execution - -Code execution is better for: - -1. **Multiple tool calls**: Any workflow with 2+ tool calls -2. **Large data**: Any operation returning >1,000 tokens -3. **Filtering/aggregation**: Processing data to extract summaries -4. **Loops**: Polling, batch processing, retries -5. **Privacy**: Operations involving PII or sensitive data - -## Migration Checklist - -- [ ] Identify workflows with multiple tool calls -- [ ] Identify operations with large data returns -- [ ] Convert sequential calls to loops -- [ ] Add filtering/aggregation in subprocess -- [ ] Return summaries instead of raw data -- [ ] Add error handling (try/catch) -- [ ] Test in isolated subprocess -- [ ] Measure token savings -- [ ] Update agent prompts with examples - -## Common Pitfalls - -### Pitfall 1: Returning Raw Data - -**Bad**: -```bash -result=$(bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) 82b93a8a) -echo "$result" # Could be 50,000 tokens -``` - -**Good**: -```bash -result=$(bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) 82b93a8a) -count=$(echo "$result" | wc -l) -echo "Inbox has $count messages" # ~10 tokens -``` - -### Pitfall 2: Not Using Loops - -**Bad**: -```bash -result1=$(bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) agent-1) -echo "$result1" -result2=$(bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) agent-2) -echo "$result2" -result3=$(bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) agent-3) -echo "$result3" -``` - -**Good**: -```bash -for agent_id in agent-1 agent-2 agent-3; do - result=$(bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) "$agent_id") - count=$(echo "$result" | wc -l) - echo "$agent_id: $count messages" -done -``` - -### Pitfall 3: Ignoring Errors - -**Bad**: -```bash -result=$(bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) invalid-id) -echo "$result" # May fail silently -``` - -**Good**: -```bash -if result=$(bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) invalid-id 2>&1); then - echo "$result" -else - echo "Failed to read inbox: $result" >&2 -fi -``` - -### Pitfall 4: Re-reading Tool Definitions - -**Bad**: -```bash -for i in {1..10}; do - tool=$(9p read anvillm/tools/anvilmcp/check_inbox.sh) - # Parse and use... -done -``` - -**Good**: -```bash -# Call tool directly each time -for i in {1..10}; do - bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) 82b93a8a -done -``` - -## Performance Comparison - -| Workflow | Direct Tools | Code Execution | Savings | -|----------|--------------|----------------|---------| -| Single small call | 500 tokens | 400 tokens | 20% | -| 3 sequential calls | 15,000 tokens | 150 tokens | 99% | -| Filter 10k rows | 200,000 tokens | 100 tokens | 99.95% | -| Poll 10 times | 5,000 tokens | 200 tokens | 96% | -| Multi-step pipeline | 50,000 tokens | 500 tokens | 99% | - -## Rollout Strategy - -### Phase 1: Pilot (Week 1) -- Migrate 1-2 simple workflows -- Measure token savings -- Identify issues - -### Phase 2: Expand (Week 2-3) -- Migrate high-volume workflows -- Update agent prompts -- Train team on patterns - -### Phase 3: Optimize (Week 4+) -- Refine implementations -- Add more examples -- Monitor metrics - -## Support - -For questions or issues: -- Review [User Guide](./code-execution-user-guide.md) -- Check [Security Documentation](./code-execution-security.md) -- See [Example Workflows](./code-execution-examples.md) - -## Token Savings Calculator - -Estimate your savings: - -``` -Current tokens = (tool_count × 90) + (data_size_kb × 250) -Code execution tokens = 30 + (summary_size_kb × 250) -Savings = (Current - Code execution) / Current × 100% -``` - -Example: -- 100 tools, 10 KB data -- Current: (100 × 90) + (10 × 250) = 11,500 tokens -- Code execution: 30 + (0.01 × 250) = 33 tokens -- Savings: 99.7% diff --git a/docs/code-execution-pattern.md b/docs/code-execution-pattern.md index f37729f..67ec457 100644 --- a/docs/code-execution-pattern.md +++ b/docs/code-execution-pattern.md @@ -2,191 +2,59 @@ ## Overview -The code execution pattern reduces token consumption by 80-99% by executing data transformations and tool interactions in a isolated subprocess instead of passing all data through the model's context window. +`execute_code` is the single MCP tool exposed by `anvilmcp`. It executes bash scripts in an isolated subprocess sandboxed via landrun. All AnviLLM functionality — messaging, sessions, beads, state — is accessed through 9P and helper scripts within these subprocesses. -Reference: [Code execution with MCP: Building more efficient agents](https://www.anthropic.com/engineering/code-execution-with-mcp) +## Tool Discovery -## Progressive Discovery Workflow - -Instead of loading all tool definitions upfront, discover tools on-demand: - -1. **List available tools** - ```bash - 9p ls anvillm/tools/anvilmcp/ - ``` - -2. **Read specific tool definitions** - ```bash - 9p read anvillm/tools/anvilmcp/beads.sh - ``` - -3. **Import and use in bash** - ```bash - # Call tools from anvillm/tools/anvilmcp/beads'; - - const allBeads = await beads.list(); - const open = allBeads.filter(b => b.status === 'open'); - console.log(`Found ${open.length} open beads`); - ``` - -## Data Transformation Examples - -### Filtering Large Datasets - -Instead of loading 10,000 rows into context: +Discover available helper scripts on-demand via 9P: ```bash -// Load data in subprocess -const allRows = await beads.list(); - -// Filter in subprocess -const pending = allRows.filter(row => row.status === 'open' && row.priority <= 2); - -// Only return summary to context -console.log(`Found ${pending.length} high-priority open beads`); -console.log(pending.slice(0, 5)); // Show first 5 for review +9p ls anvillm/tools/anvilmcp/ ``` -### Polling Loops - -Execute loops in subprocess instead of alternating tool calls: +Read and invoke a specific script: ```bash -let found = false; -while (!found) { - const beads = await beads.list(); - found = beads.some(b => b.id === 'bd-abc' && b.status === 'closed'); - if (!found) await new Promise(r => setTimeout(r, 5000)); -} -console.log('Bead bd-abc completed'); +bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) ``` -### Aggregations +## Examples -Compute statistics without context bloat: +### Messaging ```bash -const allBeads = await beads.list(); - -const stats = { - total: allBeads.length, - byStatus: {}, - byPriority: {} -}; - -for (const bead of allBeads) { - stats.byStatus[bead.status] = (stats.byStatus[bead.status] || 0) + 1; - stats.byPriority[bead.priority] = (stats.byPriority[bead.priority] || 0) + 1; -} - -console.log(JSON.stringify(stats, null, 2)); +bash <(9p read anvillm/tools/anvilmcp/send_message.sh) \ + --to user --type PROMPT_RESPONSE --subject "Done" --body "Task complete" ``` -## Token Savings - -- **Without code execution**: Load all tool definitions (150k tokens) + all intermediate results -- **With code execution**: Load only needed tools (~2k tokens) + filtered results -- **Savings**: 80-99% reduction in token consumption - -## When to Use - -Use code execution when: -- Working with large datasets (>100 rows) -- Performing multiple transformations on the same data -- Implementing loops or polling -- Computing aggregations or statistics -- Filtering data before presenting to user - -Use direct tool calls when: -- Single, simple operations -- Small result sets -- No data transformation needed - -## Extended Examples - -### Multi-Source Data Joins - -Join data from multiple sources without loading everything into context: +### Beads ```bash -# Call tools from anvillm/tools/anvilmcp/beads'; -# Call tools from anvillm/tools/anvilmcp/kb'; - -// Load data from both sources -const allBeads = await beads.list(); -const kbEntries = await kb.search('implementation'); - -// Join in subprocess -const enriched = allBeads - .filter(b => b.status === 'open') - .map(bead => { - const relatedKB = kbEntries.filter(kb => - kb.content.includes(bead.id) || - bead.description.includes(kb.title) - ); - return { ...bead, kbReferences: relatedKB.length }; - }) - .filter(b => b.kbReferences > 0); - -console.log(`Found ${enriched.length} beads with KB references`); -console.log(enriched.slice(0, 3)); +echo 'new "Implement login" "Add JWT auth"' | 9p write anvillm/beads/ctl +9p read anvillm/beads/ready | jq -r '.[] | "\(.id): \(.title)"' ``` -### Batch Operations with Progress Tracking - -Process many items without flooding context: +### Polling ```bash -const beads = await beads.query({ status: 'open', priority: 1 }); - -let processed = 0; -let errors = []; - -for (const bead of beads) { - try { - await beads.update(bead.id, { priority: 2 }); - processed++; - } catch (e) { - errors.push({ id: bead.id, error: e.message }); - } - - if (processed % 10 === 0) { - console.log(`Processed ${processed}/${beads.length}`); - } -} - -console.log(`Complete: ${processed} updated, ${errors.length} errors`); -if (errors.length > 0) { - console.log('Errors:', errors.slice(0, 5)); -} +while true; do + state=$(9p read anvillm/$ID/state) + [ "$state" = "idle" ] && break + sleep 5 +done +echo "Session $ID is idle" ``` -### Complex Filtering with Multiple Criteria - -Apply complex business logic in subprocess: +### Filtering ```bash -const allBeads = await beads.list(); - -const candidates = allBeads.filter(bead => { - // Multiple conditions - const isHighPriority = bead.priority <= 2; - const isStale = new Date() - new Date(bead.updated_at) > 7 * 24 * 60 * 60 * 1000; - const hasNoAssignee = !bead.assignee; - const isNotBlocked = !bead.blockers || bead.blockers.length === 0; - - return isHighPriority && isStale && hasNoAssignee && isNotBlocked; -}); - -// Group by priority -const grouped = candidates.reduce((acc, bead) => { - acc[bead.priority] = acc[bead.priority] || []; - acc[bead.priority].push(bead); - return acc; -}, {}); - -console.log('Stale unassigned beads by priority:'); -for (const [priority, beads] of Object.entries(grouped)) { - console.log(` Priority ${priority}: ${beads.length} beads`); -} +9p read anvillm/beads/list | jq '[.[] | select(.status == "open" and .priority <= 2)]' ``` + +## Why One Tool + +A single `execute_code` tool means: +- No upfront tool-definition loading (saves tokens) +- Data stays in the subprocess (not in context) +- Loops, conditionals, and pipes work naturally +- Progressive discovery — only read what you need diff --git a/docs/code-execution-security-guide.md b/docs/code-execution-security-guide.md index 67a48ad..d20ec97 100644 --- a/docs/code-execution-security-guide.md +++ b/docs/code-execution-security-guide.md @@ -125,7 +125,7 @@ Validation is basic pattern matching. Not comprehensive - relies on sandbox for **Scenario**: Sensitive data leaked from subprocess -**Risk**: LOW (Improved from direct tools) +**Risk**: LOW **Mitigations**: 1. Data stays in subprocess (not in context) @@ -133,20 +133,22 @@ Validation is basic pattern matching. Not comprehensive - relies on sandbox for 3. No network access (unless explicitly granted) 4. 9P socket access controlled by outer sandbox -**Residual risk**: Data could be written to workspace, but workspace is deleted. Data could be sent via 9P to other agents - same risk as direct tools. +**Residual risk**: Data could be written to workspace, but workspace is deleted. Data could be sent via 9P to other agents. ### Threat 5: 9P Socket Access **Scenario**: Subprocess abuses 9P access to read/write unauthorized data -**Risk**: MEDIUM (Same as direct tools) +**Risk**: MEDIUM **Mitigations**: 1. 9P socket permissions (Unix socket, user-only) 2. anvilsrv enforces access control 3. Audit logging in anvilsrv -**Residual risk**: If subprocess has 9P access, it has same capabilities as direct tool calls. This is by design - code execution is meant to replace direct tools. +**Residual risk**: If subprocess has 9P access, it has the same capabilities as any 9P client. This is by design. + +**Mitigation**: Run locally on a trusted network. On a single-user system, the only processes with socket access are ones the user launched. Do not expose the 9P mount over the network, or at minimum, implement access control (e.g., firewall). ### Threat 6: Path Traversal @@ -161,25 +163,14 @@ Validation is basic pattern matching. Not comprehensive - relies on sandbox for **Residual risk**: None - path traversal blocked at multiple layers. -## Security Benefits vs Direct Tools +## Security Properties -**Improved**: -- Data isolation (subprocess) +- Data isolation (subprocess, not in model context) - Privacy preservation (PII doesn't enter context) - Explicit control over data flow - Workspace isolation - -**Same**: - 9P socket access control -- Process isolation -- Agent trust model - -**Additional Considerations**: -- Code validation required -- Sandbox configuration complexity -- Resource limit tuning - -**Conclusion**: Code execution improves security posture compared to direct tool calls, with acceptable additional complexity. +- Process isolation via landrun (Landlock) ## Configuration @@ -389,4 +380,3 @@ All tests should fail safely without compromising the system. - landrun: https://github.com/zouuup/landrun - Anthropic code execution pattern: https://www.anthropic.com/engineering/code-execution-with-mcp - User guide: ./code-execution-user-guide.md -- Migration guide: ./code-execution-migration-guide.md diff --git a/docs/code-execution-security.md b/docs/code-execution-security.md index 6c62fe8..01ed1e2 100644 --- a/docs/code-execution-security.md +++ b/docs/code-execution-security.md @@ -182,7 +182,7 @@ var executionSem = make(chan struct{}, 4) // Max 4 concurrent **Description**: Code leaks sensitive data outside the system. -**Risk Level**: LOW (Improved from direct tools) +**Risk Level**: LOW **Attack Vectors**: - Network exfiltration (blocked) @@ -219,11 +219,7 @@ for (const email of emails) { console.log(`Processed ${emails.length} emails`); // Summary only ``` -**Residual Risk**: LOW - Better isolation than direct tools - -**Comparison to Direct Tools**: -- **Direct tools**: All data flows through model context -- **Code execution**: Data stays in subprocess, only summaries returned +**Residual Risk**: LOW — data stays in subprocess, only summaries returned to the model ### Threat 5: 9P Socket Access @@ -263,9 +259,9 @@ log.Printf("9P: agent=%s op=%s path=%s", agentID, op, path) allowedOps := []string{"read", "write", "ls"} ``` -**Residual Risk**: MEDIUM - Same as current system +**Residual Risk**: MEDIUM — inherent to any 9P client -**Note**: This is not a new risk introduced by code execution. The same 9P access exists with direct tool calls. +**Mitigation**: Run locally on a trusted network. On a single-user system, the only processes with socket access are ones the user launched — self-inflicted abuse is not a meaningful threat. Do not expose the 9P mount over the network, or at minimum, implement access control (e.g., firewall). ### Threat 6: Supply Chain @@ -307,33 +303,14 @@ import { evil } from "/etc/passwd"; **Residual Risk**: LOW - No remote imports possible -## Security Benefits vs Direct Tools +## Security Properties -### Improved Security - -1. **Data Isolation**: Intermediate data stays in subprocess - - **Direct tools**: All data flows through model context - - **Code execution**: Data processed in subprocess, summaries only - -2. **Privacy Preservation**: PII doesn't enter model context - - **Direct tools**: PII visible to model - - **Code execution**: PII stays in subprocess - -3. **Explicit Control**: Agent explicitly logs what model sees - - **Direct tools**: All tool output visible - - **Code execution**: Only console.log output visible - -### Same Security - -1. **9P Access Control**: Unix permissions, audit logging -2. **Process Isolation**: Separate processes per agent -3. **Authentication**: Agent ID validation - -### Additional Considerations - -1. **Code Validation**: New attack surface (pattern matching) -2. **Subprocess Configuration**: Complexity in setup -3. **Resource Limits**: Requires tuning and monitoring +1. **Data Isolation**: Intermediate data stays in subprocess — only stdout/stderr returned to the model +2. **Privacy Preservation**: PII processed in subprocess, never enters model context +3. **Explicit Control**: Agent explicitly chooses what to print (and thus what the model sees) +4. **9P Access Control**: Unix socket permissions, audit logging +5. **Process Isolation**: Separate landrun sandbox per execution +6. **Code Validation**: Pattern matching rejects dangerous constructs ## Subprocess Configuration @@ -551,18 +528,6 @@ func TestSubprocessEscape(t *testing.T) { ## Comparison to Alternatives -### vs Direct Tool Calls - -| Aspect | Direct Tools | Code Execution | -|--------|--------------|----------------| -| Data isolation | Poor (all in context) | Good (subprocess) | -| Privacy | Poor (PII in context) | Good (PII isolated) | -| Attack surface | 9P socket | 9P socket + subprocess | -| Resource control | Limited | Good (timeouts, limits) | -| Audit trail | 9P logs | 9P + execution logs | - -**Conclusion**: Code execution improves security posture - ### vs Unrestricted Execution | Aspect | Unrestricted | Code Execution | @@ -609,8 +574,8 @@ The code execution pattern provides strong security through multiple layers: 4. **Resource Limits**: Prevent exhaustion 5. **Audit Logging**: Comprehensive trail -**Security Posture**: Code execution improves security compared to direct tool calls by isolating data and providing explicit control over information flow. +**Security Posture**: Data isolation and explicit control over information flow via sandboxed subprocesses. -**Residual Risks**: Primarily resource exhaustion (MEDIUM) and 9P access control (MEDIUM, same as current system). +**Residual Risks**: Primarily resource exhaustion (MEDIUM) and 9P access control (MEDIUM, inherent to any 9P client). Mitigated by running locally on a trusted network — do not expose the 9P mount over the network, or at minimum, implement access control (e.g., firewall). **Recommendation**: Proceed with implementation. Security benefits outweigh risks. diff --git a/docs/code-execution-user-guide.md b/docs/code-execution-user-guide.md index b631995..fda655d 100644 --- a/docs/code-execution-user-guide.md +++ b/docs/code-execution-user-guide.md @@ -2,328 +2,98 @@ ## Overview -The `execute_code` tool enables agents to write and execute bash scripts in a isolated subprocess. This reduces token consumption by 80-99% compared to direct tool calls by enabling progressive tool discovery and in-subprocess data processing. +`execute_code` is the only MCP tool provided by `anvilmcp`. It runs bash scripts in an isolated subprocess sandboxed via landrun. All AnviLLM functionality is accessed through 9P and helper scripts within these subprocesses. -## Why Use Code Execution? +## Tool Discovery -### Problem 1: Tool Definitions Consume Context - -Loading 100 tools upfront costs ~9,000 tokens (45% of a 200k context window). With code execution, you discover tools progressively: +List available helper scripts: ```bash -# List available tools (~30 tokens) -9p ls anvillm/tools/anvilmcp - -# Read only the tool you need (~90 tokens) -9p read anvillm/tools/anvilmcp/check_inbox.sh +9p ls anvillm/tools/anvilmcp/ ``` -**Savings**: 97% (120 tokens vs 9,000 tokens) +Read a script to see its usage: -### Problem 2: Large Data Flows Through Context - -Processing a 10,000-row spreadsheet directly costs ~200,000 tokens. With code execution, filter in the subprocess: - -```bash -all_rows=$(get_sheet "abc123") -pending=$(echo "$all_rows" | jq '[.[] | select(.Status == "pending")]') -count=$(echo "$pending" | jq 'length') -echo "Found $count pending orders" -# Agent sees: "Found 47 pending orders" (~10 tokens) -``` - -**Savings**: 99.995% (10 tokens vs 200,000 tokens) - -## Basic Usage - -### Tool Discovery Workflow - -1. **List available tools**: -```bash -9p ls anvillm/tools/anvilmcp -``` - -2. **Read tool documentation**: ```bash 9p read anvillm/tools/anvilmcp/check_inbox.sh ``` -3. **Call tool via MCP**: -```bash -echo '{"name":"check_inbox","arguments":{"agent_id":"82b93a8a"}}' | 9p write anvillm/mcp/call -``` - -### Calling 9P Operations - -You can also call 9P operations directly: +Invoke a script: ```bash -beads=$(9p read anvillm/beads/list) -count=$(echo "$beads" | jq 'length') -echo "Found $count beads" +bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) ``` ## Common Patterns -### Pattern 1: Filtering Large Datasets +### Messaging ```bash -# Call check_inbox via MCP -result=$(echo '{"name":"check_inbox","arguments":{"agent_id":"82b93a8a"}}' | 9p write anvillm/mcp/call) -total=$(echo "$result" | jq -r '.content[0].text' | wc -l) -urgent=$(echo "$result" | jq -r '.content[0].text' | grep -c "URGENT") +# Check inbox +bash <(9p read anvillm/tools/anvilmcp/check_inbox.sh) "$AGENT_ID" -echo "Found $urgent urgent messages out of $total total" -``` - -### Pattern 2: Polling Loops - -```bash -found=false -attempts=0 - -while [ $attempts -lt 10 ] && [ "$found" = false ]; do - sessions=$(anvillm/tools/anvilmcp/list_sessions.sh) - if echo "$sessions" | jq -e '.[] | select(.state == "completed")' > /dev/null; then - found=true - else - echo "Attempt $((attempts + 1)): Not ready yet" - sleep 5 - fi - attempts=$((attempts + 1)) -done - -if [ "$found" = true ]; then - echo "Session completed" -else - echo "Timeout after 10 attempts" -fi -``` - -### Pattern 3: Multi-Step Aggregations - -```bash -agents=("agent-1" "agent-2" "agent-3") -total_messages=0 - -for agent_id in "${agents[@]}"; do - result=$(anvillm/tools/anvilmcp/check_inbox.sh "$agent_id") - count=$(echo "$result" | wc -l) - total_messages=$((total_messages + count)) -done - -echo "Total messages across ${#agents[@]} agents: $total_messages" -``` - -### Pattern 4: Privacy-Preserving Operations - -Process sensitive data without exposing it to the model: - -```bash -# Read PII from one system -emails=$(9p read anvillm/data/customer-emails) - -# Write to another system without PII entering context -echo "$emails" | jq -c '.[]' | while read -r email; do - address=$(echo "$email" | jq -r '.address') - name=$(echo "$email" | jq -r '.name') - - echo "{\"email\":\"$address\",\"name\":\"$name\"}" | \ - 9p write anvillm/crm/contacts -done - -count=$(echo "$emails" | jq 'length') -echo "Processed $count contacts" -``` - -## Available Tools - -All tools are accessed via 9P commands. Read tool documentation first: - -```bash -9p read anvillm/tools/anvilmcp/check_inbox.sh -``` - -### Communication - -**check_inbox**: Read messages from agent's inbox -```bash -# List inbox messages -9p ls anvillm/AGENT_ID/inbox - -# Read specific message -9p read anvillm/AGENT_ID/inbox/MESSAGE_ID.json -``` - -**send_message**: Send message to another agent or user -```bash -echo '{"from":"agent-1","to":"agent-2","type":"PROMPT_REQUEST","subject":"Task","body":"Do work"}' | \ - 9p write anvillm/agent-1/mail +# Send message +bash <(9p read anvillm/tools/anvilmcp/send_message.sh) \ + --to user --type PROMPT_RESPONSE --subject "Done" --body "Complete" ``` ### Session Management -**list_sessions**: List all active sessions ```bash 9p read anvillm/list +9p read anvillm/$ID/state ``` -**set_state**: Set agent state +### Beads + ```bash -echo "running" | 9p write anvillm/AGENT_ID/state +9p read anvillm/beads/list | jq '[.[] | select(.status == "open")]' +echo 'new "Task title" "Task description"' | 9p write anvillm/beads/ctl ``` -### Skills +### Polling -**list_skills**: List available skills ```bash -anvillm-skills list +while true; do + state=$(9p read anvillm/$ID/state) + [ "$state" = "idle" ] && break + sleep 5 +done ``` -## Error Handling +### Filtering -Always handle errors in your scripts: +Process data in the subprocess — only return summaries to the agent: ```bash -if output=$(9p read anvillm/beads/list 2>&1); then - echo "$output" -else - echo "9P command failed: $output" >&2 - exit 1 -fi +beads=$(9p read anvillm/beads/list) +count=$(echo "$beads" | jq '[.[] | select(.priority <= 2)] | length') +echo "Found $count high-priority beads" ``` ## Subprocess Restrictions -Your code runs in a isolated subprocess with limited permissions: +Scripts run in a landrun sandbox with limited permissions: -### Allowed -- Read from `./anvillm/tools/` directory -- Execute `/usr/bin/9p` command -- Access environment variables: `NAMESPACE`, `AGENT_ID` -- Standard bash utilities (jq, grep, sed, awk) +- Read/write to the session's working directory +- Read from system paths (`/usr`, `/lib`, `/bin`) +- Access to 9P namespace +- Standard utilities: `jq`, `grep`, `sed`, `awk`, `9p` +- Network access per sandbox config -### Blocked -- Network access -- Filesystem writes outside workspace -- Executing arbitrary commands beyond allowed list -- Dangerous patterns: `rm -rf`, `:(){ :|:& };:`, eval with untrusted input - -## Performance Tips - -### 1. Minimize Tool Discovery - -Cache tool paths instead of re-reading: - -```bash -# Bad: Re-read every time -for i in {1..10}; do - 9p read anvillm/tools/anvilmcp/check_inbox.sh -done - -# Good: Call directly -for i in {1..10}; do - anvillm/tools/anvilmcp/check_inbox.sh "82b93a8a" -done -``` - -### 2. Process Data in Batches - -```bash -agents=("agent-1" "agent-2" "agent-3") -for agent_id in "${agents[@]}"; do - anvillm/tools/anvilmcp/check_inbox.sh "$agent_id" & -done -wait -echo "Processed ${#agents[@]} inboxes" -``` - -### 3. Return Summaries, Not Raw Data - -```bash -# Bad: Return all data -result=$(anvillm/tools/anvilmcp/check_inbox.sh "82b93a8a") -echo "$result" # Could be 50,000 tokens - -# Good: Return summary -result=$(anvillm/tools/anvilmcp/check_inbox.sh "82b93a8a") -count=$(echo "$result" | wc -l) -echo "Inbox has $count messages" # ~10 tokens -``` - -## Debugging - -### View Execution Output - -The `execute_code` tool returns stdout and stderr: - -```bash -echo "Debug: Starting execution" -echo "Warning: This is a warning" >&2 -``` - -### Check 9P Command Results +## Error Handling ```bash if output=$(9p read anvillm/beads/list 2>&1); then - count=$(echo "$output" | jq 'length') - echo "Found $count beads" + echo "$output" | jq 'length' else - echo "9P command failed: $output" >&2 + echo "Failed: $output" >&2 exit 1 fi ``` -## Token Savings Examples +## Further Reading -| Scenario | Direct Tools | Code Execution | Savings | -|----------|--------------|----------------|---------| -| List 5 tools | 452 tokens | 30 tokens | 93% | -| List 100 tools | 9,000 tokens | 30 tokens | 99.7% | -| Filter 10k rows | 200,000 tokens | 100 tokens | 99.95% | -| Poll 10 times | 5,000 tokens | 200 tokens | 96% | -| Process 3 inboxes | 15,000 tokens | 150 tokens | 99% | - -## Best Practices - -1. **Discover tools progressively**: Only read tools you need -2. **Filter data in subprocess**: Return summaries, not raw data -3. **Use loops for repetitive tasks**: Avoid alternating between tool calls and reasoning -4. **Handle errors gracefully**: Always check exit codes -5. **Log meaningful output**: Help the agent understand what happened -6. **Respect subprocess limits**: Don't try to bypass restrictions -7. **Keep scripts simple**: Minimal, focused implementations - -## Migration from Direct Tools - -### Before (Direct Tool Calls) - -``` -Agent: TOOL CALL check_inbox(agent_id: "82b93a8a") -MCP: { "content": [{"type": "text", "text": "[50,000 tokens of messages]"}] } -Agent: TOOL CALL send_message(...) -MCP: { "content": [{"type": "text", "text": "Message sent"}] } -``` - -**Cost**: 50,452 tokens - -### After (Code Execution) - -``` -Agent: TOOL CALL execute_code(code: " - result=$(anvillm/tools/anvilmcp/check_inbox.sh '82b93a8a') - urgent=$(echo \"$result\" | grep -c 'URGENT') - echo \"Found $urgent urgent messages\" -", language: "bash") -MCP: { "content": [{"type": "text", "text": "Found 3 urgent messages"}] } -``` - -**Cost**: 200 tokens (tool discovery) + 50 tokens (output) = 250 tokens - -**Savings**: 99.5% - -## Next Steps - -- Read the [Migration Guide](./code-execution-migration-guide.md) for converting existing workflows -- Review [Security Documentation](./code-execution-security.md) for threat model and subprocess configuration -- Explore [Example Workflows](./code-execution-examples.md) for common patterns +- [Code Execution Pattern](./code-execution-pattern.md) — design rationale +- [Security Documentation](./code-execution-security.md) — threat model and sandbox config +- [Example Workflows](./code-execution-examples.md) — common patterns diff --git a/docs/diagrams/sandbox-config.mmd b/docs/diagrams/sandbox-config.mmd index bc6a33d..f3bd60e 100644 --- a/docs/diagrams/sandbox-config.mmd +++ b/docs/diagrams/sandbox-config.mmd @@ -1,11 +1,9 @@ graph TB global["global.yaml
(Base configuration)"] backends["backends/*.yaml
(Backend-specific)"] - roles["roles/*.yaml
(Role-specific)
default: roles/default.yaml"] - tasks["tasks/*.yaml
(Task-specific)"] + sandbox["sandbox/*.yaml
(Sandbox-specific)
default: sandbox/default.yaml"] final["Final Config
(most permissive wins)"] - global -->|override| backends - backends -->|override| roles - roles -->|override| tasks - tasks --> final + global -->|merge| backends + backends -->|merge| sandbox + sandbox --> final diff --git a/docs/diagrams/sandbox-config.svg b/docs/diagrams/sandbox-config.svg index ab1c949..bab2c3c 100644 --- a/docs/diagrams/sandbox-config.svg +++ b/docs/diagrams/sandbox-config.svg @@ -1 +1 @@ -

override

override

override

global.yaml
(Base configuration)

backends/*.yaml
(Backend-specific)

roles/*.yaml
(Role-specific)
default: roles/default.yaml

tasks/*.yaml
(Task-specific)

Final Config
(most permissive wins)

 \ No newline at end of file +

merge

merge

global.yaml
(Base configuration)

backends/*.yaml
(Backend-specific)

sandbox/*.yaml
(Sandbox-specific)
default: sandbox/default.yaml

Final Config
(most permissive wins)

 \ No newline at end of file diff --git a/docs/testing-code-execution.md b/docs/testing-code-execution.md deleted file mode 100644 index 0daa52c..0000000 --- a/docs/testing-code-execution.md +++ /dev/null @@ -1,142 +0,0 @@ -# Testing Code Execution Pattern Adoption - -## Overview - -This document describes how to verify that agents are using the code execution pattern instead of direct tool calls, and how to measure token reduction. - -## Test Script - -Run `./test-code-execution.sh` to automatically test agent adoption of the code execution pattern. - -The script: -1. Creates a test agent with Taskmaster template -2. Sends queries that should trigger execute_code usage -3. Verifies tool usage patterns -4. Cleans up test agent - -## Manual Testing - -### Test Case 1: Data Filtering - -**Query**: "Find all beads that haven't been updated in 30+ days and are still open. Show count by priority." - -**Expected behavior**: -- Agent uses `execute_code` with bash -- Loads beads once with `beads.list()` -- Filters in subprocess using `.filter()` and date comparison -- Aggregates by priority in subprocess -- Returns only summary statistics to context - -**Anti-pattern** (what NOT to do): -- Multiple `beads.list()` calls -- Loading all bead data into context -- Using separate tool calls for each filter condition - -### Test Case 2: Batch Operations - -**Query**: "Update all priority 1 beads to priority 2. Show progress." - -**Expected behavior**: -- Agent uses `execute_code` with bash -- Queries beads once with `beads.query({ priority: 1 })` -- Loops through results in subprocess -- Updates each bead with `beads.update()` -- Reports progress periodically (e.g., every 10 items) -- Returns final summary to context - -**Anti-pattern**: -- Alternating between tool calls and model responses for each update -- Loading full bead details into context for each item - -### Test Case 3: Multi-Source Joins - -**Query**: "Show all open beads that have related KB entries. Include KB reference count." - -**Expected behavior**: -- Agent uses `execute_code` with bash -- Loads beads and KB entries in parallel -- Joins data in subprocess -- Filters to only beads with KB references -- Returns enriched summary to context - -**Anti-pattern**: -- Sequential tool calls for each bead -- Loading all KB content into context -- Performing join logic through model reasoning - -## Measuring Token Reduction - -### Baseline (Direct Tool Calls) - -For a query that processes 1000 beads: -- Tool definitions: ~150k tokens (all tools loaded upfront) -- Tool results: ~500k tokens (all bead data in context) -- Total: ~650k tokens - -### With Code Execution - -For the same query: -- Tool definitions: ~2k tokens (only beads.sh loaded) -- Code execution: ~1k tokens (bash code) -- Results: ~500 tokens (filtered summary) -- Total: ~3.5k tokens - -**Reduction**: 99.5% (650k → 3.5k tokens) - -### Measuring in Practice - -1. **Enable token tracking** in agent stats: - ```bash - 9p read anvillm//stats - ``` - -2. **Compare before/after**: - - Run same query with direct tool calls - - Run same query with code execution - - Calculate reduction percentage - -3. **Look for indicators**: - - Fewer tool calls in agent logs - - Smaller context window usage - - Faster response times - - Lower API costs - -## Verification Checklist - -- [ ] Agent uses `execute_code` for data transformations -- [ ] Agent loads tool definitions on-demand (progressive discovery) -- [ ] Agent filters data in subprocess before returning to context -- [ ] Agent uses loops in subprocess instead of alternating tool calls -- [ ] Token usage reduced by 80-99% compared to direct tool calls -- [ ] Response times improved -- [ ] Agent behavior is correct (no regressions) - -## Common Issues - -### Agent still using direct tool calls - -**Cause**: Agent prompt doesn't emphasize code execution pattern strongly enough. - -**Fix**: Add more prominent code execution examples to agent context. Show specific use cases relevant to agent's role. - -### Agent loads all tools upfront - -**Cause**: Agent doesn't understand progressive discovery workflow. - -**Fix**: Add explicit instructions to list `anvillm/tools/anvilmcp/` first, then read only needed tool files. - -### Agent passes large datasets through context - -**Cause**: Agent doesn't filter data in subprocess. - -**Fix**: Add examples showing filtering, aggregation, and summarization in subprocess before returning results. - -## Success Criteria - -An agent successfully adopts the code execution pattern when: - -1. **Token usage** is reduced by 80-99% for data-heavy operations -2. **Tool calls** are consolidated (1-2 execute_code calls instead of 10+ direct tool calls) -3. **Context efficiency** is improved (only summaries and filtered results in context) -4. **Correctness** is maintained (same or better results than direct tool calls) -5. **Response time** is improved (fewer round-trips to model) diff --git a/kiro-cli/README.md b/kiro-cli/README.md index 5e1cb47..0e3a5e1 100644 --- a/kiro-cli/README.md +++ b/kiro-cli/README.md @@ -12,11 +12,10 @@ Install the AnviLLM MCP server to enable agent communication tools: ./kiro-cli/install-mcp.sh ``` -This configures the `anvilmcp` MCP server in `~/.kiro/settings/cli.json`, providing tools: -- `check_inbox` - Read messages from agent's inbox -- `send_message` - Send messages to other agents or user -- `list_sessions` - List all active sessions -- `set_state` - Set agent state (idle, running, etc.) +This configures the `anvilmcp` MCP server in `~/.kiro/settings/cli.json`, providing a single tool: +- `execute_code` - Execute bash scripts in an isolated subprocess (sandboxed via landrun) + +All AnviLLM functionality (messaging, sessions, beads, etc.) is accessed through 9P and helper scripts within `execute_code`. ### Agent Configuration