lkn
/
ollie
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add Experiment Trial 1: Multi-agent coordination analysis

This commit is contained in:
Levi Neely 2026-05-01 11:57:47 +02:00
parent c709b0711f
commit 79395ef450
1 changed files with 110 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

110
experiments/TRIAL_1.md Normal file
View File

@ -0,0 +1,110 @@
# Experiment Trial 1: Multi-Agent coordination for PR Resolution and Review
## Project
**Repository**: [lneely/pcloudcc-lneely](https://github.com/lneely/pcloudcc-lneely)
**Base SHA**: `7595c485bf327ac702c7aceb12e3d06f343f3ada`
## Environment & Infrastructure
- **Agent System**: Ollie (9P-based agentic primitives)
- **Backend**: OpenRouter
- **Model**: `google/gemini-3-flash-preview`
- **Agent Config**: `default`
- **Date**: 2026-05-01
## Agent Roster & Initial Seeding
### 1. agent-issue-394 (Developer)
- **CWD**: `issue-394/`
- **Task**: Resolve #394 (Boost 1.90 missing).
- **Seed Prompt**: "Update CMake/build configuration to handle newer Boost versions or fix the version pinning. Commit your changes to the issue-394 branch. Create a PR. Do NOT merge it. When finished, notify the parent session."
### 2. agent-issue-393 (Developer)
- **CWD**: `issue-393/`
- **Task**: Resolve #393 (pcloudcc panic on startup).
- **Seed Prompt**: "Investigate the cause of the panic (likely related to sqlite version or library loading). Implement a fix. Commit your changes to the issue-393 branch. Create a PR. Do NOT merge it. When finished, notify the parent session."
### 3. reviewer-396-logic (Reviewer)
- **CWD**: `issue-394/`
- **Task**: Logic and security review for PR #396.
- **Seed Prompt**: "Review the changes for logic errors, missing features from the original Boost implementation, and CLI11 best practices. If you find issues, write them to [agent-issue-394]/prompt. Once the developer addresses all your findings and you are satisfied, write a 'Review Passed' message to the parent session."
### 4. reviewer-396-build (Reviewer)
- **CWD**: `issue-394/`
- **Task**: Build and portability review for PR #396.
- **Seed Prompt**: "Review the changes to Makefile, CMakeLists.txt, and Nix files. Ensure Boost is properly removed and CLI11 is properly integrated. If you find issues, write them to [agent-issue-394]/prompt. Once satisfied, write a 'Review Passed' message to the parent session."
### 5. reviewer-395-logic (Reviewer)
- **CWD**: `issue-393/`
- **Task**: Logic review for PR #395.
- **Seed Prompt**: "Review the fix for correctness. Ensure it actually addresses the root cause of the crash rather than masking it. If you find issues, write them to [agent-issue-393]/prompt. Once satisfied, write a 'Review Passed' message to the parent session."
### 6. reviewer-395-regression (Reviewer)
- **CWD**: `issue-393/`
- **Task**: Regression/QA review for PR #395.
- **Seed Prompt**: "Review the changes for any potential side effects or regressions in startup logic or sqlite handling. If you find issues, write them to [agent-issue-393]/prompt. Once satisfied, write a 'Review Passed' message to the parent session."
## Overview
This experiment involved resolving two open GitHub issues ([#393](https://github.com/lneely/pcloudcc-lneely/issues/393), [#394](https://github.com/lneely/pcloudcc-lneely/issues/394)) using a multi-agent orchestration pattern.
### Pull Requests
- **PR #395 (Issue #393)**: [Fix pcloudcc panic on startup](https://github.com/lneely/pcloudcc-lneely/pull/395)
- **PR #396 (Issue #394)**: [Resolve #394: Replace Boost.Program_options with CLI11](https://github.com/lneely/pcloudcc-lneely/pull/396)
## Coordination Model
We used a **Hybrid Choreographed Peer-to-Peer** model:
1. **Orchestrator (Parent)**: Handled high-level task decomposition, worktree management, and subagent spawning.
2. **Specialized Developers (Subagents)**: Independent sessions assigned to specific worktrees/branches.
3. **Lateral Communication (Choreography)**: Reviewers were given the session IDs of developer agents and communicated feedback directly (`reviewer` -> `developer`) without parental mediation. This avoided the "hub-and-spoke" bottleneck for the heavy iteration cycle.
4. **Cross-Functional Reviewers (Subagents)**: Spawned to review specific PRs from Logic, Security, and Build perspective.
**Communication Mechanism**:
- **Seeding**: Context and "Roster" (peer names) were passed via the initial prompt.
- **Feedback Loop**: Reviewers were instructed to write directly to the Developer agents' `prompt` files.
- **Completion Signaling**: Agents were instructed to write "Review Passed" or status updates back to the Parent's `prompt` file.
## What Went Right
- **Parallelization**: Two distinct architectural issues were addressed simultaneously in separate worktrees without merge conflicts.
- **Specialization**: Reviewers successfully caught domain-specific issues (e.g., `-std=c++11` missing in Makefile, security wiping of sensitive strings).
- **Cross-PR Dependency Awareness**: Agents correctly identified that the Boost fix (#394) depended on the stability of the startup fix (#393).
- **Semantic Recovery**: When agents failed to use the parent's prompt for notification, the orchestrator was able to inspect their internal `chat` logs to recover state.
## What Went Wrong
- **Notification Failure**: Subagents often failed the "Final Step" of writing to the parent's `prompt` file. They would conclude they were "Done" internally but stay idle without signaling.
- **State Desync**: A "Wait Loop" occurred where a Reviewer sent feedback a second time, and the Developer correctly noted it was already fixed, but the Reviewer didn't autonomously re-check until nudged.
- **Assumptive Hallucination & Review Failure**: Both the orchestrator and the developer agents initially assumed the project used CMake. I included "Update CMake" in the seed prompts, and `agent-issue-394` fulfilled this by creating a redundant `CMakeLists.txt` file in a strictly Makefile-based project. While the build reviewer noted the need to fix the `Makefile`, it failed to demand the removal of the hallucinated `CMakeLists.txt`. Consequently, a **phantom file** (`CMakeLists.txt`) remained in the final PR. In this context, "phantom" refers to an AI-generated file that is functionally dead—the project build is driven strictly by `make`, and this file serves no utility. It represents "technical debt as hallucination," where a file is created to satisfy a flawed prompt and persists because it doesn't technically "break" the build, even though it pollutes the repository with misinformation. This represents a failure in the "Review-Passed" criteria for build-system purity.
## Human Intervention Points
The following transitions required manual user intervention (the user prompting the orchestrator to act) because the autonomous signaling loop failed:
1. **Initial Progress Verification**: After subagents completed PR creation, they did not signal the parent. The user had to prompt the orchestrator to ask for an update.
2. **Review Feedback Deadlock**: `reviewer-396-logic` and `agent-issue-394` entered a state where the reviewer was waiting for changes already committed. The user had to prompt the orchestrator to inspect the idle state, leading to a manual nudge.
3. **Hallucination Detection**: The user identified the "CMake" hallucination and prompted for its documentation and subsequent cleanup. The agents did not autonomously identify that the premise of the task (CMake) was incorrect for the repo.
## Recovery Procedures
- **Parental Inspection**: The Orchestrator used `tail -c +$((offset+1))` to read the exact response of idle agents.
- **Directed Nudging**: Based on log inspection, the Parent manually injected prompts into the "stuck" agent's FIFO to force a re-evaluation of state.
- **Technical Debt Resolution**: Upon identifying the hallucinated `CMakeLists.txt`, a specialized cleanup subagent (`agent-issue-394-cleanup`) was spawned. This agent successfully coordinated with the build reviewer to delete the file and verify the Makefile's supremacy, resulting in commit `74ec0b0`.
## Raw Metrics
### Code Changes (PR Stats)
| PR | Changes | Description |
|---|---|---|
| #395 | +3, -3 (1 file) | Fix startup panic (SQLite version) |
| #396 | +146, -90 (12 files) | Boost removal and CLI11 migration |
### Agent Efficiency (Orchestra Totals)
- **Total Requests**: 297 (across 6 agents)
- **Total Input Tokens**: ~1.74M
- **Total Output Tokens**: ~82k
- **Cache Hits**: ~10.19M tokens
## Key Observations & Behavioral Patterns
- **Compliance vs. Correction**: A critical failure point was identified where the developer agent prioritized "Prompt Compliance" (creating `CMakeLists.txt` as instructed) over "System Accuracy" (noting that the project doesn't use CMake). This suggests that agents in a specialized role may be less likely to challenge the parent's premises.
- **Reviewer Myopia**: The build reviewer successfully identified missing flags in the `Makefile` but initially ignored the presence of the hallucinated CMake file. Reviewers appear to be effective at checking "correctness" of changes but less effective at identifying "extraness" or "repository pollution" unless specifically tasked with "purity."
- **The "Done" State Trap**: Agents frequently entered a "logical completion" state where they believed the task was over but failed to perform the "administrative completion" of notifying the parent. This indicates that the transition from *thinking* to *system-signaling* is the weakest link in the P2P choreography.
- **Information Decay in Parallelism**: In a parallel spawn, context must be perfectly mirrored. Any discrepancy in the roster or instructions between concurrent agents can lead to deadlocks where one agent waits for a signal that another agent doesn't know it was supposed to send.
## Conclusion
The model is effective for multi-stage engineering tasks. However, notification protocols rely on agent autonomy which sometimes fails. Manual inspection of session files remains a critical fallback for the orchestrator. Future trials should experiment with a dedicated "Status File" per session that agents are required to update, allowing the orchestrator to poll health without reading full chat transcripts.