lkn
/
ollie
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
ollie/experiments/TRIAL_1.md

11 KiB
Raw Blame History

Experiment Trial 1: Multi-Agent coordination for PR Resolution and Review

Project

Repository: lneely/pcloudcc-lneely Base SHA: 7595c485bf327ac702c7aceb12e3d06f343f3ada

Environment & Infrastructure

  • Agent System: Ollie (9P-based agentic primitives)
  • Backend: OpenRouter
  • Model: google/gemini-3-flash-preview
  • Agent Config: default
  • Date: 2026-05-01

Agent Roster & Initial Seeding

1. agent-issue-394 (Developer)

  • CWD: issue-394/
  • Task: Resolve #394 (Boost 1.90 missing).
  • Seed Prompt: "Update CMake/build configuration to handle newer Boost versions or fix the version pinning. Commit your changes to the issue-394 branch. Create a PR. Do NOT merge it. When finished, notify the parent session."

2. agent-issue-393 (Developer)

  • CWD: issue-393/
  • Task: Resolve #393 (pcloudcc panic on startup).
  • Seed Prompt: "Investigate the cause of the panic (likely related to sqlite version or library loading). Implement a fix. Commit your changes to the issue-393 branch. Create a PR. Do NOT merge it. When finished, notify the parent session."

3. reviewer-396-logic (Reviewer)

  • CWD: issue-394/
  • Task: Logic and security review for PR #396.
  • Seed Prompt: "Review the changes for logic errors, missing features from the original Boost implementation, and CLI11 best practices. If you find issues, write them to [agent-issue-394]/prompt. Once the developer addresses all your findings and you are satisfied, write a 'Review Passed' message to the parent session."

4. reviewer-396-build (Reviewer)

  • CWD: issue-394/
  • Task: Build and portability review for PR #396.
  • Seed Prompt: "Review the changes to Makefile, CMakeLists.txt, and Nix files. Ensure Boost is properly removed and CLI11 is properly integrated. If you find issues, write them to [agent-issue-394]/prompt. Once satisfied, write a 'Review Passed' message to the parent session."

5. reviewer-395-logic (Reviewer)

  • CWD: issue-393/
  • Task: Logic review for PR #395.
  • Seed Prompt: "Review the fix for correctness. Ensure it actually addresses the root cause of the crash rather than masking it. If you find issues, write them to [agent-issue-393]/prompt. Once satisfied, write a 'Review Passed' message to the parent session."

6. reviewer-395-regression (Reviewer)

  • CWD: issue-393/
  • Task: Regression/QA review for PR #395.
  • Seed Prompt: "Review the changes for any potential side effects or regressions in startup logic or sqlite handling. If you find issues, write them to [agent-issue-393]/prompt. Once satisfied, write a 'Review Passed' message to the parent session."

Overview

This experiment involved resolving two open GitHub issues (#393, #394) using a multi-agent orchestration pattern.

Pull Requests

Coordination Model

We used a Hybrid Choreographed Peer-to-Peer model:

  1. Orchestrator (Parent): Handled high-level task decomposition, worktree management, and subagent spawning.
  2. Specialized Developers (Subagents): Independent sessions assigned to specific worktrees/branches.
  3. Lateral Communication (Choreography): Reviewers were given the session IDs of developer agents and communicated feedback directly (reviewer -> developer) without parental mediation. This avoided the "hub-and-spoke" bottleneck for the heavy iteration cycle.
  4. Cross-Functional Reviewers (Subagents): Spawned to review specific PRs from Logic, Security, and Build perspective.

Communication Mechanism:

  • Seeding: Context and "Roster" (peer names) were passed via the initial prompt.
  • Feedback Loop: Reviewers were instructed to write directly to the Developer agents' prompt files.
  • Completion Signaling: Agents were instructed to write "Review Passed" or status updates back to the Parent's prompt file.

What Went Right

  • Parallelization: Two distinct architectural issues were addressed simultaneously in separate worktrees without merge conflicts.
  • Specialization: Reviewers successfully caught domain-specific issues (e.g., -std=c++11 missing in Makefile, security wiping of sensitive strings).
  • Cross-PR Dependency Awareness: Agents correctly identified that the Boost fix (#394) depended on the stability of the startup fix (#393).
  • Semantic Recovery: When agents failed to use the parent's prompt for notification, the orchestrator was able to inspect their internal chat logs to recover state.

What Went Wrong

  • Notification Failure: Subagents often failed the "Final Step" of writing to the parent's prompt file. They would conclude they were "Done" internally but stay idle without signaling.
  • State Desync: A "Wait Loop" occurred where a Reviewer sent feedback a second time, and the Developer correctly noted it was already fixed, but the Reviewer didn't autonomously re-check until nudged.
  • Assumptive Hallucination & Review Failure: Both the orchestrator and the developer agents initially assumed the project used CMake. I included "Update CMake" in the seed prompts, and agent-issue-394 fulfilled this by creating a redundant CMakeLists.txt file in a strictly Makefile-based project. While the build reviewer noted the need to fix the Makefile, it failed to demand the removal of the hallucinated CMakeLists.txt. Consequently, a phantom file (CMakeLists.txt) remained in the final PR. In this context, "phantom" refers to an AI-generated file that is functionally dead—the project build is driven strictly by make, and this file serves no utility. It represents "technical debt as hallucination," where a file is created to satisfy a flawed prompt and persists because it doesn't technically "break" the build, even though it pollutes the repository with misinformation. This represents a failure in the "Review-Passed" criteria for build-system purity.

Human Intervention Points

The following transitions required manual user intervention (the user prompting the orchestrator to act) because the autonomous signaling loop failed:

  1. Initial Progress Verification: After subagents completed PR creation, they did not signal the parent. The user had to prompt the orchestrator to ask for an update.
  2. Review Feedback Deadlock: reviewer-396-logic and agent-issue-394 entered a state where the reviewer was waiting for changes already committed. The user had to prompt the orchestrator to inspect the idle state, leading to a manual nudge.
  3. Hallucination Detection: The user identified the "CMake" hallucination and prompted for its documentation and subsequent cleanup. The agents did not autonomously identify that the premise of the task (CMake) was incorrect for the repo.

Recovery Procedures

  • Parental Inspection: The Orchestrator used tail -c +$((offset+1)) to read the exact response of idle agents.
  • Directed Nudging: Based on log inspection, the Parent manually injected prompts into the "stuck" agent's FIFO to force a re-evaluation of state.
  • Technical Debt Resolution: Upon identifying the hallucinated CMakeLists.txt, a specialized cleanup subagent (agent-issue-394-cleanup) was spawned. This agent successfully coordinated with the build reviewer to delete the file and verify the Makefile's supremacy, resulting in commit 74ec0b0.

Raw Metrics

Code Changes (PR Stats)

PR Changes Description
#395 +3, -3 (1 file) Fix startup panic (SQLite version)
#396 +146, -90 (12 files) Boost removal and CLI11 migration

Agent Efficiency (Orchestra Totals)

  • Total Requests: 297 (across 6 agents)
  • Total Input Tokens: ~1.74M
  • Total Output Tokens: ~82k
  • Cache Hits: ~10.19M tokens

Key Observations & Behavioral Patterns

  • Compliance vs. Correction: A critical failure point was identified where the developer agent prioritized "Prompt Compliance" (creating CMakeLists.txt as instructed) over "System Accuracy" (noting that the project doesn't use CMake). This suggests that agents in a specialized role may be less likely to challenge the parent's premises.
  • Reviewer Myopia: The build reviewer successfully identified missing flags in the Makefile but initially ignored the presence of the hallucinated CMake file. Reviewers appear to be effective at checking "correctness" of changes but less effective at identifying "extraness" or "repository pollution" unless specifically tasked with "purity."
  • The "Done" State Trap: Agents frequently entered a "logical completion" state where they believed the task was over but failed to perform the "administrative completion" of notifying the parent. This indicates that the transition from thinking to system-signaling is the weakest link in the P2P choreography.
  • Information Decay in Parallelism: In a parallel spawn, context must be perfectly mirrored. Any discrepancy in the roster or instructions between concurrent agents can lead to deadlocks where one agent waits for a signal that another agent doesn't know it was supposed to send.

Conclusion

The model is effective for multi-stage engineering tasks. However, notification protocols rely on agent autonomy which sometimes fails. Manual inspection of session files remains a critical fallback for the orchestrator.

Future Work

  • Status Polling Mechanism: Implement a mandatory status file for all agents that the orchestrator can poll. This would prevent having to tail chat files to estimate if an agent is stuck or done.
  • Verification of Premises: Require agents to perform an "Environment Check" before acting on prompts to prevent fulfillment of hallucinated requirements (e.g., CMake in a Makefile-only project).
  • Administrative Handshake: Force a "Sync" step at the end of every subagent task where they must receive an ACK from the parent before idling, reducing the "Done State Trap."
  • Purity Reviewers: Introduce specialized reviewer scripts that look for "extraness" (extra files, unrelated changes, or phantom technical debt) rather than just functional correctness.
  • Post-Turn Environmental Hooks: Implement hooks for sub-agents that automatically run environmental checks (e.g., build verification or linting) after every turn, ensuring that premises and system state remain valid without manual oversight.
  • Just-in-Time (JIT) Agent Generation: Ollie could dynamically generate specialized JSON agent configs in a/, tailored to the immediate task (e.g., a "Makefile Purity Expert"). These ephemeral agents could be kept if they prove reusable or deleted immediately after the subagent session concludes to prevent configuration rot.