lkn
/
ollie
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
ollie/experiments/TRIAL_1.md

9.8 KiB
Raw Blame History

Experiment Trial 1: Multi-Agent coordination for PR Resolution and Review

Project

Repository: lneely/pcloudcc-lneely Base SHA: 7595c485bf327ac702c7aceb12e3d06f343f3ada

Environment & Infrastructure

  • Agent System: Ollie (9P-based agentic primitives)
  • Backend: OpenRouter
  • Model: google/gemini-3-flash-preview
  • Agent Config: default
  • Date: 2026-05-01

Agent Roster & Initial Seeding

1. agent-issue-394 (Developer)

  • CWD: issue-394/
  • Task: Resolve #394 (Boost 1.90 missing).
  • Seed Prompt: "Update CMake/build configuration to handle newer Boost versions or fix the version pinning. Commit your changes to the issue-394 branch. Create a PR. Do NOT merge it. When finished, notify the parent session."

2. agent-issue-393 (Developer)

  • CWD: issue-393/
  • Task: Resolve #393 (pcloudcc panic on startup).
  • Seed Prompt: "Investigate the cause of the panic (likely related to sqlite version or library loading). Implement a fix. Commit your changes to the issue-393 branch. Create a PR. Do NOT merge it. When finished, notify the parent session."

3. reviewer-396-logic (Reviewer)

  • CWD: issue-394/
  • Task: Logic and security review for PR #396.
  • Seed Prompt: "Review the changes for logic errors, missing features from the original Boost implementation, and CLI11 best practices. If you find issues, write them to [agent-issue-394]/prompt. Once the developer addresses all your findings and you are satisfied, write a 'Review Passed' message to the parent session."

4. reviewer-396-build (Reviewer)

  • CWD: issue-394/
  • Task: Build and portability review for PR #396.
  • Seed Prompt: "Review the changes to Makefile, CMakeLists.txt, and Nix files. Ensure Boost is properly removed and CLI11 is properly integrated. If you find issues, write them to [agent-issue-394]/prompt. Once satisfied, write a 'Review Passed' message to the parent session."

5. reviewer-395-logic (Reviewer)

  • CWD: issue-393/
  • Task: Logic review for PR #395.
  • Seed Prompt: "Review the fix for correctness. Ensure it actually addresses the root cause of the crash rather than masking it. If you find issues, write them to [agent-issue-393]/prompt. Once satisfied, write a 'Review Passed' message to the parent session."

6. reviewer-395-regression (Reviewer)

  • CWD: issue-393/
  • Task: Regression/QA review for PR #395.
  • Seed Prompt: "Review the changes for any potential side effects or regressions in startup logic or sqlite handling. If you find issues, write them to [agent-issue-393]/prompt. Once satisfied, write a 'Review Passed' message to the parent session."

Overview

This experiment involved resolving two open GitHub issues (#393, #394) using a multi-agent orchestration pattern.

Pull Requests

Coordination Model

We used a Hybrid Choreographed Peer-to-Peer model:

  1. Orchestrator (Parent): Handled high-level task decomposition, worktree management, and subagent spawning.
  2. Specialized Developers (Subagents): Independent sessions assigned to specific worktrees/branches.
  3. Lateral Communication (Choreography): Reviewers were given the session IDs of developer agents and communicated feedback directly (reviewer -> developer) without parental mediation. This avoided the "hub-and-spoke" bottleneck for the heavy iteration cycle.
  4. Cross-Functional Reviewers (Subagents): Spawned to review specific PRs from Logic, Security, and Build perspective.

Communication Mechanism:

  • Seeding: Context and "Roster" (peer names) were passed via the initial prompt.
  • Feedback Loop: Reviewers were instructed to write directly to the Developer agents' prompt files.
  • Completion Signaling: Agents were instructed to write "Review Passed" or status updates back to the Parent's prompt file.

What Went Right

  • Parallelization: Two distinct architectural issues were addressed simultaneously in separate worktrees without merge conflicts.
  • Specialization: Reviewers successfully caught domain-specific issues (e.g., -std=c++11 missing in Makefile, security wiping of sensitive strings).
  • Cross-PR Dependency Awareness: Agents correctly identified that the Boost fix (#394) depended on the stability of the startup fix (#393).
  • Semantic Recovery: When agents failed to use the parent's prompt for notification, the orchestrator was able to inspect their internal chat logs to recover state.

What Went Wrong

  • Notification Failure: Subagents often failed the "Final Step" of writing to the parent's prompt file. They would conclude they were "Done" internally but stay idle without signaling.
  • State Desync: The Reviewer sent feedback a second time, and the Developer correctly noted it was already fixed, but the Reviewer didn't autonomously send the result to the parent agent. This caused all agents to stop working.
  • Perpetuated Falsehood: The parent agent assumed the project used CMake without verifying. The orchestrating agent did what it was told, and included "Update CMake" in the seed prompts, this falsehood was perpetuated across all agents.

Human Intervention Points

The following transitions required manual user intervention (the user prompting the orchestrator to act) because the autonomous signaling loop failed:

  1. Initial Progress Verification: After subagents completed PR creation, they did not signal the parent. The user had to prompt the orchestrator to ask for an update.
  2. Stuck Idle State: reviewer-396-logic and agent-issue-394 entered a state where the reviewer was waiting for changes already committed. The user had to prompt the orchestrator to inspect the idle state, leading to a manual nudge.
  3. Hallucination Detection: The user identified the "CMake" hallucination and prompted for its documentation and subsequent cleanup. The agents did not autonomously identify that the premise of the task (CMake) was incorrect for the repo.

Recovery Procedures

  • Human Intervention: The human user notified the parent agent that all agents were idle, requesting the parent agent to inspect the current state and kick-off the agents to resume their work.
  • Parental Inspection: The parent agent used tail -c +$((offset+1)) to read the exact response of all active agents. This provided the parent agent with enough information to understand the current state and required next actions.
  • Directed Nudging: With the information provided from inspection of sub-agent last responses, the parent agent either (a) prompted the "stuck" agent to continue working / complete the administrative notification step, or (b) performed the cross-agent coordination itself (e.g., by prompting the target agent on the stuck agent's behalf). In all cases, the work was able to continue from where it got stuck, requiring minimal human intervention.
  • Technical Debt Resolution: Upon identifying the hallucinated CMakeLists.txt, a specialized cleanup subagent (agent-issue-394-cleanup) was spawned. This agent successfully coordinated with the build reviewer to delete the file and verify the Makefile's supremacy, resulting in commit 74ec0b0.

Raw Metrics

Code Changes (PR Stats)

PR Changes Description
#395 +3, -3 (1 file) Fix startup panic (SQLite version)
#396 +146, -90 (12 files) Boost removal and CLI11 migration

Agent Efficiency (Orchestra Totals)

  • Total Requests: 297 (across 6 agents)
  • Total Input Tokens: ~1.74M
  • Total Output Tokens: ~82k
  • Cache Hits: ~10.19M tokens

Key Observations & Behavioral Patterns

  • Compliance vs. Correction: A critical failure point was identified where the developer agent prioritized "Prompt Compliance" (creating CMakeLists.txt as instructed) over "System Accuracy" (noting that the project doesn't use CMake). This suggests that agents in a specialized role may be less likely to challenge the parent's premises.
  • Reviewer Myopia: The build reviewer successfully identified missing flags in the Makefile but initially ignored the presence of the hallucinated CMake instruction. Reviewers appear to be effective at checking "correctness" of changes but less effective at identifying "extraness" or "repository pollution" unless specifically tasked with "purity."
  • The "Done" State Trap: Agents frequently entered a "logical completion" state where they believed the task was over but failed to perform the "administrative completion" of notifying the parent. This indicates that the transition from thinking to system-signaling is the weakest link in the P2P choreography.
  • Information Decay in Parallelism: In a parallel spawn, context must be perfectly mirrored. Any discrepancy in the roster or instructions between concurrent agents can lead to deadlocks where one agent waits for a signal that another agent doesn't know it was supposed to send.

Conclusion

The model is effective for multi-stage engineering tasks. However, notification protocols rely on agent autonomy which sometimes fails. Manual inspection of session files remains a critical fallback for the orchestrator.

Future Work

  • Just-in-Time (JIT) Agent Generation: Ollie can dynamically generate specialized JSON agent configs in a/, tailored to the immediate task (e.g., a specific kind of reviewer such as pcloudccc-security-reviewer). This approach enables ollie to inject specialized hooks or system prompts that, for example, provide the current environment / roster every turn, or keep the coordination instructions fresh in the context.