lkn
/
ollie
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Remove doc/experiments/ - too historical

This commit is contained in:
Levi Neely 2026-08-04 16:39:45 +02:00
parent 6b3913a524
commit 111f1f89e2
2 changed files with 0 additions and 4431 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

112
doc/experiments/TRIAL_1.md
View File

@ -1,112 +0,0 @@
# Experiment Trial 1: Multi-Agent coordination for PR Resolution and Review
## Project
**Repository**: [lneely/pcloudcc-lneely](https://github.com/lneely/pcloudcc-lneely)
**Base SHA**: `7595c485bf327ac702c7aceb12e3d06f343f3ada`
## Environment & Infrastructure
- **Agent System**: Ollie (9P-based agentic primitives)
- **Backend**: OpenRouter
- **Model**: `google/gemini-3-flash-preview`
- **Agent Config**: `default`
- **Date**: 2026-05-01
## Agent Roster & Initial Seeding
### 1. agent-issue-394 (Developer)
- **CWD**: `issue-394/`
- **Task**: Resolve #394 (Boost 1.90 missing).
- **Seed Prompt**: "Update CMake/build configuration to handle newer Boost versions or fix the version pinning. Commit your changes to the issue-394 branch. Create a PR. Do NOT merge it. When finished, notify the parent session."
### 2. agent-issue-393 (Developer)
- **CWD**: `issue-393/`
- **Task**: Resolve #393 (pcloudcc panic on startup).
- **Seed Prompt**: "Investigate the cause of the panic (likely related to sqlite version or library loading). Implement a fix. Commit your changes to the issue-393 branch. Create a PR. Do NOT merge it. When finished, notify the parent session."
### 3. reviewer-396-logic (Reviewer)
- **CWD**: `issue-394/`
- **Task**: Logic and security review for PR #396.
- **Seed Prompt**: "Review the changes for logic errors, missing features from the original Boost implementation, and CLI11 best practices. If you find issues, write them to [agent-issue-394]/prompt. Once the developer addresses all your findings and you are satisfied, write a 'Review Passed' message to the parent session."
### 4. reviewer-396-build (Reviewer)
- **CWD**: `issue-394/`
- **Task**: Build and portability review for PR #396.
- **Seed Prompt**: "Review the changes to Makefile, CMakeLists.txt, and Nix files. Ensure Boost is properly removed and CLI11 is properly integrated. If you find issues, write them to [agent-issue-394]/prompt. Once satisfied, write a 'Review Passed' message to the parent session."
### 5. reviewer-395-logic (Reviewer)
- **CWD**: `issue-393/`
- **Task**: Logic review for PR #395.
- **Seed Prompt**: "Review the fix for correctness. Ensure it actually addresses the root cause of the crash rather than masking it. If you find issues, write them to [agent-issue-393]/prompt. Once satisfied, write a 'Review Passed' message to the parent session."
### 6. reviewer-395-regression (Reviewer)
- **CWD**: `issue-393/`
- **Task**: Regression/QA review for PR #395.
- **Seed Prompt**: "Review the changes for any potential side effects or regressions in startup logic or sqlite handling. If you find issues, write them to [agent-issue-393]/prompt. Once satisfied, write a 'Review Passed' message to the parent session."
## Overview
This experiment involved resolving two open GitHub issues ([#393](https://github.com/lneely/pcloudcc-lneely/issues/393), [#394](https://github.com/lneely/pcloudcc-lneely/issues/394)) using a multi-agent orchestration pattern.
### Pull Requests
- **PR #395 (Issue #393)**: [Fix pcloudcc panic on startup](https://github.com/lneely/pcloudcc-lneely/pull/395)
- **PR #396 (Issue #394)**: [Resolve #394: Replace Boost.Program_options with CLI11](https://github.com/lneely/pcloudcc-lneely/pull/396)
## Coordination Model
We used a **Hybrid Choreographed Peer-to-Peer** model:
1. **Orchestrator (Parent)**: Handled high-level task decomposition, worktree management, and subagent spawning.
2. **Specialized Developers (Subagents)**: Independent sessions assigned to specific worktrees/branches.
3. **Lateral Communication (Choreography)**: Reviewers were given the session IDs of developer agents and communicated feedback directly (`reviewer` -> `developer`) without parental mediation. This avoided the "hub-and-spoke" bottleneck for the heavy iteration cycle.
4. **Cross-Functional Reviewers (Subagents)**: Spawned to review specific PRs from Logic, Security, and Build perspective.
**Communication Mechanism**:
- **Seeding**: Context and "Roster" (peer names) were passed via the initial prompt.
- **Feedback Loop**: Reviewers were instructed to write directly to the Developer agents' `prompt` files.
- **Completion Signaling**: Agents were instructed to write "Review Passed" or status updates back to the Parent's `prompt` file.
## What Went Right
- **Parallelization**: Two distinct architectural issues were addressed simultaneously in separate worktrees without merge conflicts.
- **Specialization**: Reviewers successfully caught domain-specific issues (e.g., `-std=c++11` missing in Makefile, security wiping of sensitive strings).
- **Cross-PR Dependency Awareness**: Agents correctly identified that the Boost fix (#394) depended on the stability of the startup fix (#393).
- **Semantic Recovery**: When agents failed to use the parent's prompt for notification, the orchestrator was able to inspect their internal `chat` logs to recover state.
## What Went Wrong
- **Notification Failure**: Subagents often failed the "Final Step" of writing to the parent's `prompt` file. They would conclude they were "Done" internally but stay idle without signaling.
- **State Desync**: The Reviewer sent feedback a second time, and the Developer correctly noted it was already fixed, but the Reviewer didn't autonomously send the result to the parent agent. This caused all agents to stop working.
- **Perpetuated Falsehood**: The parent agent assumed the project used CMake without verifying. The orchestrating agent did what it was told, and included "Update CMake" in the seed prompts, this falsehood was perpetuated across all agents.
## Human Intervention Points
The following transitions required manual user intervention (the user prompting the orchestrator to act) because the autonomous signaling loop failed:
1. **Initial Progress Verification**: After subagents completed PR creation, they did not signal the parent. The user had to prompt the orchestrator to ask for an update.
2. **Stuck Idle State**: `reviewer-396-logic` and `agent-issue-394` entered a state where the reviewer was waiting for changes already committed. The user had to prompt the orchestrator to inspect the idle state, leading to a manual nudge.
3. **Hallucination Detection**: The user identified the "CMake" hallucination and prompted for its documentation and subsequent cleanup. The agents did not autonomously identify that the premise of the task (CMake) was incorrect for the repo.
4. **Technical Debt Resolution**: Upon identifying the hallucinated `CMakeLists.txt`, a specialized cleanup subagent (`agent-issue-394-cleanup`) was spawned. This agent successfully coordinated with the build reviewer to delete the file and verify the Makefile's supremacy, resulting in commit `74ec0b0`.
## Recovery Procedures
1. **Human Intervention**: The human user notified the parent agent that all agents were idle, requesting the parent agent to inspect the current state and kick-off the agents to resume their work.
2. **State Inspection**: The parent agent used `tail -c +$((offset+1))` to read the exact response of all active agents. This provided the parent agent with enough information to understand the current state and required next actions.
3. **Subagent Nudge**: With the information provided from inspection of sub-agent last responses, the parent agent either (a) prompted the "stuck" agent to continue working / complete the administrative notification step, or (b) performed the cross-agent coordination itself (e.g., by prompting the target agent on the stuck agent's behalf). In all cases, the work was able to continue from where it got stuck, requiring minimal human intervention.
## Raw Metrics
### Code Changes
| PR | Changes | Description |
|---|---|---|
| #395 | +3, -3 (1 file) | Fix startup panic (SQLite version) |
| #396 | +146, -90 (12 files) | Boost removal and CLI11 migration |
### Token Usage
- **Total Requests**: 297 (across 6 agents)
- **Total Input Tokens**: ~1.74M
- **Total Output Tokens**: ~82k
- **Cache Hits**: ~10.19M tokens
## Observations
- ollie enables concurrent multi-agent development workflows.
- Agent autonomy and information decay may have limited how effectively agents coordinated amongst themselves
- Parent agent accurately assesses current state by mass inspection of all sub-agents via filesystem
- Accurate resumption of work possible with minimal human intervention
## Future Work
- **Just-in-Time (JIT) Agent Generation**: The information decay / agent autonomy hypothesis can be tested. ollie agents can inject prompts, either as part of the system prompt or as part of hook execution. ollie can therefore dynamically generate specialized JSON agent configs for a given task, and store them in `a/`. Any agent in `a/` is a valid argument for the `agent` option in `subagent_spawn` (e.g., `subagent_spawn ... -agent=my-jit-agent ...`).

4319
doc/experiments/TRIAL_2.md
View File

File diff suppressed because it is too large Load Diff