From c44c44dae3efad60b894a57adf57a3f309f3f9bb Mon Sep 17 00:00:00 2001
From: Levi Neely <lneely@pascom.net>
Date: Fri, 9 Oct 2026 18:53:59 +0200
Subject: [PATCH] doc: add Phase 40 (JSONL chat log format) to evolution.md

---
 doc/evolution.md | 87 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 87 insertions(+)

diff --git a/doc/evolution.md b/doc/evolution.md
index db1993d..5005859 100644
--- a/doc/evolution.md
+++ b/doc/evolution.md
@@ -101,6 +101,7 @@ gantt
     Explicit tool loading          :done, 2026-08-21, 1d
     Streaming rdwr + event filters :done, 2026-08-22, 1d
     Bypass coordination + state UI :done, 2026-10-06, 1d
+    JSONL chat log format          :done, 2026-10-07, 1d
 ```
 
 ## Current Size (Oct 6)
@@ -2247,3 +2248,89 @@ ChatPane.qml            +12  onBypassResolved handler
 SessionTree.qml         +55  bypass indicator (⚠) and state indicator (dot)
 data/scripts/o          +65  bypass, approve, deny commands
 ```
+
+## Phase 40: JSONL Chat Log Format (Oct 7)
+
+Replaced the custom `[[[role#id]]]...[[[end]]]` delimiter format with JSON Lines (JSONL) — one JSON object per line. Fixes streaming partial display and eliminates brittle regex parsing.
+
+### Old format (deleted)
+
+```
+[[[user#abc12345]]]
+hello
+[[[end]]]
+[[[assistant#def67890]]]
+response text
+[[[end]]]
+```
+
+Problems:
+- `[[[end]]]` must appear on its own line, but streaming chunks don't respect line boundaries
+- Regex-based parsing is fragile and error-prone
+- No standard tooling for reading/searching
+
+### New format (JSONL)
+
+```json
+{"role":"user","id":"abc12345","content":"hello"}
+{"role":"assistant","id":"def67890","content":"response text"}
+{"role":"call","id":"aaa11111","name":"shell","content":"{\"cmd\":\"ls\"}"}
+{"role":"tool","id":"bbb22222","content":"file1.txt\nfile2.txt","format":"text"}
+```
+
+Each line is a complete JSON object with:
+- `role`: user, assistant, context, reasoning, call, tool, error, info, retry, stalled
+- `id`: 8-char hex block ID (deterministic from sha256)
+- `content`: block text
+- `name`: tool/function name (for call blocks)
+- `format`: output format hint (for tool blocks)
+- `partial`: true if streaming in progress
+
+### Server changes
+
+**format/block.go** (new): `Block` struct with JSONL marshal/unmarshal, `RenderBlock` for filtered text output.
+
+**Deleted**: `format/event.go`, `format/format.go`, `format/format_test.go` — old delimiter format entirely removed.
+
+**agent/chat.go**: Dual logs — `rawLog` (JSONL for programmatic access) and `textLog` (rendered text for humans). `AppendBlock` writes to both. Streaming starts from beginning for replay.
+
+**agent/chatlog.go**: Event handler emits JSONL blocks via `flushPartial` (partial=true per chunk) and `closeBlock` (partial=false final). Skips internal events: `state`, `usage`, `limitretry`.
+
+**fs/spec.go**: New 9P files:
+- `log.raw` — JSONL stream (blocking, suitable for live tailing)
+- `log` — Rendered text (non-blocking, last 64KB)
+- `block` — Rdwr lookup: write block ID, read JSON
+
+Deleted: `chat`, `chat.raw`, `chat.search`
+
+### GUI changes
+
+**chatblockmodel.cpp**: Complete rewrite. Uses `QJsonDocument` for parsing instead of regex state machine. Filters `context` role from display. Removed `m_inContext` tracking, `renderBlock` method, and regex patterns.
+
+**ollie9pclient.cpp**: `readLogForSession` reads from `/log.raw` (JSONL) instead of `/log` (plain text). Streaming via `startActiveAgentStreams` uses `/log.raw`.
+
+**chatblockmodel.h**: Added `Context` to `ChatBlock::Type` enum. Removed unused members.
+
+### Context blocks
+
+Context injected by the server (user prompts, matched skills, tool hints) now emits as a separate `"context"` role block, followed by the actual user message. Context blocks are:
+- Sent to the LLM (wrapped in `<context>` tags in history)
+- Filtered from GUI display (role-based, not pattern-based)
+- Filtered from rendered text log via `RenderBlock`
+
+### Source changes
+
+```text
+format/block.go           +68  new Block struct, JSONL marshal/unmarshal, RenderBlock
+format/event.go           -67  deleted
+format/format.go          -107 deleted
+format/format_test.go     -188 deleted
+agent/chat.go             +95  dual logs, AppendBlock, BlockByID, streaming from start
+agent/chatlog.go          +35  JSONL emission, skip internal events
+fs/spec.go                +45  log.raw, log, block files; removed chat/chat.raw/chat.search
+chatblockmodel.cpp        -400 JSONL parsing, removed regex state machine
+chatblockmodel.h          +3   Context type, removed m_inContext
+ollie9pclient.cpp         +5   read from log.raw
+```
+
+Net: **-680 lines** — simpler, more robust, standard format.