# HUD Protocol — Structured Activity Feed **Status:** Design — not yet implemented **Author:** Titan **Created:** 2026-03-15 --- ## Overview The HUD (Heads-Up Display) is a real-time activity feed in the webchat UI showing what the agent is doing — tool calls, reasoning, session events. It replaces the previous flat `string[]` log with a structured, hierarchical, machine-readable event stream. --- ## Goals - **Structured** — args and results are objects, not truncated strings - **Hierarchical** — tools nest inside turns; thinking nests inside turns - **Incremental** — events stream as they happen (`_start` / `_end` pairs) - **Machine-readable** — Titan can inspect full tool output from HUD panel - **Replay-aware** — history replay emits same events, flagged `replay: true` - **Extensible** — new event types add without breaking existing consumers --- ## Wire Format All HUD events are sent as WebSocket messages with `type: "hud"`. ### Base shape ```ts interface HudEvent { type: 'hud' event: HudEventKind id: string // always crypto.randomUUID() — globally unique correlationId?: string // provider call_* id (tools) or turnId (turns) — used for _start/_end pairing parentId?: string // correlationId of containing turn ts: number // Unix ms timestamp replay?: boolean // true when emitted from history replay } ``` ### Event kinds ``` tool_start tool_end think_start think_end turn_start turn_end received // instantaneous — no _end counterpart ``` --- ## Event Shapes ### turn_start / turn_end ```json { "type": "hud", "event": "turn_start", "id": "", "correlationId": "", "ts": 1741995000000 } { "type": "hud", "event": "turn_end", "id": "", "correlationId": "", "ts": 1741995004500, "durationMs": 4500 } ``` `correlationId` = `session.turnId`. Frontend pairs `turn_start` ↔ `turn_end` by matching `correlationId`. --- ### think_start / think_end ```json { "type": "hud", "event": "think_start", "id": "", "correlationId": "", "parentId": "", "ts": 1741995000050 } { "type": "hud", "event": "think_end", "id": "", "correlationId": "", "parentId": "", "ts": 1741995000820, "durationMs": 770 } ``` `correlationId` = `crypto.randomUUID()` generated at `think_start`, held in session state until `think_end`. `parentId` = containing turn's `correlationId`. --- ### tool_start / tool_end ```json { "type": "hud", "event": "tool_start", "id": "", "correlationId": "call_123f898fc88346afaec098e0", "parentId": "", "tool": "read", "args": { "path": "workspace-titan/SOUL.md" }, "ts": 1741995001000 } { "type": "hud", "event": "tool_end", "id": "", "correlationId": "call_123f898fc88346afaec098e0", "parentId": "", "tool": "read", "result": { "ok": true, "text": "# SOUL\n…", "bytes": 2048 }, "ts": 1741995001210, "durationMs": 210 } ``` `id` = always `crypto.randomUUID()`. `correlationId` = provider tool call id (`call_*`) — frontend pairs `tool_start` ↔ `tool_end` by matching `correlationId`. **Result shapes by tool:** ```ts // Shared — viewer-navigable file reference interface FileArea { startLine: number // 1-indexed, in resulting file endLine: number } interface FileMeta { path: string // raw as passed to tool viewerPath: string // normalized: /home/openclaw/.openclaw/ stripped area?: FileArea } // read args: { path: string, offset?: number, limit?: number } result: { ok: boolean, file: FileMeta, area: FileArea, text: string, bytes: number, truncated: boolean } // area inferred from offset/limit → { startLine: offset, endLine: offset+limit } // write args: { path: string, operation: 'write' } result: { ok: boolean, file: FileMeta, area: FileArea, bytes: number } // area: { startLine: 1, endLine: lineCount(written content) } // edit args: { path: string, operation: 'edit' } result: { ok: boolean, file: FileMeta, area: FileArea } // area: line range of replaced block in resulting file // append args: { path: string, operation: 'append' } result: { ok: boolean, file: FileMeta, area: FileArea, bytes: number } // area: { startLine: prevLineCount+1, endLine: newLineCount } // exec args: { command: string } result: { ok: boolean, exitCode: number, stdout: string, truncated: boolean, mentionedPaths?: FileMeta[], error?: string } // mentionedPaths: file paths parsed from command string via regex // web_search / web_fetch args: { query?: string, url?: string } result: { ok: boolean, text: string, url?: string, truncated: boolean } // browser / canvas / nodes / message / sessions_* args: { action: string, [key: string]: any } result: { ok: boolean, summary: string, raw?: any } // fallback (unknown tool) result: { ok: boolean, raw: any } ``` **UI labels derived from operation:** | Tool + operation | Label | |---|---| | `read` | `👁 path:L10–L50` | | `write` | `✏️ path (overwrite)` | | `edit` | `✏️ path:L22–L28` | | `append` | `✏️ path:L180–L195` | | `exec` | `⚡ command` | | `web_fetch` | `🌐 url` | | `web_search` | `🔍 query` | --- ### received (instantaneous) No `_end` counterpart. Emitted when the backend acknowledges a control action. ```ts interface ReceivedEvent extends HudEvent { event: 'received' subtype: ReceivedSubtype label: string // human readable description payload?: Record } type ReceivedSubtype = | 'new_session' | 'agent_switch' | 'stop' | 'kill' | 'handover' | 'reconnect' | 'message' ``` **Examples:** ```json { "type": "hud", "event": "received", "id": "", "subtype": "new_session", "label": "/new received — resetting session", "payload": { "previousAgent": "tester" }, "ts": 1741995010000 } { "type": "hud", "event": "received", "id": "", "subtype": "agent_switch", "label": "switch → titan", "payload": { "from": "tester", "to": "titan" }, "ts": 1741995020000 } { "type": "hud", "event": "received", "id": "", "subtype": "stop", "label": "stop received — aborting turn", "payload": { "state": "AGENT_RUNNING" }, "ts": 1741995030000 } { "type": "hud", "event": "received", "id": "", "subtype": "reconnect", "label": "reconnected — replaying history", "payload": { "sessionKey": "agent:titan:web:nico" }, "ts": 1741995040000 } { "type": "hud", "event": "received", "id": "", "subtype": "message", "label": "message received", "payload": { "preview": "hello world" }, "ts": 1741995050000 } ``` --- ## ID Policy All `id` fields are always `crypto.randomUUID()` — globally unique, no exceptions. `correlationId` carries the external or domain identifier used for `_start`/`_end` pairing: | Event | `correlationId` source | |---|---| | `tool_start` / `tool_end` | Provider tool call id (`call_*`) | | `turn_start` / `turn_end` | `session.turnId` | | `think_start` / `think_end` | `crypto.randomUUID()` generated at `think_start`, reused at `think_end` | | `received` | — (no pairing needed) | Frontend pairing logic: - `*_start` → create node, index by `correlationId` (or `id` if no `correlationId`) - `*_end` → look up by `correlationId` → merge result, set `state: 'done'`, set `durationMs` - FIFO fallback if `correlationId` is missing or unmatched — match oldest running node of same tool/type --- ## Frontend Data Model ```ts interface HudNode { id: string type: 'turn' | 'tool' | 'think' | 'received' subtype?: string state: 'running' | 'done' | 'error' label: string // human readable tool?: string args?: Record // full, structured result?: Record // full, structured startedAt: number endedAt?: number durationMs?: number children: HudNode[] // tools/thinks nest inside turns replay: boolean } ``` **Pairing logic:** - Maintain `Map` (pending nodes) - `*_start` → create node with `state: 'running'`, insert into map + tree - `*_end` → look up by id, merge result, set `state: 'done'`, set `durationMs`, remove from map - `received` → create complete node immediately (`state: 'done'`) - If `*_end` arrives with unknown id → FIFO fallback (match oldest running node of same tool) --- ## Emission Points | Source | Events emitted | |---|---| | `gateway.ts` `chat.tool_call` | `tool_start` | | `gateway.ts` `chat.tool_result` | `tool_end` | | `gateway.ts` `chat.thinking` | `think_start` (on first chunk) | | `gateway.ts` `chat.done` / `chat.final` | `think_end` (if thinking was open), `turn_end` | | `gateway.ts` `chat.delta` / `turn_start` | `turn_start` (on first delta) | | `server.ts` `handleNew` | `received` subtype=`new_session` | | `server.ts` `handleSwitchAgent` | `received` subtype=`agent_switch` | | `server.ts` `handleStopKill` | `received` subtype=`stop` or `kill` | | `server.ts` `handleHandoverRequest` | `received` subtype=`handover` | | `server.ts` reconnect path | `received` subtype=`reconnect` | | `session-watcher.ts` history | all of the above with `replay: true` | --- ## Rendering (HudActions.vue) - Tree view: turns at root, tools/thinks as children - Each row: `[state-dot] [icon] [label] [duration-badge]` - Expandable: click row → show args/result as formatted JSON - `replay: true` nodes rendered at reduced opacity - Running nodes animate (pulse dot) - Max visible: last 50 nodes (configurable) - History replay nodes collapsible as a group --- ## Migration Replaces: - `hudActionsLog: ref` in `sessionHistory.ts` - String-building in `handleSessionEntry` / `handleSessionHistory` - Raw string push in `useAgentSocket.ts` lines 111–114 Preserved: - `chatStore.pushSystem()` — chat bubble system messages (errors, stop confirm) — different concern - `lastSystemMsgRef` — status text in HudRow --- ## Open Questions - Should `received.message` events be emitted for every user message? (could be noisy) - Should thinking content be stored in the node (for expand) or discarded? - Cap on `result.text` size stored in node? (full fidelity vs memory)