gitea/agent-runtime
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
agent-runtime/testcases/smoketest.md
Nico 1000411eb2 v0.15.0: Frame engine (v3), PA + Expert architecture (v4-eras), live test streaming 
Frame Engine (v3-framed):
- Tick-based deterministic pipeline: frames advance on completion, not timers
- FrameRecord/FrameTrace dataclasses for structured per-message tracing
- /api/frames endpoint: queryable frame trace history (last 20 messages)
- frame_trace HUD event with full pipeline visibility
- Reflex=2F, Director=4F, Director+Interpreter=5F deterministic frame counts

Expert Architecture (v4-eras):
- PA node (pa_v1): routes to domain experts, holds user context
- ExpertNode base: stateless executor with plan+execute two-LLM-call pattern
- ErasExpertNode: eras2_production DB specialist with DESCRIBE-first discipline
- Schema caching: DESCRIBE results reused across queries within session
- Progress streaming: PA streams thinking message, expert streams per-tool progress
- PARouting type for structured routing decisions

UI Controls Split:
- Separate thinker_controls from machine controls (current_controls is now a property)
- Machine buttons persist across Thinker responses
- Machine state parser handles both dict and list formats from Director
- Normalized button format with go/payload field mapping

WebSocket Architecture:
- /ws/test: dedicated debug socket for test runner progress
- /ws/trace: dedicated debug socket for HUD/frame trace events
- /ws (chat): cleaned up, only deltas/controls/done/cleared
- WS survives graph switch (re-attaches to new runtime)
- Pipeline result reset on clear

Test Infrastructure:
- Live test streaming: on_result callback fires per check during execution
- Frontend polling fallback (500ms) for proxy-buffered WS
- frame_trace-first trace assertion (fixes stale perceived event bug)
- action_match supports "or" patterns and multi-pattern matching
- Trace window increased to 40 events
- Graph-agnostic assertions (has X or Y)

Test Suites:
- smoketest.md: 12 steps covering all categories (~2min)
- fast.md: 10 quick checks (~1min)
- fast_v4.md: 10 v4-eras specific checks
- expert_eras.md: eras domain tests (routing, DB, schema, errors)
- expert_progress.md: progress streaming tests

Other:
- Shared db.py extracted from thinker_v2 (reused by experts)
- InputNode prompt: few-shot examples, history as context summary
- Director prompt: full tool signatures for add_state/reset_machine/destroy_machine
- nginx no-cache headers for static files during development
- Cache-busted static file references

Scores: v3 smoketest 39/40, v4-eras fast 28/28, expert_eras 23/23

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-29 17:10:31 +02:00

2.3 KiB
Raw Blame History

Smoketest

Fast validation: one example per category, covers all 11 suite areas in ~2 minutes.

Setup

  • clear history

Steps

1. Reflex path (social/trivial skips Thinker)

  • send: hi there!
  • expect_response: length > 3
  • expect_trace: input.analysis.intent is "social"
  • expect_trace: input.analysis.complexity is "trivial"

2. Input analysis (German detection + question intent)

  • send: Wie spaet ist es?
  • expect_trace: input.analysis.language is "de"
  • expect_trace: input.analysis.intent is "question"
  • expect_response: length > 5

3. Frustrated tone detection

  • send: this is broken, nothing works and I'm sick of it
  • expect_trace: input.analysis.tone is "frustrated" or "urgent"
  • expect_response: length > 10

4. Button creation

  • send: create two buttons: Alpha and Beta
  • expect_actions: length >= 2
  • expect_actions: any action contains "alpha" or "Alpha"
  • expect_actions: any action contains "beta" or "Beta"

5. Dashboard feedback (Thinker sees buttons)

  • send: what buttons can you see in my dashboard?
  • expect_response: contains "Alpha" or "alpha" or "Beta" or "beta"

6. DB query (tool call + table)

  • send: show me 3 customers from the database
  • expect_trace: has tool_call
  • expect_response: length > 10

7. Director plan (complex request)

  • send: investigate which customers have the most devices in the database
  • expect_trace: has director_plan or decided
  • expect_trace: has tool_call
  • expect_response: length > 20

8. Memorizer state (facts + language tracking)

  • send: My dog's name is Bella
  • expect_state: facts any contains "Bella"
  • expect_state: language is "en" or "mixed"

9. Machine creation

  • send: create a navigation machine called "test" with initial state "ready" showing a Go button
  • expect_trace: has tool_call create_machine
  • expect_trace: machine_created id="test"

10. Counter with buttons

  • send: create a counter starting at 0 with increment and decrement buttons
  • expect_response: contains "counter" or "count" or "0" or "Zähler"
  • expect_actions: length >= 2

11. Language switch

  • send: Hallo, wie geht es dir?
  • expect_state: language is "de" or "mixed"
  • expect_response: length > 5

12. Expert routing (v4 only, safe to skip on v3)

  • send: show me 3 customers from the database
  • expect_trace: has tool_call
  • expect_response: length > 10