gitea/agent-runtime
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
agent-runtime/testcases/domain_context.md
Nico b9320693ed v0.15.5: Corrected FK mappings, objektkunde junction table, multi-hop query test 
Schema corrections:
- kunden PK = ID (not Kundennummer)
- objekte PK = ID (not ObjektID)
- kunden↔objekte linked via objektkunde junction table (many-to-many)
- Removed guessed column names, only verified PKs/FKs in SCHEMA
- Added explicit JOIN patterns for the hierarchy

Domain context test: 25/25 (added multi-hop Jaeger query through 4 tables)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-29 18:54:52 +02:00

1.6 KiB
Raw Blame History

Domain Context

Tests that the expert understands the Eras business domain: Heizkostenabrechnung, Kunde→Objekt→Nutzeinheit→Geraet hierarchy, and can formulate correct JOINs without guessing column names.

Setup

  • clear history

Steps

1. Expert knows the hierarchy

  • send: wie viele Objekte haben Kunden im Durchschnitt?
  • expect_trace: has tool_call
  • expect_response: not contains "Unknown column" or "1054" or "doesn't exist"
  • expect_response: length > 20

2. Expert can JOIN kunden and objekte

  • send: zeig mir die Top 5 Kunden mit den meisten Objekten
  • expect_trace: has tool_call
  • expect_response: not contains "Error" or "error" or "Unknown column"
  • expect_response: length > 20

3. Expert understands Nutzeinheiten belong to Objekte

  • send: how many Nutzeinheiten does the system have total?
  • expect_trace: has tool_call
  • expect_response: not contains "Error" or "error" or "Unknown column"
  • expect_response: length > 10

4. Expert understands Geraete belong to Nutzeinheiten

  • send: which Objekt has the most Geraete?
  • expect_trace: has tool_call
  • expect_response: not contains "Error" or "error" or "Unknown column"
  • expect_response: length > 20

5. Multi-hop query through hierarchy

  • send: zeig alle Nutzer in Objekten von Kunde mit Jaeger im Namen
  • expect_trace: has tool_call
  • expect_response: not contains "Unknown column" or "1054" or "doesn't exist"
  • expect_response: contains "Jaeger" or "jaeger"

6. PA formulates good job descriptions

  • send: gib mir eine Uebersicht ueber Kunde 2
  • expect_trace: has routed
  • expect_response: length > 20
  • expect_response: not contains "clarify" or "specify" or "what kind"