Hamsa Eval: an LLM-as-judge platform for voice agents

How it works

Voice agents fail in ways a demo never shows. A call sounds fine, the agent is polite, and it still books the wrong slot or claims to have sent a confirmation it never sent. Hamsa Eval exists to catch that at scale, on real production calls, without a person listening to each one.

Every finished call is hydrated into one record: transcript, the path the agent took through its conversation graph, and every tool call it made. That record runs through four layers, cheapest first. A deterministic navigation check costs nothing and runs on everything. A small model scores each node the agent visited. A larger model judges the whole conversation. A final layer scores compliance on its own.

The design choice that matters most is the split. Quality asks whether the caller was served well. Compliance asks whether the agent stayed inside its script and policy. An agent can do one without the other, and a single score hides which. Two scores tell the prompt engineer exactly where to look.

I designed and built it end to end: the REST API, the queue, the database schema and the React dashboard. It ships with regression detection across projects, so a prompt change that quietly degrades one client's agent shows up as a trend, and an MCP server so other agents can query results.

Key features

• Four-layer pipeline that runs the free deterministic check first and the expensive model last
• Quality and compliance scored independently, never blended
• Evaluates intent accuracy, hallucination, Arabic morphology and outcome success per call
• Regression detection across projects, surfaced on a React dashboard
• Background processing on a BullMQ queue so evaluation never sits on the call path
• MCP server so results can be queried by other tools and agents

Step by step

1. Call endsThe voice platform emits the event. Transcript, navigation path and tool log are fetched and joined into one record.
2. Layer 1, navigationDeterministic check of the path the agent took against the graph it should have followed. No model, no cost.
3. Layer 2, per nodeGPT-4.1-mini scores each node the agent visited for whether it did that node's job.
4. Layer 3, whole conversationGPT-4.1 judges the conversation as the caller experienced it: intent accuracy, hallucination, language, outcome.
5. Layer 4, complianceA separate pass scores script and policy adherence, so a helpful deviation and a harmful one are both visible.
6. Aggregate and storeScores land in PostgreSQL through a queue. Regression detection compares against the project's history.
7. DashboardTrends, per-call drill-down and alerts in React. Agents and tools can also query over MCP.

Why the split matters