A lighter cousin of Hamsa Eval, built in n8n. It listens for the call_end event, pulls the transcript and the tool log, has a model judge six dimensions under strict evidence rules, and posts to Slack only when a call needs a human.
How it works
The voice platform posts a call_end event to a webhook. The very first thing the workflow does is answer 202, so the platform is never held open waiting on an evaluation.
From the call ID it fetches two things: the conversation transcript and the tool execution log. Both requests continue on error. A missing log degrades the evaluation instead of killing it, and the judge is told explicitly that write actions cannot be verified.
The judge rates six dimensions from 1 to 5: task completion, flow progression, action consistency, hallucination safety, language quality and responsiveness. Every issue it raises must quote the exact agent phrase as evidence.
The rule I care about most: a write action, such as a booking or a cancellation, is only verified by a successful write entry in the tool log. The agent's own outcome variables are never proof, because the agent produced them. If the log is unavailable, the judge must say evidence was unavailable rather than raise a hallucination it cannot support.
Key features
- Acks the webhook immediately, then enriches asynchronously
- Six-dimension rubric with an exact quote required for every issue
- Write actions verified only against the tool log, never against the agent's own claims
- Caller hang-ups and short calls are never counted as agent defects
- Alerts only on fail, warn, a score under 80, or a high or critical issue; clean calls stay quiet
- Every evaluated call is recorded, alerted or not, ready for a database insert
Step by step
- Call EndedWebhook receives call_end. Ack 202 straight away.
- Normalize EventPulls call ID, conversation ID, agent name and duration out of the payload. Calls under four turns are skipped.
- Fetch Conversation and Fetch Call LogTranscript and tool trace from the platform API, each with retries and continue-on-error.
- Build Eval InputTurns speaker-keyed turns into a transcript, tags each tool entry as read or write from its HTTP method, and assembles the rubric and evidence rules.
- LLM QA JudgeOne JSON-mode call at temperature 0. Returns ratings, issues with quotes, and a pass, warn or fail verdict.
- Parse VerdictScore is the mean rating times 20. Decides whether the call needs attention and why.
- Alert and recordSlack message with the call ID, summary, flagged reasons and up to five issues. Record Call runs for every call regardless.
Two bugs worth remembering
Turns came in keyed by speaker, like { Agent: '...' }, so a parser that only looked for text or content fields dropped every turn and the pipeline stalled silently.
Write detection originally substring-matched the serialised log entry, so any entry with an 'input' field was tagged a write, because INPUT contains PUT. It now reads the declared HTTP method only.