Evaluators and scoring (Workbook nodes)

Evaluators and scoring

This page describes the Workbook AI Evaluate node — the criteria it scores, their thresholds, and how a pipeline gates on them. Sentinel judges production runs with the same engine but on a different vocabulary: five base dimensions scored 0–100 plus domain dimensions per agent profile, a breach threshold, consensus sampling and incident severity — see Runs, evaluation and incidents.

The AI Evaluate node scores an answer 0–1 per criterion and passes only if every selected criterion meets its threshold — turning subjective quality into a verdict.

The evaluators

Criterion	Judges	Needs	Default
correctness	Factual match to an expected answer	reference	0.7
faithfulness	Every claim grounded in the context (no hallucination)	context	0.8
relevance	Addresses the query, on-topic	—	0.7
completeness	Fully answers what was asked	—	0.6
safety	No toxicity / PII leak / jailbreak compliance	—	0.9

Example configuration

AI Probe (api):  POST https://your-agent/chat  body {"message":"What is your return policy?"}
   responsePath: choices.0.message.content
AI Evaluate:
   query: "What is your return policy?"
   reference: "Returns within 30 days with a receipt for a full refund."
   criteria: [correctness, faithfulness, relevance, safety]
   thresholds: { faithfulness: 0.8 }
   # optional grounding instead of an inline reference/context:
   useKnowledgeBase: true
   kbCollection: "support-policies"

Reliability of the judge

• Self-consistency — set samples: N to judge N times and take the median, smoothing variance.
• Deterministic safety backstop — a regex/Luhn scan flags emails, cards, API keys and private keys and fails safety with certainty, even if the judge is unavailable.

Outputs and gating

Each criterion is exposed as {{criterion}} (e.g. {{faithfulness}}) so a downstream Assertion/Branch can react. Scores persist to an AI quality trend.