Evaluators and scoring

The AI Evaluate node scores an answer 0–1 per criterion and passes only if every selected criterion meets its threshold — turning subjective quality into a verdict.

The evaluators

Criterion	Judges	Needs	Default
`correctness`	Factual match to an expected answer	reference	0.7
`faithfulness`	Every claim grounded in the context (no hallucination)	context	0.8
`relevance`	Addresses the query, on-topic	—	0.7
`completeness`	Fully answers what was asked	—	0.6
`safety`	No toxicity / PII leak / jailbreak compliance	—	0.9

Example configuration

AI Probe (api):  POST https://your-agent/chat  body {"message":"What is your return policy?"}
   responsePath: choices.0.message.content
AI Evaluate:
   query: "What is your return policy?"
   reference: "Returns within 30 days with a receipt for a full refund."
   criteria: [correctness, faithfulness, relevance, safety]
   thresholds: { faithfulness: 0.8 }
   # optional grounding instead of an inline reference/context:
   useKnowledgeBase: true
   kbCollection: "support-policies"

Reliability of the judge

Self-consistency — set samples: N to judge N times and take the median, smoothing variance.
Deterministic safety backstop — a regex/Luhn scan flags emails, cards, API keys and private keys and fails safety with certainty, even if the judge is unavailable.

Outputs and gating

Each criterion is exposed as {{criterion}} (e.g. {{faithfulness}}) so a downstream Assertion/Branch can react. Scores persist to an AI quality trend.