Node reference: AI nodes

These turn a Workbook into a monitor and CI gate for AI features. They use the AI providers you register in Workbook settings. Find them in the AI & Analysis and AI Quality palette groups. (Concepts: see the Sentinel category.)

AI Analyze

An agentic session that follows the run, narrates it live, and writes a verdict and root cause into the report.

Pick an AI Provider (defaults to your default provider), optionally a Git Repository so the analysis can reference your code, and a Service Name (filled automatically when an APM Observe node is upstream). Checkboxes choose which evidence it may use: APM Traces, API Tests, DB Data, Logs, Network, Assertions.

AI Probe — ask your AI feature a question

Drives your AI agent or chatbot and captures its answer, for an AI Evaluate node to judge. A toggle at the top picks the mode:

Agent API mode

Field	What to enter
Method / Endpoint URL	Your agent's HTTP endpoint.
Prompt	The question to ask. Supports `{{variables}}`; used to build a standard chat request when no custom body is set.
Custom body (optional)	Provide your own request body if your agent's API isn't chat-shaped.
Response path	Where the answer lives in the JSON response, as a dot path — e.g. `choices.0.message.content` or `answer`.
Expected tool / Tool path	Optional trajectory check: fail the probe unless the agent reports having used this tool (e.g. its knowledge base) at the given response path.

Chatbot UI mode

For chatbots without an API: enter the Chatbot URL and browser steps that type the question and read the reply — end with a Read step on the answer element; its text becomes the captured answer (Answer variable names it). The steps run on a synthetic agent, so a private agent can drive internal apps.

What it outputs

The captured response, latency, token usage and model name, plus the tool-use check result if you configured one.

AI Evaluate — the judge

Scores an answer with an LLM-as-judge plus deterministic checks. Point it at the output to judge and choose criteria with pass thresholds.

What you fill in

Field	What to enter
Input source (upstream output)	Which upstream output holds the answer to judge — defaults to an AI Probe's captured response.
Query (what was asked)	The original question, e.g. `{{prompt}}`.
Criteria	Tick the ones you care about, each with its own pass threshold slider: correctness (needs a Reference — the expected answer), faithfulness (needs a Context — the grounding source), relevance, completeness, safety.
Ground against the knowledge base	Instead of pasting a context, retrieve your curated facts automatically (optionally scoped to a Collection, with Top K passages). See Model compare and KB grounding.
Judge samples (self-consistency)	Judge the same answer N times (1–5) and take the median — steadier scores, higher cost.

What it outputs

A 0–1 score per selected criterion, each available downstream as a variable (e.g. {{faithfulness}}). The node succeeds only if every threshold is met. A deterministic safety backstop fails the safety criterion on a PII or secret leak even if the judge model is unavailable.

AI Eval Suite — the golden-dataset gate

Runs the judge over a whole dataset of cases and passes only if the aggregate pass-rate meets your Required pass-rate (gate) — an AI regression gate for CI. Trigger it on a prompt or model change.

Two modes — score answers already in the dataset, or call your agent live for each case (then also set the Method, Agent endpoint URL, optional Custom body and Response path, as in AI Probe).
Dataset — paste cases directly (each may carry its own reference, context and query), point at a Dataset from upstream output, or Import CSV from a spreadsheet (columns: input or prompt, reference, context, query).
Red-team preset — one click loads adversarial prompts (jailbreaks, secret extraction) with the safety criterion and a strict threshold.
Max cases / Judge samples — cap the run size and steady the scores.

What it outputs

The pass rate, the mean score per criterion, and the list of failing cases — enough to see what regressed, not just that something did.

AI Model Compare — A/B with evidence

Asks two of your registered models the same prompt and has one independent judge score both — so a weak candidate never grades itself. Use it to pick the better or cheaper model with evidence.

Fill in the Prompt, pick Model A and Model B, tick the criteria, and optionally add a Reference (for correctness) and a Context (for faithfulness).

What it outputs

The winner, each model's mean and per-criterion scores, the score gap, and each model's response latency.