How evals work
An eval is a question with labels or a score and instructions. Try it, read P(label), and pin it as a node on a workflow.
In short
Write what is judged, how to decide, and what the answer looks like. Try it on one input. Drop it onto a workflow as a node. The built-in evaluator can answer any eval; a custom evaluator answers only the contract it was trained for.
The question
Every eval has the same parts:
| Part | What you write | Why it matters |
|---|---|---|
| Name | A short name the dashboard and the workflow builder show | So people pick the right question |
| What is judged | One sentence, such as "How quickly this item needs a response" | This is the question the evaluator is answering |
| Instructions | How to decide, in the team's own words | This is where your nuance lives until you have a custom evaluator |
| Answer shape | Choice, score, or yes / no / unknown | So every run is comparable |
| Input fields | The request fields the eval reads, when you declare them | Try builds a form from these; a new workflow's trigger can too |
The instructions are not a prompt you keep tweaking in production. They are the written form of a judgment your desk already makes. If the written form is wrong, fix it here, Try it, then save. Published workflows keep the output contract they pinned; they do not pick up a rewritten question until you save a new workflow version.
Here is Northlight's must-have skills eval. The question is whether the CV meets the brief's required skills. The labels are met, partial, and missing. Try sits beside the editor, already filled with an Amelia Chen sample, not yet run.
Notice the labels carry the same kind of writing you would put in a playbook, attached to the answer the workflow will log. Support desks write the same shape with urgent / normal / low. Run once is how you see a verdict and P(label) without a full workflow run.
Three ways to answer
The dashboard offers three output types:
| Dashboard label | When to use it | What a run stores |
|---|---|---|
| Choice: one label from a list | A closed set you will route on, such as urgent / normal / low, or met / partial / missing | The chosen label, plus a probability per label |
| Score: a number in a range | Degree of fit or impact, with optional named levels from low to high | A number in the range you set |
| Yes / no / unknown | "Should a person look at this before anything automated happens?" | A likelihood between 0 and 1 |
Starters cover one of each:
- Urgency: choice,
urgent/normal/low - Severity: score, 0 to 4, with a five-level rubric from "no problem" to "outage, data loss or a security issue"
- Needs human review: yes / no / unknown, aimed at legal, billing, security, or a large account
You can write your own. Recruitment desks often want must-have skills as a choice, seniority as in-band / below / above, and overall as a 0 to 4 score. Support desks often want urgency plus "needs a human". The constraint is the same: one eval, one question. Three questions are three eval nodes, which a workflow can run together.
Built-in and custom evaluators
An evaluator is the model that answers the question.
- The built-in evaluator (the dashboard lists it as JEV, pinned to a release) can answer any eval: any labels, any score range, yes / no / unknown. New evals use it until you pin something else.
- A custom evaluator is trained on one workflow's eval node, on that node's output contract. It is a drop-in for that node only. It will not answer a different set of labels, and it will not run on a different workflow.
The dashboard says this next to Try: Built-in JEV answers any eval. A custom model answers only the contract it was trained on. If you pick a custom model that is not a drop-in, Try warns and refuses the call before anything is spent.
Custom evaluators exist because the built-in one is general. It has not sat on your review queue. When your labels show a pattern it keeps missing (founding-CTO "hands-on" vs "led 140", a desk that treats "cannot log in" as normal unless many users are named) you train one. Until then, the built-in evaluator is the right default.
How that training works is Fine-tuning. How you prove the new one is safe is Compare and roll out.
Try
Try is one eval call on one input. It is the fastest way to see whether the instructions do what you meant, without running the whole workflow.
- Open the eval from Evals, then Definitions.
- Choose Try.
- Pick the evaluator (built-in, or a custom model that is a drop-in for this contract).
- Fill the form built from the eval's input fields, or paste a flat JSON object if it has none.
- Run once.
You get the chosen label, P(label) for a choice, how long it took, and the cost. Try is billed against the tenant spend limit when it succeeds. A call the dashboard already knows is incompatible is blocked and not billed.
On the must-have skills eval above, Amelia Chen's sample is already in the Try form. The same loop holds for a support Urgency starter: paste a subject and body, run once, tighten the instructions, run again.
There is no separate "confidence" figure on the screen. Evaluators do report one internally, on different scales, and it is not what the dashboard shows. What you see for a choice is P(label). The review queue later shows that as P(met) 58% on a real Amelia Chen run.
Verdicts and P(label)
On a workflow run, each eval node logs a verdict: the chosen label or score, a short rationale when the evaluator gave one, and for a choice the probability of each label.
Read a choice like this:
- The value is the decision (
urgent,in-band,met). P(label)is how much of the probability mass sat on that chosen label.P(above) 69%means the evaluator pickedaboveand put 69% on it. The rest is on the other labels.- A low
P(label)is a good candidate for the review queue. The queue's "low confidence" reason is this number being under the line, not a second mystery score.
For a score, you see the number in the range, and the nearest named level when the eval has a rubric. For yes / no / unknown, you see the likelihood.
This is what makes quality measurable. You are not asking whether the workflow "felt right this week". You are asking how often the verdict matched a person, and where P(label) was low.
Evals as nodes in a workflow
A workflow may have up to three eval nodes. Each node stores:
- which eval definition it uses
- which evaluator will answer it
- a snapshot of the output contract (the labels or the score range)
That snapshot is why a published version is stable. Change the definition's labels on Tuesday, and Monday's workflow version still returns the labels it pinned. To pick up the new contract, save a new workflow version.
On a run, the eval sees:
- the request (the same input every node sees)
- upstream outputs: the trigger summary, an insight's findings, other evals that feed it
Important input fields are placed first, so the judgment is made from what you marked as important. Keep the fields that decide the answer early and short. A very long body still works; the fields that must weigh in should not be buried at the end.
Use in a new workflow, on the eval's row, opens the builder on an unsaved trigger to eval to answer draft, with the trigger fields taken from the eval's input schema. The builder can also create an eval inline when you add a node.
You cannot delete an eval while the current version of any workflow still has a node on it. Older versions do not block; their runs keep the contract they pinned.
On candidate-company-match, three evals sit between the insight and the answer: must-have skills, seniority, and overall.
What to notice: the evals are not the whole workflow. They are the judgments. The trigger and the insight assemble context from the request, from claims, and from prior work. The answer writes the result a person or an agent will read.
Where this sits
Labeling and review is how those verdicts become teaching data. Fine-tuning is how a custom evaluator learns from them. Evals is the overview, including when not to retrain.