Labeling and review
Turn real workflow runs into labels a person stands behind, then hold some back so you can tell whether a new evaluator is actually right.
In short
Logged runs are labeled by the evaluator unless a person corrects them. Those corrections, plus any examples you upload, are what let a custom evaluator beat the current one, and what let you prove that it did. The review queue asks for labels where they help most.
A model trained only on its own past answers learns to copy itself. That is fine for volume. It is useless for getting better. The way an eval gets smarter is a person saying "this run was wrong, and here is the right label", on work that actually happened.
In the dashboard this lives under Evals: Definitions, then the human-label pages (the nav says Labelled examples), Review queue, and the eval's own data page.
Every eval's counts are on that index, so you can see which questions still need work. Here they are all still Not ready.
Those labels do two jobs at once:
- They teach. Training prefers a human correction over the logged verdict on the same input.
- They examine. About 30% of labeled inputs are held back as a test set the model never trains on. That is how later comparisons can say the new evaluator is right, not merely that it imitates the old one.
Where labels come from
Two places, both counting toward the same eval.
Corrections on real runs. Open a workflow run and use Correct this verdict on the eval node. Or work through the review queue, which saves each answer as a correction immediately. A second correction on the same run replaces the first.
Uploaded examples. If you already have a golden set (a spreadsheet of tickets, a folder of CVs with an agreed label), upload CSV or JSONL from the template on the eval's data page. The template is the eval's input fields plus a label column. The same input with the same decision is counted as a duplicate and skipped. The same input with a different decision is rejected as a conflict: the eval will not store two answers for one case.
Examples live on the eval, so every workflow that uses it can train on them. A logged run has upstream context (the trigger summary, an insight). An uploaded example does not, until you replay it through a workflow. Replay estimates the cost first, then holds that amount against spend, the same way a training job does. Skip replay if you are only building a golden set; include it when those examples should join a node's dataset.
Northlight's must-have skills eval already has 33 human labels: 14 met, 11 partial, 8 missing. The grade is still Not ready, because the held-back test set has 12 of the 20 rows it needs.
What to notice: the grade is about human labels, not about how many runs the evaluator has logged. A thousand uncorrected runs still leave you with nothing to examine a new model against. Upload and replay live on this same page.
The review queue
Evals, then Review queue. Pick the workflow and the eval node. You see one run at a time: the input, what the upstream steps produced, the evaluator's verdict (with P(label) on a choice), and a key per possible answer.
The queue is ranked so a label from you is spent where it moves the needle:
| Why it is here | What that usually means |
|---|---|
| Low confidence | The chosen label's probability was low. The evaluator was unsure. |
| Models disagree | A comparison already ran, and the candidate chose a different decision. |
| Rare label | This verdict is uncommon in recent runs. Training will otherwise barely see it. |
Keys 1 to 9, then 0, match the labels in contract order. S skips. Each save moves you to the next run. The last few labels of the session stay listed underneath, so you can see the shape of what you just taught.
This is the highest-leverage hour a lead can spend on evals. You are not reviewing at random. You are labeling the unsure, the disputed, and the rare, which is exactly what a later custom evaluator needs and exactly what a held-back test set must contain.
Take Amelia Chen on Northlight's must-have skills eval. She is run 1 of 19 on candidate-company-match. The input is a Founding CTO at Juniper Labs. The built-in evaluator answered met with P(met) 58% and flagged low confidence. You press 1 to keep met, 2 for partial, or 3 for missing. That save is a human label. Seniority still has its own queue, with in-band / below / above, when you are teaching that question instead.
Notice you are not scoring the whole workflow. You are answering the same question the eval asks. The rest of the graph can stay as it is. The P(label) on screen is this probability, not a second confidence number.
Quality, before you freeze anything
Each eval has a quality report, on its data page and again in the training wizard's preview.
| Grade | Plain reading |
|---|---|
| Not ready | Too few rows to train, a label with no examples, or more than a little conflict between labels on the same input. |
| Usable | You can train, but the report will tell you what to add: more human labels, a rarer class, fewer duplicates. |
| Good | Every label shows up enough, the test set is large enough to judge correctness, a healthy share of rows are human-labeled, and conflicts are gone. |
The report also shows:
- how many distinct inputs you have (repeated tickets teach less than they appear to)
- the share of rows a person labeled
- per-label counts, so you can see a missing class
- how often the current evaluator already agrees with a person
- the test-set size, and how many it still needs before correctness is judged
Conflicts and duplicates are called out by name. Fix conflicts (one input, two human answers) before you freeze. Duplicates are waste; they do not poison the set the way a conflict does.
The held-back test set
Once an eval has 20 human labels for its current answer contract, about 30% of those labeled inputs are held back. They are never trained on. They are the rows later screens mark human-verified.
A few properties that matter in practice:
- Nothing is held back until those 20 labels exist. Below that, the UI will tell you there is not enough data to judge correctness. Label more, then train.
- An input on the test set never moves back to training, even if you edit or delete labels later. A changed answer contract (different labels, or a different score range) starts a fresh assignment.
- Logged runs whose input matches a test input are kept out of training, so the model cannot study the answer key.
The badge you will see on comparisons, on runs people chose to label, is this set: not a random sample of all traffic, the sample your reviewers actually stood behind.
Freezing a dataset
Training does not read the live run log. It reads a frozen snapshot of a date range for one workflow eval node.
In Evals, then Training, then Train a model:
- Pick the workflow and the eval node.
- Pick a date range. A small chart shows how many runs (and how many already have a human correction) sit in each day.
- Preview the dataset. You see row counts, where labels came from (human correction, evaluator, or uploaded example), warnings, the quality grade, and a cost estimate.
- Freeze. The snapshot is what the job will train on. Deleting an example later does not change a freeze that already happened.
On each row, the label is: the human correction if there is one, else the logged evaluator output. Uploaded examples keep their own label. That is the whole rule. It is why an hour on the review queue, spent before you freeze, is worth more than an hour spent after.
Training rows and test-set rows are counted separately. The minimum to start a job is about training rows. Test rows are extra, and they are the ones that later say whether the model is right. The wizard's first step, picking the workflow and eval node, is on Fine-tuning.
If the eval's output contract has moved since the workflow version pinned it, freeze and training ask you to refresh the node from the definition first. That keeps the labels, the snapshot, and the live node on the same question.
What to do this week
- Correct the runs you already know are wrong, from the run log.
- Spend a short session on the review queue for the eval you care about most.
- Upload any golden examples you already trust, and replay them through the workflow if they should join training.
- Watch the quality grade. When it leaves Not ready and the test set is filling, you are ready to fine-tune.