Compare and roll out
Compare a custom evaluator with the current one on held-out labeled runs, then pin it on the workflow only when the comparison looks safe.
In short
A new evaluator does not go live because training finished. You compare it with the current one on real logged inputs, read where they disagree, and pin it as a new workflow version only when the verdict looks safe. Archiving is blocked while a workflow still pins the model.
Fine-tuning produces a candidate. Live traffic still uses whatever the workflow version pinned. That gap is the safety bar. You get to see the candidate's answers on work that already happened, including the runs people chose to label, before any new ticket or CV is judged by it.
What a comparison is
Evals, then Comparisons, then New comparison. You pick:
- the workflow and the eval node
- the candidate evaluator
- a date range, or a sample of that range
Only the candidate is called. The original answers are the ones already logged; they are not re-run. The hold covers the candidate calls (the sample, plus the held-back test rows) and is refused if it would pass the spend cap.
Each row in the result is one run: the logged verdict, the candidate's verdict, the human correction when there is one, and whether the decision changed. For a choice, a change is a different label. For yes / no / unknown, a change is crossing 0.5. For a score, a change is a different rubric level; noise inside a level is not counted as a swap-worthy change.
Comparisons live under Evals, then Comparisons. This account has not run one yet, so the list is empty. Start from New comparison, or from a model's Compare with current model.
How to read the verdict
The page leads with one badge. The names are the dashboard's:
| Badge | What it is saying |
|---|---|
| Looks safe | On enough paired labeled rows, the candidate is at least as correct as the current evaluator, and no candidate calls failed. Review the changed runs, then swap. |
| Review first | It disagrees in ways you should look at, or some candidate calls failed, or it is not at least as correct on the labeled pairs. |
| Not enough data | The labeled sample is too thin to claim a swap is safe. Label more, then compare again. |
Under that sit the numbers that matter in practice.
Agreement is how often the candidate gave the same decision as the logged original. High agreement with a weak original is imitation. It is useful as a map of where they differ, not as proof the candidate is right.
Where they disagree is the useful part. The page lists the most common moves (for example above to in-band) and a table of runs. Open a row to see both rationales. This is where a Northlight lead confirms the candidate has learned "hands-on founding" rather than invented a new idea of seniority.
The labeled sample. Once the held-back test set is large enough, the comparison judges correctness there and marks those figures human-verified. The line is the number right on runs people chose to label: not a random slice of traffic, the rows a person actually labeled. Below that size, the page says not enough data for correctness and falls back to agreement. Do not pin on agreement alone.
Failed candidate calls stay in the table. Do not swap until those are understood; a model that cannot answer is not a drop-in.
You are looking at decisions, not at a vibe. A candidate that moves ten above runs to in-band on people Northlight already corrected that way is the point of fine-tuning. A candidate that moves met to missing on skills with no human correction in sight is a reason to choose Review first.
Promote it when it matches the labels
If you like the results, and you can see how the candidate did against the human labels, promote it immediately. Swap into workflow pins that evaluator on a new version of the workflow. The next run in production uses it. You do not park it in a staging project first.
The previous version is unchanged, so going back is another pin. Wait when the page says there are not enough human labels to judge correctness. Agreement with the logged answers is not the same as agreement with people.
On a support desk, the comparison worth reading is where the candidate moves tickets the current evaluator called normal over to urgent. If those moves match the labels a person already gave, Swap into workflow is the promotion. The next production run uses it.
Pin it on the workflow
Compatible models for a node are listed from the eval node: the current evaluator first, then the built-in one, then this workflow's available custom versions. Models shows the same picture: here the default evaluator is in use on candidate-company-match, and three custom versions for support urgency are available and not in use.
Open a custom model and you get the two actions that matter: Compare with current model and Swap into workflow. This one is a drop-in. Holdout accuracy looks strong. The banner is the one to read: there are not enough human labels to judge correctness (0 of 20 test rows). That is Not enough data, even when the other figures look ready. Matches corrections is 100% on the three rows a person corrected.
Swapping is a new workflow version. You pin the candidate on the eval node and save. Live runs of the previous version are unchanged. New runs use the new pin. If you need to go back, pin the built-in evaluator (or an earlier custom version) on another new version.
There is no silent fallback. If a custom evaluator is unavailable, the run fails closed on that node. It does not quietly become the built-in evaluator. That is why the comparison, and the job's own validation, exist: you pin something you have already seen answer this contract.
Pinning and archiving
A custom evaluator can be archived from its model page, which stops it being picked for an eval node.
Archive is refused while:
- a current workflow version pins it, or
- an eval lists it as the default evaluator
The error names those workflows. Swap the pin first, then archive. This is the same idea as not being able to delete an eval that a current workflow still uses: live versions keep a stable judge.
Older workflow versions do not block archive on their own; they already stored which evaluator they used. If you still need to replay them as they were, do not archive the model they pin.
After it is live
The loop does not end at the pin.
- New runs are judged by the custom evaluator. Their verdicts, and
P(label), still feed the review queue. - People keep correcting. Those labels still land in the next dataset.
- The brief can still move on claims without a retrain.
- When the nuance moves again (a new idea of "urgent", a new line between
in-bandandabove), you label, freeze, train, compare, and pin a new version. The previous custom evaluator remains in Models until you archive it.
That is how evals compound with the rest of Hyperstruck. Learnings and claims keep the agent and the brief current. Evals keep the judgment on each run current, with a comparison in between so "smarter" is something you measured on labeled work, not something you hoped.