Fine-tuning
Train a custom evaluator on a workflow's logged runs and human labels, then read the result before you compare it with the current one.
In short
A custom evaluator is your desk's judgment, learned from real runs and the labels people put on them. You freeze a dataset, you see the cost, you train. The job is refused if it would pass your spend limit, and the new model is only made available if it clears a safety bar against the current evaluator.
The built-in evaluator is a strong general judge. Fine-tuning is for the last mile: the way this team uses these labels on this workflow. Northlight's idea of in-band for a founding CTO is not a universal fact. It is a pattern in their corrections. A custom evaluator is how that pattern stops living only in the review queue.
You do not fine-tune to encode a new salary band or a new location rule. Those are claims. You fine-tune when the judgment itself has a shape the built-in evaluator keeps missing, and you have the labels to show it.
What it learns from
One workflow, one eval node, one frozen dataset. Each training row is:
- the state the evaluator sees on a live run (request fields, then upstream outputs)
- the label: a person's correction if they gave one, otherwise the logged verdict; an uploaded example keeps its own label
That is why labeling comes first. A dataset of mostly evaluator labels produces a model that imitates the current judge. Human labels are what let it beat that judge, and what later comparisons use to say so.
Examples you uploaded are included by default, replayed through this workflow so they have the same upstream shape as a real run. You can turn that off in the preview if you want logged runs only.
The training wizard
Evals, then Training, then Train a model. Jobs you have already run sit on that Training list. The three below are completed jobs for support-ticket-triage urgency.
Train a model opens a four-step wizard, and the URL keeps your place, so a run log can deep-link into the preview.
Workflow
Pick the workflow and the eval node. Custom evaluators belong to that node. A model trained on Northlight's seniority eval will not drop into urgency on a support workflow. The first step lists every eval node you can train, from candidate-company-match and from support-ticket-triage.
Date range
Choose which logged runs to include. The volume chart shows runs with an output for the node, and how many of those already have a human correction. Wider is not always better: a range that mixes two versions of the question will warn about schema drift.
Preview
Counts, label sources, warnings, quality, sample rows, and the cost estimate the job will hold. This is the last cheap place to go back to the review queue. If human share is low, if a label is missing, if conflicts remain, fix those and preview again.
Train
The estimate, the remaining headroom on the tenant spend limit, and Train. If the job would pass the cap, the dashboard says so and nothing starts. One training job can run at a time on a given eval node.
The job moves Queued to Training to Validating to Available. You can cancel while it is running; the hold is released. On success, the measured cost is billed, never more than the estimate that was held. On failure, the hold is released and the model is not made available.
When it finishes, those four phases are all done, and the result cards sit on the job page. The same figures appear on the model. Holdout accuracy is high, but the banner is the one to read: not enough human labels to judge correctness (0 of 20 test rows). Matches corrections is 100% on the three rows a person corrected.
The new evaluator gets an id the dashboard shows on Models. It is a drop-in only for this workflow's eval node, and only while its status is available.
Reading the result
The job page and the model page show the same cards. Read them in this order, which is also the order the dashboard leads with.
Did we have enough human labels to judge correctness? If the held-back test set is still thin, the UI tells you how many test rows you still need. Do not treat any other number as "the model is right" until that bar is met. Label more, freeze a new dataset, train again.
On the held-back human labels, how often did it give the person's answer? That is the only figure marked human-verified. It is measured on inputs the model never trained on. A range around it is shown when there is enough data; overlapping ranges with the current evaluator mean "as good", not "proven better".
On those same test rows, how does it compare with the current evaluator? This is the like-for-like. The job is not made available if the new model is clearly worse than the one you would replace, on those human labels.
How often did it match the training labels on a held-out slice of the dataset? Most of those labels are still the old evaluator's. A high number here with a test-set score no better than before means the model learned to imitate. Add human labels, especially for rare classes, and train again.
Until the test set is large enough, the dashboard will not pretend a percentage is correctness. That is deliberate. A confident number on the wrong sample is how teams pin a worse judge.
Cost, in one paragraph
Training is estimated up front, reserved against the tenant spend limit, and refused with nothing created if it would pass the cap. You see that estimate and the remaining headroom before you confirm. Failed and canceled jobs release the hold. Try, replay, and comparison runs are metered the same way, separately. There is no hidden training bill: if the wizard cannot hold the estimate, it does not start.
When to train, and when not to
Train when:
- the quality report is at least Usable, and you have a reason (a pattern in corrections the built-in evaluator keeps missing)
- the test set is on its way to being large enough to judge the result
- the eval's question is stable (same labels, same score range) across the date range
Do not train when:
- the brief changed (update claims or the request; see the FAQ on the overview)
- you have almost no human labels (you will get an imitator, and nothing honest to compare it with)
- you are still rewriting instructions every day (save, Try, and label first; then freeze)
A custom evaluator that never beats, or at least matches, the current one on human labels is not promoted. That is the next page: Compare and roll out.