Evals

Most teams already know what good looks like. A recruiter can tell whether a candidate meets the brief. A support lead can tell an outage from a noisy ticket. That judgment usually lives in people's heads, in a spreadsheet of exceptions, or in a prompt that drifts the moment the world moves.

Evals put that judgment on the work you already run. Each answer is something you can count, correct, and improve. Quality stops being a feeling about last week's queue and becomes a number on the runs that actually happened. The system gets smarter from those corrections, so fewer tickets bounce, fewer candidates go down the wrong path, and fewer people are pulled in to rework a decision the workflow should have got right.

That is the point: judgment applied to real data and real runs, not a separate scoring project that goes stale.

What you get from it

None of this asks you to replace the tools you already use. Workflows sit behind the Hyperstruck MCP, so the agents you already run can call them. Evals are the judgment those workflows apply, configured in the dashboard.

How evals support workflows

A workflow is aimed at an outcome: triage a ticket, match a candidate to a brief, decide whether a person should look before anything automated goes out. An eval is a node in that graph. It does not try to do the whole job. It answers one question the rest of the workflow can trust.

• Urgency, severity, and needs human review ship as starters, ready to drop onto a first-line support workflow.
• A recruitment match can ask three questions in parallel: must-have skills, seniority fit, and an overall score against the brief.
• The workflow's answer node then writes in your voice, using those verdicts, the trigger summary, and anything an insight node found.

The eval is reusable. One "meets the brief" definition can sit on more than one workflow. Each workflow pins the evaluator and the output contract on that version, so editing the definition later does not silently change what a published workflow returns.

How to write the question, try it, and place it on a graph is in How evals work.

How it compounds with what you already invested in

Evals do not replace learnings, claims, or documents. They sit on top of them.

Documents and finished work fill claims and learnings. Those then ride into a workflow run, optionally with an insight. The eval node turns that context into a verdict. People label the verdicts that were wrong. Those labels train a custom evaluator, which the workflow can pin after a comparison.

• Claims hold what is true now: this client's seniority band, location policy, salary range, who owns the relationship. The workflow reads them. The eval judges against them. When the brief changes, you update the claim. You do not retrain.
• Learnings hold what to do: when a required document is missing, request it before declining. They steer the agent that is assembling the run. They are not a substitute for a scored verdict.
• Documents are where those claims and learnings are earned, with a passage behind each fact.
• Insights can run inside the workflow and hand the eval prior incidents, a rediscovered conversation, or a reason to act now. The eval then judges with that extra context in view. How it all fits together is the picture of the whole loop.
• The eval is the gate that turns all of that into a label or a score on this run, so the rest of the workflow can route, score, or escalate without re-litigating the same judgment.

The compounding loop is the human one. People correct verdicts on real runs. Those labels train a custom evaluator. You compare it with the current one, then pin it. The next run is judged the way your best people already judge, using the same claims and documents they already paid to assemble.

Example: a week on a live desk

This is an example, so you can see the loop on one desk. Northlight, here, is a recruiting agency. Juniper Labs, a Sydney applied-AI studio, has asked them to find a founding CTO: hands-on, still writing code, has led 10 to 30 engineers, hybrid in Sydney, salary USD 220,000 to 320,000.

That brief already lives as claims on Northlight's client-desk agent, sourced from the document they filed for Juniper. The workflow candidate-company-match does not hard-code the numbers. It takes the CV fields on the request, looks up the company side from claims, optionally runs an insight ("has her latest conversation changed whether she would consider an early-stage role?"), then asks three evals:

Amelia Chen's CV lands. She led 140 people company-wide at Lattice Health, still codes, and a recent conversation says she would consider a founding CTO role in Sydney. The built-in evaluator, looking at "led 140", often answers seniority above with something like P(above) 69%. A Northlight recruiter who knows the brief means hands-on founding, not manager-only, opens the run and corrects it to in-band.

That correction is now a human label. The review queue will keep surfacing runs like it: low P(label), rare labels, or places a candidate model already disagreed. When there are enough labels, Northlight freezes a dataset, trains a custom evaluator, and compares it with the current one on held-out labeled runs, the ones people chose to label. Only if it holds up do they pin it on the workflow.

A candidate in Manchester who will not relocate and has no Australian work rights should score near 0 overall, because location and work rights are hard gates. So should a founding CTO in Sydney whose salary expectation sits well above the band. A two-year individual contributor who has not led a team should come back seniority below. None of those need a new model. They need the claims and the eval instructions to say what a hard miss is.

When Juniper later raises the salary band, Northlight updates the claim. The same evals, and any custom evaluator trained on how seniority is judged, keep working. Retraining is for when the nuance of the judgment changes: what "too senior" means, not what the band is this quarter.

A first-line support desk can run the same pattern with less ceremony. Install the Urgency starter, put it on a triage workflow, Try it on "Login broken for everyone", and you already have a verdict the answer node can route on. Labeling and a custom evaluator come once the queue has a shape you care about tightening.

Both desks show up on one Evals list. Northlight's must-have skills, seniority, and overall sit next to Kestrel's routing, sentiment, and urgency, plus Needs human review. Each row names the workflow that uses the eval and offers Try.

Click to enlargeTap to enlarge

Notice you do not need a separate project per team. Recruiting and support questions live together, and Try is on the row so you can judge one input without running the whole workflow.

Getting started

You can stop after step 3. A starter eval on a workflow is already applying judgment to live runs. Labeling and a custom evaluator are how that judgment becomes yours.

FAQs

Related pages