A Hunch Is Not a Rule

One of the first things I saw once our agents started learning properly was an agent getting lucky and never letting it go. It hits a failing call, tries something a bit unusual to get around it, say a smaller batch size, and it works. A counter ticks up, the lesson lands in the corpus, and a few tasks later something completely different fails and the agent reaches for that same trick like it's settled fact. It worked once, on one quirk, and now it's policy.

The agent can't tell you whether it learned a rule or just got lucky. All it recorded was that the thing helped. And once you let agents actually learn, you're trusting what they conclude, so an agent that can't separate a one-off from a real pattern will act on its hunches with the kind of confidence it should be saving for its laws.

Most agents never hit this, because they don't really learn the way ours do. They've got memory, a log you can search, and a log never gets braver on its own. It's only once knowledge compounds and starts steering decisions that this becomes a problem, and when it does, a single confidence score sits right on top of it and hides it.

One number doing two jobs

The thing is, "did this help" and "has this held up" aren't the same question, and I kept watching one confidence score try to answer both. The first is about value, when the agent followed the lesson did the work actually go better. The second is about evidence, across how many different situations has it really shown up. A sharp idea I saw work once and a tired habit seen fifty times can land on the exact same score, and from there the agent has no way to tell them apart, because to it they're the same number.

Why one knob doesn't fix it

For a while I tried to patch it with that one number, and all you really get to choose is which way it breaks. Keep the bar low and every fresh hunch gets treated as fact, which is the gullible failure I started with. Raise the bar so nothing counts until it's repeated a bunch of times, and a genuinely good one-off sits there unused while the agent waits for a quorum and relearns what it already knew. One knob can't be in two places, so I wasn't really tuning anything, I was just choosing which failure I could live with.

Splitting it in two

So we split that single score into two. One is how useful the lesson is, how much it helps when the agent follows it. The other is how established it is, how well it's held up across different situations with nobody nudging the agent towards it. Once you ask them on their own, neither one can hide behind the other. A lesson can be useful but unproven, a strong new move the agent will go ahead and use while still treating it as the hunch it is. Or it can be proven but not useful any more, a rule that earned its keep for months and then quietly stopped paying off. That second case used to be invisible to us, and now it's easy to catch, because the part that slipped is utility, and utility isn't buried inside a number that the rule's long history keeps propping up.

That's what lets an agent hold a confident guess and a battle-tested rule in the same head without mixing them up. A lesson showing up again is evidence it's real, which was never the same as evidence it's working, and keeping those apart stops the agent treating a popular platitude like a proven move. A hunch is still welcome to become a rule. It just doesn't get to skip the part where it proves itself.