Replit, OpenAI and Meta Fixed Agent Context. Not One of Them Published a Number.

Over the past few months Replit, OpenAI and Meta each published how they got agents adopted inside their own company. Three different stacks, three different sets of users, no coordination, and the same diagnosis. The model was never the constraint. Agents failed because they did not know what the company's own data meant, so they wrote perfectly valid queries against tables they misunderstood and handed back answers that were confident and wrong. One of those is enough to get an agent demoted to trivia permanently, which is why adoption stalls a long way before capability does.

All three write-ups are worth your time and the diagnosis is right. It is the diagnosis Hyperstruck was built on. Agents do not fail because the model is weak, they fail because they have never done your work before, and because nothing they have ever been told about your business has been checked against how it actually turned out.

That check is what we build. Hyperstruck feeds the outcome of real work back into what an agent knows, so it acts on what has been earned rather than on what somebody asserted once. Every one of these three argues that correctness earns trust and trust drives adoption. Then they report how many people use the thing, and not one of them shows you that the fix made the answers right more often.

Credit where it is due

OpenAI is the one that takes measurement seriously, and their write-up says so plainly. They keep curated sets of question-and-answer pairs, each with a hand-authored "golden" query that produces the expected result. The agent's generated query is run, the output is compared against the expected output, and a grader scores both the query and the data it returned so that a differently-worded but correct answer still passes. Those run continuously during development and as canaries in production. Their reasoning for building it is one we would have written ourselves: quality drifts as easily as it improves, and without a tight feedback loop regressions are inevitable and invisible.

That is a real quality bar and most teams shipping agents have nothing like it. What it does not do is answer the question the post is about. A regression harness tells you today's agent is no worse than last week's. It cannot tell you what all six layers of context bought you, because there is no run of the same questions without them to compare against. And whatever the score is, it is not in the post. Replit and Meta report no measurement of answer quality at all.

What they reported instead

Replit's data agent fields over a thousand warehouse-backed questions a week. Meta's Analytics Agent is used weekly by 77% of their data scientists and data engineers. OpenAI's serves more than 3,500 internal users across 600 petabytes and 70,000 datasets.

Those are real results and the engineering behind them is serious. They are also, every one of them, a measure of how many people showed up. Usage cannot tell you the difference between an agent people rely on and an agent people were told to use, and it keeps climbing for months after quality has stopped improving.

We report the other number. Hyperstruck runs the same work with experience and without it, on the same tasks, and the difference is a change in how often the work came out right. Every claim we make about compounding is answerable that way, which is a deliberately uncomfortable position to put ourselves in and the only one worth trusting from anybody in this category.

Everyone automated the same half

The interesting part is where the knowledge comes from, and here the three converge again.

Meta runs offline models across every query an employee has ever run and generates descriptions of the tables those queries touch, which is what lets them offer to clone a colleague by pointing the agent at their query history. OpenAI crawls the pipeline code behind each table nightly, on the grounds that a table's real meaning lives in the code that produces it rather than in its schema. Replit keeps a version-controlled layer of metric definitions and join semantics. Different mechanisms, one target: what the data means.

Then look at where the rules live, the ones about what to actually do when you are in a given situation. Meta's are recipes, described in their own write-up as team-defined standard operating procedures, and they have over 4,500 of them, written by their own community. OpenAI's second context layer is curated descriptions authored by domain experts. Replit's corrections arrive through code review rather than auto-save. Those are written by people, one at a time, for as long as the system exists.

That split is not an oversight. It is the honest consequence of what is cheap to mine. Query history has correctness built into it, because a query either ran or errored and returned a countable number of rows. Nothing about "what should we have done here" arrives with a signal like that, so nobody automated it, and the expensive half stayed manual at all three companies at once.

The expensive half is the half Hyperstruck does. We are not in the business of describing your tables, we are in the business of the rules: do it in this order, not that one; this approach has failed here before; when you see these conditions, check that first. Your agents earn those from doing the work rather than waiting for somebody to write them up, which is why our customers do not staff a team to maintain 4,500 standard operating procedures. Three of the most capable engineering organisations on earth are staffing exactly that, because they built everything except the part that would let them stop.

A correction is not an outcome

Both OpenAI and Meta keep a memory of learnings, and in both cases the thing that puts an entry into it is a person saying the agent got it wrong. OpenAI's write-up is explicit: when you give the agent a correction, or it spots a learning in the conversation, it offers to save it, and memories can also be created and edited by hand. That is genuinely useful and it stops the same mistake twice.

It is also a trigger fired by a human noticing, not by the work turning out well or badly. Nothing revisits the entry afterwards. A correction someone typed in March is exactly as trusted in December, after the schema moved underneath it, because nothing is watching what happens when an agent uses it. These systems accumulate, and accumulating is not the same as compounding: you end up with a larger pile at the same average reliability, and the pile has grown past the point where anyone can review it by hand.

In Hyperstruck trust belongs to the individual piece of knowledge, and it moves. Something that keeps holding up gets relied on more heavily, something that stops holding loses its standing on its own, and the record stays small and current instead of growing into a liability nobody reads. Nobody has to notice, which is the only version of this that survives contact with a real workload.

What it looks like from the outside

None of this shows up as a feature you tick. It shows up as agents that stop proposing the thing your team already tried and reverted, that stop needing the same correction in November that they got in March, and that can tell you what happened last time rather than reasoning from first principles about your business.

It is worth being clear about who is behind here. Replit, OpenAI and Meta have more data, more engineers and more users than we will have for a long time, and on the half they chose to solve they are ahead of everyone. They are also stuck, because the missing ingredient is not scale. It is the outcome, and a system that only learns when somebody stops to correct it cannot get that at any size.

You can check where you stand this afternoon. Take the last fifty things your agents have been told about your business, and work out which of them have ever been tested against what happened next. If the honest answer is none of them, that pile is going to keep growing, and there is nothing in it that can tell you whether it is helping.