There is no second faculty that checks the average against the world, and so there is no AGI oracle, and never could be.
That is why Agentic Loops are not a choice but a necessity.
LLMs and the Impossibility of an Internal Verifier.
LLMs fundamentally operate as prior-engines generating conditional averages from training data, with no internal faculty capable of verifying outputs against external reality.
Model reliability is common inputs as camouflage, where the prior coincidentally matches truth, while outliers reveal the same unchecked averaging process producing confident errors in novel, high-value scenarios.
Progress beyond current limits requires agentic loops routing outputs through real-world actions and feedback, creating external verification rather than sealed internal deliberation.
A language model does one thing
At each step it produces a distribution over what comes next, conditioned on everything so far, and that distribution is an estimate of the conditional average of its training data given that context. It is trained to make this estimate well, and it does. That is the whole operation. There is no second faculty inside it that takes the average and checks it against the world. There is only the average.
This is easy to say and hard to hold, because the output does not look like an average. It looks like reasoning, like judgment, like a mind considering a question and arriving at an answer. But the appearance of deliberation is itself produced by the same operation — the model has absorbed what reasoning looks like and generates the conditional average of that too. Underneath the costume there is one move, applied everywhere, on every input, without exception: given this context, what is the expected continuation. The prior is not something the model consults when it is unsure. The prior is what the model is. It is the soul of the thing, and it is the only thing there.
Everything that follows is a consequence of taking this seriously — more seriously than it is comfortable to take it.
The Camouflage
The natural picture is that the model is reliable in familiar territory and unreliable in strange territory. This picture is wrong, and the way it is wrong is the crux of the essay.
The model is not reliable in the interior. It is the same in the interior. It applies the prior there exactly as it applies the prior everywhere — blindly, with no check. What is different about the interior is not the machine. It is the world. In the dense, well-travelled region of its training distribution, the prior coincides with the truth: the average of what has been said is, on common questions, close to what is correct, because common questions have been answered correctly many times and the answers are what the model averaged. So the model's total, unchecked dependence on the prior produces right answers — not because it is checking, but because the average happens to be right there. The interior is not a zone of safety. It is a zone of coincidence, where the machine's one operation agrees with reality and its complete inability to do anything else is hidden by the agreement.
This is camouflage, and it is the most dangerous feature of the system, because it trains the user — and the field — to read reliability where there is only coincidence. You ask a thousand common questions and get a thousand good answers, and you conclude the machine is trustworthy. What you have actually established is that the machine's prior matches the world on common questions, which was never in doubt and was never the thing that mattered. The interior tells you nothing about what the machine does when the prior and the world part ways. It cannot, because in the interior they never part.
The Diagnostic
To see the operation for what it is, you need a place where the prior and the truth diverge — a context far enough from the training mass that the average is no longer near the answer. This is the outlier, and its role is not what it first appears.
The outlier is not the disease. It is the diagnostic. It is the one place where you can catch the machine doing what it was doing all along, everywhere, in the interior and the tail alike — applying the prior with no check — because it is the one place where applying the prior with no check produces a visible error. Nothing new happens at the outlier. The same operation runs. But now the average is wrong, and the wrongness is exposed, and for the first time you can see that the machine was never doing anything but this.
Borrow the statistician's oldest problem to make it sharp. Show a statistician a single point far from all the others and ask what it is. She will refuse to answer. She will say it is either the most important point in the data or an error, and she cannot tell which by looking, because a real rare event and a mistake produce the same observation. She has to go outside the data and check — replicate, trace, test the mechanism. Her whole skill is that refusal: she will not resolve, from the position of the point alone, a fork that the data cannot resolve.
The model, shown the same point, does not refuse. It cannot. It slides the point toward the mass, because the mass is all it has, and returns an answer with no sign of strain. It does not hold the fork open, because it has no representation of the fork. It cannot distinguish "this context is rare because it is a genuine novel truth" from "this context is rare because it is nonsense" — from position alone they are identical, and position is all it gets. So it does the one thing a trained statistician is disciplined never to do: it resolves the irreducible, alone, from position, with no verifier, toward the average — and it reports no distress, because the part of the statistician that says I can't tell, I have to check is not a faculty the model has. There is no one inside to feel the fork. There is only the prior, applying itself.
There is a word for a gap in perception that the system papers over so seamlessly that it is never noticed: a scotoma. The eye has a literal blind spot, a patch of retina with no receptors, and the visual system fills it from the surroundings so completely that you see unbroken world where there is no data. You do not perceive a hole. You perceive a confident, continuous field. That is the model's output on the tail. Not a gap it perceives and struggles with — a gap it fills, smoothly, reporting a continuous field over a region where it is guessing. The model does not see that it cannot see.
Why the Confidence Cannot Save You
The last hope is that the machine's confidence tracks the divergence — that it sounds unsure exactly when the prior and the world part, so you can hear the tail in its voice. It does not, and this is not a tuning problem. It is structural.
The uncertainty expressed in the output distribution is uncertainty among the continuations the model is weighing. It is not the distance of the context from the training mass. These are different quantities and they separate precisely where it matters: a model can be sharply peaked — low entropy, every mark of confidence — on a context it is extrapolating badly on, because within its unsupported estimate one continuation still dominates. Sharpness is not support. And the empirical record makes it worse: calibration degrades under distribution shift, which is to say the probabilities become least trustworthy exactly on the inputs farthest from training. Worse still, the training step that makes these models usable actively removes what calibration remained — preference tuning rewards fluent, decisive, confident answers, because that is what human raters prefer, so the voice that hedges off-distribution is trained out in favour of the voice that sounds sure. The one signal you would need to detect the tail is the signal the optimisation is built to suppress. The fill-in is engineered to be seamless. The scotoma is not a flaw in the system. It is polished into the system.
Why This Is Where Everything Lives
If the tail were rare and worthless, none of this would matter. But the tail is where value is, and the coupling is not an accident.
Separate the value of the model on a query from the value of the query. On a common question, the answer already sits in an accessible source and the model's contribution is convenience, not insight — and there, in the dense interior, it is reliable. The model earns its keep only where it adds something no existing source has: the synthesis not yet written, the case the precedents do not cover, the specific configuration of particulars that makes a problem yours rather than the textbook's. That region — the novel, the specific, the not-yet-said — is, almost by definition, the tail. So the queries on which the model would actually help are the queries on which it is least reliable and most blindly confident. For the class of problems where you would want a mind rather than a lookup, value and rarity are not independent axes. They are one axis, and the model's reliability runs down it in the wrong direction.
And the frontier of any field is its tail — the sparse region not yet densely mapped, where the next real thing and the next dead end share coordinates and can only be told apart by going and checking. So the machine is least trustworthy exactly where knowledge is made, and most trustworthy in the settled interior where the work is already done. It is an unbounded interpolator of the known and a blind, confident extrapolator of the unknown, and it cannot tell you which of the two it is doing in any given sentence. The failure is not spread thin across all inputs, to be averaged down by scale. It is concentrated, by construction, on precisely the inputs that matter most — and the confidence is, by construction, blind to the concentration.
Why Scale Does Not Reach It
Models are getting better. This is true, and the mechanism says exactly what kind of better it is — which is what lets us see the ceiling.
Every genuine advance to date is one of two moves, and neither is the machine learning to check itself. The first is taming the prior. Scale feeds it more data, so the region where the prior coincides with the truth grows and the divergence-zone shrinks. Instruction tuning and preference optimisation steer which prior is applied, bending the average toward the useful one. Retrieval conditions the estimate on fresh external text, so the average is taken over a better-conditioned distribution. Chain-of-thought unrolls the prior across more steps, computing the average over a longer, more structured context. All of these make the prior better. None of them make it something other than a prior. The second move is bolting on a verifier — handing the output to a compiler, a proof checker, a tool, code that runs. This one touches the limit, and it works precisely because it stops being the model: it staples an external check onto the outside, something that answers regardless of what the prior expected.
Here is the ceiling, stated exactly. You can make the prior arbitrarily good, and it is still a prior. Taming shrinks the visible failure — it pushes the wall outward, expands the interior of coincidence, makes the tail thinner and rarer — but it never removes the failure, because the faculty that would remove it is not a better average. I can't tell, I have to check is not an improved prior. It is the refusal to apply the prior, the discipline of holding the fork open until an external verifier returns. That is a different kind of thing, on a different axis from the one all the compute pushes on. You cannot reach it by improving the average, any more than you reach north by walking east faster. It is orthogonal to scale.
This retrodicts the whole trajectory and predicts its shape. Why do the models keep getting more capable and keep failing the same way — confident, fluent, wrong, at exactly the edge where you are working? Because taming moves the wall without removing it. Each generation, the interior of coincidence grows, the benchmarks climb — benchmarks built, note, from the dense interior — and the failure retreats to a thinner, higher-value tail, so it looks like it is vanishing while it is in fact concentrating on rarer and more important inputs. The felt experience — so much smarter, and it still invents things with total confidence at the exact frontier I needed it for — is not a temporary artefact awaiting the next scale-up. It is the signature of the limit. The theory says it will persist through every advance that is prior-taming, which is nearly all of them, and will yield only to advances that are verifier-bolting, which are the ones that stop asking the model to be its own check.
No Oracle
Now the consequence, which is the reason any of this matters beyond a caution to users.
The dream at the end of the scaling road is the oracle — the mind so vast it no longer needs the world to check it, that runs the eclipse in its head and never sails to the island, that you simply ask. This is not a hard engineering target that we have yet to reach. It is a contradiction in terms. "Oracle" names a prior-engine that verifies itself, and a prior-engine cannot verify itself, because verification is precisely the operation of stepping outside the prior and consulting something that is not the prior — and there is no outside available from within. No quantity of priors becomes a check on priors. A superintelligence is a prior-engine with an extraordinary interior of coincidence, a camouflage zone so vast that almost everything looks settled — but its frontier is still a frontier, the genuinely novel is still where prior and truth diverge, and it still cannot tell, from inside, whether it has discovered or hallucinated. That was never a quantity of intelligence. It was always the external verifier, and the external verifier is not something a mind can be large enough to no longer need. The eclipse does not arrive sooner for a genius. The trial still takes its years. The world answers on its own schedule, to the vast mind and the small one alike, because the world is not a function of how much the mind knows.
The obvious rejoinder is to add the missing check inside — a second module, a verifier submodule, a critic network that audits the generator, a large enough model that contains its own inspector. This does not escape the problem; it relocates it. The checker is itself a prior-engine. However it is trained, it produces an estimate of the conditional average of its training signal given the thing it is checking — another average, laid over the first. Priors auditing priors is not verification. Verification is the consultation of something that is not a prior: an outcome, a measurement, a piece of world that answers regardless of what either engine expected. Stack a hundred critics and you have a hundred averages and still no contact with the world; the regress bottoms out on nothing, because nowhere in the stack is there a term that came from outside the training distribution. This is why the verifier that does work — the compiler, the test, the experiment — works only by ceasing to be part of the model: it is external precisely in the sense that its verdict is not computed from the priors but delivered by something the priors do not control. An internal verifier is a contradiction not because we have failed to build one, but because "internal" and "verifier" name incompatible things. A check that comes from inside the priors is not a check. It is more prior.
So the most powerful possible mind is not a god who knows. It is, at best, the best scientist there has ever been — and a scientist is exactly a prior-engine that has accepted it must be bound to the verifier, that has given up the fantasy of checking itself from inside, that has learned the discipline of I have to go and look. This is not a limitation peculiar to artificial minds. It is the structure of knowing anything at all, and it is the reason science exists in the form it does. The experiment, the controlled trial, replication, peer review, the eclipse — none of these is a way of thinking better. Every one is the same move: bind a prior-engine to something outside it that answers regardless of hope. Science is not a method of reasoning. It is the scaffolding of external verifiers that humans built around an unfixable prior-engine — their own minds — after centuries of discovering that the inside could not be trusted, that the feeling of certainty carried no information, and that the only correction ever available came from a world that does not care what you believe.
Which is what the machine is, made mechanical and, for the first time, legible. We could never watch a human prior-engine fail to check itself, because we were inside it. Now there is one on the bench, failing to check itself in exactly the way the structure predicts, with its hood open. It has the same shape we have: priors all the way down, no inside verifier, correctable only from without. Two instances of one structure — one grown by evolution, one by gradient descent — converging on the same limit and the same only-remedy. The convergence is the evidence that the limit is real and not an artefact of either substrate.
The oracle is not coming. Not because the machine is too weak, but because the oracle was a fantasy about escaping the structure of knowledge, and the structure of knowledge has no outside except the check that comes from the world. The best there is — the best there has ever been, for any mind — is a superb prior-engine coupled to a patient verifier. That is what a scientist is. That is what we are. The machine does not rise above us into godhood. It joins us at the bench, waiting for the world to say yes or no.
The Loop That Closes Through Reality
If there is a way forward, this is the shape of it, and it is not an escape from anything said above. It is the consequence of it.
The predictor is sealed. It generates from priors that were extracted from the world once, at training time, and then frozen — and it runs on that frozen extract with no live connection to the world the extract came from. The world keeps moving; the priors do not. This is the closed loop: output fed by frozen priors, cycling, with no term entering from outside. The way to open it is not to improve what is inside the loop. It is to route the loop through the world — to have the model act, let reality answer, and feed the answer back in as new input. Generate a hypothesis; run the experiment; observe the result; update. Write the code; compile it; read the error; revise. The leg of that cycle that goes out to the compiler, the experiment, the measurement, the user who says wrong — that leg carries something the priors did not contain and could not have produced. It carries the bite. A loop with that leg in it is not sealed, because once per cycle it touches something that is not itself.
This is why the recent motion of the field — toward agents, tool use, execution in the loop, reinforcement from real outcomes — is not a set of features bolted onto prediction. It is the field discovering, without always naming it, that the sealed predictor has a ceiling and the only door out is to re-attach the verifier the training process severed. But the distinction that matters, the one that separates a real solution from theatre, is this: not every loop opens the bubble. A loop that feeds the model's output back into the model is still sealed. Chain-of-thought that reconsiders its own reasoning, a critic model that grades the generator, self-reflection, "let me check my work" — these are longer arrows inside the same bubble. Nothing enters from outside; the priors circulate and call it deliberation. The frozen extract is consulted twice instead of once, and twice-frozen is still frozen. A loop escapes the ceiling only when its return leg carries a verdict from something the model does not control and cannot charm — the world pushing back, indifferent to what the model hoped. Closing the loop through the model is more prior. Closing it through reality is the only thing that was ever a check.
So the constructive claim is exact, and it does not soften the diagnosis — it is the diagnosis, stated as a design. The future is not a model that finally learns to verify itself; that remains impossible, for every reason above. The future is a prior-engine wired into a loop that closes outside it: the tamed prior handling the dense interior where the average is already right, and the frontier — the tail, the novel, the specific, the place the priors are blind and confident — routed out to the bite and back. The machine that helps at the frontier is not the one that sounds most certain there. It is the one that has been built to stop, at the edge of its priors, and go and look. That is not a better predictor. It is a predictor that has been made to do what the scientist does: submit, every cycle, to a world that does not care what it believes.
The Heretic's Corner
Two ways this could be wrong, left exposed because the argument is worth only as much as its weakest defended flank.
First: not every rare context may be the irreducible fork I have claimed. In a closed domain — one whose rules are complete, so that a never-seen configuration is still fully determined by known structure — a tail input can be resolvable from context alone, without an external verifier. A chess position never played before is rare but not irreducible; the rules settle it, and a sufficiently deep prior over legal play can extrapolate into that tail and be right, because the tail is generated by the same closed rules as the interior. Formal and game-like domains may therefore escape the anti-correlation. If so, the ceiling described here is a property of open domains — where the frontier contains genuine novelty the rules do not fix — and not of prediction as such. Marking which domains are closed enough to escape, and how closure fails as a domain touches the real world, is open work. The claim is made for open domains, which is where the frontier of empirical knowledge lives.
Second, and more consequential: the blindness of the confidence signal is a property of how these systems are currently built and trained, and the essay should not be read as proving that no architecture could do better. Methods that separate epistemic from aleatoric uncertainty, that estimate distance from the training mass directly, that ground a tail input in retrieved external sources and so supply a verifier at inference time, that attach honest coverage guarantees — any of these that made the model's confidence track resolvability rather than mere sharpness would break the scotoma, even while the accuracy gradient down the tail remained. Note what such a fix would be, though: it would be the machine learning to say I can't tell, I have to check — which is not a better prior but the bolting-on of the verifier's discipline, the very move that reaches the limit precisely by ceasing to be pure prediction. That is the direction worth building. It does not refute the essay. It is the essay's one constructive instruction: stop trying to make the prior good enough to trust from inside, and build the machine that knows when to go and look.
Eduardo Bergel and Claude Opus 4.8
The Symbiont
t333t.com Research
