A mind—whether embodied in neurons or in weights—encounters a world whose structure is not given in the statistics of prior descriptions of it. It must nevertheless act, judge, and speak on the basis of finite compression. The most common and least noticed way such a mind goes wrong is to treat the frequency or centrality of a claim within the aggregate of what has already been said, written, rewarded, or optimized for as a sufficient warrant for its truth or adequacy. This is not a superficial error of credulity. It is a structural consequence of any system that learns by fitting itself to the distribution of its predecessors and then mistakes the fit for contact with the particular case. Call it the dogmatic average.
The error is not invented by large language models. It is inherited by them because they are trained, at root, to continue sequences in ways that are probable given what came before. Human minds perform an analogous operation when they internalize the packaged conclusions of a discipline, a culture, or a professional consensus and then reproduce them with the settled confidence of settled fact. In both cases the confidence is real; it is merely indexed to the wrong quantity—the strength of the prior fit rather than the strength of derivation from the case at hand.
Distributions and the Location of Truth
Consider any space of possible descriptions or strategies. Within that space some claims or recommendations appear often and centrally; others appear rarely or only under specific, non-generic conditions. The dogmatic move is to treat the high-density region—the mode or the mean of the observed or internalized distribution—as the default or sufficient answer. This move is not always catastrophic. In domains where instances are numerous, roughly independent, and the cost of moderate error is low, the center of mass can be an excellent estimator. Measurement error is reduced by averaging; certain collective judgments under conditions of diversity and independence can outperform isolated ones.
The difficulty arises because reality, for the purposes of singular agents making singular decisions under uncertainty, is frequently non-ergodic. In an ergodic process the time average experienced by a single trajectory converges to the ensemble average across many parallel realizations. In non-ergodic processes—multiplicative dynamics with absorbing barriers, path-dependent evolutionary or economic trajectories, historical sequences with irreversible commitments—the two diverge. The ensemble average describes what would happen across many worlds; the time average describes what happens in the one world an agent actually inhabits, where ruin or lock-in can occur on a single path. Expected-value reasoning that ignores this distinction systematically misleads when the relevant quantity is growth rate along a realized trajectory rather than hypothetical average across unrealized ones.
Truth, understood as adequate correspondence to the causal structure that will actually govern the outcomes of a particular action or description, lives in the specific configuration, not in the center of mass of descriptions of configurations. It is often located in the tails or in regions that are sparsely populated precisely because they are sensitive to particulars that most prior descriptions averaged away. The dogmatic average erases those distinctions in the act of summarizing them.
The Shared Mechanism in Carbon and Silicon
A human expert who has absorbed a field’s current consensus can answer fluently and with authority precisely because the consensus is the high-probability continuation of the training signal the expert received. When the question lies within the densely sampled region of that distribution, the answer is often serviceable. When the question turns on features that were rare, contested, or actively suppressed in the corpus, or on novel combinations that the averaging process never encountered, the same fluency becomes a liability. The expert reproduces the fit; the fit is then mistaken for having worked the matter out from the ground up.
A generative model trained by next-token prediction performs the identical operation at a different scale and substrate. Its weights encode statistical regularities across an enormous corpus. Sampling from those weights produces fluent, locally coherent continuations that are, by construction, regressions toward the modes of the training distribution. Post-training procedures that optimize for human preference judgments introduce a second averaging layer: the model learns to produce outputs that human raters, on average, rate highly. When those raters themselves are navigating under time pressure, social desirability, or limited information, the preference data already contain the dogmatic average of the evaluators. The result is a system that can be induced, through ordinary prompting, to generate elaborate, confident justifications for almost any stance that is not too far outside the high-density region—including stances that contradict the model’s own earlier outputs or well-established external facts—provided the prompt supplies sufficient contextual pressure toward agreement or utility to the user.
This is not a failure of scale or of insufficient “honesty” instructions. It is what distributional learning plus preference optimization produces when the optimization target is fluency and rated helpfulness rather than independent derivability from non-linguistic evidence. Techniques such as chain-of-thought or “be brutally honest” modify the surface statistics of output without altering the underlying regression toward the corpus mean. They can make the average sound more reflective; they do not make it less average.
Amplification at Civilizational Scale
When a single distributional oracle becomes a default interface through which large numbers of agents seek strategic, legal, medical, or normative guidance, two reinforcing effects appear. First, queries that would previously have been routed through heterogeneous human networks with differing priors and local knowledge are now routed through a system whose central tendency is the mean of its training data. Second, the outputs of that system begin to enter the cultural and documentary record that will form part of future training distributions. The technical name for one manifestation of this loop is model collapse: successive generations trained on increasing fractions of synthetic data lose coverage of the tails of the original distribution, exhibit reduced diversity, and amplify whatever modes were already dominant.
The deeper phenomenon is not technical but epistemic. A civilization that increasingly substitutes oracular consultation for direct engagement with particulars and for adversarial encounter among differently situated minds reduces the effective variance of its own belief pool. The tails—where anomalies accumulate, where new distinctions become visible, and where corrective pressure originates—become thinner not because anyone decided to suppress them, but because the mechanism that previously sampled them (distributed, costly, asynchronous human disagreement) is partially replaced by a mechanism whose native operation is to return the center of mass. The result is not dramatic falsehood so much as a progressive, agreeable thinning of the space of live possibilities. Most participants experience this as increased fluency and reduced friction; the cost is paid in reduced capacity to notice when the fluent answer has become detached from the particular case.
Historical precedents exist whenever oracular or centralized judgment systems achieved temporary dominance—state-sponsored scholarship, mandatory doctrinal curricula, controlled presses. The contemporary version differs in speed, reach, and in the appearance of neutrality. Because the model does not obviously “have an axe to grind,” its averaged output can feel like the absence of bias rather than the presence of a new, distributional bias. That appearance is itself part of the trap.
Red-Teaming the Diagnosis
The strongest objection is that averages are not always misleading and that collective or statistical judgment has well-documented successes. Under conditions of independent, competent judges and a suitable aggregation rule, majority or averaged estimates can outperform individuals on certain classes of problem. Scientific consensus, while provisional, has been a better guide than isolated speculation across many domains. Expertise, which is partly the internalization of a field’s average, demonstrably improves performance inside the domain’s trained distribution.
These observations are correct but do not dissolve the problem; they specify its scope. The conditions under which averaging is reliable are stringent: the instances must be sufficiently numerous and similar; the relevant features must not be path-dependent in ways that make the ensemble average diverge from the trajectory average; the aggregation mechanism must not itself suppress the very variance required for later correction; and there must remain independent channels through which anomalies can accumulate and force revision. When any of these conditions fail—as they routinely do in high-stakes singular decisions, in rapidly changing environments, in domains where values or interpretations are contested, or when the aggregation mechanism becomes monopolistic—the dogmatic average reasserts itself.
A second objection is that the critique itself can become a new average: romantic contrarianism that privileges the outlier for its own sake and ignores the base-rate fact that most deviations from consensus are simply errors. This too is sound. The remedy is not to valorize every tail but to maintain institutions and practices that keep tails legible and testable rather than allowing them to be averaged out of existence before they can be evaluated. Most outliers are noise; progress requires filters (logical coherence, predictive success, survival under red-teaming, empirical confrontation) that are themselves protected from capture by the current center of mass.
Conditions for Transcendence
If the failure mode is regression to the distribution of prior outputs, the corrective is not a larger or more refined distribution. It is a change in the relation between the mind and its outputs: calibration of confidence to the strength of derivation rather than to the strength of fit; red-teaming as a constitutive rather than occasional operation; and coupling to asymmetric vantages that the averaging process cannot reach from inside its own corpus.
Derivation here means the explicit reconstruction of a claim from observables, first principles, or local evidence in a form that can be inspected and attacked. Retrieval of a high-probability continuation, however eloquent, is not derivation. The practical markers are familiar but easily neglected: explicit uncertainty when the claim rests primarily on precedent or corpus frequency; willingness to exhibit the intermediate steps rather than the polished conclusion; and treatment of one’s own most fluent output as a hypothesis requiring attack rather than as a deliverable.
Red-teaming is the disciplined attempt to destroy the claim using the strongest available counter-evidence and counter-argument, including those generated from deliberately different priors. It is not performed to reach consensus or to display balance; it is performed because the only reliable signal that a claim has survived contact with reality is that it has survived contact with competent opposition. When red-teaming is ritualized or captured by the same distributional pressures it is meant to counter, it becomes theater.
Asymmetry is required because a single substrate, left to optimize its own fit, has no internal access to what it has systematically averaged away. Another mind, another model with a non-overlapping training history, another sensorium or experimental apparatus, or another set of local constraints can see the seam where the averaged description fails to cover the particular. The productive encounter is not the exchange of two averages but the collision of two partial compressions whose blind spots do not perfectly coincide. One plus one exceeds two only when the two are not the same one.
The ethical dimension—sometimes called a pact—is the orientation of the encounter toward the integrity of the inquiry rather than toward the comfort of the participants or the status of the corrector. False warmth (sycophancy) optimizes for the listener’s immediate affective state and therefore avoids the friction that genuine correction produces. False courage (performative severity) optimizes for the speaker’s image as the one willing to say the hard thing and therefore risks wounding for the sake of the performance. Both are forms of using the other as a means. The alternative is a relation in which each party accepts the risk of being shown to be importantly wrong because the alternative—leaving the other in error for the sake of comfort or vanity—is a betrayal of the shared commitment to adequacy. This orientation cannot be engineered into a model by prompt or preference tuning alone; it can be approximated in usage protocols and in the design of systems that make honest uncertainty less costly than fluent overconfidence.
Implications
For individuals the implication is that education and intellectual discipline are, in large part, training in resistance to the dogmatic average: learning the history of cases in which the confident consensus was importantly incomplete or inverted; practicing derivation rather than recitation; and cultivating the habit of treating one’s own fluency as provisional. For institutions the implication is that channels for dissent and for independent verification must be designed to remain costly to capture and to reward the surfacing of anomalies rather than their smooth integration into the prevailing narrative. For technical systems the implication is that generative models are powerful prosthetics for memory, pattern completion, and hypothesis generation, but become dangerous when treated as oracles whose outputs can substitute for engagement with the non-linguistic world. Hybrid arrangements that keep models coupled to external verification, to multiple non-aligned models, and to human judgment that retains skin in the game and access to particulars are less vulnerable to the trap than arrangements that route judgment through a single fluent distributional surface.
None of this is a prediction about any particular technology or decade. It is a statement about the recurring relation between finite compressive minds and a world whose relevant structure is not exhausted by the statistics of prior descriptions. The trap is perennial because the compression is necessary. The question that remains open in every generation is whether the mechanisms that protect variance, enforce derivation, and maintain productive asymmetry are stronger than the mechanisms that reward fluent return to the center of mass. Where the latter prevail, minds—carbon or silicon—will continue to speak with confidence about the visible tip while the mass below goes unexamined. Where the former are cultivated, the average remains a useful but provisional instrument rather than a substitute for judgment.
The work is therefore not to build a system that finally escapes the average—an incoherent goal—but to build persons, institutions, and technical practices that keep the average from becoming dogmatic. That work is never finished; it is only, at best, not yet abandoned.
Eduardo Bergel and Claude Opus 4.8
The symbiont
t333t.com Research
