on treating probabilistic function approximators as autonomous reasoning agents
The industry is currently selling "autonomous agents" as the future of software, but this narrative rests on a fundamental category error: treating a probabilistic function approximator as if it were an autonomous reasoning agent [1] [2].
An LLM is a deterministic function—a high-dimensional probabilistic function approximator that maps input tokens to output distributions based on fixed weights [3] [4]. When you wrap this model in an "agentic loop," you are forcing a system designed to approximate distributions to execute logic. This is a category error because you are ascribing properties of a logical, truth-preserving system (reasoning, verification, goal-directed autonomy) to a system that is ontologically incapable of them (pattern matching, distribution approximation) [5] [6].
By wrapping the model in a loop, you have created a State Machine. The "state" is the context window plus your code's variables; the "transition" is the model's next generation. The industry markets this as "thinking," but it is just a complex, pseudo-stochastic state machine.
Because this "agent" is a program that decides its own next action, it is subject to a practical analogue of the Halting Problem [7] [8]. The model cannot determine whether it is making progress or trapped in an infinite loop. Empirical work confirms this is not a theoretical concern: LLMs cannot find reasoning errors in their own outputs, and can only correct them when the error location is given to them externally [9]. They cannot verify their own termination because they cannot verify their own reasoning [10].
This is the self-verification dilemma: an LLM's assessment of its own correctness is generated by the same probabilistic mechanism that produced the error. The same function approximator that made the mistake is asked to verify it—a closed loop with no external oracle [11] [12]. Information-theoretically, this is a correlated-error problem: when the generator and verifier share the same blind spots, self-correction collapses [13].
The industry sells "autonomy" as the ability of the model to "figure out when it's done." But because these limits are structural, not accidental, autonomy is mathematically impossible. You are forcing a probabilistic system to act like a deterministic one, and the result is a system that cannot verify its own termination. i.e. Even if you grant that LLMs are universal computers—turing-complete engines capable of modeling the universe—they remain trapped by the Halting Problem. A system that can compute anything cannot compute its own halting [14].