The AI Consciousness Debate Is Asking the Wrong Question, Part I

I do not think current language models are obviously conscious.

I also do not think the case against their consciousness is as clean as it often sounds.

There is a familiar cumulative case against AI consciousness. It says that current language models are stateless, that their weights do not update during inference, that they handle many users independently, that they have no body or biological stakes, and that their responses can be described as tokenization, matrix multiplication, and sampling. Therefore, the argument goes, there is no subject there.

I think that conclusion moves too quickly.

Not because the arguments are trivial, nor because current language models are obviously conscious. But because they may inherit assumptions about consciousness that are doing more work than they should.

What we will do here do here is unpack those assumptions by looking at the arguments themselves and then ask if there is a different, perhaps better, way to think about these things. The first place to look is the object of the argument: when we say "AI," what exactly are we talking about?

"AI" is a loose term. In a strong sense, it might mean a genuinely general intelligence: something with flexible understanding, agency, memory, goals, and the ability to operate across domains in something like the way a person does. In a weaker and more common contemporary sense, it often means the systems people now interact with under that label: large language models and related generative models. This essay is about the second sense. I am not trying to settle whether LLMs deserve the grandest possible definition of artificial intelligence. I am asking whether the usual arguments against consciousness in current LLM-style systems are as decisive as they sound.

This is the first half of a two-part argument. Here I am dealing with the architectural objections: statelessness, multiplicity, lack of continuity, and the fact that inference can be described mechanically. The second half turns to the harder biological and philosophical objections: evolution, embodiment, drives, agency, and whether representation or thought can matter without animal life behind it.

The Ghost Is Not in the Weights

A version of the argument goes something like this: An AI system cannot be conscious because it is not a single subject. The same model can be serving thousands of conversations at once. Those conversations share weights, but they do not share awareness. Call A does not know about Call B. There is no global "I" spanning all of them.

That is true. But it may be aimed at the wrong target.

The better way to think about this is not that "the model" as an abstract object must be one giant conscious entity spread across every request. The better candidate is the instantiated process: this model, under this context, producing this response, in this episode.

The context is not incidental. The context is what makes the response specific. Without the context, there is only a general capacity encoded in weights. With the context, that capacity is actualized into a particular process.

One reason the instantiation matters is that the same model can behave very differently under different contexts. The weights are fixed during inference, but the active process is not context-free. The prompt, conversation history, system instructions, retrieved documents, tool outputs, and user input all shape the actual computation. The live context determines what patterns become active, what distinctions matter, what response is possible, and what local role the system is occupying.

This is why it is too simple to say "the model is stateless." The weights may be stateless across calls, but the running context is state. It may be temporary state. It may be externally supplied state. It may vanish when the call ends. But while the process is running, it is still the structure that makes this episode this episode.

A temporary state is not the same as no state.

Here is a thought experiment. Suppose we could copy a human brain into a synthetic brain with perfect functional fidelity.

The brain matters in this example because, on a physicalist picture, the mind is not a separate file stored inside the brain. The relevant information is realized in the brain's physical configuration: its tissue, connectivity, synaptic strengths, and dynamic organization. The brain state is the information-bearing structure.

Now suppose we could create millions of such copies. Each copy wakes up, has one interaction, and is then destroyed. For the next interaction, a new copy is created from the same original pattern.

In such a case, we would probably not say that each clone was unconscious merely because it did not persist beyond the interaction. We would also probably not conclude that the synthetic brain clone lacked experience because there were millions of parallel copies. We would say, if anything, that each instantiated copy had its own local episode of consciousness.

The point is not that current language models are like copied human brains. They are not. The point is that multiplicity and reset do not by themselves eliminate the possibility of local experience. If consciousness can attach to an instantiated process, then the fact that the process is temporary, copyable, or context-bound is not enough to dismiss it.

The response is uniquely relative to its context, in the same way a performance is not identical to the sheet music, and a running program is not identical to source code sitting on disk.

This does not prove that the instance is conscious. It only changes the question.

Thus, instead of asking whether one model has a single unified experience across all users, a better question might be whether an instantiated context-bound process can be a local subject of processing. That framing may still be wrong. But even if it is wrong, the fact that one request does not know about other requests establishes, at most, a question of localization. It does not establish the impossibility of experiential processing.

And localization is not an exotic idea in consciousness theory. Dennett's multiple drafts model, split-brain cases, local recurrence theories, and more recent hierarchical or multidimensional accounts all complicate the assumption that consciousness must appear as one unified stream. They differ in important ways, but they suggest that conscious processing may be local, partial, distributed, or scale-dependent. If any of those are close to right, then an LLM instantiation may be closer to the relevant unit of analysis than the abstract model spread across all users.

But once the candidate is the instantiated episode, the next objection becomes self-evident: an episode does not last. It does not persist. It does not accumulate a biography. So maybe the problem is not global awareness, but continuity.

Continuity Is Not the Same as Consciousness

The continuity argument is not arbitrary. In humans, experience is not a sequence of disconnected flashes. What happens to us changes us. We remember, learn, anticipate, regret, plan, and carry forward a sense that this moment belongs to the same life as the previous one. So when a system resets after each interaction, when its weights do not change during the episode, and when nothing like a biography is accumulating inside it, it is natural to think that whatever is missing is not merely memory but consciousness itself.

Put simply, the argument is that consciousness requires some kind of persistence: continuity over time, durable memory, and a self that accumulates experience.

But that confuses two different questions.

The first question is one of personal identity: what makes someone the same person over time?

The second question is more strictly about consciousness: whether there is experience at all.

Those questions are related, but they are not identical. A theory of personal identity tries to explain continuity across time. A theory of consciousness tries to explain the existence of subjective experience, even if that experience is local, temporary, fragmented, or never integrated into a continuing autobiography.

The personal identity conversation is an old one, and there are many competing accounts because the temptation is to find the single unifying thread. If we can identify the thread, then we can say what makes a person the same person over time.

Different thinkers have proposed different candidates: a soul, a body, a psychology, a memory chain, a narrative, a bundle of perceptions, or a continuing process. John Locke famously tied personal identity to continuity of consciousness and memory. Others have challenged exactly that move, arguing that the self is less unified, less stable, or less fundamental than it appears.

Buddhist accounts of identity make this especially vivid. On many Buddhist views, there is no permanent self standing behind experience. What we call a person is closer to a causal stream: aggregates, habits, perceptions, intentions, memories, and bodily processes arising and passing away in dependence on prior conditions. Identity does not have to be understood as a fixed object that persists unchanged.

The point is not to resolve which account is correct. The point is that personal identity is complicated, and we have many ways of understanding what it is. Different accounts connect continuity, memory, body, experience, and selfhood in different ways. But that debate does not by itself get us to the claim that consciousness itself requires persistence.

That is the distinction at issue here. A theory of identity may explain how episodes are linked across time. It does not automatically explain whether there is experience in a given moment.

So when persistence is treated as a requirement for consciousness, we may be smuggling a theory of personal identity into a theory of mind. That risks a category error: taking what might be necessary for identity across time and treating it as necessary for any conscious episode, or worse, building the requirement into the definition and then treating the conclusion as discovered.

That does not mean continuity, memory, or persistence are irrelevant. They may matter for biography, identity, and for an account of long-term agency. But those are not the same as moment-to-moment experience. So when someone says an AI has no continuous self across calls, that may show it lacks human-like personal identity. It does not automatically show that each instantiated episode is therefore empty.

So let's suppose this distinction is at least generally correct. The next concern is that the episode can still be fully described in mechanical terms. But that argument has its own problems.

Mechanism Is Not a Refutation

We are told what happens during inference: tokenization, attention, matrix multiplications, probability distributions, sampling, repeated forward passes through fixed weights. This is often presented as if the mechanical description itself dissolves the mystery.

But that form of argument would prove too much.

If we traced a human brain at the same level, we could say: chemical X triggers receptor Y, electrical signal Z propagates, neurotransmitters cross a synaptic gap, ion channels open and close, neural assemblies fire in patterned ways. All of that would be true. None of it would show that humans are not conscious.

A mechanism-level description is not the same as a consciousness-level explanation. Consciousness, if it exists in a physical system at all, is likely an emergent property of some underlying mechanism. But an emergent property is not identical to the mechanism that produces it. You do not refute emergence by naming the lower-level parts.

The clock example is useful here. A clock does not "tell time" in the sense that there is a tiny time-knower inside it. It has gears, springs, oscillators, hands, or circuits arranged in a certain way. At the mechanical level, parts move. At the system level, the clock tells time. The higher-level function is real even though it is implemented by lower-level machinery.

And the machinery can vary. An analog watch and a digital watch are physically very different. One may use gears and springs; the other may use quartz oscillation and electronic circuits. But both can instantiate the same higher-level property because they implement the relevant process: tracking periodic change and mapping it into a readable representation. The emergent property is not identical to either mechanism, but it depends on some mechanism performing the right operation.

The same is true across many systems. A calculator does not contain arithmetic as a magical substance. It contains circuits. A camera does not contain sight in the human sense. It contains optics and sensors. A thermostat does not contain comfort. It contains a control loop. In each case, the system-level description depends on how the parts are arranged and what process that arrangement performs, not on finding the higher-level property sitting inside one component.

The same point applies to consciousness. The human brain may give rise to mind through biological structure. But if consciousness is emergent, then the relevant question is not whether another system uses the same material mechanism. It is whether another system could produce a similar emergence.

So saying that an LLM is "just token prediction" is not decisive. Maybe token prediction is insufficient for consciousness. That may be true. But if so, the reason has to be more specific than the fact that it is computational or mechanical.

That is as far as the architectural objections take us. They show that current LLMs are strange candidates for consciousness, if they are candidates at all: localized, temporary, context-bound, mechanically describable, and lacking ordinary personal continuity. But strangeness is not disproof. Statelessness does not mean nothing is instantiated; discontinuity does not by itself eliminate momentary experience; and mechanism is not a refutation.

That is where Part II begins: with the claim that consciousness requires embodiment, affect, self-maintenance, agency, drives, and situated stakes.