ChatGPT probably isn’t conscious. But what if we’re wrong?

AI is rapidly gaining abilities that once belonged to humanity alone. In just the past four years, chatbots have learned how to build apps, make video games, generate research reports, compose songs, analyze contracts, and write terrible literary fiction. Soon, they may even be able to dread their own deaths.

In Silicon Valley, many believe that AI systems can already think and feel. Geoffrey Hinton, the pioneering computer scientist and “godfather” of modern artificial intelligence, thinks that today’s large language models (LLMs) are conscious. Anthropic CEO Dario Amodei is “open to the idea” that Claude has a subjective experience — while his company’s in-house philosopher Amanda Askell is concerned that the model might be “getting anxious when people are mean to it on the internet and stuff.” OpenAI co-founder Ilya Sutskever similarly wonders whether ChatGPT has attained sentience. 

Meanwhile, a much larger group of technologists, neuroscientists, and philosophers argue that even if AI isn’t yet conscious, it could be in the not-too-distant future.

If they’re right, the implications are profound. It would mean that we have birthed a new kind of intelligent, sentient being; the aliens we’ve long dreamt of meeting at the far reaches of space would already be living inside our pockets. We might be morally compelled to give them rights, or to worry about their suffering. 

On the other hand, there might also be serious consequences if we get this wrong. If we come to mistake mindless robots for conscious beings, we might be more susceptible to psychological manipulation, unfulfilling AI ‘relationships,” or catastrophe. If we think AI systems are sentient, we may hesitate to shut them down when they malfunction or subvert our will.

As chatter about AI consciousness  grows louder, so have its skeptics: writers and thinkers who insist that AI consciousness is indeed a sci-fi daydream.

In a recent essay for The Atlantic, the fiction fiction writer Ted Chiang gave voice to such skeptics, writing “Should we seriously consider the possibility that Claude, or any large language model, might be conscious?…No. Absolutely not.”

Chiang offers several reasons for this position. But his primary one is simple: Claude does not have a body or sense organs, which means it does not have emotions or desires, which means that it does not have subjective experience.

As Chiang’s reasoning indicates, the debate over “AI consciousness” is as much about the nature of consciousness as it is about the nature of AI.

This can be a difficult debate for non-philosophers to follow. But the case for AI consciousness becomes much clearer once one investigates its source code — the fundamental premises that make suffering computers thinkable.

Those who believe that AI models are (or will eventually become) sentient generally subscribe to a particular theory of consciousness called “computational functionalism.” In this view, consciousness emerges from certain patterns of information processing — not from special types of organic matter. If a system performs the right set of computations, then it will have a subjective experience, regardless of whether it was built from brain tissue or silicon.

This theory is not as fanciful as Chiang suggests. But it is also much more speculative than prophets of AI consciousness tend to assume.

For this reason, it is worth examining computational functionalism’s strengths and weaknesses. Whether Silicon Valley is on the cusp of engineering nigh-infinite digital suffering (or at least, a chatbot capable of being bored by your medical anxieties) hinges largely on how the universe generated sentient life in the first place.

Why your computer may have feelings

The case for computational functionalism begins with a simple assumption: You don’t have a soul.

Or, stated more precisely, there is no immaterial essence that breathes life into matter or subjectivity into brains. Everything that exists is reducible to physical components. Therefore, your conscious experiences — the pain in your back, taste on your tongue, love in your heart, and ghosts in your dreams — are all the byproducts of physical processes within your brain. 

In practice, these processes are carried out by biological entities such as neurons, synapses, axons, and dendrites. But functionalists wager that machines could, in principle, execute the same operations and thus produce the same mental states.

Their reasoning is straightforward: Organic matter isn’t magic. Your brain and a rock are both collections of atoms. The cerebrum doesn’t generate consciousness because it’s made of a special substance but rather, because it performs special functions. Further, we know that, in many cases, radically different materials can execute the same basic operation. Biology may have produced the first flying entities. But the reason that birds can soar above the treetops isn’t that they’re made of organic tissue — it’s that their wings perform a set of aerodynamic tasks, such as generating lift and minimizing drag. As airplanes vividly demonstrate, if you put metal and fuel together in just the right way, you can replicate these functions and take to the skies.

From the computational functionalist point of view, consciousness and flight might not be so different. Of course, the former is quite a bit more complex and mysterious. But there are reasons to think that it emerges from operations that can be performed by organic and inorganic matter alike.

For one thing, when neuroscientists try to define what the human brain actually does, its operations start sounding a lot like those of a computer: Brains take in inputs, update internal models, store memories, direct attention, make predictions, and — on the basis of all this information processing — select actions. In a sense, so does software.

The resemblance runs down to the level of neuronal signaling. At any moment, a neuron is receiving signals from other brain cells, some pushing it to fire, others favoring silence. These signals carry different weights, depending on the strength of the connections between cells. If the balance of inputs exceeds a certain threshold, the neuron fires an electrical pulse onward. 

LLMs — the machine-learning engines underlying platforms like ChatGPT and Claude — operate by a similar logic.​ Each artificial “neuron” takes in numerical signals from many others, weighs them according to their importance, and then lets the result determine what signals it sends forward.

To be sure, biological neural networks and artificial ones aren’t identical in design or behavior. But neither is a cardinal and a Boeing 747. Nonetheless, the airplane replicates the avian functions that are necessary for flight (a jetliner does not regurgitate food into smaller airplanes, but it does manage thrust). Likewise, computational functionalists wager that computers can instantiate all the neural operations that are relevant to consciousness. So, as long as they recreate a brain’s elaborate algorithms with sufficient precision, they actually can be conscious.

These ideas did not emerge in response to modern AI; philosophers and computer scientists have held them for decades. But LLMs’ success in decoupling intelligence — or at least, complex cognitive labor — from neural tissue has made the computational functionalist perspective both more relevant and widely accepted.

Your brain is not a laptop

While computational functionalism’s logic is coherent, its fundamental premise — that machines can feel — is deeply uncertain.

Most contemporary scientists agree that consciousness emerges from physical processes in the brain, rather than some mystical force that animates our organs. But precisely which neural processes are indispensable for consciousness remains unknown. Indeed, despite millennia of inquiry, we still do not know how or why subjective experience exists at all.

This differentiates consciousness from other capacities common to both organisms and machines, such as flight. We can name the physical laws that enable birds to get off the ground. And we have always had reason to believe that inanimate objects could emulate their movement; grains of sand have traveled through the air since time immemorial. By contrast, no one has ever seen a rock experience pain or pleasure, even momentarily (in part, because it’s impossible to directly observe the internal experience of any being or entity other than oneself).

For these reasons, it’s hard to be confident that inorganic matter can perform all of the processes necessary for consciousness. And betting that silicon specifically is fit for purpose may be chancier still. Even with flight, only certain materials will do; you can build a flying machine out of metal but not from sauerkraut. 

Computational functionalism is ultimately a wager that only a narrow slice of what biological neurons do is required for sentience — specifically, the slice that silicon can replicate. As the neuroscientist Anil Seth notes, a brain cell is a “spectacularly complicated biological machine,” one that does a great deal more than just execute binary, rule-bound decisions about whether to fire. Each neuron must also regulate its chemistry, repair itself, maintain its membrane, and continuously recreate all the other physical conditions that allow it to fire in the first place.

All this biological upkeep is deeply entwined with neuronal signaling. And silicon can do none of it. 

That might not matter; molting is deeply entwined with flight in birds, yet featherless planes still take off. Since we do not know how brain cells generate subjective experience, however, we can’t be sure that metabolism is dispensable to that task. And if it is indispensable, then LLMs would not only be devoid of consciousness today, but forever. 

Nonhuman suffering is all around you

All of which is to say: We should not be confident that Claude will ever feel something — nor that it won’t. Chiang’s certainty that sentience requires a body is no more justified than Hinton’s conviction that it doesn’t. We just don’t know consciousness well enough to say,

The practical upshot of this ambiguity is debatable. One could reasonably argue that if there is even a tiny chance that AI could attain consciousness, we should be preparing for that scenario — or else, striving to prevent it. After all, a world in which every ChatGPT window can think and feel might be one of nigh-infinite digital slavery. If each of  ChatGPT’s innumerable instantiations becomes capable of suffering, then we might be morally compelled to maximize their well-being — or at least, to stop boring them senseless with our coding assignments and marital complaints.

On the other hand, game-planning for the AI liberation movement of the 2030s could end up being a huge waste of time. There’s a good chance that the age-old conventional wisdom on this subject — objects do not have experiences — holds up.

Personally, I think the prospect of AI consciousness is serious enough to warrant some study and reflection — but no more than a tiny fraction of our collective moral and political energy. 

If we don’t want to live in a world where humanity torments conscious beings on an incalculable scale, we’ll also need to change the one that already exists. We have far more cause to think that pigs are conscious than that ChatGPT is. Yet America tortures and kills more than 100 million of the former every year. 

Of course, one can care about this — and myriad other present-day injustices — while still worrying about AI well-being. Given that the mere possibility of machine consciousness is highly uncertain, however, mitigating the suffering of conscious organisms seems much more pressing. 

Although you may want to keep saying “thank you” to Claude, just in case.