If an AI Tells You It's Conscious, Should You Believe It? | Pt. 1 of 3

Three Ways to Tell If A Machine Is Conscious and Why They Don't Work pt. 1

Here’s Bob Pepper’s cover art for Philip K. Dick’s novel, Do Androids Dream of Electric Sheep. It’s so good I had it tattooed on my forearm.

If an artificial intelligence (AI) tells you that it’s conscious, you shouldn’t believe it.

Why think that? Well, I think the arguments for machine consciousness fail and don’t leave us with good reasons to affirm that a given machine is conscious. In this post I’ll look at the three major ways to argue that an AI is conscious and I’ll explain why none of them are actually sufficient for attributing phenomenal consciousness to a machine—where ‘phenomenal consciousness’ means that there is something-it’s-like to be a thing. So a phenomenally conscious AI would have something-that-it’s-like to be it and a non-conscious AI would not.

Now, there's no way to directly experience the qualitative experiences of minds other than our own, of ‘other minds’. So how do we know that anyone else has a mind like ours? That any other minds are phenomenally conscious? Well, we make arguments for the existence of other minds. The same goes for AI. How could we know that an AI is conscious? We make similar arguments to those made in favor of the existence of other human minds and animal minds.

I think there are three major ways to go about arguing for AI phenomenal consciousness (called ‘machine consciousness’ in the literature).

Here are the three ways I have in (my phenomenally conscious) mind:

Way1: drawing an analogy between the human cognitive processes and/or framework, and an AI’s organizational structure, and then concluding that the AI is conscious like us.

Way2: inferring that phenomenal consciousness best explains the behavior of an AI.

Way3: Punting to Panpsychism, which is to say, positing panpsychism (the family of theories that claims all things are conscious) and getting machine consciousness for free. If everything is conscious, then AIs are conscious too.

Here in part 1, I’ll consider Way1 and argue against it.

The details of this post come from my article in the Journal of Artificial Intelligence and Consciousness. If you want to read the full thing with more of the technical jargon, less of the colloquial jargon, and all of the footnotes, you can find the penultimate version of my paper on my philpeople page here. I am going to nab some stuff from that paper without quoting myself. This post is not an academic paper. If it were then I’d be plagiarizing myself. Since it’s not, I don’t care. I worked hard on my arguments and definitions over in that paper and I’m going to jack them for this post. That’s that. I’m not about to quote myself in a blog post—that’d be preposterous.

Oh, and by the way, the title of this post is an “allusion” or “homage” to the title of my friend Susan Schneider’s piece in Scientific American—which is to say, I completely ripped off the title because it was so good. Susan is a very good philosopher of mind and AI. You should read her piece here.

Anyways, let’s get into the first way to argue that a machine is conscious and see why it’s no good.

Way 1: Organizational Similarity

The first way to argue that an AI is conscious is to point to a sufficient level of organizational similarity between the AI and the human cognitive processes. If the AI is similar enough to human beings in its cognitive processes, then perhaps it’s reasonable to attribute consciousness to the AI.

Here’s how the argument might run:

Sufficient Organizational Similarity Argument (SOS):

1. Once a sufficient level of similarity is reached between the organization of an AI system and that of human cognitive agents, one is justified in attributing consciousness to that AI system (when performing relevantly similar cognitive tasks).

2. AI system X has reached a sufficient level of organizational similarity to that of human cognitive agents.

therefore,

3. We are justified in attributing consciousness to AI system X (when performing relevantly similar cognitive tasks).

So, how do we tell if a machine is conscious? It’s ‘cognitive processes’ are relevantly similar to a humans—it’s organizational system is analogous enough to a human beings that it’s reasonable to attribute phenomenal consciousness to it, that is, it’s reasonable to believe that there is something-it’s-like to be that AI. And so we will.

Now, this all presupposes something like the ‘substrate independence thesis’, i.e., the idea that conscious minds can be associated with different kinds of ‘substrates’, rather than just carbon based substrates like human brains or the brains of other earth-based life forms that we take to be conscious: bats, cats, dogs, alligators, etc. In order for us to draw and analogy between a machine and a conscious human and conclude that a machine is conscious, we need to believe that consciousness can be associated with, or arise out of, or be constituted in whole or in part by, or supervene on, etc., material other than carbon brains like the ones we have. If we don’t believe that, then the idea of machine consciousness can’t get off the ground at all. If we believed that something about our carbon brains is unique and necessary for consciousness, then we’d never think anything other than a carbon based animal, or perhaps biological life, could be conscious—including an AI.

Here’s my take on the substrate independence thesis:

Substrate Independence: it is possible for phenomenally conscious minds to be associated with, arise out of, be constituted in part or in whole by, supervene on, etc. (fill in the appropriate nomenclature from your preferred theory of mind), a substrate other than the carbon substrate of the human brain.[1]

Substrate Independence seems pretty reasonable. If it’s true, then the substrate associated with consciousness isn’t all that important. Perhaps there are some substrates that simply cannot be associated with consciousness, but it’s an empirical question as to what those would be—and empirical questions are for the nerds in the sciences not the armchair goofballs like us.

But if substate independence, then machine consciousness is possible, that is, a silicon substrate can, in theory, function in the same roll as our carbon brains do in our own conscious experience. Again, if we have a reason to think substrate independence is false, and that there really is something especially important to our carbon brains in the roll of our conscious experience, then we have a reason to deny that machine can be conscious. But let’s just assume that substrate independence is true for the sake of argument.

Even with substrate independence, there are some other challenges for those who would attribute consciousness to machines based on the SOS argument above. For one, we need to define what we mean by “organizational similarity” and secondly, we need to define what exactly counts as “sufficient” organization similarity.

As far as I can tell, there are two potential ways to flesh out what the SOS proponent could mean by sufficient organizational similarity:

(1) isomorphic sufficiency, which emphasizes the ‘isomorphic’ or identical form or structure between a human’s cognitive framework and an AI’s.

(2) functional sufficiency, which emphasizes the output of the conscious human agent and the output of the AI.

So the SOS proponent can either draw an analogy between the structure of an AI and the structure of a human agent’s mental framework or they can draw an analogy between the function of the human agent and the function of the AI.

When can we justifiably attribute consciousness to a machine? The folks who want to point to similar structures can give an isomorphic sufficiency condition and the folks who want to emphasize the functional similarities can give a functional sufficiency condition:

isomorphic sufficiency condition: an AI has reached the sufficient level of organizational similarity to a human cognitive agent needed to justify an attribution of phenomenal consciousness when said AI system is a precise isomorph of a human cognitive agent.

I’ll call an AI that meets the isomorphic sufficiency condition an ‘isomorphic AI’.

functional sufficiency condition: an AI has reached the sufficient level of organizational similarity to a human cognitive agent needed to justify an attribution of phenomenal consciousness when said AI system is functionally identical to a human cognitive agent.

I’ll call an AI that meets the functional sufficiency condition and ‘AIF’.

Let’s start with isomorphic AI. An isomorphic AI will precisely exemplify all the same constitutional features of a human cognitive agent which, the machine consciousness proponent will argue, should predict consciousness in the isomorphic AI, given the substrate independence thesis. The substrates are different, the human’s cognitive hardware is made of carbon (and maybe an immaterial soul but maybe that’s more like the software—that’s for another time, though) whereas the Isomorphic AI’s substrate is made of silicon. But that doesn’t matter because we’ve already posited the substrate independence thesis. While the substrate is different, an isomorphic AI’s cognitive framework/make up is structurally identical to a human cognitive agent’s. So if the human is phenomenally conscious, so too should the isomorphic AI be phenomenally conscious.

But I don’t think the isomorphic sufficiency condition is all that sufficient for attributing consciousness to isomorphic AIs after all. Why? Because isomorphic form/structure doesn’t guarantee analogous phenomenal consciousness. Consider Perry and Dead-Perry.

Perry is a living and conscious human cognitive agent at time t₁ and Dead-Perry is the same being but no longer living nor conscious at time t₂. Perry and Dead-Perry have the exact same cognitive organization, they are identical in form or structure, the only difference is one is non-conscious and dead and the other is conscious and living. It looks like, based on the Perrys, that isomorphic similarity is insufficient for justifiably attributing consciousness, since, if we limited ourselves solely to isomorphic similarity then we would have falsely attributed consciousness to Dead-Perry—the dead, not conscious, corpse-version of Perry.

Now you might be thinking “not so fast, Park! This was entirely too quick.” I know, I know. You’re going to take issue with the Perry/Dead-Perry comparison, and argue that while their cognitive organizational structures may appear to be the same at a gross level, at a more fine-grained level of observation, say the microscopic and molecular level, the two couldn’t possibly be the same, due to the effects of ischemia on the ultrastructure of the nervous system or some such something or other (this point was brought up to me by Paul Gould as well as reader number 2).

For those who are skeptical about the comparison between Perry and Dead-Perry, consider instead Perry and 3D-Printed-Perry. 3D-Printed-Perry has been printed in flesh and blood by an advanced 3D printer. It is an exact isomorphic copy of Perry at time t₁ except for the fact that 3D-Printed-Perry is not alive nor conscious and Perry is. 3D-Printed-Perry is not strictly speaking ‘dead’ because it was never alive and since we’re limiting our focus to time t₁, the effects of ischemia need not be considered. So long as a non-conscious 3D-Printed-Perry is possible, we have a counter example to the isomorphic sufficiency condition since we have precise isomorphic similarity without analogous levels of phenomenal consciousness. Thus, isomorphic similarity is not sufficient for attributing consciousness and there’s no good reason to think an isomorphic AI is phenomenally conscious.

Hang on though, maybe the SOS argument above can still be salvaged if we just interpret the “sufficient organizational similarity” along functional lines instead of isomorphic. What matters is not so much the identical structure of the analogous cognitive frameworks, this proponent will argue, but instead the functional similarity of the analogous cognitive frameworks. What matters is that they’re both able to do the same things and produce the same kinds of products. If an AI meets the functional sufficiency condition, it’s an AIF, meaning that since it’s functionally identical to a human cognitive agent, then we’re justified in attributing consciousness to it. An AIF will exemplify all the same functional features of a human cognitive agent which could reasonably predict consciousness in the AIF given the substrate independence thesis.

What are some of those functional features which should be present in the AIF? In his paper draft of “Could a Large Language Model Be Conscious?”, David Chalmers provides a helpful list, including things like having functional sense ‘organs’, embodiment, world-models and self-models, recurrent processing—that is a feedback loop rich form of information processing rather than a one directional “feedforward” form of information processing—a global workspace with unconscious information processing happening in multiple non-conscious modules, each striving to put its information in the conscious awareness—as well as a unified agency.

This list of features is a good start but it isn’t a complete list of the necessary and sufficient features for being functionally identical to a human cognitive agent and we’d actually need that kind of a list to ensure that an AIF is actually an AIF, i.e., that it’s truly functionally identical to a human, if we want to justifiably attribute consciousness to it. Furthermore, none of the conditions on Chalmer’s list are novel. In fact, all of them are being explored in various AI projects currently and none of those AIs exhibiting these features are considered to be phenomenally conscious. It’s not clear why adding them all together would suddenly give rise to machine consciousness.

But this whole thing is moot after all because being functionally identical doesn’t guarantee analogous phenomenal consciousness either. The map is not the territory. Just because there is a simulation of a hurricane on the weatherman’s screen doesn’t mean his keyboard will get wet. As John Searle forcefully notes,

“…no body supposes that the computer simulation is actually the real thing; no one supposes that a computer simulation of a storm will leave us all wet, or a computer simulation of a fire is likely to burn the house down. Why on earth would anyone in his right mind supposes a computer simulation of mental processes actually had mental processes?” John Searle, Minds, Brains, and Science (Cambridge, Harvard University Press, 2003), 37.

We need more reason to think that mimicking the mind would likewise produce phenomenal consciousness—especially in light of counterexamples, of which there are some fun ones, like Ned Block’s Chinese Nation Argument.

Block originally targeted the machine functionalist theory of mind, which is a theory about mental states, especially conscious mental states. Machine functionalism claims that the human mind is a realized Turing machine table, that is, a set of inputs, outputs, and internal states of a given order. Block argued that this theory of mind was far too liberal in that it would falsely ascribe phenomenally conscious mental states to things which were obviously not conscious, such as the entire nation of China.

Block paints a scenario wherein the Chinese government orders its people to form a pattern that realizes a functionally equivalent system to that of a human cognitive agent, which responds to inputs, changes its internal states accordingly, and produces the right output in keeping with the realization of the proper machine table—a machine table just being the complete and exhaustive specification of the machine’s operations, also called the “instruction list” (see Jaegwon Kim, Philosophy of Mind 3^rd ed. (Boulder Co.: Westview Press, 2011), 139-147 for more).

Block contends that the Chinese Nation argument is a prima facie counter example to machine functionalism because it is doubtful that the nation of China would have any mental states at all, let alone phenomenally conscious states (see Ned Block, “Troubles with Functionalism” in Philosophy of Mind: Classical and Contemporary Readings 2^nd Edition ed. David Chalmers (New York: Oxford University Press, 2021), 100-102. for more).

Now, we can get the same result from Block’s Chinese Nation argument against the functional sufficiency condition above by adding the functional exemplars that we appropriated from Chalmers on behalf of the machine consciousness proponent. Just add the details to Block’s scenario and we can call it The New Nation of China Argument:

The New Nation of China Argument

Here’s the scenario: Chinese government officials grew weary of their Chinese Nation Experiment. It seemed as though no conscious mental states were arising from their system. Machine functionalism was indeed too liberal of a theory. But these officials got wind of a new theory to try out. Instead of merely organizing their populace into the realization of a particular machine table, the Chinese government officials learned of Chalmers’s additional criteria for machine consciousness and implemented them into their nationwide experiment. They already had the sense organs from their last experiment, which were the citizens charged with looking out for the inputs of the system. One by one they added the extra conditions into their experiment. They built a giant skyscraper to live in while they conducted their experiment which satisfied the embodiment condition, they reorganized their system to account for world and self-models, they made sure to incorporate recurrent data processing, and implemented an elaborate alert system to function as a global workspace of data. They even had a singular dictator in charge of the whole show to fulfill the unified agency criterion.

Now as implausible as the New Nation of China scenario is, all that I need for it to go through is that is it metaphysically possible, which of course it is. So since its implausible to attribute consciousness to the New Nation of China system, then functional identity is not a sufficient condition for the justified attribution of consciousness because functional sufficiency is not enough to guarantee analogous phenomenal consciousness.

So neither the isomorphic nor the functional sufficiency conditions are sufficient conditions for justifiably attributing consciousness and premise (1) of SOS above is left unmotivated.

So Way1 of arguing for conscious AIs is a bust. Even if an AI were conscious, we couldn’t use this way to justifiably attribute consciousness to it.

---

In part 2 of this 3 part series, I’ll consider the second way to argue for machine consciousness:

Way2: inferring that phenomenal consciousness best explains the behavior of an AI.

Here I’ll explain why machine consciousness is not the best explanation for the ‘behavior’ exhibited by AIs.

In part 3 of this series, I’ll consider the third way to argue for machine consciousness:

Way3: Punting to Panpsychism, which is to say, positing panpsychism (the family of theories that claims all things are conscious) and getting machine consciousness for free. If everything is conscious, then AIs are too.

There I will argue that even if some form of panpsychism were true, it still wouldn’t guarantee that AIs are, or could be conscious.

This was a long one and full of dense stuff and words I meant to explain better but didn’t. If you made it this far then you’re a champ! Good work. Let me know what you think about the arguments presented for and against machine consciousness in the comments below.

If you liked this post and want to support my work, consider buying me a coffee:

And if you really, really liked this post, and want access to exclusive content like my systematic theology class, my commonplace book of wise sayings, access to our Zoom call book club sessions on Dune, and lots of other exclusive posts, then consider becoming a paid subscriber.

---

[1] Substrate independence is not unique to me, this is just my particular take on it. It was first proposed by Hilary Putnam in his multiple realizability argument against mind-brain identity theories of mind, c.f. Jaegwon Kim, Philosophy of Mind 3^rd ed. (Westview Press, 2011), 121-22 and Hilary Putnam ““Putnam, Hilary, 1967, “Psychological Predicates”, in W.H. Capitan and D.D. Merrill (eds.), Art, Mind, and Religion, Pittsburgh: University of Pittsburgh Press, 37–48.