Great work. I think AI consciousness is physically impossible because substrate independent is false. We know that consciousness exists in organisms with highly centralized nervous systems, whereas we have no evidence that it exists anywhere else. And arguments like China brain show that functional properties aren't sufficient for consciousness. So unless something changes in that evidentiary picture, I think we should inductively conclude that consciousness probably requires a biological substrate.
You're right! I'm a straight up physicalist. Physics is the most empirically successful theory we have, and it leaves no room for non-physical causes, at least within the domain of everyday life (can't speak as categorically regarding domains like the physics of black holes that are less well understood). So I don't think there's any reason to believe in non-physical entities, and even if they did exist, they wouldn't be able to causally interact with the everyday world.
This is one of the clearest breakdowns I’ve seen of why we should be cautious about attributing phenomenal consciousness to AI. The IBE approach feels intuitive at first glance—after all, we rely on it daily to assume other humans have inner lives—but once you get into the weeds of “relevantly similar behavior,” things fall apart fast. Just because an AI can act like it's conscious doesn’t mean there’s anything it’s like to be that AI. The ghost in the machine might just be a really convincing ventriloquist.
Exactly! And thanks for the kind words! I tend to think machine consciousness is not the kind of think we can ever make, but say it were possible, unless we can get around the IBE blockers, we might make a consciousness machine that we aren't able to justifiably attribute consciousness to--which is a whole different kind of horror.
I disagree with nearly all of this, but I'll just make one point.
You quote this from Huemer, with approval: "The computer is following an extremely complicated algorithm designed by human beings to mimic the behavior of intelligent beings."
I think that contemporary LLMs are involved in mimicry and nothing else, so I agree with half of this.
The problem is that the final algorithm being executed is not designed; it is trained. The training algorithm was designed, but that is a small component of the final design. Someone with access to the details could give us a ratio of the informational content of the final LLM (in bits) with the training code (in bits), and it would be high.
If consciousness is the best way of getting some types of cognitive work done, then training might discover it emergently just as evolution discovered it emergently. I'm not saying that this is necessary, but it is at least possible. If conscious AIs arrive, which I think is very likely this century, then there is a high chance that the actual algorithm being executed will be a result of selection and training processes that the human programmers do not have full insight into.
Furthermore, perfect mimicry of conscious humans is already a very difficult task (not achieved by any current LLM), and AI development will be adding other difficult tasks in the training process, so there is no upper limit on how much value there might be in discovering novel cognitive solutions during training. (Some of the most challenging tasks will be real-time engagement with the physical world, necessitating an attention management system to prioritise cognitive resources.)
The solutions that emerge in tackling these challenges do not have to have been anticipated by the human programmers.
If evolution found it "useful" to develop consciousness, then AI training against increasingly difficult problems might do the same. It is at least plausible (I would say likely) that instantiating an internal consciousness system is the best way to perform well at some tasks, and it would not necessarily require a deliberate design decision on the part of programmers.
That said, current LLMs have no internal cognitive behaviour; they generate their outputs in a single pass. The idea that they might be conscious is silly, and they are obviously recycling our own consciousness language rather than reporting something they found on introspection.
It's becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman's Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher order consciousness, which came to only humans with the acquisition of language. A machine with only primary consciousness will probably have to come first.
What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing.
I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.
My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461, and here is a video of Jeff Krichmar talking about some of the Darwin automata, https://www.youtube.com/watch?v=J7Uh9phc1Ow
Great work. I think AI consciousness is physically impossible because substrate independent is false. We know that consciousness exists in organisms with highly centralized nervous systems, whereas we have no evidence that it exists anywhere else. And arguments like China brain show that functional properties aren't sufficient for consciousness. So unless something changes in that evidentiary picture, I think we should inductively conclude that consciousness probably requires a biological substrate.
Great thoughts here! I take it you don't believe in God/gods, angels, demons, ghosts, souls, or otherwise immaterial or spiritual entities?
You're right! I'm a straight up physicalist. Physics is the most empirically successful theory we have, and it leaves no room for non-physical causes, at least within the domain of everyday life (can't speak as categorically regarding domains like the physics of black holes that are less well understood). So I don't think there's any reason to believe in non-physical entities, and even if they did exist, they wouldn't be able to causally interact with the everyday world.
Phenomenal article and great follow up, really liked your explanation of IBE. Thanks Parker!
You're thr man🤝 thanks for reading it!
This is one of the clearest breakdowns I’ve seen of why we should be cautious about attributing phenomenal consciousness to AI. The IBE approach feels intuitive at first glance—after all, we rely on it daily to assume other humans have inner lives—but once you get into the weeds of “relevantly similar behavior,” things fall apart fast. Just because an AI can act like it's conscious doesn’t mean there’s anything it’s like to be that AI. The ghost in the machine might just be a really convincing ventriloquist.
Exactly! And thanks for the kind words! I tend to think machine consciousness is not the kind of think we can ever make, but say it were possible, unless we can get around the IBE blockers, we might make a consciousness machine that we aren't able to justifiably attribute consciousness to--which is a whole different kind of horror.
I disagree with nearly all of this, but I'll just make one point.
You quote this from Huemer, with approval: "The computer is following an extremely complicated algorithm designed by human beings to mimic the behavior of intelligent beings."
I think that contemporary LLMs are involved in mimicry and nothing else, so I agree with half of this.
The problem is that the final algorithm being executed is not designed; it is trained. The training algorithm was designed, but that is a small component of the final design. Someone with access to the details could give us a ratio of the informational content of the final LLM (in bits) with the training code (in bits), and it would be high.
If consciousness is the best way of getting some types of cognitive work done, then training might discover it emergently just as evolution discovered it emergently. I'm not saying that this is necessary, but it is at least possible. If conscious AIs arrive, which I think is very likely this century, then there is a high chance that the actual algorithm being executed will be a result of selection and training processes that the human programmers do not have full insight into.
Furthermore, perfect mimicry of conscious humans is already a very difficult task (not achieved by any current LLM), and AI development will be adding other difficult tasks in the training process, so there is no upper limit on how much value there might be in discovering novel cognitive solutions during training. (Some of the most challenging tasks will be real-time engagement with the physical world, necessitating an attention management system to prioritise cognitive resources.)
The solutions that emerge in tackling these challenges do not have to have been anticipated by the human programmers.
If evolution found it "useful" to develop consciousness, then AI training against increasingly difficult problems might do the same. It is at least plausible (I would say likely) that instantiating an internal consciousness system is the best way to perform well at some tasks, and it would not necessarily require a deliberate design decision on the part of programmers.
That said, current LLMs have no internal cognitive behaviour; they generate their outputs in a single pass. The idea that they might be conscious is silly, and they are obviously recycling our own consciousness language rather than reporting something they found on introspection.
It's becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman's Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher order consciousness, which came to only humans with the acquisition of language. A machine with only primary consciousness will probably have to come first.
What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing.
I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.
My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461, and here is a video of Jeff Krichmar talking about some of the Darwin automata, https://www.youtube.com/watch?v=J7Uh9phc1Ow