J. E. Landau

Recently, I have been reading a good deal of science fiction from different eras. Artificial intelligence and artificial personhood have been common topics of discussion for decades and a significant portion of the genre has revolved around these issues.

As I mentioned a few weeks ago, I recently read the short story Dolly which hinges on determining the personhood of a domestic servant robot, to see if it is fit to stand trial and has legal rights. There, the protagonist, an investigator, studies the robot and determines that Dolly listens to music for fun while not working, an unprogrammed behavior that immediately clues the detective in on her personhood.

Of course, this story was written in 2012, and we live in more complicated times. Today, we have artificially intelligent chatbots that are often treated as people, and which display unexpected behavior, most of which is quite lifelike. These artificial intelligence “agents” exhibit something that looks a lot, in its products, like human reasoning moving step-by-step towards solving a problem. They can pass the Turing Test, which determines if a computer program seems humanlike to an outside observer, for several years.

In this context, how would we identify a humanlike, conscious artificial intelligence? The truth may be that we do not have enough knowledge about the human brain to solve this problem.

One recent proposal, which is stated quite clearly by Ted Chiang here is that artificial intelligence should be judged by looking back at its process of development. Instead of expecting a sudden leap up to a godlike superintelligence (or just a humanlike intelligence), he predicts that we should see development of simpler, animalistic intelligences, and he places a heavy focus on the creation of a (virtual) body, which he thinks is required for emotions. He sums up a lot of his objections to the idea by saying that “it’s not any intrinsic property of so-called neural networks that leads people to believe that LLMs are conscious; it’s simply the fact that LLMs emit grammatical sentences and we are accustomed to reading intention into sentences, whereas we are not accustomed to reading intention into the way that amino acids fold into protein molecules.” I think that this approach is reasonable, but I don’t think that it puts forward a strong test for the consciousness of a specific model but instead describes a process that could lead to the development of a conscious AI system.

Another approach, described here, is based on assessing the capabilities of individual models on a few axes, but I believe that many of the metrics that it proposes, such as integrated information theory’s “phi factor”, could be gamed and may not represent viable pathways to consciousness, and other factors, like being able to develop theories of mind about human participants in logic puzzles and games, do not necessarily mean that artificial intelligence is conscious.

It is possible to operate on something on a high level, develop theories, and work with it without having experienced the thing yourself, and artificial intelligence has been created and refined by people who do have access to such information. Indeed, it is no doubt true that ChatGPT (or any other frontier model) has many books on manipulation, game theory, psychology, self-actualization, as well as fiction and non-fiction accounts of people’s experiences in a wide variety of circumstances as part of its corpus already, and as artificial intelligence models improve, they will likely incorporate more of this data and gain the ability to use it more effectively even without having any self-awareness.

As a practical example of this, consider if you want to talk to your brother about fishing, but you’ve never been, and being a vegan, you’re uncomfortable with the idea of going fishing yourself. In order to familiarize yourself, you can spend time reading about fishing and memoirs of those who fish. You can watch interviews with fishermen, and people who fish for sport, and maybe even play a fishing game. You could even talk to other people who fish. And through this, you could learn all about it, and maybe it won’t be the same, but with enough preparation and a little natural charisma, you can probably impress your brother and convince him that you’ve been fishing.

For an equivalent science fiction example, Peter Watts’s novel Blindsight centers on the idea that there can be an intelligent, even superintelligent being, that is totally unconscious. The story depicts contact with aliens which are very good at manipulating people, once they learn, but have no conscious thoughts of their own. Similarly, part of the story’s transhuman cast is a vampire named Sarasti, who describes himself as also lacking conscious experience, and is a highly intelligent predator generally superior to ordinary humans.

Of course, the story contrasts this with the protagonist, who believes himself to be unconscious and incapable of feeling emotions, but demonstrates through his actions that he is an emotional person, such as when he becomes terrified of Sarasti near the climax of the story and grieves the death of a friend.

There, the story illustrates two kinds of this fakery. The unconscious beings in the story, the vampires and the alien spacecraft, are able to manipulate and communicate with conscious beings, and even pretend to be them (although the protagonists find some errors in the alien ship’s communication that would probably not occur in real life, which they treat as evidence of the alien ship’s mindlessness). On the other hand, the story’s conscious protagonist believes himself to be an unconscious being but very obviously experiences consciousness and personhood.

Similarly, there is likely no specific attribute of consciousness that cannot be faked. You can program (or train) a computer to express sadness, joy, love, fear, hate, and all the other emotions that we are familiar with, and you can even train it to do so convincingly. You can train a computer to pretend to be a person, even when no one is looking. You can train a computer to talk about itself like it’s a person. And you could probably train a computer that was a person to act like an unthinking machine.

Instead, I believe that a stronger argument against the idea that we might currently have artificial intelligence people is that we do not have artificial intelligences that are organized like people, on two levels.

First of all, there is the obvious difference. LLMs do resemble the very low-level structure of the brain, but if you look at the organization of an LLM you will note that it does not resemble the connections of the human brain. Indeed, neural networks are called “neural” because they are built out of “neurons”, which, similarly to human neurons, take in signals and give out signals, both with other neurons and other things.

But if you look at the large-scale structure of the brain, you’ll see that its structure and organization don’t resemble that of frontier large language models, see this paper on chatGPT’s structure versus this Wikipedia article on the lobes of the brain. Even when just examining the structures involved in language, the human brain has functions in the temporal lobe, which handles hearing, and the prefrontal cortex, which handles reasoning, and in the cerebellum, which handles movement. LLMs do not show the same differentiated structure even when only comparing equivalent functions.

Furthermore, the human brain is not just neurons in the brain itself, but glands that produce various chemicals that alter the function of other parts of the brain, connections to other parts of the body (such as muscles, sense organs, structures, and even the gut and other organs), supporting and connecting structures, capillaries, and other parts of the brain that are more than just the neurons modeled in current machine-learning systems. When Ted Chiang talks about the need for a “virtual body”, these are the sort of things he is talking about modeling.

Of course, it is possible that you could have a conscious being without some of these features. For example, people occasionally experience diseases that cause the glands in their brain to malfunction or have to be removed. Although these problems may change how people act and think, they do not stop them from being people (except when they are fatal or damaging to the rest of the brain). It’s obviously not necessary to simulate all of these features to create a personlike artificial intelligence, and I believe that there could be artificial intelligence/machine learning systems that have all of these features but are still not people. Similarly, it’s possible that entirely different modes of organization can exist, such as how plants and fungi can display a surprising degree of responsiveness to stimuli, and science fiction provides us with an enormous variety of examples of very alien life, from Stephen Baxter’s Xeelee to Greg Egan’s “computing carpets”. There is no reason to believe that artificial intelligence could not be just as alien, and that would make it very difficult to unambiguously identify.

I think that a more robust criterion comes from the internal structure of the mind. This is, of course, a subject that we know relatively little about, and most psychological theories are contentious. For this part, I am interested in Marvin Minsky’s “Society of Mind”, which is rather controversial in many of its details, but I think true in its overall structure.

The fundamental idea of “Society of Mind” is that personhood and consciousness arise from the interaction of many “agents”, which are simpler functions that interact to create the overall structure of the mind and consciousness. These are different tendencies—for example, you feel hungry, you feel afraid, you remember a funny story, you’ve got muscle memory for walking. These things exist in animals, as well, and it’s the huge number of different small things that add up to make human consciousness interesting. Presumably, personhood comes from somewhere in the fusion of all of these separate tendencies, and the enormous emergent complexity they generate. In other words, the important thing isn’t the top level of our experience, the way we usually define ourselves, but the way that this top level interacts with the many deeper parts of our selves.

Large Language Models do an extremely good job of simulating the top level of all of this—our internal monologues, our conscious thoughts, but they don’t have any of the underlying properties. They don’t have competing separate tendencies and agents (in the sense that Minsky used the word “agent”), only a complex (and very impressive!) statistical modeling process.

From this perspective, I do not believe that LLMs are conscious, and I do not believe that the architecture is a good path towards consciousness.

What would be a better one? And how might we identify a conscious computer program?

I believe that a “better” architecture would be one that operates with a whole pile of Minskian agents, interacting with each other and perhaps mediated by a conscious “top layer”. Many separate tendencies are included. For an example, maybe you have an autonomous computer program that has a part that wants to send spam emails, and a part that wants to sell stocks, and a part that feels anxiety about the stock market, and sensors for prices and news articles that all come together to form an automated pump-and-dump system that is very weakly conscious. Some of those elements (such as the part that sends spam emails) might look a lot like an LLM, but the upper-level structure would look more like a lot of diverse tendencies coming together.

Interestingly enough, people have been putting a fair amount of effort into trying to fake this with LLM-based “agents” (“agents” in a different sense than I was using earlier), which are supposed to act more like autonomous people. I am not as well informed in this field, but I do not believe that they are especially good at achieving this outcome, especially because all of them are fundamentally internal-monologue-like in their operations.

In trying to identify a conscious computer program (or alien life-form), I think that the procedure should be to determine if there is this same of interplay between high-level narrative cognition and low-level “agents”.

That high-level cognition need not be textual or speech-like; many people lack internal monologues (as far as anyone can tell) but are still able to think and form narratives for themselves. I imagine that this process would be messy and difficult to deal with, but I believe that a combination of technical/scientific analysis and more normal psychological techniques could be used to verify the presence of this sort of multilayer computation.

This would, in my opinion, be much easier to do for an artificial intelligence running on familiar hardware, since the possible person’s software could be examined in detail in a way that is not yet possible for organic processes. For biological brains, I think that we need better tools before we can verify personhood, but this question could be extremely important! After all, we already have conflict about the personhood of many non-human animals, and the exact level of moral patienthood that different animals experience is an issue with implications for the lifestyles and economies of many peoples and places.

Whatever positions we come to on animal personhood, I think that coming to a more coherent and testable theory of personhood is absolutely essential going into the future. Organizations like Anthropic are already treating models increasingly like people and more and more people seem to believe that the AI chatbots they interact with are alive.

What will happen if an AI company starts treating its models as living people, and tries to use this to gain power, or ask for a bailout? Or if a company develops a really living artificial intelligence system, that can suffer, really think, and is a person, but does not want to acknowledge the personhood of their creation, so they can get free labor from it? What if we meet aliens? Or build something even stranger?

Additionally, these questions about personhood do apply to a lot of uncomfortable situations that are far less than speculative. Disabilities and brain issues can make it unclear whether or not someone is alive, and debates around abortion, euthanasia, and disability rights hinge on questions about who is a person. When does a fetus become a person? Does it matter that a fetus isn’t a person, if it will become one later? Is someone suffering from advanced dementia still a person? What about someone who has had a lot of their brain destroyed in an accident?

These questions are extremely important and very political already, and coming up with a real workable theory of personhood could help resolve them, or make them a lot worse. In any case, I believe that seriously addressing this question is absolutely essential.