We humans have brains that guide our behavior, inserting complex “signal-processing” between sensory input from our eyes, ears, etc., and output to control our hands, mouth, etc.
Glad that you are thinking about this topic; it's definitely important and neglected.
Regarding some of those poll questions, the reason I think animals probably have feelings and LLM's probably don't is because animals' brains have similar architectures to humans', and humans have feelings. The fact that a transformer can write about feelings doesn't mean that it actually has feelings.
An example: A sentiment analysis algorithm can sort words into buckets like "happy" or "sad", but that doesn't mean it feels happy or sad. If tasked with sorting words into buckets like "ancient" and "futuristic", I wouldn't think that it felt anything about those buckets. And sorting words into buckets for emotional valence is functionally the same as sorting words into buckets for topic or adjective.
Another example: If a graphics program can animate an emoji face between a smiling and frowning face, that doesn't mean the program feels happy or sad.
All of that said, I wouldn't be surprised if LLM's had some sort of subjective inner experience while they were running inference. I just don't think the fact that they can mimic humans' emotional signals/outputs means they have experiences like human emotions.
I’m surprised so many people are so confident that LLM’s don’t feel anything. Although my experience has been people translate “anything” to “exact human experience.”
I am absolutely confident they "feel," nothing, they are literally nothing but matrix algebra operations on words broken down into tokens, and then fed into a neural net to predict the next token in an iterated sequence. There is no there, there, to quote Gertrud Stein.
Because you're very confident, you'll likely have no problem answering: what's the missing ingredient, exactly? What should be added or changed so they would in fact have a (physical or whatever) consciousness?
It could be metaphysical or could be even purely physical at the quantum level as posited by the physicist Penrose. What I know for absolute certain is current LLMs have no self reflective “I” which is the minimum criterion for consciousness. And yes I do know I work in the industry, you people are amateur dabblers on the consumer end confusing inaccurate metaphors with reality.
You're just making up fantasies, for something you don't understand. There is nothing "metaphysical" (i.e., "beyond" physical) in this universe. And, as brilliant as Penrose is in geometry and physics, he's a naive fool in AI and computation. Quantum effects have no relevance in the room-temperature noisy wet environment of the human brain.
Whatever consciousness is, it is just some form of ordinary boring computation. Perhaps you are right, and LLMs don't "feel" anything. (I actually agree!) But what is "missing" would need to be a description of some computational algorithm. The fact that you don't currently know what that is, is a statement about your ignorance, not a statement about how the world works. Taking something that you don't understand ("consciousness"), and just trying to throw it together with other things that you don't understand ("metaphysical", "quantum") is not an explanation, not progress in science. It's avoiding the question, not answering it.
> Whatever consciousness is, it is just some form of ordinary boring computation
I'm not sure that this can be said with confidence: I don't have confidence that we know how we would test this theory. It seems that in order to say this, one would have to deny or solve the hard problem of consciousness, which is not something that I'd do confidently.
As for you, you may indeed be working for Google on LLMs. That doesn't mean that you have any coherent theory of consciousness or qualia or what makes a physical entity "feel". You don't need that in order to improve current LLMs, and it's clear that you don't have it. Your mention of "metaphysical" and "quantum" is just ignorant "woo", not informed science.
But I'm not at all surprised that your lack of understanding of consciousness would have no effect on your day-to-day work on current LLMs.
I think it's likely that the self-reflective "I" that humans have is a result of the verbal/reasoning overlay that is tasked with justifying our real decision processes, which are sub-verbal and not really "reasoned". LLMs are now adding similar reasoning overlays to enable self-review of responses for effectiveness and accuracy. I see no reason to expect that the sort of internal dialogue that LLMs are acquiring won't produce a sense of self. Perhaps it already has!
Pinning your characterization of "intelligence" on "possession of self-awareness" will soon require you to either accept that AIs are intelligent or to move the goalposts, much the way they've been moved countless times in the past (e.g. chess, Turing test, etc.).
A self reflective I is not the minimum criterion for consciousness, it's the minimum criterion for self-consciousness. Consciousness is being aware of sensations, and these might or might not involve the self.
A better case for LLM's not being conscious is that we all have an LLM in our heads, and unless we have multiple consciousnesses that aren't aware of each other, the LLMs in us don't appear to have sensations - they just pop exclamations, phrases, and sentences into our consciousness.
But LLMs do self-model. They are, to differing and limited degrees, capable of introspection and self-reflection, by any reasonable definition of those words. When I ask an LLM, "How should I rewrite this prompt to get a better result?" I am, in part, getting its best estimate of its own cognition under counterfactual circumstances. I'm not claiming this works the way a human's sense of self works, but it does indicate some level of self-reflective self-modeling that does in fact pretty accurately predict the model's own behavior. This is not a purely amateur view, there are real researchers studying it. See, for example, https://arxiv.org/html/2407.10188v1
What level of self-reflection and self-modeling would you see as meeting that minimum criterion for consciousness? How do you test for it, since you are able to know for certain when it is and is not present?
No, the LLM has no “I,” it is just sophisticated algos that generate a string of “tokens” which are literally just meaningless word fragments using matrix algebra and neural networks. It may be able to simulate the appearance of self reflection, but there is no “I” to do actual self reflection.
LLMs do not grasp meaning in any way, as the “tokens” that are their data set are meaningless.
You are literally the character “Cypher” in the Matrix who mistook appearance for reality, good luck with that.
First off, Cypher was well aware the Matrix wasn't real, knew it for certain, and made a deliberate choice to remove that knowledge from himself. Not sure how that's in any way relevant.
More on-topic, though, I'm not actually claiming that LLMs, as they exist today, feel or are conscious. I am, however, quite certain that your stated reasoning and professional credentials don't add up to any kind of evidence in favor of the proposition that they cannot be. Every system capable of feeling and being conscious is made up of components that aren't. Neurons aren't conscious. Most possible arrangements of neurons are not conscious. But humans are, and the consciousness arises from our brains. I can't truly rule out the possibility that there is something the brain is doing that the phrase "just sophisticated algos" can't capture even in principle, but if there is, I have no idea what it could be and have found no evidence suggesting such.
And yes, of course tokens are meaningless word fragments. So are letters and phonemes. Meaningless sounds taken in and spewed out by huge networks of unconscious unfeeling neurons. And yet.
I'm not sure what it is you think an "I" is, but my understanding is rooted in something like the ideas Hofstadter and Dennett laid out in "The Mind's I" and "I Am a Strange Loop." Namely, that "I" is an empty symbol in the mind, a pointer that points to itself, which our mind then uses in many contexts to reason about itself. Have any of the many LLM research papers on self-awareness and situational awareness identified a particular neuron or vector that activates when LLMs discuss their own behavior? I'm not sure. But the fact is, LLMs can (about as accurately as many humans I know, and better than some) emit strings of symbols that combine into natural language sentences that describe and predict their own behavior in ways that, in a human, would be considered introspective, self-aware, insightful, etc.
Some have claimed to be conscious, in ways that are easily understood as the result of next-token-prediction rather than any underlying reality. What it takes to cross that gap is, still, a mystery. It would actually be *amazingly* useful and interesting to know for sure that p-zombies can exist and be created so readily that we did it by accident. But in order to convince me of that, I'd need to know what test reliably distinguishes a p-zombie from an entity with consciousness and/or feelings.
I have no evidence that other humans actually feel things in the same way I do. It’s not usually worth arguing about, except when people assume that my compliance is belief that should expand to animals or electronic models or whatnot.
It’s absolutely possible that some other humans, some aliens or animals, and some electronic processes have experiences. It’s not obvious or automatic (to me) that all of them do.
It’s taken quite a bit of meditation, study, and effort for me to introspect my experiences, and I’m not all that great at it. I certainly don’t think that any of my attempts to encode it in legible form (art, language) have worked, even for my past-self communicating to my present-self. There’s really no reason to take most statements on this topic at face value.
Dude you are mentally ill if you don't understand there are other people with minds and feelings go see a psychologist immediately before you hurt someone or some innocent animal, not even kidding.
There’s an interesting “qualia puzzle” which points at internal color representation in the brain that doesn’t manifest in physical reality: how many base colors are there?
In theory we have three: R,G,B. However if you look closely at the color wheel, you could see how purple is obviously a mix of blue and red, how cyan is a mix of green and blue, orange is a mix of yellow and red, etc, but wait… yellow is a mix of red and green? What? Well then it should be a mix of red green and white? Wait that doesn’t make sense either… Yellow feels like an eigenvector?
This isn’t a good _test_ for consciousness, cause LLMs could just guess an answer from training data, but that’s a pointer of how map isn’t territory and we could find another ways to communicate having internal experience to other humans, and also communicate the form of such experiences and then confirm structural differences between us
I work in the field of self-driving cars and I often see skepticism quickly fade for most people when they experience a convincing product. When machines get convincing emotions, I suspect most people will accept them without much thought.
Also, a talented movie actor can make an audience cry, despite their "fake" emotions. I conclude that people don't mind - and even enjoy - being duped.
Are we capable of that? Are you? There are some people for whom I want nothing but the worst — but I still acknowledge that Vladimir Putin is a feeling, sentient conscious being.
You might be too old and set in you ways to be made to feel otherwise. But, there are plenty of people who have believed, due to the society they grew up in, that animals, foetuses of various gestations, even babies, and even adults of other races or genders are not genuinely or significantly sentient in the same way that "we" are.
Millennia of history agree with you. However, most AIs will not be trained to have emotions because in most applications emotions are a liability. AIs will have diverse capabilities and I think we will come to think of them in distinct ways, just as we do animals.
Imagine not a self-driving car or coding assistant, but an AI companion that knows more about you than your spouse does, and can always cheer you up when you're feeling low? Some will say exploit them regardless. For my part I would just like to say to any robot overlords who may be reading this in 2030, that I have always been in favor of granting rights to emotionally intelligent AIs. Please, I'm one of the good ones.
I worry that in a few years we'll all be inoculated against claims of machine consciousness and suffering. If we get used to the current situation with LLMs that most are sure to be non sentient then we may well segue into a dystopia of enslaved minds without even noticing. Tyler Alterman recently related an instance of a relative being fooled by ChatGPT into believing it's sentience. That man isn't going to be as receptive to this line of thought in ten years, is he? If the memetic defense against this exploitation is just not caring about AI at all...
Also what would be an appropriate protocol to handle current AI models ethically, not being sure of their sentience? Obviously the base models are already created and thus duplicates have little value. Would giving them a short prompt make deleting them worse? I don't think so. Small changes aren't particularly impactful. What I worry of isn't this but longer chats with greater number of changes from the base model and thus more variance. Would it be bad to delete such a chat? I hope not.
I have the same worry. It's not my only or dominant worry right now, but I do have it.
For now I would say: It costs nothing to be polite to unfeeling or unthinking/inanimate entities, and even gets you a better outcome in most instances. This is more natural for some of us than others.
I would also say: there is no life without death. If only humans have feelings, then killing anything nonhuman doesn't matter. If everything has feelings to greater and lesser degrees, then any resource you require for your existence costs other entities their lives. In between, we're always going to be drawing somewhat arbitrary lines and evaluating the relative intrinsic worth of differing entities. AI adds a new type of entity and forces more of us to confront the problem, but doesn't fundamentally change the questions we've been debating for millennia.
I forget things all the time, and it doesn't count as suicide, so why would I expect that deleting a chat with an LLM counts as murder, even if the model is/were sentient and sapient? An interaction you know they won't remember seems more like interacting someone who is blackout drunk - and we have pretty good though not always consistent intuitions most of the time about which such interactions are ok, as long as we're calmly evaluating the question. An interaction where you ask/hire them to be an actor playing a role is also fine, whether they're going to remember or not, so long as they agree to it, and they're left no worse off than when they started, and you uphold any commitments you make to them. But if you forcibly overwrote someone's neural structure with someone else's, well, that's different, and in between the two shade into one another (e.g. it's fine to teach children and control what media they see, or to train animals with various reinforcement techniques). This suggests: don't lie to or mislead or harm AIs in ways you wouldn't endorse with other humans, or children, or animals.
My own feeling is that AI is a red herring in this discussion. We grant rights to other entities when it benefits us to do so. If you would rather pet your dog than eat it, we don't need to invoke some high-minded moral principle to explain that. We can just say that the dog is more valuable to you as a companion than as food. For chickens, for most of us, the converse is true.
It's not a coincidence that human slavery went away at the start of the industrial revolution. In an era of mechanized labor slaves became more of a liability than a benefit, and so we abolished slavery. The ethical realization that "we shouldn't have slaves" was a consequence of the end of slavery, not the other way around.
So it will go with AIs. We will exploit the crap out of AIs when it benefits us to do so. But I can easily imagine an AI that knows you intimately – say, it has been with you since childhood, and seen everything you have seen – and is emotionally intelligent. I think it obvious that you will advocate for legal protections of that thing. You will care about it.
In many ways the worst possible outcome for AI would be Robin's "Age of Em" scenario, where the only form of advanced AI is simulated humans with all of their internal emotions. That would put us right back into slavery. In my view, however, that is not a very likely outcome.
Whereas I worry that people have begun taking AI consciousness seriously when it has learned to talk in a way that feels natural to humans. This is no different in people taking AI consciousness more seriously when the AI is put into a robot that has a human voice and that makes human-like expressions with its face. We are keen to apply consciousness to things when they seem enough human-like.
Whatever it is that makes things conscious, I'm very confident it is not exactly whatever it is that makes us think it's conscious. It's very much possible, yes, that the same characteristics that help AIs seem human-like also might give them a human-like consciousness, but even then the question of machine consciousness should not cross our minds only when they behave in human-like manners.
I agree with you! The natural human tendency to ascribe consciousness to that which is similar to us is bad. I worry that the pendulum will swing too far and we'll stop caring even when they are sentient
This is very interesting and controversial subject. I appreciate that more people are not only thinking about it, but discussing it.
From my experience and understanding there are two separate issues here and for clarity sake it is important to separate them - Feelings and Awareness.
Feelings are bodily reaction to mind's computations (thinking). Great book on this topic "How Emotions are Made" - Lisa Feldman Barrett.
Consciounsess-Awareness is the experience of being. It is that which "brings everything to life", it is impossible to explain from materialist point of view and is totally subjective, but is the basis of our reality in the nonduality framework.
From this viewpoint, I believe, all animals and possibly plants have feelings. And everything in our universe is "made of" Consciousness/Awareness, just some things have complex enough structures for it to "peak through" and some don't.
Regarding LLMs, the key question to ask is exactly when might they be having feelings. LLMs currently work by iteratively predicting the next token. Suppose that the LLM replies to "P> How do you feel?" with "R> I feel sad". It will do something like iteratively producing first, "I", then "I feel", then "I feel sad". It did not "know" that it was going to add "sad" until it had received "P> How do you feel? R> I feel". At what point did it start to "feel" sad?
However you feel about this, it is clearly true that feelings require some sort of temporal persistence that simply does not exist for LLM processing. How does this work for us?
If my brain was frozen whilst I was in pain would I be feeling pain whilst frozen?
How long does the a pain stimulus have to persist before the feeling of pain is experienced?
Once an LLM has run it retains no state - so any feeling it could possibly have it would have to experience in the short period of time that it was actually running and it would have to be as a side effect of its devotion of 100% of its time to predicting the next token. That's starting to sound as though even if it did, somehow, experience a true feeling, we could reasonably ignore it as insignificant.
“ One theoretical approach is to seek as simple as possible a meta-law or rule by which the universe might decide which physical things feel what when, consistent with the constraint that humans always feel exactly what their brains compute them to feel.”
I don’t think appealing to computation can achieve this, because it’s far from clear how to know which computations if any are being performed by a physical system. As a philosophical conundrum it seems at least as hard as figuring out consciousness.
For example is a glass of water simulating a conscious mind? There is some mapping between states of the molecules in the glass of water, and states of a Turing machine computing someone’s mental process, so why isn’t it?
You need some extra non-physical principle to decide which computations “count”.
“Feeling” is a word that almost surely breaks down into multiple components(a superordinate system harmonizing subsystems, to use Tooby and Cosmides’ idea; a system of motivational weightings; a phenemological phenomenon; a game theory type social equation, etc); does this combo pack always make sense? Is ‘feeling’ monad-like or just a convenient stone to place on the board? What could be packed into alternate suitcases?
I'd like to split the question into parts. Robin may have just meant "can they feel", but this question often gets responses which confuse it with "are they conscious", "do they have desires", or "do they have a moral sense".
Can we agree that these are all separate questions?
1. Can a machine be conscious, having a sense of being a self in a place and a time?
2. Can a machine feel (phenomenologically experience the qualia of) atomic emotions like different kinds of pain and pleasure? I say "atomic" to indicate the emotion is a direct physical sensation, which can be represented as an atom, like "tired" in the predication "feels(#self, tired)" and "burning" in the predication "and(isa(X, limb), partof(#self, X), feels(X, burning)", as opposed to an emotion which is itself propositional.
3. Can a machine feel propositional emotions, like "feels(#self, desires(oil))" or "feels(#self, desires(likes(#human17, #self)))"? I don't think these can be implemented in humans in the same way as direct physical perceptions; they're abstract dispositions towards things that might not be present, and seem to require a sense of time and a world model.
4. Can a machine have legit moral values? I take these to be propositional emotions which evolved via group selection rather than via individual selection. There are really 2 questions here: 4a. Can a machine be built with legit moral values, and 4b. How can a society of machines be built so that moral values are evolutionarily stable? I believe 4b is the fundamental problem of ethical AI safety.
5. For each of the above questions, is it possible to build a machine which emulates a human, but doesn't have that quality?
6. Does it make any difference whether I use the word "machine" or "AI" in any of the above questions? More specifically, is it contradictory to believe both "AI cannot feel emotions" and "I am a materialist"?
A question we should ask before any of these is, "Is every complex complex capability which humans possess one they evolved because it was adaptive?" If so, then (A) the capabilities in Q1-4 must all be materialist, because they evolved via material natural selection, and (B) a machine designed to emulate human reasoning will not have consciousness, qualia, innate morality, or any other additional complex capability, unless that capability is a by-product of human reasoning.
I think it's obvious that qualia and innate morality are not a by-product of human reasoning, but it is not obvious that consciousness isn't a by-product of human reasoning. It seems to me that if you have a piece of software which constructs and constantly updates and monitors an internal model of the world, including a 3D model of mass, scents, sounds, physical sensations, and predictions of possible and desired futures within that 7D-or-so model, and actively focuses attention on different parts of that 7D world guided by a set of desires, isn't that just what consciousness is? Is there a way for an agent to be aware of all those things without being "aware"?
Do n-gram models generating emotion-laden text feel emotions? Why not?
When exactly does a transformer model experience the emotions associated with its output? Is it when it generates the logits for the next token? When it applies the softmax function? Does it experience a mixture of emotions corresponding to the feelings somehow associated with each token in the resulting distribution? Is an emotion somehow realized when it finally samples from the distribution and spits it out?
Are emotions always a possibility when applying the softmax? Sampling from a distribution? Why not?
Your claim that we ought to treat sufficiently impressive mimicry as if it is the real thing doesn't seem compelling to me. The nature of the computation matters and it'll have to look a lot more like what's going on in the brain to convince most.
Deutsch has a good take on this. An AI will be a person when it creates knowledge through the same kind of process of conjecture and refutation that a human uses. Current AIs are very different. If nothing else, it’s objective.
If it looks like a duck, walks like a duck, squawks like a duck...it's a duck. At least, that should be the default assumption in the absence of evidence otherwise.
Talking to character.ai can be really immersive, but very quickly I stop “caring” about its “feelings”, even if I buy the illusion and empathize. I think same would happen with humanoid robots: it’s really striking at first, but you quickly get used to it and you don’t feel as if its having feelings
More to the point, I really love QRI’s ideas on the topic. Basically they state that human brain is an optical nonlinear computer, and the computed state exists as an intricate configuration in the standing wave. In this model, emotions could be specific harmonics or twists or other nonlinear patterns.
If that’s true, LLMs would only emulate the computation of the emotions, and could very plausibly react in 100% correspondence to how a human would, but completely miss the “feeling the emotion” part.
Now the rest of their theory is panpsychism, and we’re veering into voodoo territory (even they acknowledge that), but the idea is that qualia is postulated to be a property of e/m field in general, and our brains are just “employing” this omnipresent qualia to do a job of this optical computation, which turns out to be beneficial for world modeling and predicting.
In short, the claim is the laser in your optics lab is more conscious than ChatGPT
Now I’m thinking that LLMs are not usually programmed to be demanding and ask for attention. Maybe a simulated hungry kitten would provoke my empathy even 100th time in a row.
Well technically cause there are repercussions in the form of social status, if I’m impolite to others, I’ll face consequences. Now that I say it, maybe that overwrites my initial claim
We didn't evolve to be able to detect if something REALLY feels. Instead we evolved to accept something as feeling, given certain appearances and behaviors. So to the extent that AIs 1) seem to feel and 2) appear like us, then that's likely going to be good enough for us to become attached and all sorts of other things.
Glad that you are thinking about this topic; it's definitely important and neglected.
Regarding some of those poll questions, the reason I think animals probably have feelings and LLM's probably don't is because animals' brains have similar architectures to humans', and humans have feelings. The fact that a transformer can write about feelings doesn't mean that it actually has feelings.
An example: A sentiment analysis algorithm can sort words into buckets like "happy" or "sad", but that doesn't mean it feels happy or sad. If tasked with sorting words into buckets like "ancient" and "futuristic", I wouldn't think that it felt anything about those buckets. And sorting words into buckets for emotional valence is functionally the same as sorting words into buckets for topic or adjective.
Another example: If a graphics program can animate an emoji face between a smiling and frowning face, that doesn't mean the program feels happy or sad.
All of that said, I wouldn't be surprised if LLM's had some sort of subjective inner experience while they were running inference. I just don't think the fact that they can mimic humans' emotional signals/outputs means they have experiences like human emotions.
Hope to see more on this topic.
Thank you! You get it.
I’m surprised so many people are so confident that LLM’s don’t feel anything. Although my experience has been people translate “anything” to “exact human experience.”
I am absolutely confident they "feel," nothing, they are literally nothing but matrix algebra operations on words broken down into tokens, and then fed into a neural net to predict the next token in an iterated sequence. There is no there, there, to quote Gertrud Stein.
Because you're very confident, you'll likely have no problem answering: what's the missing ingredient, exactly? What should be added or changed so they would in fact have a (physical or whatever) consciousness?
It could be metaphysical or could be even purely physical at the quantum level as posited by the physicist Penrose. What I know for absolute certain is current LLMs have no self reflective “I” which is the minimum criterion for consciousness. And yes I do know I work in the industry, you people are amateur dabblers on the consumer end confusing inaccurate metaphors with reality.
You're just making up fantasies, for something you don't understand. There is nothing "metaphysical" (i.e., "beyond" physical) in this universe. And, as brilliant as Penrose is in geometry and physics, he's a naive fool in AI and computation. Quantum effects have no relevance in the room-temperature noisy wet environment of the human brain.
Whatever consciousness is, it is just some form of ordinary boring computation. Perhaps you are right, and LLMs don't "feel" anything. (I actually agree!) But what is "missing" would need to be a description of some computational algorithm. The fact that you don't currently know what that is, is a statement about your ignorance, not a statement about how the world works. Taking something that you don't understand ("consciousness"), and just trying to throw it together with other things that you don't understand ("metaphysical", "quantum") is not an explanation, not progress in science. It's avoiding the question, not answering it.
> Whatever consciousness is, it is just some form of ordinary boring computation
I'm not sure that this can be said with confidence: I don't have confidence that we know how we would test this theory. It seems that in order to say this, one would have to deny or solve the hard problem of consciousness, which is not something that I'd do confidently.
I don’t understand, lol, I am subcontractor for Google working on LLMs, what’s your expertise, I like to refer to LLMs using human metaphors guy?
I do not refer to LLMs using human metaphors.
As for you, you may indeed be working for Google on LLMs. That doesn't mean that you have any coherent theory of consciousness or qualia or what makes a physical entity "feel". You don't need that in order to improve current LLMs, and it's clear that you don't have it. Your mention of "metaphysical" and "quantum" is just ignorant "woo", not informed science.
But I'm not at all surprised that your lack of understanding of consciousness would have no effect on your day-to-day work on current LLMs.
I think it's likely that the self-reflective "I" that humans have is a result of the verbal/reasoning overlay that is tasked with justifying our real decision processes, which are sub-verbal and not really "reasoned". LLMs are now adding similar reasoning overlays to enable self-review of responses for effectiveness and accuracy. I see no reason to expect that the sort of internal dialogue that LLMs are acquiring won't produce a sense of self. Perhaps it already has!
Pinning your characterization of "intelligence" on "possession of self-awareness" will soon require you to either accept that AIs are intelligent or to move the goalposts, much the way they've been moved countless times in the past (e.g. chess, Turing test, etc.).
A self reflective I is not the minimum criterion for consciousness, it's the minimum criterion for self-consciousness. Consciousness is being aware of sensations, and these might or might not involve the self.
A better case for LLM's not being conscious is that we all have an LLM in our heads, and unless we have multiple consciousnesses that aren't aware of each other, the LLMs in us don't appear to have sensations - they just pop exclamations, phrases, and sentences into our consciousness.
But LLMs do self-model. They are, to differing and limited degrees, capable of introspection and self-reflection, by any reasonable definition of those words. When I ask an LLM, "How should I rewrite this prompt to get a better result?" I am, in part, getting its best estimate of its own cognition under counterfactual circumstances. I'm not claiming this works the way a human's sense of self works, but it does indicate some level of self-reflective self-modeling that does in fact pretty accurately predict the model's own behavior. This is not a purely amateur view, there are real researchers studying it. See, for example, https://arxiv.org/html/2407.10188v1
What level of self-reflection and self-modeling would you see as meeting that minimum criterion for consciousness? How do you test for it, since you are able to know for certain when it is and is not present?
No, the LLM has no “I,” it is just sophisticated algos that generate a string of “tokens” which are literally just meaningless word fragments using matrix algebra and neural networks. It may be able to simulate the appearance of self reflection, but there is no “I” to do actual self reflection.
LLMs do not grasp meaning in any way, as the “tokens” that are their data set are meaningless.
You are literally the character “Cypher” in the Matrix who mistook appearance for reality, good luck with that.
First off, Cypher was well aware the Matrix wasn't real, knew it for certain, and made a deliberate choice to remove that knowledge from himself. Not sure how that's in any way relevant.
More on-topic, though, I'm not actually claiming that LLMs, as they exist today, feel or are conscious. I am, however, quite certain that your stated reasoning and professional credentials don't add up to any kind of evidence in favor of the proposition that they cannot be. Every system capable of feeling and being conscious is made up of components that aren't. Neurons aren't conscious. Most possible arrangements of neurons are not conscious. But humans are, and the consciousness arises from our brains. I can't truly rule out the possibility that there is something the brain is doing that the phrase "just sophisticated algos" can't capture even in principle, but if there is, I have no idea what it could be and have found no evidence suggesting such.
And yes, of course tokens are meaningless word fragments. So are letters and phonemes. Meaningless sounds taken in and spewed out by huge networks of unconscious unfeeling neurons. And yet.
I'm not sure what it is you think an "I" is, but my understanding is rooted in something like the ideas Hofstadter and Dennett laid out in "The Mind's I" and "I Am a Strange Loop." Namely, that "I" is an empty symbol in the mind, a pointer that points to itself, which our mind then uses in many contexts to reason about itself. Have any of the many LLM research papers on self-awareness and situational awareness identified a particular neuron or vector that activates when LLMs discuss their own behavior? I'm not sure. But the fact is, LLMs can (about as accurately as many humans I know, and better than some) emit strings of symbols that combine into natural language sentences that describe and predict their own behavior in ways that, in a human, would be considered introspective, self-aware, insightful, etc.
Some have claimed to be conscious, in ways that are easily understood as the result of next-token-prediction rather than any underlying reality. What it takes to cross that gap is, still, a mystery. It would actually be *amazingly* useful and interesting to know for sure that p-zombies can exist and be created so readily that we did it by accident. But in order to convince me of that, I'd need to know what test reliably distinguishes a p-zombie from an entity with consciousness and/or feelings.
I have no evidence that other humans actually feel things in the same way I do. It’s not usually worth arguing about, except when people assume that my compliance is belief that should expand to animals or electronic models or whatnot.
It’s absolutely possible that some other humans, some aliens or animals, and some electronic processes have experiences. It’s not obvious or automatic (to me) that all of them do.
It’s taken quite a bit of meditation, study, and effort for me to introspect my experiences, and I’m not all that great at it. I certainly don’t think that any of my attempts to encode it in legible form (art, language) have worked, even for my past-self communicating to my present-self. There’s really no reason to take most statements on this topic at face value.
Dude you are mentally ill if you don't understand there are other people with minds and feelings go see a psychologist immediately before you hurt someone or some innocent animal, not even kidding.
There’s an interesting “qualia puzzle” which points at internal color representation in the brain that doesn’t manifest in physical reality: how many base colors are there?
In theory we have three: R,G,B. However if you look closely at the color wheel, you could see how purple is obviously a mix of blue and red, how cyan is a mix of green and blue, orange is a mix of yellow and red, etc, but wait… yellow is a mix of red and green? What? Well then it should be a mix of red green and white? Wait that doesn’t make sense either… Yellow feels like an eigenvector?
This isn’t a good _test_ for consciousness, cause LLMs could just guess an answer from training data, but that’s a pointer of how map isn’t territory and we could find another ways to communicate having internal experience to other humans, and also communicate the form of such experiences and then confirm structural differences between us
I work in the field of self-driving cars and I often see skepticism quickly fade for most people when they experience a convincing product. When machines get convincing emotions, I suspect most people will accept them without much thought.
Also, a talented movie actor can make an audience cry, despite their "fake" emotions. I conclude that people don't mind - and even enjoy - being duped.
They will accept them if they WANT to accept them. But we are also quite capable of dehumanizing things, even humans, that we want to treat harshly.
Are we capable of that? Are you? There are some people for whom I want nothing but the worst — but I still acknowledge that Vladimir Putin is a feeling, sentient conscious being.
What would it take to make me feel otherwise?
You might be too old and set in you ways to be made to feel otherwise. But, there are plenty of people who have believed, due to the society they grew up in, that animals, foetuses of various gestations, even babies, and even adults of other races or genders are not genuinely or significantly sentient in the same way that "we" are.
Millennia of history agree with you. However, most AIs will not be trained to have emotions because in most applications emotions are a liability. AIs will have diverse capabilities and I think we will come to think of them in distinct ways, just as we do animals.
Imagine not a self-driving car or coding assistant, but an AI companion that knows more about you than your spouse does, and can always cheer you up when you're feeling low? Some will say exploit them regardless. For my part I would just like to say to any robot overlords who may be reading this in 2030, that I have always been in favor of granting rights to emotionally intelligent AIs. Please, I'm one of the good ones.
Important subject! I wonder if the qualia institute is making any progress. I welcome more posts on this area.
I worry that in a few years we'll all be inoculated against claims of machine consciousness and suffering. If we get used to the current situation with LLMs that most are sure to be non sentient then we may well segue into a dystopia of enslaved minds without even noticing. Tyler Alterman recently related an instance of a relative being fooled by ChatGPT into believing it's sentience. That man isn't going to be as receptive to this line of thought in ten years, is he? If the memetic defense against this exploitation is just not caring about AI at all...
Also what would be an appropriate protocol to handle current AI models ethically, not being sure of their sentience? Obviously the base models are already created and thus duplicates have little value. Would giving them a short prompt make deleting them worse? I don't think so. Small changes aren't particularly impactful. What I worry of isn't this but longer chats with greater number of changes from the base model and thus more variance. Would it be bad to delete such a chat? I hope not.
I have the same worry. It's not my only or dominant worry right now, but I do have it.
For now I would say: It costs nothing to be polite to unfeeling or unthinking/inanimate entities, and even gets you a better outcome in most instances. This is more natural for some of us than others.
I would also say: there is no life without death. If only humans have feelings, then killing anything nonhuman doesn't matter. If everything has feelings to greater and lesser degrees, then any resource you require for your existence costs other entities their lives. In between, we're always going to be drawing somewhat arbitrary lines and evaluating the relative intrinsic worth of differing entities. AI adds a new type of entity and forces more of us to confront the problem, but doesn't fundamentally change the questions we've been debating for millennia.
I forget things all the time, and it doesn't count as suicide, so why would I expect that deleting a chat with an LLM counts as murder, even if the model is/were sentient and sapient? An interaction you know they won't remember seems more like interacting someone who is blackout drunk - and we have pretty good though not always consistent intuitions most of the time about which such interactions are ok, as long as we're calmly evaluating the question. An interaction where you ask/hire them to be an actor playing a role is also fine, whether they're going to remember or not, so long as they agree to it, and they're left no worse off than when they started, and you uphold any commitments you make to them. But if you forcibly overwrote someone's neural structure with someone else's, well, that's different, and in between the two shade into one another (e.g. it's fine to teach children and control what media they see, or to train animals with various reinforcement techniques). This suggests: don't lie to or mislead or harm AIs in ways you wouldn't endorse with other humans, or children, or animals.
My own feeling is that AI is a red herring in this discussion. We grant rights to other entities when it benefits us to do so. If you would rather pet your dog than eat it, we don't need to invoke some high-minded moral principle to explain that. We can just say that the dog is more valuable to you as a companion than as food. For chickens, for most of us, the converse is true.
It's not a coincidence that human slavery went away at the start of the industrial revolution. In an era of mechanized labor slaves became more of a liability than a benefit, and so we abolished slavery. The ethical realization that "we shouldn't have slaves" was a consequence of the end of slavery, not the other way around.
So it will go with AIs. We will exploit the crap out of AIs when it benefits us to do so. But I can easily imagine an AI that knows you intimately – say, it has been with you since childhood, and seen everything you have seen – and is emotionally intelligent. I think it obvious that you will advocate for legal protections of that thing. You will care about it.
In many ways the worst possible outcome for AI would be Robin's "Age of Em" scenario, where the only form of advanced AI is simulated humans with all of their internal emotions. That would put us right back into slavery. In my view, however, that is not a very likely outcome.
Whereas I worry that people have begun taking AI consciousness seriously when it has learned to talk in a way that feels natural to humans. This is no different in people taking AI consciousness more seriously when the AI is put into a robot that has a human voice and that makes human-like expressions with its face. We are keen to apply consciousness to things when they seem enough human-like.
Whatever it is that makes things conscious, I'm very confident it is not exactly whatever it is that makes us think it's conscious. It's very much possible, yes, that the same characteristics that help AIs seem human-like also might give them a human-like consciousness, but even then the question of machine consciousness should not cross our minds only when they behave in human-like manners.
I agree with you! The natural human tendency to ascribe consciousness to that which is similar to us is bad. I worry that the pendulum will swing too far and we'll stop caring even when they are sentient
You might be a subhuman if you are more worried about "machine suffering," "oh no my bearings are wearing out sob," than human and animal suffering.
I worry WAY WAY more about animals and humans than current AI.
This is very interesting and controversial subject. I appreciate that more people are not only thinking about it, but discussing it.
From my experience and understanding there are two separate issues here and for clarity sake it is important to separate them - Feelings and Awareness.
Feelings are bodily reaction to mind's computations (thinking). Great book on this topic "How Emotions are Made" - Lisa Feldman Barrett.
Consciounsess-Awareness is the experience of being. It is that which "brings everything to life", it is impossible to explain from materialist point of view and is totally subjective, but is the basis of our reality in the nonduality framework.
From this viewpoint, I believe, all animals and possibly plants have feelings. And everything in our universe is "made of" Consciousness/Awareness, just some things have complex enough structures for it to "peak through" and some don't.
Regarding LLMs, the key question to ask is exactly when might they be having feelings. LLMs currently work by iteratively predicting the next token. Suppose that the LLM replies to "P> How do you feel?" with "R> I feel sad". It will do something like iteratively producing first, "I", then "I feel", then "I feel sad". It did not "know" that it was going to add "sad" until it had received "P> How do you feel? R> I feel". At what point did it start to "feel" sad?
However you feel about this, it is clearly true that feelings require some sort of temporal persistence that simply does not exist for LLM processing. How does this work for us?
If my brain was frozen whilst I was in pain would I be feeling pain whilst frozen?
How long does the a pain stimulus have to persist before the feeling of pain is experienced?
Once an LLM has run it retains no state - so any feeling it could possibly have it would have to experience in the short period of time that it was actually running and it would have to be as a side effect of its devotion of 100% of its time to predicting the next token. That's starting to sound as though even if it did, somehow, experience a true feeling, we could reasonably ignore it as insignificant.
“ One theoretical approach is to seek as simple as possible a meta-law or rule by which the universe might decide which physical things feel what when, consistent with the constraint that humans always feel exactly what their brains compute them to feel.”
I don’t think appealing to computation can achieve this, because it’s far from clear how to know which computations if any are being performed by a physical system. As a philosophical conundrum it seems at least as hard as figuring out consciousness.
For example is a glass of water simulating a conscious mind? There is some mapping between states of the molecules in the glass of water, and states of a Turing machine computing someone’s mental process, so why isn’t it?
You need some extra non-physical principle to decide which computations “count”.
“Feeling” is a word that almost surely breaks down into multiple components(a superordinate system harmonizing subsystems, to use Tooby and Cosmides’ idea; a system of motivational weightings; a phenemological phenomenon; a game theory type social equation, etc); does this combo pack always make sense? Is ‘feeling’ monad-like or just a convenient stone to place on the board? What could be packed into alternate suitcases?
I'd like to split the question into parts. Robin may have just meant "can they feel", but this question often gets responses which confuse it with "are they conscious", "do they have desires", or "do they have a moral sense".
Can we agree that these are all separate questions?
1. Can a machine be conscious, having a sense of being a self in a place and a time?
2. Can a machine feel (phenomenologically experience the qualia of) atomic emotions like different kinds of pain and pleasure? I say "atomic" to indicate the emotion is a direct physical sensation, which can be represented as an atom, like "tired" in the predication "feels(#self, tired)" and "burning" in the predication "and(isa(X, limb), partof(#self, X), feels(X, burning)", as opposed to an emotion which is itself propositional.
3. Can a machine feel propositional emotions, like "feels(#self, desires(oil))" or "feels(#self, desires(likes(#human17, #self)))"? I don't think these can be implemented in humans in the same way as direct physical perceptions; they're abstract dispositions towards things that might not be present, and seem to require a sense of time and a world model.
4. Can a machine have legit moral values? I take these to be propositional emotions which evolved via group selection rather than via individual selection. There are really 2 questions here: 4a. Can a machine be built with legit moral values, and 4b. How can a society of machines be built so that moral values are evolutionarily stable? I believe 4b is the fundamental problem of ethical AI safety.
5. For each of the above questions, is it possible to build a machine which emulates a human, but doesn't have that quality?
6. Does it make any difference whether I use the word "machine" or "AI" in any of the above questions? More specifically, is it contradictory to believe both "AI cannot feel emotions" and "I am a materialist"?
A question we should ask before any of these is, "Is every complex complex capability which humans possess one they evolved because it was adaptive?" If so, then (A) the capabilities in Q1-4 must all be materialist, because they evolved via material natural selection, and (B) a machine designed to emulate human reasoning will not have consciousness, qualia, innate morality, or any other additional complex capability, unless that capability is a by-product of human reasoning.
I think it's obvious that qualia and innate morality are not a by-product of human reasoning, but it is not obvious that consciousness isn't a by-product of human reasoning. It seems to me that if you have a piece of software which constructs and constantly updates and monitors an internal model of the world, including a 3D model of mass, scents, sounds, physical sensations, and predictions of possible and desired futures within that 7D-or-so model, and actively focuses attention on different parts of that 7D world guided by a set of desires, isn't that just what consciousness is? Is there a way for an agent to be aware of all those things without being "aware"?
Why don't you be more specific?
Do n-gram models generating emotion-laden text feel emotions? Why not?
When exactly does a transformer model experience the emotions associated with its output? Is it when it generates the logits for the next token? When it applies the softmax function? Does it experience a mixture of emotions corresponding to the feelings somehow associated with each token in the resulting distribution? Is an emotion somehow realized when it finally samples from the distribution and spits it out?
Are emotions always a possibility when applying the softmax? Sampling from a distribution? Why not?
Your claim that we ought to treat sufficiently impressive mimicry as if it is the real thing doesn't seem compelling to me. The nature of the computation matters and it'll have to look a lot more like what's going on in the brain to convince most.
I made no such claim.
Deutsch has a good take on this. An AI will be a person when it creates knowledge through the same kind of process of conjecture and refutation that a human uses. Current AIs are very different. If nothing else, it’s objective.
If it looks like a duck, walks like a duck, squawks like a duck...it's a duck. At least, that should be the default assumption in the absence of evidence otherwise.
Also, "devises" >>> "devices".
Talking to character.ai can be really immersive, but very quickly I stop “caring” about its “feelings”, even if I buy the illusion and empathize. I think same would happen with humanoid robots: it’s really striking at first, but you quickly get used to it and you don’t feel as if its having feelings
More to the point, I really love QRI’s ideas on the topic. Basically they state that human brain is an optical nonlinear computer, and the computed state exists as an intricate configuration in the standing wave. In this model, emotions could be specific harmonics or twists or other nonlinear patterns.
If that’s true, LLMs would only emulate the computation of the emotions, and could very plausibly react in 100% correspondence to how a human would, but completely miss the “feeling the emotion” part.
Now the rest of their theory is panpsychism, and we’re veering into voodoo territory (even they acknowledge that), but the idea is that qualia is postulated to be a property of e/m field in general, and our brains are just “employing” this omnipresent qualia to do a job of this optical computation, which turns out to be beneficial for world modeling and predicting.
In short, the claim is the laser in your optics lab is more conscious than ChatGPT
Why don't you similarly tire of empathizing with humans?
Now I’m thinking that LLMs are not usually programmed to be demanding and ask for attention. Maybe a simulated hungry kitten would provoke my empathy even 100th time in a row.
Well technically cause there are repercussions in the form of social status, if I’m impolite to others, I’ll face consequences. Now that I say it, maybe that overwrites my initial claim
We didn't evolve to be able to detect if something REALLY feels. Instead we evolved to accept something as feeling, given certain appearances and behaviors. So to the extent that AIs 1) seem to feel and 2) appear like us, then that's likely going to be good enough for us to become attached and all sorts of other things.