This happened to me the other day with Jippity. It outright lied to me:
“You’re absolutely right. Although I don’t have access to the earlier parts of the conversation”.
So it says that I was right in a particular statement, but didn’t actually know what I said. So I said to it, you just lied. It kept saying variations of:
“I didn’t lie intentionally”
“I understand why it seems that way”
“I wasn’t misleading you”
etc
It flat out lied and tried to gaslight me into thinking I was in the wrong for taking that way.
It didn’t lie to you or gaslight you because those are things that a person with agency does. Someone who lies to you makes a decision to deceive you for whatever reason they have. Someone who gaslights you makes a decision to behave like the truth as you know it is wrong in order to discombobulate you and make you question your reality.
The only thing close to a decision that LLMs make is: what text can I generate that statistically looks similar to all the other text that I’ve been given. The only reason they answer questions is because in the training data they’ve been provided, questions are usually followed by answers.
It’s not apologizing you to, it knows from its training data that sometimes accusations are followed by language that we interpret as an apology, and sometimes by language that we interpret as pushing back. It regurgitates these apologies without understanding anything, which is why they seem incredibly insincere - it has no ability to be sincere because it doesn’t have any thoughts.
There is no thinking. There are no decisions. The more we anthropomorphize these statistical text generators, ascribing thoughts and feelings and decision making to them, the less we collectively understand what they are, and the more we fall into the trap of these AI marketers about how close we are to truly thinking machines.
The only thing close to a decision that LLMs make is
That’s not true. An “if statement” is literally a decision tree.
The only reason they answer questions is because in the training data they’ve been provided
This is technically true for something like GPT-1. But it hasn’t been true for the models trained in the last few years.
it knows from its training data that sometimes accusations are followed by language that we interpret as an apology, and sometimes by language that we interpret as pushing back. It regurgitates these apologies without understanding anything, which is why they seem incredibly insincere
It has a large amount of system prompts that alter default behaviour in certain situations. Such as not giving the answer on how to make a bomb. I’m fairly certain there are catches in place to not be overly apologetic to minimize any reputation harm and to reduce potential “liability” issues.
And in that scenario, yes I’m being gaslite because a human told it to.
There is no thinking
Partially agree. There’s no “thinking” in sentient or sapient sense. But there is thinking in the academic/literal definition sense.
There are no decisions
Absolutely false. The entire neural network is billions upon billions of decision trees.
The more we anthropomorphize these statistical text generators, ascribing thoughts and feelings and decision making to them, the less we collectively understand what they are
I promise you I know very well what LLMs and other AI systems are. They aren’t alive, they do not have human or sapient level of intelligence, and they don’t feel. I’ve actually worked in the AI field for a decade. I’ve trained countless models. I’m quite familiar with them.
But “gaslighting” is a perfectly fine description of what I explained. The initial conditions were the same and the end result (me knowing the truth and getting irritated about it) were also the same.
The only thing close to a decision that LLMs make is
That’s not true. An “if statement” is literally a decision tree.
If you want to engage in a semantically argument, then sure, an “if statement” is a form of decision. This is a worthless distinction that has nothing to do with my original point and I believe you’re aware of that so I’m not sure what this adds to the actual meat of the argument?
The only reason they answer questions is because in the training data they’ve been provided
This is technically true for something like GPT-1. But it hasn’t been true for the models trained in the last few years.
Okay, what was added to models trained in the last few years that makes this untrue? To the best of my knowledge, the only advancements have involved:
- Pre-training, which involves some additional steps to add to or modify the initial training data
- Fine-tuning, which is additional training on top of an existing model for specific applications.
- Reasoning, which to the best of my knowledge involves breaking the token output down into stages to give the final output more depth.
- “More”. More training data, more parameters, more GPUs, more power, etc.
I’m hardly an expert in the field, so I could have missed plenty, so what is it that makes it “understand” that a question needs to be answered that doesn’t ultimately go back to the original training data? If I feed it training data that never involves questions, then how will it “know” to answer that question?
it knows from its training data that sometimes accusations are followed by language that we interpret as an apology, and sometimes by language that we interpret as pushing back. It regurgitates these apologies without understanding anything, which is why they seem incredibly insincere
It has a large amount of system prompts that alter default behaviour in certain situations. Such as not giving the answer on how to make a bomb. I’m fairly certain there are catches in place to not be overly apologetic to minimize any reputation harm and to reduce potential “liability” issues.
System prompts are literally just additional input that is “upstream” of the actual user input, and I fail to see how that changes what I said about it not understanding what an apology is, or how it can be sincere when the LLM is just spitting out words based on their statistical relation to one another?
An LLM doesn’t even understand the concept of right or wrong, much less why lying is bad or when it needs to apologize. It can “apologize” in the sense that it has many examples of apologies that it can synthesize into output when you request one, but beyond that it’s just outputting text. It doesn’t have any understanding of that text.
And in that scenario, yes I’m being gaslite because a human told it to.
Again, all that’s doing is adding additional words that can be used in generating output. It’s still just generating text output based on text input. That’s it. It has to know it’s lying or being deceitful in order to gaslight you. Does the text resemble something that can be used to gaslight you? Sure. And if I copy and pasted that from ChatGPT that’s what I’d be doing, but an LLM doesn’t have any real understanding of what it’s outputting so saying that there’s any intent to do anything other than generate text based on other text is just nonsense.
There is no thinking
Partially agree. There’s no “thinking” in sentient or sapient sense. But there is thinking in the academic/literal definition sense.
Care to expand on that? Every definition of thinking that I find involves some kind of consideration or reflection, which I would argue that the LLM is not doing, because it’s literally generating output based on a complex system of weighted parameters.
If you want to take the simplest definition of “well, it’s considering what to output and therefore that’s thought”, then I could argue my smart phone is “thinking” because when I tap on a part of the screen it makes decisions about how to respond. But I don’t think anyone would consider that real “thought”.
There are no decisions
Absolutely false. The entire neural network is billions upon billions of decision trees.
And a logic gate “decides” what to output. And my lightbulb “decides” whether or not to light up based on the state of the switch. And my alarm “decides” to go off based on what time I set it for last night.
My entire point was to stop anthropomorphizing LLMs by describing what they do as “thought”, and that they don’t make “decisions” in the same way humans do. If you want to use definitions that are overly broad just to say I’m wrong, fine, that’s your prerogative, but it has nothing to do with the idea I was trying to communicate.
The more we anthropomorphize these statistical text generators, ascribing thoughts and feelings and decision making to them, the less we collectively understand what they are
I promise you I know very well what LLMs and other AI systems are. They aren’t alive, they do not have human or sapient level of intelligence, and they don’t feel. I’ve actually worked in the AI field for a decade. I’ve trained countless models. I’m quite familiar with them.
Cool.
But “gaslighting” is a perfectly fine description of what I explained. The initial conditions were the same and the end result (me knowing the truth and getting irritated about it) were also the same.
Sure, if you wanna ascribe human terminology to what marketing companies are calling “artificial intelligence” and further reinforcing misconceptions about how LLMs work, then yeah, you can do that. If you care about people understanding that these algorithms aren’t actually thinking in the same way that humans do, and therefore believing many falsehoods about their capabilities, like I do, then you’d use different terminology.
It’s clear that you don’t care about that and will continue to anthropomorphize these models, so… I guess I’m done here.
That’s because they aren’t “aware” of anything.
This Nobel Prize winner and subject matter expert takes the opposite view
Interesting talk but the number of times he completely dismisses the entire field of linguists kind of makes me think he’s being disingenuous about his familiarity with it.
For one, I think he is dismissing holotes, the concept of “wholeness.” That when you cut something apart to it’s individual parts, you lose something about the bigger picture. This deconstruction of language misses the larger picture of the human body as a whole, and how every part of us, from our assemblage of organs down to our DNA, impact how we interact with and understand the world. He may have a great definition of understanding but it still sounds (to me) like it’s potentially missing aspects of human/animal biologically based understanding.
For example, I have cancer, and about six months before I was diagnosed, I had begun to get more chronically depressed than usual. I felt hopeless and I didn’t know why. Surprisingly, that’s actually a symptom of my cancer. What understanding did I have that changed how I felt inside and how I understood the things around me? Suddenly I felt different about words and ideas, but nothing had changed externally, something had change internally. The connections in my neural network had adjusted, the feelings and associations with words and ideas was different, but I hadn’t done anything to make that adjustment. No learning or understanding had happened. I had a mutation in my DNA that made that adjustment for me.
Further, I think he’s deeply misunderstanding (possibly intentionally?) what linguists like Chomsky are saying when they say humans are born with language. They mean that we are born with a genetic blueprint to understand language. Just like animals are born with a genetic blueprint to do things they were never trained to do. Many animals are born and almost immediately stand up to walk. This is the same principle. There are innate biologically ingrained understandings that help us along the path to understanding. It does not mean we are born understanding language as much as we are born with the building blocks of understanding the physical world in which we exist.
Anyway, interesting talk, but I immediately am skeptical of anyone who wholly dismisses an entire field of thought so casually.
For what it’s worth, I didn’t downvote you and I’m sorry people are doing so.
I am not a linguist but the deafening silence from Chomsky and his defenders really does demand being called out.
Syntactical models of language have been completely crushed by statistics-at-scale via neural nets. But linguists have not rejected the broken model.
The same thing happened with protein folding – researchers who spent the last 25 years building complex quantum mechanical/electrostatic models of protein structure suddenly saw AlphaFold completely crush prior methods. The difference is, bioinformatics researchers have already done a complete about-face and are taking the new AI tools and running with them.
People really do not like seeing opposing viewpoints, eh? There’s disagreeing, and then there’s downvoting to oblivion without even engaging in a discussion, haha.
Even if they’re probably right, in such murky uncertain waters where we’re not experts, one should have at least a little open mind, or live and let live.
It’s like talking with someone who thinks the Earth is flat. There isn’t anything to discuss. They’re objectively wrong.
Humans like to anthropomorphize everything. It’s why you can see a face on a car’s front grille. LLMs are ultra advanced pattern matching algorithms. They do not think or reason or have any kind of opinion or sentience, yet they are being utilized as if they do. Let’s see how it works out for the world, I guess.
I think so too, but I am really curious what will happen when we give them “bodies” with sensors so they can explore the world and make individual “experiences”. I could imagine they would act much more human after a while and might even develop some kind of sentience.
Of course they would also need some kind of memory and self-actualization processes.
Interaction with the physical world isn’t really required for us to evaluate how they deal with ‘experiences’. They have in principle access to all sorts of interesting experiences in the online data. Some models have been enabled to fetch internet data and add them to the prompt to help synthesize an answer.
One key thing is they don’t bother until direction tells them. They don’t have any desire they just have “generate search query from prompt, execute search query and fetch results, consider the combination of the original prompt and the results to be the context for generating more content and return to user”.
LLM is not a scheme that credibly implies that more LLM == sapient existance. Such a concept may come, but it will be something different than LLM. LLM just looks crazily like dealing with people.
I think there’s two basic mistakes that you made. First, you think that we aren’t experts, but it’s definitely true that some of us have studied these topics for years in college or graduate school, and surely many other people are well read on the subject. Obviously you can’t easily confirm our backgrounds, but we exist. Second, people who are somewhat aware of the topic might realize that it’s not particularly productive to engage in discussion on it here because there’s too much background information that’s missing. It’s often the case that experts don’t try to discuss things because it’s the wrong venue, not because they feel superior.
I watched this entire video just so that I could have an informed opinion. First off, this feels like two very separate talks:
The first part is a decent breakdown of how artificial neural networks process information and store relational data about that information in a vast matrix of numerical weights that can later be used to perform some task. In the case of computer vision, those weights can be used to recognize objects in a picture or video streams, such as whether something is a hotdog or not.
As a side note, if you look up Hinton’s 2024 Nobel Peace Prize in Physics, you’ll see that he won based on his work on the foundations of these neural networks and specifically, their training. He’s definitely an expert on the nuts and bolts about how neural networks work and how to train them.
He then goes into linguistics and how language can be encoded in these neural networks, which is how large language models (LLMs) work… by breaking down words and phrases into tokens and then using the weights in these neural networks to encode how these words relate to each other. These connections are later used to generate other text output related to the text that is used as input. So far so good.
At that point he points out these foundational building blocks have been used to lead to where we are now, at least in a very general sense. He then has what I consider the pivotal slide of the entire talk, labeled Large Language Models, which you can see at 17:22. In particular he has two questions at the bottom of the slide that are most relevant:
- Are they genuinely intelligent?
- Or are they just a form of glorified auto-complete that uses statistical regularities to pastiche together pieces of text that were created by other people?
The problem is: he never answers these questions. He immediately moves on to his own theory about how language works using an analogy to LEGO bricks, and then completely disregards the work of linguists in understanding language, because what do those idiots know?
At this point he brings up The long term existential threat and I would argue the rest of this talk is now science fiction, because it presupposes that understanding the relationship between words is all that is necessary for AI to become superintelligent and therefore a threat to all of us.
Which goes back to the original problem in my opinion: LLMs are text generation machines. They use neural networks encoded as a matrix of weights that can be used to predict long strings of text based on other text. That’s it. You input some text, and it outputs other text based on that original text.
We know that different parts of the brain have different responsibilities. Some parts are used to generate language, other parts store memories, still other parts are used to make our bodies move or regulate autonomous processes like our heartbeat and blood pressure. Still other bits are used to process images from our eyes and other parts reason about spacial awareness, while others engage in emotional regulation and processing.
Saying that having a model for language means that we’ve built an artificial brain is like saying that because I built a round shape called a wheel means that I invented the modern automobile. It’s a small part of a larger whole, and although neural networks can be used to solve some very difficult problems, they’re only a specific tool that can be used to solve very specific tasks.
Although Geoffrey Hinton is an incredibly smart man who mathematically understands neural networks far better than I ever will, extrapolating that knowledge out to believing that a large language model has any kind of awareness or actual intelligence is absurd. It’s the underpants gnome economic theory, but instead of:
- Collect underpants
- ?
- Profit!
It looks more like:
- Use neural network training to construct large language models.
- ?
- Artificial general intelligence!
If LLMs were true artificial intelligence, then they would be learning at an increasing rate as we give them more capacity, leading to the singularity as their intelligence reaches hockey stick exponential growth. Instead, we’ve been throwing a growing amount resources at these LLMs for increasingly smaller returns. We’ve thrown a few extra tricks into the mix, like “reasoning”, but beyond that, I believe it’s clear that we’re headed towards a local maximum that is far enough away from intelligence that would be truly useful (and represent an actual existential threat), but in actuality only resembles what a human can output well enough to fool human decision makers into trusting them to solve problems that they are incapable of solving.
believing that a large language model has any kind of awareness or actual intelligence is absurd
I (as a person who works professionally in the area and tries to keep up with the current academic publications) happen to agree with you. But my credences are somewhat reduced after considering the points Hinton raises.
I think it is worth considering that there are a handful of academically active models of consciousness; some well-respected ones like the CTM are not at all inconsistent with Hinton’s statements
Sounds pretty human to me. /s
Sounds pretty human to me. no /s
However, when the participants and LLMs were asked retroactively how well they thought they did, only the humans appeared able to adjust expectations
This is what everyone with a fucking clue has been saying for the past 5, 6? years these stupid fucking chatbots have been around.
Why is a researcher with a PhD in social sciences researching the accuracy confidence of predictive text, how has this person gotten to where they are without being able to understand that LLM don’t think? Surely they came up when he started even co soldering this brainfart of a research project?
Someone has to prove it wrong before it’s actually wrong. Maybe they set out to discredit the bots
I guess, but it’s like proving your phones predictive text has confidence in its suggestions regardless of accuracy. Confidence is not an attribute of a math function, they are attributing intelligence to a predictive model.
I work in risk management, but don’t really have a strong understanding of LLM mechanics. “Confidence” is something that i quantify in my work, but it has different terms that are associated with it. In modeling outcomes, I may say that we have 60% confidence in achieving our budget objectives, while others would express the same result by saying our chances of achieving our budget objective are 60%. Again, I’m not sure if this is what the LLM is doing, but if it is producing a modeled prediction with a CDF of possible outcomes, then representing its result with 100% confindence means that the LLM didn’t model any other possible outcomes other than the answer it is providing, which does seem troubling.
Nah so their definition is the classical “how confident are you that you got the answer right”. If you read the article they asked a bunch of people and 4 LLMs a bunch of random questions, then asked the respondent whether they/it had confidence their answer was correct, and then checked the answer. The LLMs initially lined up with people (over confident) but then when they iterated, shared results and asked further questions the LLMs confidence increased while people’s tends to decrease to mitigate the over confidence.
But the study still assumes intelligence enough to review past results and adjust accordingly, but disregards the fact that an AI isnt intelligence, it’s a word prediction model based on a data set of written text tending to infinity. It’s not assessing validity of results, it’s predicting what the answer is based on all previous inputs. The whole study is irrelevant.
Well, not irrelevant. Lots of our world is trying to treat the LLM output as human-like output, so if human’s are going to treat LLM output the same way they treat human generated content, then we have to characterize, for the people, how their expectations are broken in that context.
So as weird as it may seem to treat a stastical content extrapolation engine in the context of social science, there’s a great deal of the reality and investment that wants to treat it as “person equivalent” output and so it must be studied in that context, if for no other reason to demonstrate to people that it should be considered “weird”.
I’m pretty much done with them except for some search
Not even a good use case either, especially when it spews such bullshit like “there’s no recorded instance of trump ever having used the word enigma” and “there’s 1 r in strawberry”.
LLMs are a copy paste machine, not a rationalization engine of any sort (at least as far as all the slop that we get shoved in our face, I don’t include the specialized protein folding and reconstructive models that were purpose built for very niche applications)
they’re solid starting point for shopping now that wirecutter, slant, and others are enshittified. i hate it and it makes me feel dirty to use, and you can’t just do whatever the llm says. but asking it for a list of options to then explore is currently the best way i’ve found to jump into things like outdoor basketball shoe options
But what about humans?
LLMs don’t understand anything. At all. They’re a glorified auto complete.
How do you think language in our brains work? Just like many things in tech (especially cameras), things are often inspired by how it works in nature.
Oh god I just figured it out.
It was never they are good at their tasks, faster, or more money efficient.
They are just confident to stupid people.
Christ, it’s exactly the same failing upwards that produced the c suite. They’ve just automated the process.
Oh good, so that means we can just replace the C-suite with LLMs then, right? Right?
An AI won’t need a Golden Parachute when they inevitably fuck it all up.
AI evolved their own form of the Dunning Kruger effect.
About halfway through the article they quote a paper from 2023:
Similarly, another study from 2023 found LLMs “hallucinated,” or produced incorrect information, in 69 to 88 percent of legal queries.
The LLM space has been changing very quickly over the past few years. Yes, LLMs today still “hallucinate”, but you’re not doing anyone a service by reporting in 2025 the state of the field over 2 years before.
They are not only unaware of their own mistakes, they are unaware of their successes. They are generating content that is, per their training corpus, consistent with the input. This gets eerie, and the ‘uncanny valley’ of the mistakes are all the more striking, but they are just generating content without concept of ‘mistake’ or’ ‘success’ or the content being a model for something else and not just being a blend of stuff from the training data.
For example:
Me: Generate an image of a frog on a lilypad.
LLM: I’ll try to create that — a peaceful frog on a lilypad in a serene pond scene. The image will appear shortly below.<includes a perfectly credible picture of a frog on a lilypad, request successfully processed>
Me (lying): That seems to have produced a frog under a lilypad instead of on top.
LLM: Thanks for pointing that out! I’m generating a corrected version now with the frog clearly sitting on top of the lilypad. It’ll appear below shortly.<includes another perfectly credible picture>
It didn’t know anything about the picture, it just took the input at it’s word. A human would have stopped to say “uhh… what do you mean, the lilypad is on water and frog is on top of that?” Or if the human were really trying to just do the request without clarification, they might have tried to think “maybe he wanted it from the perspective of a fish, and he wanted the frog underwater?”. A human wouldn’t have gone “you are right, I made a mistake, here I’ve tried again” and include almost the exact same thing.
But tha training data isn’t predominantly people blatantly lying about such obvious things or second guessing things that were done so obviously normally correct.
The use of language like “unaware” when people are discussing LLMs drives me crazy. LLMs aren’t “aware” of anything. They do not have a capacity for awareness in the first place.
People need to stop taking about them using terms that imply thought or consciousness, because it subtly feeds into the idea that they are capable of such.
Okay fine, the LLM does not take into account in the context of its prompt that yada yada. Happy now word police, or do I need to pay a fine too? The real problem is people are replacing their brains with chatbots owned by the rich so soon their thoughts and by extension the truth will be owned by the rich, but go off pat yourself on the back because you preserved your holy sentience spook for another day.
Is that a recycled piece from 2023? Because we already knew that.
prompting concerns
Oh you.
If you don’t know you are wrong, when you have been shown to be wrong, you are not intelligent. So A.I. has become “Adequate Intelligence”.
That definition seems a bit shaky. Trump & co. are mentally ill but they do have a minimum of intelligence.
As any modern computer system, LLMs are much better and smarter than us at certain tasks while terrible at others. You could say that having good memory and communication skills is part of what defines an intelligent person. Not everyone has those abilities, but LLMs do.
My point is, there’s nothing useful coming out of the arguments over the semantics of the word “intelligence”.
Confidently incorrect.
