Fair points. Thanks for letting me refine and discard hypotheses. While I think about those, how about this?
Language is not a complete representation of thinking.
We use language to describe symbols, not even very precisely, and we can convert imprecise language to more precise symbols in our brain, manipulate them as symbols, and only then turn them back into language.
That’s why you often cannot perfectly translate something between two languages. That’s why nine year olds, who have been trained on far less text, can learn to do math that ChatGTP never could without an API. (They don’t have to generate their output linearly - they can add the one’s column first) When Newton invented calculus he wasn’t predictively generating words by token; he performed logical manipulation of symbols in his brain first.
That’s why LLMs can’t tell you where they got a specific piece of their own output from, while a human can. This matters because LLMs can’t convert it into a symbol and think about it directly and deduce new conclusions from it, while a human can.
If fundamentally human thinking was just “LLM” we would have never generated the words to train ourselves on in the first place! And neither would any new idea that gradually built the library of human knowledge that eventually trained ChatGTP. The language is just the interface; it’s not the full essence of the thinking itself.
> We use language to describe symbols, not even very precisely, and we can convert imprecise language to more precise symbols in our brain, manipulate them as symbols, and only then turn them back into language.
I don't think that's true for all people. I know that some people manipulate words in their heads, others images, I manipulate sounds and images. Language is just a noisy medium through which we communicate the internal state of our brain or its outputs to other people / humans and ourselves.
> can learn to do math that ChatGTP never could without an API.
GPT4 does just fine in some cases and extrapolates just fine in others, e.g. ask it whether there are more wheels or doors, and try to investigate the definitions of either and see how well it adds the numbers.
> When Newton invented calculus he wasn’t predictively generating words by token;
There are very few people in history up to Newton so I don't think it's fair to hold up what is essentially a new field up to him.
> he performed logical manipulation of symbols in his brain first.
We don't know "how" he did that. We don't know that his brain manipulated symbols everything he did. We simply know that Calculus can be derived from a set of axioms following logical inference.
What you are expressing is largely true for many primates, and according to some, our brains are "just linearly scaled primate brains".
> That’s why LLMs can’t tell you where they got a specific piece of their own output from, while a human can.
I don't think that is correct. The human might provide a justification for something but that doesn't mean it is the true reason they reached a conclusion. The only way this happens is if you apply logical operators, at which point we are doing math again.
It turns out that our brains have decided long before we are even aware of the decision, such decisions may be guided by external stimulation, or even by internal stimulation since our neural networks don't have well defined components and boundaries thus neighbouring neurons can affect or even trigger circuits, and our own forward predictive models back propagate information to other circuits.
> If fundamentally human thinking was just “LLM” we would have never generated the words to train ourselves on in the first place!
I don't think that's true. Language has evolved over thousands of years in many different ways by > 80 bn humans each of whom having 80bn neurons and trillions of synapses.
Yet, we have found that models can learn to communicate with each other and derive their own languages.
I highly recommend you read Eagleman's "The Brain: The Story of You". It covers nearly everything I spoke of here and is very easy to read / listen to.
> I know that some people manipulate words in their heads, others images, I manipulate sounds and images. Language is just a noisy medium through which we communicate the internal state of our brain or its outputs to other people / humans and ourselves.
We are in agreement here. I think you are only strengthening my argument that language is too imprecise and restrictive for LLMs to be fundamentally equivalent to human thinking.
> GPT4 does just fine in some cases and extrapolates just fine in others
"just fine" is not very persuasive here. A nine-year-old can do much better than "just fine" after learning some very simple rules with very minimal examples. And I conjecture that if you removed a lot of the mathematical examples from GTP's training corpus, to be more equivalent to what a nine-year-old has seen, it would do even worse. And it's fundamentally because it cannot break out of its linear language limits and understand numbers as abstract symbols that can be manipulated before converting back into language.
> I don't think that is correct. The human might provide a justification for something but that doesn't mean it is the true reason they reached a conclusion.
Sometimes, yes. What I meant here is that a human can (sometimes - when it is acting intelligently) specifically repeat information they learned and specifically recall the source of that information. This is how our entire scientific process works; we can go back and look up exactly how we derived any piece of our collective knowledge, verify or repeat it if necessary, or build on it further. You're proving my point by the citations you are giving me! (thank you for them) As an intelligent human, you do not "hallucinate" sources, you can provide real ones and provide them directly.
And you can do this because - to go back to my original argument - your intelligence is fundamentally different than an LLM. (That's not an argument that AI is impossible, only that we work differently somehow than what we've seen so far.)
The fact that GPT understands how to use tools suggests that not only does it understand the meaning of numbers, it also understands its own limitations.
By all means, the argument around numeracy is bogus, as lots of people have numeracy issues but they know how to use a calculator.
The fact that so many people seem stuck up over the inability to write perfect math when it can do in context learning of a novel programming language, do addition over groups where addition is not defined as a+b but as a+b+c where c is a constant is incredible.
If we held humans to the same standard we hold GPT3.5+ models, the vast majority pf humans would fail.
The fact that it needs as much data as it does is simply an architectural issue and not inherent to the model itself.
As for hallucinations; I will point to whole thing that religion is, a mass psychosis, Eagleman's book goes into great detail on how we hallucinate our reality.
I don't feel like you're responding to the arguments I'm making. Yes, people have numeracy issues or suffer from mass psychosis, but we generally consider those signs of less intelligence. I'm not holding GTP to the same standards; I'm arguing that human intelligence/thinking is not fundamentally the same as an LLM (e.g. reasoning in flexible symbols rather than language), which is why LLMs appear highly intelligent in some ways but much less intelligent in other ways (like grade-school numeracy or the ability to cite sources that exist)
Language is not a complete representation of thinking.
We use language to describe symbols, not even very precisely, and we can convert imprecise language to more precise symbols in our brain, manipulate them as symbols, and only then turn them back into language.
That’s why you often cannot perfectly translate something between two languages. That’s why nine year olds, who have been trained on far less text, can learn to do math that ChatGTP never could without an API. (They don’t have to generate their output linearly - they can add the one’s column first) When Newton invented calculus he wasn’t predictively generating words by token; he performed logical manipulation of symbols in his brain first.
That’s why LLMs can’t tell you where they got a specific piece of their own output from, while a human can. This matters because LLMs can’t convert it into a symbol and think about it directly and deduce new conclusions from it, while a human can.
If fundamentally human thinking was just “LLM” we would have never generated the words to train ourselves on in the first place! And neither would any new idea that gradually built the library of human knowledge that eventually trained ChatGTP. The language is just the interface; it’s not the full essence of the thinking itself.