We don't know all the different ways humans arrive at answers to novel problems.
And while these LLMs aren't literally just copying verbatim, they are literally just token selection machines with sophisticated statistical weighting algorithms biased heavily towards their training sets. That isn't to say they are overfitted, but the sheer scale/breadth gives the appearance of generalization without the substance of it.
Here's an argument that GPT does actually build an internal representation of the game Othello, it's not just token selection: https://thegradient.pub/othello/
Keep in mind that the Othello example is model specifically trained on only Othello games. I haven’t seen any claims that general purpose models like GPT-4 have internal representations of complex abstract structures like this.
Why wouldn't they? Text-moves of Othello games are presumably a subset of the training data for a general LLM. If anything the general LLM has the chance to derive more robust internal world representations given similarly laid out board games.
It is also not surprising that if you force a system to succinctly encode input-output relationships, eventually it discovers the underlying generating process or its equivalent as implied by Kolmogorov complexity theory. Language is just a convenient encoding for inputs and outputs, not fundamental. So yes it is regurgitating statistics, but statistics are non-random because of some non-trivial underlying process, always, and if you can regurgitate those statistics consistently you're guaranteed to have learned a representation of the process. There is no difference and biological systems aren't any different.
This morning I asked GPT-4 to play duck chess with me. Duck chess is a very simple variant of chess with a duck piece (that acts like an impassable brick) that each player moves to an empty square after their normal move. [I gave GPT-4 a more thorough and formal explanation of the rules of course.]
To a human, board state in chess and in duck chess is very simple. It’s chess, but with one square that’s blocked off. Similarly, even a beginner human chess player can understand the basics of duck chess strategy (block your opponent’s development in the beginning, block your opponent’s attacks with the duck to free up your other pieces, etc.).
GPT-4 fell apart, often failing to make legal moves, and never once making a strategically coherent duck placement. To me this suggests that it does not have an internal representation of the 64 squares of the board at all. Even if you set aside the strategic aspect, the only requirement for a duck move to be legal is that you place it on an empty square, which it cannot consistently do, even at the very beginning of the game (it like to place the duck on d7 as black after 1. …e5, even when its own pawn is there).
It is a matter of degree. GPT-4 may, for various reasons some of which are artificial handicaps, have only a weak grasp of a board representation now. But if it has any such representation at all, that's already a different story than if it did not. I think all evidence points this way, even from other networks, e.g. image classification networks that learn common vision filters. It's a pretty general phenomenon.
And while these LLMs aren't literally just copying verbatim, they are literally just token selection machines with sophisticated statistical weighting algorithms biased heavily towards their training sets. That isn't to say they are overfitted, but the sheer scale/breadth gives the appearance of generalization without the substance of it.