If LLMs' internal representations are essentially one-to-one mappings of input texts with no additional structure, how can those representations be useful for tasks like object manipulation in robotics?

How is transfer learning possible when non-textual training data enhances performance on textual tasks?

I didn't mean it is a one to one mapping from tokens. But instead it might be mapping a corpus of input text to some points in some multi dimensional space, (just like the input data a linear regression), then then it just extends the line further across that space to get the output.

>How is transfer learning possible when non-textual training data enhances performance on textual tasks?

If non-textual training data can be mapped to the same multi-dimensional space ( by using them alongside textual data during training or something like that), then shouldn't it be possible to do what you describe?