Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I think the authors are using information theory to inappropriately flatten the complexity of the problem. On one hand we have “bits” of pre-processed sensory measurement data, then on the other hand we have “bits” of post-processed symbolic data: in many cases directly so via human language, but that would also include “the Terran unit moved a short distance” as a compact summary of a bunch of pixels updating in StarCraft. This even extends to the animal examples: the 10 bits/s figure applies to higher-level cognition. The crucial difference is that the sensory bits can be interpreted via the same “algorithm” in a context-independent way, whereas the higher-level cognition bits need their algorithms chosen very carefully (perhaps being modified at runtime).

So I am just not sure why 10 bits/s of symbolic data processing is especially slow in the first place. We don’t have a relevant technological comparison because none of our technology actually processes data in that fashion.



I'm running a chat LLM on my local pc. It spits out text just slightly faster than I can type, but it is using much of my CPU and redlining my GPU.

Is it processing at a dozen bits per second, or hundreds of millions?

If the text the LLM generates is "that is true", can I consider that one bit of information?

I agree, they're artificially simplifying the framing of the question to generate a lower number than is sensible.


When compared directly to the 10^9 bits/s for sensory information, which uses the same type of information, it is slow.


It's more like quantum information theory isn't it?

https://en.wikipedia.org/wiki/Quantum_information




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: