How is this technically done? How does it split the query and aggregates the res... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

kosolam 11 months ago | parent | context | favorite | on: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

How is this technically done? How does it split the query and aggregates the results?

magicalhippo 11 months ago [–]

From the readme:

More devices mean faster performance, leveraging tensor parallelism and high-speed synchronization over Ethernet.

The maximum number of nodes is equal to the number of KV heads in the model #70.

I found this[1] article nice for an overview of the parallelism modes.

[1]: https://medium.com/@chenhao511132/parallelism-in-llm-inferen...

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact