All of this is for batch size 1. | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		p12tic 8 months ago \| parent \| context \| favorite \| on: Why DeepSeek is cheap at scale but expensive to ru... All of this is for batch size 1.

qeternity 8 months ago [–]

I know. That was my point.

Throughput doesn't scale on CPU as well as it does on GPU.

p12tic 8 months ago | [–]

We both agree. Batch size 1 is only relevant to people who want to run models on their own private machines. Which is the case of OP.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact