You can batch only if you have distinct chat in parallel, | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		d3m0t3p 3 months ago \| parent \| context \| favorite \| on: Open models by OpenAI You can batch only if you have distinct chat in parallel,

mlyle 3 months ago [–]

> > if I want to run 20 concurrent processes, assuming I need 1k tokens/second throughput (on each)

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact