How can you finetune for a new language? Aren't the tokens baked in by the time ... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

1024core on July 18, 2023 | parent | context | favorite | on: Llama 2

How can you finetune for a new language? Aren't the tokens baked in by the time the model is done training?

brucethemoose2 on July 19, 2023 [–]

Apparently not. shrug

The backend does sometimes need a new tokenizer, depending on how its implemented.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact