AI models cost more when you don’t use English
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
A new study from the Coalition for Language Equity in AI Research finds that AI models need far more tokens to process some languages than others, creating what researchers call a "tokenizer tax." Speakers of less-represented languages, and languages written in scripts tokenizers rarely saw, can face higher costs, slower…