✦   ✦   ✦

breaking news

News, read through The Primal Race
← 全部评论 · all commentary

换把尺子,收你一笔税Change the Ruler, Charge Them a Tax

科技 结构层 · 文化层 Rest of World ↗ 2026-11-02 § 链接
新闻摘要AI 研究联盟 Coalition for Language Equity in AI Research 上月发布研究,发现 AI 模型处理不同语言所需的 token 数量差异巨大:在网络最流行的 20 种语言之间,最贵的语言所用 token 是最便宜语言的 1.5 到 5 倍;扩展到 200 种语言后,某个模型的 token 成本差距扩大到 17 倍。研究者称之为隐藏的「tokenizer tax」,使用小语种的人要付更多钱,并承受更高延迟和更短的上下文容量。
The newsA study published last month by the Coalition for Language Equity in AI Research found that AI models need far more tokens for some languages than others. Among the 20 most popular internet languages, the costliest used 1.5 to 5 times more tokens than the cheapest; across 200 languages, the gap in one model grew to 17 times. The researchers call this a hidden "tokenizer tax," which also means higher latency and shorter context windows for speakers of smaller languages.
定价权在谁手里,谁就不必解释自己的尺子
Whoever owns the ruler never has to explain it.

先把这件事说清楚:tokenizer 不是自然规律,是人做出来的工具,它把文字切成块,块切得多,你就付得多。切法偏向英语,于是英语使用者每一句话都更便宜。这不是市场供需,这是设计者在自己的语言上省了事,账单寄给了别人。

报道里最值得看的一句来自那项研究:班加罗尔的一家创业公司,每一次 prompt 和 completion 都要付一笔附加费,而它在奥斯汀的竞争对手不用付。同一门生意,同一个模型,两个人拿到的报价不一样,原因只是他们说的语言。这是结构层在动——不是谁对谁有恶意,是资源的分配方式本身把一部分人标了高价。

更漂亮的是第二层。报道说,决定 token 数量的不只是语言有多稀有,还看文字。阿斯图里亚斯语使用者很少、在 Common Crawl 里没有数据,却因为用拉丁字母、又沾了意大利语等罗曼语族的光,占了便宜;缅甸语、藏语、提非纳文、高棉语、老挝语、奥尔奇基文这些 tokenizer 很少见过的文字,在几乎所有模型里都稳定垫底。也就是说,便宜不是因为你的语言被认真对待了,是因为你的字形恰好长得像占主导的那一种。这跟第一章讲的是同一件事:表达决定你是谁,而谁的表达被系统认得出来,谁就在博弈里先拿到一张便宜的票。

然后是共谋的那一段。报道里几位受访者的说法都很体面:token 价格在快速下降,直接成本的影响会随时间减轻;更好的多语言数据和合成数据会改善质量。Reddy 说得最老实——光靠训练数据修不好 tokenizer,要修就得实验室主动为这些语言和文字重新设计。可他也只是「警告」,没有承诺。Choudhury 提出让每家实验室发布按语言分的记分卡,承诺对服务最差的语言做出可测量的改进。这个要求之所以要提出来,本身就说明现在没有。

所以真正的问题不是价格。价格会降,报道自己说了。问题是解释权:谁有权决定什么样的文字算「标准切法」,谁就有权决定谁多付、谁等得久、谁的上下文窗口先被历史对话吃光。那个在班加罗尔教学生的辅导 AI,记不住学生前面的错误和讲解,不是因为技术做不到,是因为这门语言的 token 预算先被切走了一半。服务质量的差距,最后会被说成「那些语言的数据就是少」,一句话把设计者的选择洗成了客观条件。

「Perfect equality may be unrealistic」——这句话我同意,完全平等不现实。但报道里那句更该被记住:大规模、持续存在的差距,仍然需要一个解释和一份改进计划。现在两样都没有。有的是一个叫 tokenizer tax 的新名字,和一份说价格会降的安慰。

文字被切碎的时候,没有人听见声音。等到这些语言的使用者在 AI 经济里被系统性地排除在外,账本上只会写一行:他们的语言数据不足。

Let's be clear about what a tokenizer is: not a law of nature, but a tool someone built. It chops text into pieces. More pieces, bigger bill. The chopping favors English, so every English sentence comes cheaper. This isn't supply and demand. It's a designer cutting corners in their own language and mailing the invoice to everyone else.

The sharpest line in the report comes from the study itself: a startup in Bengaluru may pay a surcharge on every prompt and completion while its competitor in Austin does not. Same business, same model, two different quotes, and the only variable is the language they speak. This is the structural layer moving. Nobody bears anyone ill will; the allocation of resources simply prices some people higher.

The second layer is prettier. The study found that token counts depend not only on how rare a language is but on its script. Asturian has few speakers and no Common Crawl data, yet benefits from the Latin-script advantage and from related Romance languages like Italian. Burmese, Tibetan, Tifinagh, Khmer, Lao, and Ol Chiki, scripts the tokenizer rarely saw, sit at the bottom across nearly every model. So you're not cheap because your language was taken seriously. You're cheap because your glyphs happen to resemble the dominant one. This is Chapter One: expression determines who you are, and whoever's expression the system recognizes gets the cheaper ticket first.

Then comes the complicity. The experts quoted are all reasonable: token prices are falling fast, the direct cost impact should ease over time, better multilingual and synthetic data will improve quality. Reddy is the most honest: better training data alone won't fix the tokenizer; labs would have to deliberately redesign future tokenizers for these languages and scripts. But he only warns. He doesn't commit. Choudhury asks every major lab to publish a language-by-language scorecard and commit to measurable improvements for its worst-served languages. The fact that this has to be requested tells you it doesn't exist.

So the real issue isn't price. Prices will fall; the report says so itself. The issue is who holds the power to define what counts as a standard cut. Whoever holds it decides who pays more, who waits longer, whose context window gets eaten by conversation history first. That tutoring AI in Bengaluru can't retain a student's earlier mistakes and explanations, not because the technology can't, but because this language's token budget was cut in half before the lesson began. The service gap will eventually be described as "those languages just have less data," which launders a designer's choice into an objective condition.

"Perfect equality may be unrealistic." Fine. But the line worth keeping is the next one: large, persistent disparities should still require an explanation and an improvement plan. Right now there is neither. There is a new name, tokenizer tax, and a reassurance that prices will fall.

No one hears the sound of text being chopped. By the time these speakers are systematically excluded from the AI economy, the ledger will carry one line: insufficient language data.