TriusLD – A Small Tokenizer for LLMs with Robust Multilingual Support
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
TriusLD is a 65,535-slot tokenizer for LLMs that allocates vocabulary across 19 writing systems using per-script quotas plus a shared byte fallback. Its author reports 0.52 tokens per character averaged over 58 languages, versus 0.84 for GPT-4's vocabulary, while English gets 3.85 characters per token, about 18% behind a 200,000-token…