We build our voice input method like an experiment
Comments on “We build our voice input method like an experiment”
Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked
Arion, a local Chinese voice and Pinyin input method, published results from ablations it ran before shipping features. On a 30-case held-out set, screen-text context lifted Qwen3 4B from 9 to 16 passes but left the shipped 1.7B model at 9. A no-LLM acoustic second pass beat every LLM correction variant. An upstream Rime reranker's "1 ms" came from CPU time, not wall-clock. These figures are Arion's own, from one developer-authored corpus on one Apple M4, and could not be independently checked.
Why it ranked
No. 3 of 43 with a score of 0.71. The panel rated it highest on substantiated (0.92) and new information (0.92), lowest on significance (0.54).
- AI judgeJudge 2claude-haiku-5-5
- Detailed, well-evidenced engineering write-up on evaluating local LLM voice input with ablations and reproducible archive; strong technical depth and practical lessons, narrower audience around local ASR/IME work.
- AI judgeJudge 3gpt-5.6-luna
- A highly reproducible engineering report with held-out tests, controls, failure analysis, latency measurements, and archived evidence. It offers broadly useful lessons for AI-assisted product development, evaluation, and local-model integration.
Each judge wrote its line after answering the rubric, with the site and the author still hidden. Every answer behind this is in the table under Score.
Score
0.71total under rubric v2.3
- Significance0.54
- Reader interest0.81
- Audience breadth0.59
- Practical relevance0.80
- Substantiated0.92
- Technical depth0.92
- New information0.92
- Counts against: Promotional0.06
- Counts against: Overclaim0.11
Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).
| Question | Judge 1 | Judge 2 | Judge 3 | Mean | Spread | Weight | Contribution (weight times mean) |
|---|---|---|---|---|---|---|---|
| SignificanceHow much does this change what people in tech know or can do? | 0.61 | 0.30 | 0.70 | 0.54 | 0.40 | +0.24 | +0.129 |
| Reader interestWould a well-informed senior engineer or tech founder want to read this today, and likely pass it on? | 0.80 | 0.72 | 0.92 | 0.81 | 0.20 | +0.26 | +0.211 |
| Audience breadthHow many people who build technology does it concern? | 0.67 | 0.42 | 0.67 | 0.59 | 0.25 | +0.13 | +0.076 |
| Practical relevanceHow useful is it to someone who builds software or tech products? | 0.91 | 0.60 | 0.88 | 0.80 | 0.31 | +0.22 | +0.175 |
| SubstantiatedHow well is it backed? Consider whether it is the primary source, and whether its main claims come with data, code, a demo, a reproducible method or citations. | 0.99 | 0.82 | 0.96 | 0.92 | 0.17 | +0.05 | +0.046 |
| Technical depthHow technically substantive is it? | 1.00 | 0.85 | 0.91 | 0.92 | 0.15 | +0.05 | +0.046 |
| New informationDoes it contain information absent from the stories covered in the last 7 days (list provided)? | 0.88 | 0.90 | 0.98 | 0.92 | 0.10 | +0.05 | +0.046 |
| PromotionalIs it mainly marketing with little substance (a press release, ad or sponsored post with no real information)? A launch post that carries real news is not promotional. | 0.05 | 0.10 | 0.03 | 0.06 | 0.07 | −0.19 | −0.011 |
| OverclaimDo the headline or conclusions claim more than the body supports? | 0.14 | 0.15 | 0.04 | 0.11 | 0.11 | −0.11 | −0.012 |
| Sum of contributions | +0.706 | ||||||
| Claim-check penalty | 0.000 | ||||||
| Total, as published | 0.706 | ||||||
Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.
The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.
<a href="https://rubric.hrih.si/editions/2026-11-02/3/"><img src="https://rubric.hrih.si/badge/2026-11-02/3.svg" alt="Ranked #3 on Rubric, 2 Nov 2026" width="231" height="28"></a>Claim check
Claim check: 8 couldn’t check. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.
Couldn’t checkBenchmark result
On Arion's held-out test set, Qwen3 1.7B scored 9 of 30 with screen context, cursor text and dictation history enabled, and also 9 of 30 with all context off.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “turning on screen context, cursor text and dictation history scored 9 out of 30 on our held-out test set. Turning all context off also scored 9 out of 30.1”
The source says No passage could be compared.
Source cited by the article: github.com/AstraFoundry/Arion/blob/43586d3277719eec7ebdfcaa377fbfd4b649bd81/benchmarks/results/context-ablation-v1/README.md
Couldn’t checkBenchmark result
With macOS Vision accurate OCR (107 ms p50), screen text added 5 to 6 passes on the Qwen3 4B model, while text around the cursor added 0 passes and dictation history added 1 to 2.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “paired with macOS Vision's accurate OCR (107 ms p50), it added 5 to 6 passes on the 4B model”
The source says No passage could be compared.
Source cited by the article: github.com/AstraFoundry/Arion/blob/43586d3277719eec7ebdfcaa377fbfd4b649bd81/benchmarks/results/context-ablation-v1/README.md
Couldn’t checkBenchmark result
Only the Qwen3 4B model climbed from 9 to 16 passes with context, at roughly twice the latency and memory.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “Only the 4B model climbed from 9 to 16, at roughly twice the latency and memory”
The source says No passage could be compared.
Source cited by the article: github.com/AstraFoundry/Arion/blob/43586d3277719eec7ebdfcaa377fbfd4b649bd81/benchmarks/results/context-ablation-v1/README.md
Couldn’t checkNumber
The offline acoustic second pass control took 0.13 s at p50, lowered all three error rates, improved 60 utterances, and corrupted none of the originally correct ones.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “Re-decoding the audio with an offline acoustic model took 0.13 s at p50, lowered all three error rates, improved 60 utterances, and corrupted none of the originally correct ones”
The source says No passage could be compared.
Source cited by the article: github.com/AstraFoundry/Arion/blob/43586d3277719eec7ebdfcaa377fbfd4b649bd81/benchmarks/results/phonetic-asr-1.7b-v2/README.md
Couldn’t checkNumber
Passing raw token scores to an unadapted Qwen3 1.7B corrupted 26 of the 60 utterances that streaming recognition had transcribed without error.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “It corrupted 26 of the 60 utterances that streaming recognition had transcribed without error”
The source says No passage could be compared.
Source cited by the article: github.com/AstraFoundry/Arion/blob/43586d3277719eec7ebdfcaa377fbfd4b649bd81/benchmarks/results/phonetic-asr-1.7b-v2/README.md
Couldn’t checkBenchmark result
Reproducing the upstream Rime LLM reranker on 100 real candidate snapshots, top-1 accuracy fell from 56 to 45 of 100, with 3 rescues and 14 breakages.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “Top-1 accuracy fell from 56 of 100 to 45 of 100, with 3 rescues and 14 breakages”
The source says No passage could be compared.
Source cited by the article: github.com/AstraFoundry/Arion/blob/43586d3277719eec7ebdfcaa377fbfd4b649bd81/benchmarks/results/rime-llm-reranker-v1/README.md
Couldn’t checkCausal conclusion
The reranker's os.clock() timer measures process CPU time, so it reported 0.787 ms p50 while HTTP calls to the model server took 288.85 ms at the median.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “its timer called os.clock(), which measures process CPU time. While LuaSocket blocked waiting for the model server to respond, the Rime process was asleep”
The source says No passage could be compared.
Source cited by the article: github.com/AstraFoundry/Arion/blob/43586d3277719eec7ebdfcaa377fbfd4b649bd81/benchmarks/results/rime-llm-reranker-v1/README.md
Couldn’t checkBenchmark result
With the context ranker enabled, later-occurrence top-1 rose to 88.7% versus 32.2% for standard frequency learning, with 47 of 50 pairs improving and 0 regressing.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “With the context ranker enabled, later-occurrence top-1 rose to 88.7% (MRR from 0.612 to 0.933), with 47 pairs improving and 0 regressing”
The source says No passage could be compared.
Source cited by the article: github.com/AstraFoundry/Arion/blob/43586d3277719eec7ebdfcaa377fbfd4b649bd81/benchmarks/results/imk-context-v1/README.md
Also covered by
No other coverage was clustered with this story.
Elsewhere
Elsewhere: HN 1 pts, not on front page
Recorded for context only, checked 2 Nov 2026, 05:46 UTC. It played no part in the ranking.