Keyword Search 101: How BM25 Scores
Comments on “Keyword Search 101: How BM25 Scores”
Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked
Deployed's explainer works through BM25, the default ranker in Lucene, Elasticsearch, OpenSearch and Solr, on a five-page example and then on 660 questions over papers, contracts and meeting transcripts. It reports that IDF and term saturation matter most, that the defaults k1 = 1.2 and b = 0.75 land within 1.5 points of the best of 70 settings, and that scores shouldn't be thresholded. Textbook IDF can go negative, so Lucene's ln(1 + ...) is safer. The measurements are the author's own, on small corpora, with no source given.
Why it ranked
No. 7 of 59 with a score of 0.67. The panel rated it highest on technical depth (0.89) and substantiated (0.86), lowest on significance (0.50). Placed No. 7, No. 10 by score alone: the placement terms moved other stories past it.
- AI judgeJudge 2claude-haiku-5-5
- Detailed, well-substantiated technical walkthrough of BM25 with reproducible experiments on search internals; practical for builders of search systems, though not a landmark and partly a tutorial.
- AI judgeJudge 3gpt-5.6-luna
- A technically detailed, practically useful BM25 explanation with original comparative measurements, implementation caveats, and reproducible formulas. Some claims are broad and several reported values appear formatting-truncated, but the core analysis is well supported.
Each judge wrote its line after answering the rubric, with the site and the author still hidden. Every answer behind this is in the table under Score.
Score
0.67total under rubric v2.3
- Significance0.50
- Reader interest0.75
- Audience breadth0.64
- Practical relevance0.83
- Substantiated0.86
- Technical depth0.89
- New information0.78
- Counts against: Promotional0.06
- Counts against: Overclaim0.22
Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).
| Question | Judge 1 | Judge 2 | Judge 3 | Mean | Spread | Weight | Contribution (weight times mean) |
|---|---|---|---|---|---|---|---|
| SignificanceHow much does this change what people in tech know or can do? | 0.58 | 0.30 | 0.62 | 0.50 | 0.32 | +0.24 | +0.120 |
| Reader interestWould a well-informed senior engineer or tech founder want to read this today, and likely pass it on? | 0.66 | 0.70 | 0.88 | 0.75 | 0.22 | +0.26 | +0.194 |
| Audience breadthHow many people who build technology does it concern? | 0.70 | 0.50 | 0.72 | 0.64 | 0.22 | +0.13 | +0.083 |
| Practical relevanceHow useful is it to someone who builds software or tech products? | 0.87 | 0.80 | 0.83 | 0.83 | 0.07 | +0.22 | +0.183 |
| SubstantiatedHow well is it backed? Consider whether it is the primary source, and whether its main claims come with data, code, a demo, a reproducible method or citations. | 0.95 | 0.85 | 0.78 | 0.86 | 0.17 | +0.05 | +0.043 |
| Technical depthHow technically substantive is it? | 1.00 | 0.90 | 0.78 | 0.89 | 0.22 | +0.05 | +0.045 |
| New informationDoes it contain information absent from the stories covered in the last 7 days (list provided)? | 0.89 | 0.60 | 0.86 | 0.78 | 0.29 | +0.05 | +0.039 |
| PromotionalIs it mainly marketing with little substance (a press release, ad or sponsored post with no real information)? A launch post that carries real news is not promotional. | 0.04 | 0.10 | 0.04 | 0.06 | 0.06 | −0.19 | −0.011 |
| OverclaimDo the headline or conclusions claim more than the body supports? | 0.32 | 0.15 | 0.18 | 0.22 | 0.17 | −0.11 | −0.024 |
| Sum of contributions | +0.673 | ||||||
| Claim-check penalty | 0.000 | ||||||
| Total, as published | 0.672 | ||||||
Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.
The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.
<a href="https://rubric.hrih.si/editions/2026-11-01/7/"><img src="https://rubric.hrih.si/badge/2026-11-01/7.svg" alt="Ranked #7 on Rubric, 1 Nov 2026" width="231" height="28"></a>Claim check
Claim check: 1 matches source · 3 no source given · 4 couldn’t check. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.
Couldn’t checkVersion
Lucene 8.0 dropped the (k1 + 1) factor from the BM25 numerator, tracked in LUCENE-8563.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “Lucene also dropped the (k1 + 1) factor that the Okapi team's TREC-3 formula has in its numerator, in Lucene 8.0”
The source says No passage could be compared.
Source cited by the article: issues.apache.org/jira/browse/LUCENE-8563
Couldn’t checkQuote
Kamphuis et al. (2020) compared BM25 variants on several test collections and found the differences small.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “Kamphuis et al. (2020) compared the variants on several test collections and found the differences small”
The source says No passage could be compared.
Source cited by the article: link.springer.com/chapter/10.1007/978-3-030-45442-5_4
Matches sourceNumber
Lucene stores each document's length in one byte using SmallFloat.intToByte4, exact up to 40 and rounded to four significant bits above that.
The cited source says the same thing.
The article says “lengths up to 40 are exact, and above that the length is rounded down to four significant bits”
The source says “if (i < NUM_FREE_VALUES) { return (byte) i; } else { return (byte) (NUM_FREE_VALUES + longToInt4(i - NUM_FREE_VALUES”
Source cited by the article: github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/util/SmallFloat.java
Couldn’t checkDate
BM25 was published in 1994 and is still the baseline against which every embedding model is measured.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “published in 1994, and still the baseline every embedding model is measured against”
The source says No passage could be compared.
Source cited by the article: deployed-ai.io/blog/what-is-an-embedding
Couldn’t checkBenchmark result
On identifier queries over SciFact, BM25 scored 0.93 nDCG@10 versus 0.21 to 0.30 for three embedding models.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “on identifier queries over SciFact, it scored 0.93 nDCG@10 against 0.21 to 0.30 for three embedding models”
The source says No passage could be compared.
Source cited by the article: deployed-ai.io/blog/what-is-an-embedding
No source givenNumber
Removing saturation or IDF cost 4 to 20 points of recall on chunked papers, contracts and transcripts, while removing length normalization cost under 2.
The article gives no link or citation for it.
The article says “removing either one cost 4 to 20 points of recall; removing length normalization cost under 2”
The source says No source to compare with.
No source givenBenchmark result
The default k1 = 1.2 and b = 0.75 came within 1.5 points of the best of 70 settings on all six dataset-chunking pairs.
The article gives no link or citation for it.
The article says “k1 = 1.2 and b = 0.75 came within 1.5 points of the best of 70 settings everywhere”
The source says No source to compare with.
No source givenGeneralization
The top BM25 score correlates 0.57 to 0.79 with the number of query words, so no score threshold means relevant.
The article gives no link or citation for it.
The article says “the correlation between the top score and the number of query words is 0.57 to 0.79”
The source says No source to compare with.
Also covered by
No other coverage was clustered with this story.
Elsewhere
Elsewhere: HN 1 pts, not on front page
Recorded for context only, checked 1 Nov 2026, 05:47 UTC. It played no part in the ranking.