Do you still need embeddings for semantic search? - my little experiment
Comments on “Do you still need embeddings for semantic search? - my little experiment”
Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked
Lax Meiyappan tested whether BM25 keyword search, with an LLM query rewrite and a reranker, could replace Qwen3-Embedding-0.6B on SciFact, NFCorpus and FiQA. Embeddings won or tied in every fair comparison, leading by about a point on the first two sets and eight to nine on FiQA once both used the same reranker. The rewrite added little after reranking. Replacing 54 FiQA names with invented codenames narrowed the gap to 1.3 points, but embeddings stayed ahead. These are one author's unreplicated results, and the cost comparison reflects his pricing.
Why it ranked
No. 15 of 43 with a score of 0.66. The panel rated it highest on technical depth (0.85) and substantiated (0.85), lowest on significance (0.49). Placed No. 15, No. 8 by score alone: the sixth AI & machine learning story of the day (−0.060), and close to “Keyword Search 101: How BM25 Scores”, No. 7 on Sunday 1 November 2026 (−0.018). The placement terms move stories; they are not part of the score.
- AI judgeJudge 2claude-haiku-5-5
- Hands-on, well-controlled experiment comparing embeddings against LLM-rewritten BM25 search with reranking; substantive technical detail and honest negative results, though a niche topic with a modest scope.
- AI judgeJudge 3gpt-5.6-luna
- A well-controlled, reproducible retrieval experiment compares BM25, query rewriting, reranking, embeddings, and hybrid search across public datasets, with practical cost and system-design implications. Some results are visually incomplete, but methodology and conclusions are well supported.
Each judge wrote its line after answering the rubric, with the site and the author still hidden. Every answer behind this is in the table under Score.
Score
0.66total under rubric v2.3
- Significance0.49
- Reader interest0.78
- Audience breadth0.57
- Practical relevance0.78
- Substantiated0.85
- Technical depth0.85
- New information0.81
- Counts against: Promotional0.08
- Counts against: Overclaim0.16
Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).
| Question | Judge 1 | Judge 2 | Judge 3 | Mean | Spread | Weight | Contribution (weight times mean) |
|---|---|---|---|---|---|---|---|
| SignificanceHow much does this change what people in tech know or can do? | 0.49 | 0.35 | 0.62 | 0.49 | 0.27 | +0.24 | +0.117 |
| Reader interestWould a well-informed senior engineer or tech founder want to read this today, and likely pass it on? | 0.72 | 0.72 | 0.90 | 0.78 | 0.18 | +0.26 | +0.203 |
| Audience breadthHow many people who build technology does it concern? | 0.57 | 0.55 | 0.58 | 0.57 | 0.03 | +0.13 | +0.074 |
| Practical relevanceHow useful is it to someone who builds software or tech products? | 0.81 | 0.70 | 0.82 | 0.78 | 0.12 | +0.22 | +0.171 |
| SubstantiatedHow well is it backed? Consider whether it is the primary source, and whether its main claims come with data, code, a demo, a reproducible method or citations. | 0.95 | 0.75 | 0.86 | 0.85 | 0.20 | +0.05 | +0.043 |
| Technical depthHow technically substantive is it? | 0.93 | 0.80 | 0.84 | 0.85 | 0.13 | +0.05 | +0.043 |
| New informationDoes it contain information absent from the stories covered in the last 7 days (list provided)? | 0.90 | 0.60 | 0.93 | 0.81 | 0.33 | +0.05 | +0.041 |
| PromotionalIs it mainly marketing with little substance (a press release, ad or sponsored post with no real information)? A launch post that carries real news is not promotional. | 0.06 | 0.10 | 0.08 | 0.08 | 0.04 | −0.19 | −0.015 |
| OverclaimDo the headline or conclusions claim more than the body supports? | 0.25 | 0.15 | 0.08 | 0.16 | 0.17 | −0.11 | −0.018 |
| Sum of contributions | +0.657 | ||||||
| Claim-check penalty | 0.000 | ||||||
| Total, as published | 0.657 | ||||||
Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.
The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.
<a href="https://rubric.hrih.si/editions/2026-11-02/15/"><img src="https://rubric.hrih.si/badge/2026-11-02/15.svg" alt="Ranked #15 on Rubric, 2 Nov 2026" width="238" height="28"></a>Claim check
Claim check: 1 matches source · 7 no source given. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.
No source givenGeneralization
In the author's experiment, embeddings won or tied in every fair comparison against keyword search.
The article gives no link or citation for it.
The article says “Embeddings are still hard to beat; embeddings won or tied in every fair comparison”
The source says No source to compare with.
No source givenNumber
Keyword search scored within about 1 point of embeddings when question and answer shared words or close synonyms, and dropped 8 to 9 points when the answer used completely different words.
The article gives no link or citation for it.
The article says “keyword search came within about 1% point when the question and the answer shared words or close synonyms. It dropped 8 to 9% when the answer used completely different words”
The source says No source to compare with.
Matches sourceNumber
The Query2doc paper (2023) uses the same technique of repeating the original query in the expanded query, with five repeats.
The cited source says the same thing.
The article says “Query2doc, a 2023 paper, does the same thing with five repeats”
The source says No passage could be compared.
Source cited by the article: arxiv.org/abs/2303.07678
No source givenBenchmark result
With the same reranker applied, embeddings stayed ahead of keyword search everywhere: about a point on SciFact and NFCorpus, and eight to nine points on FiQA.
The article gives no link or citation for it.
The article says “Embeddings with the reranker stayed ahead everywhere: about a point on SciFact and NFCorpus, eight to nine on FiQA”
The source says No source to compare with.
No source givenBenchmark result
Once a reranker was used, the LLM query rewrite made little difference: within about half a point on SciFact and NFCorpus, and slightly worse on FiQA.
The article gives no link or citation for it.
The article says “once the reranker was in, the rewrite barely mattered, within about half a point on SciFact and NFCorpus, and slightly worse on FiQA”
The source says No source to compare with.
No source givenBenchmark result
After swapping 54 real FiQA names for made-up codenames, embeddings lost about 10 points while keyword search barely moved, and the gap between the reranked setups shrank from 7.5 to 1.3 points.
The article gives no link or citation for it.
The article says “Embeddings lost about 10 points, while keyword search barely moved. The gap between the two reranked setups shrank from 7.5 points to 1.3”
The source says No source to compare with.
No source givenCausal conclusion
Letting the reranker read the top 200 closes about half the gap at about four times the reranking cost, and even at 500 embeddings with a reranker stay 3.5 points ahead.
The article gives no link or citation for it.
The article says “Reading the top 200 closes about half the gap, for about four times the reranking cost. After that it flattens, and even at 500, embeddings with a reranker stay 3.5…”
The source says No source to compare with.
No source givenNumber
Embedding all of FiQA costs about as much as 200 to 1,200 reranked keyword searches.
The article gives no link or citation for it.
The article says “embedding all of FiQA costs about as much as 200 to 1,200 reranked keyword searches”
The source says No source to compare with.
Also covered by
No other coverage was clustered with this story.
Elsewhere
Elsewhere: HN 2 pts, not on front page
Recorded for context only, checked 2 Nov 2026, 05:46 UTC. It played no part in the ranking.