Benchmarking Vector search in six database engines
Comments on “Benchmarking Vector search in six database engines”
Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked
Percona benchmarked HNSW vector search in six engines on 990,000 OpenAI embeddings, in one run on one machine. Time to a queryable index ranged from 1.9 minutes to 6.7 hours. pgvector scaled 21.7× at 32 clients, while the three InnoDB-based engines stayed flat. At 1% filter selectivity, pgvector's default returned too few rows. Concurrency figures for those three are partly withdrawn because of a harness lock, and three churn cells are missing.
Why it ranked
No. 7 of 127 with a score of 0.72. The panel rated it highest on technical depth (0.95) and substantiated (0.93), lowest on audience breadth (0.52). Placed No. 7, No. 12 by score alone: the placement terms moved other stories past it.
- AI judgeJudge 2claude-haiku-5-5
- Detailed, self-critical benchmark of vector search across six databases with reproducible setup and disclosed errors; strong technical depth and practical value, though single-run and narrow in scope.
- AI judgeJudge 3gpt-5.6-luna
- A highly reproducible, technically deep benchmark with raw methodology, caveats, corrections, and actionable database-selection findings. Its scope is one corpus and machine, so significance and breadth are notable but not field-wide.
Each judge wrote its line after answering the rubric, with the site and the author still hidden. Every answer behind this is in the table under Score.
Score
0.72total under rubric v2.3
- Significance0.56
- Reader interest0.82
- Audience breadth0.52
- Practical relevance0.84
- Substantiated0.93
- Technical depth0.95
- New information0.92
- Counts against: Promotional0.03
- Counts against: Overclaim0.16
Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).
| Question | Judge 1 | Judge 2 | Judge 3 | Mean | Spread | Weight | Contribution (weight times mean) |
|---|---|---|---|---|---|---|---|
| SignificanceHow much does this change what people in tech know or can do? | 0.59 | 0.40 | 0.68 | 0.56 | 0.28 | +0.24 | +0.134 |
| Reader interestWould a well-informed senior engineer or tech founder want to read this today, and likely pass it on? | 0.73 | 0.80 | 0.94 | 0.82 | 0.21 | +0.26 | +0.214 |
| Audience breadthHow many people who build technology does it concern? | 0.52 | 0.50 | 0.55 | 0.52 | 0.05 | +0.13 | +0.068 |
| Practical relevanceHow useful is it to someone who builds software or tech products? | 0.86 | 0.80 | 0.86 | 0.84 | 0.06 | +0.22 | +0.185 |
| SubstantiatedHow well is it backed? Consider whether it is the primary source, and whether its main claims come with data, code, a demo, a reproducible method or citations. | 0.99 | 0.85 | 0.95 | 0.93 | 0.14 | +0.05 | +0.047 |
| Technical depthHow technically substantive is it? | 1.00 | 0.90 | 0.96 | 0.95 | 0.10 | +0.05 | +0.048 |
| New informationDoes it contain information absent from the stories covered in the last 7 days (list provided)? | 0.88 | 0.90 | 0.97 | 0.92 | 0.09 | +0.05 | +0.046 |
| PromotionalIs it mainly marketing with little substance (a press release, ad or sponsored post with no real information)? A launch post that carries real news is not promotional. | 0.03 | 0.05 | 0.02 | 0.03 | 0.03 | −0.19 | −0.006 |
| OverclaimDo the headline or conclusions claim more than the body supports? | 0.24 | 0.15 | 0.08 | 0.16 | 0.16 | −0.11 | −0.017 |
| Sum of contributions | +0.717 | ||||||
| Claim-check penalty | 0.000 | ||||||
| Total, as published | 0.717 | ||||||
Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.
The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.
<a href="https://rubric.hrih.si/editions/2026-10-30/7/"><img src="https://rubric.hrih.si/badge/2026-10-30/7.svg" alt="Ranked #7 on Rubric, 30 Oct 2026" width="238" height="28"></a>Claim check
Claim check: 8 no source given. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.
No source givenNumber
Time to a queryable index across the six engines ran from 1.9 minutes to 6.7 hours for the same million vectors.
The article gives no link or citation for it.
The article says “Time to a queryable index ran from 1.9 minutes to 6.7 hours for the same million vectors”
The source says No source to compare with.
No source givenNumber
pgvector returns 21.7× its single-client throughput under 32 clients, while the three InnoDB-based engines return about 1×.
The article gives no link or citation for it.
The article says “pgvector returns 21.7× its single-client throughput. The three InnoDB-based engines return about 1×”
The source says No source to compare with.
No source givenFirst, only or fastest
MariaDB 12.3 is the fastest engine in the run at recall 0.99, at 277 QPS.
The article gives no link or citation for it.
The article says “MariaDB 12.3 is the fastest engine in the run at recall 0.99, at 277 QPS”
The source says No source to compare with.
No source givenFirst, only or fastest
MariaDB, AliSQL and MHNSW-based engines produce the smallest index at 3.9 GiB, versus 5.7 GiB for raw float32 vectors; this is the only measurement where MariaDB leads.
The article gives no link or citation for it.
The article says “It is the only measurement in this run where MariaDB leads”
The source says No source to compare with.
No source givenBenchmark result
At 1% selectivity pgvector's default post-filtering returns fewer than k rows, giving recall 0.1034.
The article gives no link or citation for it.
The article says “Recall 0.1034 is a short answer rather than a bad one”
The source says No source to compare with.
No source givenNumber
Percona Search loses 39.2% of throughput after churn, the largest drop in the table, while recall barely changes (0.8432 to 0.8416).
The article gives no link or citation for it.
The article says “Percona Search loses 39%, the largest drop in the table, while recall goes 0.8432 to 0.8416”
The source says No source to compare with.
No source givenCausal conclusion
MariaDB 11.8 and AliSQL showed flat throughput and a roughly twentyfold median latency increase with more clients, indicating a server-side queue.
The article gives no link or citation for it.
The article says “Offering an engine more work and getting the same throughput at twenty times the latency is a queue, and that queue was on the server side”
The source says No source to compare with.
No source givenCausal conclusion
Percona Search is the only engine that gets faster as the filter tightens, at 71.9 QPS at 10% selectivity and 117.5 at 1%, because it pre-filters.
The article gives no link or citation for it.
The article says “Percona Search pre-filters, which is why it is the only engine that gets faster as the predicate tightens: 71.9 QPS at 10% selectivity, 117.5 at 1%”
The source says No source to compare with.
Also covered by
No other coverage was clustered with this story.
Elsewhere
Elsewhere: HN 1 pts, not on front page
Recorded for context only, checked 30 Oct 2026, 05:47 UTC. It played no part in the ranking.