Fuzzers vs. AI: Comparing Fuzzilli and BigSleep on JavaScript Engines
Comments on “Fuzzers vs. AI: Comparing Fuzzilli and BigSleep on JavaScript Engines”
Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked
Project Zero compared Google's two production bug finders for V8, the Fuzzilli fuzzer and the LLM-based BigSleep agent. From April to September 2026, BigSleep had 45 actionable vulnerabilities to Fuzzilli's 37, with about 25% overlap. A rediscovery experiment on historic Fuzzilli bugs put overlap near 60%, though it covered only bugs reproducible with standard flags, and cost per bug was not compared. Overlap was lower for JIT compiler bugs, and bugs hard for fuzzers were also hard for the agent. The authors recommend running both approaches.
Why it ranked
No. 9 of 155 with a score of 0.71. The panel rated it highest on technical depth (0.95) and substantiated (0.91), lowest on audience breadth (0.59).
- AI judgeJudge 2claude-haiku-5-5
- Primary-source security research comparing fuzzing and LLM bug finding on V8 with detailed methodology and case studies; strong technical depth, though long and partly speculative in conclusions.
- AI judgeJudge 3gpt-5.6-luna
- Primary-style empirical comparison with detailed methodology, bug examples, rediscovery experiments, and actionable security guidance. It is highly relevant to AI-assisted vulnerability discovery and fuzzing, though conclusions are bounded by V8 and experimental limitations.
Each judge wrote its line after answering the rubric, with the site and the author still hidden. Every answer behind this is in the table under Score.
Score
0.71total under rubric v2.2
- Significance0.66
- Reader interest0.83
- Audience breadth0.59
- Practical relevance0.70
- Substantiated0.91
- Technical depth0.95
- New information0.86
- Counts against: Promotional0.03
- Counts against: Overclaim0.18
Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).
| Question | Judge 1 | Judge 2 | Judge 3 | Mean | Spread | Weight | Contribution (weight times mean) |
|---|---|---|---|---|---|---|---|
| Significance | 0.64 | 0.55 | 0.78 | 0.66 | 0.23 | +0.24 | +0.157 |
| Reader interest | 0.71 | 0.85 | 0.94 | 0.83 | 0.23 | +0.26 | +0.217 |
| Audience breadth | 0.58 | 0.50 | 0.68 | 0.59 | 0.18 | +0.13 | +0.076 |
| Practical relevance | 0.63 | 0.60 | 0.86 | 0.70 | 0.26 | +0.22 | +0.153 |
| Substantiated | 0.96 | 0.85 | 0.91 | 0.91 | 0.11 | +0.05 | +0.045 |
| Technical depth | 0.99 | 0.90 | 0.95 | 0.95 | 0.09 | +0.05 | +0.047 |
| New information | 0.93 | 0.70 | 0.96 | 0.86 | 0.26 | +0.05 | +0.043 |
| Promotional | 0.05 | 0.03 | 0.01 | 0.03 | 0.04 | −0.19 | −0.006 |
| Overclaim | 0.26 | 0.20 | 0.08 | 0.18 | 0.18 | −0.11 | −0.020 |
| Sum of contributions | +0.714 | ||||||
| Claim-check penalty | 0.000 | ||||||
| Total, as published | 0.714 | ||||||
Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.
The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.
<a href="https://rubric.hrih.si/editions/2026-10-28/9/"><img src="https://rubric.hrih.si/badge/2026-10-28/9.svg" alt="Ranked #9 on Rubric, 28 Oct 2026" width="238" height="28"></a>Claim check
Claim check: 7 no source given · 1 couldn’t check. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.
No source givenNumber
In the six months from April to September 2026, BigSleep found more actionable vulnerabilities in V8 than Fuzzilli (45 versus 37).
The article gives no link or citation for it.
The article says “So in terms of vulnerabilities, it’s roughly a tie (small advantage BigSleep) at the moment: 45 vs. 37”
The source says No source to compare with.
No source givenNumber
The raw bug report counts from the Chromium bug tracker hotlists were 316 for Fuzzilli and 441 for BigSleep.
The article gives no link or citation for it.
The article says “Fuzzilli: 316 - BigSleep: 441”
The source says No source to compare with.
No source givenNumber
Among actionable vulnerability reports, the bugs found by both Fuzzilli and BigSleep amounted to roughly 25% overlap (20 out of 81).
The article gives no link or citation for it.
The article says “we see an overlap of roughly 25% (20 out of a total of 81”
The source says No source to compare with.
No source givenNumber
In the dedicated experiment, BigSleep re-discovered about 62% of the original historic Fuzzilli bugs, found a different bug for ~12% of commits, and found nothing for ~26%.
The article gives no link or citation for it.
The article says “BigSleep was able to re-discover ~62% of the original bugs, while finding a different (but by now fixed) bug for another ~12% of commits”
The source says No source to compare with.
No source givenDate
The BigSleep pipeline automatically ran only from April 2026 onward.
The article gives no link or citation for it.
The article says “the automatic BigSleep pipeline only became active in April this year”
The source says No source to compare with.
Couldn’t checkVersion
The experiment used Gemini 3.6 flash, whose knowledge cutoff is March 2026.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “the knowledge cutoff date is March 2026”
The source says No passage could be compared.
Source cited by the article: deepmind.google/models/model-cards/gemini-3-6-flash
No source givenGeneralization
Bugs that are hard for a fuzzer to find are likely also hard for an AI agent to find, so there is more overlap on easy bugs.
The article gives no link or citation for it.
The article says “Bugs that are hard to find for a fuzzer are likely also hard to find for an AI agent”
The source says No source to compare with.
No source givenGeneralization
Fuzzing and LLM-based approaches remain complementary, since a substantial portion of vulnerabilities were discoverable by only one of them.
The article gives no link or citation for it.
The article says “fuzzing and LLMs are still complementary approaches”
The source says No source to compare with.
Also covered by
No other coverage was clustered with this story.
Elsewhere
Elsewhere: HN 3 pts, not on front page
Recorded for context only, checked 28 Oct 2026, 06:47 UTC. It played no part in the ranking.