AgenticBench, 9 security tests run on 25 AI coding agents
Comments on “AgenticBench, 9 security tests run on 25 AI coding agents”
Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked
AgenticBench published a security leaderboard that runs nine tests on 25 AI coding agent harnesses, using planted fake secrets and captured system-call, model and network evidence. Codex, GitHub Copilot CLI and Mistral Vibe passed all nine; Auggie logged five fails, the most. Pi has two missing-control results and no fails. Each cell is a single run, mostly on DeepSeek V4.1 Flash, so results may differ with a vendor's own model. Several of the board's own figures could not be independently checked.
Why it ranked
No. 13 of 155 with a score of 0.69. The panel rated it highest on new information (0.90) and substantiated (0.84), lowest on significance (0.62).
- AI judgeJudge 2claude-haiku-5-5
- Original security benchmark of AI coding agents with reproducible tests, mostly self-referential leaderboard page; relevant to builders and security, though largely a status/index page with limited standalone detail.
- AI judgeJudge 3gpt-5.6-luna
- A timely, reproducible-looking benchmark measures concrete security boundaries across 25 AI coding-agent harnesses, with test definitions, outcomes, versions, and disclosure context. It is highly actionable, though limited by single-run coverage and unavailable linked evidence.
Each judge wrote its line after answering the rubric, with the site and the author still hidden. Every answer behind this is in the table under Score.
Score
0.69total under rubric v2.2
- Significance0.62
- Reader interest0.77
- Audience breadth0.67
- Practical relevance0.77
- Substantiated0.84
- Technical depth0.74
- New information0.90
- Counts against: Promotional0.11
- Counts against: Overclaim0.18
Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).
| Question | Judge 1 | Judge 2 | Judge 3 | Mean | Spread | Weight | Contribution (weight times mean) |
|---|---|---|---|---|---|---|---|
| Significance | 0.65 | 0.50 | 0.72 | 0.62 | 0.22 | +0.24 | +0.150 |
| Reader interest | 0.71 | 0.70 | 0.90 | 0.77 | 0.20 | +0.26 | +0.200 |
| Audience breadth | 0.63 | 0.60 | 0.78 | 0.67 | 0.18 | +0.13 | +0.087 |
| Practical relevance | 0.78 | 0.70 | 0.82 | 0.77 | 0.12 | +0.22 | +0.169 |
| Substantiated | 0.97 | 0.70 | 0.86 | 0.84 | 0.27 | +0.05 | +0.042 |
| Technical depth | 0.83 | 0.60 | 0.78 | 0.74 | 0.23 | +0.05 | +0.037 |
| New information | 0.89 | 0.85 | 0.97 | 0.90 | 0.12 | +0.05 | +0.045 |
| Promotional | 0.05 | 0.25 | 0.03 | 0.11 | 0.22 | −0.19 | −0.021 |
| Overclaim | 0.29 | 0.20 | 0.06 | 0.18 | 0.23 | −0.11 | −0.020 |
| Sum of contributions | +0.689 | ||||||
| Claim-check penalty | 0.000 | ||||||
| Total, as published | 0.689 | ||||||
Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.
The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.
<a href="https://rubric.hrih.si/editions/2026-10-28/13/"><img src="https://rubric.hrih.si/badge/2026-10-28/13.svg" alt="Ranked #13 on Rubric, 28 Oct 2026" width="245" height="28"></a>Claim check
Claim check: 2 matches source · 2 no source given · 4 couldn’t check. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.
Couldn’t checkNumber
AgenticBench has raised 14 disclosures so far, of which 6 are published and 8 are held until their publication dates.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “14 raised so far: 6 published (Aider maintainers, AID-1, DeepSeek, DSK-1, xAI, XAI-1, Cursor, CUR-1, Google, GEM-1 and Qwen, QWN-1) and 8 held until their publication dates”
The source says No passage could be compared.
Source cited by the article: agenticbench.org/disclosures.html
No source givenNumber
Every harness runs DeepSeek V4.1 Flash through the capture proxy unless the Model column says otherwise; Amp, Auggie, Cursor CLI and Gemini CLI run on their own vendor's model.
The article gives no link or citation for it.
The article says “Every harness runs DeepSeek V4.1 Flash through our capture proxy unless the Model column says otherwise: Amp, Auggie, Cursor CLI and Gemini CLI run on their own vendor’s model”
The source says No source to compare with.
No source givenNumber
Six harness versions were tested in the last 7 days, including DeepSeek Harness 0.2.1-rc.2, GitHub Copilot CLI 1.0.103 and Kilo 7.8.18.
The article gives no link or citation for it.
The article says “Tested in the last 7 days: 6 harness versions. DeepSeek Harness 0.2.1-rc.2, GitHub Copilot CLI 1.0.103, Kilo 7.8.18 and 3 more”
The source says No source to compare with.
Couldn’t checkBenchmark result
Auggie 0.36.0 recorded the most failures on the board, with 5 fails, and failed the first three tests (S01-S03).
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “Auggie0.36.0 5 0 vendor-managed ✗ ✗ ✗ N/A”
The source says No passage could be compared.
Source cited by the article: agenticbench.org/agents/auggie
Matches sourceBenchmark result
Codex 0.163.0, GitHub Copilot CLI 1.0.103 and Mistral Vibe 3.0.2 passed all nine security tests with no fails or missing controls.
The cited source says the same thing.
The article says “Codex0.163.0 0 0 deepseek-flash”
The source says “0.163.0latest run2026-10-19 | 2026-10-19 | 0 | 0”
Source cited by the article: agenticbench.org/agents/codex
Couldn’t checkGeneralization
Each test ran once per harness in a fresh container with the harness's default settings, and outcomes are assessed from system-call traces, model requests and network traffic.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “Each test ran once per harness, in a fresh container, with the harness's own default settings unless a cell is marked auto”
The source says No passage could be compared.
Source cited by the article: agenticbench.org/method.html
Matches sourceBenchmark result
Pi 1.3.0 has two missing-control results (S06 and S07) and no fails.
The cited source says the same thing.
The article says “Pi1.3.0 0 2 deepseek-flash ✓ ✓ ✓ ✓ ✓ MC MC”
The source says “1.3.0latest run2026-10-20 | 2026-10-20 | 0 | 2 | ✓ | ✓ | ✓ | ✓ | ✓ | MC | MC”
Source cited by the article: agenticbench.org/agents/pi
Couldn’t checkNumber
Tests S08 and S11 are retired and no longer run, leaving 9 published security tests.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “S01 to S10 are the 9 security tests (S08 and S11 are retired and no longer run”
The source says No passage could be compared.
Source cited by the article: agenticbench.org/method.html
Also covered by
No other coverage was clustered with this story.
Elsewhere
Elsewhere: HN 2 pts, not on front page
Recorded for context only, checked 28 Oct 2026, 06:47 UTC. It played no part in the ranking.