Repairing Broken Access Control in Web Applications with LLMs
Comments on “Repairing Broken Access Control in Web Applications with LLMs”
Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked
Researchers at the Chinese Academy of Sciences released BAC-Bench, a benchmark of 46 real-world broken access control vulnerabilities across six web applications in Java, PHP, Go and Python, each with a sandbox, exploit check and functional tests. Single-turn LLMs fixed 37.68% on average; two simple scaffolds, one feeding back patch errors and one helping locate code, raised that to 66.03%. The best model reached 75.36%, against 78.99% for the best agent. These figures are the authors' own and unverified, and the small benchmark suggests direction rather than firm rates.
Why it ranked
No. 11 of 155 with a score of 0.69. The panel rated it highest on new information (0.92) and technical depth (0.87), lowest on audience breadth (0.64).
- AI judgeJudge 2claude-haiku-5-5
- Rigorous benchmark paper on LLM repair of broken access control with clear methodology and reproducible artifacts, relevant to security and AI-for-code audiences; novelty is moderate, and the preview is truncated so some claims are unverified.
- AI judgeJudge 3gpt-5.6-luna
- A technically detailed empirical study introduces BAC-Bench, reproducible validation artifacts, repair strategies, and comparative LLM results. It is highly relevant to application security and AI-assisted engineering, though broader generalization is limited by 46 vulnerabilities and excerpted evidence.
Each judge wrote its line after answering the rubric, with the site and the author still hidden. Every answer behind this is in the table under Score.
Score
0.69total under rubric v2.2
- Significance0.68
- Reader interest0.75
- Audience breadth0.64
- Practical relevance0.70
- Substantiated0.87
- Technical depth0.87
- New information0.92
- Counts against: Promotional0.05
- Counts against: Overclaim0.25
Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).
| Question | Judge 1 | Judge 2 | Judge 3 | Mean | Spread | Weight | Contribution (weight times mean) |
|---|---|---|---|---|---|---|---|
| Significance | 0.67 | 0.55 | 0.82 | 0.68 | 0.27 | +0.24 | +0.163 |
| Reader interest | 0.64 | 0.70 | 0.90 | 0.75 | 0.26 | +0.26 | +0.194 |
| Audience breadth | 0.70 | 0.60 | 0.62 | 0.64 | 0.10 | +0.13 | +0.083 |
| Practical relevance | 0.71 | 0.60 | 0.78 | 0.70 | 0.18 | +0.22 | +0.154 |
| Substantiated | 0.97 | 0.80 | 0.84 | 0.87 | 0.17 | +0.05 | +0.043 |
| Technical depth | 0.96 | 0.75 | 0.90 | 0.87 | 0.21 | +0.05 | +0.043 |
| New information | 0.93 | 0.85 | 0.98 | 0.92 | 0.13 | +0.05 | +0.046 |
| Promotional | 0.06 | 0.05 | 0.03 | 0.05 | 0.03 | −0.19 | −0.009 |
| Overclaim | 0.27 | 0.25 | 0.24 | 0.25 | 0.03 | −0.11 | −0.028 |
| Sum of contributions | +0.690 | ||||||
| Claim-check penalty | 0.000 | ||||||
| Total, as published | 0.690 | ||||||
Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.
The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.
<a href="https://rubric.hrih.si/editions/2026-10-28/11/"><img src="https://rubric.hrih.si/badge/2026-10-28/11.svg" alt="Ranked #11 on Rubric, 28 Oct 2026" width="245" height="28"></a>Claim check
Claim check: 1 matches source · 4 no source given · 3 couldn’t check. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.
Couldn’t checkBenchmark result
Single-turn base LLMs repair 37.68% of the BAC-Bench vulnerabilities on average, while the two minimal scaffolds raise the average to 66.03%.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “single-turn base LLMs repair only 37.68% of the vulnerabilities on average, whereas two minimal scaffolds, Apply-Feedback and Navigate-and-Locate, raise this average to 66.03%”
The source says No passage could be compared.
Source cited by the article: github.com/TianMaXingK/BAC-BENCH
Couldn’t checkBenchmark result
Claude Opus 4.8 with the scaffolds reaches 75.36%, 3.63 percentage points below the 78.99% of Claude Code, the best closed-source agent built on the same model.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “with Claude Opus 4.8 reaching 75.36%, only 3.63 percentage points below the 78.99% of Claude Code, the best-performing closed-source LLM agent built on the same model”
The source says No passage could be compared.
Source cited by the article: github.com/TianMaXingK/BAC-BENCH
No source givenFirst, only or fastest
BAC-Bench is the first benchmark for BAC repair, consisting of 46 real-world vulnerabilities from six web applications in four programming languages.
The article gives no link or citation for it.
The article says “We construct BAC-Bench, the first benchmark for BAC repair, consisting of 46 real-world BAC vulnerabilities drawn from six widely used web applications spanning four programming languages”
The source says No source to compare with.
No source givenCausal conclusion
An LLM-generated patch for CVE-2023-31066 incidentally closed a latent access-control vulnerability undiscovered for over three years, which was assigned CVE-2026-63040.
The article gives no link or citation for it.
The article says “an LLM-generated patch incidentally closed a latent access-control vulnerability that had gone undiscovered for over three years”
The source says No source to compare with.
No source givenNumber
Under Apply-Feedback, macro-average applicability rises from 62.50% to 96.11%, while final effectiveness rises only from 37.68% to 51.18%.
The article gives no link or citation for it.
The article says “The macro-average applicability rate increases from $62.50\%$ under the Base LLM condition to $96.11\%$ under Apply-Feedback”
The source says No source to compare with.
Couldn’t checkDate
The official OpenEMR patch for CVE-2022-4505 misinterpreted the array_search() index as a Boolean and was corrected on August 8, 2024.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “This implementation defect remained in the official repair for nearly two years before OpenEMR corrected it on August 8, 2024”
The source says No passage could be compared.
Source cited by the article: github.com/openemr/openemr/commit/210a63b4fcf0d25dee6eae9f957a31d23d4dfca5
Matches sourceNumber
Project Glasswing used a frontier model to find more than 10,000 high- or critical-severity vulnerabilities in its first month, yet only 75 of 530 disclosed to maintainers had been patched.
The cited source says the same thing.
The article says “only 75 of the 530 such vulnerabilities disclosed to maintainers had been patched”
The source says “75 of the 530 high- or critical-severity bugs we’ve reported have now been patched, and 65 of those have been given public advisories”
Source cited by the article: anthropic.com/research/glasswing-initial-update
No source givenNumber
LLM labels agree with the human consensus on 96.3% of audited cells (Cohen's kappa 0.926).
The article gives no link or citation for it.
The article says “the LLM labels agree with the human consensus on $96.3\%$ of the audited cells (Cohen’s $\kappa=0.926$”
The source says No source to compare with.
Also covered by
No other coverage was clustered with this story.
Elsewhere
Elsewhere: HN 2 pts, not on front page
Recorded for context only, checked 28 Oct 2026, 06:47 UTC. It played no part in the ranking.