
Semgrep and SonarQube found zero of 232 access-control bugs
Semgrep and SonarQube found zero of 232 access-control bugs
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
Application Security
Devsecops
Cybersecurity
Infosec
Semgrep and SonarQube found zero of 232 access-control bugs
Jost Faganel
5 min read·
2 hours ago
--
Both scored 0% on the vulnerability class attackers chain most. If a rule-based scanner is your main control, your backlog records what it couldn’t see. Here’s how to tell.
In the RealVuln v3.1 benchmark, two of the most widely deployed static scanners in the industry, Semgrep and SonarQube, found zero of 232 labelled broken-access-control vulnerabilities. Not a few. None. On missing authentication and authorisation, same result: zero of 173.
Broken access control is the number-one entry on the OWASP Top 10, and the class an autonomous attacker chains first. If a rule-based scanner is the main control standing between your code and that class of flaw, it isn’t catching a reduced number of them. It’s catching none.
That has a consequence most monthly security reviews get backwards. Every month a team presents a slide with one number on it: open vulnerabilities. If the number went down, the meeting goes well. But a backlog can only hold what a scanner detected, so the number on that slide answers a different question from the one the room thinks it is.
Disclosure: the figures below come from RealVuln v3.1, a benchmark built by Kolega Labs, where I work, and the metrics are adapted from a Kolega framework document. Both are public; the argument stands on the arithmetic, not on any product.
Three teams own identical codebases. Each runs a different scanner, then triages and fixes everything it reports. Which team has the most secure application?
The RealVuln corpus lets us put numbers on this. Its 140 repositories contain 4,138 labelled vulnerabilities. On that same code:
SonarQube, the other rule-based tool in the benchmark, also scored 0% on broken access control. The title’s “zero of 232” is Semgrep’s and SonarQube’s shared result on that family; the detailed per-tool figures are in the report’s Table 3.
The first team has the smallest backlog by a factor of eight, and the most unfixed vulnerabilities by a factor of nine. The backlog number says nothing about that.
Two caveats. The corpus is dense with vulnerabilities by design, so the absolute numbers don’t transfer to your estate. And these scanners aren’t your scanners. What does transfer is the shape: a queue can only hold what was detected, and on the same code, detection varies by an order of magnitude between tools.
Total findings open has a perverse property. It rises when detection improves and falls when detection degrades. Switch to a better scanner and your headline risk number goes up. Let a scanner silently fail on a large repository, as the last post showed can happen, and it goes down.
That makes it a reasonable workload indicator: it tells you how much triage and fixing sits in front of the team. As an exposure indicator it’s poor. Keep it on the slide, but label it honestly.
Remediation runs on a queue, and a queue is a register of the subset your tooling could see. That distinction becomes material when a regulator, auditor or insurer asks a simple question: how do you know the register is complete?
Join Medium for free to get updates from this writer.
If the real answer is “we don’t, because our tool is blind to the families an attacker would chain”, then the backlog isn’t a statement of your vulnerability position. It’s a statement about your instrument’s coverage. The next post looks at what the main financial regulators actually ask on this point, clause by clause.
Metrics fail audits for two reasons: they measure activity rather than exposure, or nobody can reproduce them. Each of these is defined closely enough that a third party could re-derive it from your records.
-
Detection completeness by family. The recall of your deployed scanning stack against a labelled corpus, per vulnerability family. Use a public benchmark, or the seeded-branch protocol from the last post on your own code. This is the metric that tells you what the backlog can and cannot contain.
-
Verified closure rate. Chains or findings closed by a failing re-test, divided by those confirmed in the period. A merged fix is not a closure. A later post in the series covers this.
-
Median time to verified close, by consequence band. Days from confirmation to a failing re-test. This answers the question an examiner actually asks: how long was it exploitable? CISA’s BOD 22–01 is a useful reference point: it sets deadline-based remediation for known-exploited vulnerabilities in US federal systems, measured in days.
-
Residual reachable critical outcomes. The number of distinct protected outcomes (from the routes method earlier in this series) that known routes still reach at the control-plane or crown-jewel rings, and how many routes reach each. This states exposure in consequence terms, not severity counts.
-
Closure-set coverage. The share of in-scope protected outcomes for which you have computed the smallest set of fixes that interrupts every known route. It shows that remediation is planned against routes, not tickets.
-
Chokepoint concentration. The fraction of routes to a given ring that pass through a single finding or position. High concentration means one fix removes many paths, and one failure opens many.
Download the six-metric definition card (one-page PDF)
The risk in this argument is that it sounds like a team explaining why its numbers look bad. Three habits avoid that.
NIST’s guidance on patch management and on security measurement makes the same underlying point: measure whether the risk response worked, not only whether it happened.
Next: what DORA, the FFIEC handbook and MAS’s guidelines actually say about this, quoted rather than paraphrased.
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。