A score of zero on the OWASP Benchmark, and why it is in the README

Why measure on someone else's suite

PandaWard already has a rule benchmark. pandaward benchmark runs every built-in rule against a corpus of vulnerable and safe cases, prints true and false positives and negatives per rule, and CI fails when a rule misses a vulnerable case or fires on a safe one. A rule only counts as benchmarked with precision and recall of at least 0.95 on at least three vulnerable and three safe cases.

The weakness is obvious: I wrote the rules and I wrote the corpus. A corpus written by the rule author checks that each rule does what it was written to do. It says nothing about how much of a real codebase the rules would catch.

The OWASP Benchmark for Java is a public, labelled test suite that I did not write: 2,740 test cases in 11 categories, each one file, with an expected results file that marks it as a real vulnerability or not. About half the cases in each category are deliberately not exploitable, so a tool that flags everything scores no better than one that flags nothing. The score is the true positive rate minus the false positive rate.

How it was measured

tools/owaspbench/run.sh fetches BenchmarkJava at a pinned commit into a temporary directory outside the repository. The checkout is only read; no build script, Maven goal or other code from it is run. The tool walks it with the same inventory settings as pandaward scan, runs the built-in adapters in process, and maps each finding to a test case by file name and to a Benchmark category by rule or CWE. A case counts as flagged only when a finding of its own category is in its file, as the OWASP scorecard does.

Two details matter for honesty. A category with no PandaWard rule is shown as "not covered" rather than as a zero, and the summary also gives the average with those categories counted as zero, which is how the OWASP scorecard averages a tool. And the Benchmark's test case directory is not treated as test code, so the heuristics really do analyze every case; the tool checks and reports that.

The result

Six categories have at least one PandaWard rule: command injection, weak hashing, insecure cookies, SQL injection, weak randomness and XSS. In all six the rules found nothing and flagged nothing:

cmdi          TP 0  FP 0  FN 126  TN 125  score 0.00
hash          TP 0  FP 0  FN 129  TN 107  score 0.00
securecookie  TP 0  FP 0  FN  36  TN  31  score 0.00
sqli          TP 0  FP 0  FN 272  TN 232  score 0.00
weakrand      TP 0  FP 0  FN 218  TN 275  score 0.00
xss           TP 0  FP 0  FN 246  TN 209  score 0.00

Weak ciphers, LDAP injection, path traversal, trust boundary and XPath injection have no built-in rule at all. The built-in rules did produce 14 findings on the whole checkout, none in a test case: the project's own CI workflows, Dockerfile, routes, a tracked key store and one HTML sink in a JavaScript helper.

Why it is zero

The measurement is not broken; the rules are the wrong kind of check for this suite. PandaWard's built-in code checks are single-line regular expressions for well-known risky idioms, mostly in JavaScript, Python, PHP, Ruby and Go. The Benchmark is written to defeat exactly that: every case moves the input through variables, helper classes, collections and conditionals before it reaches the sink, and the safe variants differ from the vulnerable ones only in that data flow. A few concrete misses from the write-up:

• The command injection rule looks for Runtime.getRuntime().exec( on one line. The cases store the runtime in a variable and call r.exec(...) later, or use ProcessBuilder.
• The weak hash rule needs MD5 or SHA-1 and a password-like word on the same line. The cases call MessageDigest.getInstance(...) with the algorithm from a variable or a properties file.
• The cookie rule matches configuration such as secure: false, not Java's cookie.setSecure(false).

Telling the vulnerable and safe cases apart needs inter-procedural taint tracking, which PandaWard does not do.

Why publish a zero

PandaWard's whole premise is that a security tool should say what it did not establish. Its reports never count a check that did not run as a pass, and a model cannot confirm a finding. A README that quoted only the self-written corpus, where every rule passes, would make exactly the kind of claim the tool is built to avoid.

The zero is also useful. It draws a clear line between what the built-in rules are for, such as CI workflows, infrastructure, supply chain, secrets and idioms in the languages they target, and what they are not for. And it gives any future rule change a public baseline to report against.

Takeaways

• A benchmark you wrote yourself checks intent, not accuracy. Measure on a suite someone else built, and say which one.
• Report "not covered" separately from "scored zero", and give the average both ways.
• A result of zero with zero false positives is information. Publish it with the reason and with what users should do instead.