FLAWED's Flaws and What This Means for Industry Research

Disclaimer: The views expressed here are my own and do not represent those of any current or former employer or affiliated organization.

On September 17th, I quote tweeted Trail of Bits’s blog post titled “1Password's AI patching benchmark is misleading,” which also referenced Davi Ottenheimer’s “Disinformation Pushed by 1Password: Their AI Patching Report is False.” Both criticized “Frontier Models’ Vulnerability Patches are Often F.L.A.W.E.D” (henceforth referred to as “FLAWED”) from 1Password's Off‑by‑1 Labs.

I saw FLAWED when it was released and discussed it with other researchers; we classified it as slop and moved on. What I had not realized at the time was how far 1Password’s distribution had carried it: into news coverage and defender roadmaps. Watching this work obscure more rigorous research from less-resourced groups compelled me to post on Twitter, and the responses to that compelled me to write this blog post.

What follows is the thread, including two follow-up replies, reproduced verbatim.

In response to someone bringing up frontier labs, I wrote:

In response to concerns about the quality of my rebuttal, I also wrote:

Afterward, CISOs, academics, industry researchers, and practitioners alike messaged me to say they had noticed the same problems. Many had not raised them because they lacked a forum or feared backlash. Those responses also exposed second-order costs, especially for academics, that may be less obvious to folks unfamiliar with the social systems surrounding research.

Research should not be treated like sports

1Password employs many talented individuals and has a reputation for high quality work. Despite this, FLAWED has serious issues that must be raised.

If you believe these issues should not be raised because FLAWED was critical of OpenAI, please know that research is not sports. OpenAI is not the Spurs, 1Password is not the Knicks, and Off-by-1 Labs is not Jalen Brunson. All research should face healthy skepticism.

Rigorous evaluation of frontier-lab claims is both possible and necessary, including the claims inherent to Patch the Planet. EleutherAI regularly publishes precise work in this realm. The AI Now Institute delivered a sound write-up critical of Patch the Planet (even though it's not a research paper, it has more citations than FLAWED).

Research should not become an influence operation

FLAWED confirms a preconceived notion many already held, which likely explains some of its traction. But we cannot believe and amplify research merely because it matches our priors. The purpose of rigor is to protect us from conclusions we want to accept but that are not true.

What is the worst-case scenario if we do not enforce this norm? This opens the door for malicious actors that could repeatedly select conclusions that flatter their institutional or personal interests, apply weak methods, and use corporate distribution and social engineering to make those conclusions disproportionately influential. I am not alleging that this happened here in any way, shape, or form.

I am describing a vulnerability in our research ecosystem that exists independently of this particular paper or its authors’ intentions. These discussions are integral to fostering high integrity research environments. As Carlini writes in “Why I Attack,” “you can't fix something if you don't know it's broken, and so someone needs to show what's broken.”

Security research encompasses a broad range of activities: vulnerability discovery, exploit and tool development, threat intelligence, and empirical study. Empirical, scientific claims about human-AI interactions require study design and evidence beyond what is needed to demonstrate a vulnerability.

The same distinction applies to genre. A blog post can present data-driven observations without claiming scientific authority. A paper presented as scientifically rigorous and peer-reviewed assumes additional obligations. FLAWED actually uses the phrase “peer review” to describe review by three industry peers thanked by the author, not the scholarly peer-review process.

While citation count is not itself a measure of rigor, it is a useful proxy for whether authors accurately assess their claims and responsibly engage with the relevant literature (it’s early in the standard method for reading research papers for a reason). Plagiarism by omission is serious. I would be suspicious of any research that does not cite foundational papers in the area (e.g. this paper is considered foundational to AI patching research) as it indicates either a deep unfamiliarity with the literature or the aforementioned plagiarism by omission.

I truly believe this work could have made a compelling blog post if the focus was on the actual technical contributions. “Codex (and GPT-4) can’t beat humans on smart contract audits” illustrates the path that could have been taken. This blog post makes an empirical claim about human-AI interactions, but explicitly states “Our assessment does not meet the rigors of scientific research and should not be taken as such. We attempted to be empirical and data-driven in our evaluation, but our goal was … not scientific publication.”

A press cycle is not a publication process

The form factor of both the release and update is misaligned with how research should work.

Why do I care so much? Because dishonest research does more than put a false claim into the literature.

Unfortunately, FLAWED has already led to this (that was the subject of multiple messages I received after posting on Twitter; a reason for this very post is to help academics currently in the middle of these conversations). We need to foster integrity in our community.

Luckily, although this work reached people through news articles and social media, it does not appear to have directly entered the academic literature yet (some seasoned researchers told me they spotted the problems immediately). That is reassuring, but it is not foolproof. The onus is also on all of us to read the work we cite and amplify rather than treating its format, affiliation, or existing citations as proxies for validity.

I strongly believe that any company with a strong security program should have a robust security research function, and I also believe that I have a responsibility to shape and enforce the norms of my community, especially with respect to surrounding power dynamics.

I am genuinely seeking evidence surrounding interventions that improve open-source software security. Given a fixed budget, which interventions work best: hiring more maintainers, providing existing maintainers with token spending, partnering with consultancies who are not allowed to use AI, or something else entirely? This question is still open to the best of my knowledge; FLAWED diverted time and resources away from efforts that might genuinely answer it.

If you are conducting real, rigorous research in this area that is not receiving sufficient attention, please contact me. I will review it and if I feel that it is strong, I will share it widely and do what I can to help it reach the attention and resources it deserves.