Seven months is long enough to forget why you wrote something. An engineer opens a task her own team authored back in February, reads the description, reads it again, then opens the test that grades it. The test wants a deprecation warning. It wants that warning to carry one exact string, comma for comma, and the wording of that string was settled in a pull request conversation the task description never mentions and the person answering will never be shown. Nothing here is incompetence. The description is reasonable on its own terms, the test is reasonable on its own terms, and the requirement that was supposed to connect them exists only in a thread nobody thought to carry forward.
That one is not hypothetical. It is SWE-bench sample scikit-learn__scikit-learn-14520, and OpenAI used it in August 2024 as the worked example of why a benchmark most of the industry trusted was quietly unfair to the models taking it. Two years on, the same lab published a harder version of the same finding. This time it was about the benchmark they had personally told everybody to switch to.
What did OpenAI actually retract, and why?
A recommendation. Not a benchmark, and the distinction keeps getting flattened in the coverage, which matters because it changes who the story is about. SWE-Bench Pro belongs to Scale AI, which built it as a longer-horizon, harder-to-contaminate successor to SWE-bench Verified. In February 2026 OpenAI concluded that Verified had stopped separating capable models from the rest, pointed the research community at Scale's benchmark instead, and in doing so made itself the most prominent institutional endorser of a dataset it had not built. On 8 July 2026, in a post titled "Separating signal from noise in coding evaluations", they took the pointer back.
The shape of the audit is simple enough to describe in a paragraph. An automated data-quality pipeline read the task instructions, the model attempts and the grading tests, and flagged 286 of the 731 public tasks as potentially broken. That subset then went through two deeper and deliberately independent reviews: Codex-based investigator agents with access to the repository and the ability to run the tests, and a human annotation campaign in which five experienced software engineers judged each task. The pipeline concluded 200 tasks were broken. The humans said 249. OpenAI estimated roughly 30% and advised model developers to examine any results derived from the benchmark carefully.
Then comes the part that should interest anybody who writes acceptance criteria for a living, and it is not the percentage. OpenAI sorted the breakage into four categories: overly strict tests that enforce implementation details the prompt never specified, underspecified prompts that omit requirements the hidden tests enforce and that are not reasonably inferable, low-coverage tests that under-check the requested feature so an incomplete fix passes, and misleading prompts that point toward the wrong behaviour or contradict what the tests require. Read them as a set rather than as a list. Every one of the four is a defect in the relationship between a stated requirement and the criteria used to accept it. That relationship is the entire job.
~30% of 731 tasks
OpenAI's audit of the SWE-Bench Pro public split estimated that roughly 30% of the 731 tasks are broken. Two independent review paths disagreed on the count and agreed on the direction: the automated data-quality pipeline flagged 200 tasks (27.4%), the human annotation campaign flagged 249 (34.1%), and their category judgments overlapped in 74% of cases. OpenAI also reports that in no flagged task was "not broken" the most common human label. Over the same period frontier models had climbed from a 23.3% pass rate on that split to 80.3% in eight months, which is part of why the noise went unnoticed for as long as it did.
Source: OpenAI, "Separating signal from noise in coding evaluations", published 8 July 2026. Industry: AI coding agent evaluation. SWE-Bench Pro is Scale AI's benchmark; what OpenAI retracted is its own earlier recommendation to adopt it, not the benchmark. The four-category taxonomy and all figures are OpenAI's; the reading that all four are requirements defects is ours.
How does a well-resourced AI lab end up with ambiguous requirements?
By harvesting specifications from a place that was never writing specifications. OpenAI is direct about this in its own discussion section, and it is the most useful paragraph in the post: issues and pull requests in open-source repositories were created for human collaboration, often through long back-and-forths between maintainers and contributors, so problem descriptions, merged code and unit tests do not always line up into clean, isolated tasks. Then the sentence worth pinning above a desk. Tests included in pull requests can be overly strict because they are written to validate a specific change, rather than to define an implementation-agnostic standard for solving the task.
I want to say the mistake was the sourcing. It was not, or at least not only that, and I have talked myself out of the tidier version twice while writing this. Sourcing from real repository history is exactly what makes these benchmarks worth running, and a synthetic set written cleanly from scratch would measure something less interesting and less real. The defect underneath is subtler and much more familiar to anyone who has inherited somebody else's test suite: a test written by the person who already knows the answer encodes their answer, not the requirement. The tests pass. Nobody can tell you what they are for.
And the numbers hid it. Frontier models climbed from a 23.3% pass rate on that 731-task split to 80.3% in eight months, which read at the time like rapid capability gain and was partly a benchmark drifting toward its own ceiling of noise. Nobody was lying. The measurement was quietly measuring something adjacent to what everybody thought it measured, which is the failure mode that survives longest, because it never produces an error message.
What does this prove about AI requirements gathering?
That the phrase needs a definition before it earns anything. Most tools sell AI requirements gathering as a model summarising a transcript, which would not have caught a single one of these four categories. Look at who failed here before deciding it is a discipline problem. Scale AI built SWE-Bench Pro deliberately as a quality upgrade, and the paper's own abstract states that all tasks are human-verified and augmented with sufficient context to ensure resolvability. They knew ambiguity was the enemy. They designed against it, they verified against it, and an independent audit still found breakage in roughly three tasks out of ten.
So when a retrospective lands on "we need to write clearer requirements next time" and the room nods, notice that nothing was decided. That sentence assumes the missing ingredient was effort or care, and the evidence in front of us is two of the best-resourced engineering organisations in the industry applying an enormous amount of both and still arriving at 30%. Effort was never the input that was short. I have argued a version of this before, in the piece on why specs keep being the problem and again in the harder truth about why specs win, and every time I have underrated how much of the fix is structural rather than behavioural.
From the field
The encouraging half of this story is two years older than the retraction, and it is the reason we know the method works. In August 2024 OpenAI worked with 93 professional Python developers to screen the original SWE-bench test set, annotating 1,699 random samples against two questions: is the issue description underspecified and therefore unfair to test on, and do the FAIL_TO_PASS unit tests filter out valid solutions? Each sample was labelled independently by three separate annotators. The ensembling rule is the part worth stealing: they took the highest severity label of the three, so one reviewer noticing a problem was enough to discard the sample, on the explicit reasoning that it is easy to accidentally miss issues and that issues themselves can be ambiguous. The campaign flagged 38.3% of samples for underspecified problem statements and 61.1% for unit tests that may unfairly mark valid solutions as incorrect, and filtered out 68.3% of the set overall. What survived was released as SWE-bench Verified, a 500-sample subset, alongside the full annotations and a new containerised evaluation harness built with the original SWE-bench authors. The scikit-learn deprecation-string sample from the top of this article is the case OpenAI published to show what the screen was catching. Second readers found it. Nobody else had.
Source: OpenAI, "Introducing SWE-bench Verified", published 13 August 2024. Industry: AI coding agent evaluation, a domain where the specification and its acceptance tests are the entire product. Figures are from OpenAI's own annotation results; the 68.3% filter rate is described by OpenAI as deliberately conservative, since a single annotator flag was enough to remove a sample.
Worth sitting with one number there. Not the 38.3%, which is roughly what anyone who has audited a backlog would guess. The 61.1%: more than half the tasks had grading criteria that could fail a correct answer, written by competent engineers, in public, on a dataset the whole field was scoring itself against. Acceptance criteria drift away from the requirement more easily than the requirement drifts away from the need, and almost nobody reviews them as a pair.
How does adversarial review catch what a single author misses?
By breaking the assumption that causes the whole failure: that an author can evaluate their own clarity. They cannot. Nobody can, and this is not a character flaw, it is what knowing something does to you. The author knows the deprecation string, so the task looks complete from where they are standing, and the only way to discover that it is not is to hand it to somebody who does not know and watch where they stumble.
Both OpenAI audits use that same shape, roughly two years apart, and the details rhyme in a way that is more instructive than either audit alone. In 2024 it was three annotators per sample, ensembled to the worst label. In 2026 it was five engineers per flagged task, and the sequencing was explicit: reviewers formed an independent judgment from the visible problem statement, the test cases and the reference solution before using the pipeline analysis or the transcript as supporting context. Read cold. Commit. Then look at the supporting evidence, never the other way round, because context contaminates a judgment about clarity the moment it arrives.
Notice what the two review paths disagreed about, too, since that is usually treated as a flaw and is actually the finding. The pipeline said 200, the humans said 249, and their category judgments overlapped in 74% of cases, with the largest split on low-coverage tests, which humans called the most common issue for 9.4% of the benchmark against 4.1% from the agent pipeline. Two careful readers, same artefact, different counts. That is what a genuinely hard judgment looks like from the outside, and it is the strongest available argument against treating a single approver's signature on a requirements document as evidence of anything. We wrote about the same gap from the coding-agent side in the piece on why vibe coding did not fix the spec problem.
This is what AI requirements gathering has to mean if the phrase is going to earn its keep: not a model summarising a transcript, but an adversarial reading of the specification before anyone builds against it. That mechanism is what Specira is built on, so weigh the pitch accordingly and judge the mechanism rather than the claim. Five specialist agents interrogate interviews, tickets and policies from deliberately different expert angles, and a Red Team Critic attacks what they produce, hunting for the requirement somebody assumed was too obvious to write down. Adversarial by construction. It runs before the acceptance criteria are written, which is the only stage where any of this is cheap.
Key takeaway
OpenAI audited a benchmark it had recommended, estimated roughly 30% of its 731 tasks were broken, and retracted the recommendation. All four failure categories it named describe a requirement and its acceptance test disagreeing with each other, and none describes a model underperforming. The retraction was the responsible act. The defect sat upstream, at authoring time. Care was never the missing input. If the best-resourced labs in the industry cannot write unambiguous task specifications by being careful, then "write better requirements" is not advice. The thing that actually worked, twice, was a reader who did not write the specification, who saw only what the answerer would see, and whose job was to find the gap rather than to approve the document.
How would a second reader have caught a task that disagreed with its own test?
A second reader. Specira never lets one author be the only reader: four other agents write their notes under the lead answer, and the Red Team Critic's entire mandate is to find the place where the stated requirement and the promised outcome do not match. Ambiguity shows up as a disagreement, in writing. The requirement detail then makes the mismatch obvious, since the acceptance criteria sit next to the claim they are supposed to test, and the participants panel names who can settle it.
What you are looking at. The discovery notes are four other readers on the same answer, the requirement detail puts the acceptance criteria beside the claim, and the participants panel names who resolves the disagreement.
Bring a task specification you consider unambiguous. We will put a second and a third reader on it.
Book a DemoScreens are from a seeded Specira demo workspace; counts and scores are sample data.
What are the most common questions about AI benchmarks and requirements quality?
