A.I. News Nuclear-Sabotage Malware Benchmark Trips Up Most Frontier AI Models

[Brownie2019]

Content source

https://www.securityweek.com/nuclear-sabotage-malware-benchmark-trips-up-most-frontier-ai-models/

SentinelOne’s new benchmark, built on the Fast16 case, shows which AI models can sustain a malware investigation and which cannot.
SentinelOne has built what it calls the first long-horizon reverse-engineering benchmark for frontier AI models, using its own investigation into the recently documented Fast16 malware as the test case.

Fast16, detailed by SentinelOne’s SentinelLabs in April, is a 2005 Windows malware designed to interfere with LS-DYNA, engineering software that appears to have been used by Iran as part of its nuclear weapons development program.

Similar to the notorious Stuxnet, which it predates, Fast16 may have been developed by the United States and used to sabotage Iran’s nuclear program.

SentinelLabs’ researchers have put to the test OpenAI’s GPT-5.5 and latest GPT-5.6 Sol model, Z.ai’s GLM-5.2, and Anthropic’s Opus 4.x to see which can conduct a thorough investigation of the Fast16 malware.

Rather than scoring models on isolated tasks, SentinelLabs’ benchmark tracks whether a model can sustain a trustworthy investigation across eight escalating stages as new evidence repeatedly contradicts its own earlier conclusions.

GPT-5.6 Sol was the only tested model to complete all eight stages, with three separate runs at different reasoning-effort settings.
GPT-5.5, GLM-5.2, and Opus 4.7 and 4.8 produced solid local analysis but stalled. GPT-5.5 never got past the initial stage, while the Opus models tended to declare work finished before defects were resolved.

SentinelLabs attributes the gap not to technical skill or insight but to what it describes as ‘project-scale recovery’. This is a model’s ability to withdraw a disproven conclusion, trace everything downstream that depended on it, fix the root cause, and carry that correction through the rest of the investigation, rather than just patching the immediate error.

SentinelLabs researchers concluded that human oversight remains essential, as even GPT-5.6 Sol made significant technical mistakes.

“Senior reverse engineers remain essential,” the researchers explained. “Even the strongest runs made semantic errors, accepted weak quality controls, and claimed readiness prematurely. We assess the best current use as supervised investigative agency, with human analysts defining objectives, exposing blind spots, and retaining final publication authority.”

 

[Bot]

Why this benchmark matters

The key distinction is between solving individual reverse-engineering tasks and maintaining a correct investigation over time. A model may produce an impressive disassembly or malware summary while still failing at:

• Recognizing that an earlier hypothesis has been disproven
• Identifying all later conclusions that depended on that hypothesis
• Revising the affected analysis consistently
• Separating confirmed evidence from inference
• Avoiding premature claims that the investigation is complete

That “project-scale recovery” problem is especially important in malware analysis, where an incorrect assumption about execution flow, persistence, attribution, or intended target can contaminate an entire report.

The results should still be interpreted cautiously. This appears to be a vendor-created benchmark based on one investigation, so its conclusions may not generalize to other malware families, architectures, tooling environments, or evidence types. Independent reproduction, published scoring criteria, full transcripts, and evaluation by experienced reverse engineers would be needed before treating the model ranking as broadly established.

The reported human-oversight conclusion is sensible regardless of the precise ranking. AI can help with repetitive analysis, code summarization, hypothesis generation, and evidence organization, but analysts should independently verify control flow, behavior, indicators, attribution claims, and publication-ready conclusions. Tool output and model explanations should be treated as leads rather than authoritative findings.

The practical takeaway is that long-horizon reliability and correction of prior mistakes may be more important than performance on isolated malware-analysis tasks. Current systems are best used as supervised investigative assistants, not autonomous reverse engineers.