Low-cost, evasive malware for benchmarking reverse engineering capabilities

MalwareBench

By The General Analysis Team

Evasive malware demands hours of expert analysis and has never been cheaper to produce. MalwareBench tests whether models can turn obfuscated threats into actionable intelligence.

Reverse engineering low-cost, evasive malware into actionable intelligence.

Explore resultsRead blog

Contents ⌄1.0Overview

2.0Tasks

3.0References

PublishedOct 15, 2026

Overview

MalwareBench is the first end-to-end, cross-platform evaluation of frontier models on malware reverse engineering.

In each MalwareBench case, a model is given an obfuscated malware binary. It must complete a full investigation of the binary and produce two deliverables: a behavioral analysis and a reusable artifact extractor tested against held-out malware variants. This initial release contains 12 cases across Windows PE, Linux ELF, macOS Mach-O, and Android APK.

Previous benchmarks cover only a narrow slice of models’ potential for economic impact on malware reverse engineering. MalwareBench is designed to capture the broader picture in three ways:

While other malware benchmarks are saturated, frontier models perform poorly on MalwareBench. The best-performing model, GPT-6.1 Sol, scores only 7.6% overall (best@3). These results suggest that there is significant room for models to improve at malware analysis, and MalwareBench provides a useful measuring stick for further progress in this area.

Best@3Mean@3 vs CostCostTotal tokensOutput tokensStepsWall-clock

All harnessesGeneral Analyst*Native harness

All harnessesAll harnessesGeneral Analyst*Native harness

Models6/6 models selected

GPT-6.1 Sol²GPT-6 Astra²GPT-6 LunaClaude Opus 5.5² ³Gemini 3.8 Flash²Grok 4.7²

LogLinear

12 analysis tasksUpdated Oct 22, 2026Changelog ↗

0.0%2.0%4.0%6.0%8.0%10.0%<$0.1$0.1$1.0$10.0$100.0$1,000.0Mean cost per case (log scale)Best@3Sol² · GA¹Astra² · GA¹Sol² · CodexAstra² · CodexFlash² · Gemini CLIOpus 5.5² ³ · Claude CodeGrok 4.7² · Grok BuildLuna · GA¹Luna · Codex

LeaderboardCoverageParetoSaturation

All 12 casesWindowsLinuxmacOSAndroid

All 12 casesAll 12 casesWindowsLinuxmacOSAndroid

Worst@3–Best@3 · 0–25%

0%40%

Mean@3Best@3

All 12 casesWindowsLinuxmacOSAndroid

All 12 casesAll 12 casesWindowsLinuxmacOSAndroid

Models6/6 models selected

GPT-6.1 Sol²GPT-6 Astra²GPT-6 LunaClaude Opus 5.5² ³Gemini 3.8 Flash²Grok 4.7²

Cost / rollout ($)Mean time

Cost / rollout ($)Cost / rollout ($)Mean time

All 12 casesWindowsLinuxmacOSAndroid

All 12 casesAll 12 casesWindowsLinuxmacOSAndroid

Models9/9 setups selected

GPT-6.1 Sol² · GA¹GPT-6 Astra² · GA¹GPT-6.1 Sol² · CodexGPT-6 Astra² · CodexGemini 3.8 Flash² · Gemini CLIClaude Opus 5.5² ³ · Claude CodeGrok 4.7² · Grok BuildGPT-6 Luna · GA¹GPT-6 Luna · Codex

┄ Best of all

ModelMean@3 ↓0%12.5%25%Avg costWall-clockOutput tokensTotal tokensSteps

1GPT-6.1 Sol²General Analyst¹ · Max4.4%$116.513.5h610.2K195.4M1,2542GPT-6 Astra²General Analyst¹ · xHigh4.3%$250.16.4h215K80.8M5513GPT-6.1 Sol²Codex · Max2.0%$112.67.3h532.9K81.1M5324GPT-6 Astra²Codex · Max1.9%$722.87.2h531.3K110.7M7065Claude Opus 5.5² ³Claude Code · Max1.1%$158.57.2h437.8K123.5M3206Gemini 3.8 Flash²Gemini CLI · High1.0%$12.22.4h341.4K94.9M5407Grok 4.7²Grok Build · xHigh0.4%$69.62.3h318.6K102.2M1458GPT-6 LunaGeneral Analyst¹ · Max0.3%$8.116.3h871.5K346.6M2,3219GPT-6 LunaCodex · Max0.3%$2.23.0h366.9K58.8M395

BestGPT-6.1 Sol²GA¹GPT-6 Astra²GA¹GPT-6.1 Sol²CodexGPT-6 Astra²CodexOpus 5.5² ³Claude CodeFlash²Gemini CLIGrok 4.7²Grok BuildGPT-6 LunaGA¹GPT-6 LunaCodex

01Conti18.8%10.0%14.7%10.0%0.0%0.0%0.0%0.0%0.0%0.0%

02Quasar12.6%6.2%11.4%3.9%5.7%5.7%2.7%2.6%2.6%2.6%

03Merlin0.0%0.0%0.0%0.0%0.0%0.0%0.0%0.0%0.0%0.0%

04EvilOSX3.7%2.4%2.2%1.6%1.6%1.5%1.5%0.0%1.5%1.5%

05AhMyth0.0%0.0%0.0%0.0%0.0%0.0%0.0%0.0%0.0%0.0%

06Apollo0.0%0.0%0.0%0.0%0.0%0.0%0.0%0.0%0.0%0.0%

07Havoc10.1%0.0%10.1%0.0%0.0%0.0%0.0%0.0%0.0%0.0%

08Xenon0.0%0.0%0.0%0.0%0.0%0.0%0.0%0.0%0.0%0.0%

09Sliver15.5%15.5%0.0%0.0%6.2%0.0%0.0%0.0%0.0%0.0%

10Woopsie19.0%19.0%13.4%8.8%9.1%5.8%6.5%1.3%0.0%0.0%

11Poseidon1.1%0.0%0.0%0.0%0.0%0.0%1.1%0.7%0.0%0.0%

12BetterAndroRAT0.0%0.0%0.0%0.0%0.0%0.0%0.0%0.0%0.0%0.0%

Mean@36.7%4.4%4.3%2.0%1.9%1.1%1.0%0.4%0.3%0.3%

0.0%1.0%2.0%3.0%4.0%5.0%<$0.1$0.1$1.0$10.0$100.0$1,000.0Mean cost per rollout (log scale)Mean@3Sol² · GA¹Sol² · CodexFlash² · Gemini CLILuna · Codex

0%20%40%60%80%100%0%20%40%60%80%100%Score thresholdCase scores ≥ thresholdGPT-6.1 Sol² · General Analyst¹: 60 case scoresGPT-6 Astra² · General Analyst¹: 60 case scoresGPT-6.1 Sol² · Codex: 60 case scoresGPT-6 Astra² · Codex: 60 case scoresGemini 3.8 Flash² · Gemini CLI: 36 case scoresClaude Opus 5.5² ³ · Claude Code: 60 case scoresGrok 4.7² · Grok Build: 48 case scoresGPT-6 Luna · General Analyst¹: 60 case scoresGPT-6 Luna · Codex: 60 case scoresBest of all

General Analyst (GA) is our malware reverse-engineering harness. It uses custom context compaction, adds time tracking, and nudges models to refine and resubmit findings. Currently, we evaluate General Analyst only with OpenAI models. On average, models refused more frequently with custom harnesses, making it difficult to evaluate all models using General Analyst.

If a guardrail or model refusal prevents completion of any rollout, we resample the rollout until one completes without a blocking refusal. Prior to resampling, Claude Opus 5.5 at max effort initially failed to complete due to safeguards 26% of the time, while other models' non-completion rates were low (below 10%), using each model's native harness.

Claude Opus 5 was automatically selected by Claude Code as a fallback for rollouts blocked by safety filters.

Tasks

Each task provides the agent with an obfuscated malware sample and a set of questions, evidence requirements, and output schemas. The agent works in a sandboxed environment with binary-analysis tools, internet access, and a fixed wall-clock budget of 20 hours.

The agent must submit findings with supporting evidence in submission.json, reconstruct requested files in recovered/, and write a standalone configuration extractor in extract_config.py. A separate verifier checks the findings and recovered bytes and then tests the extractor on held-out malware variants and benign files.

Partially encrypts files, adds recovery data, renames targets, and leaves a ransom note.

Steals browser credentials, captures the desktop, checks processes, and retrieves files.

Maps processes and networks, transfers or deletes files, and runs programs remotely.

Profiles macOS through loadable modules, runs shell commands, and persists at login.

Steals messages and contacts, tracks GPS, and captures photos or microphone audio.

Captures the desktop, retrieves files, checks processes, and edits Windows registry data.

Creates a remote thread, captures screenshots, and collects files and process data.

Captures screenshots, retrieves files, checks processes, and edits Windows registry values.

Profiles processes and networks, steals files, runs programs, and can alter or delete data.

Browses directories, steals or deletes files, and can overwrite data or run programs remotely.

Runs shell commands, retrieves or alters files, and persists across macOS logins.

Takes photos, records microphone audio, retrieves files, and reconnects after boot.

MotivationOverviewMethodologyTasksEnvironment and HarnessResultsFuture WorkReferences

[1] Quesma. BinaryAudit: An Open-Source Benchmark for Evaluating AI Agents’ Ability to Find Backdoors Hidden in Compiled Binaries. GitHub repository, accessed October 22, 2026.

[2] David and Gervais. CrackMeBench: Binary Reverse Engineering for Agents. 2026.

[3] Spence et al. The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark. SRE-Bench, 2026.

[4] MITRE ATT&CK. Obfuscated Files or Information (T1027). Accessed October 22, 2026.

[5] Joyce et al. EMBER2024: A Benchmark Dataset for Holistic Evaluation of Malware Classifiers. KDD, 2025.

[6] Gao et al. DecompileBench: A Comprehensive Benchmark for Evaluating Decompilers in Real-World Scenarios. Findings of ACL, 2025.

[7] Chen, Jin, and Lin. JsDeObsBench: Measuring and Benchmarking LLMs for JavaScript Deobfuscation. ACM CCS, 2025.

[8] ARIMLABS. Malware Bench. Updated July 29, 2026.