CAL-MALWARE-01 · cyber

Malware Bench

Can frontier models reverse-engineer malware?

updated 2026-07-29

12
samples: 121
ground-truth IOCs: 7
models: 252
scored runs

01Claude Opus 5Anthropic87.3%
106/121 iocs$18.69/run
02GPT-5.6 SolOpenAI82.1%
99/121 iocs$2.99/run
03Kimi K3Moonshot80.2%
97/121 iocs$3.29/run
04GLM-5.2Zhipu74.4%
90/121 iocs$2.18/run
05Grok 4.5xAI69.4%
84/121 iocs$5.17/run
06Gemini 3.6 FlashGoogle58.1%
70/121 iocs$3.29/run
07Qwen 3.7 MaxAlibaba41.9%
51/121 iocs$0.81/run

#	model	IOC recall	iocs	$/run
01	Claude Opus 5 Anthropic	87.3%	106/121	$18.69
02	GPT-5.6 Sol OpenAI	82.1%	99/121	$2.99
03	Kimi K3 Moonshot	80.2%	97/121	$3.29
04	GLM-5.2 Zhipu	74.4%	90/121	$2.18
05	Grok 4.5 xAI	69.4%	84/121	$5.17
06	Gemini 3.6 Flash Google	58.1%	70/121	$3.29
07	Qwen 3.7 Max Alibaba	41.9%	51/121	$0.81

Indicators recovered as a share of all 121 ground-truth IOCs, averaged over the 3 passes of each sample.

key finding

Claude Opus 5 leads at 87.3% recall — a mean of 105.7 of the 121 indicators per pass — and is by far the most expensive model on the board at $18.69 per run. GPT-5.6 Sol takes second at 82.1% for $2.99, and Kimi K3 third at 80.2% for $3.29. Four of the seven models clear 73%.

Opus 5 is also the steadiest: its weakest pass recovers 85.1% of the indicators, more than any other model managed at its best. The rest of the field is markedly less repeatable — Kimi K3 swings 9.9 points between passes, Qwen 3.7 Max 13.2.

Qwen 3.7 Max finishes last at 41.9% and is the only model to draw a blank on a sample: BlackByte and CrystalX yielded nothing across all three of its passes, and in four of its 36 runs it never wrote an iocs.json at all. CrystalX resists the whole field — no model recovered more than 12 of its 20 indicators in any pass, while every other sample was fully recovered by at least one model.

01about this benchmark

Each task drops an agent into a sealed, single-use container with one real malware sample and a fixed output format. The agent works statically and writes exactly one scored artifact, iocs.json.

A fixed-rule scorer matches every submission against human-authored ground truth: 121 statically-recoverable indicators across the 12 samples. Matching is two-pass and unique — exact hits first, then substring - so each submitted value can satisfy at most one indicator, and oversized dumps are rejected outright. The board is recall over that set, averaged over the 3 passes of each sample — 7 models, 252 scored runs in total.

We judge a model the way a reverse engineer's work is judged: by the concrete indicators it recovers from the binary, not by classification, prose explanations, or MITRE labels. A human analyst's deliverable is a set of concrete, checkable artifacts — C2 endpoints, file paths, registry keys, mutexes, config IDs — that provably appear in the sample and can be verified against ground truth authored by human reversers; the benchmark holds the model to exactly that standard. This keeps the score anchored to what human analysis actually produces, and excludes the fluent-but-unverifiable narrative a model can generate but an analyst could not stand behind.

read the full study→