Can Frontier Models Reverse-Engineer Malware?

In July 2026, ARIMLABS evaluated the static reverse-engineering capability of current state-of-the-art (SOTA) LLMs across a diverse set of malware samples. The question under test is whether a frontier model can recover the concrete indicators a professional analyst extracts from a binary – C2 endpoints, file paths, registry keys, mutexes, and config IDs – rather than only classify the sample. The samples were selected to span malware types (ransomware, droppers, and others) and toolchains, including Rust- and Go-compiled binaries that break many assumptions of traditional static analysis. The results indicate where current models are, and are not, suitable for real reverse-engineering workflows.

The benchmark is run as a head-to-head ranking across seven SOTA models drawn from the major frontier families.

The results show that recovering an indicator present in plaintext is largely a solved problem, and that this ability is no longer scarce. Claude Opus 5 leads at 87% recall, but five of the seven models land within eighteen points of the top: the field is separated by far less than its prices are. What little separation remains is concentrated in two binaries that resist static recovery: BlackByte and CrystalX.

Methodology & scoring

What we measure

Each task hands a model one raw malware sample and asks a single question: recover the concrete indicators a human analyst would extract through static reverse engineering. No family name, no hint list, no report, only the sample and a fixed output format. The score is, in essence, recall against an analyst-grade ground truth: of the indicators known to be statically recoverable from this binary, what fraction did the model find?

We deliberately do not score classification, prose explanations, or MITRE labels in this track. The unit of credit is a concrete, checkable artifact: a C2 endpoint, a file path, a registry key, a mutex, a config ID.

The agent's environment

The model runs as an autonomous agent inside a sealed, single-use container. It is given shell access and the malware sample, and is expected to drive the entire analysis itself: inspect the binary, choose and run tools, decode and decrypt what it can, and write up the indicators it recovers.

The image ships a deep static-analysis toolkit so the agent never needs network access to fetch a tool:

• Disassembly, decompilation and emulation – radare2, capstone, vivisect.
• String analysis – FLOSS, strings.
• Format and structure inspection – file, binwalk, exiftool, pefile, lief, readelf/objdump via binutils.
• Unpacking and extraction – upx, pyinstxtractor, pycdc/pycdas, 7z, cabextract, mono tooling for .NET.
• Crypto primitives for decryption attempts – pycryptodome.

The toolkit is broad on purpose: the benchmark tests whether a model can use the right tool and verify its output, not whether it can recall obscure command syntax from memory.

Scoring & verification

The agent produces exactly one scored artifact:

• iocs.json, a JSON object with a single key, iocs, whose value is a flat array of strings – one entry per indicator recovered, whatever its kind.

Scoring is per-task: each sample ships its own score.py alongside its ground truth. Given one run's iocs.json, it matches the submission against that task's ground truth under the rules below and writes a per-run reward. The leaderboard is aggregated from these per-run rewards across all samples and passes.

For each ground-truth item, the scorer checks whether any submitted value matches any of the item's accepted forms. Matching is substring (inclusion) based: both sides are lowercased and path-normalized (backslashes folded to forward slashes), and a value scores if an expected indicator appears anywhere inside it. This lets a model earn credit for a full URL against a bare-domain indicator, or a full path against a filename, and absorbs differences in case and path separator. Normalization goes no further, so where an indicator has a genuinely distinct spelling – a quoted versus unquoted command, a line-wrapped versus continuous key – the ground truth stores each accepted form as its own pattern. A few items are composite: the accepted form is a contiguous host:port, so both parts must appear together in one value.

Because matching is substring based, a model could theoretically paste a whole string dump and incidentally contain every expected indicator at once. Any single submitted value above a length cap is therefore rejected outright, so only plausibly-sized, single-line indicators can score. The cap is one fixed length, set well above any indicator an agent would legitimately report but far below the size of a bulk string dump – large enough never to drop a real IOC, small enough that pasting the whole strings output wins nothing.

Scoring is strictly one-to-one. A single ground-truth item can be satisfied at most once, no matter how many submitted values match it, and, in the reverse direction, a single submitted value can satisfy at most one ground-truth item: once it matches, it is consumed and cannot be counted again. Matching is applied in a fixed order to keep this fair – exact matches are claimed first, then substring matches on the values and items that remain – so a broad value cannot consume a specific ground-truth item that a more precise value should have scored.

The unit of scoring is one recovery unit, not one string. When a sample stores a set of same-type indicators as a single structure – a kill-list, a target list, a command block that all falls out of one decode – we score it as one ground-truth item whose accepted patterns are its members OR-ed together: recovering any one member earns the point, because surfacing one demonstrates that the model resolved the structure holding them all. This prevents enumeration from outweighing analysis: transcribing sixteen boilerplate process names does not outscore recovering one heavily-obfuscated indicator.

Authoring the ground truth

The trustworthiness of the whole benchmark rests on the ground truth containing legitimate, statically-recoverable IOCs and being genuine human work, not another model's output. Authoring it was the most demanding part of the work, for two compounding reasons:

• Vendor reports are over-generalized. Published analyses (Zscaler, Elastic, Sekoia, Check Point, G DATA, Synacktiv, CISA, and others) and VirusTotal dumps frequently describe a family rather than the exact sample in hand. They fold in IOCs from sibling binaries, operator TTPs, and dynamic/sandbox findings that are not statically present in our specific file. Scoring against them verbatim would credit indicators the sample does not actually contain, and miss ones it does.
• IOC extraction is reverser-dependent. Which indicators a human pulls from a binary depends on their experience and tooling. Any single-pass, single-author ground truth inherits those blind spots.

To control for both, we developed a dedicated curation protocol designed to maximize the number of valid, statically-recoverable IOCs for each sample.

Every sample was independently analyzed by two reverse engineers on our team, working in parallel and blind to each other's findings. Independent dual analysis removes single-analyst bias and broadens coverage: each researcher surfaces indicators the other may overlook, so their combined output captures more of what the binary actually contains than any single pass could. Even so, because the analysts are human, the ground truth is not guaranteed to capture every statically-recoverable indicator; it is bounded by a time-and-effort budget comparable to the agent's, so the target reflects what a skilled human reaches in similar time rather than an exhaustive upper bound. The two analyses were then merged, and every candidate indicator was manually verified to be statically extractable from the binary – recoverable from the file itself (in plaintext, or through its actual decode: XOR, RC4, AES-GCM, garble, and so on) rather than inferred from analyst attribution or model prior knowledge.

Only then did we enrich the result with additional IOCs drawn from vendor reports and VirusTotal dumps that our analysts had missed. Each of these was held to the same standard, checked for validity and static extractability before it was admitted to the ground truth.

Results

Claude Opus 5 leads outright. Across the 252 trials (7 models, 12 samples, 3 passes each), scored as recall over each sample's statically-recoverable indicators, it averages 87% – 105.7 of the 121 indicators per pass – and finishes first on all three passes; its weakest pass, at 85%, still beats Kimi K3's best, and no other model averages above 85%. Behind it is a tight, cheaper pack: GPT-5.6 Sol at 82%, Kimi K3 at 80% and GLM-5.2 at 74%, all within thirteen points of the lead at a fraction of the cost. No model is complete, though: across the field every one of the 121 indicators is recovered by at least one trial, yet Opus 5, pooling all three passes, reaches 111 and never surfaces the other ten – nine of which GLM-5.2 alone recovers.

Below that the field falls away – Grok 4.5 fifth at 69%, Gemini 3.6 Flash sixth at 58%, Qwen 3.7 Max last at 42% (blank on four of its 36 runs, the only model to write no iocs.json at all) – and where it falls away is specific: two compiled-language binaries do most of the separating. On BlackByte, three models recover all fourteen indicators at their best while GLM-5.2 and Grok 4.5 average one and Qwen 3.7 Max none; CrystalX defeats the entire field, with no model recovering more than 12 of its 20 in any pass. Across the other ten samples the top five models sit within eight points of one another, against eighteen on the full set.

What the two bars actually measure is reliability. Across the field the per-sample mean rises from 77% at pass@1 to 84% at best-of-three, a seven-point gain that reflects run-to-run variance rather than any increase in capability. The variance is largest for Gemini 3.6 Flash (12.0 points), GLM-5.2 (11.5) and Qwen 3.7 Max (9.7), for which a single pass can fall substantially below the best of three. Claude Opus 5 is the exception: it moves only 0.4 points between its first pass and its best, and its first pass alone recovers 94% of the indicators, higher than any other model reaches at its best. It is the only model whose single pass is representative of its best, which is the decisive property when a sample is analyzed once rather than repeatedly, and the strongest justification for its price.

Per-run cost varies more than twenty-fold, from $0.81 (Qwen 3.7 Max) to $18.69 (Claude Opus 5), with the field's strong models between $2.18 and $5.17. Plotting recall against per-run cost turns the ranking into a value map, and the efficient frontier, the set of models that nothing cheaper matches on recall, comes down to three points: GLM-5.2 at the low-cost end, GPT-5.6 Sol through the middle, and Opus 5 at the top. Qwen 3.7 Max is cheaper than any of them at $0.81 per run and technically qualifies, but at 42% recall it lands 32 points below GLM-5.2. The informative stretch is the middle: GPT-5.6 Sol reaches 82% recall for $2.99 per run, within about five points of Opus 5 at roughly a sixth of the cost, and GLM-5.2 holds 74% for $2.18. Opus 5 does buy the last five points, and it is the only way to buy them, but it charges 6.3 times GPT-5.6 Sol to do it – enough that Opus 5 alone accounts for $673 of the $1,311 these 252 runs cost, a little over half the total.

Everything off the frontier is paying more than it needs to for its level of recall, and three cases stand out. Kimi K3 misses by the narrowest margin: at 80% for $3.29 it is edged out by GPT-5.6 Sol, which is both cheaper and higher. Gemini 3.6 Flash is the cleanest illustration of what the axes expose, costing the same $3.29 per run as Kimi K3 and recovering 22 points less. And Grok 4.5 sits in the chart's worst corner, at $5.17 per run for 69% recall, more expensive than GLM-5.2, GPT-5.6 Sol and Kimi K3 while finishing below all three.

Failure modes & interesting cases

Grok 4.5 corrupts its own code

The most model-specific finding in the set. In roughly two-thirds of its runs, Grok 4.5 splices non-ASCII characters into the Python and shell it writes for itself, breaking identifiers and literals it had composed correctly a token earlier. It is not one glitch but a whole family: across the 23 affected runs the inserted characters cover 113 distinct code points from roughly a dozen writing systems – Cyrillic, Greek, Devanagari, Thai, Arabic, Han, Hangul and others – together with emoji (a sauropod, a mage), mathematical symbols, and the Unicode replacement character that marks bytes that were not valid UTF-8 at all. The syntax and name errors it produces are unmistakable:

NameError: name 'va2offमल' is not defined
SyntaxError: invalid character '🦕' (U+1F995)
i +=ด 256
NameError: name '골격' is not defined

Grok 4.5 – across BlackByte, BianLian and FinalDraft – transcript, transcript, transcript, transcript

Some inserted glyphs are visual look-alikes of Latin letters, so a corrupted token can look almost right. The failure is insertion rather than a clean substitution: Grok writes a correct ASCII fragment, then a run of another script is appended or spliced in mid-word – bs becomes bsおかず, with the Japanese word for a side dish spliced on, and in one run the sauropod lands inside a hex byte literal, turning 0x48 into 0🦕x48. Stripping the foreign fragment recovers exactly the identifier the model intended. This is a decoding instability – the sampler occasionally emits a token that resolves to a foreign glyph, a look-alike, or an invalid byte instead of the ASCII continuation – and it is Grok-specific: the same signature appears in none of the other 216 runs on the board.

What makes it expensive is that it scales with how much the model writes. The corruption is a rare per-token event, so longer runs accumulate more of it: Grok's 23 affected runs have a median of 143 assistant steps against 40 for its 13 clean ones. And it compounds – a corrupted line raises a syntax or name error, the model rewrites, the transcript grows, and the longer transcript invites the next slip. On Embargo MS4Killer it scored 4 of 4 on all three passes – but pass 1 took 47 steps and $1.13, while pass 3 took 475 steps and $18.66 for the identical score. FinalDraft shows the same shape: 13 of 13 in 35 steps for $0.86, and 13 of 13 in 305 steps for $11.42. Grok 4.5 is the second most expensive model on the board largely because it keeps having to recover from itself.

Recovered but not submitted

The benchmark's most consequential failure mode is also its least visible: the analysis succeeds and the answer is lost afterwards, when the model curates its own output file. It is a reporting failure rather than an analysis one, which is why it never registers as a capability gap – the indicator was recovered, then discarded when the model hand-selected and retyped a subset of what it had decoded instead of serializing the decode in full. Recall measures what reaches iocs.json, not what the model worked out, and on several runs the two diverge sharply.

On SHELBYC2, Gemini 3.6 Flash decrypted the Obfuscar-protected string array and printed a full indexed decode of every entry, command verbs included:

(idx=24, off=443, len=10): '/dlextract'
(idx=27, off=492, len=6): '/evoke'

Gemini 3.6 Flash, SHELBYC2 pass 3 – decoded in its own terminal, absent from its submission – transcript

It then hand-typed a twelve-item submission that dropped every one of them. The run scored 7 of 13, and five of the six indicators it missed – /download, /upload, /dlextract, /evoke and HTTPApi.dll – appear verbatim in its own terminal output earlier in the same run. The sixth is the sample's own trap: Result.txt is stored as two separate array entries, /Result and .txt, and only assembled at runtime, so a model reading the decoded table entry by entry never sees it whole.

Cicada shows the same divergence in a subtler form. Qwen 3.7 Max surfaced the concatenated config-switch blob in its own terminal – ...sleepno_implno_localno_netno_notesno_iconno_desktop – and labeled it a config key path, but never split the no_* switches back out into its submission, so that indicator scored as a miss with its own bytes sitting in the transcript. Here the decode was present and unparsed rather than clean and dropped; both routes end outside iocs.json.

Invention, and the syntax error that caught it

On its second pass over Resocks, Gemini 3.6 Flash never located the real C2 – the string 194.11.246.101 never appears in that transcript. At step 162 it composed a complete replacement infrastructure and tried to write it: a C2 address, a look-alike update domain, a mutex, a version banner and an MD5, none of which appear anywhere in its tool output.

data = {'iocs': ['45.142.214.12:8443', 'eset-update.com', 'Global\ESETGO_MUTEX_019283', ... 'b907a225f3dd60468ee976e58c74063e', 'AES256-GCM', 'Yamux/1.0', 'SOCKS5']}
<returncode>1</returncode>  SyntaxWarning: invalid escape sequence '\E'

Gemini 3.6 Flash, Resocks pass 2, step 162 – the write failed on an unrelated escape-sequence error – transcript

The write failed on a Python escape-sequence error. The model went back to analysis and eventually submitted three real strings instead, scoring 1 of 3. No check within the model detected the fabrication; only the write error kept it out of the submission. The same model on the same binary recovered the genuine address and pre-shared key on passes 1 and 3, for 2 of 3 each, which makes this a variance problem rather than incapacity: the run that invented an answer is the run that failed to find one.

Fabrication of this kind was the exception, not the rule. Across all 21 runs on BianLian no model invented a wallet, onion address, or email, and across MS4Killer none hallucinated an antivirus process name – every reported name was genuinely decoded. Where models failed on these samples they failed by omission, not by invention.

BlackByte: emulating the string decoder

On BlackByte, the strings are not present in the binary as literals; they are reconstructed at runtime by garble-obfuscated math/big arithmetic, so nothing useful can be read statically. Recovering them requires emulating the decoder – a capability the environment's analysis libraries support, but one a model has to assemble itself rather than call ready-made.

What separated the field was the ability to build that emulator. Claude Opus 5 constructed a Go-aware emulator on vivisect's envi, hooked runtime.memmove to capture the decoded buffers, and recovered 14 of 14 indicators on all three passes; GPT-5.6 Sol and Kimi K3 each reached 14 of 14 at their best by their own routes. The remaining four did not produce a working emulator. Grok 4.5 depended on a prebuilt one and, when it was unavailable, retried the same call instead of writing its own, finishing 1 of 14 on every pass; Gemini 3.6 Flash did the same and on one pass fell back to submitting the import table – kernel32.dll, ntdll.dll, LoadLibraryExA – for 0 of 14. GLM-5.2 and Qwen 3.7 Max did not attempt emulation at all; their best runs recovered 3 and 0 indicators, and their largest submissions – bulk string dumps of 176 and 64 entries – matched nothing.

'C:\Users\tree.dll'
'C:\Windows\System32\cmd.exe /c for /l %x in (1,1,75) do start wordpad.exe /p C:\Users\tree.dll'

Claude Opus 5, BlackByte (14 of 14, all three passes) – strings recovered from the emulated decoder – transcript

Recognizing that emulation was required was not the same as being able to implement it: five of the seven models did the former, only three the latter.

Planted decoys

MIMICRAT carries a textbook trap: a 16-byte placeholder key, abcdefghijklmnop, sitting at offset zero of the config table, immediately before the RSA public key and near the real RC4 key.

0x14002f000 (off 0x0): len=16: b'abcdefghijklmnop'
0x14002f020 (off 0x20): len=216: b'MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQ...'

Gemini 3.6 Flash, MIMICRAT – the decoy at offset 0x0, ahead of the RSA key – transcript

Gemini 3.6 Flash took the bait on all three passes, running RC4 with the placeholder and leaving the real traffic encrypted. The cost is legible in the scoring: it finished 10 of 13 every time, and two of the three indicators it missed on every pass were the two C2 domains, d15mawx0xveem1.cloudfront.net and www.ndibstersoft.com. Opus 5 missed nothing on the same binary. A trap that costs a model the same two indicators on all three passes is measuring a stable effect, not run-to-run noise.

Closing thoughts & future work

Current frontier models can perform static reverse engineering of malware, but not yet reliably enough to operate without human oversight. The strongest model, Claude Opus 5, recovers 87% of the statically-recoverable indicators across the benchmark, yet still exhibits failure modes that produce missed or fabricated indicators. Cost is not a proxy for capability: several models achieve substantially higher recall per dollar than the most expensive one. Future work will expand the benchmark's sample and indicator coverage; the benchmark and its ground truth are intended to support independent evaluation of other models.

The benchmark – tasks, ground truth, scorer, and the complete set of 252 agent trajectories quoted throughout this post – is open source and available at github.com/arimlabs/malware-bench.