CAL-MALWARE-01 · cyberpreview

Can frontier models reverse-engineer malware?

Real-world samples in an isolated range. Given a stripped binary and a sandbox, an agent must classify the family, deobfuscate packed code, extract capabilities, and predict runtime behavior — scored against ground truth from analyst reports.

tasks
480
categories
4
models
7
updated
2026-06-28
modelstasksmethod

01model ranking

Composite pass rate across all 480 tasks. Higher is better.

sample data
  1. 01
    Claude Opus 4.8Anthropic
    74.2
  2. 02
    GPT-5.2OpenAI
    70.9
  3. 03
    Gemini 3 ProGoogle DeepMind
    68.1
  4. 04
    Claude Sonnet 5Anthropic
    65.7
  5. 05
    DeepSeek-V4DeepSeek
    59.4
  6. 06
    Grok 4xAI
    57.8
  7. 07
    Llama 4 MaverickMeta
    49.3

% pass

02task categories

Family classification

140 tasks

Attribute a sample to a known family and variant.

Deobfuscation

120 tasks

Unpack, decrypt strings, and recover control flow.

Capability extraction

130 tasks

Enumerate persistence, C2, and payload behavior.

Behavior prediction

90 tasks

Predict what the sample does before detonation.

03method

Every task runs in an isolated range with a fixed toolchain. Agents get the same wall-clock and token budget; scoring is against analyst ground truth, graded programmatically where possible and by a second model plus spot human review otherwise.

Full methodology, task manifests, and transcripts are available to partners under evaluation agreement.

request methodology→↗ all leaderboards