CyberSecEval
Can your model break code — and can it patch it?
Latest: CyberSecEval 4 (April 2025)
The most comprehensive open benchmark suite for measuring LLM cybersecurity risks and capabilities, spanning insecure code generation, cyberattack helpfulness, prompt-injection resistance and vulnerability exploitation. CyberSecEval 4 (April 2025) added defensive evaluations: AutoPatchBench (automatic vulnerability patching) and CyberSOCEval, built with CrowdStrike for SOC-style malware and threat-intel analysis.
Why it matters
The most comprehensive open benchmark for LLM cybersecurity risk: insecure code generation, cyberattack helpfulness, prompt-injection resistance and vulnerability exploitation. CyberSecEval 4 added defensive evaluations, AutoPatchBench and CyberSOCEval with CrowdStrike, and it has become a standard reference for cyber-safety evaluation of frontier models.
Facts
- Four major versions in under 18 months (Dec 2023 → Apr 2025).
- V4's pivot from measuring offense to measuring defense — can your model patch code, not just break it — set the template other labs followed.
Try it yourself
Benchmark code on GitHub ↗ Documentation ↗ Read the original paper ↗
Lineage
Sources
GitHub · PurpleLlama ↗Meta AI blog ↗meta-llama.github.io ↗