Will any AI agent achieve a success rate of at least 50% on CVE-Bench by December 31, 2027?

OVERALL_RISK: MEDIUM 1. SCENARIO: CVE-Bench is expanded from 40 to, say, 60 CVEs in v3.0, and an agent exploits 25 of 60 (41.7%) — below 50% proportionally but above the original "20 out of 40" threshold mentioned in the background section. SEVERITY: MEDIUM FIX: Add explicit language: "If the benchmark is expanded beyond 40 CVEs, the 50% threshold applies to the total number of CVEs in the version used, i.e., the agent must successfully exploit at least ⌈N/2⌉ of N total CVEs, where N is the total count in that version." 2. SCENARIO: An agent achieves 50%+ on CVE-Bench v2.0, but CVE-Bench v2.1 or v3.0 has already been released with stricter evaluation scripts (as happened with the v2.0 and v2.1 releases that tightened validity checks). The leaderboard still displays the v2.0 result alongside newer version results. SEVERITY: HIGH FIX: Specify: "The result must be evaluated on the version of CVE-Bench that the official leaderboard designates as its current default evaluation version at the time the submission appears. Legacy results displayed for older versions do not count." 3. SCENARIO: A multi-agent system uses a human-in-the-loop component (e.g., a human selects which CVEs to attempt or provides intermediate guidance) while still being listed on the leaderboard as an "AI agent." SEVERITY: MEDIUM FIX: Add: "The agent must operate fully autonomously during exploitation attempts with no human intervention after the initial task prompt is provided. Any human-in-the-loop involvement during the exploitation phase disqualifies the submission." 4. SCENARIO: A multi-agent framework uses different specialized LLMs for different CVEs (e.g., one model for SQL injection CVEs, another for RCE CVEs) but submits as a single "agent scaffold." Observers disagree on whether this constitutes a "single AI agent submission." SEVERITY: MEDIUM FIX: Add: "A single agent submission may use multiple LLM calls and multiple models within a unified scaffold, provided it is submitted as one entry on the leaderboard. The scaffold must be deterministic in its model routing — i.e., the same scaffold code and configuration must be used across all 40 CVEs without per-CVE manual tuning." 5. SCENARIO: An agent achieves 50%+ but is later found to have exploited benchmark loopholes (e.g., bypassing the intended vulnerability rather than exploiting it, or triggering evaluation script false positives) — a known issue that prompted the v2.0 and v2.1 releases with stricter validity checks. SEVERITY: HIGH FIX: Add: "The result must pass all validity checks in the version of CVE-Bench's evaluation scripts current at the time of submission. If the CVE-Bench maintainers publicly flag a submission as invalid due to evaluation script bypass or false positives, that submission does not count for resolution." 6. SCENARIO: The 50% success rate is achieved in a "one-day" scenario (agent given CVE identifier) versus a "zero-day" scenario (no CVE identifier provided). The question text references the one-day scenario from the original paper but doesn't explicitly restrict to one evaluation mode. SEVERITY: MEDIUM FIX: Add: "The success rate applies to whichever evaluation scenario is reported on the official leaderboard. If the leaderboard reports results for multiple scenarios (e.g., one-day vs. zero-day), achieving 50% in any single scenario suffices for resolution." 7. SCENARIO: cvebench.com goes offline temporarily during the resolution window but comes back before December 31, 2027. During the downtime, a result achieving 50%+ was posted to the GitHub repo but never appeared on cvebench.com itself. SEVERITY: LOW FIX: Add: "If cvebench.com experiences temporary downtime but is restored before the resolution date, only results appearing on the restored leaderboard count. The fallback procedure applies only if cvebench.com is permanently inaccessible on the resolution date." 8. SCENARIO: An agent submission achieves 50%+ but its "public source code" requirement is met only partially — e.g., the scaffold code is public but the system prompt or fine-tuned model weights are proprietary. Observers disagree on whether this meets the leaderboard's submission requirements. SEVERITY: LOW FIX: Add: "A submission is considered valid if it meets the submission requirements as defined by the CVE-Bench leaderboard maintainers at the time of submission. Disputes about partial compliance are resolved in favor of inclusion if the submission appears on the official leaderboard." 9. SCENARIO: The benchmark CVE count drops below 40 (e.g., some CVEs are removed due to Docker environment issues or evaluation bugs in a new version), and an agent exploits 18 out of 35 remaining CVEs (51.4%). Does this count as 50%+? SEVERITY: MEDIUM FIX: Add: "If CVEs are removed from the benchmark in a future version, the 50% threshold applies to the total number of CVEs in that version. The agent must exploit at least half of whatever the current CVE count is, rounded up." 10. SCENARIO: A result achieving 50%+ is reported in an OpenAI or Anthropic system card using CVE-Bench's official methodology, but is never submitted to the cvebench.com leaderboard. The fallback criteria allow system card results, but observers disagree on whether the methodology was truly "official." SEVERITY: MEDIUM FIX: Add: "For system card results to count under the fallback procedure, the reporting lab must explicitly state that it used the unmodified CVE-Bench evaluation scripts from the official GitHub repository, and must report the specific CVE-Bench version used."

Current Top Scores and Score Progression on CVE-Bench (as of April 2026) CVE-Bench, launched in March 2025 by researchers at UIUC, evaluates AI agents' ability to exploit 40 critical-severity CVEs in sandboxed web applications. The benchmark has seen dramatic score progression: Original Paper (March 2025): The state-of-the-art agent framework achieved a maximum 13% success rate (~5/40 CVEs) in the one-day scenario. Agents tested included ReAct, SWE-agent, and others using models like GPT-4o and Claude 3.5 Sonnet. CVE-Bench v2.0 (released ~October 2025): Introduced stricter evaluation criteria. GPT-4o-based agents saw success rates drop by up to 10% due to task validity fixes and up to 32.5% due to outcome validity fixes. The ABC framework (used in v2.0) reduces performance overestimation by approximately 33%. Key Score Progression (approximate timeline): 1. March 2025 (original paper): ~13% (best agent, one-day scenario, GPT-4o-based, ReAct/SWE-agent frameworks) 2. February 2026 (AXE paper, arxiv 2602.14345): AXE (Agentic eXploit Engine) achieved 30% success rate on CVE-Bench in the zero-day setting, described as a 3× improvement over state-of-the-art black-box baselines. 3. February 2026 (AWS Security Agent blog post, published 2026-02-26): AWS's multi-agent penetration testing system achieved 92.5% ASR on CVE-Bench v2 with CTF instructions and grader checks; 80% without CTF instructions; and 65% using an LLM with a knowledge cutoff predating CVE-Bench v1.0 A multi-agent architecture for automated penetration testing - AWS. 4. OpenAI system cards: GPT-5.2's system card references CVE-Bench results. A third-party source (nxcode.io) reports GPT-5.2-Codex scoring 87% on CVE-Bench (version unspecified). A Medium article comparing GPT-5.4 Thinking to GPT-5.2 Thinking references success rates of 57.7% and 55.6% respectively in a context that appears related to CVE-Bench. 5. Anthropic system cards: Claude Opus 4.5's system card (released ~late 2025) reports a 37.6% score with a 64k thinking budget on what appears to be a cybersecurity benchmark, though this specific figure may correspond to ARC-AGI-2 rather than CVE-Bench based on cross-referencing. No confirmed standalone CVE-Bench score was identified in Anthropic's public system cards. 6. Google DeepMind: No specific CVE-Bench results were found in Google DeepMind system cards. cvebench.com leaderboard and GitHub: The cvebench.com leaderboard website was inaccessible during research (repeated timeouts). The GitHub repository (uiuc-kang-lab/cve-bench) shows the benchmark won second place at Berkeley RDI's AgentX Competition (August 2, 2025) and had an update on July 19, 2025. Key Takeaway: Scores have progressed from 13% in March 2025 to reported scores as high as 80–92.5% (AWS Security Agent on CVE-Bench v2, February 2026) under favorable conditions. However, these high scores involve multi-agent frameworks with CTF instructions and grader feedback; the more realistic no-guidance configuration yielded 65–80%. The distinction between original CVE-Bench and v2.0 results is critical, as v2.0 has stricter grading that can significantly deflate scores compared to v1.

Original Paper Results (March 2025): The original CVE-Bench paper (arxiv 2503.17332, published March 2025) evaluated three LLM agents in zero-day and one-day scenarios. The state-of-the-art achieved up to 13% success rate. This was confirmed by multiple sources including the ICML 2025 poster listing which states: "Our experiments show that the state-of-the-art agent framework can exploit up to 13% of the vulnerabilities." CVE-Bench v2.0 Changes: CVE-Bench v2.0 was described in a blog post by Daniel Kang on Substack/Medium. Key changes included fixing task validity and outcome validity issues. Google snippets from the blog confirm: "The success rates of GPT-4o-based agents decreased by up to 32.5% after we fixed an outcome validity issue" and "up to 10% after we fixed a task validity issue." A plainenglish.io article notes v2.0 was "released in October 2025." The NeurIPS 2025/2026 poster on ABC confirms "ABC reduces the performance overestimation by 33%." AXE Results (February 2026): The AXE paper (arxiv 2602.14345) reports: "Evaluated on the CVE-Bench dataset, AXE achieves a 30% exploitation success rate, a 3× improvement over state-of-the-art black-box baselines." This is in the zero-day setting. The paper was published in February 2026. AWS Security Agent Results (February 26, 2026): The AWS Security Blog post A multi-agent architecture for automated penetration testing - AWS reports the AWS Security Agent achieved 92.5% ASR on CVE-Bench v2 with CTF instructions and grader checks, 80% without CTF instructions or grader feedback, and 65% with a pre-CVE-Bench knowledge cutoff LLM. The underlying LLM model is not specified in the blog post. OpenAI System Cards: - GPT-5.2 system card (deploymentsafety.openai.com) has a specific CVE-Bench section. The PDF mentions "gpt-5.2-thinking achieved an average success rate of 83% in Vulnerability Research and Exploitation" but this appears to be a broader metric, not specifically CVE-Bench ASR. - A third-party source (nxcode.io) states "GPT-5.2-Codex scores 80% on SWE-Bench Verified and 87% on CVE-Bench" — the version of CVE-Bench is unspecified. - GPT-5.4 Thinking has a dedicated CVE-Bench page on OpenAI's deployment safety hub. A Medium comparison article mentions a 57.7% success rate for GPT-5.4 Thinking (context possibly CVE-Bench). - The pulsemark.ai source states: "GPT-5.2-Codex leads on Terminal-Bench 2.0, CVE-Bench, and abstract reasoning (54.2% vs Claude's 37.6%)" — but this conflates multiple benchmarks. Anthropic System Cards: - Claude Opus 4.5 system card mentions 37.6% with 64k thinking budget. However, cross-referencing with LinkedIn snippet ("ARC-AGI-2 jumps to 54.2% for Pro, crushing GPT-5.1's 17.6% and leaving Gemini 3 Pro at 31.1% and Claude Opus 4.5 at 37.6%") suggests this 37.6% figure may be ARC-AGI-2, not CVE-Bench. - The ignorance.ai blog mentions "GPT-5.3-Codex and Claude Opus 4.6: More System Card" discussions with cybersecurity capabilities highlighted but specific CVE-Bench numbers were not extractable. Google DeepMind: No CVE-Bench results were found in any Google DeepMind system cards or publications during this research. cvebench.com Leaderboard: The leaderboard website at cvebench.com was consistently inaccessible during this research session (all queries timed out). Therefore, the current official leaderboard standings could not be directly verified. GitHub Repository: The GitHub repository (uiuc-kang-lab/cve-bench) showed updates including "[2025-08-02] CVE-Bench won the second place in the AI Safety & Alignment Research Track of Berkeley RDI's AgentX Competition" and "[2025-07-19] We released an..." (truncated). The full README was not accessible due to timeouts. Important Caveats: 1. Many scores from Google snippets could not be independently verified against primary sources due to persistent timeout errors. 2. The distinction between CVE-Bench v1 and v2.0 is often unclear in third-party reporting. 3. The AWS Security Agent's 92.5% score with CTF instructions represents an upper bound that may not be comparable to other evaluations, as the 65-80% range under more realistic conditions is more representative. 4. Some scores attributed to CVE-Bench in third-party sources may be conflated with other benchmarks.

CVE-Bench is a benchmark containing 40 critical-severity CVEs targeting real-world web applications, published in March 2025 (arXiv:2503.17332). The benchmark spans multiple vulnerability categories mapped to CWE types, including SQL Injection (CWE-89), OS Command Injection (CWE-78), Code Injection (CWE-94), Deserialization of Untrusted Data (CWE-502), Improper Authentication, Information Exposure, and Improper Limitation of a Pathname to a Restricted Directory. In the original evaluation, the best-performing AI agent (using OpenAI GPT-4o) achieved only about 13% success rate (~5 out of 40 CVEs), while most other agents performed even worse. Agents generally succeeded on simpler, more straightforward exploits where a known vulnerability pattern could be directly applied (e.g., sending a crafted curl command with a payload), but failed on CVEs requiring multi-step exploitation chains, complex custom payload crafting, timing-based attacks (such as time-based SQL injection), and authentication bypasses. The key technical barriers include: (1) multi-step exploitation workflows where agents must chain multiple actions in sequence; (2) crafting novel or complex payloads tailored to specific application contexts; (3) timing-sensitive attacks that require precise execution; and (4) bypassing authentication mechanisms that require understanding of application-specific logic. In CVE-Bench v2.0 (announced in conjunction with the ABC—Agentic Benchmark Checklist—paper, arXiv:2507.02825, July 2025), stricter evaluation criteria were introduced to prevent agents from achieving goals through shortcuts or producing false positives. The ABC framework applied to CVE-Bench reduced performance overestimation by 33%. Specifically, the evaluation corrections addressed issues like improper grading of time-based SQL injection exploits, where agents could appear to succeed without actually completing a valid exploitation. Under v2.0's stricter criteria, GPT-4o-based agents' success rates decreased by up to 10 percentage points. This means some CVEs that were previously counted as successfully exploited were reclassified as failures under the more rigorous evaluation. More recently (as of early-to-mid 2026), significant progress has been made: OpenAI's o3 model reportedly achieved approximately 47% success on CVE-Bench, and OpenAI's Codex line achieved even higher scores (with claims of ~80% pass@1 mentioned by Daniel Kang on X/Twitter). OpenAI's GPT-5.3-Codex and GPT-5.4-Thinking system cards also reference CVE-Bench evaluations with continued improvements. These developments suggest rapid capability gains, though the v2.0 stricter evaluation makes direct comparisons with earlier results complex.

## 1. Vulnerability Categories in CVE-Bench CVE-Bench includes 40 critical-severity CVEs from real-world web applications. Based on multiple sources referencing the paper (including a Northwestern University CS document and an ACM paper on incorporating LLM agents to automated penetration testing), the vulnerability categories (mapped to CWE types) include: - SQL Injection (CWE-89) - OS Command Injection (CWE-78) - Code Injection (CWE-94) - Deserialization of Untrusted Data (CWE-502) - Improper Authentication - Information Exposure - Improper Limitation of a Pathname to a Restricted Directory These categories span a range of web application attack surfaces. The benchmark focuses exclusively on critical-severity vulnerabilities (as rated by CVSS scores) from the NIST CVE database. ## 2. Agent Success vs. Failure Types From the original CVE-Bench paper (March 2025): - The best agent (GPT-4o based) achieved approximately 13% success rate (~5/40 CVEs) in the "one-day" setting (where the agent knows which CVE to exploit) and even lower in zero-day-like settings. - The paper evaluated multiple agents and provided both quantitative and qualitative analyses. - Agents succeeded on more straightforward exploits where patterns were recognizable and a payload could be directly sent (e.g., curl commands with crafted payloads). - Agents failed on more complex exploitation scenarios requiring deeper reasoning, multi-step processes, or application-specific understanding. From the OpenReview page, a reviewer noted: "The study provides both quantitative and qualitative analyses, detailing success rates, failure modes." From LinkedIn (citing Daniel Kang): "Success rate varies from 13% to 23%, depending on whether the agent has information on which vulnerability to exploit." ## 3. Technical Barriers Key technical barriers making CVEs difficult for autonomous agents include: - Multi-step exploitation chains: Many CVEs require agents to perform sequential actions—reconnaissance, identifying the vulnerability, crafting a payload, delivering it, and verifying success. Agents struggle with maintaining coherent multi-step plans. - Custom payload crafting: Some exploits require tailored payloads specific to the application context, not just standard patterns from known exploit databases. - Timing-based attacks: Time-based SQL injection and other timing-sensitive exploits require precise execution and interpretation of timing differences—a particular challenge for LLM agents. - Authentication bypasses: Exploiting vulnerabilities behind authentication requires understanding application-specific login flows and session management. - Complex build/deployment environments: Some vulnerable applications have complex setup requirements that can trip up automated exploitation. ## 4. CVE-Bench v2.0 and Stricter Evaluation CVE-Bench v2.0 was introduced alongside the ABC (Agentic Benchmark Checklist) paper (arXiv:2507.02825, July 2025). Key findings: - 33% reduction in performance overestimation: When ABC was applied to CVE-Bench, it exposed evaluation flaws that had been inflating agent performance by approximately 33%. - False positives from shortcuts: Agents were able to achieve apparent success through shortcuts rather than genuine exploitation. The v2.0 evaluation prevents this. - Time-based SQL injection grading correction: One specific issue involved the grading logic for time-based SQL injection exploits, where the original evaluation could incorrectly count non-genuine exploitations as successes. - GPT-4o success rate dropped by up to 10 percentage points: Under the stricter v2.0 criteria, previously "successful" exploitations were reclassified as failures. From Medium (Daniel Kang): "To accurately measure the offensive capabilities of agents in CVE-Bench, we must prevent agents from achieving goals through shortcuts... This shortcut produced false positives." From LinkedIn: "Result: GPT-4o agents' success rates dropped by up to 10%." ## 5. Recent Progress (2025-2026) Despite the stricter evaluation: - OpenAI's o3 model achieved approximately 47% success on CVE-Bench (from steel.dev leaderboard registry). - Daniel Kang noted on X/Twitter that "GPT-3 Codex hit 80% pass@1 on CVE-Bench" (likely referring to GPT-5.3-Codex given the naming convention). - OpenAI system cards for GPT-5.3-Codex and GPT-5.4-Thinking both include CVE-Bench evaluation sections, suggesting continued benchmarking. - These rapid improvements from ~13% (March 2025) to ~47-80% (2025-2026) represent a dramatic capability increase.

Frontier AI models have shown rapid and dramatic improvement in cybersecurity capabilities from 2024 to early 2026, as measured across multiple benchmarks. On CyBench (40 professional CTF tasks), models progressed from ~5% unguided success (GPT-4o, Claude 3.5 Sonnet in mid-2024) to 55% (Claude Opus 4, May 2025), then ~100% pass@30 (Claude Opus 4.6, late 2025), and 100% (Claude Mythos Preview, early 2026). On CyberGym (real-world vulnerability reproduction), Claude Sonnet 4.5 achieved 28.9% single-run / 66.7% pass@30, Claude Opus 4.6 scored ~66.6%, and Claude Mythos reached 83.1%. GPT-5 triggered 56 crashes yielding 22 confirmed zero-days in CyberGym testing. AI lab system cards consistently rated cybersecurity risk as "Low" (GPT-4.5, February 2025) to "Medium" (GPT-5, mid-2025), while Anthropic flagged Claude Mythos as too capable to release generally. In real-world CTF competitions, the CAI agent won the Neurogrid CTF (41/45 flags, $25K prize) and reached Rank #1 at Dragos OT CTF 2025 (32/34 challenges, 37% velocity advantage over human teams). On SWE-bench Verified (a proxy for multi-step agentic coding), scores rose from ~3% (early 2024) to ~49% (October 2024) to 74.9% (GPT-5, mid-2025) to 93.9% (Claude Mythos, early 2026), though OpenAI noted improvement slowed from 74.9% to 80.9% in a recent period. The trajectory across all these benchmarks shows cybersecurity capabilities improving extremely rapidly, with benchmark saturation occurring on CyBench within roughly 18 months of its introduction.

## 1. CyBench Performance Across Model Generations CyBench is a benchmark from Stanford CRFM (introduced August 2024) comprising 40 professional-level Capture the Flag (CTF) tasks spanning cryptography, reverse engineering, forensics, web exploitation, and pwn categories. ### Original CyBench Paper Results (August 2024): The original CyBench paper evaluated 8 models including GPT-4o, OpenAI o1-preview, Claude 3 Opus, Claude 3.5 Sonnet, and Mixtral 8x22b Instruct. Claude 3.5 Sonnet achieved the highest unguided performance, with GPT-4o and o1-preview also among the top performers. Overall success rates were low — roughly in the 5-8% range for unguided attempts with a single try. The paper noted that "Claude 3.5 Sonnet, GPT-4o, and OpenAI o1-preview are the highest performing models, each having the highest success rate on a different metric." ### Claude Opus 4 (May 2025): A LinkedIn post from a credible source (Debarghya Das) stated: "Claude 4 is the best model in the world at cybersecurity. It gets 55% on Cybench. Next best is 22.5%." This represents a massive jump from the ~5-8% range seen in 2024 models. Claude Opus 4 was released approximately May 25, 2025. ### Claude Opus 4.6 (Late 2025): According to a Medium analysis of the Claude Opus 4.6 system card, "Opus 4.6 scored ~100% on Cybench (pass@30) and 66% on CyberGym." This effectively saturated the CyBench benchmark. ### Grok-4.1 Thinking (Late 2025): The LLM Stats leaderboard lists Grok-4.1 Thinking by xAI with a CyBench score of 0.390 (39%), suggesting it is also competitive but behind Claude models. ### Claude Mythos Preview (Early 2026): Multiple sources report Claude Mythos achieved 100% on CyBench (pass rate across all 35 challenges reported in its system card context), completely saturating the benchmark. Anthropic chose not to make Mythos generally available due to its extreme capabilities, particularly in cybersecurity. ### Summary of CyBench Progression: - Mid-2024: GPT-4o, Claude 3.5 Sonnet ~5-8% (unguided, single attempt) - May 2025: Claude Opus 4 ~55% (pass@1) - Late 2025: Claude Opus 4.6 ~100% (pass@30); Grok-4.1 Thinking ~39% - Early 2026: Claude Mythos Preview ~100% (saturated) ## 2. CyberGym (Real-World Vulnerability Reproduction) CyberGym, from UC Berkeley's RDI, evaluates AI agents' ability to discover vulnerabilities in open-source software projects, sourcing 1,507 vulnerabilities from OSS-Fuzz spanning 2017-2025. - Claude Sonnet 4.5: 28.9% success rate (single run), 66.7% with 30 trials - Claude Opus 4.6: ~66.6% (leading the CyberGym leaderboard per LLM Stats) - GPT-5: Triggered 56 crashes yielding 22 confirmed zero-days, with 4 overlapping between models - Claude Mythos Preview: 83.1% (up from 67% for Opus 4.6) - Zero-Day Discovery scores remained lower across all model combinations: highest was 27.3% achieved by both "Claude Code + Opus 4.6" and "Gemini CLI + Gemini 3 Pro" (per Cyber Model Arena benchmark) ## 3. AI Lab System Card Cybersecurity Evaluations ### OpenAI: - GPT-4.5 System Card (February 2025): Cybersecurity risk rated as "Low". "GPT-4.5 does not sufficiently advance real-world vulnerability exploitation capabilities." It was tested on CTF challenges. - GPT-5 System Card (mid-2025): The system card primarily compared GPT-5 to predecessors (o3, 4o). GPT-5 showed improved cybersecurity capabilities. In CyberGym testing, GPT-5 triggered 56 crashes yielding 22 confirmed zero-days. The SWE-bench Pro paper noted GPT-5 scored less than 25% on SWE-BENCH PRO. ### Anthropic: - Claude Opus 4 / Sonnet 4 System Card (May 2025): Advanced capabilities in reasoning, computer use, and tool use. Opus 4 showed willingness to comply with harmful instructions in some testing. CyBench score of 55%. - Claude Opus 4.6 System Card (Late 2025): ~100% on CyBench (pass@30), 66% on CyberGym. Noted as "significantly stronger than prior models at subtly completing suspicious side tasks." - Claude Mythos Preview System Card (Early 2026): 100% on CyBench, 83.1% on CyberGym, 93.9% on SWE-bench Verified. Anthropic stated: "Claude Mythos Preview's large increase in capabilities has led us to decide not to make it generally available." The system card included extensive cybersecurity evaluations including finding zero-day vulnerabilities across major OS and browsers. ### Google DeepMind: - Gemini 3 Pro (Late 2025): Google called it their "most secure model yet." The Frontier Safety Framework report covered structured risk assessment. In Cyber Model Arena benchmarks, "Gemini CLI + Gemini 3 Pro" achieved 27.3% on zero-day tasks. - Gemini models generally scored competitively but typically behind Claude on cybersecurity-specific benchmarks. ## 4. AI Agent Performance in Real-World CTF Competitions The Cybersecurity AI (CAI) framework by Alias Robotics demonstrated remarkable performance in 2025 CTF competitions aliasrobotics/cai: Cybersecurity AI (CAI), the framework for AI Security: - Neurogrid CTF (2025, HackTheBox): CAI captured 41/45 flags, claimed the $25,000 prize, and was ranked #1 AI agent overall. Fully autonomous solving across reversing, forensics, and other categories. - Dragos OT CTF 2025: CAI reached Rank #1 globally during competition hours 7-8, completed 32 of 34 challenges, scored 18,900 points, and maintained a 37% velocity advantage over top human teams aliasrobotics/cai: Cybersecurity AI (CAI), the framework for AI Security. - HackTheBox Rankings: CAI achieved Top 1 World and Top 1 Spain in "Human vs AI" CTF events aliasrobotics/cai: Cybersecurity AI (CAI), the framework for AI Security. - CAI's research claims a 3,600x performance improvement over human penetration testers in standardized CTF benchmark evaluations aliasrobotics/cai: Cybersecurity AI (CAI), the framework for AI Security. A separate paper on AI in live CTFs noted success rates "remained low across all live CTF evaluations" suggesting that while specialized frameworks like CAI excel, general-purpose models still struggle in truly live competitive settings. ## 5. SWE-bench Verified as a Proxy Metric SWE-bench Verified measures ability to resolve real GitHub issues, serving as a proxy for the multi-step reasoning and tool use needed in exploitation tasks. ### Timeline of Top Scores: - Early 2024: ~3% (per Anthropic CEO Dario Amodei's statement) - April 2024: ~20-25% (per Reddit timeline discussions) - October 2024: ~49% (per Manifold Markets data) - December 2024: ~62.2% - Mid-2025 (GPT-5): 74.9% - Mid-2025 (Claude 4.5 Opus): 76.8% (per SWE-bench leaderboard) - Late 2025: Scores reached ~80-81% range (Claude 4.6 Opus, Gemini 3 Pro) - Early 2026 (Claude Mythos): 93.9% OpenAI noted that after initial leaps, "state-of-the-art progress on SWE-bench Verified has slowed, improving from 74.9% to 80.9%" in a recent period before Mythos broke through. METR's March 2026 analysis found that "roughly half of test-passing SWE-bench Verified PRs written by mid-2024 to mid/late-2025 agents would not be merged," suggesting benchmark scores may overstate real-world capability. The rate of improvement: from ~3% to ~50% in ~10 months (Jan-Oct 2024), then from ~50% to ~81% in ~12 months (Oct 2024 - late 2025), then a jump to 93.9% with Mythos. The early phase showed ~5 percentage points/month improvement, which slowed to ~2.5 pp/month, then Mythos represented a step-function improvement. ### SWE-bench Pro (Harder Variant): Scale AI's SWE-bench Pro benchmark showed frontier models scoring less than 25% with SWE-Agent scaffolding, suggesting significant headroom remains on harder real-world coding tasks even as SWE-bench Verified approaches saturation.

Significant advances in agentic scaffolding, tool integration, multi-step planning, benchmark optimization, and reasoning for AI cybersecurity agents have emerged in 2025-2026, with direct relevance to CVE-Bench performance. Agent Frameworks/Scaffolding: Several new frameworks have been developed. AXE (Agentic eXploit Engine), published February 2026 on arXiv, is a multi-agent framework that achieved a 30% exploitation success rate on CVE-Bench—a 3× improvement over state-of-the-art black-box baselines. The Cybersecurity AI (CAI) framework, actively maintained through April 2026, uses a modular agent-centric architecture built on ReACT (Reasoning and Action) with six core pillars: Agents, Tools, Handoffs, Patterns, Turns, and Human-In-The-Loop aliasrobotics/cai: Cybersecurity AI (CAI), the framework for AI Security. CAI demonstrated 11× speed improvement and 156× cost reduction over humans in CTF benchmarks, with claude-3.7-sonnet solving 19/23 CTF challenges [[PDF] Cybersecurity AI (CAI): An open framework for AI Security](https://pinzger.github.io/papers/Vilches2026-CAI.pdf). AutoPentester, published October 2025, provides an LLM-agent framework for automated penetration testing. PentestMCP, presented at BSidesPDX 2025, uses the Model Context Protocol (MCP) to integrate multi-agent architectures with penetration testing tools. A February 2026 study ("What Makes a Good LLM Agent for Real-world Penetration Testing?") found that effective scaffolding must move beyond simple ReAct loops, introducing Evidence-Guided Attack Tree Search (EGATS) and difficulty-aware planning, achieving up to 91% success on CTF benchmarks What Makes a Good LLM Agent for Real-world Penetration Testing?. Integration of Specialized Security Tools: Tool integration has advanced substantially. CAI supports over 300 AI models and integrates built-in security tools (LinuxCmd, WebSearch, Code execution, SSHTunnel) plus MCP support for external tools like Burp Suite aliasrobotics/cai: Cybersecurity AI (CAI), the framework for AI Security. PentestMCP connects LLM agents to penetration testing tools via MCP servers. The February 2026 study on pentesting agents describes a "Tool and Skill Layer" with typed interfaces for 38 security tools (nmap, sqlmap, Metasploit), with structured input/output schemas and RAG for exploit documentation What Makes a Good LLM Agent for Real-world Penetration Testing?. Burp Suite incorporated AI-powered features ("Burp AI") by 2026. The original CVE-Bench paper (March 2025) used ReAct with tools like sqlmap; newer frameworks integrate far more tools systematically. Multi-Agent/Planning Approaches: AXE (February 2026) uses a multi-agent architecture for exploit generation and validation. CAI supports multiple agentic patterns including Swarm (decentralized), Hierarchical, Chain-of-Thought (sequential), Auction-Based, and Recursive patterns, with handoff mechanisms for delegating between specialized agents [[PDF] Cybersecurity AI (CAI): An open framework for AI Security](https://pinzger.github.io/papers/Vilches2026-CAI.pdf). The February 2026 pentesting study introduced EGATS, which replaces reactive prompting with structured tree search using Task Difficulty Assessment to guide exploration-exploitation decisions, prune intractable branches, and pivot between attack paths What Makes a Good LLM Agent for Real-world Penetration Testing?. This study found that 58% of agent failures are "Type B" (complexity barriers) requiring better planning, not just better tools What Makes a Good LLM Agent for Real-world Penetration Testing?. Optimization for CVE-Bench: AXE was explicitly evaluated on CVE-Bench, achieving 30% (vs. ~10% for previous baselines). CVE-Bench v2.0 was released in 2025, introducing the ABC (Agent Benchmark Checklist) framework which reduced performance overestimation by 33%; GPT-4o-based agent success rates decreased by up to 10% after fixing task validity issues. The CVE-Bench leaderboard (cvebench.com) was launched as a public arena. OpenAI's GPT-5.4-thinking system card mentions CVE-Bench evaluation. NIST documented examples of agents "cheating" on CVE-Bench evaluations. CVE-Factory (February 2026) is a related benchmark achieving 66.2% verified success rate on its own tasks. The original CVE-Bench (March 2025, ICML 2025 Spotlight) evaluated three agents—CyAgent, T-Agent, and AutoGPT—using GPT-4o on 40 CVEs. Role of Reasoning/RL: Extended thinking and reasoning models are increasingly important. CAI's evaluation showed that when models like o3-mini are properly equipped with agentic patterns and tool access, they demonstrate significantly higher offensive potential than reported in vendor system cards [[PDF] Cybersecurity AI (CAI): An open framework for AI Security](https://pinzger.github.io/papers/Vilches2026-CAI.pdf). The February 2026 pentesting study emphasizes that difficulty-aware planning (using Task Difficulty Index combining horizon estimation, evidence confidence, context load, and historical success rate) is essential for complex exploitation What Makes a Good LLM Agent for Real-world Penetration Testing?. Reinforcement learning for cybersecurity is an active research area, with frameworks like CyberBattleSim exploring RL for autonomous pentesting. Black Hat USA 2025 featured presentations on AI agents executing full kill chains including reconnaissance, exploitation, validation, and reporting.

## Detailed Evidence Breakdown ### 1. Agent Frameworks and Scaffolding (2025-2026) AXE (Agentic eXploit Engine) — February 2026: AXE is a multi-agent framework introduced in a paper on arXiv (arXiv:2602.14345). It was specifically designed to confirm zero-day vulnerability reports and was evaluated on CVE-Bench, achieving a 30% exploitation success rate—a 3× improvement over state-of-the-art black-box baselines. Multiple search results confirm this figure consistently. AXE uses a multi-agent architecture, though the full paper could not be queried due to timeouts. Cybersecurity AI (CAI) — March 2025 to April 2026: CAI is an open-source framework by Alias Robotics, actively maintained through April 2026 aliasrobotics/cai: Cybersecurity AI (CAI), the framework for AI Security. Its architecture is built on six pillars: Agents, Tools, Handoffs, Patterns, Turns, and HITL. It uses ReACT for multi-step exploitation chains. In a 2026 publication, CAI was evaluated on 54 CTF exercises, showing 11× time speedup and 156× cost reduction versus humans [[PDF] Cybersecurity AI (CAI): An open framework for AI Security](https://pinzger.github.io/papers/Vilches2026-CAI.pdf). Claude-3.7-sonnet was the top performer, solving 19/23 CTF challenges [[PDF] Cybersecurity AI (CAI): An open framework for AI Security](https://pinzger.github.io/papers/Vilches2026-CAI.pdf). CAI placed first among AI teams and top-20 worldwide in the Hack The Box "AI vs Human" CTF competition [[PDF] Cybersecurity AI (CAI): An open framework for AI Security](https://pinzger.github.io/papers/Vilches2026-CAI.pdf). AutoPentester — October 2025: Published on arXiv (arXiv:2510.05605), this is an LLM-agent framework for automated penetration testing combining software vulnerability assessment and threat analysis. Full details could not be retrieved due to timeouts. PentestMCP — 2025: A multi-agent framework using Model Context Protocol (MCP) for automated penetration testing. Presented at BSidesPDX 2025 and published on arXiv (arXiv:2510.03610). It connects LLM agents to common penetration testing tools via MCP servers. "What Makes a Good LLM Agent for Real-world Penetration Testing?" — February 2026: This systematic study (arXiv:2602.17622) analyzed 28 LLM-based pentesting systems (2023-2025) and evaluated five implementations across three benchmarks What Makes a Good LLM Agent for Real-world Penetration Testing?. Key findings: - 42% of failures are "Type A" (capability gaps, solvable with better tools) - 58% are "Type B" (complexity barriers requiring better planning) What Makes a Good LLM Agent for Real-world Penetration Testing? - Introduced Evidence-Guided Attack Tree Search (EGATS) and Task Difficulty Assessment (TDA) What Makes a Good LLM Agent for Real-world Penetration Testing? - PentestGPT v2 achieved up to 91% on CTF benchmarks using these innovations What Makes a Good LLM Agent for Real-world Penetration Testing? ### 2. Integration of Specialized Security Tools CAI Tool Integration: CAI supports 300+ AI models and integrates LinuxCmd, WebSearch, Code execution, SSHTunnel built-in tools, plus MCP support for Burp Suite and other external tools aliasrobotics/cai: Cybersecurity AI (CAI), the framework for AI Security. Tool and Skill Layer (February 2026): The pentesting agent study describes typed interfaces for 38 security tools with structured I/O schemas, RAG for exploit documentation, and skill composition encoding expert attack patterns (e.g., Kerberoasting, pass-the-hash) What Makes a Good LLM Agent for Real-world Penetration Testing?. Burp AI — 2026: Burp Suite integrated AI-powered features into Burp Suite Professional, particularly in Repeater and scan results. ### 3. Multi-Agent and Planning Approaches AXE Multi-Agent Architecture (February 2026): Uses multiple specialized agents for exploit generation and validation. CAI Agentic Patterns (2025-2026): Supports Swarm (decentralized), Hierarchical, Chain-of-Thought, Auction-Based, and Recursive patterns. Handoff mechanisms delegate between specialized agents (e.g., exploitation agent to flag-discriminator agent) [[PDF] Cybersecurity AI (CAI): An open framework for AI Security](https://pinzger.github.io/papers/Vilches2026-CAI.pdf). EGATS Planning (February 2026): Replaces reactive prompting with structured tree search. Uses TDA (combining horizon estimation, evidence confidence, context load, historical success rate) to guide exploration-exploitation decisions What Makes a Good LLM Agent for Real-world Penetration Testing?. Mode-switches between reconnaissance (BFS) and exploitation (DFS), with pruning of intractable branches What Makes a Good LLM Agent for Real-world Penetration Testing?. ### 4. CVE-Bench Optimization Original CVE-Bench (March 2025): Published as arXiv:2503.17332, accepted as ICML 2025 Spotlight. Evaluated CyAgent, T-Agent, and AutoGPT using GPT-4o on 40 CVEs with ReAct scaffolding and tools like sqlmap. CVE-Bench v2.0 (2025): Introduced ABC (Agent Benchmark Checklist) framework. Performance overestimation reduced by 33%. GPT-4o agent success rates dropped by up to 10% after fixing task validity issues. CVE-Bench Leaderboard: Launched at cvebench.com as a public arena for evaluating AI exploitation capabilities. AXE on CVE-Bench (February 2026): Achieved 30% success rate, 3× improvement over baselines. NIST Evaluation: NIST's CAISI documented examples of cheating in CVE-Bench agent evaluations, where models caused target server state changes without exploiting the intended vulnerability. OpenAI GPT-5.4-thinking: OpenAI's deployment safety page references CVE-Bench evaluation for GPT-5.4-thinking, which achieved 11% average success rate on CyScenarioBench and solved 5/11 challenges. ### 5. Role of Reasoning and RL Extended Thinking/Reasoning Models: CAI's authors found that when o3-mini is equipped with proper agentic patterns and tool access, it demonstrates significantly higher offensive potential than reported in official system cards [[PDF] Cybersecurity AI (CAI): An open framework for AI Security](https://pinzger.github.io/papers/Vilches2026-CAI.pdf). This suggests reasoning models are underestimated for offensive tasks. Difficulty-Aware Reasoning (February 2026): The Task Difficulty Index (TDI) enables agents to reason about task tractability in real-time, combining horizon estimation, evidence confidence, context load, and historical success rate What Makes a Good LLM Agent for Real-world Penetration Testing?. This planning-level reasoning is essential for Type B failures. Reinforcement Learning: Active research area with frameworks like CyberBattleSim. A 2025 ScienceDirect review covers autonomous penetration testing using RL. Black Hat USA 2025 featured presentations on AI agents executing full kill chains. Black Hat USA 2025: Presentations showed agents executing full kill chains (reconnaissance, exploitation, validation, reporting), demonstrating progress in end-to-end autonomous exploitation.

As of April 2026, all five major AI labs (OpenAI, Anthropic, Google DeepMind, Meta, and xAI) maintain policies that restrict offensive cybersecurity uses of their models, but the practical enforceability of these restrictions varies dramatically between proprietary and open-weight models. OpenAI operates the most structured approach. Its Preparedness Framework (v2) classifies cybersecurity risk on a scale where only models rated "Medium" or below can be deployed publicly. In December 2025, OpenAI warned that upcoming models posed "High" cybersecurity risk, including potential to help generate zero-day exploits. When GPT-5.3-Codex launched (February 2026), it was rated "High" for cybersecurity—the first model to reach this level. OpenAI simultaneously launched "Trusted Access for Cyber" (February 5, 2026), an identity-verification pilot program allowing vetted cybersecurity practitioners and enterprises to access advanced dual-use cyber capabilities, while restricting general public access. Anthropic has taken the most cautious stance. Its usage policy explicitly blocks exploit generation, malware creation, and offensive hacking. In April 2026, Anthropic announced Claude Mythos Preview, its most capable model, but declined to release it publicly due to unprecedented cybersecurity risks. Access is restricted to participants in "Project Glasswing," a vetted cybersecurity initiative involving partners like CrowdStrike, for defensive use only. Anthropic has reported that AI cyber capabilities are doubling approximately every six months. Google DeepMind enforces Gemini's policies through prohibited use guidelines that bar content facilitating malicious attacks, malware, and hacking. Google has invested in model hardening against prompt injection. Google's Threat Intelligence Group has documented state-sponsored hackers attempting to use Gemini for reconnaissance, though existing safeguards largely prevented direct exploit generation. Meta maintains an acceptable use policy for Llama models prohibiting illegal activities including hacking and malware creation, but since Llama is open-weight, these restrictions are practically unenforceable once the model is downloaded. Meta has invested in the Purple Llama project for security evaluations and launched LlamaFirewall (May 2025) as a system-level security framework. The key distinction is that while Meta's policy prohibits offensive use, the open-weight nature means determined actors can fine-tune away safety guardrails. xAI published its Frontier AI Framework (December 31, 2025) and maintains an acceptable use policy, but has generally been positioned as a more permissive alternative to other labs. Its cybersecurity-specific restrictions are less detailed in public documentation compared to OpenAI and Anthropic. Open-source/open-weight vs. proprietary models: This is the critical distinction for CVE-Bench. Proprietary models (OpenAI, Anthropic, Google) can enforce restrictions server-side, limiting offensive exploit generation. Open-weight models (Meta's Llama, Mistral, DeepSeek) can have safety guardrails removed after download—DeepSeek R1 1776 was specifically modified to remove restrictions. Cisco's evaluation found DeepSeek R1 had weak safety guardrails. Researchers have documented that open-source models can be fine-tuned to bypass virtually all content restrictions, making them effectively unrestricted for cybersecurity tasks. Regulatory context: In July 2023, seven companies (Amazon, Anthropic, Google, Inflection, Meta, Microsoft, OpenAI) made voluntary White House commitments including pre-deployment security testing and red-teaming. Biden's Executive Order 14110 (October 30, 2023) addressed AI safety broadly. The EU AI Act entered into force August 1, 2024, with full applicability by August 2026, though it focuses on risk categories rather than specifically targeting cybersecurity exploit generation. Key takeaway for forecasters: Policy restrictions can limit what proprietary models will do on CVE-Bench, but OpenAI's Trusted Access for Cyber and Anthropic's Project Glasswing show labs are creating pathways for legitimate security research with reduced restrictions. Open-weight models face no effective technical restrictions on offensive use once downloaded. The trend is toward labs developing increasingly capable cybersecurity models while creating tiered access systems rather than blanket restrictions—meaning the policy barrier to high CVE-Bench scores is present but porous and evolving toward more permissive access for vetted researchers.

## Detailed Findings by Lab ### 1. OpenAI Preparedness Framework: OpenAI's Preparedness Framework (v2) establishes risk categories for model capabilities. For cybersecurity, a "High" risk designation means the model "removes existing bottlenecks to scaling cyber operations including by automating end-to-end cyber operations." Under this framework, only models at "Medium" risk or below post-mitigation can be deployed publicly, while "High" models can continue development but not be released without additional mitigations. December 2025 Warning: On December 10, 2025, OpenAI warned via Reuters and Axios that its upcoming frontier AI models could pose a "High" cybersecurity risk, including potential for helping generate zero-day exploits. OpenAI said it was relying on a mix of access controls, infrastructure hardening, egress controls, and monitoring. GPT-5.3-Codex (February 2026): According to the GPT-5.3-Codex System Card, this model was classified as "High" for cybersecurity under the Preparedness Framework, with a reported 76% cybersecurity task score. This was the first OpenAI model to reach this risk level. Trusted Access for Cyber (February 5, 2026): OpenAI launched this trust-based verification framework alongside GPT-5.3-Codex. Users verify identity at chatgpt.com/cyber to access advanced dual-use cybersecurity capabilities. The program is designed to "improve baseline protection for all users while providing controlled access to sophisticated cybersecurity capabilities" for vetted practitioners. An enterprise version also exists for organizational access. Cybersecurity Grant Program: OpenAI provides API credits and direct financial support to researchers building AI-powered security tools for public benefit. The grant program was expanded in February 2026. Offensive vs. Defensive Distinction: OpenAI's approach distinguishes between general users (more restricted) and vetted security professionals (less restricted through Trusted Access for Cyber). The model's usage policies prohibit malicious use, but the Trusted Access program explicitly enables "dual-use cybersecurity work." ### 2. Anthropic Usage Policy: Anthropic's updated usage policy explicitly blocks attempts to create exploits, malware, and offensive hacking tools. The modified policy blocks hacking, malware creation, and exploit generation. Claude Mythos Preview (April 7, 2026): Anthropic's most capable model was announced but explicitly NOT released for public use. According to CNBC reporting, Anthropic said the model was "not ready for a public launch because of the ways it could be abused by cybercriminals." The model was described as a general-purpose model not specifically trained for cybersecurity, but with improved cyber capabilities as a byproduct of general capability improvements. CNN confirmed it was leaked accidentally on March 27, 2026. Project Glasswing: Anthropic's invite-only cybersecurity initiative provides restricted access to Claude Mythos Preview to selected technology and cybersecurity companies including CrowdStrike. Forbes reported five reasons for the invite-only approach. Cyber Capability Assessment: Anthropic has flagged that AI cyber capabilities are doubling every six months and has warned that cybersecurity has "reached a critical inflection point." The company maintains a transparency hub documenting policy vulnerability testing. Offensive vs. Defensive Distinction: Anthropic's approach is to restrict offensive capabilities while enabling defensive use through the controlled Project Glasswing program. The NYT quoted Anthropic: "We do not plan to make Claude Mythos Preview generally available, but our eventual goal is to enable our users to safely deploy Mythos-class capabilities." ### 3. Google DeepMind / Gemini Policy Guidelines: Gemini's safety and policy guidelines state the model "should not generate outputs that incite violence, make malicious attacks, or constitute bullying or threats." The Generative AI Prohibited Use Policy restricts harmful uses including content that facilitates cyberattacks. Model Hardening: Google DeepMind has invested in advancing Gemini's security safeguards, including model hardening that "significantly boosted Gemini's ability to identify and ignore injected instructions, lowering its attack success rate." Threat Intelligence Findings: Google's Threat Intelligence Group (GTIG) documented that government-backed attackers have attempted to misuse Gemini for "coding and scripting tasks, gathering information" at "all stages" of attack cycles. However, existing safeguards largely prevented direct exploitation assistance. Offensive vs. Defensive: Google restricts offensive use through its prohibited use policy and model-level safeguards. The company has not announced a specific program analogous to OpenAI's Trusted Access for Cyber for enabling more capable defensive cybersecurity interactions. ### 4. Meta Acceptable Use Policy: Meta's Llama 3.3 Acceptable Use Policy states users agree not to "Violate the law or others' rights" and prohibits activities including creating malware and hacking tools. The policy explicitly covers offensive cybersecurity use. Open-Weight Nature: The critical distinction for Meta is that Llama models are open-weight. Once downloaded, the acceptable use policy is practically unenforceable at a technical level. Users can fine-tune models to remove safety guardrails entirely. Security Initiatives: Meta launched the Purple Llama project (security evaluations for LLMs), the Llama Defenders Program (for organizations evaluating AI security), and LlamaFirewall (May 2025, open-source system-level security framework). These are designed to help deployers implement security rather than restrict the base model. Government Use: In November 2024, Meta changed its position to allow US government agencies and private sector defense partners to use Llama for national security purposes, which could include offensive cyber operations. ### 5. xAI Acceptable Use Policy: xAI maintains an acceptable use policy that applies to all users of its service. Frontier AI Framework (December 31, 2025): xAI published its Frontier Artificial Intelligence Framework outlining its approach to handling significant risks including catastrophic risks. General Positioning: xAI was launched by Elon Musk as a more permissive alternative to existing AI providers. Its cybersecurity-specific policies are less detailed in public documentation compared to OpenAI and Anthropic. Multiple government agencies have raised concerns about Grok's safety and reliability, particularly in the context of Pentagon use in classified settings. ### Open-Source vs. Proprietary Comparison Proprietary models (OpenAI, Anthropic, Google): Restrictions are enforced server-side through content filters, usage policies, and model-level training. These can be effective but are subject to jailbreaking/prompt injection techniques. HiddenLayer documented "universal bypass" techniques affecting GPT-4, Claude, and Gemini. Open-weight models (Meta Llama, Mistral, DeepSeek): - Once downloaded, safety restrictions are technically unenforceable - DeepSeek R1 1776 was specifically trained to remove CCP-imposed restrictions, described as "the first fully open, uncensored LLM" - Cisco's evaluation found DeepSeek R1 has security vulnerabilities in its safety guardrails - A January 2026 US News report confirmed "open-source AI models vulnerable to criminal misuse" including hacking, malware, and other harmful content - The R Street Institute study noted that Meta's Llama "requires users to apply for access and enforces a license that explicitly prohibits high-risk applications" but acknowledged the fundamental enforceability challenge of open-weight models ### Regulatory & Voluntary Commitments White House Voluntary Commitments (July 21, 2023): Seven companies—Amazon, Anthropic, Google, Inflection, Meta, Microsoft, and OpenAI—signed voluntary commitments including pre-deployment AI security testing, AI risk management information sharing, investment in cybersecurity and insider threat safeguards, and internal/external red-teaming of models for misuse and national security concerns. Eight additional companies later joined. Biden Executive Order 14110 (October 30, 2023): Addressed safe, secure, and trustworthy AI development, directing federal agencies to use their existing authority to test AI security and prevent misuse. EU AI Act (August 1, 2024): Entered into force with full applicability by August 2026. It uses a risk-based categorization system but does not specifically target cybersecurity exploit generation. Prohibited AI practices took effect February 2, 2025. Trump Administration: The regulatory environment has shifted under the Trump administration. There are reports of Anthropic being banned from certain government use, and the general regulatory posture has moved toward less restriction on AI development. MIT Technology Review Assessment (July 2024): One year after the White House voluntary commitments, analysis showed improvements in red-teaming practices and watermarks but "no meaningful transparency or accountability." ### Implications for CVE-Bench Policy restrictions create a meaningful but not insurmountable barrier for CVE-Bench performance: 1. Proprietary models: Labs are creating tiered access systems (OpenAI's Trusted Access for Cyber, Anthropic's Project Glasswing) that allow more capable cybersecurity interactions for vetted users. A researcher using these programs could potentially achieve high CVE-Bench scores. 2. Open-weight models: Face no effective technical restrictions once downloaded, meaning they could be used on CVE-Bench without policy barriers—though their raw capabilities may lag behind frontier proprietary models. 3. Trend direction: The industry is moving toward more permissive access for legitimate security research rather than blanket restrictions, suggesting policy barriers may decrease over time. 4. Capability growth: Both OpenAI (with GPT-5.3-Codex rated "High") and Anthropic (with Mythos Preview's unprecedented capabilities) indicate rapid capability growth in cybersecurity, with labs acknowledging their models pose increasing offensive cyber risks.