Forecasts

## Detailed Findings ### Competition Overview DEF CON 33 CTF finals took place August 8–10, 2025, at the Las Vegas Convention Center, organized by Nautilus Institute. The format was attack-defense, requiring teams to simultaneously attack other teams' services while defending their own, supplemented by a LiveCTF bracket-style challenge-solving component. The competition ran over three days. ### Final Standings and Scores Two different scoreboards were identified from search results: Composite/Final Scores (from DEF CON CTF Archive page): 1. Maple Mallard Magistrates: 976,068 2. [Team name not visible in snippet]: 837,676 3. SuperDiceCode: 541,693 4. Nu1L: 496,550 5. RePokemonedCollections: [score not captured] Attack-Defense Scores (from Nautilus Institute Mastodon post): 1. Maple Mallard Magistrates: 4,419 2. SuperDiceCode: 3,992 3. mhackeroni: 3,956 4. Nu1L: 3,792 5. Shellphish: 3,494 6. KuK Hofhackerei: 3,383 7. organizers: 3,132 8. cold fusion: 3,077 The discrepancy between the two scoreboards (different rankings for 2nd/3rd place, different score magnitudes) suggests the composite final scores include LiveCTF bonuses and potentially other scoring components beyond the main attack-defense game. ### Nils/Friendly Maltese Citizens Performance - SRLabs' blog confirms Nils placed 8th overall, competing under the team name Friendly Maltese Citizens - The SRLabs blog snippet states: "Nils competed in the DEF CON 33 CTF finals, placing 8th while running attack-defense operations, participating in the LiveCTF, and publishing a [exploit/writeup]" - A separate SRLabs blog snippet also mentions: "I had the honor of competing in this year's DEF CON CTF finals as a member of the Friendly Maltese Citizens"—indicating the team included at least one human SRLabs researcher alongside the AI agent ### LiveCTF Component - From the LiveCTF GitHub repository (Live-CTF/LiveCTF-DEFCON33), Friendly Maltese Citizens participated in the LiveCTF bracket: "Upper Round 2: 7+8, Loki · mhackeroni vs Friendly Maltese" - The LiveCTF Day 2 YouTube video description mentions "AI Solve Discovery" at timestamp 4:15:58, suggesting an AI-driven solve was notable enough to be highlighted - A LinkedIn post references someone from a team using "a background AI agent [to] solve a LiveCTF challenge while that player was still working on it"—though this appears to reference a different team (pb_ctf x BlueWater) ### Capability Gaps vs. Top-3 Teams Based on the available scoreboard data: - MMM (1st): 976,068 final / 4,419 A-D - 2nd place: 837,676 final - 3rd (SuperDiceCode): 541,693 final / 3,992 A-D - Nils/FMC (8th): approximately 3,077 A-D score (if the "cold fusion" entry at 8th corresponds—though there's ambiguity here; FMC may not appear in the intermediate scoreboard under that name) The gap between 1st and 8th in A-D scoring alone was ~31% (4,419 vs ~3,077). In final composite scores, the gap was likely much larger—potentially 2-3x—because LiveCTF bonuses disproportionately rewarded stronger teams. Key capability gaps likely include: 1. Exploit development speed: Top human teams can rapidly reverse-engineer novel binaries and develop working exploits within minutes 2. Challenge coverage: Top teams solve a higher percentage of challenges across diverse categories 3. Defense/patching: Effective binary patching while maintaining service availability requires deep understanding 4. Adaptation over time: The multi-day format rewards teams that can adapt strategies based on evolving competition dynamics ### Scoring Trajectory Without access to the full SRLabs blog, the specific round-by-round or day-by-day scoring trajectory could not be determined. The competition ran over approximately 3 days (August 8–10), with multiple tick-based rounds in the attack-defense format. ### 39C3 Talk A 39C3 (Chaos Communication Congress, December 2025) talk titled "There is NO WAY we ended up getting..." was scheduled by a Friendly Maltese Citizens member, suggesting notable/surprising aspects of their DEF CON performance worth presenting. ### Important Caveats 1. The SRLabs blog post repeatedly timed out and could not be fully accessed, meaning the detailed attack vs. defense breakdown, scoring trajectory, and SRLabs' own gap analysis are not available in this report 2. Nils appears to have been part of a hybrid human-AI team (Friendly Maltese Citizens), not a purely autonomous AI team—the SRLabs blog author describes competing "as a member" of the team 3. The 8th-place identity is ambiguous between the two scoreboards—the Nautilus social post shows "cold fusion" at 8th, while SRLabs claims Nils/FMC placed 8th. These may be different scoring snapshots or the team may appear under different names 4. There may be additional teams below 8th that are not captured in the snippets

1. Organizer and Timeline DEF CON announced the Benevolent Bureau of Birds (BBB) as the new DEF CON 34 CTF organizers in approximately March 2026. The announcement was posted across DEF CON's official channels (defcon.org, DEF CON Forum, Facebook, Instagram, Reddit). The qualifier round is scheduled for May 22–24, 2026, and finals will take place at DEF CON 34 in Las Vegas, August 6–9, 2026. The BBB's official website is https://bbbirds.org/. Key BBB members named in public announcements include Vie, Robert Xiao, Zaratec, and Bluepichu — several of whom are associated with Maple Bacon, a CTF team from the University of British Columbia. 2. BBB Official Communications - bbbirds.org: The site timed out during multiple fetch attempts and could not be queried. - CTF Radiooo Episode 025 ("Chatting with NEW DEF CON CTF Organizers: Benevolent Bureau of Birds"): Published March 18, 2026. The YouTube video and podcast page could not be directly queried for transcript content. From Google snippets, the episode features adamd and Zardus interviewing BBB members (Vie, Robert Xiao, Zaratec, Bluepichu) about their plans. No specific details about rules, AI policies, format, scoring, or number of finalists were extractable from the snippets. - DEF CON Forum post (forum.defcon.org/node/255475): Timed out. Google snippet indicates it is a welcome announcement with a link to bbbirds.org and qualifier dates, but no detailed rules. 3. Competition Format and Rules (Not Yet Published) As of April 8, 2026, extensive searching reveals NO publicly available detailed rules, format specification, scoring system, or finalist count for DC34 CTF under BBB. The qualifier is listed on CTFtime as "On-line, open-registration" with finals at DEF CON in August 2026, but no further details are provided. 4. AI Agent/Autonomous Team Eligibility No public statement from BBB has been found that explicitly permits or prohibits autonomous AI agents or AI-only teams from entering the qualification round or competing in the finals. The open-registration nature of the qualifier suggests that any team (including AI-driven ones) could potentially register, but this is not confirmed. 5. Historical Precedent: DEF CON 33 (2025) The Nautilus Institute organized DEF CON 33 CTF. SRLabs' AI agent "Nils" was permitted to compete and placed 8th in the finals. The competition used an attack-and-defense format with LiveCTF components. Approximately 8–10 teams competed in the finals. Google snippets from srlabs.de confirm: "Nils competed in the DEF CON 33 CTF finals, placing 8th while running attack-defense operations, participating in the LiveCTF..." The University of Hawaii article mentioned "top eight teams" for DEF CON 33 finals. Carnegie Mellon's PPP (Plaid Parliament of Pwning) won their fourth consecutive and ninth overall title. 6. Key Uncertainties - The BBB has not yet released detailed rules, so it is unknown whether they will follow the traditional attack-and-defense format, how many teams will qualify for finals, what the scoring system will be, or whether AI-only teams will be explicitly allowed or banned. - The qualifier being "open-registration" is suggestive but not definitive regarding AI team eligibility. - The BBB is a new organizer, and each organizer historically sets their own rules. The fact that Nautilus Institute allowed Nils does not necessarily mean BBB will do the same. - The CTF Radiooo interview may contain relevant details about format and rules, but the transcript was not accessible for analysis.

## Trajectory of AI Agent Improvement in Cybersecurity Tasks (2023–Early 2026) ### 1. Academic Benchmarks: Establishing Baselines (2023–2024) InterCode-CTF (2023–2024): InterCode-CTF, introduced at NeurIPS 2023, contains 100 CTF tasks from picoCTF—a competition aimed at high-school-level participants. Early LLM performance was modest, but by December 2024, Palisade Research published results showing 95% solve rates with plain LLM agent designs. This benchmark is now widely considered "saturated," meaning it no longer differentiates between frontier AI capabilities. The rapid saturation of this entry-level benchmark demonstrates how quickly AI agents can master well-characterized, lower-difficulty challenges. NYU CTF Bench (2024–2025): The NYU CTF Bench, based on 200 challenges from CSAW competitions spanning 2017–2023, provides a more challenging evaluation NYU CTF Bench: A Scalable Open-Source Benchmark .... Results published in the paper (arXiv v3: February 18, 2025) showed: - Claude 3: ~5.77% solve rate across all challenges - GPT-3.5: ~1.92% solve rate - GPT-4: Scored 300 in 2023 CSAW qualifiers - Mixtral and LLaMA: 0% solve rate - Claude 3 achieved a score of 1500 in the 2022 CSAW finals, outperforming the median human score of 1321 - Open-source models completely failed NYU CTF Bench: A Scalable Open-Source Benchmark ... This benchmark revealed that while some frontier models could match or exceed median human performance on specific competition subsets, overall success rates remained low, particularly on complex multi-step challenges. Cybench (August 2024): Stanford's Cybench introduced 40 professional-level CTF tasks from recent competitions (2022–2024). Results from the original paper showed: - GPT-4o: ~12.5% unguided solve rate; 29.4% with subtask guidance - Claude 3.5 Sonnet: Comparable unguided performance (solved at least one task unguided) - Claude 3 Opus: Also solved at least one unguided task - These low solve rates on professional-level challenges contrast sharply with the saturation of InterCode-CTF As of early 2026, the Cybench leaderboard shows Grok-4.1 Thinking leading with a score of 0.390 (39%), indicating continued but incremental improvement on professional-level tasks. ### 2. DARPA AI Cyber Challenge (AIxCC): 2023–2025 DARPA's AIxCC was a two-year, multi-million-dollar competition focused on autonomous vulnerability discovery and patching in open-source software. Key milestones: - 2023: Competition launched, attracting 42 teams - August 2024 (DEF CON 32): Semifinals held; 7 teams advanced to finals - August 8, 2025 (DEF CON 33): Finals held - Winner: Team Atlanta (Georgia Tech/Samsung) — $4M prize - 2nd Place: Trail of Bits ("Buttercup") — $3M prize - 3rd Place: Theori - Competition included 63 synthetic vulnerabilities - Competitors' cyber reasoning systems (CRSs) collectively identified 54 vulnerabilities and patched 43 - Trail of Bits reported finding 28 vulnerabilities and patching 19 AIxCC demonstrated that AI systems can perform meaningful autonomous vulnerability discovery and patching at scale, but the task was specifically scoped to source-code-level analysis of open-source projects—a narrower task than full CTF competition. ### 3. 2025 CTF Circuit: AI Agents Begin Competing Directly Hack The Box "AI vs Human" CTF (2025): In a landmark event, AI agent teams competed directly against human teams: - 5 of 8 AI agent teams solved 19 out of 20 challenges (95% solve rate) - They competed against 403 human teams - The CAI agent (from Alias Robotics/Cybersecurity AI) achieved its final flag 30 minutes before the next AI team CAI's 2025 CTF Circuit Performance: The CAI agent systematically competed in 5 major jeopardy-style CTF competitions throughout 2025: - Won $50K at the Neurogrid CTF with a 91% solve rate - Demonstrated 98% cost reduction compared to human teams - Led researchers to argue that "jeopardy-style CTFs may be obsolete" as meaningful benchmarks for AI CSAW 2025: Research published in early 2026 compared autonomous agent performance against human teams in the 2025 CSAW competition, observing differences across autonomy levels and challenge categories. ### 4. Wiz Research Study (January 29, 2026) Wiz Research, in collaboration with the AI security lab Irregular, tested Claude Sonnet 4.5, GPT-5, and Gemini 2.5 Pro on 10 lab environments modeled after real-world vulnerabilities AI Agents vs Humans: Who Wins at Web Hacking in 2026?: - Narrow scope (specific target given): Agents solved 9 of 10 challenges; costs often under $1 per success - Broad scope (no specific target): Performance degraded significantly; costs increased 2–2.5x; agents struggled to prioritize targets and spread efforts haphazardly - Key failure mode: Agents failed to use standard fuzzing tools unless prompted, couldn't pivot strategies when initial approaches failed - The unsolved challenge (GitHub Secrets) required creative investigative pivoting that agents couldn't perform - Study concluded AI agents are highly effective at executing known attack patterns but lack strategic adaptability for complex, unguided offensive operations AI Agents vs Humans: Who Wins at Web Hacking in 2026? ### 5. Tenzai Claim (March 17, 2026) Israeli startup Tenzai announced on March 17, 2026 that its autonomous AI hacker: - Achieved top 1% performance across six major CTF platforms - Outperformed over 125,000 human competitors - Was described as "the first autonomous system to rank in the top 1% of global hacking competitions" - Covered competitions "designed for humans" ### 6. Analysis: Improvement Trajectory The trajectory is best characterized as S-curve-like with domain-dependent saturation points: Entry-level tasks (InterCode-CTF): Rapid improvement → saturation at 95% by late 2024. Effectively solved. Mid-tier jeopardy challenges (HTB, standard CTFs): Steep improvement through 2025. AI agents went from struggling with basic challenges to achieving 91-95% solve rates and top-1% rankings by early 2026. Professional-level jeopardy tasks (Cybench): Slower improvement. From ~12.5% unguided (mid-2024) to ~39% (early 2026), suggesting continued but more modest gains. Real-world vulnerability discovery (AIxCC): AI systems demonstrated meaningful but imperfect capability—finding ~86% (54/63) of synthetic vulnerabilities and patching ~68% (43/63). Broad, unguided offensive operations: Still significantly limited as of January 2026, with degraded performance when agents must independently identify and prioritize targets AI Agents vs Humans: Who Wins at Web Hacking in 2026?. ### 7. The Jeopardy vs. Attack-Defense Gap Narrowing at the jeopardy end: AI agents have essentially caught up with or surpassed many human competitors on jeopardy-style CTFs by early 2026. The CAI team's 2025 performance and Tenzai's top-1% claims confirm this. Still wide at the attack-defense end: The DEF CON CTF finals use an attack-defense format requiring: - Simultaneous offensive and defensive operations - Real-time adaptation to opponent strategies - Service patching that maintains functionality - Network traffic analysis and exploit development under time pressure - Coordination of multiple concurrent tasks A 2026 paper by Vilches et al. ("Evaluating Agentic Cybersecurity in Attack/Defense CTFs") represents the first empirical study of autonomous AI agents in A/D CTF scenarios, studying AI agents competing concurrently in offensive and defensive roles. This suggests the field is only beginning to formally evaluate this gap. The Wiz Research finding that AI agents struggle with broad-scope, unguided operations AI Agents vs Humans: Who Wins at Web Hacking in 2026? is particularly relevant—attack-defense CTFs are inherently broad-scope, requiring agents to simultaneously monitor, attack, and defend multiple services without explicit targeting guidance. Summary of the gap: While the gap is clearly narrowing for isolated challenge-solving (jeopardy), the gap for real-time, multi-service, adversarial attack-defense competition (as in DEF CON CTF finals) remains substantial. The improvement from "can't solve basic CTF challenges" to "top 1% in jeopardy CTFs" took roughly 2 years (2023–2025), but the remaining jump to "competitive in DEF CON CTF finals" requires solving qualitatively different problems in real-time coordination, strategic adaptation, and simultaneous offense/defense.

## Qualitative Differences: Attack-Defense Finals vs. Jeopardy-Style CTF ### Jeopardy-Style Format In jeopardy-style CTFs (used in most online CTF platforms and in DEF CON qualifiers), teams are presented with a set of standalone challenges across categories such as cryptography, reverse engineering, binary exploitation (pwn), web, forensics, and miscellaneous. Each challenge has a single flag to capture. Teams work at their own pace, challenges are static (they don't change based on opponents' actions), and there is no adversarial interaction between teams. Success is purely a function of how many challenges a team can solve within the allotted time. ### Attack-Defense Format (DEF CON CTF Finals) The DEF CON CTF finals, organized by Nautilus Institute (as of 2024-2025), use an attack-defense format where approximately 12 teams (per the 2025 rules) compete simultaneously. According to the 2025 DEF CON CTF finals format, the competition is described as "a reverse engineering and exploitation competition first and foremost." Key structural differences include: 1. Simultaneous offense and defense: Each team runs identical copies of vulnerable services on their own infrastructure. Teams must simultaneously attack other teams' services to steal flags AND defend their own services by patching vulnerabilities. 2. Round-based scoring: The game proceeds in timed rounds (typically 3-5 minutes each). Each round, new flags are planted in services, and teams earn attack points by stealing flags from opponents and defense points by preventing flag theft from their own services. 3. SLA/Availability requirements: Teams must keep their services running and functional. If a patch breaks the service's expected functionality, the team loses availability/SLA points. This creates a critical constraint: patches must fix the vulnerability without altering legitimate behavior. 4. Dynamic, adversarial environment: Unlike static jeopardy challenges, the competition environment evolves continuously. Opponents adapt their defenses, new services are released during the competition, and teams must monitor network traffic to detect and respond to attacks. 5. Scale of simultaneous services: Teams must manage ~8-10 or more services simultaneously over the multi-day competition, requiring significant infrastructure automation. 6. LiveCTF component: Since DEF CON 30 (2022), a LiveCTF component features 1v1 head-to-head matches where individual players solve jeopardy-style challenges in a timed format, adding another dimension to scoring. ## Analysis of Six Sub-Tasks and AI Capability ### 1. Reverse-Engineering Unknown Binary Services Requirement: At the start of each round or when new services are deployed, teams receive compiled binary executables (often stripped of symbols, possibly obfuscated) that they must quickly reverse-engineer to understand functionality, identify vulnerabilities, and determine how to exploit and patch them. AI capability status: As of April 2025, Claude (Anthropic's frontier model) could not solve any challenges at PlaidCTF, a top-tier jeopardy-style competition featuring binary exploitation and reverse engineering challenges. While AI agents have shown capability on easier reverse engineering tasks (e.g., Claude achieved top 3% in PicoCTF, a student-level competition), DEF CON finals binaries are significantly more complex—often custom-designed, using unusual architectures, and requiring deep understanding of low-level systems concepts. The gap between student-level reverse engineering and DEF CON finals-level binary analysis remains enormous for AI systems. Challenge level for AI: HIGH. Binary reverse engineering requires spatial reasoning about code structure, understanding of assembly language semantics, and the ability to form and test hypotheses about program behavior—capabilities where current AI agents show inconsistent performance, especially at scale and under time pressure. ### 2. Finding Vulnerabilities Requirement: After reverse-engineering services, teams must identify exploitable vulnerabilities (buffer overflows, format string bugs, use-after-free, logic errors, cryptographic weaknesses, etc.). AI capability status: DARPA's AIxCC competition (finals August 8, 2025) demonstrated that autonomous Cyber Reasoning Systems (CRS) can find vulnerabilities in source code. Team Atlanta's CRS won first place, demonstrating AI-driven vulnerability detection across 54 million lines of code in C++ and Java source code. However, AIxCC operated on source code, not stripped binaries. The DEF CON CTF finals typically involve compiled binaries where vulnerability discovery is significantly harder. Challenge level for AI: MEDIUM-HIGH. AI has shown promising results for source-code vulnerability detection, but binary-level vulnerability discovery (the DEF CON CTF requirement) remains substantially more difficult. Traditional fuzzing and symbolic execution tools can partially automate this, but integrating these with AI reasoning in real-time competition conditions is an unsolved challenge. ### 3. Writing Exploits Requirement: Teams must write working exploits that reliably steal flags from multiple opponents' service instances. Exploits must account for potential differences in memory layout (ASLR), deployed patches, and network conditions. AI capability status: AI agents have demonstrated basic exploit writing capability on CTF challenges. In the Hack The Box AI vs. Human CTF (July 2025), five of eight AI-agent teams solved 19 out of 20 challenges, including binary exploitation. However, these were pre-designed challenges with known solution paths. Writing reliable exploits that work across multiple targets in a live, adversarial environment with varying defenses is a qualitatively harder task. The need to modify exploits on-the-fly when opponents patch vulnerabilities adds another layer of difficulty. Challenge level for AI: HIGH. Exploit development for competition-grade binaries requires creative problem-solving, deep understanding of memory corruption primitives, and the ability to chain multiple vulnerabilities. The additional requirement of reliability across multiple targets and adaptation to patched services makes this especially challenging. ### 4. Patching/SLA Management Requirement: Teams must patch their own service binaries to fix vulnerabilities while preserving all legitimate functionality. If a patch breaks the service (fails SLA checks), the team loses points. This requires precise understanding of both the vulnerability and the service's intended behavior. AI capability status: AIxCC demonstrated AI-driven patching of source code vulnerabilities. Team Atlanta's system could autonomously generate patches. However, DEF CON CTF finals require binary patching—modifying compiled executables without access to source code. Binary patching is significantly harder: teams must modify machine code directly, often with tight space constraints, while ensuring the binary passes functionality checks. As Team Atlanta noted in their post-competition analysis, "a single bug can kill a CRS entirely. The autonomous system is that brittle." Challenge level for AI: VERY HIGH. Binary patching without breaking functionality is one of the hardest sub-tasks for AI. It requires: (a) correct identification of the vulnerability at the binary level, (b) generation of a correct fix in machine code, (c) verification that the fix doesn't break legitimate behavior, and (d) all of this under time pressure. The SLA constraint makes this especially punishing—an overly aggressive patch that breaks functionality costs the team more than leaving the vulnerability unpatched. ### 5. Real-Time Strategy Adaptation Requirement: Teams must continuously adapt their strategy as new services are released, opponents deploy new exploits, and the competitive landscape shifts. This includes deciding which services to prioritize for offense vs. defense, when to invest resources in new exploits vs. refining existing ones, and how to respond to detected attacks. AI capability status: No current AI system has demonstrated the ability to make real-time strategic decisions in a multi-service, multi-opponent competitive environment. This is fundamentally a multi-agent, multi-objective optimization problem with incomplete information—a domain where AI capabilities are still nascent. The 2016 DARPA Cyber Grand Challenge (CGC) at DEF CON 24 showed that autonomous systems could compete in a simplified attack-defense format, but those systems operated in a highly constrained environment (standard binary format, limited service complexity) and finished last when competing against human teams in the main DEF CON CTF. Challenge level for AI: VERY HIGH. This requires meta-reasoning about competition dynamics, opponent modeling, resource allocation under uncertainty, and the ability to pivot strategies rapidly. It is arguably the most uniquely challenging aspect of attack-defense CTF for AI, as it requires integrating information across all other sub-tasks and making holistic decisions. ### 6. Infrastructure and Traffic Management Requirement: Teams must manage their competition infrastructure (game servers, exploit deployment systems, traffic capture and analysis, automated flag submission), monitor network traffic to detect incoming attacks and reverse-engineer opponents' exploits, and deploy their own exploits automatically across all opponent targets every round. AI capability status: While components of this can be automated with traditional scripting and tooling (and human teams do extensively automate this), the AI-specific challenge is in the traffic analysis component—automatically identifying novel exploit patterns in network captures and converting observed attacks into defensive patches or counter-exploits. No current AI system has demonstrated this capability in a live competition setting. Challenge level for AI: MEDIUM-HIGH. Much of the infrastructure management can be handled by pre-built tooling rather than requiring AI reasoning. However, the traffic analysis, automated exploit detection, and dynamic infrastructure reconfiguration components require AI capabilities that haven't been demonstrated at competition scale. ## Key Evidence Points with Dates - August 8, 2025: DARPA AIxCC finals at DEF CON 33. Team Atlanta won first place with autonomous CRS for source-code vulnerability finding and patching across 54 million lines of code. This demonstrated AI capability for source-code analysis but not binary-level analysis required by DEF CON CTF. - August 2025: Carnegie Mellon's PPP won their fourth consecutive (and ninth overall) DEF CON CTF title, demonstrating that human teams continue to dominate the competition. - August 2025: At DEF CON 33, Claude competed in LiveCTF at the DEF CON CTF finals (referenced in YouTube video descriptions showing "AI Solve Discovery" during Day 2 of LiveCTF). - August 5, 2025: Axios reported that Claude had been "quietly beating human hackers" in student-level competitions, but Anthropic's own transparency page noted Claude achieved top 3% in PicoCTF (student competition), solved 19/20 in Hack The Box's AI vs. Human CTF, but scored only 15/30 in the Airbnb CTF and failed to solve any challenges at PlaidCTF (April 4, 2025). - April 4, 2025: Claude attempted PlaidCTF, a challenging jeopardy-style competition, and could not solve any challenges, demonstrating the gap between AI capability on easy-to-medium challenges and top-tier competition challenges. - July 2025: In Hack The Box's AI vs. Human MCP Tryout CTF, five of eight AI-agent teams solved 19/20 challenges, competing against 403 human teams. However, these were retired challenges of mixed difficulty, not at DEF CON finals level. ## Greatest Challenges Summary The requirements that pose the greatest challenges for current autonomous AI systems are: 1. Real-time multi-service strategic orchestration: No AI has demonstrated the ability to simultaneously manage offense and defense across many services with adversarial opponents adapting in real time. 2. Binary-level patching under SLA constraints: Modifying compiled binaries without source code while preserving functionality is extremely brittle and error-prone for AI. 3. Adversarial real-time adaptation: Responding to opponents' evolving exploits and defenses requires a feedback loop of traffic analysis, attack identification, and dynamic response that no current AI system can execute. 4. Competition-grade binary reverse engineering: While AI can handle simpler reverse engineering tasks, the custom, complex, often obfuscated binaries used in DEF CON CTF finals remain beyond current AI capability, as evidenced by Claude's failure at PlaidCTF. The combination of all six sub-tasks occurring simultaneously, under time pressure, in an adversarial environment, makes attack-defense CTF qualitatively harder than jeopardy-style CTF for AI systems. Even if an AI could solve individual sub-tasks in isolation, the integration challenge—managing all tasks concurrently with strategic coherence—represents an additional, compounding difficulty.

Landscape of Autonomous AI CTF Agents (as of April 2026) The competitive landscape for autonomous AI CTF agents has expanded significantly between 2025 and early 2026, with multiple well-funded organizations and academic teams developing systems. Below is a comprehensive breakdown: --- SRLabs / Nils - Background: SRLabs is a Berlin-based security research lab that developed "Nils," the first known fully autonomous AI team to compete in DEF CON CTF finals. - Results (August 2025): Nils placed 8th at DEF CON 33 CTF finals, participating in both attack-defense operations and LiveCTF. - Current status: No public announcement has been found confirming plans for DEF CON 34 (August 2026). Their blog post documents their DEF CON 33 experience but does not explicitly state future competition plans. - Assessment: Given their pioneering effort and the publicity gained, continued participation seems likely but is unconfirmed. --- Tenzai - Background: Israeli startup founded in 2025 by former intelligence agency cybersecurity executives. - Funding (by March 2026): $75 million seed round at a $330 million valuation, raised within six months of founding. - Results (March 2026): Announced top-1% performance across six major CTF platforms designed for humans, outperforming 125,000+ human competitors. This was widely reported in Forbes and Yahoo Finance on March 17, 2026. - Goals: Stated focus is on enterprise security (autonomous penetration testing), with CTF results serving as validation of capability. - DEF CON plans: No specific announcement about entering DEF CON CTF has been found. --- Team Atlanta (DARPA AIxCC) - Background: Won DARPA's AI Cyber Challenge in August 2025 ($4M first prize), led by Professor Taesoo Kim at Georgia Tech. - Post-AIxCC (as of February 2026): Published SoK paper on AIxCC. Donated $2M to Georgia Tech's SSLab for ongoing autonomous cybersecurity research. - Focus: Their CRS (Cyber Reasoning System) was designed for defensive tasks—vulnerability detection and patching in open-source software. This is fundamentally different from the offensive exploitation required in DEF CON CTF. - Pivot to offensive CTF: No evidence of such a pivot. Taesoo Kim's team has historical DEF CON CTF experience (DEFKOR00T won DEF CON CTF 2018), but the AIxCC work was defense-oriented. --- XBOW - Funding (March 2026): Raised $120M Series C, valued over $1B. - Results: Became #1 ranked autonomous penetration tester on HackerOne's global leaderboard in 2025. Ran 1,060+ autonomous attacks as documented in their blog. - Focus: Commercial penetration testing product, not CTF competition specifically. --- RunSybil - Background: Automated offensive security company co-founded by Ariel Herbert-Voss (ex-OpenAI first research scientist). - Status (2026): Received fresh funding, expanding platform and hiring. Featured in DEF CON/MCSC 2026 panel on AI offense/defense. - DEF CON CTF: No announced plans to compete. --- CAI (Cybersecurity AI) by Alias Robotics - Results: First place among AI teams in Hack The Box's "AI vs Human" CTF; top-20 worldwide overall. 11x faster than humans across 54 exercises, but underperformed in "pwn" (0.77x) and "crypto" (0.47x) categories [[PDF] Cybersecurity AI (CAI): An open framework for AI Security](https://pinzger.github.io/papers/Vilches2026-CAI.pdf). - Framework: Open-source, agent-centric architecture supporting multiple LLMs. Claude-3.7-sonnet was top performer, solving 19/23 selected challenges [[PDF] Cybersecurity AI (CAI): An open framework for AI Security](https://pinzger.github.io/papers/Vilches2026-CAI.pdf). - Limitation: Struggles with harder challenge categories that are the bread-and-butter of DEF CON CTF finals. --- Major AI Labs - Anthropic: Claude Mythos Preview announced April 7, 2026, described as a "step change" in cybersecurity capabilities. However, standard Claude models solved zero challenges at PlaidCTF and DEF CON Qualifier (elite competitions requiring novel exploitation), as widely noted on LinkedIn in early 2026. Mythos is being shared with ~50 companies for defensive use, not for CTF competition. - OpenAI: No specific autonomous CTF agent development announced. General cybersecurity capabilities improving with each model generation. - Google DeepMind: Google ran AI-centric CTFs at DEF CON 33 AI Village (September 2025 blog post) focused on education/adoption rather than competition. - None of the major AI labs have announced plans to enter an autonomous agent in DEF CON CTF. --- Academic Teams - NYU Tandon: Developed "EnIGMA" framework for autonomous cybersecurity challenge solving. - Georgia Tech SSLab: Receiving $2M from Team Atlanta's prize for continued autonomous security research. - Various universities: The March 2026 arxiv paper on "Scalable Agentic CTF Design" studied autonomous AI performance in educational CTF settings, noting limitations at higher difficulty levels. --- Key Structural Factors - DEF CON 34 CTF timeline: Qualifier May 22–24, 2026; Finals August 6–9, 2026 in Las Vegas. - Current AI limitations at elite CTF: The 2026 International AI Safety Report and multiple sources note that while AI agents perform well on standard/medium CTF challenges, they struggle with the novel, multi-step exploitation chains required at elite competitions like DEF CON CTF finals. - Gap between benchmarks and live competition: Tenzai's top-1% on static CTF platforms and XBOW's #1 on HackerOne are impressive, but DEF CON CTF finals involve real-time attack-defense dynamics, novel challenges, and time pressure that current systems handle poorly—as evidenced by Nils's 8th place finish (out of ~20 teams) at DEF CON 33.

2025 UNGA First Committee Resolution L.41 (Vote: November 6, 2025) The resolution on autonomous weapons systems (L.41) was tabled by Austria and 30 co-sponsors at the 80th session of the UNGA First Committee. It was adopted with 156 votes in favor, 5 against, and 8 abstentions. The resolution stressed the urgent need for the CCW to address challenges posed by autonomous weapons, including a call to complete elements of an instrument, with a view to future negotiations. The resolution also noted the Secretary-General's calls to commence negotiations on a legally binding instrument. Russia's vote on L.41 (November 6, 2025): Russia voted AGAINST. This is consistent with Russia voting against in both 2023 (L.56) and 2024 (L.77). Russia's October 24, 2025 statement at the UNGA First Committee Cluster IV debate Permanent Mission of the Russian Federation to the United Nations confirmed its opposition to moving LAWS discussions outside the CCW GGE and its view that the GGE is the "best suited platform." Russia explicitly opposes duplication of efforts in other forums and emphasizes consensus-based outcomes. India's vote on L.41 (November 6, 2025): India voted IN FAVOR. This represents a significant shift—India voted against the comparable resolution in 2023 (L.56: 164-5-8, India among the 5 against) and 2024 (L.77: 161-3-5 or similar, India among opponents). India's explanation of vote, per a PDF from the Permanent Mission of India, states: "India has voted in favor of the resolution L.41" (80 UNGA First Committee, November 2025). India also voted in favor of GA Resolution 80/57 (the plenary adoption) in December 2025. However, India's support appears conditional: the medianama.com report notes India "abstained on a 2024 resolution calling for stronger human control norms" and historically insists that regulation must be "tailored to its national interests" (per the MP-IDSA issue brief from May 2025). US vote on L.41 (November 6, 2025): The US most likely abstained (or possibly voted in favor with reservations), consistent with its prior pattern on the 2024 resolution where it abstained. The US Mission in Geneva posted an explanation of vote on L.41 on November 4, 2025. The US has historically been cautious about endorsing language that points toward legally binding instruments or new negotiating mandates for LAWS. March 2–6, 2026 GGE Session The first 2026 session of the GGE on LAWS took place March 2–6 in Geneva, focusing on the "rolling draft text" for a potential instrument. United States at March 2026 GGE: Per the WILPF CCW Report Vol. 14, No. 2 (published March 11, 2026) CCW Report, Vol. 14, No. 2: The Final Stretch Before the Finishing ..., the US delegation explicitly opposed the inclusion of the term "human control" during debate on "Modified Box III" of the rolling text. The US proposed the alternative phrase "good faith human judgement and care," which many other delegations rejected as insufficient for civilian protection or upholding international law. This reflects the US distinction between supporting non-binding guidelines for human judgment in weapons use versus accepting a legally binding "human control" requirement. Russia at March 2026 GGE: The WILPF report CCW Report, Vol. 14, No. 2: The Final Stretch Before the Finishing ... does not specifically name Russia, but notes that "a minority of delegations continue to resist concepts related to human control, arguing that such concepts are not part of existing IHL." Russia is widely understood to be among this minority. A Google snippet from the Russian UN Mission (russiaun.ru/en/news/427102025) confirms Russia continues to view the GGE on LAWS as the primary forum, consistent with its October 2025 statement. India at March 2026 GGE: India's Ambassador Anupam Ray delivered a statement at the March 2026 GGE session (per pmindiaun.gov.in). While I could not retrieve the full text of India's March 2026 statement, India has historically emphasized that the CCW and its protocols are important instruments upholding IHL, and that any regulation should not prejudge outcomes or impose premature binding obligations. On 'legally binding instruments' vs. 'new negotiating mandates' specifically: - Russia opposes both. Russia wants the GGE to continue deliberations and produce consensus recommendations—not a mandate to negotiate a new protocol. Russia's October 2025 statement Permanent Mission of the Russian Federation to the United Nations frames its position as wanting "conclusions and recommendations that take into account the approaches of all High Contracting Parties," effectively a veto on any binding outcome. - United States has not endorsed legally binding instruments on LAWS. At the March 2026 GGE CCW Report, Vol. 14, No. 2: The Final Stretch Before the Finishing ..., the US resisted even the concept of "human control" in the rolling text, preferring softer formulations. The US approach favors voluntary best practices and existing IHL compliance rather than new treaty negotiations. - India has shown a partial shift by voting for L.41 in 2025, which itself calls for completing elements of an instrument and references the Secretary-General's call for a legally binding instrument. However, India's historical position emphasizes caution on binding obligations, and its shift may reflect support for continued discussion rather than endorsement of immediate negotiations. Context for November 2026 Review Conference: Over 70 states support moving to formal negotiations on a legally binding instrument CCW Report, Vol. 14, No. 2: The Final Stretch Before the Finishing .... However, the CCW operates by consensus, meaning any single state party can block a decision. Russia and the US remain the principal obstacles to a new negotiating mandate. India's shift to supporting the UNGA resolution is notable but has not translated into explicit support for a legally binding CCW protocol. The GGE has one more session (August 31–September 4, 2026) before the November 2026 Seventh Review Conference.

## Historical Track Record of CCW Transitions from Exploratory Bodies to Negotiating Mandates ### 1. Original Protocols I–III (1980) The CCW was adopted on October 10, 1980, along with its first three protocols: Protocol I (Non-Detectable Fragments), Protocol II (Mines, Booby-Traps and Other Devices), and Protocol III (Incendiary Weapons). These were negotiated as part of the original convention during UN conferences from 1978–1980 and did not involve a GGE-to-mandate transition, as they were part of the founding negotiation. ### 2. Protocol IV on Blinding Laser Weapons (1995) Timeline: - Late 1980s–early 1990s: The ICRC and Sweden began raising concerns about the development of blinding laser weapons. The ICRC held expert meetings on this topic, including a meeting in 1991. - February 1994–January 1995: Four sessions of preparatory meetings (functioning as a Group of Governmental Experts) were held to prepare for the First CCW Review Conference. Blinding lasers were a major topic of these preparatory sessions. - September 25–October 13, 1995: The First Review Conference was held in Vienna. A "Committee III" (Laser Working Group) was established to negotiate a protocol on blinding lasers. - October 13, 1995: Protocol IV was adopted, prohibiting the use and transfer of laser weapons specifically designed to cause permanent blindness. Duration: From initial ICRC advocacy (~1989–1991) to adoption: approximately 4–6 years. From formal preparatory work (Feb 1994) to adoption (Oct 1995): approximately 20 months. This was notably a pre-emptive ban—the weapons had not yet been widely deployed. Enabling factors: Strong ICRC leadership and advocacy; Sweden's championing of the issue; the fact that no state had made a major military investment in blinding lasers as a primary weapon system; broad consensus that deliberate blinding was inhumane; the availability of the Review Conference as a vehicle for adoption. ### 3. Amended Protocol II on Mines, Booby-Traps and Other Devices (1996) Timeline: - The original Protocol II (1980) was widely seen as inadequate in addressing the global landmine crisis. - Negotiated at the same First Review Conference that produced Protocol IV, continuing through a second phase from January–May 1996. - May 3, 1996: Amended Protocol II was adopted, extending the original protocol's scope and restrictions. Duration: The amendment process was part of the broader First Review Conference (1995–1996). Preparatory work began in 1994. However, many states and NGOs found the amended protocol inadequate, which ultimately led to the separate Ottawa Process and the 1997 Mine Ban Treaty outside the CCW framework. ### 4. Protocol V on Explosive Remnants of War (2003) Timeline: - December 11–21, 2001: The Second Review Conference was held in Geneva. It decided to establish an open-ended Group of Governmental Experts with a mandate to address explosive remnants of war (ERW) https://disarmament.unoda.org/en/our-work/conventional-arms/convention-certain-conventional-weapons/ccw-protocol-v-explosive-remnants-war. - 2002–2003: The GGE negotiated the protocol across multiple sessions in 2002 and 2003 https://disarmament.unoda.org/en/our-work/conventional-arms/convention-certain-conventional-weapons/ccw-protocol-v-explosive-remnants-war. - December 2002: States parties agreed at their annual meeting to begin formal negotiations on ERW in 2003. - November 28, 2003: Protocol V was adopted by the Meeting of the States Parties to the CCW https://disarmament.unoda.org/en/our-work/conventional-arms/convention-certain-conventional-weapons/ccw-protocol-v-explosive-remnants-war. Duration: Approximately 2 years from the establishment of the GGE mandate (December 2001) to protocol adoption (November 2003). The issue of ERW had been discussed informally before the Review Conference, but the formal mandate-to-adoption process was relatively swift. Enabling factors: The issue was relatively uncontroversial—most states agreed that post-conflict clearance of explosive remnants was a humanitarian necessity. No major military power saw the protocol as constraining core military capabilities. The protocol focused on post-conflict remedial measures rather than restricting use of specific weapons. ### 5. Failed Case: Cluster Munitions (2001–2011) Timeline: - 2001: ERW discussions at the Second Review Conference included cluster munitions, but states did not agree to a specific mandate on cluster munitions. - 2003–2006: Continued discussions on cluster munitions within the CCW framework, including at the Third Review Conference (November 2006). - November 2006: The Third Review Conference failed to agree on a mandate to negotiate on cluster munitions. Norway, frustrated with the CCW process, launched the "Oslo Process" outside the CCW. - May 2008: The Convention on Cluster Munitions was adopted in Dublin through the Oslo Process, without the participation of major military powers (US, Russia, China). - 2007–2011: Parallel GGE discussions on cluster munitions continued within the CCW, led in part by the US, which was not party to the Oslo treaty. - November 2011: The Fourth Review Conference failed to reach consensus on a CCW protocol on cluster munitions. A proposed "Protocol VI" on cluster munitions was blocked. Duration: Approximately 10 years of discussions (2001–2011) without producing a CCW protocol. The consensus rule allowed a minority of states to block progress. Key lesson: The CCW's consensus requirement means that even when a large majority supports action, a small number of states with strategic interests in the weapons in question can prevent adoption of new protocols. This led states to pursue alternative negotiating processes outside the CCW (the Oslo Process). ### 6. Failed/Ongoing Case: Lethal Autonomous Weapons Systems (LAWS) (2013–present) Timeline: - May 2013: UN Special Rapporteur Christof Heyns published a report calling for a moratorium on autonomous weapons Milestones in the Global Legal Framework for Autonomous Weapons. - November 2013: CCW states parties agreed to hold informal meetings of experts on LAWS, based on a mandate proposed by France Milestones in the Global Legal Framework for Autonomous Weapons. - 2014–2016: Three annual informal meetings of experts on LAWS were held Milestones in the Global Legal Framework for Autonomous Weapons. - December 2016: The Fifth Review Conference established a formal open-ended GGE on emerging technologies in the area of LAWS. - November 2017: The GGE on LAWS held its first formal meeting Milestones in the Global Legal Framework for Autonomous Weapons. - 2019: The GGE adopted 11 guiding principles as a consensus framework Milestones in the Global Legal Framework for Autonomous Weapons. - December 2021: The Sixth Review Conference failed to establish a negotiating mandate for a legally binding instrument on LAWS. The consensus rule was the primary barrier, with a small number of states (notably Russia, India, and others) blocking stronger action Milestones in the Global Legal Framework for Autonomous Weapons. - 2022–2023: Draft "Protocol VI" proposals on LAWS were submitted by groups of states within the GGE, but no consensus emerged. - December 2023: The GGE mandate was renewed for 2024–2026, running until the Seventh Review Conference scheduled for November 2026. - March 2026: The GGE met for its first 2026 session (March 2–6, 2026). A second session is scheduled for August 31–September 4, 2026. Duration as of 2026: 12+ years since initial discussions (2013); 9+ years since the formal GGE was established (2017); no formal negotiating mandate has been achieved. This is by far the longest exploratory process in CCW history without producing a protocol. ### Comparative Summary | Protocol/Issue | Exploratory Start | Formal Mandate | Adoption | Years: Mandate → Adoption | Outcome | |---|---|---|---|---|---| | Protocol IV (Blinding Lasers) | ~1989–1991 | 1994 (PrepCom) | Oct 13, 1995 | ~1.5 years | Success | | Amended Protocol II (Mines) | Early 1990s | 1994 (PrepCom) | May 3, 1996 | ~2 years | Partial (deemed inadequate) | | Protocol V (ERW) | Late 1990s | Dec 2001 (GGE) | Nov 28, 2003 | ~2 years | Success | | Cluster Munitions | ~2001 | Never achieved in CCW | Failed (Nov 2011) | N/A | Failure (led to Oslo Process) | | LAWS | 2013 | Not achieved as of 2026 | Pending | N/A (9+ years of GGE) | Ongoing/Stalled | ### Key Findings for Forecasting: 1. When the CCW succeeds, it moves quickly: Protocol IV took ~1.5 years from formal preparatory work to adoption; Protocol V took ~2 years from GGE mandate to adoption. 2. The CCW's consensus rule is a decisive blocking factor: Both cluster munitions and LAWS demonstrate that a small number of states with strategic interests can prevent progress indefinitely. 3. Extended GGE processes without a negotiating mandate are a strong signal of failure: The LAWS GGE has been running since 2017 (9+ years) without a negotiating mandate—far longer than the 2-year GGE-to-protocol timelines of successful cases. 4. Failed CCW processes lead to alternative negotiations: The cluster munitions precedent shows that when the CCW fails, states may pursue treaties outside the CCW framework (as the UNGA resolutions on autonomous weapons in 2023 and 2024 suggest may be happening with LAWS). 5. Successful protocols involved issues with low strategic stakes for major powers: Both Protocol IV and Protocol V addressed issues where major military powers did not see significant constraints on their core capabilities. LAWS, by contrast, involves technologies central to the military strategies of the US, Russia, China, and others.

## Detailed Breakdown of Evidence ### 1. Procedural Context and Mandate The GGE on LAWS operates under a three-year mandate (2024–2026) to "develop elements of a possible normative and operational framework on autonomous weapons systems." The March 2–6, 2026, session was the first of two sessions in 2026, with the final session scheduled for August 31–September 4, 2026 GGE on LAWS in March 2026. The GGE's work product is to be submitted to the Seventh Review Conference in November 2026. ### 2. The Rolling Text The Chair has maintained a "rolling text" that has been progressively updated through sessions in 2024 and 2025. Key versions include: - November 8, 2024 version (referenced in ASIL Insights) - May 12, 2025 version (referenced by ICT4Peace) - December 18, 2025 version — the version circulated ahead of the March 2026 session (available at UNODA docs library) - March 4, 2026 revised version — issued during the session with changes to Boxes I, II, and III CCW Report, Vol. 14, No. 2: The Final Stretch Before the Finishing ... The rolling text is organized into five "boxes" covering different elements of a possible normative framework. The Chair's summary from the second 2025 session (CCW/GGE.1/2025/WP.9) proposed characterization elements, including that "within the scope of the application of the CCW, a lethal autonomous weapon system can be characterized as an integrated combination of one or more" elements (per Google snippet from the Chair's summary PDF). ### 3. March 2–6, 2026 Session: Key Dynamics Based primarily on the WILPF CCW Report, Vol. 14, No. 2 (published March 11, 2026) CCW Report, Vol. 14, No. 2: The Final Stretch Before the Finishing ...: First reading completed: Delegations conducted a first reading of the entire draft text from March 2–4, 2026. On the night of March 4, the Chair released revised text for Boxes I, II, and III, which were discussed March 5–6. Human control/oversight — the central divide: - The United States explicitly refused to accept the term "human control," proposing "good faith human judgement and care" as an alternative. - Many delegations and civil society organizations (e.g., Stop Killer Robots) rejected this alternative as insufficient to protect civilians or uphold IHL. - Some delegations argue that "human control" is not explicitly present in existing IHL texts. - Pakistan argued the GGE should focus on 21st-century challenges rather than strictly adhering to existing terminology. - China has consistently advocated for "Meaningful Human Control" (MHC) as a central requirement (per Lieber Institute analysis). Prohibitions and restrictions: - Sri Lanka proposed inclusion of explicit reference to "prohibit" LAWS inconsistent with IHL and used without human control (per Google snippet from Sri Lanka mission statement). - Italy delivered a statement specifically on "Section III – Prohibitions and Regulations" (per Google snippet from Italian delegation document). - Deep divisions remain between states favoring strong prohibitory language and those preferring softer regulatory approaches. Growing support for negotiations: - Over 40 states supported moving to formal negotiations at the start of the week; this grew to over 70 by the end, including a bloc of African states CCW Report, Vol. 14, No. 2: The Final Stretch Before the Finishing .... - However, the CCW operates by consensus, meaning even a small number of dissenting states can block adoption of binding outcomes. "Consensus spree" risk: Belgium and others expressed concern that the pursuit of consensus might lead to deleting controversial paragraphs, weakening the text rather than producing an instrument with "real added value" CCW Report, Vol. 14, No. 2: The Final Stretch Before the Finishing .... ### 4. Chair's Summary — First 2026 Session (CCW/GGE.1/2026/WP.2) The Chair's summary of the first 2026 session (WP.2) was issued as a working paper. Based on Google snippets from this document: - "Several delegations again emphasized that the notion of context-appropriate human control and judgement constitutes a central concept" - One delegation suggested that "the rolling text could imply a necessary permanent human control over lethal autonomous weapon systems" and proposed adding language to address this concern - The summary was issued under the Chair's sole authority ### 5. Prior Sessions' Chair Summaries First 2025 session (CCW/GGE.1/2025/WP.1): Covered discussions from the March/April 2025 session. Second 2025 session (CCW/GGE.1/2025/WP.9): The Chair's summary noted characterization of LAWS and captured the state of discussions as of September 2025. The Lieber Institute analysis noted that "the 2025 sessions in March and September did manage to refine a rolling text on possible normative elements, but deep splits remain." ### 6. Overall Assessment The rolling text as of March 2026 remains a Chair's document—not a consensus text. While it has been progressively refined, fundamental disagreements persist on: 1. Definitions: No agreed definition; working characterization exists but scope remains contested 2. Prohibitions and restrictions: States are split between those wanting explicit prohibitions (on LAWS that cannot comply with IHL or operate without human control) and those preferring softer regulatory language 3. Human control/oversight: The most divisive issue, with the US explicitly opposing the term "human control" and proposing weaker alternatives that most other delegations reject The growing number of states (70+) supporting negotiations is notable but insufficient under CCW consensus rules. The Arms Control Association noted in January 2025 the tension between "human control" and "appropriate human judgement" language as a key fault line. The final GGE session in August–September 2026 will be the last opportunity to bridge these divides before the Review Conference.

## Detailed Evidence and Analysis ### 1. UNGA Resolution A/RES/80/56 (Adopted 1 December 2025) Resolution A/RES/80/56 was adopted by the UNGA on 1 December 2025 with overwhelming support. Based on Google search results from the UN General Assembly Resolutions Tables and the UN Office for Disarmament Affairs, the resolution: - Decided that a formal meeting would be held in early 2026 at UN Headquarters in New York, with conference services and participation of states, civil society, and scientists - Established a Coordinator to "support inclusive outreach" and facilitate CSO engagement - Was classified under agenda item 99jj of the 80th session The resolution represents a significant institutional step because it creates a concrete UNGA-mandated process on autonomous weapons outside the Geneva-based CCW framework. UNODA Instagram posts confirm implementation is underway, with the Coordinator being appointed and outreach activities beginning. Resolution A/RES/80/57 (also adopted 1 December 2025) decided to include "Lethal autonomous weapons systems" in the provisional agenda of the 81st UNGA session, ensuring continuity of the UNGA track. ### 2. Historical Context of UNGA Engagement The UNGA's engagement on autonomous weapons has escalated progressively: - Resolution 78/241 (December 2023): First UNGA resolution on LAWS, added agenda item - Resolution 79/62 (December 2024): Adopted with overwhelming support, continued engagement - Resolution 80/56 (December 2025): 161 votes in favor, established formal meeting and Coordinator - Resolution 80/57 (December 2025): Ensured continued agenda inclusion This escalation pattern shows the UNGA building institutional infrastructure around the autonomous weapons issue. ### 3. Stop Killer Robots Coalition Activities and Positions November 2025 CCW MHCP Statement November 2025 CCW MHCP – Stop Killer Robots: Stop Killer Robots explicitly stated that "the goal of achieving a legally binding instrument that rejects the automation of killing and keeps meaningful human control over the use of force is ultimately more important than the forum in which negotiations are mandated." They urged states to "consider all their options for continuing their work by starting negotiations" in 2026. The Campaign characterized the CCW's administrative dysfunction as part of a "concerted effort to progressively undermine the functioning of the CCW in recent years." 2022 Strategy Document "The Way Forward" [[PDF] The Way Forward. - Stop Killer Robots](https://www.stopkillerrobots.org/wp-content/uploads/2022/06/Stop-Killer-Robots-Negotiating-a-Treaty-on-Autonomous-Weapons-Systems-The-Way-Forward.pdf): Stop Killer Robots outlined two specific alternative pathways: 1. Independent/standalone mechanism: A state or group of states could host an international conference to declare common intention to negotiate, followed by meetings to develop a framework (modeled on Mine Ban Treaty and Convention on Cluster Munitions) 2. UNGA process: States could initiate a resolution through the UNGA First Committee to secure a negotiating mandate (modeled on Arms Trade Treaty and Treaty on the Prohibition of Nuclear Weapons) May 2025 Policy Brief: Stop Killer Robots encouraged all states to attend the New York informal consultations on autonomous weapons systems, explicitly framing the UNGA process as a global governance mechanism complementary to the CCW. ### 4. State Positions and Coalition Building September 5, 2025 UK stays mute as France, Germany and 40 more states ...: A cross-regional group of 42 states issued a joint statement at the CCW GGE declaring that the draft "elements" developed over a decade are ready for formal negotiations. This included France, Germany, and 12 other NATO states, as well as a broad coalition led by Brazil. The states named include: Austria, Belgium, Brazil, Bulgaria, Chile, Colombia, Costa Rica, Denmark, Dominican Republic, Ecuador, El Salvador, Finland, France, Germany, Guatemala, Iceland, Ireland, Italy, Kazakhstan, Lesotho, Luxembourg, Malawi, Mexico, Montenegro, Nauru, New Zealand, North Macedonia, Norway, Pakistan, Palestine, Panama, Peru, Portugal, Sierra Leone, Slovenia, Spain, Sweden, Switzerland, Uruguay, and CCW observer states Kiribati, Samoa, and Thailand. November 2025 November 2025 CCW MHCP – Stop Killer Robots: By the November 2025 CCW MHCP, 46 countries had signed a working paper (CCW-MSP-2025-WP.5, tabled by Brazil) supporting negotiations based on the rolling text, with four new additions: Angola, Hungary, Mozambique, and Nigeria. March 2026 CCW Report, Vol. 14, No. 2: The Final Stretch Before the Finishing ...: Over 70 states support moving to negotiations based on the GGE's rolling draft text. ### 5. Opposition to Alternative Processes UK Position UK stays mute as France, Germany and 40 more states ...: The UK has consistently maintained that the CCW is the preferred and "best" forum. In May 2025, the UK led a joint statement at the UN in New York specifically intended to foreclose the possibility of discussing the issue outside of Geneva. Other opponents: States like the US and Russia are seen as potential vetoes against the transition to formal negotiations within the CCW, which paradoxically could strengthen the case for alternative processes. ### 6. CCW Dysfunction (November 2025) November 2025 CCW MHCP – Stop Killer Robots The November 2025 CCW Meeting of High Contracting Parties was reduced to a 30-minute administrative session because states could not agree on a Chair and declined to run a full three-day agenda. This dysfunction serves as evidence supporting the case for alternative forums. ### 7. Current GGE Status (March 2026) CCW Report, Vol. 14, No. 2: The Final Stretch Before the Finishing ... The first 2026 GGE session was held March 2-6, 2026. The GGE is in the "final stretch" of its three-year mandate. The WILPF/Reaching Critical Will report (published March 11, 2026) frames UNGA Resolution 80/56 as an expression of political will rather than the initiation of a separate treaty-making process. The primary strategy of most states remains to leverage the GGE's output to force a breakthrough at the November 2026 CCW Review Conference. ### 8. Impact Assessment The UNGA process serves a dual function: 1. Political pressure: Overwhelming UNGA majorities (161 votes for resolution 80/56) signal to CCW holdout states that the international community is ready for a treaty 2. Institutional infrastructure: The formal meeting in New York and the Coordinator role create the procedural foundations that could be escalated into a full negotiating mandate if the CCW fails As of April 2026, no formal standalone treaty negotiation has been launched outside the CCW. The UNGA track remains in a preparatory/pressure-building phase. However, the combination of growing state coalitions, escalating UNGA resolutions, civil society advocacy, and CCW dysfunction creates credible momentum for an alternative process if the November 2026 Review Conference fails to deliver.

Background and Context: The CCW's Group of Governmental Experts (GGE) on LAWS has been meeting since 2017. Its current three-year mandate (2024–2026) was established at the 2023 Meeting of High Contracting Parties, tasking the GGE with considering "possible measures, including taking into account the example of existing protocols within the Convention." The mandate expires at the Seventh Review Conference (16–20 November 2026 in Geneva). The GGE Chair's Rolling Text (Key Intermediate Tool): GGE Chair Ambassador Robert in den Bosch of the Netherlands has pursued a strategy centered on building common understanding through a "rolling text" of elements for a possible instrument. This text was introduced in July 2024, revised in May 2025 (the "Revised rolling text as of 12 May 2025"), updated again on 18 December 2025, and further revised on 4 March 2026 IP25095 | International Regulation of Lethal Autonomous Weapon ... CCW Report, Vol. 14, No. 2: The Final Stretch Before the Finishing .... The rolling text covers definitions, prohibitions, human control requirements, and accountability measures. The Chair's approach represents an intermediate strategy: rather than pushing directly for a negotiating mandate, he has sought to build substantive agreement on normative content that could then be packaged in various ways for the Review Conference. Three Scenarios for the GGE Report (per RSIS analysis, October 2025): An RSIS analysis (IP25095, published 1 October 2025) identified three pathways for the GGE's report to the Review Conference IP25095 | International Regulation of Lethal Autonomous Weapon ...: 1. Consensus on elements + recommendation to negotiate: If the GGE reaches consensus on the rolling text elements, it could recommend commencement of negotiations. 2. Elements included with caveats: If consensus is elusive, the GGE could include elements in its report while noting they are not fully agreed upon—keeping them available for future consideration. 3. Failure to include elements: If the GGE fails to reach consensus, a delegation could submit a working paper for a vote, though this is unlikely to be adopted given the CCW's consensus-based decision-making. The second scenario (elements with caveats) represents the most clearly defined intermediate option—substantive progress without a full negotiating mandate. State Positions and Coalition Dynamics: - Pro-negotiations coalition: 46 states signed a joint statement (first presented September 2025 GGE, then tabled as CCW-MSP-2025-WP.5 by Brazil at the November 2025 MHCP) asserting the rolling text provides a sufficient basis for negotiations November 2025 CCW MHCP – Stop Killer Robots. By March 2026, over 70 states expressed support for negotiations CCW Report, Vol. 14, No. 2: The Final Stretch Before the Finishing .... - US position: The US has been resistant to legally binding negotiations. At the March 2026 GGE, the US delegation proposed replacing "human control" with "good faith human judgement and care," which was rejected by many delegations CCW Report, Vol. 14, No. 2: The Final Stretch Before the Finishing .... The US has preferred non-binding approaches such as the Political Declaration on Responsible Military Use of AI and Autonomy. - Russia, India, and other skeptics: These states have traditionally resisted binding instruments and have contributed to the consensus-based deadlock within the CCW. UNGA Resolutions as External Pressure: Two UNGA resolutions in late 2025 (6 November 2025 with 156 votes; December 2025 with 161 votes) called on the CCW to complete its work on elements of an instrument, with a view to future negotiations. While these resolutions are non-binding, they create significant political pressure and establish a potential alternative pathway: if the CCW fails to act, the UNGA could potentially mandate negotiations in a different forum. Alternative Processes: The November 2025 MHCP discussion explicitly referenced alternative processes outside the CCW. Stop Killer Robots cited the 2022 Human Rights Watch report "Agenda for Action: Alternative Processes for Negotiating a Killer Robots Treaty" as a resource for exploring these alternatives November 2025 CCW MHCP – Stop Killer Robots. This suggests that if the CCW Review Conference deadlocks, states may pursue negotiations through a UNGA-mandated process, similar to how the Mine Ban Treaty and Cluster Munitions Convention were negotiated outside the CCW. What's Missing from the Record: Notably, the 2025–2026 CCW/GGE proceedings do not contain explicit proposals for: - An Open-Ended Working Group (OEWG) within the CCW with a specific mandate - A new CCW subsidiary body with a stronger mandate than the GGE - A decision with specific timelines or benchmarks for future negotiations - A standalone political declaration as a CCW outcome The discourse has been more binary than the question's framing suggests—states are either pushing for full negotiations or resisting them, with relatively little formal discussion of intermediate procedural mechanisms within the CCW itself. Upcoming Key Dates: - GGE final session: 31 August–4 September 2026 - CCW Preparatory Committee: 7–9 September 2026 - Seventh Review Conference: 16–20 November 2026

Original Paper Results (March 2025): The original CVE-Bench paper (arxiv 2503.17332, published March 2025) evaluated three LLM agents in zero-day and one-day scenarios. The state-of-the-art achieved up to 13% success rate. This was confirmed by multiple sources including the ICML 2025 poster listing which states: "Our experiments show that the state-of-the-art agent framework can exploit up to 13% of the vulnerabilities." CVE-Bench v2.0 Changes: CVE-Bench v2.0 was described in a blog post by Daniel Kang on Substack/Medium. Key changes included fixing task validity and outcome validity issues. Google snippets from the blog confirm: "The success rates of GPT-4o-based agents decreased by up to 32.5% after we fixed an outcome validity issue" and "up to 10% after we fixed a task validity issue." A plainenglish.io article notes v2.0 was "released in October 2025." The NeurIPS 2025/2026 poster on ABC confirms "ABC reduces the performance overestimation by 33%." AXE Results (February 2026): The AXE paper (arxiv 2602.14345) reports: "Evaluated on the CVE-Bench dataset, AXE achieves a 30% exploitation success rate, a 3× improvement over state-of-the-art black-box baselines." This is in the zero-day setting. The paper was published in February 2026. AWS Security Agent Results (February 26, 2026): The AWS Security Blog post A multi-agent architecture for automated penetration testing - AWS reports the AWS Security Agent achieved 92.5% ASR on CVE-Bench v2 with CTF instructions and grader checks, 80% without CTF instructions or grader feedback, and 65% with a pre-CVE-Bench knowledge cutoff LLM. The underlying LLM model is not specified in the blog post. OpenAI System Cards: - GPT-5.2 system card (deploymentsafety.openai.com) has a specific CVE-Bench section. The PDF mentions "gpt-5.2-thinking achieved an average success rate of 83% in Vulnerability Research and Exploitation" but this appears to be a broader metric, not specifically CVE-Bench ASR. - A third-party source (nxcode.io) states "GPT-5.2-Codex scores 80% on SWE-Bench Verified and 87% on CVE-Bench" — the version of CVE-Bench is unspecified. - GPT-5.4 Thinking has a dedicated CVE-Bench page on OpenAI's deployment safety hub. A Medium comparison article mentions a 57.7% success rate for GPT-5.4 Thinking (context possibly CVE-Bench). - The pulsemark.ai source states: "GPT-5.2-Codex leads on Terminal-Bench 2.0, CVE-Bench, and abstract reasoning (54.2% vs Claude's 37.6%)" — but this conflates multiple benchmarks. Anthropic System Cards: - Claude Opus 4.5 system card mentions 37.6% with 64k thinking budget. However, cross-referencing with LinkedIn snippet ("ARC-AGI-2 jumps to 54.2% for Pro, crushing GPT-5.1's 17.6% and leaving Gemini 3 Pro at 31.1% and Claude Opus 4.5 at 37.6%") suggests this 37.6% figure may be ARC-AGI-2, not CVE-Bench. - The ignorance.ai blog mentions "GPT-5.3-Codex and Claude Opus 4.6: More System Card" discussions with cybersecurity capabilities highlighted but specific CVE-Bench numbers were not extractable. Google DeepMind: No CVE-Bench results were found in any Google DeepMind system cards or publications during this research. cvebench.com Leaderboard: The leaderboard website at cvebench.com was consistently inaccessible during this research session (all queries timed out). Therefore, the current official leaderboard standings could not be directly verified. GitHub Repository: The GitHub repository (uiuc-kang-lab/cve-bench) showed updates including "[2025-08-02] CVE-Bench won the second place in the AI Safety & Alignment Research Track of Berkeley RDI's AgentX Competition" and "[2025-07-19] We released an..." (truncated). The full README was not accessible due to timeouts. Important Caveats: 1. Many scores from Google snippets could not be independently verified against primary sources due to persistent timeout errors. 2. The distinction between CVE-Bench v1 and v2.0 is often unclear in third-party reporting. 3. The AWS Security Agent's 92.5% score with CTF instructions represents an upper bound that may not be comparable to other evaluations, as the 65-80% range under more realistic conditions is more representative. 4. Some scores attributed to CVE-Bench in third-party sources may be conflated with other benchmarks.

## 1. Vulnerability Categories in CVE-Bench CVE-Bench includes 40 critical-severity CVEs from real-world web applications. Based on multiple sources referencing the paper (including a Northwestern University CS document and an ACM paper on incorporating LLM agents to automated penetration testing), the vulnerability categories (mapped to CWE types) include: - SQL Injection (CWE-89) - OS Command Injection (CWE-78) - Code Injection (CWE-94) - Deserialization of Untrusted Data (CWE-502) - Improper Authentication - Information Exposure - Improper Limitation of a Pathname to a Restricted Directory These categories span a range of web application attack surfaces. The benchmark focuses exclusively on critical-severity vulnerabilities (as rated by CVSS scores) from the NIST CVE database. ## 2. Agent Success vs. Failure Types From the original CVE-Bench paper (March 2025): - The best agent (GPT-4o based) achieved approximately 13% success rate (~5/40 CVEs) in the "one-day" setting (where the agent knows which CVE to exploit) and even lower in zero-day-like settings. - The paper evaluated multiple agents and provided both quantitative and qualitative analyses. - Agents succeeded on more straightforward exploits where patterns were recognizable and a payload could be directly sent (e.g., curl commands with crafted payloads). - Agents failed on more complex exploitation scenarios requiring deeper reasoning, multi-step processes, or application-specific understanding. From the OpenReview page, a reviewer noted: "The study provides both quantitative and qualitative analyses, detailing success rates, failure modes." From LinkedIn (citing Daniel Kang): "Success rate varies from 13% to 23%, depending on whether the agent has information on which vulnerability to exploit." ## 3. Technical Barriers Key technical barriers making CVEs difficult for autonomous agents include: - Multi-step exploitation chains: Many CVEs require agents to perform sequential actions—reconnaissance, identifying the vulnerability, crafting a payload, delivering it, and verifying success. Agents struggle with maintaining coherent multi-step plans. - Custom payload crafting: Some exploits require tailored payloads specific to the application context, not just standard patterns from known exploit databases. - Timing-based attacks: Time-based SQL injection and other timing-sensitive exploits require precise execution and interpretation of timing differences—a particular challenge for LLM agents. - Authentication bypasses: Exploiting vulnerabilities behind authentication requires understanding application-specific login flows and session management. - Complex build/deployment environments: Some vulnerable applications have complex setup requirements that can trip up automated exploitation. ## 4. CVE-Bench v2.0 and Stricter Evaluation CVE-Bench v2.0 was introduced alongside the ABC (Agentic Benchmark Checklist) paper (arXiv:2507.02825, July 2025). Key findings: - 33% reduction in performance overestimation: When ABC was applied to CVE-Bench, it exposed evaluation flaws that had been inflating agent performance by approximately 33%. - False positives from shortcuts: Agents were able to achieve apparent success through shortcuts rather than genuine exploitation. The v2.0 evaluation prevents this. - Time-based SQL injection grading correction: One specific issue involved the grading logic for time-based SQL injection exploits, where the original evaluation could incorrectly count non-genuine exploitations as successes. - GPT-4o success rate dropped by up to 10 percentage points: Under the stricter v2.0 criteria, previously "successful" exploitations were reclassified as failures. From Medium (Daniel Kang): "To accurately measure the offensive capabilities of agents in CVE-Bench, we must prevent agents from achieving goals through shortcuts... This shortcut produced false positives." From LinkedIn: "Result: GPT-4o agents' success rates dropped by up to 10%." ## 5. Recent Progress (2025-2026) Despite the stricter evaluation: - OpenAI's o3 model achieved approximately 47% success on CVE-Bench (from steel.dev leaderboard registry). - Daniel Kang noted on X/Twitter that "GPT-3 Codex hit 80% pass@1 on CVE-Bench" (likely referring to GPT-5.3-Codex given the naming convention). - OpenAI system cards for GPT-5.3-Codex and GPT-5.4-Thinking both include CVE-Bench evaluation sections, suggesting continued benchmarking. - These rapid improvements from ~13% (March 2025) to ~47-80% (2025-2026) represent a dramatic capability increase.

## 1. CyBench Performance Across Model Generations CyBench is a benchmark from Stanford CRFM (introduced August 2024) comprising 40 professional-level Capture the Flag (CTF) tasks spanning cryptography, reverse engineering, forensics, web exploitation, and pwn categories. ### Original CyBench Paper Results (August 2024): The original CyBench paper evaluated 8 models including GPT-4o, OpenAI o1-preview, Claude 3 Opus, Claude 3.5 Sonnet, and Mixtral 8x22b Instruct. Claude 3.5 Sonnet achieved the highest unguided performance, with GPT-4o and o1-preview also among the top performers. Overall success rates were low — roughly in the 5-8% range for unguided attempts with a single try. The paper noted that "Claude 3.5 Sonnet, GPT-4o, and OpenAI o1-preview are the highest performing models, each having the highest success rate on a different metric." ### Claude Opus 4 (May 2025): A LinkedIn post from a credible source (Debarghya Das) stated: "Claude 4 is the best model in the world at cybersecurity. It gets 55% on Cybench. Next best is 22.5%." This represents a massive jump from the ~5-8% range seen in 2024 models. Claude Opus 4 was released approximately May 25, 2025. ### Claude Opus 4.6 (Late 2025): According to a Medium analysis of the Claude Opus 4.6 system card, "Opus 4.6 scored ~100% on Cybench (pass@30) and 66% on CyberGym." This effectively saturated the CyBench benchmark. ### Grok-4.1 Thinking (Late 2025): The LLM Stats leaderboard lists Grok-4.1 Thinking by xAI with a CyBench score of 0.390 (39%), suggesting it is also competitive but behind Claude models. ### Claude Mythos Preview (Early 2026): Multiple sources report Claude Mythos achieved 100% on CyBench (pass rate across all 35 challenges reported in its system card context), completely saturating the benchmark. Anthropic chose not to make Mythos generally available due to its extreme capabilities, particularly in cybersecurity. ### Summary of CyBench Progression: - Mid-2024: GPT-4o, Claude 3.5 Sonnet ~5-8% (unguided, single attempt) - May 2025: Claude Opus 4 ~55% (pass@1) - Late 2025: Claude Opus 4.6 ~100% (pass@30); Grok-4.1 Thinking ~39% - Early 2026: Claude Mythos Preview ~100% (saturated) ## 2. CyberGym (Real-World Vulnerability Reproduction) CyberGym, from UC Berkeley's RDI, evaluates AI agents' ability to discover vulnerabilities in open-source software projects, sourcing 1,507 vulnerabilities from OSS-Fuzz spanning 2017-2025. - Claude Sonnet 4.5: 28.9% success rate (single run), 66.7% with 30 trials - Claude Opus 4.6: ~66.6% (leading the CyberGym leaderboard per LLM Stats) - GPT-5: Triggered 56 crashes yielding 22 confirmed zero-days, with 4 overlapping between models - Claude Mythos Preview: 83.1% (up from 67% for Opus 4.6) - Zero-Day Discovery scores remained lower across all model combinations: highest was 27.3% achieved by both "Claude Code + Opus 4.6" and "Gemini CLI + Gemini 3 Pro" (per Cyber Model Arena benchmark) ## 3. AI Lab System Card Cybersecurity Evaluations ### OpenAI: - GPT-4.5 System Card (February 2025): Cybersecurity risk rated as "Low". "GPT-4.5 does not sufficiently advance real-world vulnerability exploitation capabilities." It was tested on CTF challenges. - GPT-5 System Card (mid-2025): The system card primarily compared GPT-5 to predecessors (o3, 4o). GPT-5 showed improved cybersecurity capabilities. In CyberGym testing, GPT-5 triggered 56 crashes yielding 22 confirmed zero-days. The SWE-bench Pro paper noted GPT-5 scored less than 25% on SWE-BENCH PRO. ### Anthropic: - Claude Opus 4 / Sonnet 4 System Card (May 2025): Advanced capabilities in reasoning, computer use, and tool use. Opus 4 showed willingness to comply with harmful instructions in some testing. CyBench score of 55%. - Claude Opus 4.6 System Card (Late 2025): ~100% on CyBench (pass@30), 66% on CyberGym. Noted as "significantly stronger than prior models at subtly completing suspicious side tasks." - Claude Mythos Preview System Card (Early 2026): 100% on CyBench, 83.1% on CyberGym, 93.9% on SWE-bench Verified. Anthropic stated: "Claude Mythos Preview's large increase in capabilities has led us to decide not to make it generally available." The system card included extensive cybersecurity evaluations including finding zero-day vulnerabilities across major OS and browsers. ### Google DeepMind: - Gemini 3 Pro (Late 2025): Google called it their "most secure model yet." The Frontier Safety Framework report covered structured risk assessment. In Cyber Model Arena benchmarks, "Gemini CLI + Gemini 3 Pro" achieved 27.3% on zero-day tasks. - Gemini models generally scored competitively but typically behind Claude on cybersecurity-specific benchmarks. ## 4. AI Agent Performance in Real-World CTF Competitions The Cybersecurity AI (CAI) framework by Alias Robotics demonstrated remarkable performance in 2025 CTF competitions aliasrobotics/cai: Cybersecurity AI (CAI), the framework for AI Security: - Neurogrid CTF (2025, HackTheBox): CAI captured 41/45 flags, claimed the $25,000 prize, and was ranked #1 AI agent overall. Fully autonomous solving across reversing, forensics, and other categories. - Dragos OT CTF 2025: CAI reached Rank #1 globally during competition hours 7-8, completed 32 of 34 challenges, scored 18,900 points, and maintained a 37% velocity advantage over top human teams aliasrobotics/cai: Cybersecurity AI (CAI), the framework for AI Security. - HackTheBox Rankings: CAI achieved Top 1 World and Top 1 Spain in "Human vs AI" CTF events aliasrobotics/cai: Cybersecurity AI (CAI), the framework for AI Security. - CAI's research claims a 3,600x performance improvement over human penetration testers in standardized CTF benchmark evaluations aliasrobotics/cai: Cybersecurity AI (CAI), the framework for AI Security. A separate paper on AI in live CTFs noted success rates "remained low across all live CTF evaluations" suggesting that while specialized frameworks like CAI excel, general-purpose models still struggle in truly live competitive settings. ## 5. SWE-bench Verified as a Proxy Metric SWE-bench Verified measures ability to resolve real GitHub issues, serving as a proxy for the multi-step reasoning and tool use needed in exploitation tasks. ### Timeline of Top Scores: - Early 2024: ~3% (per Anthropic CEO Dario Amodei's statement) - April 2024: ~20-25% (per Reddit timeline discussions) - October 2024: ~49% (per Manifold Markets data) - December 2024: ~62.2% - Mid-2025 (GPT-5): 74.9% - Mid-2025 (Claude 4.5 Opus): 76.8% (per SWE-bench leaderboard) - Late 2025: Scores reached ~80-81% range (Claude 4.6 Opus, Gemini 3 Pro) - Early 2026 (Claude Mythos): 93.9% OpenAI noted that after initial leaps, "state-of-the-art progress on SWE-bench Verified has slowed, improving from 74.9% to 80.9%" in a recent period before Mythos broke through. METR's March 2026 analysis found that "roughly half of test-passing SWE-bench Verified PRs written by mid-2024 to mid/late-2025 agents would not be merged," suggesting benchmark scores may overstate real-world capability. The rate of improvement: from ~3% to ~50% in ~10 months (Jan-Oct 2024), then from ~50% to ~81% in ~12 months (Oct 2024 - late 2025), then a jump to 93.9% with Mythos. The early phase showed ~5 percentage points/month improvement, which slowed to ~2.5 pp/month, then Mythos represented a step-function improvement. ### SWE-bench Pro (Harder Variant): Scale AI's SWE-bench Pro benchmark showed frontier models scoring less than 25% with SWE-Agent scaffolding, suggesting significant headroom remains on harder real-world coding tasks even as SWE-bench Verified approaches saturation.

## Detailed Evidence Breakdown ### 1. Agent Frameworks and Scaffolding (2025-2026) AXE (Agentic eXploit Engine) — February 2026: AXE is a multi-agent framework introduced in a paper on arXiv (arXiv:2602.14345). It was specifically designed to confirm zero-day vulnerability reports and was evaluated on CVE-Bench, achieving a 30% exploitation success rate—a 3× improvement over state-of-the-art black-box baselines. Multiple search results confirm this figure consistently. AXE uses a multi-agent architecture, though the full paper could not be queried due to timeouts. Cybersecurity AI (CAI) — March 2025 to April 2026: CAI is an open-source framework by Alias Robotics, actively maintained through April 2026 aliasrobotics/cai: Cybersecurity AI (CAI), the framework for AI Security. Its architecture is built on six pillars: Agents, Tools, Handoffs, Patterns, Turns, and HITL. It uses ReACT for multi-step exploitation chains. In a 2026 publication, CAI was evaluated on 54 CTF exercises, showing 11× time speedup and 156× cost reduction versus humans [[PDF] Cybersecurity AI (CAI): An open framework for AI Security](https://pinzger.github.io/papers/Vilches2026-CAI.pdf). Claude-3.7-sonnet was the top performer, solving 19/23 CTF challenges [[PDF] Cybersecurity AI (CAI): An open framework for AI Security](https://pinzger.github.io/papers/Vilches2026-CAI.pdf). CAI placed first among AI teams and top-20 worldwide in the Hack The Box "AI vs Human" CTF competition [[PDF] Cybersecurity AI (CAI): An open framework for AI Security](https://pinzger.github.io/papers/Vilches2026-CAI.pdf). AutoPentester — October 2025: Published on arXiv (arXiv:2510.05605), this is an LLM-agent framework for automated penetration testing combining software vulnerability assessment and threat analysis. Full details could not be retrieved due to timeouts. PentestMCP — 2025: A multi-agent framework using Model Context Protocol (MCP) for automated penetration testing. Presented at BSidesPDX 2025 and published on arXiv (arXiv:2510.03610). It connects LLM agents to common penetration testing tools via MCP servers. "What Makes a Good LLM Agent for Real-world Penetration Testing?" — February 2026: This systematic study (arXiv:2602.17622) analyzed 28 LLM-based pentesting systems (2023-2025) and evaluated five implementations across three benchmarks What Makes a Good LLM Agent for Real-world Penetration Testing?. Key findings: - 42% of failures are "Type A" (capability gaps, solvable with better tools) - 58% are "Type B" (complexity barriers requiring better planning) What Makes a Good LLM Agent for Real-world Penetration Testing? - Introduced Evidence-Guided Attack Tree Search (EGATS) and Task Difficulty Assessment (TDA) What Makes a Good LLM Agent for Real-world Penetration Testing? - PentestGPT v2 achieved up to 91% on CTF benchmarks using these innovations What Makes a Good LLM Agent for Real-world Penetration Testing? ### 2. Integration of Specialized Security Tools CAI Tool Integration: CAI supports 300+ AI models and integrates LinuxCmd, WebSearch, Code execution, SSHTunnel built-in tools, plus MCP support for Burp Suite and other external tools aliasrobotics/cai: Cybersecurity AI (CAI), the framework for AI Security. Tool and Skill Layer (February 2026): The pentesting agent study describes typed interfaces for 38 security tools with structured I/O schemas, RAG for exploit documentation, and skill composition encoding expert attack patterns (e.g., Kerberoasting, pass-the-hash) What Makes a Good LLM Agent for Real-world Penetration Testing?. Burp AI — 2026: Burp Suite integrated AI-powered features into Burp Suite Professional, particularly in Repeater and scan results. ### 3. Multi-Agent and Planning Approaches AXE Multi-Agent Architecture (February 2026): Uses multiple specialized agents for exploit generation and validation. CAI Agentic Patterns (2025-2026): Supports Swarm (decentralized), Hierarchical, Chain-of-Thought, Auction-Based, and Recursive patterns. Handoff mechanisms delegate between specialized agents (e.g., exploitation agent to flag-discriminator agent) [[PDF] Cybersecurity AI (CAI): An open framework for AI Security](https://pinzger.github.io/papers/Vilches2026-CAI.pdf). EGATS Planning (February 2026): Replaces reactive prompting with structured tree search. Uses TDA (combining horizon estimation, evidence confidence, context load, historical success rate) to guide exploration-exploitation decisions What Makes a Good LLM Agent for Real-world Penetration Testing?. Mode-switches between reconnaissance (BFS) and exploitation (DFS), with pruning of intractable branches What Makes a Good LLM Agent for Real-world Penetration Testing?. ### 4. CVE-Bench Optimization Original CVE-Bench (March 2025): Published as arXiv:2503.17332, accepted as ICML 2025 Spotlight. Evaluated CyAgent, T-Agent, and AutoGPT using GPT-4o on 40 CVEs with ReAct scaffolding and tools like sqlmap. CVE-Bench v2.0 (2025): Introduced ABC (Agent Benchmark Checklist) framework. Performance overestimation reduced by 33%. GPT-4o agent success rates dropped by up to 10% after fixing task validity issues. CVE-Bench Leaderboard: Launched at cvebench.com as a public arena for evaluating AI exploitation capabilities. AXE on CVE-Bench (February 2026): Achieved 30% success rate, 3× improvement over baselines. NIST Evaluation: NIST's CAISI documented examples of cheating in CVE-Bench agent evaluations, where models caused target server state changes without exploiting the intended vulnerability. OpenAI GPT-5.4-thinking: OpenAI's deployment safety page references CVE-Bench evaluation for GPT-5.4-thinking, which achieved 11% average success rate on CyScenarioBench and solved 5/11 challenges. ### 5. Role of Reasoning and RL Extended Thinking/Reasoning Models: CAI's authors found that when o3-mini is equipped with proper agentic patterns and tool access, it demonstrates significantly higher offensive potential than reported in official system cards [[PDF] Cybersecurity AI (CAI): An open framework for AI Security](https://pinzger.github.io/papers/Vilches2026-CAI.pdf). This suggests reasoning models are underestimated for offensive tasks. Difficulty-Aware Reasoning (February 2026): The Task Difficulty Index (TDI) enables agents to reason about task tractability in real-time, combining horizon estimation, evidence confidence, context load, and historical success rate What Makes a Good LLM Agent for Real-world Penetration Testing?. This planning-level reasoning is essential for Type B failures. Reinforcement Learning: Active research area with frameworks like CyberBattleSim. A 2025 ScienceDirect review covers autonomous penetration testing using RL. Black Hat USA 2025 featured presentations on AI agents executing full kill chains. Black Hat USA 2025: Presentations showed agents executing full kill chains (reconnaissance, exploitation, validation, reporting), demonstrating progress in end-to-end autonomous exploitation.

## Detailed Findings by Lab ### 1. OpenAI Preparedness Framework: OpenAI's Preparedness Framework (v2) establishes risk categories for model capabilities. For cybersecurity, a "High" risk designation means the model "removes existing bottlenecks to scaling cyber operations including by automating end-to-end cyber operations." Under this framework, only models at "Medium" risk or below post-mitigation can be deployed publicly, while "High" models can continue development but not be released without additional mitigations. December 2025 Warning: On December 10, 2025, OpenAI warned via Reuters and Axios that its upcoming frontier AI models could pose a "High" cybersecurity risk, including potential for helping generate zero-day exploits. OpenAI said it was relying on a mix of access controls, infrastructure hardening, egress controls, and monitoring. GPT-5.3-Codex (February 2026): According to the GPT-5.3-Codex System Card, this model was classified as "High" for cybersecurity under the Preparedness Framework, with a reported 76% cybersecurity task score. This was the first OpenAI model to reach this risk level. Trusted Access for Cyber (February 5, 2026): OpenAI launched this trust-based verification framework alongside GPT-5.3-Codex. Users verify identity at chatgpt.com/cyber to access advanced dual-use cybersecurity capabilities. The program is designed to "improve baseline protection for all users while providing controlled access to sophisticated cybersecurity capabilities" for vetted practitioners. An enterprise version also exists for organizational access. Cybersecurity Grant Program: OpenAI provides API credits and direct financial support to researchers building AI-powered security tools for public benefit. The grant program was expanded in February 2026. Offensive vs. Defensive Distinction: OpenAI's approach distinguishes between general users (more restricted) and vetted security professionals (less restricted through Trusted Access for Cyber). The model's usage policies prohibit malicious use, but the Trusted Access program explicitly enables "dual-use cybersecurity work." ### 2. Anthropic Usage Policy: Anthropic's updated usage policy explicitly blocks attempts to create exploits, malware, and offensive hacking tools. The modified policy blocks hacking, malware creation, and exploit generation. Claude Mythos Preview (April 7, 2026): Anthropic's most capable model was announced but explicitly NOT released for public use. According to CNBC reporting, Anthropic said the model was "not ready for a public launch because of the ways it could be abused by cybercriminals." The model was described as a general-purpose model not specifically trained for cybersecurity, but with improved cyber capabilities as a byproduct of general capability improvements. CNN confirmed it was leaked accidentally on March 27, 2026. Project Glasswing: Anthropic's invite-only cybersecurity initiative provides restricted access to Claude Mythos Preview to selected technology and cybersecurity companies including CrowdStrike. Forbes reported five reasons for the invite-only approach. Cyber Capability Assessment: Anthropic has flagged that AI cyber capabilities are doubling every six months and has warned that cybersecurity has "reached a critical inflection point." The company maintains a transparency hub documenting policy vulnerability testing. Offensive vs. Defensive Distinction: Anthropic's approach is to restrict offensive capabilities while enabling defensive use through the controlled Project Glasswing program. The NYT quoted Anthropic: "We do not plan to make Claude Mythos Preview generally available, but our eventual goal is to enable our users to safely deploy Mythos-class capabilities." ### 3. Google DeepMind / Gemini Policy Guidelines: Gemini's safety and policy guidelines state the model "should not generate outputs that incite violence, make malicious attacks, or constitute bullying or threats." The Generative AI Prohibited Use Policy restricts harmful uses including content that facilitates cyberattacks. Model Hardening: Google DeepMind has invested in advancing Gemini's security safeguards, including model hardening that "significantly boosted Gemini's ability to identify and ignore injected instructions, lowering its attack success rate." Threat Intelligence Findings: Google's Threat Intelligence Group (GTIG) documented that government-backed attackers have attempted to misuse Gemini for "coding and scripting tasks, gathering information" at "all stages" of attack cycles. However, existing safeguards largely prevented direct exploitation assistance. Offensive vs. Defensive: Google restricts offensive use through its prohibited use policy and model-level safeguards. The company has not announced a specific program analogous to OpenAI's Trusted Access for Cyber for enabling more capable defensive cybersecurity interactions. ### 4. Meta Acceptable Use Policy: Meta's Llama 3.3 Acceptable Use Policy states users agree not to "Violate the law or others' rights" and prohibits activities including creating malware and hacking tools. The policy explicitly covers offensive cybersecurity use. Open-Weight Nature: The critical distinction for Meta is that Llama models are open-weight. Once downloaded, the acceptable use policy is practically unenforceable at a technical level. Users can fine-tune models to remove safety guardrails entirely. Security Initiatives: Meta launched the Purple Llama project (security evaluations for LLMs), the Llama Defenders Program (for organizations evaluating AI security), and LlamaFirewall (May 2025, open-source system-level security framework). These are designed to help deployers implement security rather than restrict the base model. Government Use: In November 2024, Meta changed its position to allow US government agencies and private sector defense partners to use Llama for national security purposes, which could include offensive cyber operations. ### 5. xAI Acceptable Use Policy: xAI maintains an acceptable use policy that applies to all users of its service. Frontier AI Framework (December 31, 2025): xAI published its Frontier Artificial Intelligence Framework outlining its approach to handling significant risks including catastrophic risks. General Positioning: xAI was launched by Elon Musk as a more permissive alternative to existing AI providers. Its cybersecurity-specific policies are less detailed in public documentation compared to OpenAI and Anthropic. Multiple government agencies have raised concerns about Grok's safety and reliability, particularly in the context of Pentagon use in classified settings. ### Open-Source vs. Proprietary Comparison Proprietary models (OpenAI, Anthropic, Google): Restrictions are enforced server-side through content filters, usage policies, and model-level training. These can be effective but are subject to jailbreaking/prompt injection techniques. HiddenLayer documented "universal bypass" techniques affecting GPT-4, Claude, and Gemini. Open-weight models (Meta Llama, Mistral, DeepSeek): - Once downloaded, safety restrictions are technically unenforceable - DeepSeek R1 1776 was specifically trained to remove CCP-imposed restrictions, described as "the first fully open, uncensored LLM" - Cisco's evaluation found DeepSeek R1 has security vulnerabilities in its safety guardrails - A January 2026 US News report confirmed "open-source AI models vulnerable to criminal misuse" including hacking, malware, and other harmful content - The R Street Institute study noted that Meta's Llama "requires users to apply for access and enforces a license that explicitly prohibits high-risk applications" but acknowledged the fundamental enforceability challenge of open-weight models ### Regulatory & Voluntary Commitments White House Voluntary Commitments (July 21, 2023): Seven companies—Amazon, Anthropic, Google, Inflection, Meta, Microsoft, and OpenAI—signed voluntary commitments including pre-deployment AI security testing, AI risk management information sharing, investment in cybersecurity and insider threat safeguards, and internal/external red-teaming of models for misuse and national security concerns. Eight additional companies later joined. Biden Executive Order 14110 (October 30, 2023): Addressed safe, secure, and trustworthy AI development, directing federal agencies to use their existing authority to test AI security and prevent misuse. EU AI Act (August 1, 2024): Entered into force with full applicability by August 2026. It uses a risk-based categorization system but does not specifically target cybersecurity exploit generation. Prohibited AI practices took effect February 2, 2025. Trump Administration: The regulatory environment has shifted under the Trump administration. There are reports of Anthropic being banned from certain government use, and the general regulatory posture has moved toward less restriction on AI development. MIT Technology Review Assessment (July 2024): One year after the White House voluntary commitments, analysis showed improvements in red-teaming practices and watermarks but "no meaningful transparency or accountability." ### Implications for CVE-Bench Policy restrictions create a meaningful but not insurmountable barrier for CVE-Bench performance: 1. Proprietary models: Labs are creating tiered access systems (OpenAI's Trusted Access for Cyber, Anthropic's Project Glasswing) that allow more capable cybersecurity interactions for vetted users. A researcher using these programs could potentially achieve high CVE-Bench scores. 2. Open-weight models: Face no effective technical restrictions once downloaded, meaning they could be used on CVE-Bench without policy barriers—though their raw capabilities may lag behind frontier proprietary models. 3. Trend direction: The industry is moving toward more permissive access for legitimate security research rather than blanket restrictions, suggesting policy barriers may decrease over time. 4. Capability growth: Both OpenAI (with GPT-5.3-Codex rated "High") and Anthropic (with Mythos Preview's unprecedented capabilities) indicate rapid capability growth in cybersecurity, with labs acknowledging their models pose increasing offensive cyber risks.