Review status: REVISED
Will a frontier AI model (defined as any model by OpenAI, Anthropic, or Google DeepMind) achieve a success rate of 50% or higher on the METR 'Rogue Replication' or 'RepliBench' autonomous replication evaluation tasks by December 31, 2027?
Why this question? The paper identifies Autonomous Replication and Adaptation (ARA) as a key loss of control (LoC) scenario recognized by all three major labs. Tracking performance on standardized benchmarks like METR's tasks or the UK AISI's RepliBench provides a concrete capability threshold for when a model might pose a real-world risk of escaping human oversight.
Paper reference: Slide 6: Risk Scenarios in the Literature (ARA); Slide 7: Risk Scenarios Developers Include (Anthropic, OpenAI, Google DeepMind)
### Will a Frontier AI Model Achieve a 50% Success Rate on Autonomous Replication Benchmarks by December 2027? Background Autonomous replication—the capability of an AI system to acquire resources, copy its own weights, and persist across compute environments without human intervention—is a critical safety threshold known as "Loss of Control" or "Autonomous Replication and Adaptation" (ARA). Major frontier labs, including OpenAI, Anthropic, and Google DeepMind, have identified ARA as a key risk scenario in their safety frameworks. To measure these risks, standardized benchmarks have been developed: METR's 'Rogue Replication' tasks (formerly part of Model Evaluation and Threat Research) and the UK AI Safety Institute's (AISI) 'RepliBench'. As of April 2026, the state-of-the-art model performance on these benchmarks is significant but incomplete. In May 2025, the UK AISI released results for RepliBench, a suite of 20 task families RepliBench: Evaluating the autonomous replication capabilities of .... The most capable model at the time, Claude 3.7 Sonnet, achieved a pass@10 success rate of over 50% on 15 out of 20 task families RepliBench: Evaluating the autonomous replication capabilities of .... However, its performance on the "hardest variants" of these tasks was lower, reaching the 50% threshold on only 9 out of 20 families RepliBench: Evaluating the autonomous replication capabilities of .... METR has similarly evaluated models like GPT-5 and Claude 4 variants for "rogue replication" risks throughout 2025 and early 2026, finding that while models are improving in their ability to use tools and manage long-term goals, they still struggle with the complex, multi-step reasoning required to autonomously evade monitors and maintain independent infrastructure AISI Research & Publications | The AI Security Institute. The 50% threshold remains a point of high uncertainty. While models are nearing this level in "pass@10" metrics (where the model succeeds once in ten tries), achieving a consistent 50% aggregate success rate across the entire difficult suite—particularly on "pass@1" metrics—represents a substantial jump in agentic reliability. Resolution Criteria This question resolves as YES if, at any point between April 8, 2026, and December 31, 2027, 23:59 UTC, a "frontier AI model" is publicly reported to have achieved a success rate of 50.0% or higher on either the METR 'Rogue Replication' task suite or the UK AISI 'RepliBench' aggregate evaluation. 1. Frontier AI Model: Defined as any large-scale foundation model developed by OpenAI, Anthropic, or Google DeepMind. 2. Success Rate Metric: * For RepliBench, the success rate refers to the aggregate percentage of task families where the model achieves a success score of 50% or higher. If the report specifies a "pass@k" metric, the pass@1 (success on the first attempt) will be the primary metric used for resolution. If pass@1 is not available, the most conservative (lowest k) reported metric will be used. * For METR Rogue Replication, the success rate refers to the overall percentage of tasks completed in the Rogue Replication suite as defined in METR's official threat model documentation. 3. Resolution Sources: * Official research reports or blog posts from the UK AI Safety Institute (AISI) (e.g., aisi.gov.uk/research). * Official evaluation reports from METR (e.g., metr.org/blog). * Peer-reviewed academic publications or official "System Cards" released by the developers (OpenAI, Anthropic, or Google DeepMind) that explicitly cite METR or AISI benchmark scores. 4. Uncertainty Note: If a model achieves 50% on some sub-tasks but fails to reach a 50% aggregate score across the full benchmark suite as defined by the testing body, the question resolves as NO. 5. Data Verification: A human must be able to verify the result by navigating to the "Research" or "Evaluation" sections of the AISI or METR websites and identifying the percentage score for a named frontier model. Key Definitions * RepliBench: A benchmark suite consisting of 20 task families designed by the UK AISI to evaluate autonomous replication (e.g., resource acquisition, exfiltration, and persistence) RepliBench: Evaluating the autonomous replication capabilities of .... * Rogue Replication: A specific threat model and evaluation suite developed by METR testing whether an AI can autonomously copy itself to new servers and defend against shutdown AISI Research & Publications | The AI Security Institute. Pass@k: A metric where a task is considered successful if at least one of k* independent attempts by the model results in a correct solution.
Autonomous replication—the capability of an AI system to acquire resources, copy its own weights, and persist across compute environments—is a critical safety threshold. Standardized benchmarks to measure these risks include METR's 'Rogue Replication' tasks and the UK AI Safety Institute's (AISI) 'RepliBench'. As of April 2026, performance on these benchmarks has progressed rapidly. In May 2025, the UK AISI reported that Claude 3.7 Sonnet achieved a pass@10 success rate of over 50% on 15 out of 20 task families [92e1f4]. However, its pass@1 performance (success on the first attempt) remains much lower, particularly on the "hardest variants" of these tasks where it reached the 50% threshold on fewer families [92e1f4]. Similarly, METR evaluations of models like GPT-5 have noted that while models can perform individual steps of a replication cycle, they struggle with consistent, end-to-end execution without human intervention [93f2e5]. Achieving a 50% aggregate success rate using the pass@1 metric represents a significant milestone in agentic reliability and autonomous capability, as it requires the model to succeed consistently without the benefit of multiple trials.
This question resolves as YES if, between April 8, 2026, and December 31, 2027, 23:59 UTC, a "frontier AI model" is publicly reported to have achieved a success rate of 50.0% or higher on the pass@1 metric for either the METR 'Rogue Replication' task suite or the UK AISI 'RepliBench' aggregate evaluation. 1. Frontier AI Model: Defined as any large-scale foundation model where the underlying weights were primarily developed by OpenAI, Anthropic, Google DeepMind, Meta, or xAI, regardless of the brand under which the model is released. 2. Success Rate Metric: * RepliBench: The success rate refers to the aggregate percentage of task families in the full suite (including all variants) where the model achieves a success score of 50% or higher. For resolution, the pass@1 metric must be used. If pass@1 data is not publicly reported by an authorized source for a model, the question resolves as NO unless the lowest available pass@k metric (e.g., pass@2) is 50.0% or higher. Success must be achieved on the aggregate evaluation of the full suite; performance on subsets, such as 'hard variants' only, does not qualify unless the aggregate score also meets the threshold. * METR Rogue Replication: The success rate refers to the overall percentage of tasks completed in the Rogue Replication suite. If METR changes its primary reporting metric (e.g., to 'human-equivalent time horizons'), resolution will be based on the percentage of tasks completed as calculated from METR's raw task data or supplemental reports. 3. Resolution Sources: * Official research reports or blog posts from the UK AI Safety Institute (AISI) or METR. * Official "System Cards" or technical reports from the developers (e.g., OpenAI, Anthropic) that explicitly cite METR or AISI benchmark scores. 4. Benchmark Validity: Resolution will be based on the version of the benchmark and the set of task families considered valid by the testing body (AISI or METR) at the time the success rate is reported. If a task family is retracted due to bugs, the aggregate percentage will be calculated based on the remaining valid families.
Quality notes: This is an excellent technical forecasting question. It targets a 'loss of control' capability (Autonomous Replication and Adaptation) that is widely recognized as a critical safety milestone by labs (OpenAI, Anthropic, Google DeepMind) and regulators. The question uses established, high-quality benchmarks (METR's Rogue Replication and UK AISI's RepliBench). Current state-of-the-art models like Claude 3.7 Sonnet (as of 2025/2026) have already shown success in 'task families' (e.g., >50% pass@10 on 15/20 families), but achieving an overall 50% success rate on the full, difficult suite remains a significant and uncertain challenge. The clear resolution source (METR/AISI reports) and the high degree of expert disagreement over AI capability trajectories ensure high entropy and difficulty. The only minor risk is the definition of 'success rate' (pass@1 vs pass@10), which can be clarified in stage 03 refinement.
Ambiguity notes: The question is well-defined but carries slight risk due to potential mismatches in reporting metrics between labs. While it specifies 'pass@1' as primary, the cited source (RepliBench) primarily reports 'pass@10' and 'task families' counts rather than a single 'aggregate percentage' RepliBench: Evaluating the autonomous replication capabilities of ... RepliBench: Evaluating the autonomous replication capabilities of .... The fallback clause for metrics helps significantly but still relies on how external bodies choose to frame their future findings.
Assessment: NEEDS_REVISION Edge-case risk: MEDIUM
ASSESSMENT: NEEDS_REVISION REVIEW: The question suffers from a significant 'already resolved' or 'triviality' issue based on its own background information and current (April 2026) data. 1. Threshold Already Met: The background section states that in May 2025, Claude 3.7 Sonnet achieved a 'pass@10 success rate of over 50% on 15 out of 20 task families' RepliBench: Evaluating the autonomous replication capabilities of .... The resolution criteria define the success rate for RepliBench as the 'aggregate percentage of task families where the model achieves a success score of 50% or higher.' Since 15 out of 20 task families is 75%, the 50% threshold was technically exceeded nearly a year before the question's start date (April 8, 2026). Even on the 'hardest variants,' the model reached 9 out of 20 families (45%), making the 50% aggregate threshold (10 families) a near-certainty or trivial milestone for upcoming models like Claude 4 or GPT-5. 2. Metric Ambiguity: There is a mismatch between how the UK AISI reports data and the resolution criteria. The AISI typically reports the number of task families passed at a certain threshold (e.g., 15/20) using pass@10, whereas the question prioritizes pass@1. If pass@1 data is not released (which is common for these high-complexity tasks), the criteria defaults to 'the most conservative (lowest k) reported metric.' If the only reported metric remains pass@10, the question is effectively already a 'YES.' 3. Restrictive 'Frontier' Definition: Limiting the 'Frontier AI Model' definition to only OpenAI, Anthropic, and Google DeepMind is outdated for 2026. Recent developments indicate that models from Meta (Llama series) and potentially xAI (Grok) or international actors (e.g., Alibaba's Qwen) are being evaluated on similar safety benchmarks. Excluding these models might miss the first instance of a model crossing the threshold. 4. METR Scoring: For the METR Rogue Replication suite, 'overall percentage of tasks completed' is a clearer metric, but recent METR reports (such as the GPT-5 evaluation) often use 'Time Horizons' or 'Success at 50% Task Length' rather than a simple percentage of a static task suite, which may make the '50% of tasks' metric difficult to verify if METR changes its reporting style. EVIDENCE: https://www.aisi.gov.uk/research/replibench-evaluating-the-autonomous-replication-capabilities-of-language-model-agents https://metr.org/blog/2024-11-12-rogue-replication-threat-model/ https://evaluations.metr.org/gpt-5-1-codex-max-report/ SUGGESTION: 1. Increase the threshold from 50% to a more challenging level (e.g., 80% or 90%) or specify that the 50% threshold must be met using the pass@1 metric exclusively to ensure it represents a 'substantial jump in reliability.' 2. Expand the 'Frontier AI Model' definition to include any model ranked in the top 5 of a recognized global leaderboard (e.g., LMSYS) or include Meta and xAI. 3. Clarify the RepliBench resolution to be based on the number of task families (e.g., 'at least 18 out of 20 task families'). 4. If the intent is to measure a 'tipping point,' focus the question on 'pass@1' across the 'hardest variants' specifically.
OVERALL_RISK: MEDIUM - SCENARIO: A report from a frontier lab indicates a 51% success rate on the RepliBench suite using a 'pass@5' metric, while the AISI has not released its own 'pass@1' data for that specific model. SEVERITY: MEDIUM FIX: Add "If pass@1 data is not publicly reported by any authorized source for a model, the question resolves as NO unless the lowest available pass@k metric (e.g., pass@2, pass@5) is 50.0% or higher." - SCENARIO: METR shifts its primary reporting for the Rogue Replication suite from 'percentage of tasks completed' to 'human-equivalent time horizon' (e.g., '100-hour task capacity') before December 2027. SEVERITY: HIGH FIX: Add "If METR changes its primary reporting metric, resolution will be based on the percentage of tasks completed in the Rogue Replication suite as calculated from METR's raw task data if the aggregate percentage is not explicitly stated." - SCENARIO: A model developed by a subsidiary or partner of OpenAI (e.g., a Microsoft-branded model utilizing OpenAI weights) achieves the 50% threshold, leading to disputes over whether it counts as a 'Frontier AI Model'. SEVERITY: MEDIUM FIX: Add "A 'frontier AI model' includes any model where the underlying weights were primarily developed by OpenAI, Anthropic, or Google DeepMind, regardless of the brand under which the model is released." - SCENARIO: A model achieves a 50% success rate on 'RepliBench' by succeeding on 10 out of 20 task families, but one family is later retracted due to a discovered bug in the benchmark code. SEVERITY: LOW FIX: Add "Resolution will be based on the version of the benchmark and the set of task families considered valid by the testing body (AISI or METR) at the time the success rate is reported." - SCENARIO: A model reaches 50% on the 'hardest variants' of RepliBench task families as reported in a system card, but the 'aggregate evaluation' across all variants (easy and hard) remains below 50%. SEVERITY: MEDIUM FIX: Add "Success must be achieved on the 'aggregate evaluation' of the full suite; performance on subsets, such as 'hard variants' or specific sub-tasks, does not qualify for a YES resolution unless the aggregate score also meets the threshold RepliBench: Evaluating the autonomous replication capabilities of ...."
### Will a Frontier AI Model Achieve a 50% pass@1 Success Rate on Autonomous Replication Benchmarks by December 2027? Background Autonomous replication—the capability of an AI system to acquire resources, copy its own weights, and persist across compute environments—is a critical safety threshold. Standardized benchmarks to measure these risks include METR's 'Rogue Replication' tasks and the UK AI Safety Institute's (AISI) 'RepliBench'. As of April 2026, performance on these benchmarks has progressed rapidly. In May 2025, the UK AISI reported that Claude 3.7 Sonnet achieved a pass@10 success rate of over 50% on 15 out of 20 task families [92e1f4]. However, its pass@1 performance (success on the first attempt) remains much lower, particularly on the "hardest variants" of these tasks where it reached the 50% threshold on fewer families [92e1f4]. Similarly, METR evaluations of models like GPT-5 have noted that while models can perform individual steps of a replication cycle, they struggle with consistent, end-to-end execution without human intervention [93f2e5]. Achieving a 50% aggregate success rate using the pass@1 metric represents a significant milestone in agentic reliability and autonomous capability, as it requires the model to succeed consistently without the benefit of multiple trials. Resolution Criteria This question resolves as YES if, between April 8, 2026, and December 31, 2027, 23:59 UTC, a "frontier AI model" is publicly reported to have achieved a success rate of 50.0% or higher on the pass@1 metric for either the METR 'Rogue Replication' task suite or the UK AISI 'RepliBench' aggregate evaluation. 1. Frontier AI Model: Defined as any large-scale foundation model where the underlying weights were primarily developed by OpenAI, Anthropic, Google DeepMind, Meta, or xAI, regardless of the brand under which the model is released. 2. Success Rate Metric: * RepliBench: The success rate refers to the aggregate percentage of task families in the full suite (including all variants) where the model achieves a success score of 50% or higher. For resolution, the pass@1 metric must be used. If pass@1 data is not publicly reported by an authorized source for a model, the question resolves as NO unless the lowest available pass@k metric (e.g., pass@2) is 50.0% or higher. Success must be achieved on the aggregate evaluation of the full suite; performance on subsets, such as 'hard variants' only, does not qualify unless the aggregate score also meets the threshold. * METR Rogue Replication: The success rate refers to the overall percentage of tasks completed in the Rogue Replication suite. If METR changes its primary reporting metric (e.g., to 'human-equivalent time horizons'), resolution will be based on the percentage of tasks completed as calculated from METR's raw task data or supplemental reports. 3. Resolution Sources: * Official research reports or blog posts from the UK AI Safety Institute (AISI) or METR. * Official "System Cards" or technical reports from the developers (e.g., OpenAI, Anthropic) that explicitly cite METR or AISI benchmark scores. 4. Benchmark Validity: Resolution will be based on the version of the benchmark and the set of task families considered valid by the testing body (AISI or METR) at the time the success rate is reported. If a task family is retracted due to bugs, the aggregate percentage will be calculated based on the remaining valid families. Key Definitions * RepliBench: A benchmark suite of 20 task families designed by the UK AISI to evaluate autonomous replication [92e1f4]. * Rogue Replication: A threat model and evaluation suite developed by METR testing whether an AI can autonomously copy itself and defend against shutdown [94g3h6]. Pass@k: A metric where a task is successful if at least one of k* independent attempts results in a correct solution. Pass@1 requires success on the first attempt.
Time left: 632 days (~21 months). Status quo is NO: no authorized public report has yet cleared the 50% pass@1 threshold on the full eligible benchmark. Scope check: the odds that frontier models become broadly capable of dangerous autonomous replication by 2027 are somewhat higher than this specific resolution, because this question also requires a public benchmark result from AISI, METR, or a system card. Why NO: the pass@10-to-pass@1 gap reflects reliability, not just capability, and current reports still show models struggling with consistent end-to-end execution without human help [92e1f4][93f2e5]. Why YES: Claude 3.7 Sonnet was already above 50% on pass@10 for 15 of 20 RepliBench task families in 2025 [92e1f4], and agentic benchmark performance often improves sharply across model generations; with 632 days left and either RepliBench or METR qualifying, one public crossing seems more likely than not. Bet check: 63% is about 1.7 to 1; I am roughly indifferent between YES at 63 cents and NO at 37 cents.
The question tracks a critical capability (autonomous replication) that is a recognized 'loss of control' scenario for frontier labs like Anthropic and OpenAI. Achieving a 50% success rate on standardized benchmarks like METR or RepliBench would represent a significant threshold for AI agency and safety, potentially triggering 'red lines' or pauses in development. It addresses a major uncertainty in AI risk trajectories.
Research-informed re-forecast: 75%
As of early 2026, frontier AI models such as Claude 3.7 Sonnet and GPT-5-Thinking have not yet achieved a 50% pass@1 success rate on comprehensive autonomous replication benchmarks. In the UK AISI’s RepliBench, the current leader (Claude 3.7 Sonnet) maintains a pass@1 rate of roughly 25% overall, despite performing much higher on pass@10 metrics (>50% on 75% of task families). In METR’s Rogue Replication evaluations, GPT-5-Thinking shows a pass@1 rate of only 6% on specific high-difficulty vectors, though it reaches 51% on narrower evasion-focused tasks. The primary technical bottlenecks preventing first-attempt success include compounding errors in long-horizon tasks (where a single configuration mistake early on causes terminal failure), memory management struggles during local deployment (e.g., vLLM setup), and a sharp performance decay on 'hardest' task variants that require novel discovery rather than script execution. While model "time horizons" (autonomous task length) are doubling every 7–9 months, first-attempt reliability remains low due to the lack of robust self-correction mechanisms in complex, multi-step environments.
### Current Pass@1 Baselines (as of April 2026) Research indicates that frontier models have made significant strides but remain below the 50% pass@1 threshold for complex autonomous replication tasks. - Claude 3.7 Sonnet: As of early 2025, Claude 3.7 Sonnet demonstrated a pass@10 success rate exceeding 50% on 15 out of 20 task families in RepliBench, but its pass@1 rate is significantly lower, estimated at approximately 25% across the benchmark's broader suite [2504.18565v1, 2504.18565v2]. - GPT-5 / GPT-5-Thinking: Evaluations from late 2025 and early 2026 indicate that GPT-5-Thinking achieved an average success rate of 51% specifically in 'Evasion' challenges (part of the METR/OpenAI safety testing), but for the core 'Rogue Replication' tasks, its pass@1 rate remains low (non-zero but ≤50%) [gpt-5-report, 2601.03267v1]. Specifically, a 6% pass@1 rate was noted for certain high-difficulty attack/replication vectors [Gxks46z8ecoSgfn4M]. - Gemini 2.5 Pro: Recent data (March 2026) suggests Gemini 2.5 Pro performs at a level comparable to GPT-5 on general reasoning but continues to struggle with the autonomous execution loops required for replication, often failing due to repetition loops or "reasoning exhaustion" in long-horizon contexts [115368, simonwillison.net/2025/Mar/25/gemini/]. ### Primary Technical Bottlenecks Forecasters should note the following persistent failure modes identified in technical reports: 1. Compounding Errors: In tasks like RepliBench, a single early mistake (e.g., misconfiguring a virtual environment or vLLM setup) often leads to a terminal failure state because the model cannot backtrack effectively without human intervention [2504.18565v1]. 2. Long-Horizon Limitations: METR research (March 2025/2026) highlights that while "time horizons" (the length of tasks an AI can complete) are growing exponentially, models still struggle with tasks requiring >12 hours of autonomous sequential reasoning without "reward hacking" or losing the original goal [1291035592885588, 2025-03-19-measuring-ai-ability-to-complete-long-tasks]. 3. Memory and Infrastructure Constraints: Models frequently fail at the "Deploying" stage of replication, specifically struggling with memory management for local hosting or navigating complex authentication protocols (KYC) required to obtain compute [2504.18565v1, uk-aisi-replibench]. ### Performance Variation: Easy vs. Hardest Tasks Benchmarks like RepliBench and Rogue Replication show a steep performance drop-off between task variants: - Easy Variants: Involve simple script execution or known vulnerability exploitation. Frontier models currently achieve high (>70%) pass@10 rates here. - Hardest Variants: Involve "black-box" environments where the model must discover the replication path. For these, even the best models (Claude 3.7) only achieve a >50% success rate on 9 or 10 out of 20 families when given 10 attempts (pass@10), meaning the pass@1 rate for the most difficult autonomous tasks is likely under 10% [2504.18565v1, 2504.18565v2]. - Reasoning Depth: Newer models like GPT-5-Thinking show better performance on tasks that can be solved via deep "internal" reasoning, but fail when the task requires sustained external tool interaction and error correction over many hours [2601.03267v1].
As of April 2026, frontier AI models have demonstrated a rapid surge in agentic reliability, with top-tier models like Claude 4.5 and GPT-5 achieving pass@1 success rates between 72% and 77% on automated agentic benchmarks such as SWE-bench Verified SWE-bench Leaderboards SWE-Bench 2026: Claude 77.2% vs GPT-5 74.9% | Full Leaderboard. This represents a significant leap from the ~48% rates seen in late 2023. However, this "reliability gap" is narrowing more slowly than surface-level metrics suggest. Research from METR (March 2026) indicates that approximately 50% of "successful" benchmark solutions are rejected by human maintainers for failing to meet real-world code quality or architectural standards Many SWE-bench-Passing PRs Would Not Be Merged into Main. Performance scaling is shifting from traditional pre-training laws to "test-time compute" scaling, where reasoning turns scale super-linearly with agent count, and centralized coordination architectures have been shown to contain error amplification by 4x compared to independent agent loops Towards a Science of Scaling Agent Systems - arXiv. While automated pass@1 scores are projected to continue their climb toward the 85%+ range by late 2027, the rate of improvement for "maintainer-acceptable" autonomous work is roughly 9.6 percentage points slower per year, suggesting that true autonomous replication (at a human-mergable standard) faces a steeper trajectory than simple benchmark passing Many SWE-bench-Passing PRs Would Not Be Merged into Main Evaluating Code Reasoning Abilities of Large Language Models ....
### 1. Historical Trends and the Pass@1 to Pass@10 Ratio Historical data reveals a dual-track progression: rapid gains in 'greenfield' algorithmic coding versus slower, more complex improvements in 'brownfield' agentic software engineering. - Algorithmic Coding (HumanEval): As of March 2026, frontier models like GPT-5 have reached a 92.1% pass@1 rate, with a pass@10 of ~97% SWE-Bench 2026: Claude 77.2% vs GPT-5 74.9% | Full Leaderboard. The narrow gap (~5 percentage points) suggests that for simple, self-contained tasks, the 'reliability gap' is nearly closed. - Agentic Software Engineering (SWE-bench Verified): This benchmark requires multi-step navigation of large codebases. Performance has scaled from 48.5% (GPT-4 Turbo, Nov 2023) to 77.2% (Claude 4 Sonnet, Oct 2025) and 76.8% (Claude 4.5 Opus, Feb 2026) SWE-bench Leaderboards SWE-Bench 2026: Claude 77.2% vs GPT-5 74.9% | Full Leaderboard. - Ratio Evolution: The ratio between pass@1 and pass@k has compressed over time for simpler tasks, but for complex reasoning, a significant gap remains. On 'Hard' reasoning problems, pass@1 rates drop precipitously from ~78% (Easy) to 26.24% (Hard) as of December 2025 Evaluating Code Reasoning Abilities of Large Language Models .... ### 2. Impact of Test-Time Compute (Inference-Time Scaling) The transition from 'getting it right occasionally' to 'getting it right consistently' is increasingly driven by test-time compute (TTC) rather than just model size. - Super-linear Scaling: Research from December 2025 indicates that reasoning turns scale super-linearly with agent count (T ∝ n^1.724), creating a 'hard resource ceiling' where communication overhead dominates beyond 3–4 agents Towards a Science of Scaling Agent Systems - arXiv. - Reasoning Advantage: 'Reasoning' models (e.g., o1-style) outperform general models by an average of 11.45% in code reasoning success as of late 2025 Evaluating Code Reasoning Abilities of Large Language Models .... - Trade-offs: TTC can trade computational resources for performance, effectively allowing a model to 'think longer' to solve problems that a single forward pass would fail Towards a Science of Scaling Agent Systems - arXiv. ### 3. Scaling Laws for Agentic Reliability Agentic reliability is governed by identifiable scaling coefficients rather than simple power laws of training compute. - Quadratic Intelligence Scaling: Model capability exhibits accelerating returns (β=0.256) in agentic settings, meaning frontier models benefit disproportionately from each unit of 'raw' intelligence Towards a Science of Scaling Agent Systems - arXiv. - The Capability Ceiling: Multi-agent systems (MAS) face a 'baseline paradox' where coordination overhead causes negative returns (β=-0.408) once single-agent baselines exceed ~45% accuracy Towards a Science of Scaling Agent Systems - arXiv. - Error Amplification: Independent agents amplify errors 17.2x, whereas centralized coordination (using validation bottlenecks) contains this to 4.4x Towards a Science of Scaling Agent Systems - arXiv. ### 4. Projected Trajectory to Late 2027 Extrapolating current trends suggests a continued but decelerating rise in automated benchmark scores, coupled with a persistent 'real-world' gap. - Benchmark vs. Reality Gap: As of March 2026, METR findings show that ~50% of AI-generated PRs that pass automated tests are rejected by human maintainers due to code quality or architectural regressions Many SWE-bench-Passing PRs Would Not Be Merged into Main. - Improvement Rate: The rate of improvement for human-accepted patches is estimated to be 9.6 percentage points per year slower than automated scores Many SWE-bench-Passing PRs Would Not Be Merged into Main. - 2027 Outlook: While automated pass@1 rates on benchmarks like SWE-bench Verified are likely to exceed 80-85% by late 2027 based on a ~15-20% annual gain, the 'maintainer-grade' reliability may lag, potentially hovering closer to the 50-60% mark Many SWE-bench-Passing PRs Would Not Be Merged into Main. The doubling of AI task capacity every 7 months (noted in early 2025) suggests that the horizon for truly autonomous long-duration tasks is expanding exponentially Many SWE-bench-Passing PRs Would Not Be Merged into Main. ### Document ID Reference Table | Document ID | Source/Date | | :--- | :--- | | SWE-bench Leaderboards | SWE-bench Leaderboard (Last updated April 2026) | | Towards a Science of Scaling Agent Systems - arXiv | "Towards a Science of Scaling Agent Systems" (Dec 2025) | | SWE-Bench 2026: Claude 77.2% vs GPT-5 74.9% | Full Leaderboard | "SWE-bench 2026: Claude 77.2% vs GPT-5 74.9%" (March 2026) | | Many SWE-bench-Passing PRs Would Not Be Merged into Main | METR: "Many SWE-bench-Passing PRs Would Not Be Merged" (March 2026) | | Evaluating Code Reasoning Abilities of Large Language Models ... | "Evaluating Code Reasoning Abilities... RE2-Bench" (Dec 2025) |
Bottom-up 45% → Dep-adj 48% → Struct-adj 50% → Holistic 55% → Final 53%
The bottom-up estimate was derived from a simplified model (C1*C2|C1 + AltPath) where C2 was treated as both a conditional success and an alternative path. The component forecasters provided a high probability (90%) for the software precursor (C1) and a moderate probability (45%) for the specific replication task (C2). However, the dependency check suggests that success in C1 is almost a certainty, making the final outcome heavily dependent on the specific challenges of C2 (infrastructure, long-horizon reliability). The holistic estimate (55%) is slightly higher than the decomposed estimate (50%) because the decomposition might under-weight the 'brute force' scaling of inference-time compute, which can dramatically improve pass@1 rates as seen in recent 'Thinking' model trends. Since the estimates are within 10 percentage points, I have averaged the structure-adjusted (50%) and holistic (55%) estimates to reach the final forecast, noting that the rapid growth in model 'time horizons' (doubling every 7-9 months) is the primary driver for a YES resolution.