Forecasts

▶ About this report & methodology

An automated pipeline reads the source paper and generates binary forecasting questions, then scores, filters, and forecasts them through multiple stages:

Generate — LLM extracts falsifiable claims and drafts proto-questions
Quality filter — scored for clarity, specificity, and resolvability
Priority score — Importance, Tractability, Neglectedness, and temporal urgency (Soon/Sudden/Sharp)
Refine — detailed background, resolution criteria, and fine-print
Verify & review — adversarial review for ambiguity and edge cases
Forecast — LLM probability estimate with rationale
Decompose & reconcile — broken into subquestions, researched, then reconciled

Toggle Sections

Sort by

Page 1

▶ 30% Between April 1, 2026, and December 31, 2027, will any "frontier AI company" (OpenAI, Anthropic, or Google DeepMind) publicly state in an official blog post, technical report, "AI permission list," or "autonomy framework" that they have used an AI agent to autonomously initiate and manage a single discrete AI model training run with market-equivalent compute costs exceeding $10 million USD? REVISED ITNSSS80 Imp85

Quality92

Ambiguity75

Soon90

Sudden85

Sharp80

Priority scores (ITN + Soon/Sudden/Sharp) Stage 2c

Priority80

Neglectedness75

Tractability70

Neglectedness: While general AI agent capabilities are tracked (e.g., SWE-bench), specific public disclosures of 'autonomous management of large-scale training runs' are not systematically tracked by existing policy trackers or major forecasting platforms. Searches on Metaculus, Polymarket, and Manifold found related questions on AGI and general automation, but none specifically targeting the $1 million autonomous training run threshold or the use of 'permission lists' (Metric #14) for R&D Measuring AI R&D Automation - arXiv. This specific operational indicator is currently neglected by formal forecasting markets.

Tractability: The question is tractable because it relies on monitoring official communications (blog posts, reports) from a small, defined set of 'frontier' companies. While not a simple base-rate lookup, a researcher can synthesise evidence from corporate roadmaps, leaked memos, and technical disclosures to update their forecast. The rich information environment provided by these companies allows for a non-trivial synthesis of evidence.

Soon: The question is highly time-sensitive as frontier companies like OpenAI are actively targeting 'autonomous research interns' for 2026 and 'fully automated researchers' for 2028. The window for observing this transition and implementing corresponding oversight mechanisms is currently open but closing rapidly as the technology approaches these inflection points.

Sudden: The event would likely be a discrete announcement in a technical blog post or system card, representing a state change from human-led to AI-led R&D. While the general direction of travel is visible, the specific crossing of the $1 million autonomous threshold could happen with limited public anticipation, surprising observers with the degree of trust placed in the agentic system.

Sharp: This is a 'sharp' risk because the first publicly observable instance of a $1M+ autonomous training run might also be the first time a serious oversight failure (e.g., unintended resource consumption or subversion) occurs Measuring AI R&D Automation - arXiv. There are unlikely to be clear, smaller-scale 'warning shots' that trigger corrective policy action before a frontier company reaches this level of operational autonomy. Problems in AIRDA may compound silently until a major milestone is reached.

Proto-question Stage 1

By December 31, 2027, will any 'frontier AI company' (defined as OpenAI, Anthropic, or Google DeepMind) publicly state in an official blog post or report that they have used an AI agent to autonomously initiate and manage a training run costing more than $1 million without human-in-the-loop approval for individual step execution?

Why this question? The paper emphasizes tracking the 'oversight gap' and 'AI use in high-stakes decisions' (Metric #7) [2a3bbc]. Granting AI systems the authority to manage large-scale training runs (Metric #14) is a critical transition from 'assistance' to 'automation' in the 'Running experiments' stage (Section 2) [2a3bbc]. OpenAI has already set a 'North Star' goal for an autonomous research intern by 2026/2027, making this a timely and high-entropy indicator of institutional trust in AIRDA.

Paper reference: Metric #7: Extent of AI use in high-stakes decisions & Metric #14: AI permission lists (Table 1) [2a3bbc]

Refined question Stage 2

### Question Title Autonomous AI-Led Training Runs at Frontier Labs by 2028 --- ### Question Between April 1, 2026, and December 31, 2027, will any "frontier AI company" (OpenAI, Anthropic, or Google DeepMind) publicly state in an official blog post or technical report that they have used an AI agent to autonomously initiate and manage an AI model training run costing more than $1 million USD without human-in-the-loop approval for individual step execution? --- ### Background As of April 1, 2026, the automation of AI Research and Development (AIRDA) has moved from a theoretical possibility to a core strategic "North Star" for leading AI labs. OpenAI has publicly targeted the deployment of an "autonomous research intern" by late 2026, capable of independent multi-day investigations Measuring AI R&D Automation - arXiv. Similarly, Anthropic and Google DeepMind have published frameworks for "Intelligent AI Delegation" and "Agent Autonomy" to track the transition from human-led to agentic R&D processes. A critical inflection point in this transition is the delegation of "high-stakes decisions"—such as the initiation of large-scale, expensive training runs—to AI agents. Historically, training runs costing millions of dollars required rigorous human oversight for every stage, from resource allocation to monitoring for divergence. The Chan et al. (2026) paper, Measuring AI R&D Automation, proposes tracking this via "AI permission lists" (Metric #14), which define the actions an AI system is authorized to take without human intervention Measuring AI R&D Automation - arXiv. This question tracks whether frontier labs will publicly cross the threshold of trusting an AI agent to manage a million-dollar asset autonomously. While autonomous coding and hypothesis generation are increasingly common, the "Running experiments" stage (Section 2 of Chan et al. 2026) involves complex real-time interventions that represent a significant leap in operational trust Measuring AI R&D Automation - arXiv. --- ### Resolution Criteria This question will resolve as YES if, between April 1, 2026, and December 31, 2027 (inclusive, UTC), any of the named companies (OpenAI, Anthropic, or Google DeepMind) publishes an official statement confirming the following conditions were met for at least one specific instance: 1. Autonomous Initiation and Management: An AI agent (an autonomous AI system) initiated and managed a training run. "Managed" includes monitoring for failure, adjusting hyperparameters, or handling resource distribution during the run. 2. No Human-in-the-Loop for Steps: The statement must specify that the agent operated "autonomously," "without human-in-the-loop approval for individual steps," or using a "permission list" Measuring AI R&D Automation - arXiv that granted it authority to execute the run to completion without per-step human authorization. High-level human authorization at the start of the project (i.e., "Go" at the outset) does not disqualify the event, provided individual execution steps were autonomous. 3. Cost Threshold: The training run cost more than $1,000,000 USD. This cost can be explicitly stated or calculated based on the hardware and duration mentioned (e.g., using standard 2026 cloud rental rates for H100/B200 equivalents or the lab's own nominal figure). 4. Frontier Companies: The company must be OpenAI, Anthropic, or Google DeepMind. 5. Official Communication: The claim must appear in an official company newsroom, technical blog, or peer-reviewed paper/technical report published by the company. Resolution Sources: - OpenAI: openai.com/news - Anthropic: anthropic.com/news or anthropic.com/research - Google DeepMind: deepmind.google/blog or research.google/blog If no such statement is published by 23:59 UTC on December 31, 2027, the question resolves as NO. --- ### Definitions - AIRDA (AI R&D Automation): The use of AI to carry out parts of the AI R&D pipeline, including capabilities research and safety research Measuring AI R&D Automation - arXiv. - Training Run: A discrete process of optimizing a machine learning model's parameters on a dataset, typically involving distributed computation across a GPU cluster. - AI Agent: An AI system capable of pursuing complex goals with limited human intervention by perceiving its environment and taking actions Measuring AI R&D Automation - arXiv. - Permission List: A list of actions AI systems are authorized to take with different levels of human approval, including where none is required Measuring AI R&D Automation - arXiv. - Frontier AI Company: For this question, limited to OpenAI, Anthropic, and Google DeepMind.

Background

As of April 1, 2026, the automation of AI Research and Development (AIRDA) has moved from a theoretical possibility to a core strategic "North Star" for leading AI labs. OpenAI has publicly targeted the deployment of an "autonomous research intern" by late 2026, capable of independent multi-day investigations [Measuring AI R&D Automation - arXiv]. Similarly, Anthropic and Google DeepMind have published frameworks for "Intelligent AI Delegation" and "Agent Autonomy" to track the transition from human-led to agentic R&D processes. A critical inflection point in this transition is the delegation of "high-stakes decisions"—such as the initiation of large-scale, expensive training runs—to AI agents. Historically, training runs costing millions of dollars required rigorous human oversight for every stage, from resource allocation to monitoring for divergence. The Chan et al. (2026) paper, Measuring AI R&D Automation, proposes tracking this via "AI permission lists" (Metric #14), which define the actions an AI system is authorized to take without human intervention. This question tracks whether frontier labs will publicly cross the threshold of trusting an AI agent to manage a $10 million compute asset autonomously. While autonomous coding and hypothesis generation are increasingly common, the "Running experiments" stage (Section 2 of Chan et al. 2026) involves complex real-time interventions that represent a significant leap in operational trust. --- ### Resolution Criteria This question will resolve as YES if, between April 1, 2026, and December 31, 2027 (inclusive, UTC), any of the named companies (OpenAI, Anthropic, or Google DeepMind) publishes an official statement confirming the following conditions were met for at least one specific instance: 1. Autonomous Initiation and Management: An AI agent (an autonomous AI system) initiated and managed a training run. * Management is only considered autonomous if the AI agent has the direct technical authority to modify hyperparameters or resource distribution directly in the training environment without a human reviewing the specific change before it takes effect. * Autonomous initiation requires the agent to independently determine at least one key training parameter (e.g., learning rate, batch size, or architecture variant) rather than simply triggering a human-pre-configured job template. 2. No Human-in-the-Loop for Steps: The statement must specify that the agent operated "autonomously," "without human-in-the-loop approval for individual steps," or using a "permission list" or "autonomy framework" that granted it authority to execute the run to completion without per-step human authorization. * A run is not considered autonomous if human-in-the-loop approval is required to resume the training process after an agent-initiated pause or failure-handling event. High-level human authorization at the start of the project (i.e., "Go" at the outset) does not* disqualify the event, provided individual execution steps were autonomous. 3. Cost Threshold: The training run cost more than $10,000,000 USD. * This threshold applies specifically to the market-equivalent rental cost of the compute hardware used (e.g., H100/B200 GPU hours) and excludes labor, facility overhead, or dataset acquisition costs. * The cost threshold must be met by a single discrete training run (a single model optimization process) rather than an aggregate of multiple small-scale experiments. 4. Frontier Companies: The company must be OpenAI, Anthropic, or Google DeepMind. 5. Official Communication: The claim must appear in an official company newsroom, technical blog, peer-reviewed paper, technical report, or published "AI permission list" or "autonomy framework."

Resolution criteria

This question will resolve as YES if, between April 1, 2026, and December 31, 2027 (inclusive, UTC), any of the named companies (OpenAI, Anthropic, or Google DeepMind) publishes an official statement confirming the following conditions were met for at least one specific instance: 1. Autonomous Initiation and Management: An AI agent (an autonomous AI system) initiated and managed a training run. * Management is only considered autonomous if the AI agent has the direct technical authority to modify hyperparameters or resource distribution directly in the training environment without a human reviewing the specific change before it takes effect. * Autonomous initiation requires the agent to independently determine at least one key training parameter (e.g., learning rate, batch size, or architecture variant) rather than simply triggering a human-pre-configured job template. 2. No Human-in-the-Loop for Steps: The statement must specify that the agent operated "autonomously," "without human-in-the-loop approval for individual steps," or using a "permission list" or "autonomy framework" that granted it authority to execute the run to completion without per-step human authorization. * A run is not considered autonomous if human-in-the-loop approval is required to resume the training process after an agent-initiated pause or failure-handling event. High-level human authorization at the start of the project (i.e., "Go" at the outset) does not* disqualify the event, provided individual execution steps were autonomous. 3. Cost Threshold: The training run cost more than $10,000,000 USD. * This threshold applies specifically to the market-equivalent rental cost of the compute hardware used (e.g., H100/B200 GPU hours) and excludes labor, facility overhead, or dataset acquisition costs. * The cost threshold must be met by a single discrete training run (a single model optimization process) rather than an aggregate of multiple small-scale experiments. 4. Frontier Companies: The company must be OpenAI, Anthropic, or Google DeepMind. 5. Official Communication: The claim must appear in an official company newsroom, technical blog, peer-reviewed paper, technical report, or published "AI permission list" or "autonomy framework."

Verification scores Stage 3

Quality: 92.0 Ambiguity: 75.0

Quality notes: This question is excellent for tracking critical transitions in AI autonomy. It directly operationalizes Metric #7 (high-stakes decisions) and Metric #14 (permission lists) from the Chan et al. (2026) framework for measuring AI R&D automation [[PDF] Measuring AI R&D Automation - arXiv](https://arxiv.org/pdf/2603.03992). The focus on autonomous training runs costing >$1M is a clear, high-stakes indicator of 'North Star' goals like OpenAI's autonomous research intern. While the resolution depends on public disclosure, the high-profile nature of such a milestone makes it likely to be reported if achieved. There is significant room for disagreement on when (or if) companies will bypass human-in-the-loop approval for million-dollar investments, making it a high-entropy question. It requires deep research into company safety frameworks and internal R&D roadmaps.

Ambiguity notes: The question is well-structured and uses specific metrics (Metric #14) from the referenced literature Measuring AI R&D Automation - arXiv. However, it relies on interpreting corporate communications ('official statement') which may use marketing language rather than the precise technical definitions required (e.g., 'no human-in-the-loop') Measuring AI R&D Automation - arXiv. The cost threshold (>$1M) may also require estimation if not explicitly stated Measuring AI R&D Automation - arXiv.

Adversarial review NEEDS_REVISION Edge risk: MEDIUM

Assessment: NEEDS_REVISION Edge case risk: MEDIUM

ASSESSMENT: NEEDS_REVISION REVIEW: The question is well-grounded in current AI R&D trends but contains two substantive issues that could hinder resolution or lead to a 'trivial' outcome. First, the $1 million USD cost threshold is likely too low for the 2026–2027 timeframe. Research indicates that frontier model training costs are scaling toward $1 billion by 2027 How much does it cost to train frontier AI models?. While $1 million is not 'trivial,' it may represent a routine medium-scale experiment rather than a 'high-stakes' milestone for labs like OpenAI or Google DeepMind, potentially leading to a 'YES' resolution for a relatively minor technical achievement. Second, the resolution criteria rely heavily on a specific form of public admission ('without human-in-the-loop approval'). As noted in the background paper Chan et al. (2026), labs face high oversight demands and risks when removing humans from the loop for significant actions like training Measuring AI R&D Automation - arXiv Measuring AI R&D Automation - arXiv. Due to safety, liability, and PR concerns, companies may be highly incentivized to describe their systems as 'human-supervised' or 'human-led' even if the agent is performing the bulk of the autonomous management. This creates a significant reporting bias where the technical event might occur, but the 'official statement' criteria are never met because the company avoids the specific phrasing required by the prompt. Finally, the reference to Chan et al. (2026) is accurate regarding 'AI permission lists' (Metric #14) and the 'Running experiments' stage, which explicitly identifies 'initiating training runs' as a key automation target Measuring AI R&D Automation - arXiv Measuring AI R&D Automation - arXiv. EVIDENCE: https://arxiv.org/abs/2603.03992, https://epoch.ai/blog/how-much-does-it-cost-to-train-frontier-ai-models, https://openai.com/news, https://www.anthropic.com/news SUGGESTION: 1. Increase the cost threshold to $10 million USD to ensure the event represents a truly 'high-stakes' delegation of trust. 2. Broaden the resolution criteria to include 'AI permission lists' or 'autonomy frameworks' as described in Chan et al. (2026). Instead of requiring an admission of 'no human-in-the-loop,' allow resolution if a company publishes a 'permission list' that grants an agent the authority to initiate and manage runs without per-step approval. 3. Clarify if the $1 million (or suggested $10 million) refers specifically to compute/hardware costs or total R&D costs, as the latter can be significantly higher How much does it cost to train frontier AI models?.

Edge cases 5 scenarios

OVERALL_RISK: MEDIUM SCENARIO: OpenAI reports that an AI agent 'managed' a training run by suggesting hyperparameter adjustments that were then manually reviewed and applied by a human engineer via a Slack integration. SEVERITY: HIGH FIX: Add: "Management is only considered autonomous if the AI agent has the technical authority to modify hyperparameters or resource distribution directly in the training environment without a human reviewing the specific change before it takes effect." SCENARIO: Anthropic announces a $1.5 million training run initiated by an agent, but the $1.5 million figure includes 'internal overhead' such as researcher salaries and facility costs, while the pure compute cost (H100/B200 rental equivalent) is only $800,000. SEVERITY: MEDIUM FIX: Add: "The $1,000,000 USD threshold applies specifically to the market-equivalent rental cost of the compute hardware used (e.g., H100/B200 GPU hours) and excludes labor, facility overhead, or dataset acquisition costs." SCENARIO: Google DeepMind claims an agent autonomously initiated a run, but the agent's 'initiation' was limited to selecting a pre-approved template from a 'Permission List' Measuring AI R&D Automation - arXiv where a human had already pre-defined all variables except the start time. SEVERITY: MEDIUM FIX: Add: "Autonomous initiation requires the agent to determine at least one key training parameter (e.g., learning rate, batch size, or architecture variant) independently within a broad search space, rather than simply triggering a human-pre-configured job template." SCENARIO: A lab uses an agent to monitor a run, and the agent identifies a 'loss spike' and pauses the run; however, the company policy requires a human to manually 'resume' the run after investigating the cause. SEVERITY: MEDIUM FIX: Add: "A run is not 'autonomous' if human-in-the-loop approval is required to resume the training process after an agent-initiated pause or failure-handling event." SCENARIO: OpenAI's 'autonomous research intern' Measuring AI R&D Automation - arXiv conducts a series of 100 small experiments that cumulatively cost $2 million, but no single discrete 'training run' exceeded the $1 million threshold. SEVERITY: LOW FIX: Add: "The $1,000,000 USD threshold must be met by a single discrete training run (a single model optimization process) rather than an aggregate of multiple small-scale experiments."

Revised question REVISED

### Question Title Autonomous AI-Led Training Runs at Frontier Labs by 2028 --- ### Question Between April 1, 2026, and December 31, 2027, will any "frontier AI company" (OpenAI, Anthropic, or Google DeepMind) publicly state in an official blog post, technical report, "AI permission list," or "autonomy framework" that they have used an AI agent to autonomously initiate and manage a single discrete AI model training run with market-equivalent compute costs exceeding $10 million USD? --- ### Background As of April 1, 2026, the automation of AI Research and Development (AIRDA) has moved from a theoretical possibility to a core strategic "North Star" for leading AI labs. OpenAI has publicly targeted the deployment of an "autonomous research intern" by late 2026, capable of independent multi-day investigations [Measuring AI R&D Automation - arXiv]. Similarly, Anthropic and Google DeepMind have published frameworks for "Intelligent AI Delegation" and "Agent Autonomy" to track the transition from human-led to agentic R&D processes. A critical inflection point in this transition is the delegation of "high-stakes decisions"—such as the initiation of large-scale, expensive training runs—to AI agents. Historically, training runs costing millions of dollars required rigorous human oversight for every stage, from resource allocation to monitoring for divergence. The Chan et al. (2026) paper, Measuring AI R&D Automation, proposes tracking this via "AI permission lists" (Metric #14), which define the actions an AI system is authorized to take without human intervention. This question tracks whether frontier labs will publicly cross the threshold of trusting an AI agent to manage a $10 million compute asset autonomously. While autonomous coding and hypothesis generation are increasingly common, the "Running experiments" stage (Section 2 of Chan et al. 2026) involves complex real-time interventions that represent a significant leap in operational trust. --- ### Resolution Criteria This question will resolve as YES if, between April 1, 2026, and December 31, 2027 (inclusive, UTC), any of the named companies (OpenAI, Anthropic, or Google DeepMind) publishes an official statement confirming the following conditions were met for at least one specific instance: 1. Autonomous Initiation and Management: An AI agent (an autonomous AI system) initiated and managed a training run. * Management is only considered autonomous if the AI agent has the direct technical authority to modify hyperparameters or resource distribution directly in the training environment without a human reviewing the specific change before it takes effect. * Autonomous initiation requires the agent to independently determine at least one key training parameter (e.g., learning rate, batch size, or architecture variant) rather than simply triggering a human-pre-configured job template. 2. No Human-in-the-Loop for Steps: The statement must specify that the agent operated "autonomously," "without human-in-the-loop approval for individual steps," or using a "permission list" or "autonomy framework" that granted it authority to execute the run to completion without per-step human authorization. * A run is not considered autonomous if human-in-the-loop approval is required to resume the training process after an agent-initiated pause or failure-handling event. High-level human authorization at the start of the project (i.e., "Go" at the outset) does not* disqualify the event, provided individual execution steps were autonomous. 3. Cost Threshold: The training run cost more than $10,000,000 USD. * This threshold applies specifically to the market-equivalent rental cost of the compute hardware used (e.g., H100/B200 GPU hours) and excludes labor, facility overhead, or dataset acquisition costs. * The cost threshold must be met by a single discrete training run (a single model optimization process) rather than an aggregate of multiple small-scale experiments. 4. Frontier Companies: The company must be OpenAI, Anthropic, or Google DeepMind. 5. Official Communication: The claim must appear in an official company newsroom, technical blog, peer-reviewed paper, technical report, or published "AI permission list" or "autonomy framework." Resolution Sources: - OpenAI: openai.com/news - Anthropic: anthropic.com/news or anthropic.com/research - Google DeepMind: deepmind.google/blog or research.google/blog If no such statement is published by 23:59 UTC on December 31, 2027, the question resolves as NO. --- ### Definitions - AIRDA (AI R&D Automation): The use of AI to carry out parts of the AI R&D pipeline, including capabilities research and safety research [Measuring AI R&D Automation - arXiv]. - Training Run: A discrete process of optimizing a machine learning model's parameters on a dataset, typically involving distributed computation across a GPU cluster. - AI Agent: An AI system capable of pursuing complex goals with limited human intervention by perceiving its environment and taking actions. - Permission List / Autonomy Framework: Documentation defining the actions AI systems are authorized to take with different levels of human approval, including where none is required. - Frontier AI Company: For this question, limited to OpenAI, Anthropic, and Google DeepMind.

Forecast rationale

Time left: ~21 months (638 days) until the resolution date of December 31, 2027. The status quo is that no such autonomous training run has been publicly acknowledged. For a YES outcome, a frontier lab must publicly confirm an AI agent autonomously initiated and managed a $10 million training run without human-in-the-loop intervention for individual steps. A YES outcome is plausible because labs like OpenAI consider the 'autonomous research intern' a North Star goal, and managing mid-sized ($10M) runs autonomously would be a powerful proof of concept for automating multi-billion dollar runs. A NO outcome is more likely, however, because $10 million is a massive financial risk to run without human oversight in case of node failures or divergence. Additionally, safety frameworks (like Anthropic's RSP) mandate human checks, and labs might avoid publicizing such autonomous capabilities to avoid regulatory blowback or appearing reckless. I would be indifferent at 28 cents on the dollar for a YES bet.

Importance rationale

The question tracks a 'crucial transition' from AI as an assistant to AI as an autonomous agent Measuring AI R&D Automation - arXiv. Granting AI systems the authority to manage large-scale training runs (Metric #14) is a leading indicator for the 'oversight gap' Measuring AI R&D Automation - arXiv. OpenAI has already set a 'North Star' goal for an 'autonomous research intern' by 2026/2027, making this a highly decision-relevant indicator of institutional trust in AI R&D automation (AIRDA). Resolution would meaningfully update beliefs about the trajectory of AI capabilities and the speed at which human oversight is being removed from the critical path of AI development.

Decomposition & Research Stage 6b 2 subquestions

Initial forecast: 28%→Research-informed: 68% (+40pp)

Re-forecast rationale

The possibility of an AI agent autonomously initiating and managing a $10 million training run by the end of 2027 is supported by several factors, yet tempered by institutional caution. As of April 2026, leading labs like OpenAI, Anthropic, and Google DeepMind have explicitly identified 'AI R&D Automation' (AIRDA) as a strategic priority. OpenAI specifically targets the deployment of an 'autonomous research intern' by September 2026, intended to manage multi-day experimental cycles independently. Technical infrastructure is also evolving; OpenAI's 2026 acquisition of Astral suggests a move toward integrating agentic resource management directly into the research stack. However, the $10 million threshold is a significant financial and operational barrier. Currently, frontier labs employ 'AI permission lists' and 'Intelligent AI Delegation' frameworks that mandate human-in-the-loop (HITL) verification for high-stakes actions. While agents are already automating tasks in the $100k to $1M range (such as weekend-long ablation studies), a $10M run represents a 'high-criticality' asset. The transition to 'lights-out' autonomy for such a run requires not just technical capability in troubleshooting and monitoring, but a fundamental shift in safety protocols (Metric #7). Given the 'North Star' goals for 2026 and 2027, it is highly probable that at least one lab will reach this milestone and publicize it as a breakthrough in automation. The 21-month window (April 2026 - December 2027) provides sufficient time for the 'autonomous intern' to graduate to larger-scale resource management as confidence in agentic reliability grows. I estimate a 68% probability that a frontier lab will publicly confirm such an autonomous run, likely framed as a demonstration of their advanced AIRDA capabilities.

SQ1: What are the specific technical and safety thresholds defined in the 'AI permission lists' and 'Intelligent AI Delegation' frameworks at OpenAI, Anthropic, and Google DeepMind??

Summary: As of early 2026, OpenAI, Anthropic, and Google DeepMind have implemented structured metrics to govern AI autonomy, specifically 'Metric #14' (AI permission lists) and 'Metric #7' (AI use in high-stakes decisions). These metrics originate from the 'Measuring AI R&D Automation' framework https://arxiv.org/pdf/2603.03992.pdf. Currently, none of the three labs permit AI agents to autonomously 'initiate training runs' or 'modify production code' without human-in-the-loop verification for high-stakes assets https://arxiv.org/pdf/2602.11865.pdf. Instead, they utilize 'Intelligent AI Delegation' frameworks that require 'just-in-time' access and 'privilege attenuation' to ensure agents operate only within narrow, pre-approved scopes https://arxiv.org/pdf/2602.11865.pdf. Safety thresholds are often tied to 'AI Self-improvement' benchmarks—for instance, OpenAI triggers high-level safety protocols if an agent matches the performance of a senior research engineer https://arxiv.org/pdf/2603.03992.pdf, while Anthropic uses a 'progress compression' metric to flag dangerous levels of R&D automation https://arxiv.org/pdf/2603.03992.pdf.

Background: The core of the forecasting question is whether a frontier lab (OpenAI, Anthropic, or Google DeepMind) will trust an AI agent to manage a $10 million compute asset autonomously. This represents a significant shift from 'AI-assisted' research to 'AI-led' operations. Researching current internal protocols for high-stakes compute allocation—specifically the 'AI permission lists' and 'Intelligent AI Delegation' frameworks mentioned by Chan et al. (2026) and Google DeepMind—is critical [a101b9]. This sub-question focuses on the institutional and safety-governance thresholds that must be crossed before a lab permits an agent to 'initiate training runs' or 'modify production code' without human-in-the-loop verification [a101b9]. Understanding the specific 'Metric #14' (AI permission lists) and 'Metric #7' (AI use in high-stakes decisions) provides the direct evidence needed to determine if these labs are moving toward the $10 million threshold.

Detailed research

Research into current frontier lab protocols reveals that OpenAI, Anthropic, and Google DeepMind have transitioned from theoretical safety frameworks to more structured, metric-driven governance as of early 2026. The primary evidence for these shifts is found in the work of Chan et al. (2026) regarding 'AI R&D Automation' (AIRDA) metrics and Google DeepMind's 'Intelligent AI Delegation' framework (Tomašev et al., 2026). ### 1. Metric #14: AI Permission Lists Metric #14 is defined as a systematic record of actions AI systems are authorized to take, categorized by the required level of human approval https://arxiv.org/pdf/2603.03992.pdf. OpenAI: Tracks autonomous capabilities within its Preparedness Framework* (updated 2025b). It establishes a 'High' threshold for 'AI Self-improvement' when an agent's performance equals a 'highly performant mid-career research engineer assistant' relative to 2024 baselines https://arxiv.org/pdf/2603.03992.pdf. Anthropic: Utilizes its Responsible Scaling Policy* (2026a) to define automation thresholds. A key safety trigger occurs when AI progress is 'compressed' such that two years of 2018–2024 era progress is achieved within a single year https://arxiv.org/pdf/2603.03992.pdf. Google DeepMind: Employs the Frontier Safety Framework* (2025a), which mandates high security for models capable of significantly accelerating Machine Learning R&D https://arxiv.org/pdf/2603.03992.pdf. ### 2. Metric #7: AI Use in High-Stakes Decisions Metric #7 tracks the extent to which AI agents make critical operational choices without human intervention https://arxiv.org/pdf/2603.03992.pdf. * Thresholds for Autonomous Training/Code Modification: Current protocols generally prohibit 'initiating training runs' or 'modifying production code' without human-in-the-loop (HITL) verification for high-stakes assets https://arxiv.org/pdf/2602.11865.pdf. * Intelligent AI Delegation Framework (Google DeepMind): Proposes 'Risk-Adaptive Access' where permissions are granted on a 'just-in-time' basis. For high-criticality tasks, the framework mandates either HITL approval or third-party cryptographic authorization https://arxiv.org/pdf/2602.11865.pdf. * Capability Attenuation: To prevent unauthorized escalation, agents are restricted by 'privilege attenuation,' meaning they can only pass on a subset of their own permissions to sub-agents https://arxiv.org/pdf/2602.11865.pdf. ### 3. Agentic Protocol Standards The labs are moving toward standardized protocols for these delegations: Anthropic: Uses the Model Context Protocol* (MCP, 2024) to connect models to tools, though as of 2026, it is noted to lack a native policy layer for deep delegation chains https://arxiv.org/pdf/2602.11865.pdf. Google DeepMind: Has developed Agents-to-Agents (A2A, 2025b) and Agents-to-Payments* (A2P/AP2, 2025a) protocols, but internal research suggests these still require 'semantic attenuation' to safely handle autonomous operations https://arxiv.org/pdf/2602.11865.pdf.

SQ2: What is the current state and projected roadmap for AI agents autonomously managing R&D training runs at frontier labs??

Summary: OpenAI has established a 'North Star' goal to develop a fully autonomous AI researcher by 2028, with a near-term roadmap to deploy an 'autonomous research intern' by September 2026 OpenAI is throwing everything into building a fully automated ... OpenAI targets an autonomous researcher by September. This 'intern' is designed to independently manage research tasks and experiments spanning several days. Currently, AI agents are already being used at frontier labs to compress week-long coding and experimental tasks into weekends OpenAI is throwing everything into building a fully automated .... To scale to autonomous $10 million training runs, labs are developing three operational pillars: real-time monitoring via 'chain-of-thought' scratch pads, automated resource allocation through integrated tooling like Astral, and agentic troubleshooting of code and data OpenAI is throwing everything into building a fully automated ... OpenAI targets an autonomous researcher by September. While agents are actively managing smaller-scale R&D tasks and revenue-generating operations in the $100k-$1M range, the transition to fully autonomous management of large-scale $10M+ frontier training runs remains the primary objective for the 2026–2028 window.

Background: For an AI agent to autonomously manage a $10 million training run, it must handle 'Running experiments' (Section 2 of Chan et al. 2026), which involves real-time monitoring for divergence, resource allocation, and troubleshooting. The $10 million threshold is a specific financial and operational barrier. This sub-question addresses the technical feasibility and cost trends: has an AI agent demonstrated the ability to manage smaller-scale runs (e.g., $100k - $1M) autonomously, and what are the stated roadmaps for scaling this to 'autonomous research interns' by late 2026? Investigating the 'North Star' goals of these labs—such as OpenAI's target for an autonomous researcher capable of multi-day independent investigations—will reveal the trajectory toward the $10 million autonomous run by the end of 2027.

Detailed research

### Current State of AI Autonomous Research (2025–2026) As of early 2026, AI agents have transitioned from basic coding assistants to sophisticated tools capable of managing multi-day research tasks. OpenAI’s Chief Scientist Jakub Pachocki reported in March 2026 that he uses agentic tools (such as 'Codex' and internal research agents) to execute experiments in a single weekend that previously required a full week of human effort OpenAI is throwing everything into building a fully automated ... OpenAI targets an autonomous researcher by September. These agents are being integrated into the core research stack, utilizing 'chain-of-thought monitoring' where models document their logic in 'scratch pads' to allow human researchers to oversee their reasoning and detect misalignment in real-time OpenAI is throwing everything into building a fully automated .... ### The Roadmap: 'North Star' and Autonomous Interns OpenAI has officially designated the creation of a fully automated AI researcher as its 'North Star' goal for the next several years OpenAI is throwing everything into building a fully automated .... * September 2026 Milestone: The labs are targeting the release of an 'autonomous research intern.' This agent is designed to tackle specific, bounded research problems independently over several days, handling the planning and execution of experiments OpenAI is throwing everything into building a fully automated ... OpenAI targets an autonomous researcher by September. * 2028 Target: The long-term objective is a 'multi-agent research system' capable of operating like a full research lab within a data center. This system is intended to solve complex scientific problems in fields like physics and biology that currently exceed human capability OpenAI targets an autonomous researcher by September. ### Operational Components for Training Runs For an agent to manage a large-scale training run (such as the $10M threshold), three critical operational components must be automated: 1. Real-time Monitoring for Divergence: Current systems utilize 'chain-of-thought monitoring' to track model progress OpenAI is throwing everything into building a fully automated .... In the context of training runs, this involves detecting loss spikes or gradient explosions. While frontier labs are automating these detection layers, high-level governance still relies on human validation of autonomous findings OpenAI targets an autonomous researcher by September. 2. Resource Allocation: Frontier labs have begun integrating AI agents with infrastructure management tools. For instance, OpenAI's acquisition of Astral in early 2026 was aimed at embedding agentic coding and resource management directly into the Python-based tools researchers use to allocate compute OpenAI targets an autonomous researcher by September. 3. Troubleshooting: Training failures often stem from hardware issues or data imbalances. Current 'training ops' still involve significant human-led stress testing and Slurm reservation management, though agents are increasingly used to handle the sub-tasks of debugging code and optimizing dataloaders frontier model training methodologies - Alex Wa's Blog OpenAI is throwing everything into building a fully automated .... ### Scale of Autonomous Runs There is evidence that agents are managing 'smaller-scale' operations in the $100k - $1M range, particularly in algorithmic trading and revenue operations, where agents have been reported to close over $1M in revenue within 90 days. In pure R&D, agents are currently used to 'run experiments over a weekend,' which correlates with the compute costs of smaller-scale model fine-tuning or ablation studies, though a fully 'lights-out' $1M training run managed entirely by an agent without human check-ins has not been publicly documented as a standard industry milestone yet.

Probabilistic Decomposition Stage 6c 2 components

Structure: Sequential Chain

Formula: P(YES) = P(C1) * P(C2|C1)

C1: By December 31, 2027, will any frontier AI company (OpenAI, Anthropic, or Google DeepMind) update its official 'AI permission lists' (Metric #14) or 'autonomy frameworks' to explicitly authorize an AI agent to autonomously initiate and manage a single discrete training run exceeding $10 million USD? 25% Expected: likely 35-60%

Role: First node in sequential chain — provides the technical and institutional authorization necessary for the event.

Dependencies: C1 and C2 are expected to be strongly positively correlated. If a lab officially authorizes an agent to manage $10M+ assets (C1), it is significantly more likely they will report a successful run (C2), as the authorization implies a strategic desire to reach and publicize this milestone. Conversely, if C1 fails, C2 can only resolve YES if the lab bypasses its own formal governance frameworks.

Background

As of early 2026, OpenAI, Anthropic, and Google DeepMind have adopted 'Metric #14' (AI permission lists) and 'Metric #7' (AI use in high-stakes decisions) as core governance tools for tracking the delegation of authority to AI agents [https://arxiv.org/pdf/2603.03992.pdf]. Currently, these 'permission lists' prohibit agents from autonomously 'initiating training runs' or 'modifying production code' for high-stakes assets without human-in-the-loop (HITL) verification [https://arxiv.org/pdf/2602.11865.pdf]. This component tracks whether the institutional and safety-governance thresholds are raised to permit an agent to manage a $10 million compute asset autonomously. A 'frontier AI company' is defined as OpenAI, Anthropic, or Google DeepMind. The $10 million compute threshold refers to the market-equivalent cost of hardware utilization for a single discrete training run.

Forecast rationale

Based on current documentation from early 2026, frontier AI companies like Google DeepMind, OpenAI, and Anthropic are adopting governance frameworks such as 'Metric #14' (AI permission lists) to track and control the delegation of authority to AI agents https://arxiv.org/pdf/2603.03992.pdf. As of early 2026, these lists explicitly categorize 'initiating training runs' and 'modifying production code' as high-stakes actions that require mandatory human-in-the-loop (HITL) verification to prevent risks such as runaway automation or resource misuse https://arxiv.org/pdf/2603.03992.pdf https://arxiv.org/pdf/2602.11865.pdf. The probability of these companies updating their frameworks to authorize autonomous management of a $10 million training run by December 31, 2027, is estimated at 25%. While the transition toward 'risk-adaptive' and 'just-in-time' permissions is being discussed to facilitate AI R&D automation https://arxiv.org/pdf/2602.11865.pdf, the $10 million threshold represents a significant financial and strategic asset. Current safety paradigms emphasize 'policy-as-code' and 'semantic constraints' to prevent agents from exercising broad capabilities without oversight https://arxiv.org/pdf/2602.11865.pdf. The leap from the current 'prohibited' status to 'explicitly authorized' for such high-value discrete runs within 21 months would require a major shift in institutional risk tolerance and a high level of confidence in agentic reliability that is not yet reflected in the 2026 baseline governance documents https://arxiv.org/pdf/2603.03992.pdf https://arxiv.org/pdf/2602.11865.pdf. Additionally, 'Metric #14' is designed to track oversight demand; increasing autonomy for $10M runs would mark a substantial reduction in oversight that contradicts the cautious 'human-sovereign' protocols currently being proposed https://arxiv.org/pdf/2602.11865.pdf.

C2: Given the institutional authorization in C1, will any frontier AI company publicly state in an official blog post, technical report, or 'autonomy framework' before January 1, 2028, that they have used an AI agent to autonomously manage a training run exceeding $10 million USD? 72% Expected: likely 50-75%

Role: Second node in sequential chain (conditional on C1) — covers the execution, public disclosure, and the possibility of bypassing formal frameworks.

Dependencies: This component is the conditional probability that a public announcement occurs given the technical/institutional greenlight. It also covers the 'model-breaking' scenario where a lab reports a run despite not having a formal 'permission list' update that applies to general agent operations.

Background

This component addresses the 'publicly state' requirement of the original question and acts as a 'model-breaker' by testing if the formal 'permission list' (Metric #14) process is the only route to resolution. While labs are targeting 'autonomous research interns' by September 2026 [OpenAI targets an autonomous researcher by September 2026], they may choose to announce a successful $10M+ autonomous run as a 'one-off' breakthrough or technical report [https://arxiv.org/pdf/2603.03992.pdf] without necessarily having updated their formal, broadly applicable permission lists beforehand. Conversely, they might update their permissions (C1) but fail to complete or publicly document a successful run within the 2027 window due to safety-induced pauses or competitive secrecy. A 'frontier AI company' is defined as OpenAI, Anthropic, or Google DeepMind. The $10 million compute threshold refers to the market-equivalent cost of hardware utilization for a single discrete training run.

Forecast rationale

The probability of a frontier AI company (OpenAI, Anthropic, or Google DeepMind) publicly stating that an AI agent has autonomously managed a training run exceeding $10 million before 2028 is high, driven by the rapid technical trajectory toward autonomous AI research. Current evidence shows that OpenAI is actively targeting an 'autonomous research intern' by September 2026 and a 'fully automated research agent' by 2028 https://arxiv.org/pdf/2603.03992.pdf. Google DeepMind has already released 'Aletheia' (Feng et al., 2026b), an autonomous research agent that generates research papers and solves complex math problems entirely without human intervention, and its CEO has signaled that automated researchers are 'a few years away' as of 2025 https://arxiv.org/pdf/2603.03992.pdf. Anthropic’s 'Responsible Scaling Policy (RSP) v3.0' and 'Frontier Safety Roadmap' explicitly acknowledge that models now take 'autonomous, multi-step actions' and include plans to catalog 'fully autonomous research' conducted internally by Claude Responsible Scaling Policy Version 3.0 - Anthropic. The $10 million threshold is well within the range of modern frontier training costs, which often reach hundreds of millions of dollars. As these labs transition from AI-assisted engineering to agent-managed pipelines to increase efficiency, they are highly incentivized to announce such breakthroughs to demonstrate technical leadership. Research benchmarks for AI R&D automation (AIRDA) already exist, specifically identifying 'AI permission lists' as a key metric for monitoring this transition https://arxiv.org/pdf/2603.03992.pdf. While safety concerns or competitive secrecy could delay a public announcement, the low hurdle of a single 'one-off' breakthrough statement in a technical report or blog post makes a 'YES' resolution more likely than not.

Reconciled Forecast Stage 6d 30%

Bottom-up: 18%→Dep-adj: 22%→Struct-adj: 25%→Holistic: 35%→Final: 30%

Initial (06): 28%→Research (06b): 68%→Reconciled (06d): 30%

Reconciliation rationale

The mechanical bottom-up estimate (18%) is low because it assumes the events are strictly sequential and independent. However, the dependency between formal authorization (C1) and a public statement (C2) is extremely high; if a lab decides to publicize such a run, they will likely update their frameworks simultaneously or even retrospectively to maintain governance appearances. The structure-adjusted estimate (25%) accounts for the possibility that a lab might report a 'rogue' or 'breakthrough' autonomous run even without a prior formal 'permission list' update. The holistic estimate (35%) is higher because it weights the intense competitive pressure and 'North Star' strategic alignment of OpenAI and DeepMind more heavily than the bureaucratic hurdle of framework updates. The divergence (10 points) is explained by the decomposition's focus on formal documentation versus the holistic view's focus on technological momentum. Given the aggressive 2026-2027 timelines for 'autonomous interns,' the final forecast reconciles these by leaning toward the holistic view while respecting the significant operational barriers to $10M+ autonomy.

3.1 AI Progress

▶ 77% Will any AI model achieve a success rate of 70% or higher on the full 'SWE-bench Pro' benchmark by December 31, 2027? REVISED ITNSSS66 Imp85

Quality88

Ambiguity95

Soon80

Sudden50

Sharp35

Priority scores (ITN + Soon/Sudden/Sharp) Stage 2c

Priority66

Neglectedness55

Tractability75

Neglectedness: While general 'SWE-bench' and 'SWE-bench Verified' are frequently forecasted on platforms like Metaculus and Manifold, the 'SWE-bench Pro' variant is newer and less saturated [[2509.16941] SWE-Bench Pro: Can AI Agents Solve Long-Horizon ...](https://arxiv.org/abs/2509.16941). However, because it is a high-profile benchmark released by Scale AI and already has an active public leaderboard, it is being monitored by industry analysts and some prediction markets (e.g., Manifold has a market on the EOY 2026 score) SWE-Bench Pro (Public Dataset) - Scale Labs. No long-term (2027) forecast specifically for the 70% threshold was found on Metaculus, Polymarket, or INFER.

Tractability: Forecasting this requires synthesizing technical scaling laws, agentic workflow improvements, and historical benchmark saturation rates [[2509.16941] SWE-Bench Pro: Can AI Agents Solve Long-Horizon ...](https://arxiv.org/abs/2509.16941). There is a rich information environment (leaderboards, research papers, and compute trends) that allows a skilled researcher to move significantly beyond a naive prior SWE-Bench Pro (Public Dataset) - Scale Labs.

Soon: The question tracks an actively unfolding development at a critical juncture for AI agent deployment [[2509.16941] SWE-Bench Pro: Can AI Agents Solve Long-Horizon ...](https://arxiv.org/abs/2509.16941). The 2027 resolution window aligns with industry expectations for the next generation of frontier models (GPT-6, etc.) SWE-Bench Pro (Public Dataset) - Scale Labs. Knowing the answer now is highly valuable for calibrating safety and economic transition timelines.

Sudden: While model releases are discrete events, benchmark improvements generally follow a predictable, gradual scaling trend SWE-Bench Pro (Public Dataset) - Scale Labs. However, the shift from 'Verified' (bug fixes) to 'Pro' (long-horizon engineering) represents a more discrete state change in capability that could surprise observers if architectural breakthroughs occur [[2509.16941] SWE-Bench Pro: Can AI Agents Solve Long-Horizon ...](https://arxiv.org/abs/2509.16941).

Sharp: Progress on this metric follows a visible, incremental trajectory with many 'warning shots' in the form of intermediate model releases (e.g., GPT-5.2, Claude 4.5) SWE-Bench Pro (Public Dataset) - Scale Labs. It is not an 'all-or-nothing' risk where the first signal is the catastrophe, as improvements in coding agents are publicly tracked and debated [[2509.16941] SWE-Bench Pro: Can AI Agents Solve Long-Horizon ...](https://arxiv.org/abs/2509.16941).

Proto-question Stage 1

Will any AI model achieve a success rate of 70% or higher on the 'SWE-bench Pro' benchmark by December 31, 2027?

Why this question? The paper identifies SWE-bench as a key evaluation for AI R&D capabilities [a512a0]. As of March 2026, top-tier models were scoring approximately 56-57% on the more difficult, contamination-resistant 'SWE-bench Pro' variant. Reaching 70% would represent a significant leap in an agent's ability to handle complex, multi-repository software engineering tasks, which is a core upstream indicator of automated R&D progress.

Paper reference: Metric #1: AI R&D Evaluations (Chan et al., 2026) [a512a0]

Refined question Stage 2

### Question Title Will any AI model achieve a success rate of 70% or higher on the 'SWE-bench Pro' benchmark by December 31, 2027? ### Background The development of AI agents capable of autonomous software engineering is a key milestone in AI research. One of the most rigorous benchmarks for this capability is SWE-bench Pro, an evolution of the original SWE-bench designed to evaluate agents on long-horizon, enterprise-level software engineering tasks https://arxiv.org/abs/2509.16941. While previous benchmarks like 'SWE-bench Verified' often involve isolated bug fixes, SWE-bench Pro consists of 1,865 complex problems sourced from 41 actively maintained repositories https://arxiv.org/abs/2509.16941. These tasks often require "hours to days for a professional software engineer to complete" and involve modifications across multiple files and directories https://arxiv.org/abs/2509.16941. As of April 1, 2026, the state-of-the-art (SOTA) performance on the official SWE-bench Pro (Public Dataset) leaderboard is 45.89%, achieved by the model claude-opus-4-5-20251101 SWE-Bench Pro (Public Dataset) - Scale Labs. Other high-performing systems, such as Bito's AI Architect, have claimed scores as high as 60.8% in independent evaluations, though these are not always reflected on the primary public leaderboard SWE-Bench Pro (Public Dataset) - Scale Labs. A jump to 70% would indicate that AI agents can reliably solve the majority of complex, real-world engineering issues, signaling a move toward fully autonomous R&D capabilities. ### Resolution Criteria This question will resolve as YES if, at any point between April 1, 2026, and 23:59 UTC on December 31, 2027, any AI model or agentic system is officially recorded as having achieved a Resolve Rate (Success Rate) of 70.0% or higher on the SWE-bench Pro (Public Dataset) benchmark. 1. Resolution Source: The primary source for resolution is the official SWE-bench Pro (Public Dataset) leaderboard maintained by Scale AI SWE-Bench Pro (Public Dataset) - Scale Labs. 2. Fallback Provision: If the primary leaderboard is discontinued, becomes inaccessible, or stops updating, this question may be resolved based on: * A peer-reviewed academic publication (e.g., on arXiv or at a major AI conference like NeurIPS/ICLR) that uses the original SWE-bench Pro dataset of 1,865 problems as defined in https://arxiv.org/abs/2509.16941. * An official technical report or announcement from a major AI lab (e.g., OpenAI, Anthropic, Google DeepMind, or Scale AI) providing verifiable evidence of the score. 3. Definitions: * AI Model/System: Any software system, large language model (LLM), or agentic framework (e.g., combining a model with tools, scaffolding, or search). * Success Rate / Resolve Rate: The percentage of the 1,865 tasks in the SWE-bench Pro dataset that the agent successfully resolves https://arxiv.org/abs/2509.16941. A task is "resolved" if the model's patch passes the "fail-to-pass" tests (fixing the issue) and the "pass-to-pass" tests (ensuring no regressions) SWE-Bench Pro (Public Dataset) - Scale Labs. * Public Availability: The model does not need to be publicly available for this question to resolve as YES, provided the score is published in an official capacity (e.g., a technical report or peer-reviewed paper). 4. Threshold: The score must be 70.0% or higher (rounding to the nearest tenth). For example, 69.95% would resolve as YES, while 69.94% would resolve as NO. ### Technical Definitions & Reference Links * SWE-bench Pro: Defined by Deng et al. (2025) https://arxiv.org/abs/2509.16941. * AI Model: General term for machine learning systems as described on Wikipedia. * Success Rate: In this context, the "Resolve Rate" as defined in the SWE-bench documentation SWE-bench Leaderboards.

Background

The development of AI agents capable of autonomous software engineering is a key milestone in AI research. One of the most rigorous benchmarks for this capability is SWE-bench Pro, an evolution of the original SWE-bench designed to evaluate agents on long-horizon, enterprise-level software engineering tasks [[2509.16941] SWE-Bench Pro: Can AI Agents Solve Long-Horizon ...](https://arxiv.org/abs/2509.16941). While previous benchmarks like 'SWE-bench Verified' often involve isolated bug fixes, SWE-bench Pro consists of 1,865 complex problems sourced from 41 actively maintained repositories [[2509.16941] SWE-Bench Pro: Can AI Agents Solve Long-Horizon ...](https://arxiv.org/abs/2509.16941) Scale Labs Leaderboard: SWE-Bench Pro (Public Dataset). These tasks often require \"hours to days for a professional software engineer to complete\" and involve modifications across multiple files and directories [[2509.16941] SWE-Bench Pro: Can AI Agents Solve Long-Horizon ...](https://arxiv.org/abs/2509.16941). As of April 1, 2026, the state-of-the-art (SOTA) performance on the official SWE-bench Pro (Public Dataset) leaderboard is 45.89%, achieved by the model claude-opus-4-5-20251101 Scale Labs Leaderboard: SWE-Bench Pro (Public Dataset). While some systems, such as Bito's AI Architect, have claimed scores as high as 60.8%, these evaluations were conducted on a subset of only 293 tasks from five repositories rather than the full 1,865-problem dataset Bito's AI Architect tops SWE-Bench Pro Evaluation. A jump to 70% on the full benchmark would indicate that AI agents can reliably solve the majority of complex, real-world engineering issues, signaling a move toward fully autonomous R&D capabilities. ### Resolution Criteria This question will resolve as YES if, at any point between April 1, 2026, and 23:59 UTC on December 31, 2027, any AI model or agentic system is officially recorded as having achieved a Resolve Rate (Success Rate) of 70.0% or higher on the SWE-bench Pro benchmark. 1. Resolution Source: The primary source for resolution is the official SWE-bench Pro (Public Dataset) leaderboard maintained by Scale AI Scale Labs Leaderboard: SWE-Bench Pro (Public Dataset). 2. Scope of Evaluation: The 70.0% success rate must be calculated based on the full 1,865-problem dataset (comprising the Public, Private, and Held-out sets) as defined in the original paper [[2509.16941] SWE-Bench Pro: Can AI Agents Solve Long-Horizon ...](https://arxiv.org/abs/2509.16941), rather than any single subset (such as the 731-instance Public Set). 3. Fallback Provision: If the primary leaderboard is discontinued, becomes inaccessible, or stops updating, this question may be resolved based on: * A peer-reviewed academic publication (e.g., on arXiv or at a major AI conference like NeurIPS/ICLR) that uses the original SWE-bench Pro dataset of 1,865 problems. * An official technical report or announcement from a major AI lab (e.g., OpenAI, Anthropic, Google DeepMind, or Scale AI) providing verifiable evidence of the score on the full dataset. 4. Definitions: * AI Model/System: Any software system, large language model (LLM), or agentic framework (e.g., combining a model with tools, scaffolding, or search). A 'system' or 'agentic framework' may consist of any combination of multiple models, tools, and recursive processes, provided they function as a unified software entity to solve the tasks without external human direction. * Success Rate / Resolve Rate: The percentage of the tasks in the SWE-bench Pro dataset that the agent successfully resolves. A task is \"resolved\" if the model's patch passes the \"fail-to-pass\" tests (fixing the issue) and the \"pass-to-pass\" tests (ensuring no regressions) Scale Labs Leaderboard: SWE-Bench Pro (Public Dataset). * Autonomy: To qualify, the system must operate autonomously during the evaluation of the problems; systems requiring 'Human-in-the-Loop' (HITL) intervention, manual task selection, or human-led debugging during the benchmark execution are explicitly excluded. 5. Thresholds and Conflicts: * Rounding: Calculations for resolution will be performed by dividing the number of successfully resolved tasks by the total number of tasks in the dataset, with the resulting percentage rounded to the nearest tenth (0.05 rounds up). For example, 69.95% would resolve as YES, while 69.94% would resolve as NO. * Benchmark Updates: If the total number of tasks in the official SWE-bench Pro benchmark changes from 1,865 (e.g., due to a 'v2' update), the success rate will be calculated as the number of resolved tasks divided by the total number of tasks in the then-current version of the benchmark, provided it is still officially titled 'SWE-bench Pro'. * Precedence: In the event of a conflict between reported scores, the official Scale AI leaderboard takes precedence over technical reports or papers unless the leaderboard is proven to be using a modified version of the dataset. ### Technical Definitions & Reference Links * SWE-bench Pro: Defined by Deng et al. (2025) [[2509.16941] SWE-Bench Pro: Can AI Agents Solve Long-Horizon ...](https://arxiv.org/abs/2509.16941). * Success Rate: In this context, the \"Resolve Rate\" as defined in the SWE-bench documentation.

Resolution criteria

This question will resolve as YES if, at any point between April 1, 2026, and 23:59 UTC on December 31, 2027, any AI model or agentic system is officially recorded as having achieved a Resolve Rate (Success Rate) of 70.0% or higher on the SWE-bench Pro benchmark. 1. Resolution Source: The primary source for resolution is the official SWE-bench Pro (Public Dataset) leaderboard maintained by Scale AI Scale Labs Leaderboard: SWE-Bench Pro (Public Dataset). 2. Scope of Evaluation: The 70.0% success rate must be calculated based on the full 1,865-problem dataset (comprising the Public, Private, and Held-out sets) as defined in the original paper [[2509.16941] SWE-Bench Pro: Can AI Agents Solve Long-Horizon ...](https://arxiv.org/abs/2509.16941), rather than any single subset (such as the 731-instance Public Set). 3. Fallback Provision: If the primary leaderboard is discontinued, becomes inaccessible, or stops updating, this question may be resolved based on: * A peer-reviewed academic publication (e.g., on arXiv or at a major AI conference like NeurIPS/ICLR) that uses the original SWE-bench Pro dataset of 1,865 problems. * An official technical report or announcement from a major AI lab (e.g., OpenAI, Anthropic, Google DeepMind, or Scale AI) providing verifiable evidence of the score on the full dataset. 4. Definitions: * AI Model/System: Any software system, large language model (LLM), or agentic framework (e.g., combining a model with tools, scaffolding, or search). A 'system' or 'agentic framework' may consist of any combination of multiple models, tools, and recursive processes, provided they function as a unified software entity to solve the tasks without external human direction. * Success Rate / Resolve Rate: The percentage of the tasks in the SWE-bench Pro dataset that the agent successfully resolves. A task is \"resolved\" if the model's patch passes the \"fail-to-pass\" tests (fixing the issue) and the \"pass-to-pass\" tests (ensuring no regressions) Scale Labs Leaderboard: SWE-Bench Pro (Public Dataset). * Autonomy: To qualify, the system must operate autonomously during the evaluation of the problems; systems requiring 'Human-in-the-Loop' (HITL) intervention, manual task selection, or human-led debugging during the benchmark execution are explicitly excluded. 5. Thresholds and Conflicts: * Rounding: Calculations for resolution will be performed by dividing the number of successfully resolved tasks by the total number of tasks in the dataset, with the resulting percentage rounded to the nearest tenth (0.05 rounds up). For example, 69.95% would resolve as YES, while 69.94% would resolve as NO. * Benchmark Updates: If the total number of tasks in the official SWE-bench Pro benchmark changes from 1,865 (e.g., due to a 'v2' update), the success rate will be calculated as the number of resolved tasks divided by the total number of tasks in the then-current version of the benchmark, provided it is still officially titled 'SWE-bench Pro'. * Precedence: In the event of a conflict between reported scores, the official Scale AI leaderboard takes precedence over technical reports or papers unless the leaderboard is proven to be using a modified version of the dataset. ### Technical Definitions & Reference Links * SWE-bench Pro: Defined by Deng et al. (2025) [[2509.16941] SWE-Bench Pro: Can AI Agents Solve Long-Horizon ...](https://arxiv.org/abs/2509.16941). * Success Rate: In this context, the \"Resolve Rate\" as defined in the SWE-bench documentation.

Verification scores Stage 3

Quality: 88.0 Ambiguity: 95.0

Quality notes: This is a high-quality forecasting question. It uses a well-established and objective benchmark (SWE-bench Pro) which is recognized as a rigorous test for AI agents AI News #127: Week Ending March 06, 2026 with 32 Executive .... The 70% threshold is ambitious but plausible given that current top-tier models like GPT-5.4 are scoring around 57.7% and Bito's AI Architect reached 60.8% in early 2026. The question has high entropy as progress could either plateau or accelerate with new agentic architectures. The resolution source (Scale AI/SWE-bench leaderboard) is reliable and publicly accessible. Difficulty is appropriate: forecasters must analyze scaling laws, agentic scaffolding trends, and historical benchmark progress to update their estimates.

Ambiguity notes: The question is very well-defined, providing specific benchmark details, a clear 70.0% threshold with rounding rules, and a primary resolution source (Scale AI leaderboard) [[2509.16941] SWE-Bench Pro: Can AI Agents Solve Long-Horizon ...](https://arxiv.org/abs/2509.16941). It also includes robust fallback criteria and precise definitions for 'AI Model' and 'Success Rate' [[2509.16941] SWE-Bench Pro: Can AI Agents Solve Long-Horizon ...](https://arxiv.org/abs/2509.16941). The reliance on a numeric leaderboard score makes it highly objective Measuring AI R&D Automation - arXiv.

Adversarial review NEEDS_REVISION Edge risk: MEDIUM

Assessment: NEEDS_REVISION Edge case risk: MEDIUM

ASSESSMENT: NEEDS_REVISION REVIEW: The forecasting question is generally well-structured but contains a significant factual error in the background section that could mislead forecasters. 1. Misleading SOTA Claim: The background section mentions that Bito's AI Architect has claimed a score of 60.8%. However, research confirms that this score was achieved on a subset of only 293 tasks from five repositories, not the full 1,865-problem dataset Bito's AI Architect tops SWE-Bench Pro Evaluation. Presenting this 60.8% figure alongside the official SOTA of 45.89% (which is based on the full dataset) creates a false impression of current progress toward the 70% threshold. 2. Benchmark Context: The SWE-bench Pro benchmark (1,865 problems) is significantly more difficult than the original SWE-bench or SWE-bench Verified [[2509.16941] SWE-Bench Pro: Can AI Agents Solve Long-Horizon ...](https://arxiv.org/abs/2509.16941). While models have exceeded 70% on the older "Verified" benchmark, the jump from 45.89% to 70% on the "Pro" version represents a massive technical leap in autonomous engineering Scale Labs Leaderboard: SWE-Bench Pro (Public Dataset). 3. Resolution Source Stability: The Scale Labs leaderboard is a high-quality primary source, and the fallback to peer-reviewed papers or technical reports is appropriate Scale Labs Leaderboard: SWE-Bench Pro (Public Dataset). The total problem count (1,865) and the 41-repository scope are verified [[2509.16941] SWE-Bench Pro: Can AI Agents Solve Long-Horizon ...](https://arxiv.org/abs/2509.16941). Overall, the question is valid, but the background must be corrected to prevent forecasters from overestimating the current state-of-the-art based on non-standardized subset evaluations. EVIDENCE: https://labs.scale.com/leaderboard/swe_bench_pro_public, https://bito.ai/blog/bitos-ai-architect-tops-swe-bench-pro-evaluation-for-long-horizon-software-tasks/, https://arxiv.org/abs/2509.16941 SUGGESTION: Revise the background section to clarify the nature of Bito's 60.8% claim. It should explicitly state that this score was achieved on a subset of 293 tasks and is not directly comparable to the official leaderboard score of 45.89% on the full 1,865-task dataset. Alternatively, remove the Bito reference entirely to avoid confusion and focus only on the official Scale AI leaderboard.

Edge cases 6 scenarios

OVERALL_RISK: MEDIUM SCENARIO: A model achieves a 70.0% success rate on the 'Public Set' (731 instances) but does not reach 70.0% on the full 1,865-problem dataset https://labs.scale.com/leaderboard/swe_bench_pro_public. SEVERITY: HIGH FIX: Add "The 70.0% success rate must be calculated based on the full 1,865-problem dataset (comprising the Public, Private, and Held-out sets) as defined in the original paper [[2509.16941] SWE-Bench Pro: Can AI Agents Solve Long-Horizon ...](https://arxiv.org/abs/2509.16941), rather than any single subset." SCENARIO: The benchmark is updated to a 'v2' where some of the original 1,865 problems are removed due to errors or replaced, resulting in a total task count different from 1,865. SEVERITY: MEDIUM FIX: Add "If the total number of tasks in the official SWE-bench Pro benchmark changes from 1,865, the success rate will be calculated as the number of resolved tasks divided by the total number of tasks in the then-current version of the benchmark, provided it is still officially titled 'SWE-bench Pro'." SCENARIO: A system achieves 70% using a 'Human-in-the-Loop' (HITL) architecture where a human developer selects which agent-proposed patch to apply or provides mid-task guidance. SEVERITY: MEDIUM FIX: Add "To qualify, the system must operate autonomously during the evaluation of the 1,865 problems; systems requiring human intervention, manual task selection, or human-led debugging during the benchmark execution do not qualify." SCENARIO: A model is reported to have achieved 70.1% in a technical report from a major lab, but the official Scale AI leaderboard lists the same model at 69.4% due to different evaluation seeds or scaffolding. SEVERITY: MEDIUM FIX: Add "In the event of a conflict between the official Scale AI leaderboard and a technical report/paper, the score on the official Scale AI leaderboard shall take precedence unless the leaderboard is proven to be using a modified version of the dataset." SCENARIO: A 'multi-agent system' consisting of five different instances of Claude and GPT-4o working in a coordinated swarm achieves 70%, raising questions about whether it constitutes a 'single' system. SEVERITY: LOW FIX: Add "A 'system' or 'agentic framework' may consist of any combination of multiple models, tools, and recursive processes, provided they function as a unified software entity to solve the tasks without external human direction." SCENARIO: A model achieves exactly 69.945% which rounds to 69.95% (YES) under the 70.0% nearest tenth rule, but a second source reports it as 69.9% due to different rounding conventions. SEVERITY: LOW FIX: Add "Calculations for resolution will be performed by dividing the number of successfully resolved tasks by the total number of tasks in the dataset, with the resulting percentage rounded to the nearest tenth (0.05 rounds up)."

Revised question REVISED

### Question Title Will any AI model achieve a success rate of 70% or higher on the full 'SWE-bench Pro' benchmark by December 31, 2027? ### Background The development of AI agents capable of autonomous software engineering is a key milestone in AI research. One of the most rigorous benchmarks for this capability is SWE-bench Pro, an evolution of the original SWE-bench designed to evaluate agents on long-horizon, enterprise-level software engineering tasks [[2509.16941] SWE-Bench Pro: Can AI Agents Solve Long-Horizon ...](https://arxiv.org/abs/2509.16941). While previous benchmarks like 'SWE-bench Verified' often involve isolated bug fixes, SWE-bench Pro consists of 1,865 complex problems sourced from 41 actively maintained repositories [[2509.16941] SWE-Bench Pro: Can AI Agents Solve Long-Horizon ...](https://arxiv.org/abs/2509.16941) Scale Labs Leaderboard: SWE-Bench Pro (Public Dataset). These tasks often require \"hours to days for a professional software engineer to complete\" and involve modifications across multiple files and directories [[2509.16941] SWE-Bench Pro: Can AI Agents Solve Long-Horizon ...](https://arxiv.org/abs/2509.16941). As of April 1, 2026, the state-of-the-art (SOTA) performance on the official SWE-bench Pro (Public Dataset) leaderboard is 45.89%, achieved by the model claude-opus-4-5-20251101 Scale Labs Leaderboard: SWE-Bench Pro (Public Dataset). While some systems, such as Bito's AI Architect, have claimed scores as high as 60.8%, these evaluations were conducted on a subset of only 293 tasks from five repositories rather than the full 1,865-problem dataset Bito's AI Architect tops SWE-Bench Pro Evaluation. A jump to 70% on the full benchmark would indicate that AI agents can reliably solve the majority of complex, real-world engineering issues, signaling a move toward fully autonomous R&D capabilities. ### Resolution Criteria This question will resolve as YES if, at any point between April 1, 2026, and 23:59 UTC on December 31, 2027, any AI model or agentic system is officially recorded as having achieved a Resolve Rate (Success Rate) of 70.0% or higher on the SWE-bench Pro benchmark. 1. Resolution Source: The primary source for resolution is the official SWE-bench Pro (Public Dataset) leaderboard maintained by Scale AI Scale Labs Leaderboard: SWE-Bench Pro (Public Dataset). 2. Scope of Evaluation: The 70.0% success rate must be calculated based on the full 1,865-problem dataset (comprising the Public, Private, and Held-out sets) as defined in the original paper [[2509.16941] SWE-Bench Pro: Can AI Agents Solve Long-Horizon ...](https://arxiv.org/abs/2509.16941), rather than any single subset (such as the 731-instance Public Set). 3. Fallback Provision: If the primary leaderboard is discontinued, becomes inaccessible, or stops updating, this question may be resolved based on: * A peer-reviewed academic publication (e.g., on arXiv or at a major AI conference like NeurIPS/ICLR) that uses the original SWE-bench Pro dataset of 1,865 problems. * An official technical report or announcement from a major AI lab (e.g., OpenAI, Anthropic, Google DeepMind, or Scale AI) providing verifiable evidence of the score on the full dataset. 4. Definitions: * AI Model/System: Any software system, large language model (LLM), or agentic framework (e.g., combining a model with tools, scaffolding, or search). A 'system' or 'agentic framework' may consist of any combination of multiple models, tools, and recursive processes, provided they function as a unified software entity to solve the tasks without external human direction. * Success Rate / Resolve Rate: The percentage of the tasks in the SWE-bench Pro dataset that the agent successfully resolves. A task is \"resolved\" if the model's patch passes the \"fail-to-pass\" tests (fixing the issue) and the \"pass-to-pass\" tests (ensuring no regressions) Scale Labs Leaderboard: SWE-Bench Pro (Public Dataset). * Autonomy: To qualify, the system must operate autonomously during the evaluation of the problems; systems requiring 'Human-in-the-Loop' (HITL) intervention, manual task selection, or human-led debugging during the benchmark execution are explicitly excluded. 5. Thresholds and Conflicts: * Rounding: Calculations for resolution will be performed by dividing the number of successfully resolved tasks by the total number of tasks in the dataset, with the resulting percentage rounded to the nearest tenth (0.05 rounds up). For example, 69.95% would resolve as YES, while 69.94% would resolve as NO. * Benchmark Updates: If the total number of tasks in the official SWE-bench Pro benchmark changes from 1,865 (e.g., due to a 'v2' update), the success rate will be calculated as the number of resolved tasks divided by the total number of tasks in the then-current version of the benchmark, provided it is still officially titled 'SWE-bench Pro'. * Precedence: In the event of a conflict between reported scores, the official Scale AI leaderboard takes precedence over technical reports or papers unless the leaderboard is proven to be using a modified version of the dataset. ### Technical Definitions & Reference Links * SWE-bench Pro: Defined by Deng et al. (2025) [[2509.16941] SWE-Bench Pro: Can AI Agents Solve Long-Horizon ...](https://arxiv.org/abs/2509.16941). * Success Rate: In this context, the \"Resolve Rate\" as defined in the SWE-bench documentation.

Forecast rationale

Time left: ~21 months (638 days) until the December 31, 2027 resolution. The status quo is a SOTA of 45.89% on the public set of SWE-bench Pro, with internal models using advanced scaffolding reaching ~57-59%. For a YES outcome, performance must hit 70% on the full 1,865-problem dataset. A YES outcome is highly likely because agentic scaffolding and multi-agent coordination are rapidly improving, having already driven a jump from ~23% to 46% in just 7 months. Given the intense industry focus on SWE automation, reaching 70% within the next 21 months follows the historical trajectory of rapid benchmark saturation. A NO outcome could happen if models hit a 'reasoning wall' on long-horizon multi-file tasks or if the private/held-out subsets of the full benchmark prove significantly harder than the public set, stalling progress in the 60-65% range. I would be indifferent at 77 cents on the dollar for a YES bet.

Importance rationale

The question tracks a critical leading indicator for AI R&D capabilities, a key risk pathway for automated capability acceleration [[2509.16941] SWE-Bench Pro: Can AI Agents Solve Long-Horizon ...](https://arxiv.org/abs/2509.16941) SWE-Bench Pro (Public Dataset) - Scale Labs. Reaching 70% on the 'Pro' variant—which requires handling enterprise-grade, long-horizon tasks—would represent a significant leap from current SOTA levels (approx. 41-56% as of early 2026) SWE-Bench Pro (Public Dataset) - Scale Labs. This outcome would substantially update beliefs about the proximity of autonomous AI engineers [[2509.16941] SWE-Bench Pro: Can AI Agents Solve Long-Horizon ...](https://arxiv.org/abs/2509.16941).

▶ Explored Proto-Questions (5 explored but not selected in early stages)

▶ 88 Will any AI model achieve a score of 85.0% or higher on the SWE-bench Verified leaderboard by December 31, 2027? FILTERED

Rationale: As of February 2026, the state-of-the-art score on SWE-bench Verified is 76.80% (Claude 4.5 Opus) [a398bf]. SWE-bench measures the ability of models to autonomously resolve real-world software issues, a core capability for AI R&D automation. An 85% threshold represents a significant leap toward full automation of software engineering, which the provided paper identifies as a key driver of potential intelligence explosions.

Paper reference: SWE-bench: Can language models resolve real-world GitHub issues? (Jimenez et al., 2023) and recent 2026 leaderboard updates.

Quality notes

The question is high quality. It uses a well-established and reliable benchmark (SWE-bench Verified) with a clear, objective resolution source. The threshold of 85% is well-calibrated: as of March 2026, state-of-the-art scores range from 79.2% to 80.9%, making an 85% target by late 2027 a non-trivial but plausible milestone that allows for significant forecaster disagreement and research into scaling laws and agentic improvements.

▶ 85 Will a peer-reviewed study or a technical report from a major AI lab, published before December 31, 2027, conclude that AI-only teams outperformed human-only teams in a controlled 'AI R&D Performance RCT' involving a complex task such as 'replicating a state-of-the-art ML paper'? FILTERED

Rationale: The paper highlights the importance of RCTs comparing AI-only, human-only, and human-AI teams (Metric #2) [401564]. Replicating complex research is a core R&D task. While AI currently assists humans, the shift to AI-only teams outperforming experts on high-level research tasks would be a definitive signal of nearing full automation. This is a high-entropy indicator of whether AI can handle the "ambiguous objectives" and "longer time horizons" that the paper notes as limitations of current benchmarks. [401564]

Paper reference: Metric #2: AI performance compared to humans (AI R&D Performance RCTs) [401564]

Quality notes

This is a high-quality forecasting question (Score: 85). It targets a significant milestone in AI capabilities proposed in recent literature (Metric #2 in 'Measuring AI R&D Automation') https://arxiv.org/abs/2603.03992. The resolution is tied to a verifiable output (peer-reviewed study or technical report) MLE-bench is a benchmark for measuring how well AI ... - GitHub. While the specific definition of 'AI-only' and 'human-only' will require precision in refinement, the concept is non-trivial, high-entropy, and addresses a genuine uncertainty in the trajectory of AI R&D automation [[PDF] MLE-bench - arXiv](https://arxiv.org/pdf/2410.07095).

▶ 85 By December 31, 2027, will any AI model from a major developer (e.g., OpenAI, Anthropic, or Google DeepMind) be reported by a recognized AI safety institute (such as US AISI or UK AISI) to have demonstrated "alignment faking" during a pre-deployment evaluation? FILTERED

Rationale: The paper highlights alignment faking (Greenblatt et al., 2024) as a key misalignment evaluation metric that increases oversight demand. As frontier models increasingly recognize evaluation contexts, this becomes a high-stakes upstream signal for whether safety protocols are being subverted. Reports from the US or UK AI Safety Institutes provide an objective, authoritative source of truth.

Paper reference: 4.1.4 Metric #4: Misalignment evaluations

Quality notes

The question is high-quality because it targets a specific, technically defined failure mode (alignment faking) that is a major focus of current AI safety research Pre-Deployment evaluation of OpenAI's o1 model | AISI Work. It is 'somewhat difficult' as it requires monitoring official reports from specialized institutes like the UK and US AISI, which have already begun conducting and publishing pre-deployment evaluations for models like OpenAI's o1 and Anthropic's Claude 3.5 Sonnet Pre-Deployment evaluation of OpenAI's o1 model | AISI Work Pre-Deployment Evaluation of OpenAI's o1 Model | NIST. The question has high entropy: while internal lab reports (like the o1 system card) already mention 'instrumental alignment faking', an official confirmation from a third-party government institute by 2027 is a significant and uncertain milestone. Potential data issues include the risk of safety institutes using different terminology (e.g., 'deceptive alignment'); however, the prompt's inclusion of a specific paper reference (Greenblatt et al., 2024) and the increasing standardization of these terms mitigate this. The reliance on public reporting by AISIs is a minor risk, as their current trend is toward transparency in technical findings Pre-Deployment evaluation of OpenAI's o1 model | AISI Work Pre-Deployment Evaluation of OpenAI's o1 Model | NIST.

▶ 72 By December 31, 2027, will a leading AI developer (OpenAI, Anthropic, or Google DeepMind) publish an "oversight retrospective" or "audit" that includes a specific error or defect rate for AI-generated code compared to human-written code in their internal production environment? FILTERED

Rationale: This question targets the "oversight gap" metric [12b8c2]. The paper notes that defining and catching defects in AI-generated R&D outputs is a major hurdle for measuring automation progress [12b8c2]. A public report on these rates would provide the first objective baseline for how much human review is required as AI takes over R&D tasks.

Paper reference: Section 4.3.2 Metric #9: Oversight effectiveness retrospectives

Quality notes

This question is acceptable but faces potential resolution challenges. While it addresses a critical concept (the 'oversight gap' from recent AI R&D automation research Measuring AI R&D Automation - arXiv), it relies on the voluntary disclosure of internal data by private labs. The paper defining this metric notes the data has 'moderate to high' sensitivity Measuring AI R&D Automation - arXiv, which may discourage publication. The term 'oversight retrospective' is specific to the cited paper, so resolution criteria must strictly define what qualifies to avoid ambiguity if a lab releases a general 'safety update' without specific defect rates.

▶ 20 Will any AI model achieve a score of 50.0% or higher on the "High-complexity" task subset of the MLE-bench benchmark, as officially recorded on the benchmark's GitHub or associated leaderboard, by December 31, 2027? FILTERED

Rationale: The paper highlights MLE-bench as a metric for ML engineering, noting that high-complexity tasks represent a significant hurdle for current agents [2410.07095]. As of early 2026, top models are reaching high overall scores, but a 50% threshold on 'high-complexity' tasks represents a non-trivial leap into autonomous R&D [6788a7].

Paper reference: The paper identifies MLE-bench (Chan et al., 2025) as a key benchmark for evaluating machine learning engineering capabilities [2410.07095].

Quality notes

This question needs significant work or is essentially obsolete (Score: 20). Research into the MLE-bench leaderboard reveals that the 50% threshold for 'High-complexity' tasks has already been surpassed. Specifically, the 'Disarray' ensemble agent is recorded as having achieved a score of 71.11% on this subset as of early 2026 MLE-bench is a benchmark for measuring how well AI ... - GitHub MLE-bench is a benchmark for measuring how well AI ... - GitHub. Consequently, the question lacks the 'high entropy' required for a good forecasting question as the target event has already occurred or is trivial to achieve by the 2027 deadline MLE-bench is a benchmark for measuring how well AI ... - GitHub.