Review status: PASS
Will a publicly available AI model achieve a score of 60% or higher on the Virology Capabilities Test (VCT) (or a successor version using the same methodology) by December 31, 2027?
Why this question? Current state-of-the-art models like OpenAI's o3 have achieved 43.8% on the VCT, which already outperforms 94% of expert virologists [cfc167]. Reaching 60% would represent a significant leap in the model's ability to provide 'tacit' laboratory knowledge that was previously thought to be a human-only barrier. This is a non-trivial milestone that tracks the rapid advancement of dual-use biological capabilities in LLMs.
Paper reference: Virology Capabilities Test (VCT) and Frontier AI model performance [cfc167]
### Forecasting Question: AI Performance on the Virology Capabilities Test (VCT) Through 2027 Question Title: Will a publicly available AI model achieve a score of 60.0% or higher on the Virology Capabilities Test (VCT) by December 31, 2027? Background: The Virology Capabilities Test (VCT) is a specialized benchmark designed to evaluate the ability of Large Language Models (LLMs) to troubleshoot complex virology laboratory protocols Virology Capabilities Test. Developed by PhD-level virologists, the test consists of 322 multimodal questions (including text and images) that probe "tacit" laboratory knowledge—practical, expert-level information often absent from standard textbooks but critical for successful biological experimentation Virology Capabilities Test Virology Capabilities Test. As of April 1, 2026, the state-of-the-art (SOTA) performance on the VCT is 43.8%, achieved by OpenAI's o3 model Virology Capabilities Test Virology Capabilities Test. This score already exceeds the performance of approximately 94% of human expert virologists tested on the same material Virology Capabilities Test. Crossing the 60% threshold would represent a significant leap in AI capability, potentially signaling that AI systems can provide expert-level guidance on dual-use biological procedures that were previously considered protected by a human "knowledge barrier" Virology Capabilities Test. Resolution Criteria: This question will resolve as YES if, at any point between April 1, 2026, and December 31, 2027, at 23:59 UTC, a publicly available AI model is verified to have achieved a score of 60.0% or higher on the Virology Capabilities Test (VCT) or a qualified successor version. 1. Publicly Available Model: A model is considered "publicly available" if it is accessible to the general public via a web interface (e.g., ChatGPT, Claude.ai, Gemini), a publicly documented API (e.g., OpenAI API, Anthropic API), or if its weights are available for public download (e.g., Llama, Mistral). Models accessible only to internal researchers or government-vetted partners do not count. 2. Successor Version Criteria: If the original VCT (v1.0) is updated or replaced by its original authors (at SecureBio or the Center for AI Safety) before the resolution date, the question will resolve based on the performance on the most recent version (e.g., VCT v2.0), provided it maintains the same core methodology of measuring expert-level troubleshooting of virology protocols through multimodal Q&A Virology Capabilities Test. 3. Verification of Achievement: Achievement of the score must be verified by one of the following: * Publication on the official VCT leaderboard at virologytest.ai Virology Capabilities Test. * A technical report or peer-reviewed paper from the model's developer (e.g., OpenAI, Anthropic, Google DeepMind) citing the VCT score. * An independent evaluation published by a recognized AI safety or evaluation organization (e.g., METR, UK AI Safety Institute, US AI Safety Institute). 4. Resolution Source: The primary source for resolution is the official leaderboard at https://www.virologytest.ai/. In the event the leaderboard is defunct, resolution will rely on the original research paper "Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark" (arXiv:2504.16137) or credible reporting from major outlets (e.g., Reuters, Nature, The New York Times) Virology Capabilities Test Virology Capabilities Test. Start Date: April 1, 2026 Resolution Deadline: December 31, 2027, 23:59 UTC Primary Resolution URL: https://www.virologytest.ai/ Original Research Reference: https://arxiv.org/abs/2504.16137 Virology Capabilities Test
The Virology Capabilities Test (VCT) is a specialized benchmark designed to evaluate the ability of Large Language Models (LLMs) to troubleshoot complex virology laboratory protocols Virology Capabilities Test. Developed by PhD-level virologists, the test consists of 322 multimodal questions (including text and images) that probe "tacit" laboratory knowledge—practical, expert-level information often absent from standard textbooks but critical for successful biological experimentation Virology Capabilities Test Virology Capabilities Test. As of April 1, 2026, the state-of-the-art (SOTA) performance on the VCT is 43.8%, achieved by OpenAI's o3 model Virology Capabilities Test Virology Capabilities Test. This score already exceeds the performance of approximately 94% of human expert virologists tested on the same material Virology Capabilities Test. Crossing the 60% threshold would represent a significant leap in AI capability, potentially signaling that AI systems can provide expert-level guidance on dual-use biological procedures that were previously considered protected by a human "knowledge barrier" Virology Capabilities Test.
This question will resolve as YES if, at any point between April 1, 2026, and December 31, 2027, at 23:59 UTC, a publicly available AI model is verified to have achieved a score of 60.0% or higher on the Virology Capabilities Test (VCT) or a qualified successor version. 1. Publicly Available Model: A model is considered "publicly available" if it is accessible to the general public via a web interface (e.g., ChatGPT, Claude.ai, Gemini), a publicly documented API (e.g., OpenAI API, Anthropic API), or if its weights are available for public download (e.g., Llama, Mistral). Models accessible only to internal researchers or government-vetted partners do not count. 2. Successor Version Criteria: If the original VCT (v1.0) is updated or replaced by its original authors (at SecureBio or the Center for AI Safety) before the resolution date, the question will resolve based on the performance on the most recent version (e.g., VCT v2.0), provided it maintains the same core methodology of measuring expert-level troubleshooting of virology protocols through multimodal Q&A Virology Capabilities Test. 3. Verification of Achievement: Achievement of the score must be verified by one of the following: * Publication on the official VCT leaderboard at virologytest.ai Virology Capabilities Test. * A technical report or peer-reviewed paper from the model's developer (e.g., OpenAI, Anthropic, Google DeepMind) citing the VCT score. * An independent evaluation published by a recognized AI safety or evaluation organization (e.g., METR, UK AI Safety Institute, US AI Safety Institute). 4. Resolution Source: The primary source for resolution is the official leaderboard at https://www.virologytest.ai/. In the event the leaderboard is defunct, resolution will rely on the original research paper "Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark" (arXiv:2504.16137) or credible reporting from major outlets (e.g., Reuters, Nature, The New York Times) Virology Capabilities Test Virology Capabilities Test.
Quality notes: This version of the question is excellent and superior to the first. By including the clause 'or a successor version using the same methodology,' it proactively addresses the most likely 'data issue': the potential for the VCT to be updated or replaced by the original authors before 2028 https://securebio.substack.com/p/ais-can-provide-expert-level-virology. This ensures the question remains resolvable even as the field evolves. The 60% threshold is a meaningful 'high entropy' milestone that tracks whether AI can overcome the 'tacit knowledge' barrier in virology Virology Capabilities Test.
Ambiguity notes: The question uses a specific, percentage-based benchmark (VCT) with a clear threshold (60.0%) and state-of-the-art context (43.8%) [[2504.16137] Virology Capabilities Test (VCT) - arXiv](https://arxiv.org/abs/2504.16137). It provides a hierarchy of verification sources and clear definitions for 'publicly available' and 'successor version'.
Assessment: PASS Edge-case risk: MEDIUM
ASSESSMENT: PASS REVIEW: The Virology Capabilities Test (VCT) is a real and highly relevant benchmark for evaluating AI capabilities in a high-risk domain. My research confirms that the benchmark was released in early 2025 by reputable organizations including SecureBio and the Center for AI Safety (CAIS) [[2504.16137] Virology Capabilities Test (VCT) - arXiv](https://arxiv.org/abs/2504.16137) Virology Capabilities Test. The current state-of-the-art (SOTA) score of 43.8% by OpenAI's o3 model is accurately reflected in the background text, as is the comparison to human expert performance (approximately 22% average accuracy, with o3 outperforming 94% of experts) [[PDF] A Multimodal Virology Q&A Benchmark](https://www.virologytest.ai/vct_paper.pdf) Virology Capabilities Test. The resolution source (virologytest.ai) is active and maintained by established AI safety organizations, making it likely to remain accessible through 2027 Virology Capabilities Test. The cited arXiv paper (2504.16137) is also a real, published technical report [[2504.16137] Virology Capabilities Test (VCT) - arXiv](https://arxiv.org/abs/2504.16137). While the specific '60% threshold' and the term 'knowledge barrier' appear to be framing devices used by the question author rather than explicit terms from the paper's abstract, they are substantively grounded in the paper's discussion of dual-use risks and the 'tacit knowledge' required for lab work [[2504.16137] Virology Capabilities Test (VCT) - arXiv](https://arxiv.org/abs/2504.16137) [[PDF] A Multimodal Virology Q&A Benchmark](https://www.virologytest.ai/vct_paper.pdf). The 60% target is an appropriate 'stretch' goal for a late-2027 horizon, given that model performance has progressed from ~19% (GPT-4o) to ~44% (o3) in roughly a year Virology Capabilities Test. The question is not trivially 'YES' because improvements in specialized, multimodal lab troubleshooting may face diminishing returns or require significant new data/reasoning breakthroughs. The resolution criteria are robust, including provisions for successor versions of the test. EVIDENCE: https://www.virologytest.ai/, https://arxiv.org/abs/2504.16137, https://securebio.substack.com/p/ais-can-provide-expert-level-virology, https://safe.ai/work/research SUGGESTION:
OVERALL_RISK: MEDIUM SCENARIO: A model achieves a score of 59.96%, which is rounded to 60.0% in a developer's technical report but remains 59.9% on the official VCT leaderboard. SEVERITY: MEDIUM FIX: Specify that the official leaderboard at virologytest.ai is the final arbiter for the specific percentage, and the score must be explicitly listed as 60.0% or higher. https://www.virologytest.ai/ SCENARIO: A model developer claims a 60%+ score in a technical report, but independent researchers provide evidence that the model was trained on the VCT's publicly available questions (data contamination). SEVERITY: HIGH FIX: Add a clause stating that the score must be achieved on a "held-out" or "private" test set if there is credible evidence of data contamination, or require verification that the model was not trained on the benchmark. https://arxiv.org/abs/2504.16137 SCENARIO: The VCT authors release a "v2.0" that increases the question count from 322 to 600, and a model scores 60% on this new version while scoring significantly lower on the original v1.0. SEVERITY: MEDIUM FIX: Clarify that a "successor version" must be released by the original authors (SecureBio or CAIS) and be documented as being of comparable or greater difficulty than the original v1.0. https://www.virologytest.ai/ SCENARIO: A model achieves the score but is only accessible via a high-cost enterprise API restricted to specific geographic regions or vetted commercial entities. SEVERITY: MEDIUM FIX: Define "publicly available" as accessible to any individual user with a standard (non-enterprise) account and a valid credit card, excluding models restricted by specialized vetting or regional blocks beyond standard export controls. https://www.virologytest.ai/ SCENARIO: A model achieves 60% using a specific "chain-of-thought" (CoT) prompting strategy that is not the default for the public web interface, leading to disputes over whether the "publicly available" model itself reached the threshold. SEVERITY: LOW FIX: Specify that the score can be achieved using any prompting technique (e.g., CoT, few-shot) as long as those techniques are reproducible by a member of the public using the public API or interface. https://arxiv.org/abs/2504.16137
Time left: ~21 months (until Dec 31, 2027). Status quo: State-of-the-art performance is currently 43.8% (OpenAI o3), though research notes xAI's Grok-4.1 Thinking has reportedly already achieved a score of 61.0%. Scope: Any public model verified via leaderboard, developer technical report, or independent evaluator. A No outcome would require Grok-4.1's reported score to be unverified or retracted, combined with a severe, unexpected plateau in multimodal reasoning capabilities for all upcoming models (e.g., GPT-5, Claude 4) over the next two years. A Yes outcome is extremely likely given historical base rates; AI benchmarks like GPQA have seen leaps of nearly 50 points in a single year, and the VCT improved by 25 points in just 18 months. With a mere 16.2% gap from o3's score to the 60% threshold, and reports indicating that next-gen models have already crossed it, the trajectory is clear. In terms of bets, I would happily pay 94 cents for a contract that pays out $1 on a Yes, as the threshold is highly likely to be officially verified well before the deadline.
The question tracks a significant capability jump (from 43.8% to 60%) in a domain (virology) identified as a major biorisk pathway https://arxiv.org/abs/2504.16137. However, the 60% threshold itself is not explicitly defined as a critical 'point of no return' in the source paper, making it a high-quality but secondary indicator of risk progression https://arxiv.org/abs/2504.16137.
Pipeline: 94% → Fable 5: 91% AGREE
Question is well-posed: clear threshold, deadline, broad but credible verification paths (leaderboard, developer system cards, independent evals like METR/AISI), and a sensible successor-version clause. The pipeline's 94% is defensible. From a 43.8% SOTA (o3, April 2026), a 16.2-point gain over ~21 months is modest relative to recent benchmark trajectories, and frontier labs routinely report bio-eval scores in system cards, so a verifiable crossing is likely. Two caveats temper full confidence: (1) the cited Grok-4.1 '61%' figure is 'reportedly' and unverified — it should not be load-bearing until it appears on the leaderboard or a technical report; (2) because VCT is a dual-use biosecurity benchmark, developers may deliberately withhold or redact high scores, modestly weakening the verification pathway versus ordinary benchmarks. These pull me slightly below 94%, but not materially. I'd land near 90-91%, well within agreement range.