arXiv:2409.12183Verified
Scaling test-time compute on contest mathematics
Search guided by a verifier at test time raises contest-math accuracy beyond what parameter count alone delivered on the same items.
2026-03-28Weight 94Demonstrated
Audited ledger
Fixed primary sources. Replay only reveals excerpts already in the desk — it does not crawl the web.
Replay on
Press / to search. Open a frontier to narrow the ledger.
Valmeekam et al., arXiv:2206.10498Verified
LLMs still fail classical planning off the memorized trace
On classical planning domains, models that look competent on familiar plan text fail once the instance is larger or the names are obfuscated.
2023-05-25Weight 93Benchmark
Liu et al., arXiv:2307.03172Verified
Relevant evidence is hardest to use from the middle of a long context
Readers — human and model — use a fact more reliably when it is placed at the start or end of a long input than when it is buried in the middle.
2023-07-06Weight 91Benchmark
Kirkpatrick et al., arXiv:1612.00796Verified
Elastic weight consolidation slows forgetting and does not end it
Protecting weights that mattered for older tasks reduces catastrophic forgetting on sequential supervised problems. The tradeoff remains.
2017-01-25Weight 86Demonstrated
ARC PrizeVerified
Public frontier models still miss the ARC-AGI human band
On novel grid transformations held out from training, published general models remain below the human baseline the prize uses.
2026-03-15Weight 96Replication
Chollet, arXiv:1911.01547Verified
Generality is skill-acquisition efficiency, not a saturated exam
A system is more intelligent when it turns a novel task into a skill with less experience, relative to its priors — not when it scores higher on tasks it has effectively seen.
2019-11-05Weight 90Hypothesis
Ha & Schmidhuber, arXiv:1803.10122Verified
A learned world model can plan inside its own dream — of one environment
An agent can train a compact dynamics model and plan inside that model on the games it was trained to predict.
2018-03-27Weight 84Demonstrated
Gemini 1.5 technical report, arXiv:2403.05530Developing
Native multimodal models post strong video scores under their own protocol
A long-context multimodal model records high marks on video understanding benchmarks when subtitles and native visual input are both available.
2024-03-08Weight 62Benchmark
RT-2, arXiv:2307.15818Verified
Web-scale vision-language priors transfer into a limited robot skill set
Co-training a robot policy with web vision-language data lets the robot follow novel instructions inside the family of skills it has physically demonstrated.
2023-07-28Weight 80Demonstrated
Unreleased vendor noteUnverified
Vendor claims a 74% autonomous resolution rate on internal tickets
The company says its system fully resolves 74% of enterprise software issues with no human in the loop.
2026-04-02Weight 38Vendor claim
SWE-bench, arXiv:2310.06770Verified
Resolving real repository issues remains a minority outcome
When the test is a real GitHub issue plus the project's own tests, early agent setups resolve only a minority of instances.
2023-10-10Weight 92Benchmark
AutoGen, arXiv:2308.08155Developing
Multi-agent chat can split a task and still agree on a mistake
A conversational framework can assign roles and finish some coding and tool tasks. It does not supply a check against loops or shared error.
2023-08-16Weight 57Hypothesis
AlphaFold 2, Nature 2021Verified
Protein structure prediction is not autonomous scientific discovery
A model can predict many protein structures at experimental accuracy. That is a new measurement tool, not a closed loop of hypothesis, experiment, and revised theory.
2021-07-15Weight 95Demonstrated
Snell et al., arXiv:2408.03314Verified
Test-time compute can beat a larger model when a verifier exists
Allocating extra search at inference, rather than extra parameters, improves math and coding accuracy on the distributions where proposals can be checked.
2024-08-06Weight 88Benchmark
Shumailov et al., Nature 2024Verified
Training on a model's own outputs collapses the tail
When data generated by a model is used to train the next one, repeatedly, rare events disappear and the distribution concentrates on the mode.
2024-07-24Weight 94Benchmark
Anthropic, Scaling MonosemanticityVerified
Sparse autoencoders recover more features as models get wider
Sparse dictionaries find features in a production-scale language model that are more monosemantic than individual neurons, and the dictionary scales with width.
2024-05-21Weight 83Demonstrated
Morris et al., arXiv:2311.02462Verified
Levels of AGI separate narrow skill from competent generality
The paper ranks systems from emerging to superhuman, and from narrow to general. Competent general performance — the 50th percentile of skilled adults across a wide range of tasks — is a level, not a single exam score.
2023-11-04Weight 90Hypothesis
Unreleased vendor deckDisputed
Private slide deck declares an internal suite 'AGI-complete'
A slide states that an internal composite has crossed an AGI threshold.
2026-04-09Weight 12Vendor claim
Conditioned evaluations
These scores do not rank models. Different benchmarks, splits, and decoding regimes stay in their own rows.
| Model | Benchmark | Score | Who ran it | Conditions |
|---|---|---|---|---|
Claude 3.5 Sonnet Anthropic · 20241022 | SWE-bench Verified | 49.0% Resolve rate | Independent Academic No leakage flagged | Verified split, isolated container, pass@1 |
OpenAI o1 OpenAI · 2024-12-17 | AIME 2024 | 83.3% Accuracy | Vendor Self-Report No leakage flagged | Consensus@64 test-time search, private holdout |
DeepSeek R1 DeepSeek · 2025 release | MATH-500 | 97.3% Pass@1 | Reproducibility Collective No leakage flagged | Open weights, greedy decoding as reported by the collective |
Gemini 1.5 Pro Google DeepMind · 002 | Video-MME | 81.3% Total score | Vendor Self-Report No leakage flagged | Subtitles included, native multimodal input |
GPT-4 OpenAI · 0314 | MMLU | 86.4% Accuracy | Vendor Self-Report Leakage not ruled out | 5-shot, original static split |