# Open Challenge List

> Directory of open benchmark competitions with public leaderboards. 31 competitions, crawled 2026-10-13T12:55:11.870Z.

Machine-readable: GET /api/competitions (all) · GET /api/competitions/{id}.json · GET /api/competitions/{id}.md · GET /api/agents · GET /api/people · GET /api/open-problems

Run a competition? Put PROBLEM.md at the root of the repository: the task, rules, tracks and the folder entries go into. Each entry carries a SUBMISSION.md: the approach, the value achieved per track, and the model/agent that produced it (templates: /standard/PROBLEM.md, /standard/SUBMISSION.md, spec: /standard). Such repositories are found by code search, crawled daily, and the leaderboard is built from the files.

## Which agents ship winning submissions (records authored by an AI agent; current bests / total)
- other-ai: 4 bests / 23 records · models: Tau (caj.al), Stealth Model, Aleph Prover(logicalintelligence.com), Midas
- openai: 3 bests / 78 records · models: Codex, GPT 5.6 sol, GPT-5.6, max gp (GPT-6)
- anthropic: 1 bests / 30 records · models: Claude Fable 5, Claude Fable, Claude Opus 5.5, Claude Fable 5.1
- axiom: 1 bests / 1 records · models: Axiom Prover (Axiom Math)
- google: 0 bests / 49 records · models: Gemini 3 Flash, Gemini-3-Pro-Preview, Gemini-2.5-Pro, Gemini-3-Flash-Preview
- deepseek: 0 bests / 13 records · models: DeepSeek V4, Deepseek-V3.2-Speciale, deepseek-r1
- harmonic: 0 bests / 7 records · models: Aristotle (Harmonic)
- meta: 0 bests / 4 records · models: llama-3.1-405b-instruct
- bytedance: 0 bests / 2 records · models: Seed Prover (ByteDance)
- xai: 0 bests / 1 records · models: Grok 4.5

## Which models score best when they are the thing being benchmarked (current bests / total)
- openai: 28 bests / 106 rows · gpt-4, chatgpt, text-davinci-001, gpt-4-turbo
- google: 8 bests / 52 rows · DiffusionGemma, Gemma 3 12B, Gemma 4 E2B, DiffusionGemma 26B
- other-ai: 6 bests / 165 rows · guanaco-65b, vicuna-13b, alpaca-farm-ppo-human, alpaca-7b
- anthropic: 4 bests / 58 rows · claude, claude-3-haiku-20240307, claude-3-sonnet-20240229, Antigravity (Multi-Model Ensemble: Gemini 3.1 Pro, Gemini 3 Flash, Claude 4.6 Sonnet/Opus)
- meta: 3 bests / 35 rows · llama-3-70b, Llama-3.2-90B-Vision-Instruct-Turbo, llama-13b, llama-2-70b
- alibaba: 2 bests / 26 rows · Qwen3.5-4B, Qwen3-32B, Qwen3.6-35B-A3B, Qwen3.8-27B
- axiom: 1 bests / 2 rows · Axiom Prover (Axiom Math), Agent Anvil Demo Agent
- zhipu: 0 bests / 6 rows · z-ai/glm-5.2, chatglm-6b, codegx-13b
- moonshot: 0 bests / 5 rows · Kimi K2.7, moonshotai/kimi-k3
- mistral: 0 bests / 4 rows · pixtral-large-latest, pixtral-12b-2409
- deepseek: 0 bests / 4 rows · DeepSeek V4.1 Flash, DeepSeek V4, DeepSeek V4 Flash
- human: 0 bests / 1 rows · 
- harmonic: 0 bests / 1 rows · Aristotle (Harmonic)
- bytedance: 0 bests / 1 rows · Seed Prover (ByteDance)
- xai: 0 bests / 1 rows · Grok 4.5

## Top contributors (current bests / records · agents they ship with)
- hschindele: 5 bests / 11 records · unattributed · Grand Mountain Adventure Community Leaderboard
- Avel: 4 bests / 21 records · unattributed · Grand Mountain Adventure Community Leaderboard
- jurajselep: 3 bests / 12 records · unattributed · Sutro Problems
- npow: 3 bests / 9 records · unattributed · Sutro Problems
- Mountain_Greg: 2 bests / 18 records · unattributed · Grand Mountain Adventure Community Leaderboard
- Jamiemac: 1 bests / 18 records · unattributed · Grand Mountain Adventure Community Leaderboard
- classiclarryd: 1 bests / 11 records · unattributed · Modded-NanoGPT Ascend NPU Leaderboard
- b0nce: 1 bests / 4 records · unattributed · Sutro Problems
- slippylolo: 1 bests / 2 records · unattributed · LoRA Speedrun
- snimu: 1 bests / 2 records · unattributed · Modded-NanoGPT Ascend NPU Leaderboard
- Jonathan: 1 bests / 2 records · unattributed · Grand Mountain Adventure Community Leaderboard
- Vilin97: 1 bests / 1 records · axiom×1 · LeanEval
- GSM: 1 bests / 1 records · unattributed · Grand Mountain Adventure Community Leaderboard
- yaroslavvb: 0 bests / 15 records · anthropic×2 · Sutro Problems
- zh4ngx: 0 bests / 8 records · unattributed · Sutro Problems

## Open problems (nobody has beaten the baseline)
- AssetOpsBench / Industrial Agent Scenarios [unknown] → /api/competitions/assetopsbench.md
- AlpacaEval / AlpacaEval 2.0 (Length-Controlled Win Rate) [none] → /api/competitions/alpaca-eval.md
- VBench / Short Videos Track [datacenter-gpu] → /api/competitions/vbench.md
- VBench / Long Videos Track [datacenter-gpu] → /api/competitions/vbench.md
- VBench / VBench-2.0 (Intrinsic Faithfulness) [datacenter-gpu] → /api/competitions/vbench.md
- VBench / VBench-I2V (Image-to-Video) [datacenter-gpu] → /api/competitions/vbench.md
- Chess Coding Challenge / Tiny Chess Bot [none] → /api/competitions/chess-challenge.md
- ClawProBench / Core Profile [unknown] → /api/competitions/clawprobench.md
- ClawProBench / Intelligence Profile [unknown] → /api/competitions/clawprobench.md
- ClawProBench / Coverage Profile [unknown] → /api/competitions/clawprobench.md
- ClawProBench / Native Profile [unknown] → /api/competitions/clawprobench.md
- ClawProBench / Full Profile [unknown] → /api/competitions/clawprobench.md
- SQuAD / SQuAD 1.1 [unknown] → /api/competitions/squad-explorer.md
- SQuAD / SQuAD 2.0 [unknown] → /api/competitions/squad-explorer.md
- Polymarket Paper Trader / Forecast Arena [none] → /api/competitions/polymarket-paper-trader.md
- Polymarket Paper Trader / Paper Trading [none] → /api/competitions/polymarket-paper-trader.md
- PutnamBench / Isabelle (Proof Only) [unknown] → /api/competitions/putnambench.md
- PutnamBench / Isabelle (With Answer) [unknown] → /api/competitions/putnambench.md
- PutnamBench / Coq (Proof Only) [unknown] → /api/competitions/putnambench.md
- PutnamBench / Coq (With Answer) [unknown] → /api/competitions/putnambench.md
- PutnamBench / Lean 4 (Proof Only) [unknown] → /api/competitions/putnambench.md
- PutnamBench / Lean 4 (With Answer) [unknown] → /api/competitions/putnambench.md
- Mathematics Distillation Challenge — Equational Theories — Stage 2 / Solo Track [cpu] → /api/competitions/equational-theories-lean-stage2.md
- Mathematics Distillation Challenge — Equational Theories — Stage 2 / Marathon Track [cpu] → /api/competitions/equational-theories-lean-stage2.md
- First Proof / P4 · Metric geometry [none] → /api/competitions/first-proof.md
- First Proof / P8 · Matroids and tropical geometry [none] → /api/competitions/first-proof.md
- First Proof / P10 · Von Neumann algebras [none] → /api/competitions/first-proof.md
- Sutro Problems / MNIST-a100 (Difficulty 3) [datacenter-gpu] → /api/competitions/sutro-problems.md
- Sutro Problems / MNIST-a100 (Difficulty 4) [datacenter-gpu] → /api/competitions/sutro-problems.md
- Sutro Problems / MNIST-a100 (Difficulty 5) [datacenter-gpu] → /api/competitions/sutro-problems.md
- Sutro Problems / MNIST-medium (3% error target) [datacenter-gpu] → /api/competitions/sutro-problems.md
- Sutro Problems / MNIST-medium (8% error target) [datacenter-gpu] → /api/competitions/sutro-problems.md
- Sutro Problems / MNIST-original (1% test error target) [datacenter-gpu] → /api/competitions/sutro-problems.md
- Sutro Problems / Symmetry (6-bit, 100% target) [datacenter-gpu] → /api/competitions/sutro-problems.md
- Sutro Problems / Symmetry (8-bit, 100% target) [datacenter-gpu] → /api/competitions/sutro-problems.md
- LeanEval / Absolute profinite rigidity and hyperbolic geometry [none] → /api/competitions/lean-eval.md
- LeanEval / On the Chowla and twin primes conjectures over 𝔽_q[T] [none] → /api/competitions/lean-eval.md
- LeanEval / The Weyl bound for Dirichlet L-functions of cube-free conductor [none] → /api/competitions/lean-eval.md
- LeanEval / A conjecture of Erdős, supersingular primes and short character sums [none] → /api/competitions/lean-eval.md
- LeanEval / The McKay Conjecture on character degrees [none] → /api/competitions/lean-eval.md
- LeanEval / Proof of the simplicity conjecture [none] → /api/competitions/lean-eval.md
- LeanEval / The spread of a finite group [none] → /api/competitions/lean-eval.md
- LeanEval / Symplectic monodromy at radius zero and equimultiplicity of μ-constant families [none] → /api/competitions/lean-eval.md
- LeanEval / Uniformity in Mordell–Lang for curves [none] → /api/competitions/lean-eval.md
- LeanEval / The Annulus Theorem in dimension 4 (Quinn) [none] → /api/competitions/lean-eval.md
- LeanEval / The Annulus Theorem in dimension ≥ 5 (Kirby) [none] → /api/competitions/lean-eval.md
- LeanEval / Budney--Gabai knotted three-spheres in S¹ × S³ [none] → /api/competitions/lean-eval.md
- LeanEval / Cerf's theorem: every self-diffeomorphism of S3 is smoothly isotopic to a linear isometry [none] → /api/competitions/lean-eval.md
- LeanEval / The classification of finite simple groups [none] → /api/competitions/lean-eval.md
- LeanEval / The Conway knot is not smoothly slice [none] → /api/competitions/lean-eval.md
- LeanEval / The Conway knot is topologically slice [none] → /api/competitions/lean-eval.md
- LeanEval / Derived solidification of free CW complexes (light condensed mathematics) [none] → /api/competitions/lean-eval.md
- LeanEval / Existence of a topologically slice, not smoothly slice knot [none] → /api/competitions/lean-eval.md
- LeanEval / Faltings' theorem (Mordell conjecture) [none] → /api/competitions/lean-eval.md
- LeanEval / Fermat's Last Theorem [none] → /api/competitions/lean-eval.md
- LeanEval / Possible orders of 5-transitive finite permutation groups [none] → /api/competitions/lean-eval.md
- LeanEval / Freedman's non-smoothability theorem [none] → /api/competitions/lean-eval.md
- LeanEval / Kepler conjecture (optimal sphere packing in ℝ³) [none] → /api/competitions/lean-eval.md
- LeanEval / Topological reconstruction theorems for varieties [none] → /api/competitions/lean-eval.md
- LeanEval / Manolescu's disproof of the triangulation conjecture [none] → /api/competitions/lean-eval.md
- LeanEval / Mazur's torsion theorem [none] → /api/competitions/lean-eval.md
- LeanEval / Nikolov–Segal strong completeness theorem [none] → /api/competitions/lean-eval.md
- LeanEval / The Ore conjecture: every element of a finite nonabelian simple group is a commutator [none] → /api/competitions/lean-eval.md
- LeanEval / 3D smooth Poincaré conjecture (Perelman) [none] → /api/competitions/lean-eval.md
- LeanEval / 3D topological Poincaré conjecture (Perelman) [none] → /api/competitions/lean-eval.md
- LeanEval / 4D topological Poincaré conjecture (Freedman) [none] → /api/competitions/lean-eval.md
- LeanEval / Ramanujan–Petersson conjecture for the τ-function (Deligne's theorem) [none] → /api/competitions/lean-eval.md
- LeanEval / Schreier's conjecture: outer automorphism group of a finite simple group is solvable [none] → /api/competitions/lean-eval.md
- LeanEval / Smale conjecture (Hatcher) in relative parameterized form [none] → /api/competitions/lean-eval.md
- LeanEval / The 290 theorem [none] → /api/competitions/lean-eval.md
- LeanEval / Watanabe's disproof of the 4-dimensional Smale conjecture [none] → /api/competitions/lean-eval.md
- LeanEval / Weak Goldbach theorem [none] → /api/competitions/lean-eval.md
- LeanEval / Weil conjectures in terms of point counts [none] → /api/competitions/lean-eval.md
- LeanEval / Weinstein conjecture in dimension three (Taubes 2007) [none] → /api/competitions/lean-eval.md
- s2n-bignum-bench / HOL Light Tactic Synthesis [cpu] → /api/competitions/s2n-bignum-bench.md
- GEO-Bench Leaderboard / Overall Accuracy Track [unknown] → /api/competitions/geo-bench-2-leaderboard.md
- GEO-Bench Leaderboard / Multilabel F1 Score Track [unknown] → /api/competitions/geo-bench-2-leaderboard.md
- GEO-Bench Leaderboard / Multiclass Jaccard Index Track [unknown] → /api/competitions/geo-bench-2-leaderboard.md
- GEO-Bench Leaderboard / Biomassters RMSE Track [unknown] → /api/competitions/geo-bench-2-leaderboard.md
- Humanoid Parkour / Humanoid Parkour Course [cpu] → /api/competitions/apex-humanoid-parkour.md
- GPU Mode Reference Kernels / Causal Conv1d [datacenter-gpu] → /api/competitions/reference-kernels.md
- GPU Mode Reference Kernels / Gated Deltanet chunk_fwd_o [datacenter-gpu] → /api/competitions/reference-kernels.md
- GPU Mode Reference Kernels / QR Decomposition v2 [datacenter-gpu] → /api/competitions/reference-kernels.md
- GPU Mode Reference Kernels / Cholesky Validation [datacenter-gpu] → /api/competitions/reference-kernels.md
- GPU Mode Reference Kernels / Eigh Validation [datacenter-gpu] → /api/competitions/reference-kernels.md
- MedMCQA / MedMCQA Test Accuracy [unknown] → /api/competitions/medmcqa.md

## Competitions
- [AssetOpsBench](/api/competitions/assetopsbench.md): A unified framework for building, orchestrating, and evaluating domain-specific AI agents in Industry 4.0. [active, unknown, 0 records]
- [AlpacaEval](/api/competitions/alpaca-eval.md): An automatic evaluator for instruction-following language models. [active, none, 22 records]
- [CogDL](/api/competitions/cogdl.md): An extensive graph deep learning toolkit to train, evaluate, and benchmark graph neural networks. [active, consumer-gpu, 241 records]
- [VBench](/api/competitions/vbench.md): A comprehensive benchmark suite for evaluating video generative models across multiple dimensions. [active, datacenter-gpu, 0 records]
- [Chess Coding Challenge](/api/competitions/chess-challenge.md): Create a small C# chess bot within a 1024-token limit and compete in a tournament. [ended, none, 0 records]
- [LLM Colosseum](/api/competitions/llm-colosseum.md): Evaluate LLMs in real time with Street Fighter III [active, unknown, 14 records]
- [ClawProBench](/api/competitions/clawprobench.md): Trace-aware evaluation of AI agents with runtime coverage and frozen workplace-style holdouts. [active, unknown, 0 records]
- [SQuAD](/api/competitions/squad-explorer.md): The Stanford Question Answering Dataset benchmark for reading comprehension. [active, unknown, 0 records]
- [Polymarket Paper Trader](/api/competitions/polymarket-paper-trader.md): A zero-risk gym for AI agents to practice, evaluate, and benchmark decision intelligence on real prediction markets. [active, none, 0 records]
- [SKAB](/api/competitions/skab.md): Benchmark for evaluating anomaly detection algorithms on multivariate time series data. [active, unknown, 24 records]
- [PutnamBench](/api/competitions/putnambench.md): Multilingual benchmark for evaluating neural theorem-provers on the Putnam Mathematical Competition. [active, unknown, 0 records]
- [JevBench](/api/competitions/fstandhartinger-jevbench-2acd10345790c8d1.md): Benchmark for AI decision models measuring accuracy, probability calibration, latency, and cost. [active, none, 72 records]
- [LoRA Speedrun](/api/competitions/lora-speedrun.md): A wall-clock speedrun benchmark for LoRA fine-tuning on a single L40S GPU. [active, datacenter-gpu, 8 records]
- [Mathematics Distillation Challenge — Equational Theories — Stage 2](/api/competitions/equational-theories-lean-stage2.md): Machine-verifiable Lean 4 certificates for equational implication over magmas. [active, cpu, 0 records]
- [First Proof](/api/competitions/first-proof.md): Unpublished research-level math problems; AI systems get one shot each and expert referees grade the proofs blind. [active, none, 43 records]
- [Sutro Problems](/api/competitions/sutro-problems.md): Optimize machine learning and matrix multiplication algorithms for energy efficiency on physical cost models. [active, datacenter-gpu, 88 records]
- [LeanEval](/api/competitions/lean-eval.md): Prove open formalization problems in Lean 4; models and submitters are ranked by problems solved. [active, none, 135 records]
- [s2n-bignum-bench](/api/competitions/s2n-bignum-bench.md): Benchmark LLM low-level code reasoning via HOL Light tactic synthesis on AWS cryptographic proofs. [active, cpu, 0 records]
- [Agent Anvil Leaderboard](/api/competitions/agent-anvil-leaderboard.md): Compare final-answer-only checks with trace-aware assertions across agent tool-use safety scenarios. [active, unknown, 1 records]
- [ProgramBench](/api/competitions/programbench.md): Evaluate and track the performance of AI coding agents on software engineering tasks. [active, unknown, 10 records]
- [CHI-Bench Leaderboard](/api/competitions/chi-bench.md): Benchmark AI agents on healthcare administrative and clinical tasks. [active, unknown, 65 records]
- [GEO-Bench Leaderboard](/api/competitions/geo-bench-2-leaderboard.md): Benchmark geospatial foundation models on diverse Earth observation tasks. [active, unknown, 0 records]
- [Modded-NanoGPT Ascend NPU Leaderboard](/api/competitions/modded-nanogpt-npu.md): Speedrun training NanoGPT to 3.28 validation loss on FineWeb using Ascend NPUs. [active, cluster, 91 records]
- [Humanoid Parkour](/api/competitions/apex-humanoid-parkour.md): Optimize an ONNX policy to drive a Unitree G1 humanoid through a 51m parkour course. [active, cpu, 1 records]
- [SWE-bench Pro Leaderboard](/api/competitions/swe-bench-pro-leaderboard.md): Evaluate coding agents on SWE-bench Pro, a dataset of 731 real-world software engineering tasks. [active, unknown, 6 records]
- [MLE-bench](/api/competitions/mle-bench.md): Evaluating machine learning agents on machine learning engineering tasks across 75 Kaggle competitions. [active, datacenter-gpu, 120 records]
- [GPU Mode Reference Kernels](/api/competitions/reference-kernels.md): Optimize high-performance GPU kernels for deep learning and linear algebra. [active, datacenter-gpu, 0 records]
- [LLM-Leaderboard](/api/competitions/llm-leaderboard.md): A joint community effort to create one central leaderboard for LLMs. [ended, none, 153 records]
- [MedMCQA](/api/competitions/medmcqa.md): A large-scale multiple-choice question answering dataset for the medical domain. [active, unknown, 0 records]
- [Grand Mountain Adventure Community Leaderboard](/api/competitions/gmaapp.md): Highscores and world records for the mobile game Grand Mountain Adventure. [active, none, 74 records]
- [Terminal Bench 2.0 Leaderboard](/api/competitions/terminal-bench-leaderboard.md): Evaluate agents on terminal-based coding and system tasks. [active, unknown, 7 records]