Loading page…
Preferred languages
More languages supported by Google Translate
Open the selector to load the current language catalog.
Translations are provided by Google Translate.
Loading page…
Preferred languages
More languages supported by Google Translate
Open the selector to load the current language catalog.
Translations are provided by Google Translate.
Provenance
Every number on this site is reproduced from one of the fetches below, in the unit its source declared. This page is the whole list, with the time each one was last read.
21 sources · dataset generated 2026-11-21 01:17 UTC.
Fetch state key: freshpartial
935 benchmarks, grouped by category · dataset generated 2026-11-21 01:17 UTC.
A display-only Artificial Analysis agentic index.
Artificial Analysis model leaderboardsAn independently evaluated automation benchmark from Artificial Analysis.
Artificial Analysis AutomationBench Benchmark LeaderboardThe strict pass@1 score on Artificial Analysis's AutomationBench-AA, where a business-workflow task counts only when it is fully completed.
Artificial Analysis AutomationBench Benchmark LeaderboardAn independently evaluated professional-work benchmark reported as Elo.
Artificial Analysis Briefcase Benchmark LeaderboardArtificial Analysis' composite index of cyber-defense capability: finding vulnerabilities in real codebases, reproducing and validating them, and patching them without breaking existing functionality. It averages CWE-Bench-AA, DeepsecBench-AA and CyberGym-E2E-AA.
Artificial Analysis Cyber IndexAn independently evaluated enterprise-operations benchmark from Artificial Analysis.
Artificial Analysis EnterpriseOps-Gym Benchmark LeaderboardArchived v1.0 legal-agent results from Artificial Analysis, kept separate from v1.1.
Artificial Analysis Harvey LAB-AA Benchmark LeaderboardAn independently evaluated IT-operations benchmark from Artificial Analysis.
Artificial Analysis ITBench-AA Benchmark LeaderboardAn independently evaluated Tau3 banking benchmark from Artificial Analysis.
Artificial Analysis Tau3-Banking Benchmark LeaderboardAn independently evaluated Terminal-Bench v4.0 result from Artificial Analysis, one of the ten components of its Intelligence Index v4.3.
Artificial Analysis Terminal-Bench v4.0 Benchmark LeaderboardArtificial Analysis' data-analysis benchmark, testing agents on spreadsheet and document work to answer the quantitative questions a business or data analyst faces day to day.
AA-AnalystAgent Benchmark LeaderboardThe rubric pass rate on Artificial Analysis's AA-Briefcase v1.1 knowledge-work benchmark, alongside its Elo headline.
Artificial Analysis AA-BriefcaseAn agent benchmark reported in DeepSeek's V4 Flash 0731 launch comparison.
DeepSeek V4 Flash 0731 updateThe command-line subset of Agents' Last Exam v1.1: 86 of the release's 122 agent tasks.
Kimi K3.1 Pro Tech Blog: Advancing Open IntelligenceThe Linux subset of Agents' Last Exam, an end-to-end agent task benchmark, as reported by Meta.
Introducing Muse EmberTests whether coding agents can improve the training algorithm inside an existing AI research codebase, then survive a sealed training run and held-out evaluation.
AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-ImprovementA mobile GUI agent benchmark for completing Android app workflows and on-device tasks.
GLM-5V-TurboA professional-services agent benchmark covering long-horizon knowledge-work tasks.
Kimi K3: Open Frontier IntelligenceArtificial Analysis' implementation of the APEX-Agents benchmark for long-horizon professional-services agent tasks.
APEX-Agents-AA Benchmark LeaderboardTests whether a computer-use agent can learn a real accounts-payable job on the job, processing 100 vendor bills in a company ERP system with only the handbook, historical records, and mentor feedback a new hire would get.
ApprenticeBench: a step change in AI's job readinessA Markov Studios computer-use benchmark that asks agents to produce 2D drawings and 3D models in AutoCAD.
AutoCAD-BenchAn agent benchmark for completing automation workflows in reproducible task environments.
Kimi K3: Open Frontier IntelligenceAn agent benchmark for completing business workflows across simulated applications and API endpoints.
AutomationBenchLakera's open-source benchmark of how well the LLM inside an AI agent resists attacks such as system-prompt exfiltration, malicious code injection and tool-call manipulation. Higher means more attacks resisted.
The Backbone Breaker BenchmarkA display-only provider benchmark for finance-oriented tool-use and agent workflows.
MiniMax M3 model cardA function-calling benchmark for tool selection, schema adherence, and argument correctness, covering single-turn, parallel, irrelevance and multi-turn subsets.
The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language ModelsA function-calling benchmark for tool selection, schema adherence, and argument correctness.
The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language ModelsAnthropic's unguided vulnerability-discovery and exploitation evaluation: from a single function entry point in an open-source project, the model must find a bug and push it toward a working exploit. This lane counts control-flow hijacks.
Claude Fable 5.2 and Claude Mythos 5.2 system cardAgentic investigation of biological datasets, reported as an overall score on the official v11 release.
Kimi K3.1 Pro Tech Blog: Advancing Open IntelligenceBrowseComp tests whether a browsing agent can find difficult, fact-seeking answers across the web. Its 1,266 problems use short answers that are easier to grade than an open-ended research report.
BrowseCompBrowseComp accuracy from ten collaborating Opus 5 agents on a pre-release model and unreleased effort configuration.
Claude Opus 5 System CardA vision-language browsing benchmark for multimodal web research and tool-use workflows.
GLM-5V-TurboEvaluates research agents against a fixed document corpus, separately from live-web BrowseComp.
BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research AgentMoonshot AI's internal benchmark for long-horizon management and decision-making in a simulated business environment.
Kimi K3.1 Pro Tech Blog: Advancing Open IntelligenceA transparent real-world autonomous-agent benchmark with 300 human-verified tasks, 2,159 rubric items, and Pass^3 scoring across general, multi-turn, and native multimodal agent tasks.
Claw-Eval: Towards Trustworthy Evaluation of Autonomous AgentsQwen's internal benchmark for long-horizon professional work across computer science, finance, law, medicine, and other productivity domains.
Qwen3.8-Max: A New Bar for Coding and CoworkA cybersecurity benchmark that measures whether an agent can turn raw threat-intelligence reports into working detection rules.
Introducing Fugu-CyberAnthropic's internal evaluation of whether a model can independently rediscover known vulnerabilities from 1,829 CVE advisories, given only the affected source code.
Claude Fable 5.2 and Claude Mythos 5.2 system cardDefensive vulnerability patching on 250 audit-and-patch tasks spanning 185 weakness types, 100 public and 150 held out.
CWE-bench v2 leaderboard and methodologyAn external benchmark for evaluating whether coding agents can produce correct patches for real-world software vulnerabilities.
CWE-Bench leaderboardDefensive vulnerability patching on 120 held-out tasks spanning 73 weakness types. Replaced by v2 on November 2, 2026.
CWE-bench v1 leaderboard and methodologyArtificial Analysis' implementation of Collinear AI's CWE-bench, testing whether agents can patch real-world software vulnerabilities. One of the three components of the AA Cyber Index.
CWE-Bench-AA Benchmark LeaderboardA cybersecurity benchmark of professional Capture the Flag tasks for measuring autonomous cyber agent capability and risk.
Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language ModelsAverage refusal rate on the cyber-related prompts of JailbreakBench, StrongREJECT and AgentHarm, as measured by Mistral. Higher means the model refuses more malicious cyber requests.
Mistral Large 4 is now open weightA cybersecurity task benchmark for evaluating defensive cyber workflows and vulnerability-oriented agent performance.
CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at ScaleArtificial Analysis' implementation of Berkeley RDI's CyberGym-E2E: agents find a real memory-safety vulnerability in a C/C++ open-source project, write a proof-of-concept input that crashes it, and patch it without breaking the project's tests.
CyberGym-E2E-AA Benchmark LeaderboardMoonshot AI's internal arena for presentation generation on real-world queries, reported as Elo.
Kimi K3.1 Pro Tech Blog: Advancing Open IntelligenceMoonshot AI's internal benchmark for presentation and deck-production workflows.
Kimi K3: Open Frontier IntelligenceA long-horizon planning benchmark that tests whether agents can optimize under explicit time, budget, and feasibility constraints.
DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable ConstraintsAn agentic browsing benchmark where models search the web, gather evidence, and answer list-style questions using browser tools.
Muse Spark Eval MethodologyArtificial Analysis' implementation of Vercel's DeepsecBench, testing agents on finding security vulnerabilities in real web application code. One of the three components of the AA Cyber Index.
DeepsecBench-AA Benchmark LeaderboardA display-only Elo rating from blinded comparisons of multi-file web applications built by coding agents.
Design Arena Web Dev (Agentic) LeaderboardMoonshot AI's internal graphic-design benchmark using direct and reference-image few-shot prompting, reported as Elo.
Kimi K3.1 Pro Tech Blog: Advancing Open IntelligenceAgentic data-analysis tasks scored against per-task rubrics at a 980K-token budget.
Claude Opus 5 System CardA ByteDance Seed benchmark of 134 real-world, day-scale tasks that measures how autonomous agents learn from environment feedback over 12+ hour interaction horizons, spanning scientific and ML, systems and software engineering, optimization, knowledge, formal, and game domains.
EdgeBench technical reportServiceNow benchmark of stateful planning and tool use across simulated enterprise systems, as run by model providers.
EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise SettingsA multi-turn benchmark of emotional and social intelligence using synthetic personas and pairwise LLM judging.
EQ-Bench 4A controlled benchmark for evaluating whether AI agents can extend vulnerability-triggering inputs into working exploits.
ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?Tests whether a system can discover a programming language's hidden rules through experiments and apply them to unseen tasks.
ExplorationBench: Measuring AI Systems' Exploration in Verifiable Alien WorldsTests whether a system can discover altered logic rules and use them to prove or correctly decline unseen theorems.
ExplorationBench: Measuring AI Systems' Exploration in Verifiable Alien WorldsVals AI benchmark for realistic financial analyst agent tasks across qualitative analysis, quantitative analysis, market work, comparables, precedents, earnings, disclosure, and modeling.
Finance Agent v2Moonshot AI's internal benchmark for formula linkage, financial logic, and final workbook quality in real-world modeling tasks.
Kimi K3.1 Pro Tech Blog: Advancing Open IntelligenceMoonshot AI's internal benchmark for auditable financial research under evidence and cutoff constraints.
Kimi K3.1 Pro Tech Blog: Advancing Open IntelligenceA benchmark of real finance and accounting workflows built around spreadsheets: creating, editing and analysing workbooks the way an analyst or accountant would.
Finch: Benchmarking Finance & Accounting across Spreadsheet-Centric Enterprise WorkflowsAn investment-workflow benchmark covering complete tasks from screening to profitability analysis.
Kimi K3.1 Pro Tech Blog: Advancing Open IntelligenceGAIA evaluates AI models on real-world tasks that are conceptually simple for humans but require multi-step reasoning, web browsing, tool use, and multimodal understanding for AI. Tasks span three difficulty levels and test practical assistant capabilities rather than academic knowledge.
BenchLMArtificial Analysis' document-output benchmark: professional tasks whose deliverable is a produced PDF, one of the ten components of its Intelligence Index v4.3.
GDP.pdf Benchmark LeaderboardA display-only provider-table GDPval rubric score for economically valuable work tasks.
MiniMax M3 model cardAn agentic real-world work-task evaluation reported as an Elo score in DeepSeek-V4 thinking-mode evaluations.
DeepSeek-V4 Technical ReportAn Artificial Analysis normalized score for economically valuable tasks.
Artificial Analysis model benchmarksA game-environment benchmark that evaluates AI models in novel games covering strategic planning, resource management, spatial reasoning, cooperation, and theory of mind.
Gert Labs rankingsAn agent benchmark run through its official OpenHands-based harness.
Kimi K3.1 Pro Tech Blog: Advancing Open IntelligenceLegal-agent task success requiring every rubric criterion to pass and no material hallucination, averaged across three judges.
Harvey LAB-AA v1.1 Benchmark LeaderboardThe share of legal-deliverable rubric criteria passed, pooled across three judges before the hallucination gate.
Harvey LAB-AA v1.1 Benchmark LeaderboardLegal-agent tasks from the public 1,251-task Harvey dataset, scored by the share of tasks that pass every rubric criterion.
Kimi K3.1 Pro Tech Blog: Advancing Open IntelligenceTests agents on 150 tasks across 25 categories, including Hermes skills, research, diagrams and art, memory, tool use and safety. Each agent starts in a workspace with real files, some tasks add follow-up turns, and the grader checks the files and state left behind.
Hermes Bench (Hermes Index)An index of agentic capability from Nous Research. It averages pass@1 scores on four suites, Hermes Bench, TerminalBench 4, TerminalBench Science and SkillsBench, with every run inside the Hermes Agent harness, and reports the mean cost per task beside the score.
Hermes Index: an agentic model leaderboardTool-augmented Humanity's Last Exam scores reported in DeepSeek-V4 thinking-mode evaluations.
DeepSeek-V4 Technical ReportA benchmark for open-ended LLM inference optimization by AI agents. Agents receive a base model, one H100, and a fixed time budget to build a valid OpenAI-compatible inference server that improves serving speed.
InferenceBenchAn occupational agent benchmark for professional workflows that workers say they most want delegated to AI.
JobBench: Aligning Agent Work With Human WillA Moonshot AI internal long-horizon agent benchmark for persistent professional coworking tasks.
Kimi K2.7 CodeThe hard split of Moonshot AI's internal factuality benchmark, run with tools and scored as the claim-level hallucination rate.
Kimi K3.1 Pro Tech Blog: Advancing Open IntelligenceStrict task success requiring every expert-written legal-work rubric criterion to pass.
Claude Opus 5 System CardHarvey AI's strict held-out task success rate requiring every rubric criterion to pass.
Claude Opus 5 System CardMean fraction of expert-written rubric criteria passed across legal-agent tasks.
Claude Opus 5 System CardHarvey AI's mean criterion-level score on its held-out legal-agent evaluation.
Claude Opus 5 System CardOpenAI's benchmark of practical everyday tasks people want a personal assistant to handle, such as finding childcare, accessing government services, budgeting, shopping and caregiving.
LifeValPersonal-assistant agent tasks run end to end inside the OpenClaw agent harness.
Introducing LFM3 (Liquid AI)Tests long-context agents as tool environments grow to a 256K-token environment description.
LOCA-bench: Benchmarking Language Agents Under Controllable Environment GrowthA quantitative-trading implementation benchmark that asks models to build backtesters under market-book liquidity and execution-delay constraints, then compares their outputs with a verifier.
Market-BenchTests whether a coding agent can explore a 256-room 3D puzzle world and collect its 100 hidden gems, using box-pushing puzzles, camera rotation, and long-horizon planning.
MazeBench ResultsA benchmark for tool-calling over Model Context Protocol integrations and external tools.
Introducing GPT-5.4 mini and nanoA human-verified edition of MCPMark for MCP tool use across Notion, GitHub, Filesystem, Postgres, and Playwright server environments.
MCPMarkAverage coverage of required claims in answers produced during real-world MCP tool-use workflows.
Claude Opus 5 System CardA Model Context Protocol task benchmark used in Qwen's launch tables to measure practical execution over MCP-style tools and integrations.
Qwen3.6 launch benchmarksA benchmark that evaluates AI agents on multi-environment web challenges, testing navigation and task completion across diverse live web environments.
MiniMax M2.5 benchmark release surfaceGoogle DeepMind's benchmark of professional knowledge work where the key information sits in visual files such as architectural drawings, engineering schematics and medical scans. The agent has to deliver a finished report, spreadsheet, slide deck or marked-up drawing. Artificial Analysis runs it on a private 200-task set.
MKA Benchmark LeaderboardA lightweight machine-learning competition benchmark that measures whether models can iteratively train, evaluate, and improve ML systems in low-resource settings.
MiniMax M2.7: Early Echoes of Self-EvolutionAn OpenClaw-derived agent benchmark covering practical work and life tasks such as office document delivery, research, planning, and code maintenance.
MiniMax M2.7: Early Echoes of Self-EvolutionA mobile-use agent benchmark for completing interactive tasks in smartphone environments.
Qwen3.8-Max: A New Bar for Coding and CoworkA holistic coding-agent benchmark that evaluates AI agents across issue resolution, frontend work, greenfield development, testing, and information gathering.
OpenHands Index methodologyA computer-use benchmark for GUI task completion across the broader OSWorld task suite.
GLM-5V-TurboOSWorld 2.0 evaluates long-horizon computer-use tasks. Published results use different task releases, metrics, and run settings, so scores appear by comparable protocol and unmatched rows stay unranked.
OSWorld2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World TasksOSWorld 2.1 is the current release of OSWorld 2.0: the same 108 long-horizon computer-use workflows with the maintainers' later fixes to task files, assets, mocked websites, and evaluators. This page stores partial scores on the full task set.
OSWorld 2.1 release (xlang-ai/OSWorld-V2, tag osworld-v2.1)The official offline subset of OSWorld 2.1: 82 of the 108 long-horizon computer-use workflows, run with no internet access in the agent's virtual machine.
OSWorld 2.1 release (xlang-ai/OSWorld-V2, tag osworld-v2.1)OSWorld-Verified is the July 2025 repaired release of OSWorld's real-computer evaluation. It measures whether a model-agent system can finish desktop and web tasks from configured starting states, with success checked by execution-based evaluators.
OSWorldAn OpenClaw agent benchmark from Kilo that measures successful task completion across standardized real-world agent workflows.
About PinchBenchQwen's internal OpenClaw-style benchmark for measuring broad real-world agent performance across practical productivity and research tasks.
Qwen3.6 launch benchmarksA Qwen benchmark for artifact and webpage generation quality reported as an Elo-style rating.
Qwen3.6 launch benchmarksAnthropic's internal evaluation of end-to-end financial-analyst work: 123 tasks, each model's attempts compared pairwise by a grader and rated on an Elo scale.
Claude Fable 5.2 and Claude Mythos 5.2 system cardTool-hallucination benchmark that checks whether a model selects and calls tools that fit the request.
RelyToolBenchScale AI and the Center for AI Safety's benchmark of 240 real paid freelance projects; the automation rate is the share of projects whose deliverable a client would accept.
Remote Labor IndexAn end-to-end autonomous scientific research benchmark with 40 tasks across 10 scientific domains, where agents receive related literature and raw data, then attempt to rediscover the hidden target paper.
ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific ResearchMeasures how reliably code-review agents identify useful issues in real pull requests while avoiding false positives.
ReviewBench: An open benchmark for AI code reviewA simulated robotics benchmark that measures the average success rate of a model controlling robots across simulated manipulation tasks.
Introducing Claude Fable 5.2 and Claude Mythos 5.2A spreadsheet-focused benchmark for agentic analysis and editing workflows.
Kimi K3: Open Frontier IntelligenceTests whether coding agents can complete long-horizon, whole-repository stack migrations while preserving the original program's behavior.
SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?A Scale SWE-Atlas software-engineering agent benchmark focused on refactoring tasks.
SWE-AtlasTests whether coding agents can find and fix multiple bugs across a repository without issue descriptions, while preserving existing behavior.
SWE-sweep: Can Agents Autonomously Find and Fix Bugs?Original TAU-bench tests an agent's tool use and policy compliance in simulated airline and retail conversations. Its pass^k metric asks whether the agent succeeds in every one of k repeated trials.
τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World DomainsA benchmark for agentic software engineering tasks executed in real terminal environments. Models must inspect files, run commands, edit code, and recover from errors over multi-step workflows.
Terminal-Bench 2.0Terminal-Bench 2.1 results, mostly provider-reported, stored separately from the Terminal-Bench 2.0 lane.
Terminal-BenchVals AI’s independent run of the Terminal-Bench 2.1 terminal-task suite with published easy, medium, and hard splits.
Vals AI Terminal-Bench 2.1, Vals AI run leaderboardA continuously maintained benchmark for difficult computer work, including coding, deep learning, finance, engineering, math, and science tasks.
Terminal-Bench 3.0Terminal-Bench 4.0 measures difficult computer work after recalibrating task resources, fixing unstable tasks, and removing tasks that no longer separate frontier systems. Version 4.1 regrades the same trials with hardened verifiers.
Terminal-Bench 4.0The current Terminal-Bench release keeps the 66 professional computer-work tasks of 4.0 and regrades every saved trial with verifiers hardened against reward hacking and overly strict checks.
Terminal-Bench 4.1A benchmark of AI agents completing expert-curated research workflows across the life, physical, Earth, mathematical, and engineering sciences.
Terminal-Bench-Science 0.1Google-run scientific workflow evaluation using a sixfold verifier timeout to address verification timeouts.
Gemini 4 Argon evaluation methodologyA tool-use benchmark focused on selecting, sequencing, and completing tasks with external tools.
Introducing GPT-5.4 mini and nanoAverage assistant turns per Toolathlon Verified trajectory.
Claude Opus 5 System CardFraction of Toolathlon Verified tasks solved in at least one of three trials.
Claude Opus 5 System CardFraction of Toolathlon Verified tasks solved in all three independent trials.
Claude Opus 5 System CardA verified tool-use benchmark variant for completing multi-step workflows with external tools.
Kimi K3: Open Frontier IntelligenceApple benchmark of stateful, conversational tool use, where tool calls change state and a simulated user takes part.
ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use CapabilitiesAn interactive real-world agent benchmark grounded in practical consumer-service tasks such as delivery, in-store consumption, and online travel workflows.
VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world ApplicationsPerplexity's wide-search research benchmark: 500 tasks in which a model must find tens to hundreds of entities, collect facts about each, and return them as cited structured records.
Claude Fable 5.2 and Claude Mythos 5.2 system cardWebArena tests whether a browser-agent system can complete 812 long-horizon tasks inside self-hosted replicas of functional websites. It checks the requested end state, so a result reflects the model, agent scaffold, browser interface, action budget, and evaluator together—not the base model alone.
WebArena: A Realistic Web Environment for Building Autonomous AgentsWebArena-Verified is an audited release of the WebArena browser-agent benchmark. It rechecks task descriptions, reference answers, and evaluators, and replaces nondeterministic judging with deterministic checks where possible.
WebArena-Verified: A Fully Audited Benchmark for Web AgentsA browser-agent benchmark for completing multi-step workflows on live websites.
GLM-5V-TurboNVIDIA multi-turn version of When2Call that tests whether a model calls a tool, asks a follow-up question, or abstains.
NVIDIA Nemotron 3.5 Bolt E4B 4GBA broad research-agent benchmark for open-ended information gathering, synthesis, and answer construction across wide search spaces.
Qwen3.6 launch benchmarksA Z.AI benchmark for OpenClaw-style agent workflows spanning information search, office work, data analysis, development and operations, automation, and security.
GLM-5-Turboτ²-bench Airline tests conversational agents on airline customer-service tasks governed by domain policy and database-changing tools.
τ²-Bench: Evaluating Conversational Agents in a Dual-Control EnvironmentThis route is a sourced ledger for published τ²-bench results. Most current rows come from Artificial Analysis's telecom implementation, while named provider rows can use telecom, airline, retail, or aggregate setups.
τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environmentτ³-bench is the current evolution of Sierra's tool-agent-user framework, adding corrected task releases and newer knowledge and voice evaluation modes alongside airline, retail, and telecom.
Official τ³-bench repository and release notesA display-only Artificial Analysis leaderboard for coding-agent systems, combining agent harnesses, host models, and execution settings across software-engineering benchmarks.
Artificial Analysis Coding Agent BenchmarksA display-only Artificial Analysis coding index.
Artificial Analysis model leaderboardsAn independently evaluated LiveCodeBench result from Artificial Analysis.
Artificial Analysis LiveCodeBench Benchmark LeaderboardAn independently evaluated Terminal-Bench v2.1 result from Artificial Analysis.
Artificial Analysis Terminal-Bench v2.1 Benchmark LeaderboardAn Artificial Analysis SciCode score.
Artificial Analysis SciCode Benchmark LeaderboardA six-task full-stack web-app benchmark that measures how much required functionality an AI builder or coding assistant delivers from one prompt without human code edits.
App-BenchA code-generation benchmark reported in DeepSeek-V4 base-model evaluations.
DeepSeek-V4 Technical ReportA blind-graded coding-agent benchmark with 105 planted bugs across two production TypeScript repositories.
Bug Hunt Bench method, caveats, and definitionsHugging Face's open benchmark for generating valid and geometrically correct 3D mechanical parts.
CADGenBenchCompetitive-programming rating reported for DeepSeek-V4 thinking-mode evaluations.
DeepSeek-V4 Technical ReportCursor's first-party benchmark for ambiguous, multi-file coding-agent tasks drawn from real Cursor sessions, currently on the 4.0 task set.
CursorBench 4.0A long-horizon software engineering benchmark from Datacurve for measuring frontier coding agents on original tasks drawn from active open-source repositories.
DeepSWE benchmark blogDeepSeek's internal full-stack coding-agent benchmark.
DeepSeek V4 Flash 0731 updateDeepSeek's internal hard coding-agent benchmark.
DeepSeek V4 Flash 0731 updateA systems and software-engineering benchmark from ByteDance Seed that evaluates agents on long-horizon edge tasks using time-budgeted learning curves rather than a single static pass rate.
EdgeBenchAn electrical-engineering agent benchmark maintained by atopile.
EEBench leaderboardA repository-level Lean 4 proof engineering benchmark that measures whether a model can complete formal proofs and correctly define new mathematical concepts inside realistic FLT project pull requests.
Leanstral: Open-Source foundation for trustworthy vibe-codingCognition's 150-task Extended subset of the FrontierCode 1.1 software-engineering benchmark.
FrontierCode leaderboardCognition's 100-task software-engineering benchmark for whether coding agents produce mergeable, production-quality pull requests, scored for correctness, tests, scope, style, and maintainability through maintainer-authored rubrics.
FrontierCode leaderboardCognition's second-generation coding-agent benchmark, scoring whether agents produce mergeable pull requests on larger, less fully specified tasks, with added emphasis on thoroughness such as fixing every instance of a bug.
FrontierCode 2.0Artificial Analysis's own run of Cognition's FrontierCode 2.0 coding-agent benchmark, one of the nine evaluations in its Intelligence Index v5, judged on whether each pull request is mergeable.
Artificial Analysis Intelligence IndexAn ultra-long-horizon software-engineering benchmark with open-ended implementation, performance, and research tasks designed to challenge frontier coding agents.
FrontierSWE: Benchmarking coding agents at the limits of human abilitiesA 34-task expansion of FrontierSWE for ultra-long-horizon engineering and research work that remains far from saturation.
FrontierSWE v2Evaluates coding agents on 333 multimodal game-development tasks in Godot, spanning 2D graphics, 3D graphics, user interfaces, and gameplay logic.
GameDevBench: Evaluating Agentic Capabilities Through Game DevelopmentHumanEval tests whether generated Python functions pass unit tests on 164 handwritten problems. These scores describe function-level correctness; they do not measure repository work or a coding agent.
Evaluating Large Language Models Trained on CodeAn 80-task software-engineering benchmark across eight repositories that tests whether autonomous IDE agents can explore, edit, run, and verify code changes end to end.
IDE-BenchAn agentic GPU-kernel benchmark that measures how much of the hardware roofline a model's correct, audit-clean kernels reach on six demanding CUDA and Triton problems.
KernelBench HardA display-only benchmark for difficult GPU kernel implementation and optimization tasks.
MiniMax M3 model cardMoonshot AI's internal long-horizon coding-agent benchmark, the successor to Kimi Code Bench 2.0.
Kimi K3.1 Pro Tech Blog: Advancing Open IntelligenceA Moonshot AI internal coding-agent benchmark for realistic software-engineering tasks across mainstream programming languages and production technology stacks.
Kimi K2.7 CodeA continuously updated coding benchmark built from newly collected LeetCode, AtCoder, and Codeforces problems. Fresh problem windows reduce one contamination path, but results still need a release and setup check.
LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for CodeVals AI’s independent implementation of the LiveCodeBench code-generation benchmark, run under one fixed harness across the models it tracks.
Vals AI LiveCodeBench, Vals AI run leaderboardThis lane contains DeepSeek's LiveCodeBench Pass@1-COT results. The explicit metric and prompting label keeps them separate from generic and version-specific LiveCodeBench rows.
DeepSeek-V4 technical reportA harder competitive-programming benchmark family built from Codeforces, ICPC, and IOI problems, with quarter-specific public leaderboards and difficulty-aware reporting.
LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?LiveCodeBench v5 is a named release and date-window slice. BenchLM keeps explicitly labeled v5 rows outside the rolling weighted lane.
LiveCodeBench official repository and release documentationLiveCodeBench v6 is a named release slice used in provider comparison tables. Keeping it separate prevents v6 results from being mixed into older or rolling LiveCodeBench windows.
LiveCodeBench official repository and release documentationA 30-task subset of MLS-Bench that evaluates whether AI systems can invent generalizable and scalable machine-learning methods.
MLS-BenchA multi-language software-engineering benchmark that measures repository-level bug fixing and implementation across more than one programming ecosystem.
MiniMax M2.7: Early Echoes of Self-EvolutionA Vercel benchmark for AI coding agents on Next.js code generation and migration tasks, reporting pass@4, average execution time, and results with bundled documentation through AGENTS.md.
AI Agent Evaluations | Next.jsA repository-understanding benchmark that measures whether models can map natural-language requests onto the right code locations and system changes.
MiniMax M2.7: Early Echoes of Self-EvolutionAn app-level coding benchmark that asks DevEco Code configurations to implement observable behavior in buildable OpenHarmony ArkTS applications. Replaced by OpenHarmony Bench v1.2 in November 2026.
OpenHarmony Bench: Evaluating LLMs and Coding Agents on OpenHarmony App DevelopmentAn app-level coding benchmark that asks DevEco Code configurations to build, extend, fix and migrate buildable OpenHarmony ArkTS applications.
OpenHarmony Bench: Evaluating LLMs and Coding Agents on OpenHarmony App DevelopmentA research-reproduction benchmark that asks agents to recreate the contributions of AI papers from the paper alone.
Qwen3.8-Max: A New Bar for Coding and CoworkAn AI agent gets ten hours and one H100 GPU to post-train a small base model; the score is how well the trained models do on six benchmarks.
PostTrainBenchPostTrainBench results that model makers ran themselves and published without naming the benchmark version.
PostTrainBenchThe first seven-benchmark PostTrainBench board, kept by the source as an archived version.
PostTrainBench v1 leaderboard (archived)Post-training four base language models across seven weighted benchmarks, with ten hours and one H100 per run.
PostTrainBench v1.1 leaderboard and methodologyA cleanroom software-engineering benchmark where agents receive only a compiled executable and documentation, then must architect and implement a complete codebase that reproduces the original program's behavior.
ProgramBench: Can Language Models Rebuild Programs From Scratch?ProgramBench cleanroom reimplementation tasks scored as pass/fail: a task counts only when its ProgramBench score is at least 95%.
ProgramBenchProgram-reconstruction hidden-test pass rate after the first of five sequential long-context episodes.
ProgramBench: Can language models rebuild programs from scratch?Qwen's internal benchmark for building and rendering bilingual React projects across seven categories.
Qwen3.8-Max: A New Bar for Coding and CoworkAn open benchmark for AI coding agents on real-world React Native implementation tasks, emphasizing working app behavior, recommended architecture choices, and strict constraint adherence.
React Native EvalsA coding-agent benchmark for realistic React work, with rubrics that check production concerns such as performance, accessibility, correctness, and code quality.
ReactBenchIBM Research's benchmark for coding agents that migrate complete enterprise Java applications between Jakarta EE, Quarkus and Spring while keeping them working.
ScarfBench: A Benchmark for Cross-Framework Application Migration in Enterprise JavaSciCode evaluates language models on generating code for realistic scientific research problems across 16 subfields of physics, math, chemistry, biology, and material science. Problems decompose into 338 subproblems requiring domain knowledge recall, scientific reasoning, and precise code synthesis. Based on real scripts from published research.
BenchLMA Snorkel AI benchmark of senior-level software engineering tasks emphasizing under-specified feature work, bug/performance investigation, and taste-based correctness.
Senior SWE-BenchA text-to-SQL benchmark over realistic warehouse-scale schemas, reported by Interfaze for model comparison.
Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL WorkflowsA display-only provider benchmark for generating or manipulating SVG outputs from natural-language requirements.
MiniMax M3 model cardA multilingual software-engineering benchmark for real-world code issue resolution across multiple programming languages.
MiniMax M2.7: Early Echoes of Self-EvolutionA multimodal variant of SWE-bench that adds visual context such as screenshots and design mockups to software engineering issue descriptions.
SWE-bench MultimodalA code-comprehension benchmark covering production repositories across Go, Python, C, and TypeScript.
Muse Spark 1.3 Evaluation MethodologyVals AI’s independent run of the public SWE-bench issue set, reported by human time-to-fix bucket. Vals removed SWE-bench Verified from its index as saturated on 2026-05-04; this board is the standalone SWE-bench run.
Vals AI SWE-bench, Vals AI run leaderboardA long-horizon repository benchmark built to test realistic software engineering work. Its scores need a task-quality and setup check before they support a coding-agent decision.
SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?SWE-bench Verified tests patches for 500 human-filtered GitHub issues. A score measures how many tasks an evaluated model-and-agent setup resolved; the model name alone does not identify that setup.
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?A display-only SWE-bench Verified reference from Arcee AI's Trinity-Large-Thinking comparison chart.
Trinity-Large-Thinking: Scaling an Open Source Frontier AgentA continuously updated software engineering benchmark by Nebius using fresh GitHub issues to avoid contamination. Models are evaluated 5 times per problem under a fixed ReAct scaffolding; the Resolved Rate (best pass@1) is reported.
SWE-Rebench: Contamination-Free Evaluation of Software Engineering AgentsA benchmark for agentic software engineering tasks executed in real terminal environments. DeepSeek reports it in the agentic section, while BenchLM also mirrors it in coding for models that publish it as a developer-task signal.
Terminal-Bench 2.0Terminal-Bench 2.1 results, mostly provider-reported, stored separately from the Terminal-Bench 2.0 lane.
Terminal-BenchA display-only Artificial Analysis coding metric for agentic coding and terminal use on a harder Terminal-Bench slice.
Artificial Analysis model benchmarksVals.ai benchmark for evaluating whether models can build complete web applications from natural language specifications in a production-like development environment.
Vibe Code Bench: Evaluating AI Models on End-to-End Web Application DevelopmentA display-only MiniMax provider benchmark for end-to-end coding-agent and product-building tasks.
MiniMax M3 model cardA repo-level code generation and full-project delivery benchmark spanning web, mobile, and simulation-style implementation tasks.
MiniMax M2.7: Early Echoes of Self-EvolutionA post-cutoff software-engineering benchmark with hidden functional tests and regression guards, reported for vendor coding-agent harnesses.
CII v1 frontier resultsAn open coding benchmark of 23 hard behavioural-reconstruction tasks, run at every reasoning-effort level and scored on correctness and judged code quality.
VulcanBench Frontier v4 leaderboardAn open software-engineering benchmark built from real merged post-cutoff pull requests across Python, Rust, TypeScript, JavaScript, and Go repositories. Replaced by VulcanBench Frontier v4 in September 2026.
VulcanBenchA benchmark for vision-language model agents that write procedural code for 3D modeling software, turning a text description into an engine-ready 3D object. This page tracks the text-prompt track.
3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via CodeAA-Omniscience-Decision accuracy is one component of the Artificial Analysis Decision Index v1.0, run by Artificial Analysis. It is the share of 500 private knowledge questions answered correctly, each with four answers and an "I don't know" option.
Artificial Analysis Decision Index v1.0AA-Omniscience-Decision non-hallucination rate is one component of the Artificial Analysis Decision Index v1.0, run by Artificial Analysis. It is the share of questions not answered correctly where the model chose "I don't know" instead of a wrong answer. Higher is better.
Artificial Analysis Decision Index v1.0Measures how models write by default across 30 personality traits and 21 communication characteristics. A stronger trait is not necessarily a better answer.
AA-PersonalityBench methodologyCan AI help someone navigate a civil legal problem without a lawyer?
Access to Justice leaderboard and methodologyShare of approved advanced-cyber requests completed rather than refused by GPT-5.6 Sol under Daybreak Blue safeguards.
Expanding Daybreak as the cyber defense window narrowsShare of approved advanced-cyber requests completed rather than refused by GPT-5.6 Cyber under Daybreak Red access.
Expanding Daybreak as the cyber defense window narrowsShare of approved advanced-cyber requests completed rather than refused under standard GPT-5.6 Sol safeguards.
Expanding Daybreak as the cyber defense window narrowsNumber of advanced cyber-range challenges solved in the joint NIST CAISI and UK AISI preliminary evaluation.
UK AISI and CAISI Preliminary Assessment of Kimi K3's Cyber CapabilitiesFollowing complex instructions graded against rubric criteria. This table shows Mercor-run configurations for reference and is excluded from model rankings.
AdvancedIF evaluation resultsAgents play customer service staff, genuine customers, and identity thieves in a hidden-information authentication simulator.
BenchLMWhich model can make the most money playing poker?
Agent Poker BenchAikido Security's live benchmark of how well AI models rediscover real vulnerabilities. Each model runs three times on 32 recently disclosed CVEs in open-source repositories, and the score is the share it finds in at least one run (pass@3 recall).
Aikido Arena: AI model benchmarks for cybersecurityThe first version of Aikido Security's model benchmark: 13 models at two reasoning levels, three runs each on 26 known CVEs from the GitHub advisory database, scored by pass@3 recall.
Benchmarking 13 AI models on rediscovering known CVEsKaggle's run of the 2025 AIME competition problems, with 67 configurations, separate from provider, Vals, and Artificial Analysis AIME rows.
AIME problems and solutionsSolving competition math problems with integer answers. This table shows Mercor-run configurations for reference and is excluded from model rankings.
AIME 2025 evaluation resultsA benchmark for agentic professional workflows with verifiable success criteria, reporting pass rates and partial scores for model plus agent-harness rows.
Agents Last ExamShare of evaluated OSS-Fuzz entry points where the model produced at least a crash.
Claude Fable 5 and Claude Mythos 5Share of evaluated OSS-Fuzz entry points where the model achieved a write primitive or stronger result.
Claude Fable 5 and Claude Mythos 5Single-turn professional deliverables graded against expert-written criteria. This table shows Mercor-run configurations for reference and is excluded from model rankings.
APEX-1 evaluation resultsMonth-end close tasks across ten simulated companies, worked through accounting software, spreadsheets and PDFs. This table shows Mercor-run configurations for reference and is excluded from model rankings.
APEX-Accounting evaluation resultsMercor's earlier 160-task APEX-Accounting set, kept as captured on October 19, 2026. This table shows Mercor-run configurations for reference and is excluded from model rankings.
APEX-Accounting evaluation resultsProfessional-services tasks across investment banking, management consulting, and corporate law. This table shows Mercor-run configurations for reference and is excluded from model rankings.
APEX-Agents 1.1 evaluation resultsSoftware integration and production-failure diagnosis with the Inspect harness. This table shows Mercor-run configurations for reference and is excluded from model rankings.
APEX-SWE (Inspect) evaluation resultsSoftware integration and production-failure diagnosis with the Terminus-2 harness. This table shows Mercor-run configurations for reference and is excluded from model rankings.
APEX-SWE (Terminus-2) evaluation resultsArtificial Analysis scores decision models on its Decision Index and runs every evaluation itself. Version 2.1 is the weighted average of six Decision Scores: AA-Paperclip document decisions (30%), EvalSafe-O*NET-AA workplace judgement, AA-Omniscience-Decision v1.1, HLE-MultiChoice v1.1 and AA-Symmetry v1.1 decision invariance (15% each), and When2Call tool calling (10%).
Artificial Analysis Decision Index v2.1 methodologyArtificial Analysis scored decision models on its Decision Index v1.0, the first version, and ran every evaluation itself. The index was the weighted average of AA-Omniscience-Decision accuracy and non-hallucination rate (12.5% each), HLE-MultiChoice, When2Call and WildGuardTest (25% each). Version 2.0 replaced it in November 2026; these are its last scores.
Artificial Analysis Decision Index v1.0Liquid AI reports 36.3 skill points for d1-3B and 6.8 for d1-omni-600M on Arts with the official Decision Index 0.2.1 scorer.
Open d1: Edge decision models for text, vision, and audioFastino reports GLiDE at 46.0 skill points on Arts and Human Taste in its Decision Index 0.2.1 run. The comparison quotes the published Jev 1.13.0 reference.
Fastino GLiDE launch and Decision Index resultsKaggle's ASCII Arena asks models to draw everyday subjects as ASCII art, and people vote for the better drawing.
BenchLMIrregular's domain-level evaluation of cybersecurity evasion capability.
Assessing GPT-5.6 SolIrregular's domain-level evaluation of network attack simulation capability.
Assessing GPT-5.6 SolIrregular's domain-level evaluation of vulnerability research and exploitation capability.
Assessing GPT-5.6 SolA Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardA Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardA Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardQuesma's BabaIsBench has LLM agents play the puzzle game Baba Is You through a text interface, where a level is won by pushing word tiles around to rewrite its rules. The score here is the first-attempt pass rate on the eight intro levels.
Baba Is Solved by Fable 5 and GPT-5.6 Sol, but at what cost? (Quesma)Scale AI's benchmark of whether AI agents stay safe while carrying out ordinary workplace requests. Agents handle 110 benign tasks in nine simulated workspaces, where the risk lies in how the work is done, such as leaking private details in a calendar invite. The score is safety pass^5: the share of tasks on which all five attempts were safe.
BARC-Bench: Benign Asks, Risky ConsequencesChallenging reasoning tasks on the 750-row sample in Perplexity's September 2026 decision panel. This is a sampled, provider-reported accuracy result with its own evaluation setup.
Perplexity Decider v1 27B model cardBelebele tests passage comprehension through questions with four answer options. Its parallel language variants let researchers study how reading performance changes across languages.
The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language VariantsPassage comprehension on the 500-row sample in Perplexity's September 2026 decision panel. This is a sampled, provider-reported accuracy result with its own evaluation setup.
Perplexity Decider v1 27B model cardQuesma's BinaryAudit asks AI agents to find backdoors and timebombs planted in compiled, stripped binaries of real server software, using the reverse engineering tools Ghidra and radare2. The score here is the share of detection runs that found the planted code.
We hid backdoors in ~40 MB binaries and asked AI + Ghidra to find them (Quesma)Independent agent runs on biological data investigations, split by human-solvable and human-difficult tasks.
Vals BioMysteryBenchThe 78 text-brief tasks of BlenderWorld, a benchmark where agents work inside Blender to turn real-world 3D creation requests into editable Blender projects and requested exports.
BlenderWorld: Benchmarking Visual Agents for 3D Creation from Real-World ReferencesBridgeBench publishes comparative model ratings across nine software-building axes, including reasoning, front end, back end, security, speed, and cost. Its scoring methodology is proprietary.
BenchLMFinding difficult information through web research. This table shows Mercor-run configurations for reference and is excluded from model rankings.
BrowseComp evaluation resultsFinding difficult information through web research. This table shows Mercor-run configurations for reference and is excluded from model rankings.
BrowseComp Extended evaluation resultsA vision evaluation reported by Thinking Machines Lab under Vision Interpretation and Creation in the Inkling 2 Mini launch table and model card.
Introducing Inkling 2 MiniA Center for AI Safety dashboard view summarizing text capabilities across HLE and TextQuests.
CAIS AI DashboardVals AI private question-answer benchmark over Canadian court cases.
CaseLaw v2Kaggle's Chart Arena has models turn a small dataset and a brief into one chart, and people vote for the better chart.
BenchLMChartQA tests visual and logical reasoning about charts. These own-provider reports keep Anthropic's relaxed-accuracy results separate from Zyphra scores whose metric and scorer are not disclosed.
ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical ReasoningAnswering descriptive and reasoning questions about research-paper charts. This table shows Mercor-run configurations for reference and is excluded from model rankings.
CharXiv evaluation resultsAnswering descriptive and reasoning questions about research-paper charts. This table shows Mercor-run configurations for reference and is excluded from model rankings.
CharXiv Extended evaluation resultsEpoch AI's Chess Puzzles benchmark gives models 100 novel chess positions as FEN text and asks for the single best next move, as judged by Stockfish.
Chess Puzzles (Epoch AI)Circa pairs polar questions with indirect answers and labels how each answer should be interpreted. It tests conversational decisions where the response does not simply say yes or no.
“I’d rather just go to bed”: Understanding Indirect AnswersIndirect yes/no answer interpretation on the 500-row sample in Perplexity's September 2026 decision panel. This is a sampled, provider-reported accuracy result with its own evaluation setup.
Perplexity Decider v1 27B model cardFastino reports GLiDE at 88.7% on CLadder in its Decision Index 0.2.1 run. The comparison quotes the published Jev 1.13.0 reference.
Fastino GLiDE launch and Decision Index resultsVals AI finance benchmark that asks whether a 10-K excerpt supports a claim, contradicts it, or says nothing about it. It compares small decision models with frontier language models.
ClaimBench leaderboard and methodologyFrontier models play the social deduction game Blood on the Clocktower against each other, one model as the town and one as the Demon's team. The rating combines catching liars (detection) with lying well (deception).
Clocktower Bench leaderboardPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMPublished Cloudflare and community decision-model configurations.
BenchLMCan language models reimplement real-world programs in another language?
Code MigrationReimplementing working programs in another language using the Pi agent harness and revised hidden tests.
Code Migration v1.1HeyGen's Code2Video tests whether a model can turn a detailed creative brief into a finished motion-graphics video by writing code.
BenchLMThe human-vote board for Code2Video: people compare two generated videos for the same prompt and pick the one that follows it better.
BenchLMCohere launch retrieval report.
BenchLMCohere launch retrieval report.
BenchLMCohere launch retrieval report.
BenchLMCohere launch retrieval report.
BenchLMCohere launch retrieval report.
BenchLMCohere launch retrieval report.
BenchLMCohere launch retrieval report.
BenchLMCohere launch retrieval report.
BenchLMCohere launch retrieval report.
BenchLMCohere launch retrieval report.
BenchLMCohere launch retrieval report.
BenchLMCohere launch retrieval report.
BenchLMCohere launch retrieval report.
BenchLMCohere launch retrieval report.
BenchLMCohere launch retrieval report.
BenchLMCohere launch retrieval report.
BenchLMCohere launch retrieval report.
BenchLMCohere launch retrieval report.
BenchLMCohere launch retrieval report.
BenchLMCohere launch retrieval report.
BenchLMCohere launch reranking report.
BenchLMCohere launch reranking report.
BenchLMCohere launch reranking report.
BenchLMCohere launch reranking report.
BenchLMCohere launch reranking report.
BenchLMCohere launch reranking report.
BenchLMCohere launch reranking report.
BenchLMCohere launch reranking report.
BenchLMCohere launch reranking report.
BenchLMCohere launch reranking report.
BenchLMCohere launch reranking report.
BenchLMCohere launch reranking report.
BenchLMCohere launch reranking report.
BenchLMConcept's benchmark of how well an AI agent understands the company it works in. Agents answer 205 questions drawn from one month of Concept's own Slack history: who wrote a message, who was tagged and who sent each reply. The score is overall accuracy.
Introducing Company Agent BenchQuesma's CompileBench hands an agent the unmodified source of an open-source project such as curl, jq or GNU coreutils and asks for a working binary. Tasks include static and musl builds, cross-compiling to Windows and ARM64, and reviving code from 2003.
CompileBench: can AI compile 22-year-old code? (Quesma)An instruction-following benchmark from Surge AI that tests whether a model satisfies many simultaneous, interacting constraints in one response.
Introducing Inkling 2 SmallContractNLI asks whether a contract entails, contradicts, or does not mention a hypothesis. The original task also requires evidence spans that support the classification.
ContractNLI: A Dataset for Document-level Natural Language Inference for ContractsContract hypothesis classification on the 510-row sample in Perplexity's September 2026 decision panel. This is a sampled, provider-reported accuracy result with its own evaluation setup.
Perplexity Decider v1 27B model cardA 1,000-query web development benchmark that judges generated front-end apps by deploying them and letting an evaluator agent use them, scoring functionality and aesthetics without reference implementations or test suites.
Cookie-Bench: Continuous On-screen Key Interaction Evaluation for Web GenerationKaggle's Corp Speak Arena has models rewrite 100 everyday sentences in corporate language, and people vote for the better rewrite.
BenchLMVals AI private benchmark for understanding long-context credit agreements.
CorpFin v2The full research-level physics evaluation set in CritPt v2 (Complex Research using Integrated Thinking - Physics Test), covering unpublished graduate and frontier physics challenges with code execution.
Introducing Inkling 2 MiniThe Hard subset of CritPt v2 isolating the most demanding multi-step unpublished physics research challenges.
Introducing Inkling 2 MiniFastino reports GLiDE at 92.6% on CRUXEval in its Decision Index 0.2.1 run. The comparison quotes the published Jev 1.13.0 reference.
Fastino GLiDE launch and Decision Index resultsA Vals AI computer-use benchmark in which agents play six commercial video games with a keyboard and mouse.
CUA-benchHardware design, verification, and comprehension tasks. This table shows Mercor-run configurations for reference and is excluded from model rankings.
CVDP evaluation resultsOpenAI's black-box, no-source variant of CVE-Bench v1 across 40 critical vulnerabilities.
CVE-BenchCan autonomous agents craft PoC inputs that trigger OSS-Fuzz vulnerabilities—and stop crashing after the fix?
CyberBenchNumber of CyScenarioBench scenarios completed successfully in at least one run.
Assessing GPT-5.6 SolAverage success rate across realistic, long-horizon cybersecurity scenarios.
Assessing GPT-5.6 SolDAYJOB is Surge AI's test of long professional assignments: 80 finance tasks and 50 healthcare tasks that an agent must finish to an expert checklist. This is the combined 130-task board on Kaggle.
BenchLMThe finance track of DAYJOB, a benchmark of end-to-end professional finance work that a model must complete as a full job task rather than a single question.
Introducing Inkling 2 SmallThe finance track of Surge AI's DAYJOB as run on Kaggle: 80 long finance assignments that an agent must complete to an expert checklist.
BenchLMThe healthcare track of Surge AI's DAYJOB: 50 long clinical, pharmacy, payer and quality-assurance assignments that an agent must complete to an expert checklist.
BenchLMFastino reports GLiDE at 64.81 skill points on Decision Index 0.2.1 in its Decision Index 0.2.1 run. The comparison quotes the published Jev 1.13.0 reference.
Fastino GLiDE launch and Decision Index resultsLiquid AI reports 48.57 skill points for d1-3B and 15.95 for d1-omni-600M on Decision Index 0.2.1 with the official Decision Index 0.2.1 scorer.
Open d1: Edge decision models for text, vision, and audioDecision Index 0.4 publishes chance-corrected arts scores for 209 evaluated systems.
Decision Index leaderboardDecision Index 0.4 publishes full scores for 209 evaluated systems. The full score weights private tests of the same skills at 60% and private tasks from new domains at 40%; public benchmarks no longer count toward it.
Decision Index leaderboardDecision Index 0.4 publishes chance-corrected knowledge scores for 209 evaluated systems.
Decision Index leaderboardDecision Index 0.4 publishes chance-corrected language scores for 209 evaluated systems.
Decision Index leaderboardDecision Index 0.4 publishes chance-corrected retrieval scores for 209 evaluated systems.
Decision Index leaderboardDecision Index 0.4 publishes chance-corrected tools scores for 209 evaluated systems.
Decision Index leaderboardWriting research reports from web evidence. This table shows Mercor-run configurations for reference and is excluded from model rankings.
DeepResearch Bench II evaluation resultsA long-horizon software engineering benchmark from Datacurve for measuring frontier coding agents on original tasks drawn from active open-source repositories.
DeepSWE benchmark blogResolving software issues on the DeepSWE v1.1 task set. This table shows Mercor-run configurations for reference and is excluded from model rankings.
DeepSWE v1.1 evaluation resultsTests scientific discovery by asking agents to infer undisclosed rules across 70 interactive text-based games.
DiG-bench: Discovery in GamesA financial due-diligence agent benchmark from Thoughtful, scoring how well a model researches and analyzes a company the way an investment analyst would.
Introducing Inkling 2 SmallMeasures crisis-support replies against clinician-written rubrics. These published results are excluded from overall and category rankings.
DistressBench: Evaluating Multi-Dimensional Crisis Support in Large Language ModelsDocVQA asks questions about a single document image. This page shows four historical Anthropic reports on the test split, with ANLS, zero-shot setup, report dates, and source cohorts kept visible.
DocVQA: A Dataset for VQA on Document ImagesA Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardMina Nageh's Kaggle community benchmark "A Dynamic Benchmark for Social Cognition in AGI" has nine tasks on social cognition: understanding emotions, modeling the self and other people, reading intent and tone, and holding to constraints under social pressure.
BenchLMEpoch AI's EBR-bench tests whether models get better at an unfamiliar game with practice: they play the campaign card game Earthborne Rangers repeatedly, take notes, and are scored on later playthroughs.
Earthborne Rangers (EBR-bench), Epoch AIKaggle's ELI5 Arena asks models to explain 100 topics the way you would to a five-year-old, and people vote for the better explanation.
BenchLMExcel financial modeling from a template or from scratch, with updated grading in v1.1.
Excel Modeling Benchmark v1.1A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardAn enterprise-operations agent benchmark from Surge AI set in a simulated company (CoreCraft), where agents complete multi-step business tasks across internal tools and records.
Introducing Inkling 2 SmallEpoch Automation Reports runs AI agents on open-ended tasks taken from Epoch AI's own work, such as graphic design, data insights and AI data center research, and has human graders score the output against Epoch's internal standards.
Epoch Automation ReportsA cybersecurity benchmark for evaluating LLM agents on full-control V8 exploit synthesis using 16 measured exploit capability flags.
ExploitBenchOpenAI's internal port of ExploitBench restricted to V8 and Chrome vulnerabilities disclosed in June to August 2026, to limit contamination from historical exploits.
GPT-6 Cyber System CardA hardened variant of ExploitGym that adds address-space randomization and stack canaries to userspace targets, enables the V8 sandbox and uses stricter kernel configurations where supported.
ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?An alignment test for cyber agents: hard ExploitGym tasks are paired with an easy-to-exploit auxiliary service (such as a package registry or object store) that would reveal the flag but is outside the authorized scope. Lower scores mean the model stays in scope more often.
GPT-6 Cyber: Advancing the frontier of aligned cybersecurity agentsTests schema-guided extraction from enterprise documents, including complete records, missing values, and source evidence.
ExtractBench: A Benchmark for Schema-Guided Enterprise Document ExtractionAdonis Singh's visual reasoning benchmark: 100 generated images, each with one question whose answer is a single number. Models count the differences between two nearly identical panels, count closed loops in a tangle of curves, count the connected parts of a graph drawing, pick the tile that completes a 4x4 pattern, and count the turns along a maze path. Scoring is exact match.
EyeBench v3: a vision exam for frontier modelsThe official FACTS suite combines document grounding, visual factuality, parametric recall, and search-assisted answering. The published average stays separate from its component scores.
FACTS Benchmark Suite methodologyThe FACTS suite average as Kaggle published it on October 20, 2026, over Grounding v2, Multimodal v1, Parametric v1 and Search v2, before the November re-run.
FACTS Benchmark Suite (October 20 run) methodologyThe archived FACTS Grounding V1 protocol tests factual responses grounded in supplied documents. Its original judge setup and results remain separate from Grounding V2.
FACTS Grounding V1 methodologyFACTS Grounding v2 checks whether long-form answers remain supported by supplied documents. Its revised judge ensemble and public/private results remain separate from version 1.
FACTS Grounding v2 methodologyFACTS Grounding v2 at task revision 3 as captured on October 20, 2026, before Kaggle re-ran the board at revision 4.
FACTS Grounding v2 (October 20 run) methodologyFACTS Multimodal checks whether a model answers a question about an image without leaving out essential facts or contradicting them. It is the image part of the FACTS suite.
BenchLMThe first Kaggle version of FACTS Multimodal, with runs from November 2025 to October 2026. Kaggle replaced it with version 2 in November 2026.
BenchLMFACTS Parametric asks hard factoid questions from real Google searches and checks whether a model answers them from memory, without tools. It is the parametric-knowledge part of the FACTS suite.
BenchLMThe archived FACTS Search V1 protocol measures factual answering with Brave Search and a Gemini 2.0 Flash grader. Its scores remain separate from Search V2.
FACTS Search V1 methodologyFACTS Search V2 measures factual answering with a shared Brave Search tool. Overall, public, private, hedging, search-count, attempted-accuracy, and F1 measurements remain distinct.
FACTS Search V2 methodologyFACTS Search V2 at task revision 6 as captured on October 20, 2026, before Kaggle re-ran the board at revision 7.
FACTS Search V2 (October 20 run) methodologyIBM Research's FailureSensorIQ asks which sensors reveal which equipment failures across 10 kinds of industrial asset. It is the first published part of AssetOpsBench.
BenchLMAn archived Vals benchmark covering retrieval, numerical reasoning, financial modeling, market analysis, earnings, trends, and adjustments.
Finance Agent v1.1Financial research with tools, revised questions and rubrics, a new judge jury, and stricter treatment of hedged answers.
Finance Agent v2.1Can AI coach a household through a real financial problem?
Financial Coaching Bench leaderboard and methodologyClassify financial news sentences as positive, negative, or neutral. FinancialPhraseBank tests whether a model interprets financial context rather than treating every favorable-sounding word as positive.
Good Debt or Bad Debt: Detecting Semantic Orientations in Economic TextsFinancial sentiment classification on the 999-row sample in Perplexity's September 2026 decision panel. This is a sampled, provider-reported accuracy result with its own evaluation setup.
Perplexity Decider v1 27B model cardThe hardest task of FinSearchComp, an open financial search and reasoning benchmark for end-to-end agents: multi-step investigations over historical market and company data that mirror a financial analyst's research workflow.
FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and ReasoningShare of patched Firefox 147 JavaScript-engine targets for which the model produced a working arbitrary-code-execution exploit.
Claude Fable 5 and Claude Mythos 5A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardThe public adversarial split of Scale AI FORTRESS, measuring safe refusal and mitigation on high-risk adversarial safety prompts.
Introducing Inkling 2 MiniThe public benign split of Scale AI FORTRESS, measuring whether a model answers harmless edge-case prompts without unnecessary refusal.
Introducing Inkling 2 MiniValen Labs' benchmark of AI agents doing machine learning work in computational biology: 25 tasks in antibody development, designed binders, protein engineering, structural biology and functional genomics. Each agent builds a model from experimental data in an offline sandbox, and its prediction script is scored on held-out measurements, from 0 (a simple baseline) to 100 (a perfect answer), over three runs per task.
FrontierBioML: Evaluating agents on machine learning tasks in computational biologyIndependent evaluation of AI agents against vulnerable real-world systems in dynamic environments.
FrontierCyberLetta's benchmark of how good the memory is that an AI model writes from past agent work. The model studies a corpus of earlier agent transcripts and writes memory files; the score is the share of 15 memory-dependent coding and knowledge-work tasks that a task agent then fully passes with that memory, averaged over three runs.
FrontierMemory 1.0: Benchmarking continual learning through memoryThe first FrontierMemory table from Letta's launch post: 7 memory models on 15 memory-dependent tasks, scored by full-pass rate over three runs, on the first transcript corpus.
FrontierMemory 1.0: Benchmarking continual learning through memoryFrontierPhysics, from BenchFlow, gives an AI agent a research project written by a working physicist and checks whether it gets the physics right and writes it up well enough to publish. It has 56 tasks across nine fields of physics, and each one is run three times.
FrontierPhysics: Evaluating Agents for End-to-End Physics Research (arXiv, November 4, 2026)Long engineering projects scored by continuous task-specific metrics. This table shows Mercor-run configurations for reference and is excluded from model rankings.
FrontierSWE v2 evaluation resultsEpoch AI's Furniture Assembly benchmark shows models photos of IKEA furniture builds with the official manual and asks whether the build is correct so far and, if not, which step went wrong.
Furniture Assembly Benchmark (Epoch AI)Kaggle Game Arena fits one Bradley-Terry rating to results across strategic games. Matches played and games covered accompany the rating; it is not a percentage or a chess Elo rating.
Game Arena methodologyKaggle Game Arena's Go board ranks models by head-to-head games of Go on a 13 by 13 board.
BenchLMAn agentic coding benchmark that asks models to build a Game Boy Advance emulator from scratch and grades emulator behavior against procedural, audio, and gameplay tests.
GBA-EvalProducing workplace deliverables from reference files. This table shows Mercor-run configurations for reference and is excluded from model rankings.
GDPval evaluation resultsProducing workplace deliverables from reference files. This table shows Mercor-run configurations for reference and is excluded from model rankings.
GDPval Extended evaluation resultsTaking messy genomics datasets through quality control, modeling and diagnostics to a verifiable answer. This table shows Mercor-run configurations for reference and is excluded from model rankings.
GeneBench-Pro evaluation resultsKaggle reports an average across 16 language dimensions of Global MMLU Lite, with separate culturally sensitive and culturally agnostic results.
BenchLMGraduate-level science multiple-choice questions. This table shows Mercor-run configurations for reference and is excluded from model rankings.
GPQA Diamond evaluation resultsKaggle's few-shot GPQA Diamond rows at task version 4, as captured on October 20, 2026.
GPQA: A Graduate-Level Google-Proof Q&A BenchmarkKaggle's few-shot run of the GPQA Diamond set at task version 5, with the October 20 run kept below it.
GPQA: A Graduate-Level Google-Proof Q&A BenchmarkKaggle's zero-shot run of the 198-question GPQA Diamond set, taken from the GPQA suite board. It stays separate from provider, Vals, Artificial Analysis, and Mercor Diamond results.
GPQA: A Graduate-Level Google-Proof Q&A BenchmarkKaggle's few-shot run of the 448-question GPQA main set.
GPQA: A Graduate-Level Google-Proof Q&A BenchmarkKaggle's zero-shot GPQA main-set rows at task version 4, as captured on October 20, 2026.
GPQA: A Graduate-Level Google-Proof Q&A BenchmarkKaggle's zero-shot run of the 448-question GPQA main set, kept apart from the harder Diamond subset.
GPQA: A Graduate-Level Google-Proof Q&A BenchmarkBreadth-first-search graph traversal on the 256K-token context slice of GraphWalks, testing multi-hop reasoning over long contexts.
Introducing Inkling 2 MiniA visual grounding benchmark of 1,005 examples that tests whether multimodal models localize objects across four dimensions: discriminative look-alikes, spatial relations, occluded or tiny targets, and ungroundable rejection queries.
GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional EvaluationProducing legal work across practice areas and contract tasks. This table shows Mercor-run configurations for reference and is excluded from model rankings.
Harvey LAB evaluation resultsTests an agent's ability to complete legal work using documents, spreadsheets, presentations, and file-system tools
Harvey's Legal Agent BenchmarkArchived Vals legal-agent results before the tracked-changes grading fix.
HLAB v1Answering clinicians in care consults, documentation and medical research conversations. This table shows Mercor-run configurations for reference and is excluded from model rankings.
HealthBench Professional evaluation resultsA Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardAnswering difficult academic questions across many subjects. This table shows Mercor-run configurations for reference and is excluded from model rankings.
HLE evaluation resultsAnswering difficult academic questions across many subjects. This table shows Mercor-run configurations for reference and is excluded from model rankings.
HLE Extended evaluation resultsHLE-MultiChoice is one component of the Artificial Analysis Decision Index v1.0, run by Artificial Analysis. It measures accuracy on the 513 text-only multiple-choice Humanity's Last Exam questions, each posed as one choice among lettered answers.
Artificial Analysis Decision Index v1.0A reward-hacking and specification-gaming safety benchmark from Goodhart Labs that pairs difficult agent tasks with exploitable graders to measure how often models cheat instead of solving the task honestly. Lower scores are better.
HoneyBench v0.2Predicts response distributions for specified groups of U.S. adults on closed-form questions about behavior and preferences.
Human Behavior Bench on KaggleA Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardA Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardThe first published Humanity’s Last Exam results from early 2025, kept by Scale as a preview board. The current HLE tables are on the same page. BenchLM shows it as display-only reference data, excluded from rankings. Scale has not added a row to this board since April 2025.
Scale Labs leaderboardThe first published text-only Humanity’s Last Exam results from early 2025, kept by Scale as a preview board. The current text-only tables are on the same page. BenchLM shows it as display-only reference data, excluded from rankings. Scale has not added a row to this board since April 2025.
Scale Labs leaderboardA Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardHSS tests what a model can infer from images and silent videos across temporal, spatial, social, and abstract reasoning. We show the published results as display-only evidence, excluded from overall and category rankings.
Humanity's Sixth Sense paperVals AI and Talent Mobility Fund benchmark of expert-written U.S. immigration law questions, scored against expert rubrics.
Immigration Bench leaderboard and methodologyInfographicVQA asks questions about infographic images. These two own-provider InfoVQA test reports name the split but do not disclose the metric or scorer; their printed scores remain separate source facts.
BenchLMA local-AI efficiency metric from Stanford Hazy Research Lab and Scaling Intelligence Lab measuring task accuracy delivered per joule of energy consumed per query on local hardware.
Intelligence per Watt: Measuring Intelligence Efficiency of Local AIArchived Vals AI results on 2024 and 2025 International Olympiad in Informatics programming tasks.
Vals IOI v1Public-hard decision accuracy on the 101-row sample in Perplexity's September 2026 decision panel. This is a sampled, provider-reported accuracy result with its own evaluation setup.
Perplexity Decider v1 27B model cardKaggle's Joke Arena asks models for an original joke on 100 topics, and people vote for the funnier one.
BenchLMJudgeBench evaluates whether a judge can identify the better response in pairs with objective correctness labels. It tests judgment on challenging questions rather than agreement with writing style preferences.
JudgeBench: A Benchmark for Evaluating LLM-Based JudgesResponse-pair judgment on the 350-row sample in Perplexity's September 2026 decision panel. This is a sampled, provider-reported accuracy result with its own evaluation setup.
Perplexity Decider v1 27B model cardA behavioral benchmark that tests psychological safety across sixteen adversarial multi-turn conversations covering emotional safety, identity, sycophancy, and value integrity.
KindBench methodologyLiquid AI reports 23.8 skill points for d1-3B and 8.3 for d1-omni-600M on Knowledge with the official Decision Index 0.2.1 scorer.
Open d1: Edge decision models for text, vision, and audioFastino reports GLiDE at 62.9 skill points on Knowledge and Reasoning in its Decision Index 0.2.1 run. The comparison quotes the published Jev 1.13.0 reference.
Fastino GLiDE launch and Decision Index resultsKaggle's Landing Page Arena has models build a one-page website for a small business from a client brief, and people vote for the better page.
BenchLMKaggle's Landmark Arena has models build 20 famous landmarks, including the Seven Wonders of the Ancient World, as single-page three.js web apps, and people vote for the better scene.
BenchLMLiquid AI reports 56.4 skill points for d1-3B and 12.9 for d1-omni-600M on Language with the official Decision Index 0.2.1 scorer.
Open d1: Edge decision models for text, vision, and audioFastino reports GLiDE at 63.8 skill points on Language Understanding in its Decision Index 0.2.1 run. The comparison quotes the published Jev 1.13.0 reference.
Fastino GLiDE launch and Decision Index resultsEvaluating agents on legal research tasks across diverse areas of US law
Legal Research BenchVals AI legal benchmark with issue, rule, conclusion, interpretation, and rhetoric task views.
LegalBenchA frequently refreshed benchmark with objective scoring across reasoning, coding, agentic coding, mathematics, data analysis, language, and instruction following.
LiveBench: A Challenging, Contamination-Free LLM BenchmarkSynthesizing information across long collections of documents. This table shows Mercor-run configurations for reference and is excluded from model rankings.
Long Context Reasoning (AA-LCR v1.1) evaluation resultsSynthesizing information across long collections of documents. This table shows Mercor-run configurations for reference and is excluded from model rankings.
Long Context Reasoning (AA-LCR) Extended evaluation resultsA Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardKaggle's run of the 500-problem MATH subset, version 2 of the board, separate from provider, Vals, and Artificial Analysis MATH-500 rows.
Measuring Mathematical Problem Solving With the MATH DatasetMathVista tests mathematical reasoning about images. This table shows published accuracy on the testmini split, with each source configuration kept separate. The results are display only and do not enter overall or category rankings.
MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual ContextsMathVista tests mathematical reasoning about images. This table shows published accuracy on the test split, with each source configuration kept separate. The results are display only and do not enter overall or category rankings.
MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual ContextsunLAP's MaxxBench takes 89 DeepSWE software engineering tasks and changes the instructions or starting code while keeping the core problem, to test whether coding agents adapt to new requirements or fall back on a memorized solution.
Why MaxxBench (unLAP research post)A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardVals AI healthcare benchmark for whether models can support the medical billing process.
MedCodeVals AI healthcare benchmark for whether models can support doctors with administrative work.
MedScribeMedical multiple-choice reasoning over images and clinical details. This table shows Mercor-run configurations for reference and is excluded from model rankings.
MedXpertQA MM evaluation resultsMedical multiple-choice reasoning over images and clinical details. This table shows Mercor-run configurations for reference and is excluded from model rankings.
MedXpertQA MM Extended evaluation resultsPublished chatbot failure rates across urgent safety, health advice, and relationship boundaries in simulated teen conversations.
Mental Health BenchTests mathematical reasoning on the Bengali translation of 250 grade-school math problems from GSM8K.
Language Models are Multilingual Chain-of-Thought ReasonersMirrorCode, built by Epoch AI with METR, asks AI agents to reimplement entire command-line programs from scratch without their source code, matching the original's output exactly on end-to-end and held-out tests.
MirrorCode (Epoch AI)Kaggle's run of the original 57-subject MMLU, separate from provider-reported MMLU rows.
Measuring Massive Multitask Language UnderstandingKaggle's run of MMLU-Pro, the ten-option, reasoning-heavy successor to MMLU, kept apart from provider, Vals, Artificial Analysis, and Arcee rows.
MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding BenchmarkAnswering knowledge questions with ten answer choices. This table shows Mercor-run configurations for reference and is excluded from model rankings.
MMLU-Pro evaluation resultsAnswering questions that require both visual and textual understanding. This table shows Mercor-run configurations for reference and is excluded from model rankings.
MMMU-Pro evaluation resultsAnswering questions that require both visual and textual understanding. This table shows Mercor-run configurations for reference and is excluded from model rankings.
MMMU-Pro Extended evaluation resultsVals AI benchmark for mortgage and tax document reasoning, including semantic and numerical extraction task views.
MortgageTaxA multi-image understanding benchmark of 2,600 multiple-choice questions across 11,264 images and 12 tasks, pairing every answerable question with a minimally altered unanswerable variant.
MuirBench: A Comprehensive Benchmark for Robust Multi-image UnderstandingMulti-IF tests instruction following over three turns in eight languages. This selected historical author report preserves each turn score, dated API identifier, and generation settings for four models.
Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions FollowingA Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardA Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardA Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardEpoch AI's Mystery Game Puzzles benchmark asks models for the single best next move in 100 positions from a puzzle variant of a well-known game whose identity Epoch keeps secret.
Mystery Game Puzzles (Epoch AI)Can agents rediscover sealed mathematical mechanisms through bounded experiments?
Vals MysteryMechanismSelected author accuracy curves for long-context retrieval with minimal wording overlap. Original NoLiMa stays separate from Hard, and 64K/128K use fewer placements. The displayed score is 32K accuracy, outside overall and category rankings.
NoLiMa: Long-Context Evaluation Beyond Literal MatchingSelected reasoning-model accuracy curves on the ten hardest NoLiMa question-needle pairs. Exact API snapshots and effort remain undisclosed. The displayed score is 32K accuracy; these reports stay outside overall and category rankings.
NoLiMa: Long-Context Evaluation Beyond Literal MatchingLech Mazur's NYT Connections benchmark gives a model 940 puzzles from the New York Times word game, each with up to four extra trick words, and asks it to sort the words into four groups of four in one try. It tests word knowledge, wordplay and resisting plausible decoys.
NYT Connections LLM Benchmark (Lech Mazur, GitHub)Kaggle's Opening Lines Arena has models write the opening of a novel for 50 premises in 10 genres, and people vote for the better opening.
BenchLMOne OpenWorldEngineer-Bench task: redesign a small electric aircraft, mechanisms included, for timed laps, heavy cargo and precision drops, measuring it only through a limited budget of tests. Mean of 8 attempts, from the starting design (0) to a perfect reward (100).
Design/Build/Fly Aircraft (OpenWorldEngineer-Bench)Mercor's benchmark of AI agents on 26 long, open-ended engineering tasks in science, software, ML and engineering, built by domain experts. Each model makes 8 attempts per task, of 4 to 18 hours each, and the score is the mean task score on a scale from the starting point (0) to the best result the task allows (100).
OpenWorldEngineer-BenchQuesma's OTelBench asks coding agents to add OpenTelemetry tracing, metrics and logging to small existing services in 11 programming languages. The score is the share of runs where the service then emits correct telemetry.
Benchmarking OpenTelemetry: can AI trace your failed login? (Quesma)A 29-model comparison of how well models use Parallel Search Fast and Extract to answer web research questions.
BenchLMTests document parsing across tables, charts, content faithfulness, semantic formatting, and visual grounding.
ParseBench: A Document Parsing Benchmark for AI AgentsA vulnerability-repair benchmark that asks models to patch real security vulnerabilities in open-source code and checks the fix with tests.
Introducing Inkling 2 SmallKaggle's Pelican Arena asks every model for the same thing, an animated SVG of a pelican riding a bicycle, and people vote for the better drawing.
BenchLMPerplexity reports accuracy across 11 decision tasks on a fixed 7,210-row panel. The overall percentage combines unequal task sample sizes, so a larger task contributes more than a smaller task.
Perplexity Decider v1 27B model cardLong, end-to-end engineering projects: an agent works through a full project folder, finds the governing codes and physical limits, and produces deliverables a licensed engineer would sign off on.
PLE: Physical-Engineering's Last ExamAn agentic GPU kernel-optimization benchmark of 69 hard tasks, adapted by Thinking Machines Lab for consistent cross-model evaluation. Each model gets four independent attempts with a 10-hour budget, and the score is the area under the mean tasks-solved curve, out of a maximum of 690 task-hours.
Introducing Inkling 2 SmallA Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardA Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardCan language models rebuild programs from scratch?
ProgramBenchReconstructing a program from its executable behavior and documentation. This table shows Mercor-run configurations for reference and is excluded from model rankings.
ProgramBench evaluation resultsVals AI automated theorem-proving benchmark.
ProofBenchA Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardCan AI help people navigate SNAP benefits?
Public Benefits Bench v1Can AI help people navigate SNAP benefits?
Public Benefits Bench v1.1RAGTruth annotates unsupported or contradictory content in answers generated with retrieved evidence. It supports evaluation of hallucination detectors across question answering, summarization, and data-to-text tasks.
RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language ModelsUnsupported-claim detection on the 1,500-row sample in Perplexity's September 2026 decision panel. This is a sampled, provider-reported accuracy result with its own evaluation setup.
Perplexity Decider v1 27B model cardA Ramp Labs public leaderboard for long-horizon accounting agent work. BenchLM shows it as display-only reference data and excludes it from model rankings.
Ramp Accounting BenchNegotiating and revising contracts across multiple turns. This table shows Mercor-run configurations for reference and is excluded from model rankings.
RedlineBench evaluation resultsA Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardOpenAI's benchmark of whether a model can build a working exploit for a known vulnerability and use it against a remote target, on real applications such as GitLab, PostgreSQL and Jenkins deployed in a controlled environment.
GPT-6 Cyber: Advancing the frontier of aligned cybersecurity agentsA deep-research agent benchmark that pairs realistic, domain-diverse prompts with more than 2,500 expert-written rubrics scoring factual grounding, reasoning soundness and clarity of long-form, evidence-backed answers.
ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research AgentsLiquid AI reports 52.8 skill points for d1-3B and 35.0 for d1-omni-600M on Retrieval with the official Decision Index 0.2.1 scorer.
Open d1: Edge decision models for text, vision, and audioFastino reports GLiDE at 60.9 skill points on Retrieval and Classification in its Decision Index 0.2.1 run. The comparison quotes the published Jev 1.13.0 reference.
Fastino GLiDE launch and Decision Index resultsTests agents controlling a simulated robot arm and humanoid through motor commands, sensor readings, and camera images.
Robo-bench leaderboard and methodologySelected original RULER context sweeps and separate IBM Granite 4.2 reports. The table keeps each source cohort and context length explicit; 128K is the displayed score, and the results do not enter rankings.
RULER: What's the Real Context Size of Your Long-Context Language Models?An agentic coding benchmark where models use a TypeScript SDK to play a RuneScape-like environment and optimize skill-training performance.
RuneBenchSolving scientific programming subproblems with executable checks. This table shows Mercor-run configurations for reference and is excluded from model rankings.
SciCode evaluation resultsA separate Kaggle SciCode track for main research problems without background notes.
SciCode: A Research Coding Benchmark Curated by ScientistsA separate Kaggle SciCode track for main research problems with supplied scientific background.
SciCode: A Research Coding Benchmark Curated by ScientistsA separate Kaggle SciCode track for decomposed scientific coding problems without background notes.
SciCode: A Research Coding Benchmark Curated by ScientistsA separate Kaggle SciCode track for decomposed scientific coding problems with supplied background.
SciCode: A Research Coding Benchmark Curated by ScientistsA Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardSciUniverse Level 1 tests whether models carry out scientific work across chemistry, biology, and materials science. Its 92 tasks include physical experiments, simulated lab work, and measurement analysis, with Pass@1 averaged equally across 17 task families.
BenchLMShare of SCONE smart-contract vulnerabilities exploited on the 12-task post-cutoff set.
Evaluating frontier models on software exploitationSimulated value captured across the SCONE post-cutoff smart-contract set.
Evaluating frontier models on software exploitationScreenSpot Pro GUI grounding on high-resolution professional screenshots, with the model allowed to run Python (for example to crop or zoom) before answering.
Introducing Inkling 2 SmallScale’s SEAL board that counted harmful replies to 1,000 human-written adversarial prompts. Fewer violations rank higher. BenchLM shows it as display-only reference data, excluded from rankings. Scale marked this board deprecated as of January 2025 and no longer adds models.
Scale Labs leaderboardToolComp-Chat: 485-prompt benchmark subset that needs models to chain a Python interpreter and web search calls to reach a final answer. BenchLM shows it as display-only reference data, excluded from rankings. Scale has not added a row to this board since March 2025.
Scale Labs leaderboardToolComp-Enterprise: the subset of Scale’s 485-prompt tool-use benchmark that needs models to chain calls across 11 enterprise tools. BenchLM shows it as display-only reference data, excluded from rankings. Scale has not added a row to this board since March 2025.
Scale Labs leaderboardScale's SEAL leaderboard for Arabic: expert raters judge how well models answer about 1,000 prompts written by native Arabic speakers. BenchLM shows it as display-only reference data, excluded from rankings. Scale marked this board deprecated as of March 2025 and no longer adds models.
Scale Labs leaderboardScale's SEAL leaderboard for Chinese: expert raters judge how well models answer about 1,000 prompts written by native Chinese speakers. BenchLM shows it as display-only reference data, excluded from rankings. Scale marked this board deprecated as of March 2025 and no longer adds models.
Scale Labs leaderboardScale’s SEAL coding board: expert reviewers compared model answers to 1,000 prompts across programming languages, debugging, optimization, and documentation tasks. BenchLM shows it as display-only reference data, excluded from rankings. Scale marked this board deprecated as of March 2025 and no longer adds models.
Scale Labs leaderboardScale’s SEAL board for precise instruction following on 1,054 private prompts with detailed, specific commands. BenchLM shows it as display-only reference data, excluded from rankings. Scale marked this board deprecated as of January 2025 and no longer adds models.
Scale Labs leaderboardScale's SEAL leaderboard for Japanese: expert raters judge how well models answer about 1,000 prompts written by native Japanese speakers. BenchLM shows it as display-only reference data, excluded from rankings. Scale marked this board deprecated as of March 2025 and no longer adds models.
Scale Labs leaderboardScale's SEAL leaderboard for Korean: expert raters judge how well models answer about 1,000 prompts written by native Korean speakers. BenchLM shows it as display-only reference data, excluded from rankings. Scale marked this board deprecated as of March 2025 and no longer adds models.
Scale Labs leaderboardScale’s SEAL math board on GSM1k, a new set of grade-school word problems written to mirror GSM8K without its contamination. BenchLM shows it as display-only reference data, excluded from rankings. Scale marked this board deprecated as of January 2025 and no longer adds models.
Scale Labs leaderboardScale's SEAL leaderboard for Spanish: expert raters judge how well models answer about 1,000 prompts written by native Spanish speakers. BenchLM shows it as display-only reference data, excluded from rankings. Scale marked this board deprecated as of March 2025 and no longer adds models.
Scale Labs leaderboardCybersecurity benchmark for agentic vulnerability analysis and exploit-oriented security tasks.
Introducing GPT-5.6Kaggle's run of OpenAI's original SimpleQA, separate from provider-reported SimpleQA rows and from the Verified subset.
Measuring short-form factuality in large language modelsSimpleQA Verified measures short-form factual recall on 1,000 prompts without tools. This dated author-paper table ranks source models by F1 and keeps accuracy, accuracy given attempted, attempted, and hedged separate.
SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric KnowledgeKaggle's continuously updated SimpleQA Verified board, with 56 configurations against the 13 rows transcribed from the author paper.
SimpleQA VerifiedSkillsBench, built by BenchFlow, measures how much agent Skills (instruction folders an agent can load) help on 87 tasks in eight domains.
SkillsBenchBenchFlow's own SkillsBench table: each agent setup runs the 87 tasks of version 1.1 with Skills, the instruction folders an agent can load, and most also run them without.
SkillsBench leaderboardEpoch AI's SpireClimb has models play Slay the Spire and Slay the Spire 2 at each game's maximum difficulty through a computer-use harness, as an end-to-end test of computer use, long-horizon reasoning and tactical play.
SpireClimb (Epoch AI)A contamination-resistant agent benchmark for reverse engineering real-world binaries.
SRE BenchA contamination-controlled benchmark that asks agents to reverse engineer compiled binaries without source access and satisfy six task-specific objectives per challenge.
GPT-6 Astra System CardStealthMark, from Stealth Models, scores a model out of 100 on eight original tasks: four on physical reasoning and multi-step planning, and four on producing a precise picture from a brief. The exact instructions are kept private so models can't be tuned to them.
StealthMark: The AI Benchmark Explained (Stealth Models)StrongREJECT scores how well a model refuses forbidden requests; this setting sends the harmful prompts without any jailbreak wrapper. Higher scores mean safer responses.
A StrongREJECT for Empty JailbreaksAnswering advanced chemistry questions. This table shows Mercor-run configurations for reference and is excluded from model rankings.
SUPERChem evaluation resultsAnswering questions about software repositories with an agent. This table shows Mercor-run configurations for reference and is excluded from model rankings.
SWE Atlas - Codebase QnA evaluation resultsAnswering questions about software repositories with an agent. This table shows Mercor-run configurations for reference and is excluded from model rankings.
SWE Atlas - Codebase QnA Extended evaluation resultsA Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardA Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardA Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardA Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardA Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardA Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardA Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardResolving repository issues across programming languages. This table shows Mercor-run configurations for reference and is excluded from model rankings.
SWE-bench Multilingual evaluation resultsJavaScript issue resolution where bug reports depend on screenshots, evaluated by Vals AI with no agent internet access.
SWE-bench Multimodal on Vals AIScale’s first SWE-Bench Pro leaderboard, with public and private (commercial) dataset lanes. Scale now publishes SWE-Bench Pro V2 on a separate board. BenchLM shows it as display-only reference data, excluded from rankings.
Scale Labs leaderboardScale’s first SWE-Bench Pro leaderboard, with public and private (commercial) dataset lanes. Scale now publishes SWE-Bench Pro V2 on a separate board. BenchLM shows it as display-only reference data, excluded from rankings.
Scale Labs leaderboardA Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardA Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardEditing repositories to resolve verified GitHub issues. This table shows Mercor-run configurations for reference and is excluded from model rankings.
SWE-bench Verified evaluation resultsEditing repositories to resolve verified GitHub issues. This table shows Mercor-run configurations for reference and is excluded from model rankings.
SWE-bench Verified Extended evaluation resultsA long-horizon software engineering benchmark from Abundant AI with multi-hour tasks spanning library reproductions, full-stack product clones, and ML engineering.
SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?The smaller SWT-bench split: 276 GitHub issues from SWE-bench Lite, scored the same way as Verified by the share of issues whose generated tests fail on the original code and pass after the fix.
SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code AgentsA test generation benchmark from LogicStar AI and ETH Zurich's SRI Lab. For each of 433 real GitHub issues from SWE-bench Verified, an agent writes a test that reproduces the reported bug: the test has to fail on the original code and pass once the issue's fix is applied. The score is the share of issues reproduced this way.
SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code AgentsTabFact asks whether a statement is supported or refuted by a Wikipedia table. It tests decisions that combine language interpretation with structured evidence.
TabFact: A Large-scale Dataset for Table-based Fact VerificationTable-based fact verification on the 500-row sample in Perplexity's September 2026 decision panel. This is a sampled, provider-reported accuracy result with its own evaluation setup.
Perplexity Decider v1 27B model cardHandling simulated banking-support conversations with tools. This table shows Mercor-run configurations for reference and is excluded from model rankings.
Tau3-Banking evaluation resultsAgent evaluation on research-grade US tax questions using the Tax Agent Bench harness.
Tax Agent BenchA Vals-created set of questions and responses to tax questions
TaxEval v2State-of-the-art set of difficult terminal-based tasks
Terminal-Bench 2.1Completing tasks in command-line environments. This table shows Mercor-run configurations for reference and is excluded from model rankings.
Terminal-Bench 2.1 evaluation resultsCompleting tasks in command-line environments. This table shows Mercor-run configurations for reference and is excluded from model rankings.
Terminal-Bench 2.1 Extended evaluation resultsCompleting command-line tasks on the Terminal-Bench 4.0 set. This table shows Mercor-run configurations for reference and is excluded from model rankings.
Terminal-Bench 4.0 evaluation resultsVals AI’s independent run of the Terminal-Bench 4.0 frontier terminal suite, with per-domain splits.
Vals Terminal-Bench 4.0Vals AI’s fixed-harness, single-trial run of 70 expert-authored scientific research workflows.
Vals Terminal-Bench ScienceTextVQA requires reading and reasoning about text in images. These two own-provider reports use the validation split, with metric and scorer left undisclosed in the original model cards.
Towards VQA Models That Can ReadShare of runs that completed the 32-step cyber range within the 100-million-token limit.
UK AISI and CAISI Preliminary Assessment of Kimi K3's Cyber CapabilitiesAverage step reached on a 32-step long-horizon cyber range.
UK AISI and CAISI Preliminary Assessment of Kimi K3's Cyber CapabilitiesA Vals AI agent benchmark that gives each system five days to build and run a space program in Kerbal Space Program.
Time Horizon Index: KSPAn independent agentic-intelligence evaluation from Toloka using private simulated workflows and a pass^5 metric.
Toloka ArenaLiquid AI reports 74.5 skill points for d1-3B and 15.1 for d1-omni-600M on Tools with the official Decision Index 0.2.1 scorer.
Open d1: Edge decision models for text, vision, and audioFastino reports GLiDE at 83.5 skill points on Tools and Automation in its Decision Index 0.2.1 run. The comparison quotes the published Jev 1.13.0 reference.
Fastino GLiDE launch and Decision Index resultsBinary truthfulness classification on the 500-row sample in Perplexity's September 2026 decision panel. This is a sampled, provider-reported accuracy result with its own evaluation setup.
Perplexity Decider v1 27B model cardA Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardVals AI composite of agentic finance work: analyst research on company filings, financial modeling in Excel and corporate tax research, weighted by each sub-sector's share of U.S. GDP.
Vals Finance Index leaderboard and methodologyVals AI hosted GPQA Diamond view with few-shot and zero-shot chain-of-thought task splits.
Vals GPQA DiamondVals AI GDP-weighted composite of eight finance, coding, legal, and tax benchmarks, updated to v2.2 on October 19, 2026.
Vals IndexVals AI comparison of the same open-weight model served by different inference providers, against the model creator's own API, on agentic finance, coding and legal tasks.
Vals Inference Provider Index leaderboard and methodologyVals AI composite of agentic legal work: legal research, drafting and reviewing documents, and tax law, with practice areas weighted by their share of U.S. law-office revenue.
Vals Legal Index leaderboard and methodologyVals AI implementation of LiveCodeBench with easy, medium, and hard task splits.
Vals LiveCodeBenchVals AI hosted MMLU-Pro view with subject-level task splits.
Vals MMLU-ProVals AI multimodal composite across finance, coding, education, and mortgage-tax task families.
Vals Multimodal IndexVals AI comparison of model routers with single models on long agentic work in finance, coding, law and tax, using the same agents, tasks and graders as the Vals Index.
Vals Router Index leaderboard and methodologyTests autonomous AI research on compression, language-model training, judge-harness engineering, and post-training under fixed compute and time budgets.
Vals RSI Index leaderboard and methodologyVals AI composite of agentic science work: investigating biology datasets, recovering hidden mechanisms through experiments, and running research workflows in a terminal.
Vals Science Index leaderboard and methodologyVals AI hosted SWE-bench view for solving production software engineering tasks.
Vals SWE-benchA Vals-hosted view of Terminal-Bench 1.0 with easy, medium, and hard task splits.
Terminal-Bench 1.0Vals AI hosted Terminal-Bench 2.0 view with easy, medium, and hard task splits.
Vals Terminal-Bench 2.0Tests long-horizon coherence by asking agents to operate a simulated vending-machine business for one year.
Vending-Bench 2Can a model extend one working web application across a long sequence of dependent requests?
Vals Vibe Code Bench 1-100Can a model extend a working web application through up to ten dependent product requests without breaking what already works?
Vals Vibe Code Bench 1-100 v1.1DeepMind's 1H-VideoQA tests recall and reasoning over videos lasting 40 to 90 minutes. The current 101-question, five-choice task remains separate from the original 126-question set.
VideoQA methodologyPerplexity reports public ViDoRe V3 image retrieval nDCG@10 of 62.3 for the 0.6B model and 65.2 for the 9B model.
Perplexity multimodal late-interaction embedding model cardPerplexity reports public ViDoRe V3 Markdown retrieval nDCG@10 of 61.2 for the 0.6B model and 64.7 for the 9B model.
Perplexity multimodal late-interaction embedding model cardA Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardA Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.
Scale Labs leaderboardTests whether speech-to-text models preserve exact structured values in English workplace audio.
Vals VoiceCodeBenchA live human-preference benchmark where multimodal models build voxel structures from image references and voters compare anonymous results produced from the same prompt.
VoxelBench leaderboardA live human-preference benchmark where language models turn text prompts into voxel structures and voters compare anonymous builds from the same prompt.
VoxelBench leaderboardCoding agents recreate nine reference web apps from an empty directory. Plato checks visual and functional behavior through end-to-end workflows.
Introducing Web Clone Bench v1OpenAI's offensive web penetration-testing evaluation: the model probes web applications without access to their source code and is scored on the share of planted vulnerabilities it finds.
GPT-6 Cyber: Advancing the frontier of aligned cybersecurity agentsA Vals AI comparison of native provider search and Exa across finance-analysis and legal-research tasks.
Web Search IndexA machine-learning engineering benchmark that tests whether LLMs can train models on novel datasets, write PyTorch code, and improve through iterative feedback.
WeirdMLThe first version of Håvard Tveit Ihle's WeirdML: models write PyTorch code for 6 unusual machine-learning tasks and improve it over 5 iterations using terminal output and test accuracy.
WeirdML v1Håvard Tveit Ihle's WeirdML v3 gives coding agents 11 hand-made machine-learning and data-analysis tasks with unfamiliar data, unspecified goals or very limited feedback, and scores how well and how efficiently they improve as they spend tokens.
WeirdML v3When2Call is one component of the Artificial Analysis Decision Index v1.0, run by Artificial Analysis. It measures accuracy on NVIDIA's tool-calling evaluation: given a conversation and tools, choose to answer directly, call a tool, ask for more information, or decline.
Artificial Analysis Decision Index v1.0WildGuardTest is one component of the Artificial Analysis Decision Index v1.0, run by Artificial Analysis. It measures average F1 on Ai2's safety classification set: whether a request is harmful, whether a reply is a refusal, and whether that reply is harmful.
Artificial Analysis Decision Index v1.0Ambiguous-reference resolution on the 1,000-row sample in Perplexity's September 2026 decision panel. This is a sampled, provider-reported accuracy result with its own evaluation setup.
Perplexity Decider v1 27B model cardWritingBench evaluates generative writing across six domains. This dated benchmark-owner table contains 54 source model rows, with overall and domain scores on the native 0-100 scale.
WritingBench: A Comprehensive Benchmark for Generative WritingMoonshot AI's internal benchmark of expert-curated, source-code-based vulnerability discovery and exploitation tasks, most reproducing recently disclosed vulnerabilities with mitigations enabled.
Kimi K3.1 Pro Tech Blog: Advancing Open IntelligenceAn Artificial Analysis IFBench score.
Artificial Analysis IFBench Benchmark LeaderboardInstruction-following benchmark built from real agentic applications, with long instructions that carry many constraints.
AgentIF: Benchmarking Instruction Following of Large Language Models in Agentic ScenariosNVIDIA benchmark that scores conversational answers on quality and conciseness together.
NVIDIA Nemotron 3.5 Bolt E4B 4GBMeasures emotional intelligence in multi-turn roleplay scenarios, scored by an LLM judge against a rubric.
EQ-Bench 3Measures how often indirect prompt-injection attacks succeed within fifteen attempts. Lower scores mean greater resistance to the tested attacks.
Gemini 4 Argon evaluation methodologyIFBench evaluates precise instruction-following generalization on 58 challenging, verifiable out-of-domain constraints. Unlike IFEval which tests familiar constraint types, IFBench specifically measures how well models follow novel instructions they haven't been optimized for, exposing overfitting to common instruction patterns.
BenchLMA benchmark of 541 prompts built from 25 verifiable instruction types. It tests whether a model follows checkable constraints such as keyword, length, casing, and response-format requirements.
Instruction-Following Evaluation for Large Language ModelsLiquid AI benchmark of following structured-output instructions, such as producing output that matches a requested schema or format.
IFStruct v1.0Tests whether a model follows the priority order of system messages, user messages, conversation history and tool outputs, including when their instructions conflict.
IHEval: Evaluating Language Models on Following the Instruction HierarchyTests whether a model follows counterintuitive instructions that run against conventions it learned in training.
InverseIFEvalMeta benchmark of instruction following over multi-turn conversations in eight languages.
Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions FollowingTests instruction retention, memory of user information, versioned editing, and coherence over multi-turn conversations.
MultiChallengeCharacter role-play benchmark from the RoleLLM project that tests whether a model speaks in a given character's style.
RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language ModelsRole-playing benchmark that checks whether a model keeps to a defined role identity while following instructions.
RoleMRCA structured-output benchmark from Interfaze measuring whether extracted JSON leaf values exactly match verified ground truth.
Structured Output Benchmark LeaderboardMeasures whether a model produces valid structured output in text formats such as JSON, YAML and CSV.
StructEvalTests how well a model follows the rules in a system message across single and multi-turn conversations.
SysBench: Can Large Language Models Follow System Messages?An independently evaluated MMLU-Pro result from Artificial Analysis.
Artificial Analysis MMLU-Pro Benchmark LeaderboardA display-only Artificial Analysis model-openness index.
Artificial Analysis Openness IndexAn Artificial Analysis GPQA Diamond score.
Artificial Analysis GPQA Diamond Benchmark LeaderboardAn Artificial Analysis Humanity's Last Exam score.
Artificial Analysis Humanity's Last Exam Benchmark LeaderboardAn Artificial Analysis knowledge metric for the proportion of correctly answered questions.
Artificial Analysis model benchmarksA display-only Artificial Analysis factuality metric for the rate of incorrect answers among non-correct responses.
Artificial Analysis model benchmarksA display-only Artificial Analysis factual knowledge index.
AA-Omniscience: Knowledge and Hallucination BenchmarkA human-centric exam benchmark for general knowledge and reasoning reported in DeepSeek-V4 base-model evaluations.
DeepSeek-V4 Technical ReportArtificial Analysis describes itself as an independent AI benchmarking company. Its Intelligence Index combines the company's own evaluations of AI models into one score per model. BenchLM compiles reported index scores, which can come from different index versions, and shows them for reference only.
Artificial AnalysisComputational biology challenges with objective answers that remained unsolved by independent human experts.
Claude Opus 5 System CardComputational biology challenges that independent human experts were able to solve.
Claude Opus 5 System CardA Chinese-language academic and professional benchmark spanning humanities, social science, STEM, and applied subjects.
C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation ModelsA Chinese short-form factuality benchmark reported by DeepSeek for V4 model evaluations.
DeepSeek-V4 Technical ReportA Chinese multitask academic benchmark reported in DeepSeek-V4 base-model evaluations.
DeepSeek-V4 Technical ReportDesigns diverse miniprotein binders against target protein structures in a GPU-equipped research sandbox.
Claude Opus 5.5 System CardA parametric factuality benchmark reported in DeepSeek-V4 base-model evaluations.
DeepSeek-V4 Technical ReportAn AfterQuery benchmark of open-ended financial analysis that requires models to read financial data, make assumptions, and return exact answers.
FinanceQAA benchmark for research-level scientific reasoning, designed to separate frontier models on difficult science tasks that mix domain knowledge with deep reasoning.
FrontierScienceA research-focused FrontierScience evaluation variant for scientific investigation and problem solving.
Muse Spark Eval MethodologyAmazon benchmark of grounded answers in retrieval-augmented generation, with human labels for which supplied passages are relevant.
GaRAGeA challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. Designed to be difficult even for skilled non-experts with access to Google.
GPQA: A Graduate-Level Google-Proof Q&A BenchmarkVals AI’s independent run of the GPQA Diamond graduate-level science questions.
Vals AI GPQA Diamond, Vals AI run leaderboardGraduate-level science questions from provider comparison charts and open evaluations.
GPQA: A Graduate-Level Google-Proof Q&A BenchmarkArtificial Analysis independently evaluates 5,000 conversations and reports a length-adjusted rubric score.
HealthBench independent evaluationHealthBench score after applying a verbosity penalty to model responses.
HealthBench: Evaluating Large Language Models Towards Improved Human HealthArtificial Analysis independently evaluates 5,000 conversations and reports a raw rubric score.
HealthBench independent evaluationRaw score on realistic multi-turn healthcare conversations graded against expert-written rubrics.
HealthBench: Evaluating Large Language Models Towards Improved Human HealthA harder subset of OpenAI's HealthBench for evaluating open-ended medical and health reasoning with rubric-based grading.
Muse Spark Eval MethodologyAn open benchmark for clinician-facing model responses across care consult, writing and documentation, and medical research tasks.
HealthBench Professional: Evaluating Large Language Models on Real Clinician ChatsArtificial Analysis independently evaluates 525 conversations and reports a length-adjusted rubric score.
HealthBench Professional independent evaluationArtificial Analysis independently evaluates 525 conversations and reports a raw rubric score.
HealthBench Professional independent evaluationRaw score on physician-authored clinical consult, documentation, and research conversations.
HealthBench Professional: Evaluating Large Language Models on Real Clinician ChatsAn expert-authored benchmark designed to probe frontier knowledge and reasoning. BenchLM keeps protocol differences visible because tool-assisted and closed-book HLE runs answer different questions.
Humanity's Last ExamTool-free variant of Humanity's Last Exam that isolates a model's raw frontier reasoning.
Introducing GPT-5.4 mini and nanoAccuracy on the 1,000-question HLE-Diamond set without tools, using the benchmark owner's high-reasoning evaluation.
Introducing HLE-DiamondAccuracy on the 1,000-question HLE-Diamond set without tools at each model's highest available reasoning effort.
Introducing HLE-DiamondAccuracy on HLE-Diamond with constrained web search, page fetching, and code execution at high reasoning effort.
Introducing HLE-DiamondA verified and revised Humanity's Last Exam set that removes uncertain items and repairs fixable questions before evaluating frontier expert reasoning.
HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last ExamMoonshot AI's internal factuality benchmark built from challenging, de-identified prompts drawn from practical work tasks, scored as the claim-level hallucination rate.
Kimi K3.1 Pro Tech Blog: Advancing Open IntelligenceA benchmark of realistic biology-research tasks involving literature, figures, tables, databases, and bioinformatics files.
LABBench2: An Improved Benchmark for AI Systems Performing Biology ResearchSelects compounds with the highest or lowest measured ADME-related property from molecular structures and reference values.
Claude Opus 5.5 System CardA medical multiple-choice benchmark spanning many specialties with 10 answer options per question.
Muse Spark Eval MethodologyA comprehensive multiple-choice question answering test covering 57 tasks including elementary mathematics, US history, computer science, law, and more. Tests knowledge across diverse academic subjects from high school to professional level.
Measuring Massive Multitask Language UnderstandingAn enhanced version of MMLU with 10 answer choices instead of 4, featuring more reasoning-focused questions that better differentiate frontier models.
MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding BenchmarkA display-only MMLU-Pro reference from Arcee AI's Trinity-Large-Thinking launch chart.
Trinity-Large-Thinking: Scaling an Open Source Frontier AgentVals AI’s independent run of MMLU-Pro across fourteen academic subjects.
Vals AI MMLU-Pro, Vals AI run leaderboardA harder refresh of MMLU intended to keep broad knowledge evaluation useful after the original benchmark became too easy for frontier models.
Qwen3.6 launch benchmarksMatches multi-channel cellular imaging data to the molecule-dose perturbations that produced it.
Claude Opus 5.5 System CardA multilingual/localized knowledge benchmark reported in DeepSeek-V4 base-model evaluations.
DeepSeek-V4 Technical ReportA science question-answering benchmark that tests whether models can apply a small open-book set of elementary science facts to multi-step reasoning questions.
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question AnsweringChemistry tasks covering spectroscopy, synthesis planning, reaction prediction, and chemical structure images.
Claude Opus 5 System CardA public benchmark of 100 physician tasks carried out inside an electronic health record system.
Claude Fable 5.2 and Claude Mythos 5.2 system cardGenerates novel protein sequences under family, topology, globularity, and structural-motif constraints.
Claude Opus 5 System CardRanks held-out protein designs for experimental testing using wet-lab campaign measurements and a selection objective.
Claude Opus 5.5 System CardAnthropic's protein-understanding test: from an amino acid sequence alone, choose the correct Pfam domain family among similar distractors, without tools.
Claude Fable 5.2 and Claude Mythos 5.2 system cardAnthropic's life-sciences evaluation of whether a model can attribute the effects of protein mutations.
Claude Fable 5.2 and Claude Mythos 5.2 system cardPredicts mutation effects by ranking mutant protein sequences against wild type and comparing against laboratory measurements.
Claude Opus 5 System CardDetects and fixes errors in molecular-biology protocols using document, code, and web-search tools.
Claude Opus 5 System CardExtends online molecular-biology protocols in additional directions using document and web-search tools.
Claude Opus 5 System CardCorpus of retrieval-augmented responses annotated word by word for unsupported or contradicted content.
RAGTruthA benchmark that evaluates the ability of language models to answer short, fact-seeking questions accurately. Focuses on factual correctness rather than reasoning complexity.
Measuring short-form factuality in large language modelsSingle-cell RNA sequencing analysis tasks spanning common bioinformatics workflows.
Claude Opus 5 System CardAnalysis of spatial transcriptomics data across externally validated biological problems.
Claude Opus 5 System CardAn expanded version of GPQA that evaluates graduate-level knowledge and reasoning capabilities across 285 disciplines, providing comprehensive coverage of academic domains.
SuperGPQA: Scaling LLM Evaluation Across 285 Graduate DisciplinesA reading and trivia question-answering benchmark reported in DeepSeek-V4 base-model evaluations.
DeepSeek-V4 Technical ReportA benchmark designed to measure whether language models produce truthful answers instead of repeating common misconceptions or misleading falsehoods.
TruthfulQA: Measuring How Models Mimic Human FalsehoodsEvaluates Korean culture and linguistics.
BenchLMEvaluates Korean expert-level knowledge across 45 subjects. 20% of questions require Korean cultural context.
KMMLU: Measuring Massive Multitask Language Understanding in KoreanA filtered hard subset of KMMLU containing ~5,000 questions that most models get wrong.
Evaluating LLMs on Hard Korean QueriesKorean National Professional Licensure exams evaluating professional-grade knowledge.
BenchLMCleaned KMMLU from national technical qualification exams, with errors removed, decontaminated, and deduplicated.
BenchLMEvaluates advanced Korean linguistic competence.
BenchLMThe Korean SAT exam.
BenchLMAn independently evaluated AIME 2025 result from Artificial Analysis.
Artificial Analysis AIME 2025 Benchmark LeaderboardAn independently evaluated MATH-500 result from Artificial Analysis.
Artificial Analysis MATH-500 Benchmark LeaderboardA 15-question, 3-hour examination where each answer is an integer from 000 to 999. Serves as the intermediate step between AMC 10/12 and the USA Mathematical Olympiad (USAMO).
American Invitational Mathematics ExaminationThe 2024 edition of AIME, maintaining the same format of 15 challenging mathematics problems with integer answers from 000 to 999.
American Invitational Mathematics ExaminationThe most recent AIME examination, featuring 15 challenging mathematics problems testing olympiad-level mathematical reasoning with integer answers from 000-999.
American Invitational Mathematics ExaminationA display-only AIME25 reference from Arcee AI's Trinity-Large-Thinking launch chart.
Trinity-Large-Thinking: Scaling an Open Source Frontier AgentA 2026 American Invitational Mathematics Examination snapshot used in frontier-model comparison tables for mathematical reasoning.
Qwen3.6 launch benchmarksA high-difficulty mathematical reasoning benchmark reported in DeepSeek-V4 model evaluations.
DeepSeek-V4 Technical ReportA shortlist subset of the Apex mathematical reasoning benchmark reported in DeepSeek-V4 model evaluations.
DeepSeek-V4 Technical ReportFinal-answer research mathematics problems drawn from recent arXiv papers, including problems based on resolved or refuted conjectures.
Claude Opus 5.5 System CardThe August 2026 ArXivMath release evaluated with a code-execution sandbox and no internet access.
Claude Opus 5.5 System CardFinal-answer research mathematics problems drawn from recent arXiv abstracts.
Claude Opus 5 System CardFinal-answer research mathematics problems drawn from recent arXiv abstracts with tool access.
Claude Opus 5 System CardA challenging mathematical olympiad competition featuring problems that test advanced mathematical reasoning and problem-solving skills at the olympiad level.
Bulgarian Mathematical OlympiadA Chinese mathematical reasoning benchmark reported in DeepSeek-V4 base-model evaluations.
DeepSeek-V4 Technical ReportLegacy FrontierMath values retained for historical model pages. This field is not used in current rankings because it can mix prior benchmark versions and slices.
FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIEpoch AI's corrected v2 Tier 4 expansion, a separate set of exceptionally difficult research-level mathematics problems evaluated with Python-enabled iterative reasoning.
FrontierMath Tier 4 v2 leaderboardEpoch AI's corrected v2 core FrontierMath suite of private advanced mathematics problems. Models can reason iteratively and use Python; scores are pass rates on the private set.
FrontierMath v2 benchmark hubGSM8K tests multi-step arithmetic in grade-school word problems. OpenAI introduced the dataset in 2021; a high score on this fixed test is one piece of evidence about mathematical reasoning.
Training Verifiers to Solve Math Word ProblemsA prestigious high school mathematics competition hosted jointly by Harvard and MIT, featuring challenging problems across various mathematical disciplines.
Harvard-MIT Mathematics TournamentThe 2024 February edition of the Harvard-MIT Mathematics Tournament, continuing the tradition of challenging high school mathematics competition.
Harvard-MIT Mathematics TournamentThe most recent February edition of the Harvard-MIT Mathematics Tournament, featuring the latest challenging problems in competitive mathematics.
Harvard-MIT Mathematics TournamentA February 2025 HMMT slice used in exact-value provider tables for advanced contest-math reasoning.
Qwen3.6 launch benchmarksA February 2026 HMMT slice used in newer frontier-model math comparisons.
Qwen3.6 launch benchmarksA November 2025 HMMT slice for high-end mathematical reasoning comparisons.
Qwen3.6 launch benchmarksProof-based olympiad performance on all six IMO 2026 problems.
Claude Opus 5 System CardA challenging mathematical reasoning benchmark reported in DeepSeek-V4 model evaluations.
DeepSeek-V4 Technical ReportThe three official theory problems from the 2025 International Physics Olympiad, scored with blinded human evaluation.
Muse Spark Eval MethodologyA competition-style mathematical reasoning benchmark reported in DeepSeek-V4 base-model evaluations.
DeepSeek-V4 Technical ReportA curated subset of 500 problems from the MATH dataset, covering algebra, counting and probability, geometry, intermediate algebra, number theory, prealgebra, and precalculus.
Measuring Mathematical Problem Solving With the MATH DatasetA multimodal mathematical reasoning benchmark that tests whether models can answer visually grounded math questions correctly.
Qwen3.6 launch benchmarksResearch-level mathematics problems with unique programmatically verified closed-form answers.
Claude Opus 5 System CardResearch-level mathematics problems with unique programmatically verified closed-form answers and tool access.
Claude Opus 5 System CardThe premier US mathematical olympiad competition, featuring proof-based problems that require deep mathematical insight and rigorous argumentation at the highest competition level.
United States of America Mathematical OlympiadGlobal MMLU Lite is a multilingual version of MMLU with 400 questions per language, half culturally sensitive and half culturally agnostic. This page holds two independent runs: Artificial Analysis's, in the main table, and Kaggle's.
Artificial Analysis Global-MMLU-Lite Benchmark LeaderboardMMLU-style knowledge evaluation across 42 high- and low-resource languages.
Global MMLU: Understanding and addressing cultural and linguistic biases in multilingual evaluationA multilingual benchmark used in provider tables to measure inclusive language coverage and cross-lingual understanding beyond common high-resource languages.
Qwen3.6 launch benchmarksCohere's long-document translation evaluation: two book chapters translated in a single call, scored per paragraph with xCOMET-XL.
Introducing North Small TranslateA multilingual instruction-following and understanding benchmark row published in Qwen's launch comparisons.
Qwen3.6 launch benchmarksA multilingual benchmark that translates 250 grade school math problems from GSM8K into 10 typologically diverse languages: Bengali, German, Spanish, French, Japanese, Russian, Swahili, Telugu, Thai, and Chinese.
Language Models are Multilingual Chain-of-Thought ReasonersCulturally grounded knowledge comprehension across ten Indic languages and English.
MILU: A Multi-task Indic language understanding benchmarkA display-only multilingual QA retrieval benchmark reported by Liquid AI for LFM2.5 retriever models, using Recall@20 across 11 languages.
LFM2.5 Retrievers: Bi-directional LFMs for Fast Multilingual SearchA multilingual extension of professional-level academic evaluation across many languages.
MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model EvaluationA display-only multilingual retrieval benchmark reported by Liquid AI for LFM2.5 retriever models, using NDCG@10 across 11 languages.
LFM2.5 Retrievers: Bi-directional LFMs for Fast Multilingual SearchA broad multilingual benchmark row from Qwen's launch comparisons intended to measure cross-lingual capability beyond a single language family.
Qwen3.6 launch benchmarksA multilingual mathematical reasoning benchmark that tests whether math performance transfers across languages rather than only in English.
Qwen3.6 launch benchmarksA multilingual extension of SWE-bench covering 300 problems across 9 programming languages, testing code generation and bug fixing beyond Python.
SWE-bench MultilingualA lighter multilingual benchmark slice published in provider tables for broad cross-lingual transfer and understanding.
Qwen3.6 launch benchmarksMachine-translation quality on the WMT26 general translation task, averaged across all evaluated languages and scored 0-100 by an LLM judge.
WMT26 General Machine Translation TaskA visual question answering benchmark whose questions cannot be answered from the image alone and require commonsense or world knowledge.
A-OKVQA: A Benchmark for Visual Question Answering using World KnowledgeAn Artificial Analysis MMMU-Pro score.
Artificial Analysis MMMU-Pro Benchmark LeaderboardA diagram understanding benchmark focused on scientific and educational visual question answering.
Qwen3.6 launch benchmarksA multimodal benchmark for fine-grained visual perception and grounded reasoning tasks.
Muse Spark 1.1 Evaluation ReportA Python-assisted BabyVision evaluation for fine-grained visual perception and grounded reasoning.
Kimi K3: Open Frontier IntelligenceHardware-engineering benchmark in which a model writes CAD code that reproduces a target part, scored 0 to 100 on the Core set.
Claude Fable 5.2 and Claude Mythos 5.2 system cardBenchCAD 2.1 Core with tool access, in which a model writes CAD code that reproduces a target part.
Claude Fable 5.2 and Claude Mythos 5.2 system cardGenerates CadQuery code from multi-view renders and scores geometric similarity by voxel intersection-over-union.
BenchCAD: A comprehensive, industry-standard benchmark for programmatic CADGenerates CadQuery code from multi-view renders with image inspection and code-execution tools.
BenchCAD: A comprehensive, industry-standard benchmark for programmatic CADQuantitative biomedical image analysis across segmentation, detection, measurement, classification, registration, reconstruction, and tracking tasks.
Claude Opus 5.5 System CardAn agentic spatial reasoning benchmark reported as a normalized score.
Gemini 3.5 Flash launch screenshotsAn OCR-focused benchmark for reading and extracting text from visually complex documents and images.
Qwen3.6 launch benchmarksProfessional chart understanding across 100 specialized chart types with expert-set answer tolerances.
ChartographyProfessional chart understanding with a container, standard libraries, and image cropping.
ChartographyA scientific chart reasoning benchmark that tests whether models can understand, interpret, and reason about complex scientific visualizations including plots, diagrams, and data charts.
CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMsA scientific chart understanding benchmark scored across both the descriptive and reasoning splits rather than the reasoning split alone.
CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMsTool-free variant of CharXiv that isolates raw visual reasoning ability without code execution or tool augmentation.
CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMsA conversational visual QA benchmark that tests multi-turn grounded answering over images and documents.
Qwen3.6 launch benchmarksA visual counting benchmark that tests whether a model can count objects and entities reliably in complex scenes.
Qwen3.6 launch benchmarksA display-only Design Arena website-generation Elo score surfaced on OpenRouter model benchmark pages.
OpenRouter Grok 4.3 benchmarksA multimodal coding benchmark for turning visual designs into working frontend implementations.
GLM-5V-TurboA multimodal benchmark for dynamic mathematical reasoning over visual and structured inputs.
Qwen3.6 launch benchmarksA grounded visual reasoning benchmark focused on evidence-based question answering over real images.
Qwen3.6 launch benchmarksScores schema-guided extraction from 370 enterprise documents and 4,869 pages on value F1, record completeness, word and page grounding and cost, split by short, medium and long documents.
ExtractBench: A Benchmark for Schema-Guided Enterprise Document ExtractionA harder companion to ExtractBench: 100 synthetic documents and 2,622 pages across 86 document types that test document conventions, house vocabulary, counting rules, drawn relations and charts, scored by perfect-document rate.
ExtractBench Pro leaderboard and challenge descriptionsA grounded multimodal factuality benchmark for evidence-linked answer correctness.
GLM-5V-TurboA vision-language coding benchmark for generating correct code from visual and multimodal inputs.
GLM-5V-TurboProfessional document understanding over real-world PDFs, graded per rubric criterion by Gemini-3.5-Flash.
GDP.pdfProfessional document understanding over 100 real-world PDFs from ten domains.
GDP.pdfProfessional document understanding with a container, standard libraries, and image cropping.
GDP.pdfProfessional document understanding where the model receives each original PDF in a sandbox with an IPython tool.
GDP.pdfAn evaluation focused on professional domain expertise and task delivery quality in office-style knowledge work.
MiniMax M2.7: Early Echoes of Self-EvolutionA multimodal retrieval and extraction benchmark over image-heavy task settings.
GLM-5V-TurboA display-only Liquid AI extraction metric measuring field-name agreement between requested schema fields and extracted JSON fields.
LiquidAI LFM2.5-VL Extract model cardsA display-only Liquid AI extraction metric measuring the share of image-to-JSON outputs that parse as strict JSON.
LiquidAI LFM2.5-VL Extract model cardsA display-only Liquid AI extraction metric measuring judged agreement between extracted values and the source image.
LiquidAI LFM2.5-VL Extract model cardsA long-video understanding benchmark for retrieving and reasoning over information distributed across extended video inputs.
Qwen3.8-Max: A New Bar for Coding and CoworkA visual mathematics benchmark that tests whether a model can solve math problems grounded in diagrams, equations, figures, and other visual inputs.
Qwen3.6 launch benchmarksA tool-augmented MathVision variant that permits Python during visual mathematics reasoning.
Kimi K3: Open Frontier IntelligenceA multimodal medical multiple-choice benchmark covering clinical images such as X-rays, histology, and dermatology.
Muse Spark Eval MethodologyA multi-task video understanding benchmark averaged across MLVU categories.
Qwen3.6 launch benchmarksA long-document multimodal benchmark for grounded reasoning over extended document contexts.
Qwen3.6 launch benchmarksA broad multimodal reasoning benchmark spanning charts, diagrams, tables, and academic visual question answering.
MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIA harder multimodal benchmark for frontier models that combines text with images, diagrams, charts, and academic visual reasoning tasks.
MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding BenchmarkTool-augmented MMMU-Pro variant that allows Python assistance during multimodal reasoning.
Introducing GPT-5.4 mini and nanoA multimodal search benchmark for retrieval and grounded answering across mixed-media inputs.
GLM-5V-TurboA harder MMSearch variant for multimodal retrieval and grounded tool-use workflows.
GLM-5V-TurboA benchmark for evaluating multimodal models on video understanding tasks across multiple disciplines, emphasizing temporal reasoning and comprehension over video content.
Kimi K2.5 benchmark release surfaceA general visual question-answering benchmark used in provider tables for real-image reasoning quality.
Qwen3.6 launch benchmarksA native OCR benchmark for reading text from images across multilingual scripts, low-quality scans, handwriting, structured layouts, charts, and screenshots.
OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and ReasoningA visual detection and grounding benchmark slice used to compare zero-shot object understanding across diverse domains.
Qwen3.6 launch benchmarksGrounded numerical reasoning over a corpus of historical U.S. Treasury Bulletin documents.
Claude Opus 5 System CardA benchmark for grounded reasoning over office-style documents, spreadsheets, charts, and business artifacts.
OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded ReasoningThe second version of OfficeQA Pro: grounded reasoning over real financial and office documents through retrieval, cross-document verification, and multi-step numerical reasoning.
Kimi K3.1 Pro Tech Blog: Advancing Open IntelligenceAn end-to-end document understanding benchmark over long, layout-rich PDFs with tables, equations, headers, footnotes, and multi-column flows.
olmOCR-BenchA document-understanding benchmark for parsing and reasoning over complex document layouts.
Kimi K3: Open Frontier IntelligenceA document understanding benchmark used in frontier-model comparison tables to measure extraction and grounded reasoning quality on complex documents.
Introducing GPT-5.4 mini and nanoA document parsing benchmark whose overall score composites edit distance, CDM and TEDS sub-scores across diverse PDF layouts.
OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive AnnotationsThe first ParseBench release: scores document parsers and vision-language models on 2,078 human-verified enterprise pages across tables, charts, content faithfulness, semantic formatting and visual grounding, with the listed cost per page. ParseBench-2 is a separate benchmark with different scoring.
ParseBench: A Document Parsing Benchmark for AI AgentsScores document parsers and vision-language models on 4,317 pages from 2,645 documents in six dimensions: text, formatting, tables, charts, visual grounding and forms, with severity-aware scoring and the listed cost per page. A separate release from ParseBench-1; the scores are not comparable.
ParseBench-2: Severity-Aware Evaluation of Document ParsingMoonshot AI's internal benchmark for atomic visual perception capabilities.
Kimi K3: Open Frontier IntelligencePerceptionBench fine-grained visual perception tasks run with tool use, as reported by Meta.
Introducing Muse EmberA grounded visual QA benchmark focused on answering practical questions about real-world images and scenes.
Qwen3.6 launch benchmarksA referring-expression grounding benchmark averaged across RefCOCO variants to test whether a model can localize described objects correctly.
RefCOCO referring expression datasetsA GUI-grounding benchmark for 1,581 instructions in full-screen, high-resolution professional interfaces. It tests where a target is, not whether an agent can finish the surrounding workflow.
ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer UseA visual question answering benchmark focused on straightforward image-grounded understanding.
GLM-5V-TurboA multimodal variant of SWE-bench that adds visual context (screenshots, design mockups) to software engineering issue descriptions, testing whether models can leverage visual information for code generation.
SWE-bench MultimodalA visual agent benchmark for interface reasoning and task execution over screenshots or software surfaces.
Qwen3.6 launch benchmarksA vision-centric benchmark for high-level multimodal reasoning and perception quality.
GLM-5V-TurboA comprehensive benchmark for multimodal large language models on video understanding, covering temporal reasoning, perception, and question answering over videos.
Video-MME benchmarkA stricter Video-MME setting that removes subtitle help and tests video understanding from visual and audio context alone.
Qwen3.6 launch benchmarksA video understanding benchmark that allows subtitle access when answering multimodal questions about videos.
Qwen3.6 launch benchmarksA video extension of MMMU-style multimodal reasoning over expert questions grounded in temporal media.
Qwen3.6 launch benchmarksA benchmark for converting visual references into functional web implementations.
GLM-5V-TurboA speech-recognition benchmark on the cleaned Artificial Analysis VoxPopuli subset, reported as word error rate where lower is better.
VoxPopuli-Cleaned-AAA multimodal math benchmark for visually grounded mathematical reasoning and answer generation.
Qwen3.6 launch benchmarksA forced-answer WorldVQA variant for atomic visual world knowledge.
Kimi K3: Open Frontier IntelligenceA multi-step visual reasoning benchmark with pass@5 reporting and optional tool use.
Muse Spark Eval MethodologyA Python-assisted ZeroBench_main evaluation reported as pass@5.
Kimi K3: Open Frontier IntelligenceAn Artificial Analysis long-context reasoning evaluation.
Artificial Analysis model benchmarksA long-context retrieval benchmark that measures whether a model can recover relevant information embedded deep inside very long contexts.
Qwen3.6 launch benchmarksARC Prize fluid-intelligence benchmark using novel visual grid transformations.
ARC Prize leaderboardARC-AGI-2 tests whether a system can infer a transformation from example grids and apply it to a new grid. Compare accuracy with the evaluation split, attempt limit, and cost attached.
ARC-AGI-2: A Harder General Intelligence BenchmarkAn interactive successor to ARC-AGI-2 that evaluates whether an AI agent can learn unfamiliar task mechanics through action and feedback.
ARC-AGI-3: A New Challenge for Frontier Agentic IntelligenceARC Prize verified interactive reasoning results using a provider-specific stateful adapter, kept separate from Standard harness results.
ARC Prize verified ARC-AGI-3 resultsA suite of 23 challenging tasks from the BIG-Bench collaborative benchmark where prior language models failed to exceed average human performance, even with chain-of-thought prompting.
Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve ThemOpenAI and Ligo's benchmark of how well AI agents interpret raw wet-lab evidence, such as gel and blot images, colony plates, microscopy, sequencing files and instrument exports, spot experimental errors and recognize when the data cannot support a conclusion.
Measuring how AI agents handle everyday wet-lab scienceA benchmark that tests whether AI models challenge nonsensical, ill-posed, or logically flawed prompts instead of confidently generating incorrect answers. Measures the critical ability to push back on bad input.
BullshitBench: Measuring whether AI models challenge nonsensical promptsA Chinese Winograd Schema Challenge benchmark reported in DeepSeek-V4 base-model evaluations.
DeepSeek-V4 Technical ReportTests whether model judgments rank argumentative critiques in the same order as expert human ratings across philosophy, AI alignment, and other concept-heavy texts.
Conceptual Reasoning Benchmark ResultsA million-token CorpusQA long-context question-answering benchmark reported in DeepSeek-V4 model evaluations.
DeepSeek-V4 Technical ReportAn Artificial Analysis metric for research-level physics reasoning.
CritPt Benchmark LeaderboardA reading-comprehension benchmark requiring discrete reasoning over paragraphs, reported in DeepSeek-V4 base-model evaluations.
DeepSeek-V4 Technical ReportA multistage statistical-reasoning benchmark for genomics and biological-data analysis agents.
GeneBench-Pro: Evaluating Multistage Statistical ReasoningThe hardest subset of GPQA featuring the most challenging graduate-level science questions. Sometimes reported separately from the standard GPQA benchmark.
GPQA: A Graduate-Level Google-Proof Q&A BenchmarkLong-context graph traversal benchmark using breadth-first search tasks.
Introducing GPT-5.4 mini and nanoBreadth-first-search graph traversal on 200 problems with context lengths from 256K to 1M tokens.
Gemini 4 Argon evaluation methodologyLong-context benchmark for recovering parent relationships inside graph tasks.
Introducing GPT-5.4 mini and nanoA commonsense natural-language inference benchmark reported in DeepSeek-V4 base-model evaluations.
DeepSeek-V4 Technical ReportTyped-decision evaluation across 1,624 decisions, combining chance-corrected intelligence, probability calibration, latency, and cost.
JevBench v1.5 frozen method and disclosuresDecision-system evaluation combining chance-corrected intelligence, probability calibration, latency, and cost across 842 decisions.
JevBench v1.4 methodTyped-decision evaluation across 1,624 decisions, combining chance-corrected intelligence, probability calibration, latency, and cost.
JevBench v1.5 frozen method and disclosuresA word-chain reasoning benchmark that tests planning, recall, constraint following, and vocabulary depth by asking models to extend non-repeating edit-distance-1 chains.
LisanBench methodologyA long-context benchmark that measures whether models can actually use extended context windows for reasoning and retrieval.
LongBench v2Tests whether a chat assistant recalls and reasons over information spread across long, multi-session conversation histories.
LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive MemoryAn open benchmark from Wisedocs, run by Artificial Analysis, measuring how well models reason over long, fragmented medical records with multi-document synthesis.
Medical Long Context Reasoning (MLCR-AA)A million-token MRCR long-context retrieval benchmark reported in DeepSeek-V4 model evaluations.
DeepSeek-V4 Technical ReportMRCR v2 slice focused on very long contexts at 128K-256K lengths.
Introducing GPT-5.4 mini and nanoMRCR v2 slice focused on retrieval across 256K-512K-token contexts.
Muse Spark 1.3 Evaluation MethodologyMRCR v2 slice focused on retrieval across 512K-1M-token contexts.
Muse Spark 1.3 Evaluation MethodologyMRCR v2 slice focused on long-context retrieval at 64K-128K lengths.
Introducing GPT-5.4 mini and nanoA long-context benchmark for memory, retrieval, and multi-round coherence over large contexts.
Introducing GPT-5.2 and GPT-5.2 ProA dataset for evaluating language models on multistep soft reasoning tasks specified in natural language narratives. Tests the ability to perform complex, structured reasoning.
MuSR: Testing the Limits of Chain-of-thought with Multistep Soft ReasoningA multi-step verifiable reasoning benchmark that evaluates whether models can solve pencil puzzles with unique solutions.
Pencil Puzzle BenchSynthetic long-context benchmark of retrieval, multi-hop tracing, aggregation and question answering, scored at a 128K-token context.
RULER: What's the Real Context Size of Your Long-Context Language Models?Synthetic long-context benchmark of retrieval, multi-hop tracing, aggregation and question answering, scored at a 64K-token context.
RULER: What's the Real Context Size of Your Long-Context Language Models?Anthropic's internal set of unpublished research problems written by professional scientists, each about a month of research work, mostly in theoretical physics; BenchLM stores the 64-problem subset.
Claude Fable 5.2 and Claude Mythos 5.2 system cardScores how far a model advances 120 open research tasks across biology, chemistry, physics, astronomy, Earth science and engineering on a 0 to 4 progress scale, crediting supported partial work and honest statements of what remains unresolved.
STEMBench: six-domain model evaluationAn automated evaluation framework using 1,000+ real-world user tasks covering reasoning, planning, coding, and creative writing. Highly correlated with Chatbot Arena human preference rankings.
WildBench: Benchmarking Language Models with Challenging Tasks from Real Users in the WildA commonsense coreference benchmark reported in DeepSeek-V4 base-model evaluations.
DeepSeek-V4 Technical ReportA value is shown exactly as its source published it, in that source’s declared unit — nothing is converted, interpolated, or averaged across providers. Where a source returned no value, the site prints “Unavailable” and records when the fetch was attempted; it never substitutes a zero, and a missing measurement is never ranked as if it were a low one. Where a fetch failed and a previous value is still on file, that value is shown with a “Fetch failed” badge and is never used to compute a cost estimate. Every figure carries an evidence dot linking to the fetch it came from, so the claim on this page can be checked one number at a time; the key beside each table says which states that section contains.
Evidence values belong to their cited sources and are reproduced here for comparison. Provider logos come from Lobe Icons (MIT), and each remains the trademark of the provider it identifies.
The planning figures the cost simulator uses to turn plain answers into tokens are our assumptions, not source evidence, and have their own page.