Artificial Analysis Intelligence Index v5.0

Artificial AnalysisK

Artificial AnalysisModels

Coding Agents

Image, Speech, Video

Inference

Leaderboards

About

AI Trends

K

Benchmarking MethodologyOn this page

Artificial Analysis Intelligence Index

Additional Evaluations

Intelligence Evaluation Principles

General Testing Parameters

Artificial Analysis Intelligence Index Evaluations

Knowledge Work

AA-Briefcase v1.1

AutomationBench-AA

GDP.pdf

Coding

Terminal-Bench 4.0

FrontierCode 2.0

Science

CritPt v2 (Hard Subset)

Terminal-Bench-Science 0.1

General

AA-Omniscience v1.1

HLE-Diamond

Additional Evaluation Details

Knowledge Work

GDPval-AA v2.1

Harvey LAB-AA v1.1

APEX-Agents-AA

AA-AnalystAgent

EnterpriseOps-Gym-AA

Coding

ITBench-AA

Code2Video-AA

Science

MLCR-AA (Medical Long Context Reasoning)

CritPt v2

HealthBench / HealthBench Professional

General

IFBench

Global-MMLU-Lite

MMMU Pro

Legacy Evaluations

SciCode

AA-LCR v1.1

HLE (Humanity's Last Exam)

CritPt

GPQA Diamond (Graduate-Level Google-Proof Q&A Benchmark)

𝜏³-Banking

FrontierCode 1.1

Terminal-Bench 2.1

Terminal-Bench Hard

𝜏²-Bench Telecom

MATH-500

AIME 2025 (American Invitational Mathematics Examination)

MMLU-Pro (Multi-Task Language Understanding Benchmark, Pro version)

LiveCodeBench

Prompt Templates, Answer Extraction and Evaluation

Multiple Choice Questions (GPQA, MMLU-Pro)

Multiple Choice Extraction Regex

Equality Checker LLM

HLE (Humanity's Last Exam)

HLE Equality Checker Prompt

LCR Equality Checker Prompt

Mathematical Questions (AIME 2025)

Mathematical Equality Checker Prompt

Code Generation Tasks

SciCode

LiveCodeBench

Code Extraction Regex

Version History

Artificial Analysis Intelligence Benchmarking Methodology

Artificial Analysis Intelligence Index v5.0

Artificial Analysis Intelligence Index

Artificial Analysis Intelligence Index combines a comprehensive suite of evaluation datasets to assess language model capabilities across reasoning, knowledge, math, and programming.

It is a helpful synthesis of overall language model intelligence and can be used to compare language models. Like all evaluation metrics, it has limitations and may not apply directly to every use case. However, we are confident that it is a more useful synthesis comparison between language models than any other metric in existence today.

Artificial Analysis Intelligence Index v5.0 incorporates 9 evaluations: AA-Briefcase v1.1, AutomationBench-AA, GDP.pdf, FrontierCode 2.0, Terminal-Bench 4.0, AA-Omniscience v1.1, HLE-Diamond, CritPt v2 (Hard Subset), and Terminal-Bench-Science 0.1.0. Our methodology emphasizes fairness and real-world applicability.

The Artificial Analysis Intelligence Index is a primarily text-based, English-language evaluation suite. We benchmark models for image inputs, speech inputs and multilingual performance separately to the Intelligence Index evaluation suite.

The Intelligence Index is calculated as a weighted average across four categories: Knowledge Work (25%), Coding (25%), Science (25%), and General (25%). Category membership and per-evaluation weights are shown below.

Beyond the Intelligence Index suite, we run a range of additional evaluations covering multilingual, visual, mathematical, and other capabilities. These are reported separately and are not included in the Intelligence Index score.

Artificial Analysis Multilingual Index: Represents the multilingual ability of models. This is based on the Global-MMLU-Lite evaluation across supported languages. We support the following languages:

Our evaluation approach is guided by four core principles:

We test all evals with the following settings:

We maintain internal copies of all evaluation datasets. The sources of our selected datasets are listed below.

For the Artificial Analysis Intelligence Index evaluations, we use the token counts reported by each model's API provider where available, to accurately report the cost of running the Intelligence Index. In the rare cases where provider token counts are unavailable, we use a canonical-tokenizer fallback. This contrasts with the approach in performance benchmarking, where we use client-side token counts from the o200k_base tokenizer to standardize token counts for the same text across models. When reporting cache hit rates and cost, we combine these token counts with live measurements of the model's typical cache hit rate, rather than relying on the one-off measurement when the evaluation was run.

We use e2b as our primary sandbox provider for agentic benchmarks.

Evaluations that make up the current Artificial Analysis Intelligence Index, grouped by category.

AA-Briefcase v1.1

Status: Included in Artificial Analysis Intelligence Index v5.0 at 15% weighting.

Description: AA-Briefcase is a new benchmark for testing models on realistic knowledge work tasks in complex projects built by industry experts. Models are evaluated on multi-week knowledge work projects, each with many linked tasks and thousands of input source files. AA-Briefcase combines rubric and pairwise grading to evaluate verifiable task success, analytical quality, and presentation quality, giving a holistic view of overall agentic capability in knowledge work.

Changes from AA-Briefcase v1: v1.1 changes only how Elo ratings are fitted. We fit the ratings with a Crowd-BT model. For a comparison of submissions and with fitted strengths and , and annotator quality :

We define , giving:

where is fitted per scope on past AA-Briefcase judgments. Rubric grading is decided by a deterministic comparison rather than a judge, so for that scope. While Elo scores shift, rank ordering is largely preserved.

Example dataset: https://huggingface.co/datasets/ArtificialAnalysis/AA-Briefcase-Lite

Agent harness: https://github.com/ArtificialAnalysis/Stirrup

Implementation:Each AA-Briefcase scenario is a realistic multi-week business problem, organized as a multi-week workflow that the agent works through in sequence, with 2-5 tasks per week. Although tasks within a scenario share files and context across weeks, models currently complete each task in an independent run, without carrying over their own prior submissions. The agent receives the task description and accessible source files, then produces final deliverable files without live interaction or iterative feedback during execution.

Scenario source pools include shared files and week-specific files, mixing real, augmented, and synthetic materials. Source files are designed to include realistic professional artifacts such as Slack exports, spreadsheets, PDFs, interview transcripts, market research, standards documents, app-store pages, board materials, emails, and other business records. Later-week tasks may receive standardized base-case files (the same reference work products given to every model), so each task stays independently runnable while preserving continuity across the week.

Model submissions are run with Stirrup in a week-scoped E2B sandbox.Turns: Agents run for up to 500 turns per task.

Tools: The agent is given a single code-execution tool that runs shell commands and code inside the sandbox, plus the finish tools below (and a view-image tool when the model supports vision). The sandbox has no internet access, so the agent can only use the provided source files.

Sandbox: Each scenario/week sandbox is built from that week's source files, with standard Python packages and system tools for document processing and scientific computing pre-installed.

Finish tools: A finish tool, which the agent calls to submit a summary and the absolute paths of its deliverables (validated to be actual files, not directories or missing paths), and an abandon_task_finish (give-up) tool, which it calls with a reason only when it concludes the task is genuinely impossible.

Prompts: The prompts used across generation and grading:Agent system prompt:You are an AI agent working on a specific task within a multi-week simulated workplace scenario. Each task is part of a longer workflow; your job is to complete the current task using the tools provided in up to 500 steps, then submit your deliverables.

When you are done you must call the finish tool as your final step, passing a brief summary of what you accomplished and a list of absolute paths for every deliverable file.

If you have genuinely concluded that the task cannot be completed — for example because required inputs are missing, a hard dependency is unavailable, or the request itself is incoherent — call the abandon_task_finish tool with a brief reason instead. Do not use it to escape difficulty.

You cannot interact with the user during the task. Record any clarifying assumptions you made in your finish summary.

Agent task prompt:<execution_context>

Sandbox

You operate inside an isolated Linux container through the code_exec tool, which runs shell commands and lets you read, create, and edit files. Commands run as the unprivileged user user (UID 1000), starting from /home/user. Passwordless sudo exists but is rarely needed, since your home directory is fully writable.

Every command runs independently: no working directory, environment variable, or other shell state carries over from one call to the next. Prefer absolute paths for both files and commands, and do not navigate with cd across calls — a cd in one command is gone by the next, so relying on it leaves you silently operating in the wrong place. When a step genuinely needs a different directory, chain it into the same command (e.g. cd /home/user/work && python build.py).

No network

The container has no outbound connectivity, and there is no proxy, allowlist, or flag that can turn it on — treat the environment as permanently offline. Anything that reaches for the internet will fail, including package installs (pip, npm, apt), remote git operations, and any HTTP/HTTPS client request from any language.

Identify a network block by its error signature rather than by guessing: failed name resolution (Could not resolve host, Temporary failure in name resolution), an unreachable route (Network is unreachable, a refused or timed-out connection to a public host), or a stalled TLS handshake. When you see these, the failure is structural — do not retry the same call and do not hunt for a workaround (mirrors, alternate hosts, cached copies). Re-plan using only what is already installed and what ships inside your workspace.

Filesystem

• Writable: everything under /home/user/ plus /tmp. Use these for deliverables, intermediate files, and caches.
• Read-only inputs:
• /home/user/shared/ — reference material shared across the whole scenario
• /home/user/week/ — documents specific to this week's tasks Copy these into a working folder before transforming them rather than editing them in place.

Runtime

A broad scientific-computing and document-processing stack is already installed, so confirm what is present before assuming a gap:

• Python 3.13 with the usual data stack (numpy, pandas, polars, scipy), plotting (matplotlib, plotly), the scikit-learn ML family, and document tooling (python-docx, python-pptx, openpyxl, PyMuPDF, pdfplumber, reportlab, weasyprint, Pillow, opencv), plus Playwright.
• System tools include LibreOffice, Pandoc, Tesseract, FFmpeg, ImageMagick, Ghostscript, TeX Live, OpenJDK, Chromium, jq, and git.
• Check availability with pip show <pkg> or which <tool> instead of installing — installs fail offline, but almost anything you would reach for is already here.
• matplotlib runs headless (MPLBACKEND=Agg): write figures to files; never call plt.show().
• Commands are terminated after 20 minutes. Keep them bounded, persist intermediate results to disk, and split long jobs into smaller steps.

Submitting your work

Finish by calling the finish tool — anything not submitted through it is not graded. Your call must include:

1. A short summary of what you accomplishe