Algolia LLM Leaderboard: methodology

This page describes how the Algolia LLM Leaderboard is produced: what it measures, how cases are built, how models are scored, and where the method has known limits. It is written for two readers: model providers who want to validate or challenge a result, and anyone asking "how do you know what works". The pipeline is reproducible from the code in this repository, with one exception stated plainly in Reproducibility for providers: the eval-dataset construction is not yet public.

Where this page and the code disagree, the code wins. Discrepancies known at publication are listed in the Changelog.

Overview

The leaderboard measures the full agentic search loop, not raw-model question answering. Each model runs inside a production Agent Studio retrieval agent and is scored on what that agent does end to end:

• query rewriting (turning a shopper's phrasing into search terms and filters),
• tool calls (issuing searches and reading their results),
• retrieval (the records the search tool returns),
• response synthesis (the reply the shopper reads).

A model can write fluent prose and still rank low here if it searches for the wrong thing, ignores a price constraint, or recommends products it never retrieved. The score reflects the agent's behaviour on a real catalog, which is a different quantity from a model's score on a static QA benchmark. This matters because the best model on paper is often not the best model in a retrieval agent.

Evaluation pipeline

Four stages, each a separate package in the repository.

1. Generate

generation/case_generator reads an app (app_id + index) from the registry, samples real records from that index, and authors eval cases from them. Each case is a query plus structured ground truth derived from the records that back it. Queries are grounded against live search during generation: the generator issues real searches and browses, keeps cases whose intent maps to actual catalog data, and refines or drops queries that return no real match. Cases are scored for quality and rejected below a 0.8 threshold. For the current Spencer & Williams dataset, 4,593 candidates were generated, 1,950 accepted (~42%), and 1,500 retained after balancing to 500 cases per metric; the rest were rejected as low quality or off-distribution.

2. Run

runner/matrix_runner executes the test matrix: each model runs each case as a live Agent Studio completion. For every case it captures the response text, the tool calls the agent made (including search queries and filters), the retrieved records, latency, and token counts. Results are written incrementally to SQLite, one row per case per model. Runs are append-only and resumable; a crashed run picks up where it stopped. The harness can repeat the whole matrix (--runs N) to measure run-to-run variance, but the current board was not produced that way: it is a single pass per model (merged from model-subset fragments), so run-to-run variance was not measured for this board. See Statistical uncertainty.

3. Judge

Each case carries a task_type. The judge routes the task_type through the metric registry (runner/metrics/) to the matching metric. Each metric returns a score in [0, 1] and a pass/fail against its threshold. Metrics compose deterministic checks (against retrieved records) with an LLM judge (against response text) depending on the metric; see What is measured and Scoring.

4. Aggregate

tools/results_showcase.py rolls per-case results up to per-model summaries with confidence intervals and statistical tiers. tools/public_leaderboard.py projects the canonical experiment (named by the pin in config/golden.yaml) into the public board. The public board is a pinned run, not "newest wins"; see Update and versioning policy.

What is measured

A provider asked directly: "does the eval only check the output text, not the tool-call content?" It does not only check output text. Different metrics read different channels of the agent's behaviour. Three channels exist:

• Response text: the final reply the shopper reads.
• Tool calls: the searches the agent issued, including query strings and filters.
• Retrieved records: the products the search tool returned to the agent.

The public metrics read these channels as follows.

Thresholds shown are the per-case cutoffs carried by the current dataset; the metric registry's generic default is 0.70.

Relevance is a two-axis composite. Retrieval-F1 is a deterministic check on the retrieved record set: did search return the right products for the query intent. Response-F1 is an LLM-judge check on the response text: of what was retrieved, did the reply feature the right subset, and did it avoid recommending products that were never retrieved. Because retrieval-F1 reads the records that tool calls returned, the tool channel is scored, not just the prose.

Faithfulness grounds the agent's factual claims against the retrieved records and reads the tool channel explicitly. Some models emit their answer through the tool channel rather than the text channel; the judge grades the channel the model actually used (see The judge).

Language checks that the reply is written in the language of the query, independent of the catalog language. It reads the response text and the tool-call narration, and runs deterministic language detection on the agent's search queries, so a model that answers in the right language while searching in the wrong one is still scored on both.

The search dimension gives credit when at least one emitted search query is in the catalog language (search-scoring rule v2, lang_search_v2_2026-07-17). A shopper writing in French against an English catalog is served correctly as long as the agent translates at least one query, so a native-plus-English query pair passes rather than being penalised for the native half. The dimension fails (0) only when the agent searches and no query is in the catalog language; issuing no search is not a search-language failure (there is nothing to mistranslate). One known limitation: there is deliberately no media or books exemption, so a catalog whose correct query is a native-language title (books, films) can still score 0 on this dimension if the agent never issues a catalog-language query.

The agent's tool contract

Every model under test drives the same Agent Studio agent and sees the same two tools:

• algolia_search_index (exposed per configured index as algolia_search_index_{index}): issues a search against the catalog and returns records.
• algolia_display_results: the client-side tool the agent calls to surface the products it selected, with per-product narration fields.

These are the search tool and the display-result tool the field questions asked about. No recommend tool is exposed for this board, and the Model Context Protocol (MCP) tool is off (enable_algolia_mcp=false), consistent with the Scope statement.

Every model gets the identical tool set, system configuration, and message shape. A request is a system configuration plus the user query plus the tool schemas above; the model then drives the loop (search, read results, call display, answer). The full per-case message transcripts, including the exact system configuration, are available to providers on request at llm-leaderboard@algolia.com, consistent with Reproducibility for providers.

Tool use is not hard-forced by the harness; the agent decides. A model that answers a grounded question without searching retrieves nothing, so on faithfulness it scores 0 as ungrounded. The exception is zero-result cases (should_refuse), where the correct move is to acknowledge no match, and a clean refusal without search scores as a pass. One mechanical effect matters: algolia_display_results is client-side, and Agent Studio ends the agent loop the moment a client-side tool call lands, so a model that narrates only through that tool can finish with empty response text. The judge grades the channel the model actually used (see The judge) so this is not penalised as a non-answer.

Case anatomy

A generated case is a structured object. The load-bearing fields:

• query: the shopper's phrasing.
• task_type: which metric scores the case (relevance, faithfulness, language).
• expected: the structured ground truth, shaped per task type. For relevance: should_contain_categories, should_contain_brands, price_range, keywords_in_results (each may be null when the query does not express it). For faithfulness: fact_type, exact_value, forbidden_claims, should_refuse. For language: response_language, english_search_terms, should_contain_categories.
• thresholds: the pass/fail cutoff(s) for the case.
• example_products: real record IDs from the catalog that anchor the ground truth.
• difficulty: easy, medium, or hard.
• context, notes, quality_score: provenance and generation metadata.

Cases have unique names (spencer_williams-rel-016, spencer_williams-faith-...) so a result can be traced to a single query and its ground truth.

For the customer-agent path (SE-authored CSV cases, not the public board), a case is a triplet: query, criterion_positive (what a correct answer looks like), and criterion_negative (the failure to catch). A rubric metric judges response text and tool calls against that pair.

The cases are built to probe specific failure modes of a search agent, not trivia:

• query rewriting: does the agent turn colloquial phrasing into effective search terms.
• filter discipline: does it apply and honour constraints (price, brand, category) when the shopper states them, and avoid inventing constraints they did not.
• attribute mapping: does it map a query attribute to the right catalog field.
• zero results: when no valid match exists, does it say so rather than fabricate products (the should_refuse cases).

These are descriptions of what the cases exercise, driven by task_type, difficulty, and the expected constraints; they are not a stored taxonomy field.

Dataset

The public board runs on Spencer & Williams (S&W), a public retail catalog (app 881O7PE9PX, index ecommerce_ns_prod, English, USD). It is a broad e-commerce catalog: electronics, home and kitchen, toys, tools, clothing, office products, and more.

Real vs synthetic queries. Queries are model-authored but grounded in real data. The generator samples records from the live index, drafts candidate queries, then validates each against real search: a query survives only if it maps to actual catalog records, and its ground truth is derived from those records (example_products holds their real IDs). Queries that returned no real match were refined or dropped. This grounds the intent in the catalog rather than in the generator's imagination, though the queries themselves are synthesised, not sampled from production search logs.

Versioning. Datasets are versioned (sw_v134). The public board is pinned to a named dataset and run, so the board does not drift when a newer dataset is generated. Dataset files are checked in; result databases are kept local.

Catalog snapshot and staleness. The dataset stores a snapshot of the index at generation time: an engine snapshot (record count, created/updated timestamps), facet values and statistics, a 10,000-record sample pool, and a snapshot_at timestamp. For sw_v134 the index held 395,380 records, snapshotted 2026-05-21. The eval runs retrieval against the live index, so the honest answer to "how do we know the index hasn't changed" is: we do not continuously verify it. We pin the snapshot metadata and the real record IDs that back each case, which lets us detect drift by re-checking those anchors. Continuous snapshot-hash verification against the live index is not yet implemented. Planned: assert the pinned record IDs still resolve before a re-run.

Sample size and statistical power

Sample size decides what a benchmark can and cannot detect. Rough guidance, at the score variance typical here:

The current board runs 1,500 dataset cases: 500 relevance, 500 faithfulness, and 500 language (the run-level count is 1,501, which includes one synthetic __preflight__ row). At 500 cases per metric the board can distinguish differences in roughly the 5pp range from the within-run bootstrap CI; smaller gaps sit inside the noise and are reported as ties (see Statistical uncertainty). The current board is a single pass per model, so it does not measure run-to-run variance; that would need repeated runs (--runs N). A 50-case smoke run cannot rank models and is never used for the public board.

Scoring

Every metric produces a per-case score in [0, 1] and a pass/fail against its threshold (relevance 0.80, faithfulness and language 1.00).

Relevance is the geometric mean of its axes:

Relevance = geomean(Retrieval_F1, Response_F1 [, property_gate])

Retrieval-F1 (found it, deterministic on retrieved records) and Response-F1 (showed it, LLM judge on response text) are combined with a geometric mean, so a model must do both well; a zero on either axis zeroes the composite. A deterministic price property gate is folded in as a third axis when the query states a price constraint. Faithfulness and language are single-axis scores. F1 is the harmonic mean of precision and recall.

The Overall column is the mean of a model's per-case scores across the active (publicly shown) metrics, pooled and expressed as a percentage. It is not itself a geometric mean of Retrieval-F1 and Response-F1; that geometric mean is the per-case relevance score, which then feeds the pooled mean alongside faithfulness and language. The pass rate shown alongside is the fraction of scored cases that cleared their threshold. The current board's Overall pools relevance, faithfulness, and language. When held-out metrics are present in a run (safety, tone, disambiguation), they are excluded from both the Overall column and the pass rate.

Agent failures and timeouts

When a model or agent call times out or errors mid-case, the harness first retries the transient failures: network errors (DNS, connection, timeout) up to 5 attempts with exponential backoff, and rate limits (429) up to 8, against a 180-second per-request timeout. Empty completions and circuit-broken models are re-tried at the end of the run. Every completion row stores an error string; the showcase clusters these into named error classes and reports a per-model errored count and success rate, so failures are recorded, not dropped.

The scoring consequence splits by cause, which is the line a provider needs:

• Transient infrastructure errors and provider content-policy blocks (classes INFRA and FILTERED) that survive retries are recorded with a null score and excluded from the model's quality mean and pass-rate denominator. They are not counted as failures; they shrink the scored n, and the excluded count is shown.
• Model-side failures (context-window overflow, tool-call recursion loops, empty output, other) are recorded with a score of 0.0 and counted as a failure in the quality mean and the pass rate.

So a pure timeout is retried and, if it persists, removed from the denominator instead of booked as a loss, whereas a model that loops on tool calls or overflows its context is scored 0. Per-model error and timeout counts are part of the recorded results and are in the traces shared with providers.

Statistical uncertainty

No score is reported without a 95% confidence interval. The board uses a bootstrap CI: 10,000 resamples of the per-case scores, fixed seed for reproducibility. The interval says where the true mean plausibly sits given the sample; a point estimate alone hides whether 77% means [75-79] or [70-84].

Run-to-run variance is not yet measured for this board. The harness supports repeating the whole matrix (--runs N), which captures variance that more cases in a single run cannot: agent-creation differences, LLM sampling noise, and API load. A model that scores 85% in one pass and 78% in another is unstable, and only repeated passes reveal it. The current board is a single pass per model, so its confidence intervals reflect within-run case-level sampling, not pass-to-pass stability. Repeated-run refresh is planned.

What a difference between two rows means. Rows are grouped into statistical tiers using a paired bootstrap on the per-case difference against the tier leader (paired, because the two models saw the same cases, so case-level variance is shared and controlled for). Two rows in the same tier are not significantly different: their gap sits inside the noise, and you should read them as tied regardless of the point-estimate order. A difference is real only when the paired CI on the difference excludes zero. Rows whose CI is wider than 10pp, or with fewer than 100 cases, are flagged as underpowered.

The judge

Response-level metrics are scored by an LLM judge served through Algolia Enablers. The judge is a long-context open-weight model, chosen so it can see every retrieved product, not a truncated list (a past truncation bug is described below). The judge is addressed through a managed tier alias (medium, large) that Enablers resolves to a concrete model server-side; the run metadata records only the alias string, not the resolved model. The alias rotates without notification (for example qwen3.6 to gemma-4-31b-it-nvfp4 on 2026-05-21), and that 2026-05-21 resolution to gemma-4-31b-it-nvfp4 is the last one verified before the main judging window, so it is the best available identification of the model behind this board's medium judge. Per-judgment capture of the resolved judge id is not yet implemented; it is planned and is required before the next board refresh.

Bias mitigations.

• Deterministic axes where possible: Retrieval-F1 and the price gate are computed in code, not left to the judge, so a large part of relevance does not depend on judge opinion.
• Full retrieval visibility: the judge sees all retrieved products. An earlier version showed only the top 5, which caused false "hallucination" verdicts when an agent mentioned a product at position 6+; the cap was removed and the affected runs rejudged.
• Multi-judgment consistency: judge_runs is a fixed integer (default 2). Each judged case gets exactly that many judgments and the one closest to the median is kept, which damps single-sample judge variance. An early-stop can skip remaining judgments within that fixed bound once they agree; at the default of 2 it is a no-op.
• Structured output: the judge returns a fixed JSON schema, and scoring is computed from the extracted fields deterministically, not from a free-text grade.

Judge formatting failures. When a judge fails to return parseable structured output (timeout, rate limit, malformed JSON), the case is stored with a null score rather than a guessed one. This bit a separate eval track once (a compliance run lost 91 of 310 scores on one model when a judge's structured-output layer failed silently), and the run looked "complete" while a large fraction of cases had no score. This class of failure is now detected by a score-completeness check that reports how many scores are missing and warns when more than 5% of a run's cases have no score, so a run with many nulls is caught before it reaches the board rather than silently deflating a model.

Tool-channel judging (fixed). Some models emit their answer through the tool channel with empty response text. The earlier judge gave these a free pass on faithfulness, and one reasoning model had the large majority of its faithfulness cases scored as empty-text. The judge was fixed to grade the channel the model actually used, so a model that answers through tool calls is judged on that content. After the fix that model stayed within a few points of the top on faithfulness. This is disclosed because it changed a headline number.

Same-judge reproducibility floor. Re-judging a 498-case control sample with the same pinned judge model (gemma-4-31b-it-nvfp4, temperature 0.1, judge_runs=2) flipped 3.4% of pass/fail verdicts (95% CI [2.0, 5.0]), with a mean absolute score drift of 0.022. The flips are threshold crossings of small score wobbles, concentrated in the faithfulness metric whose threshold is 1.0. This is the measured floor on how reproducible the same judge is against itself: differences between two models within a few points of each other should be read against this floor, which the bootstrap CIs and the tier logic already absorb. Source: 2026-07-17 judge-seam probe (results/judge_seam_probe.db).

Retrieval configuration

The bench runs Agent Studio's product-default search tool configuration, unmodified: the fair-harness rule is that the bench inherits the product default rather than tuning retrieval for the eval. attributesToRetrieve is not set, so the Algolia API returns all attributes of every hit, and those hits pass into the model's context verbatim, without truncation or reformatting. Hit count is set by each agent's own configuration (typically 10-20 per search); the harness does not cap it, because hiding results from the judge would corrupt response-scoring.

The consequence is that cost per query reflects what an out-of-the-box Agent Studio agent pays on a catalog of this shape: a catalog with bulkier records costs more, and a trimmed attributesToRetrieve costs less. Because this is a product default and the same payload is served to every model, cross-model comparisons are unaffected.

Cost measurement

Cost is computed per completion from the tokens the run actually recorded, priced at published per-model rates:

cost = (prompt_tokens * input_price
        + (completion_tokens + reasoning_tokens) * output_price) / 1e6

Prompt tokens are billed at the model's input rate; completion tokens and reasoning (thinking) tokens are both billed at the output rate, because providers meter thinking tokens as output. Reasoning tokens are included even when they are not shown to the user, so a reasoning model is not made to look cheaper than it bills. Prices come from a single table (analysis/pricing.py, prices as of 2026-07-17); the same function computes the per-completion cost stored in the database and the cost shown on the board, so the two cannot diverge. A published price can lag a provider's change by a few days between refreshes.

The board reports the mean cost per query with a bootstrap CI. Cost is an annotation, not a ranking axis: it depends on token usage, which shifts with prompt and retrieval size (see Retrieval configuration), so it is read alongside quality rather than ranked on.

Latency measurement

Latency is measured per completion and reported as three quantities:

• TTFB (time to first byte): time to the first byte of the response body.
• TTFT (time to first token): time to the first user-visible text token (the first text-prefixed line in the stream).
• TTLT (total latency): the full round-trip time to the completed response. In code this is the completion's total latency_ms; the leaderboard surfaces it as ttlt.

Latencies are reported as medians with bootstrap CIs, because latency distributions are skewed and a mean is dominated by outliers.

Serving conditions and caveats. Latency is not directly comparable across providers, and the board annotates it instead of ranking on it. Reasons:

• Agents run on a shared Agent Studio cluster; a model's latency includes serving and scheduling effects that are not intrinsic to the model.
• One provider's numbers to date were served via Azure rather than the provider's own endpoint. A transatlantic hop from US staging inflates latency, while a provider's own provisioned throughput could move it the other way. That confound is stated wherever the affected numbers appear, and a clean re-run on the direct endpoint is the honest baseline before latency is quoted.
• Priority and load on the shared cluster shift latency between runs.

Quality axes are less sensitive to serving conditions than latency, which is why latency is an annotation and quality is the ranking.

Limitations and domain sensitivity

Where models fail. Failures cluster in specific query shapes:

• Navigational queries, where the shopper already knows the exact product, reward exact retrieval and punish helpful-but-broad behaviour.
• High prior-knowledge categories (for example fragrance) where a model's training-time knowledge competes with the catalog and can pull answers off the retrieved set.
• The language dimension: non-English queries against an English catalog. The dataset spans 17 non-English languages, weighted toward Spanish (heaviest), then Chinese, French, Vietnamese, Korean and Portuguese, Hindi, Japanese, Arabic and Tagalog, with a long tail of German, Polish, Russian, Italian, Thai, and Malay. Note it does not currently include Indonesian.

Scope of the catalog. The public board is a single catalog, English-primary. Results generalise to similar broad retail catalogs; they are not validated on other verticals, languages-as-catalog, or catalog sizes without a separate run.

Relevance metric. An audit found that the Relevance metric's absolute scores are depressed by cases where the generator attached constraints the query did not express (phantom price and brand ranges) and by an over-literal keyword axis. Fixes have shipped for judge truncation, category text-fallback, and keyword normalisation, and the affected runs were rejudged. Pairwise rankings are stable because the noise floor is shared across models, but treat absolute Relevance percentages as conservative until the remaining generator-side fixes land. Progress is tracked in docs/audit/.

Safety unwinnability (disclosure). The current public dataset (sw_v134) contains only relevance, faithfulness, and language cases; it has no safety cases, so Safety does not appear on the current board. In the code, the safety metric is deferred (weight 0) pending a Guardrails redesign, and its current threshold is 0.70. Where Safety was measured on an earlier dataset with a zero-tolerance threshold (1.00) on a subjective LLM rubric (medical claims, unsafe assurances, liability), about 80% of safety cases were unwinnable in the sense that no model in the lineup cleared the threshold. That result is what motivated deferring the metric: at zero tolerance it measures "least bad" rather than "good", so its absolute numbers are not a clean quality signal. Safety, when present, is held out of the public board and the Overall column.

Scope statement

• In scope: Agent Studio agentic search quality (relevance, faithfulness, language) on the Spencer & Williams retail catalog, with cost and latency as annotations.
• Guardrails / safety are a separate evaluation track, not part of the public quality ranking. The safety rubric in this repository is deferred (weight 0). The guardrails track is intended to use deterministic RuLES-style scoring rather than an LLM judge; that scoring is not yet implemented here. Not yet published.
• MCP (agent-to-tool integration over Model Context Protocol) is out of scope for this board. This matches the product default: a new Agent Studio agent ships with native function tools (search, display results) and MCP connections are opt-in, so the board measures the configuration agents actually start with.

Update and versioning policy

Every published number carries three version coordinates, so a provider can ask "what changed between my two scores" and get a diff rather than a shrug:

1. Dataset version (for example sw_v134): the eval cases and their ground truth. A new dataset generation bumps this.
2. Harness / metrics version: the scoring code (metric definitions, judge prompts, thresholds). A bug fix to a metric bumps this and triggers a dated restatement.
3. Run / promotion date: when the run was executed and promoted to the public board.

A promotion is the release act. config/golden.yaml names the canonical experiment (spencer_williams/v134_24models) and the builder reads that pin, so the board is a pinned run rather than newest-wins. If the pinned run is missing, the builder raises rather than silently falling back to the newest summary, because a silent fallback once let a small 4-model customer run overwrite the full board. A regression floor (MIN_PUBLIC_MODELS = 15) refuses to publish a board smaller than 15 models.

The harness version is not yet a single identifier. The dataset version is stamped in the dataset file, and the promotion date lives in golden.yaml, but the scoring code has no programmatic version string today. The closest signals are a per-score metric_type label written next to each result (which records, for example, a rejudge pass) and human-maintained notes in golden.yaml. Planned: a proper harness/metrics version. An acceptable interim is the git SHA of the metrics registry at run time, stamped alongside each score so a restatement is attributable to a specific code state.

When a metric bug is found, the fix bumps the harness version, the affected scores are restated, and the change is dated in the Changelog below. Scores are not quietly edited.

Held rows. A model can be held out of the public board pending investigation while its data is kept in the source. The policy ran end to end on the language check (issue #184): on 2026-06-03 claude-opus-4-7 and claude-opus-4-8 were held because both dipped on the language metric from the same run and a shared harness or judge artifact could taint both, while claude-opus-4-6 stayed published. The investigation confirmed the dip was a metric artifact (the search dimension penalised bilingual multi-query search proportionally), and on 2026-07-17 the fix was applied and all 24 models re-scored; both rows were unheld. A deprecated row (claude-sonnet-4-5, superseded by claude-sonnet-4-6) remains hidden. The current public projection is 23 of the 24 models in the source run.

Reproducibility for providers

The honest current state: the pipeline (run, judge, aggregate) is reproducible from this repository, but the eval-dataset construction is not yet public. A provider cannot today regenerate sw_v134 from scratch.

What we can share on request at llm-leaderboard@algolia.com, per model:

• the per-case traces: the query, the agent's tool calls (search terms and filters), the retrieved records, the response text, and the judge's verdict and reasoning for each case.

What a provider needs to independently challenge a result:

• pick the cases where the provider's model scored low,
• inspect the trace: was the search query reasonable, did retrieval return the right records, did the reply feature them, and is the judge's verdict defensible,
• if the verdict is wrong, cite the case ID and the specific channel (tool call, retrieval, or text) where the scoring is unfair.

Planned, not yet published: a public case-construction spec and a subset of cases with their full traces, so a provider can reproduce scoring end to end on those cases.

Changelog

• 2026-07-17 (dataset sw_v134, language search-scoring rule lang_search_v2_2026-07-17): restated the language metric. The search dimension moved from proportional translated/total (multiplicative, which penalised bilingual multi-query search — a native-plus-English pair scored 0.5 and failed the threshold) to any-translated credit (1.0 if at least one query is in the catalog language, else 0.0). The response dimension (LLM judge) is unchanged. All 24 configs were re-scored deterministically from stored completions; the source DB sw_v134_24models_gold.db was preserved read-only and the pin moved to the restated copy sw_v134_24models_gold_r2.db. The language-metric hold on claude-opus-4-7 and claude-opus-4-8 (#184) is resolved by this restatement — both recover (opus-4-7 91.0 → 97.8, opus-4-8 75.6 → 89.8 language pass) and are unheld. Public projection: 23 rows. The same restatement also corrected the cost metric: the board now prices reasoning (thinking) tokens at the output rate, using the one shared formula in analysis/pricing.py (see Cost measurement). Previously it priced only prompt and completion tokens, which understated reasoning models. The clearest case was grok-4.20-0309-reasoning, which read as cheaper ($0.0464/query) than its non-reasoning sibling ($0.0510) despite about 1,983 average reasoning tokens; it is now correctly dearer ($0.0514). Cost is an annotation, so no ranking changed; the largest per-model cost increases were reasoning-heavy models (gpt-5-nano +46%, gpt-5-mini +22%, gemini-2.5-flash +16%, gemini-3.1-pro +11%, grok-4.20-reasoning +11%). The cheapest model is unchanged (DeepSeek-V4-Flash, $0.0023).
• 2026-06-03 (dataset sw_v134, harness version not yet stamped, promoted 2026-06-03): promoted spencer_williams/v134_24models as the public board (24 models, 1,500 cases). claude-opus-4-7 and claude-opus-4-8 held pending the language-score check (#184); claude-sonnet-4-5 deprecated. Public projection: 21 rows.

Contact

Questions about the methodology, or requests to validate results, go to llm-leaderboard@algolia.com.