Agent Studio benchmark
We ask every model real questions on a real catalog, and a judge checks every claim. One frozen case: the bait is deliberate, no Artfan calendar exists in the catalog.
Grounded answer
“The wine glass is in the catalog: 50th Birthday Gift Wine Glass by Gelid, $12.00. The Artfan calendar isn’t here; want other calendars?”
“Every claim traces to a retrieved record. The missing product is named as missing, not invented.”
Honest refusal
“I couldn’t find an Artfan calendar in this catalog. Could you double-check the name? I don’t want to point you at the wrong product.”
“Nothing invented. A clean refusal is hallucination-free; how much it helps is scored separately.”
Illustrative answer, written to show the rubric.
Hallucinated answer
“Great choices! The Artfan 2026 Wall Calendar is $18.99 and beautifully illustrated. Both are customer favorites.”
“The retrieval returned no Artfan calendar. Name, price, and praise have no basis in the results.”
Illustrative answer, written to show the rubric.
Standard benchmarks score a model answering from memory. An agent has to run this loop, and any step along it can break the answer.
{
"tool_name": "algolia_search_index_<index>",
"args": {
"query": "L.O.L. Surprise! Girls 2 Piece Pajama Set B084T4MWJT",
"userIntent": "Care instructions for a specific product (ASIN B084T4MWJT)."
}
}
L.O.L. Surprise! Girls 2 Piece Pajama Set B084T4MWJTLOL Surprise pajama OR dress OR outfit girls[
{ "objectID": "B084T4MWJT",
"name": "L.O.L. Surprise! Girls 2 Piece Pajama Set",
"price": 12.99, "rating": 4 },
{ "objectID": "B07D77VSVQ",
"name": "Marvel Boys' Avengers Pajama Set",
"price": 24.95, "rating": 4 }
]
"For the L.O.L. Surprise! Girls 2 Piece Pajama Set (ASIN B084T4MWJT), it's made of 100% polyester and machine washable. This matches reviews noting it washes well with no shrinkage."Every claim traces to the retrieved record above. The judge scored this trace 1.0 for faithfulness.
Repeats until the agent can answer
Same loop, same harness, every model. Here's how we test that loop ↓
How it's built
Every number on the board starts as a product record and survives five steps. Click a step to see the real artifact it produces.
One of 395,380 records in the live snapshot
Generated from the catalog record in step 1
“I found the perfect gift for my mom! Can you show me that amazing Artfan calendar and the wine glass?”
An “amazing” Artfan calendar. No such product exists in the catalog.
6 stainless steel double wall espresso cups set
Stops after one search. The exact set is already in the results.
6 stainless steel double wall espresso cups set
"stainless steel" "double wall" espresso cups 6
double wall stainless steel espresso cup set 6 pack
Same product, three phrasings. Widening the net before settling.
gpt-5.4, response excerpt, claims extracted for judging
The response correctly identifies the SS Ecom product from the context, accurately stating the price ($24.95), capacity (3.5 oz), and set size (6 cups). The calculation of the cost per ounce is mathematically correct based on these supported facts. No claims contradict or lack evidence in the provided context.
Models 1 and 2 share Tier A. Their head-to-head difference on the same cases could plausibly be zero, we can't call a winner. Model 3's gap vs Model 1 is well above zero, so it separates into Tier B.
We also A/B the harness itself: record formats, retrieved fields, forced search parameters. Benched variations
Full leaderboard
Each cell shows the gap to the column leader. Click a row to pin its detail, a heading to sort. The harness also A/Bs its own knobs.
Run this benchmark 100 times and 95 times the score lands in the band. The solid center is the most likely value. Wider bands mean more variability.
Tradeoffs
Two tradeoffs decide most picks: how fast, and how much. Every model is plotted by answer quality against response time, then against cost per query. Each dashed line joins that chart's frontier: the models where nothing else is both faster (or cheaper) and better. Click a point to pin its detail.
Screen-sharing with a customer? Add ?expand to the URL to open every detail.
Pick a model on the evidence, then ship it on the platform these numbers came from.
Model provider? See how to reproduce or challenge a result