Listen to the brief:

As developers, we were taught to build modular, testable functions with one correct output for a given input. We knew they worked and continued working because our test suite ran them regularly and flagged any outputs that deviated from expectations.

But in the agentic era, those assumptions break down. LLMs are non-deterministic, meaning they can produce different results for the same input. For example, a shopper might ask the search agent on your site, “Show me tropical shorts for vacation”. One response might recommend linen shorts with palm prints, while another might recommend Hawaiian-pattern swim trunks. Neither of those answers are necessarily wrong, but if we tried to test the agent the same way we’d test any old software, it would likely mark the latter as wrong because it’s less likely to match the exact output we were expecting.

The same problem comes up in support. When a customer asks about a delayed order, a return policy, or account recovery, the agent has a lot of response options. It might ask clarifying questions, look up various support articles, or just use different wording to explain a complex concept like supply chain delays. That support agent doesn’t need to be predictable word-for-word; it needs to be reliably useful. Exact match testing obscures the agent’s performance since it confuses consistency with quality, rewarding sameness instead of usefulness.

This brings up an interesting question. If exact match testing doesn’t cut it, how can we actually evaluate the quality of non-deterministic agents? The answer: understand what makes a test case meaningful, generate a whole dataset of cases from our product catalog, and pull insights from those results with statistics. Let’s break each of these down.

Define cases around outcomes, not exact answers

A useful agent evaluation starts by defining a case, or what exactly we’re testing. A case includes three components:

1. The input. What query, prompt, or trigger is the agent responding to? What conversation history is included? What context is the agent provided?
2. The expected outcome. What means success in this scenario? What conclusion are we hoping the agent reaches from its context, data retrieval, and reasoning?
3. The evaluation focus. How can we evaluate if the agent produced the expected outcome? If surfacing products, which of their attributes would inform the evaluator’s decision? If surfacing support information, what could we compare the response to in order to judge its accuracy?

For example, on a retail ecommerce site, we might see this scenario play out:

1. The input. The user types into the AI-enabled search bar, “I need affordable leather wallets for my wife”.
2. The expected outcome. The agent should infer that the shopper is looking for women’s wallets at accessible price points, produce a brief introductory text response, then recommend some matching products.
3. The evaluation focus. The evaluator can assess how well the recommended products match the query using the product_type, category, gender, price, and availability keys in our product catalog dataset.

In this scenario, the expected outcome is not too strict. We haven’t specified a few individual wallets that the agent will be rewarded for recommending, and we haven’t penalized the agent for recommending anything outside of that narrow set. Instead, we’ve defined the general goal more flexibly, and the evaluation focus matches that level of specificity.

A sample good response given the query, “I need affordable leather wallets for my wife”. It includes a very brief introductory comment from the agent, then three highlighted wallets which are all in stock, affordable, and designated as leather and for women.

A sample bad response given the query, “I need affordable leather wallets for my wife”. It includes a whole paragraph of irrelevant text and eight search results. One is not a wallet, another is not designed for women, one isn’t in stock, and two aren’t leather.

In the bad example image, the agent arguably created more friction than it removed. Although three of the eight recommended products might actually match the query — the same absolute count as the good agent’s response — those matches are buried in irrelevant results, meaning that the shopper probably didn’t save time, effort, or emotional energy by using the agent. Note that the evaluation focus for this case was designed to penalize this perceived “pointlessness” above all else, since that’s what actually affects the sale directly.

This framework works outside of sales, too. For example, in a support agent:

1. The input. The user asks the support agent, “How do I reset my password if I can’t access my email?”
2. The expected outcome. The agent should look up the approved account recovery process from the company policies dataset and explain it in clear, simple terms.
3. The evaluation focus. The evaluator should compare the steps in the response against the documented policy and look for accuracy, helpfulness, and grounding in the policy. Points are deducted for any invented steps.

Note again that neither the expected outcome nor the evaluation focus contain a rigid transcript of the perfect answer. They just describe what success looks like and how we’ll know the agent produced a successful response.

Good success criteria are specific enough to catch failures, but flexible enough to allow valid variation. Some examples:

However, in each of these use cases, there are potentially infinite queries, each of which could have various valid responses. How can we create a set of cases that will efficiently test some model across all of those situations?

Generating and testing the case dataset

You already have a dataset of all the things your agent needs to know; why not use that to generate evaluation cases? Instead of starting with a query and trying to guess the right answer, start with content where the expected answer is already implied. Then, generate the kinds of questions users might ask to reach that answer. This is a much more scalable approach than defining them all manually.

For example, for each product in the product catalog index, you could run it through a sufficiently modern LLM with the prompt below to generate test cases.

Create ecommerce search and recommendation evaluation cases from this product record.

Generate 1–6 realistic shopper queries that this product should satisfy, depending on how many distinct intents are supported by the product data. Cover different intents when available: category search, attribute search, occasion/use case, style preference, price sensitivity, compatibility, gift intent, seasonal need, or brand/model-specific lookup.

For each query, define the expected outcome broadly enough that equivalent or substitute products can also pass, but specifically enough that irrelevant results should fail.

Product:
{{PRODUCT_RECORD}}

Return an array of JSON objects each representing a single test case.

If you use a model capable of outputting deterministically structured JSON, defining the output as an array of objects lets you just join that array to a running list of every product’s test cases. The end result is a thorough dataset of test cases designed specifically for your product index.

Here’s a similar prompt, assuming you’re starting with an index full of support articles:

Create support-agent evaluation test cases from this help center article.

Generate only the realistic user questions this article can answer or route. Do not create cases that require information outside the article. Include simple, ambiguous, and edge-case questions when the article supports them.

For each test case, return:
- user_input: a natural customer question
- expected_outcome: what the agent should explain, do, or escalate based on the article
- evaluation_focus: the specific policy details, steps, constraints, exceptions, or escalation rules the evaluator should check

Article:
{{SUPPORT_ARTICLE}}

Return valid JSON.

This setup also assumes that your support articles are well-structured. Cases where the agent would need to combine the contents of multiple articles together are unlikely to be generated by this prompt by design. If your current help center structure requires that the agent access multiple articles regularly, you could restructure those articles, manually add select combination cases to test, or even run a new prompt for every combination of articles.

With a fleshed-out dataset of test cases, you can run an agent using a particular model through all of them and assess its responses. Then, switch the model the agent is using and assess its responses to the same cases, and repeat for as many models as you’d like to test. It can be very helpful here to have some framework for switching models efficiently, so here at Algolia, we use Agent Studio. Setting up each model as its own provider abstracts away the complexity of managing API keys and provider-specific data structures. With Agent Studio, one set of credentials and one clearly-documented API lets you send prompts to an agent that already has the ability to search through the indexes you’ve granted it access to.

Then for each entry in the test case dataset, we can use a different LLM to compare each agent’s answer against the expected outcome using that flexible evaluation focus we defined earlier. Instead of matching by specific wording or arbitrary requirements, we wrote those evaluation metrics to judge whether response satisfies the intended meaning. For non-deterministic agents, meaning is the unit of quality.

Compare versions statistically, then improve the system

Once we’ve judged each model’s responses semantically, we can aggregate those judgments to actually compare them. Non-deterministic systems vary across both cases and repeated runs, so a screenshot of one good or bad response in isolation doesn’t tell you much about the underlying model. Before drawing conclusions, ask: How often, and under what conditions, does this version produce a successful response? A sufficiently large sample can reveal patterns that individual examples cannot, so trust the distribution over a demo.

Because we’re balancing several dimensions of quality, both the overall score and the breakdowns are meaningful. If a model’s overall score rose, we’d still want to know whether it:

• improved quality at the cost of becoming more verbose,
• performed better on common requests but invented more details in edge cases, or
• improved across most product categories while making one important category significantly worse.

In these cases, there’s more to the story than just the one “model score”. That average tells you whether performance moved, but the breakdown tells you why. Breakdowns can also highlight when a change in results may have come from the experiment itself rather than from the model (for example, when changes to the product catalog produce different test cases between runs).

The comparison is only useful if it informs the next version. That creates a simple improvement loop:

1. Inspect where a configuration gained or lost quality.
2. Adjust the prompt, model, retrieval, ranking, data, or interface.
3. Run the same evaluation again.

Individual responses can help explain why a score moved, but the next decision should come from the broader pattern instead of whichever example looks most convincing.

Agent Studio makes this process easier by letting teams configure and test different models and agent setups against the same search indexes. The goal is not to find one perfect response, just to establish a repeatable way to make each version measurably better than the last.

Measuring what matters

Non-deterministic agents do not require lower standards; they just require better definitions of quality. By evaluating outcomes instead of exact wording, generating cases from the data your agent actually uses, and comparing performance across a meaningful sample, you can replace subjective demos with repeatable evidence. The result is not a test that proves an agent will always produce the same answer. Instead, you’ll have a system for determining whether the agent is reliably useful and improving across iterations.