Evaluations

Agent Studio Evaluations assess the quality of your agent’s answers before users see them. Generate test cases from an Algolia index, run them against an agent snapshot, and use the scores to identify areas to improve.

​

How evaluations work

An evaluation has three phases:

1. Snapshot: Agent Studio records the agent’s configuration, so every result maps back to an exact version.
2. Test cases: Agent Studio generates the test cases from the agent’s Algolia Search index in the background. You don’t upload a dataset.
3. Run and score: each test case is a conversation with your agent, which takes a few large language model (LLM) calls to search and answer. A scoring layer then grades each answer against the selected metrics.

Each metric you select gets its own set of test cases and its own score.

​

Before you begin

• If you’re new to Agent Studio, start with the overview.
• You need an Algolia application with Agent Studio access. Evaluations is available to everyone with that access, so you don’t need to enable anything extra.
• The agent you evaluate must have an LLM provider set and an Algolia Search tool with a configured index. Agents that don’t meet these requirements show as incompatible in the agent picker.

​

Create and run an evaluation

1. Go to the Algolia dashboard and select your Algolia application.
2. On the sidebar, select Generative AI, go to Agent Studio, and select Evaluations.
3. On the evaluations list, select Create evaluation.
4. Under Target, select the agent to evaluate and the index to use for test case generation.
5. Under Metrics to evaluate, select one or more metrics. Each metric shows examples of answers that pass and fail. Use Select all to select every available metric.
6. Under Cases per metric, choose how many test cases to run for each metric. Start with at least 30 test cases per metric. Scores remain estimates, and more cases can reduce uncertainty. If the results show Low confidence, create another evaluation with more cases per metric. You can run up to 10,000 test cases per evaluation.
7. Optional. Under Details, enter a name and description. If you leave the name empty, Agent Studio generates one.
8. Review the Summary card. It shows the total number of test cases and an estimated token range for your LLM provider. Select Start evaluation and confirm.

​

Evaluation metrics

If your application has custom metrics, they also appear in the metric list.

​

Track progress

The evaluation page shows live progress: analyzing your agent and index, generating test cases, then running and scoring them. The status badge on the evaluations list uses the same states:

​

Read the results

When the evaluation completes, the Results card shows:

• Average score: the average test case score, out of 100, with a range that shows uncertainty in the estimate. For example, a range from 78 to 82 indicates less uncertainty than a range from 60 to 100.
• Pass rate: the share of cases that scored at or above their metric’s threshold. The default threshold is 70 out of 100.
• Judged: how many cases are graded so far, out of the total.
• Error rate: the share of cases the agent or the scoring layer couldn’t process.

The Metrics table breaks the results down per metric: its score, pass rate, and the number of cases it ran.

​

Inspect test cases

The Cases table lists every test case with its query, metric, difficulty (Easy, Medium, or Hard), status, and score. Select a case to open its detail page, which shows:

• User input: the query the agent had to answer.
• Verdict: whether the answer passed or failed, its score, and the passing threshold. For answers scored by an AI judge, this section also includes the judge’s explanation. For relevance, it also shows which terms were found in the retrieved records and in the response.
• Usage: input, output, and reasoning tokens, and the duration.
• Agent response: the full text of the reply.
• Tool calls and Retrieved records: the searches the agent made and the records it used.

​

Low-confidence results

When the evaluation doesn’t have enough consistent data to pin down exact scores, the results show a Low confidence badge with the message “This result is a rough estimate”. This happens when the evaluation judged fewer than 30 cases, or when scores vary a lot across cases. Treat low-confidence scores as rough estimates. Create a new evaluation with more cases per metric to reduce uncertainty.

​

Manage evaluations

On the evaluations list, each row shows the agent, the status, the score with its likely range, and when it was created. From the evaluation page you can:

• Retry: run the evaluation again using previously generated test cases.
• Cancel run: stop the evaluation after the case it’s currently running. A canceled evaluation can’t be restarted or resumed.
• Delete: permanently remove the evaluation, along with all its runs and results.

​

Limits and troubleshooting

​

Troubleshooting

• An agent is marked incompatible: the agent must have a configured LLM provider and an Algolia Search tool with a configured index.
• The evaluation failed: open the evaluation page for error details, then retry it. If an individual test case encounters an error, the remaining test cases continue to run. Check the Cases table for error details.
• The scores don’t match your expectations: check the per-case verdicts before changing the agent. Each verdict includes the judge’s explanation of the score, and small case counts make scores unreliable, so add cases before drawing conclusions.