Skip to main content
Algolia A/B testing uses Bayesian statistics to help you read your results. This page covers how the Bayesian approach works, how to read its output, and how it differs from the frequentist approach.

​
How Bayesian experimentation works

A Bayesian test combines what it sees in your data with what you knew before the test began. It asks one question: “given the data so far, what’s the chance that variant B beats variant A?” A frequentist test asks something else: “how odd is this data if the variants are in fact the same?” Every Bayesian test starts with a prior. A prior is a curve that shows what you believe about a metric before new data arrives. As the test runs, Algolia folds new data into that prior. The result is a posterior. The posterior is an updated curve that reflects the prior and the results so far. Algolia redoes this math with each batch of new data. This is why you can look at a Bayesian test whenever you want. Each check gives you a sound answer about which variant leads. You don’t have to wait for a set number of users, and a mid-test check doesn’t break the math.

​
How to interpret Bayesian results

Bayesian results are probabilities, so you can act on them without translating them first. Your test results include:
  • Probability to be better. For example, “Variant B has a 95% probability of beating the control.” This is a plain statement about the thing you care about. We recommend 95% as the bar for a result you can trust.
  • Credible interval. A range that holds the true value of a metric, with a stated chance. For example, “There’s a 95% chance that the true uplift falls between 1.2% and 4.8%.” That metric is often the gap in conversion rate between the variants. You can’t read a frequentist confidence interval this way.
  • Evidence status. An evidence status of “enough” means the test has gathered enough data. For Bayesian tests, this status rests on e-values. The bar is an e-value of 20 or more.
An e-value weighs your data against the case for no real difference. It builds up as data arrives, and it holds up no matter how often you check it. That’s what makes it safe to watch a test as it runs. Higher is stronger: an e-value of 20 means the data fits a real difference 20 times better than no difference. To learn more, see Safe testing. Read these numbers as you’d expect. A higher probability to be better makes a stronger case for a variant. Since the numbers change with each batch of data, you can watch a trend build instead of judging each check on its own. For the other things to weigh before you act, see How to read your A/B test results.

​
Benefits of Bayesian experimentation

  • Direct answers. A probability that “variant B beats the control” answers the business question as asked.
  • Frequent checks, no added risk. Frequentist tests suffer the “peeking problem”: each extra look raises the odds of a false positive. Bayesian tests don’t, so you can check as often as you like.
  • No set duration up front. A frequentist test needs a sample size fixed before you start. A Bayesian test lets you follow it and stop once the evidence is strong.

​
How Bayesian and frequentist testing compare

Both methods are sound ways to test an idea, but they answer different questions. Neither is the better one. Frequentist testing has a long track record and firm guarantees, but its rules are strict. Bayesian testing gives up some of that rigor for room to move and results you can read at a glance. That trade fits online A/B testing, where data keeps coming in and you keep iterating. For how Algolia computes frequentist confidence, see How A/B test scores are calculated.
Last modified on August 31, 2026