A/B Testing for AI Models

An A/B test splits users randomly into two groups: the control group (Version A, usually the baseline model) and the treatment group (Version B, the new candidate model).

To ensure validity, user assignment must be deterministic and persistent, and we must define a clear taxonomy of metrics before launching the test.

In a standard A/B test, users are randomly allocated to two groups: the control group (A), which receives predictions from the existing baseline model, and the treatment group (B), which receives predictions from the new candidate model. To ensure valid inference, traffic allocation must be random and persistent. We achieve this using a cryptographic hash of the user's ID concatenated with a test-specific salt, taking the modulo: $\text{hash}(user\_id + salt) \pmod 2$. This prevents user selection bias, maintains independence between buckets, and ensures a user experiences a consistent model throughout their session. We also run A/A tests prior to the A/B test (routing both groups to the baseline model) to confirm the randomization algorithm works correctly without pre-existing group imbalances.

Before launching an A/B test, we define a clear metric taxonomy:

1. Core Business Metrics: The primary targets we want to improve (e.g., click-through rate, conversion rate, revenue per user).

2. Guardrail Metrics: Crucial metrics that must not degrade (e.g., unsubscribe rate, uninstall rate).

3. Operational Metrics: Metrics measuring system performance (e.g., inference latency, error rates). If Model B increases conversions but doubles latency, it may violate operational guardrails, making it undeployable in production environments.

To ensure that the results of an A/B test are mathematically sound, we must pre-calculate the required sample size using power analysis, preventing false negatives and the pitfalls of early stopping.

Stopping an experiment early because the current results look promising invalidates the test statistic, leading to inflated false positives.

The sample size $n$ required per group depends on: the baseline conversion rate $p_1$, the Minimum Detectable Effect (MDE, $\Delta = p_2 - p_1$) which is the smallest change we care to detect, the significance level $\alpha$, and the power $1-\beta$. For comparing two proportions, the sample size is approximated as:

$$n \approx \frac{2 \left( z_{1-\alpha/2} + z_{1-\beta} \right)^2 \bar{p}(1-\bar{p})}{\Delta^2}$$

where $\bar{p} = (p_1 + p_2)/2$ and $z$ represents standard normal quantiles. An A/B test must run until this sample size is reached to ensure that if there is a real difference of size MDE, we have an $80\%$ chance of detecting it. Running tests without reaching this size yields underpowered experiments with high false negative rates.

A major experimental error is peeking—checking the $p$-value daily and stopping the test as soon as $p < 0.05$. Because the $p$-value fluctuates randomly, peeking dramatically increases the Type I error rate (false positive rate) from $5\%$ to over $30\%$. If daily peeking is necessary, we must use sequential testing frameworks, such as Wald's Sequential Probability Ratio Test (SPRT). SPRT computes the likelihood ratio after each observation $n$:

$$\Lambda_n = \prod_{i=1}^{n} \frac{P(x_i \mid H_1)}{P(x_i \mid H_0)}$$

We define two stopping boundaries based on the target Type I error $\alpha$ and Type II error $\beta$: $A = (1-\beta)/\alpha$ and $B = \beta/(1-\alpha)$. The decision rules are:

1. If $\Lambda_n \ge A$, reject $H_0$ and stop the test.

2. If $\Lambda_n \le B$, accept $H_0$ (fail to reject) and stop the test.

3. If $B < \Lambda_n < A$, continue collecting data. This bounds the FWER and allows early stopping only when the evidence is overwhelmingly strong.

Traditional A/B testing is static, routing $50\%$ of traffic to a potentially inferior model for weeks. Multi-Armed Bandit (MAB) algorithms offer a dynamic alternative that minimizes opportunity cost.

By updating route allocations in real-time based on accumulating rewards, bandits balance the trade-off between exploring new options and exploiting current winners.

A/B testing separates exploration (finding the best model) from exploitation (deploying it). MAB algorithms merge them, dynamically routing traffic in real-time. We quantify performance using cumulative regret—the difference between the reward from always pulling the optimal arm and the expected reward from the chosen strategy:

$$R(T) = T \mu^* - \sum_{t=1}^{T} E[\mu_{a_t}]$$

where $\mu^*$ is the expected reward of the optimal arm, and $a_t$ is the arm pulled at time $t$. Bandits aim to minimize cumulative regret over time, reducing the traffic routed to lower-performing algorithms during the testing phase.

Two primary bandit strategies are used in AI systems:

1. Upper Confidence Bound (UCB1): An optimistic algorithm that selects the arm maximizing the upper bound of its confidence interval:

$$a_t = \arg\max_{k} \left( \hat{\mu}_k + c \sqrt{\frac{\ln t}{N_k(t)}} \right)$$

where $\hat{\mu}_k$ is the empirical mean reward of arm $k$, $N_k(t)$ is the number of times arm $k$ was pulled, and $t$ is the total steps. The exploration term is derived from Hoeffding's Inequality, ensuring that arms with high uncertainty are pulled to gather information.

2. Thompson Sampling: A Bayesian algorithm. We model the success rate of each arm $k$ using a Beta distribution $\text{Beta}(\alpha_k, \beta_k)$, where $\alpha_k$ and $\beta_k$ are success and failure counts. For each user, we sample a probability $\theta_k$ from the Beta distribution of each arm, and route the user to the arm with the highest sample:

$$\theta_k \sim \text{Beta}(\alpha_k, \beta_k) \quad \text{and} \quad a = \arg\max_k \theta_k$$

After observing the user's action (success/failure, $r \in \{0, 1\}$), we update the corresponding Beta distribution's parameters: $\alpha_a \leftarrow \alpha_a + r$ and $\beta_a \leftarrow \beta_a + 1 - r$. This automatically shifts traffic to the winning model while maintaining exploration of other arms based on their parameter uncertainty.

Production AI systems are subject to environmental dynamics that can violate experimental assumptions and distort results.

User adaptation, spillover effects in graphs, and structural segment imbalances can make a superior model look inferior when aggregated.

The novelty effect occurs when users interact with a new model simply because it is different, creating a temporary spike in metrics that decays over time. The test must run long enough to discount this. In social networks or two-sided marketplaces, network effects (spillover) violate the assumption that users behave independently. If a treatment user's recommendations affect a control user's behavior, the treatment effect is underestimated. We mitigate this using cluster-based randomization. Additionally, many A/B test metrics are ratio metrics (e.g., Click-Through Rate = total clicks / total impressions), where the numerator and denominator are both random variables. To calculate the variance of a ratio metric correctly for hypothesis testing, we must apply the Delta Method, which uses a Taylor expansion to approximate the variance:

$$\text{Var}\left(\frac{X}{Y}\right) \approx \frac{E[X]^2}{E[Y]^2} \left( \frac{\text{Var}(X)}{E[X]^2} - \frac{2\text{Cov}(X,Y)}{E[X]E[Y]} + \frac{\text{Var}(Y)}{E[Y]^2} \right)$$

Using simple sample variance on ratios without this correction leads to severely deflated standard errors and false positives.

Simpson's Paradox occurs when a trend appears in different sub-groups of data but reverses when they are aggregated. Suppose Model B has a higher click-through rate than Model A on mobile users and a higher rate on desktop users. However, when we pool all users, Model A appears to perform better. This happens if the proportions of mobile/desktop users are unequal between the control and treatment groups due to randomization failure, highlighting the danger of reporting overall averages without verifying segment balances. Mathematically, let $C$ and $T$ represent control and treatment. We can have $P(Y=1 \mid T, X=\text{mobile}) > P(Y=1 \mid C, X=\text{mobile})$ and $P(Y=1 \mid T, X=\text{desktop}) > P(Y=1 \mid C, X=\text{desktop})$ but $P(Y=1 \mid T) < P(Y=1 \mid C)$ due to severe sample size imbalances.