P-Values and Statistical Significance

A p-value is a probability, but it is not the probability that the null hypothesis is true or that your model is correct.

Understanding the exact probability statement behind p-values keeps researchers from drawing overconfident conclusions from marginal signals.

The $p$-value is defined as the probability, assuming the Null Hypothesis ($H_0$) is true, of obtaining a test statistic at least as extreme as the one calculated from the observed sample:

$$p = P(T \ge t \mid H_0)$$

where $T$ is the test statistic random variable and $t$ is its observed value. Let us prove that under $H_0$, the $p$-value is uniformly distributed on $[0, 1]$. Let $T$ be a continuous test statistic with cumulative distribution function $F(t) = P(T \le t \mid H_0)$ under $H_0$. By definition, the $p$-value is $p = 1 - F(T)$. We want to find the CDF of the $p$-value, $P(p \le x \mid H_0)$ for $x \in [0, 1]$:

$$P(p \le x \mid H_0) = P(1 - F(T) \le x \mid H_0) = P(F(T) \ge 1 - x \mid H_0)$$

Since $F(t)$ is a monotonic function, we apply its inverse $F^{-1}$:

$$P(p \le x \mid H_0) = P(T \ge F^{-1}(1 - x) \mid H_0) = 1 - P(T \le F^{-1}(1 - x) \mid H_0)$$

By definition of $F$, $P(T \le F^{-1}(u)) = F(F^{-1}(u)) = u$. Thus:

$$P(p \le x \mid H_0) = 1 - (1 - x) = x$$

This proves that $P(p \le x) = x$, which is the exact definition of a Uniform distribution $U(0, 1)$. This uniform distribution explains why setting $\alpha = 0.05$ guarantees a false positive rate of exactly $5\%$ under the null hypothesis.

A $p$-value is NOT the probability that the null hypothesis is true: $P(H_0 \mid \text{Data}) \neq P(\text{Data} \mid H_0)$. It is also NOT the probability that the alternative hypothesis is false. Crucially, a $p$-value does not measure the size or importance of an effect. A very large sample size can produce a highly significant $p$-value (e.g., $p < 0.0001$) for a minuscule and practically useless difference in model performance. Therefore, reporting p-values must always be accompanied by effect sizes and confidence intervals.

When testing multiple hypotheses simultaneously, the probability of encountering a false positive increases exponentially, leading to false discoveries.

To control this risk, we must adjust individual significance thresholds using correction algorithms that control either the family-wise error rate or the false discovery rate.

If we test a single hypothesis at $\alpha = 0.05$, the probability of a false positive is $5\%$. However, if we test $m$ independent hypotheses simultaneously (e.g., checking 100 features for correlation with a target), the probability of committing at least one Type I error (the Family-Wise Error Rate, FWER) is:

$$\text{FWER} = 1 - (1 - \alpha)^m$$

For $m = 100$ and $\alpha = 0.05$, the FWER is $1 - (0.95)^{100} \approx 99.4\%$. This means we are virtually guaranteed to find a statistically significant result purely by chance. In data science, testing many features or model parameters without adjusting for this is a form of 'p-hacking' or 'data dredging', which produces models that capture noise and fail to generalize.

To control FWER, the Bonferroni correction adjusts the significance threshold for each individual test to $\alpha_{\text{new}} = \alpha / m$. While simple, this is highly conservative and reduces statistical power, leading to many Type II errors. A more powerful alternative is the Benjamini-Hochberg (BH) procedure, which controls the False Discovery Rate (FDR)—the expected proportion of false positives among all rejected hypotheses. The BH procedure sorts the $p$-values $p_{(1)} \le p_{(2)} \le \dots \le p_{(m)}$ and finds the largest index $k$ such that:

$$p_{(k)} \le \frac{k}{m} q$$

where $q$ is the target FDR. It then rejects all null hypotheses $H_{(1)}, \dots, H_{(k)}$. Modern genomics and target discovery pipelines also use Storey's q-value, which estimates the proportion of true null hypotheses $\pi_0$ from the data to increase the statistical power of FDR control, avoiding overly conservative corrections when many hypotheses are false.

Significance tests only tell us if an observed difference is likely due to chance, but they do not measure the magnitude of the difference. We use effect sizes to assess practical importance.

A result can be highly statistically significant yet completely trivial in terms of real-world value, depending on the sample size.

An effect size is a standardized, scale-free metric that quantifies the magnitude of an experimental effect. For comparing the means of two groups, Cohen's d is defined as:

$$d = \frac{\mu_1 - \mu_2}{\sigma_{\text{pooled}}}$$

where the pooled standard deviation is:

$$\sigma_{\text{pooled}} = \sqrt{\frac{(n_1-1)s_1^2 + (n_2-1)s_2^2}{n_1 + n_2 - 2}}$$

A Cohen's $d$ of $0.2$ is considered small, $0.5$ medium, and $0.8$ large. Unlike $p$-values, effect size is independent of the sample size $n$, providing a direct measure of the practical strength of the relationship. In ML, reporting Cohen's $d$ along with test accuracy guarantees that our performance gains represent meaningful changes in the prediction distributions.

Suppose a new click-through rate (CTR) prediction model increases CTR from $2.10\%$ to $2.11\%$ in an A/B test with 10 million users. Because the sample size is huge, the $p$-value will be tiny (highly significant). However, the effect size is extremely small. In production, deploying a massive new deep learning model to capture this $0.01\%$ difference might cost more in computation and latency than the revenue generated, demonstrating the crucial difference between statistical and practical significance.

A confidence interval provides a range of plausible values for a population parameter, offering a clearer picture of uncertainty than a single p-value.

When analytical formulas for standard errors are unavailable for complex ML metrics, we use computational methods like bootstrapping and permutation tests to map uncertainty boundaries.

A $(1-\alpha)$ confidence interval for a parameter $\theta$ is an interval $[L, U]$ computed from the sample such that if we repeated the sampling process infinitely, $(1-\alpha)\%$ of the generated intervals would contain the true parameter. For a sample mean $\bar{x}$ with standard deviation $s$ and sample size $n$, the confidence interval is:

$$\text{CI} = \bar{x} \pm t_{1-\alpha/2, n-1} \left( \frac{s}{\sqrt{n}} \right)$$

where $s/\sqrt{n}$ is the standard error of the mean. If the $95\%$ confidence interval for the accuracy difference between two models includes 0, we fail to reject the null hypothesis of no difference, which matches a two-tailed p-value check.

In machine learning, we often evaluate models using complex metrics like the Area Under the ROC Curve (AUC-ROC) or F1-Score, for which analytical formulas of standard error are complex or non-existent. We use bootstrapping to compute confidence intervals. We repeatedly sample the test dataset with replacement to create $B$ bootstrap samples (e.g., $B=1000$). We compute the metric on each sample, obtaining an empirical distribution. The $95\%$ confidence interval is then obtained by taking the $2.5$-th and $97.5$-th percentiles of this distribution, providing a robust estimate of the metric's variance without assuming a normal distribution.

To calculate the p-value of a metric difference without parametric assumptions, we use Permutation Testing. We pool the predictions from both models, randomly shuffle the model labels $B$ times, and compute the difference in performance for each shuffle. The empirical p-value is the proportion of shuffles where the shuffled difference exceeds the observed difference, providing an exact, distribution-free statistical check.