Population vs. Sample Data

At the heart of statistical inference lies the distinction between a population and a sample. A population is the complete set of all elements under study, whose size $N$ can be finite or infinite. Because measuring the entire population is typically impossible, we draw a sample of size $n$ ($n \ll N$). Any value computed from the population is a parameter, which is a fixed constant. Any value computed from a sample is a statistic, which is a random variable that varies from sample to sample.

Understanding this distinction is the cornerstone of statistical estimation. When we estimate a parameter using a sample statistic, we must quantify the uncertainty introduced by the sampling process. This uncertainty is characterized by the sampling distribution of the statistic, which describes how the statistic behaves over infinite repeated sampling trials.

For a finite population of size $N$, the population mean $\mu$ and population variance $\sigma^2$ are computed using the formulas:

$$\mu = \frac{1}{N} \sum_{i=1}^{N} x_i$$

$$\sigma^2 = \frac{1}{N} \sum_{i=1}^{N} (x_i - \mu)^2$$

When the population is infinite, we model the population using a probability density function $f(x)$ or probability mass function $P(x)$. In this case, the parameters are defined using mathematical expectations:

$$\mu = E[X] = \int_{-\infty}^{\infty} x f(x) \, dx$$

$$\sigma^2 = E[(X - \mu)^2] = \int_{-\infty}^{\infty} (x - \mu)^2 f(x) \, dx$$

These parameters represent the true, often unknowable characteristics of the entire population distribution, which we aim to approximate using sample data. For example, if we model a coin flip with a Bernoulli distribution where $P(X=1) = p$, the population parameter is $p$, representing the true probability of heads, and the expected value is $E[X] = 1 \cdot p + 0 \cdot (1-p) = p$. The population variance is $Var(X) = E[X^2] - (E[X])^2 = p - p^2 = p(1-p)$.

Since we only observe a sample of size $n$, we estimate the parameters using sample statistics, such as the sample mean:

$$\bar{x} = \frac{1}{n} \sum_{i=1}^{n} x_i$$

Because the sample is drawn randomly, the statistic $\bar{x}$ is itself a random variable. If we were to draw multiple independent samples of size $n$ from the same population, the values of $\bar{x}$ would vary. The probability distribution of these sample means is known as the sampling distribution. Let us derive the variance of the sampling distribution of the mean, assuming $x_i$ are drawn i.i.d. with mean $\mu$ and variance $\sigma^2$:

$$\text{Var}(\bar{x}) = \text{Var}\left( \frac{1}{n} \sum_{i=1}^{n} x_i \right) = \frac{1}{n^2} \text{Var}\left( \sum_{i=1}^{n} x_i \right)$$

Because the $x_i$ are independent, the variance of the sum is the sum of the variances:

$$\text{Var}(\bar{x}) = \frac{1}{n^2} \sum_{i=1}^{n} \text{Var}(x_i) = \frac{1}{n^2} \sum_{i=1}^{n} \sigma^2 = \frac{1}{n^2} (n \sigma^2) = \frac{\sigma^2}{n}$$

According to the Central Limit Theorem (CLT), as $n$ becomes large ($n \ge 30$), the sampling distribution of the sample mean approaches a normal distribution with mean $\mu$ and variance $\sigma^2/n$, regardless of the population's underlying distribution shape:

$$\sqrt{n}(\bar{x}_n - \mu) \xrightarrow{d} \mathcal{N}(0, \sigma^2)$$

This mathematical property allows us to quantify the uncertainty of our estimates, construct confidence intervals, and perform hypothesis testing. The standard deviation of the sampling distribution, known as the standard error of the mean, is $\text{SE} = \sigma/\sqrt{n}$, showing that our estimation error shrinks as the sample size increases.

To draw valid inferences, our sample must accurately represent the population. An estimator is a mathematical function of the sample data used to approximate a population parameter. We evaluate the quality of an estimator using properties like bias and consistency.

A representative sample is one where every subset of the population has a known, non-zero probability of being selected. When this condition is violated, the sample is unrepresentative, and estimators computed from it will be biased, leading to systematic errors that cannot be corrected by simply increasing the sample size.

Let $\hat{\theta}$ be an estimator for a population parameter $\theta$. The bias of $\hat{\theta}$ is defined as the difference between its expected value and the true parameter value:

$$\text{Bias}(\hat{\theta}) = E[\hat{\theta}] - \theta$$

An estimator is said to be unbiased if $\text{Bias}(\hat{\theta}) = 0$, meaning that on average, the estimator equals the true parameter. Additionally, an estimator is consistent if it converges in probability to the true parameter as the sample size $n$ approaches infinity:

$$\lim_{n \to \infty} P(|\hat{\theta}_n - \theta| > \epsilon) = 0$$

for any $\epsilon > 0$. Unbiasedness ensures that our predictions are not systematically off-center, while consistency guarantees that our estimations become arbitrarily precise as we collect more data. For example, the sample mean $\bar{x}$ is both an unbiased and consistent estimator of the population mean $\mu$ because $E[\bar{x}] = \mu$ and $\text{Var}(\bar{x}) = \sigma^2/n \to 0$ as $n \to \infty$. By Chebyshev's Inequality, we can prove consistency because:

$$P(|\bar{x}_n - \mu| > \epsilon) \le \frac{\text{Var}(\bar{x}_n)}{\epsilon^2} = \frac{\sigma^2}{n\epsilon^2}$$

As $n \to \infty$, the term $\frac{\sigma^2}{n\epsilon^2} \to 0$, which proves that $\bar{x}_n$ converges in probability to $\mu$.

In machine learning, training datasets are samples. If the sampling process is flawed, the model inherits sampling bias, leading to poor generalization. Common types include:

1. Selection Bias: Occurs when certain members of the population are systematically more likely to be selected in the sample than others. For instance, training a self-driving car only on highway data creates selection bias, making it fail in city streets.

2. Survivorship Bias: Occurs when the sample includes only 'surviving' or successful observations while ignoring failures. For example, building a financial forecasting model using only companies that are currently active in the stock market leads to overestimating future returns.

3. Reporting Bias: Occurs when events that are rare or dramatic are more frequently documented than common, ordinary events, distorting the model's perception of real-world frequencies. This causes machine learning systems to over-predict rare anomalies.

4. Class Imbalance: A common machine learning sampling anomaly where one class is heavily underrepresented in the sample compared to the true distribution. If a medical model is trained on a sample containing only $0.1\%$ positive cancer cases because positive cases are rare in clinical logs, the model will struggle to learn the features of the rare class unless we apply resampling or synthetic data techniques (like SMOTE).

In machine learning, we do not have access to the entire population distribution $P(X, Y)$. Instead, we only have a training sample of size $n$. Statistical learning theory provides the mathematical framework for understanding how well a model trained on a sample will generalize to the population.

The central question is how to bound the difference between the model's error on the observed sample and its expected error on the unobserved population. By analyzing the complexity of the model's hypothesis space, we can establish probabilistic guarantees on its generalization performance.

Let $D = \{(x_1, y_1), \dots, (x_n, y_n)\}$ be a training sample drawn independently and identically distributed (i.e. i.i.d.) from the true population distribution $P(X, Y)$. The goal of learning is to find a function $f$ that minimizes the true risk (expected loss over the population):

$$R(f) = E_{(X,Y) \sim P}[L(f(X), Y)] = \iint L(f(x), y) P(x, y) \, dx \, dy$$

Because we cannot calculate the true risk directly, we instead minimize the empirical risk (the average loss over the training sample):

$$\hat{R}(f) = \frac{1}{n} \sum_{i=1}^{n} L(f(x_i), y_i)$$

This strategy is known as Empirical Risk Minimization. The discrepancy between empirical risk and true risk ($R(f) - \hat{R}(f)$) is the generalization error. Overfitting occurs when the model minimizes empirical risk by learning the sample-specific noise, causing the true risk to spike.

To guarantee that a model will generalize, statistical learning theory establishes mathematical bounds on the generalization error. For a fixed hypothesis $f$ mapping to bounded loss values $L \in [0, 1]$, Hoeffding's Inequality states that:

$$P(|\hat{R}(f) - R(f)| \ge \epsilon) \le 2e^{-2n\epsilon^2}$$

By setting the right side to a confidence parameter $\delta = 2e^{-2n\epsilon^2}$, we solve for $\epsilon$ to get $\epsilon = \sqrt{\frac{\ln(2/\delta)}{2n}}$. This proves that with probability at least $1 - \delta$:

$$R(f) \le \hat{R}(f) + \sqrt{\frac{\ln(2/\delta)}{2n}}$$

This bound shows that as the sample size $n$ increases, the gap between the sample error and population error shrinks at a rate of $\mathcal{O}(1/\sqrt{n})$. For an infinite hypothesis space $\mathcal{H}$ (like neural networks or support vector machines), a simple union bound fails. We replace this with bounds based on complexity measures such as the Vapnik-Chervonenkis (VC) Dimension or Rademacher Complexity. The VC dimension measures the capacity of a classification model by checking the maximum number of points it can shatter (classify in all possible ways). For a hypothesis class with VC dimension $d$, the generalization bound with probability $1-\delta$ is:

$$R(f) \le \hat{R}(f) + \sqrt{\frac{8d \ln(2en/d) + 8\ln(4/\delta)}{n}}$$

This demonstrates that generalization depends on the balance between model capacity ($d$) and sample size ($n$).

A fundamental assumption of machine learning is that the training sample and the deployment population are drawn from the same probability distribution. When this assumption is violated, models suffer from distribution shifts, leading to performance degradation.

These shifts are common in production systems where user behavior, seasonal trends, or data-collection pipelines change. Diagnosing and correcting distribution shifts is one of the most critical challenges in deploying robust AI systems.

We analyze distribution shift by decomposing the joint probability distribution using Bayes' theorem: $P(X, Y) = P(Y|X)P(X) = P(X|Y)P(Y)$. This leads to distinct types of shift:

1. Covariate shift occurs when the input distribution changes between the training sample and the deployment population, but the conditional probability of the label remains constant:

$$P_{\text{train}}(X) \neq P_{\text{deploy}}(X) \quad \text{while} \quad P(Y|X) \text{ remains unchanged}$$

To correct for covariate shift, we can use importance weighting during training. The loss function is modified by multiplying each training instance's loss by an importance weight $w(x) = P_{\text{deploy}}(x)/P_{\text{train}}(x)$. Let us prove that the weighted empirical risk under the training distribution is an unbiased estimator of the true risk under the deployment distribution:

$$E_{X \sim P_{\text{train}}}[w(X) L(f(X), Y)] = \int w(x) L(f(x), y) P_{\text{train}}(x) \, dx$$

$$= \int \frac{P_{\text{deploy}}(x)}{P_{\text{train}}(x)} L(f(x), y) P_{\text{train}}(x) \, dx = \int L(f(x), y) P_{\text{deploy}}(x) \, dx = E_{X \sim P_{\text{deploy}}}[L(f(X), Y)]$$

2. Target shift (or prior probability shift) occurs when the label distribution changes, but the conditional feature distribution remains constant:

$$P_{\text{train}}(Y) \neq P_{\text{deploy}}(Y) \quad \text{while} \quad P(X|Y) \text{ remains unchanged}$$

This is common in medical diagnostics when a disease becomes more or less prevalent over time. We can estimate the updated prior $P_{\text{deploy}}(Y)$ using confusion matrix correction methods and apply importance weights $w(y) = P_{\text{deploy}}(y)/P_{\text{train}}(y)$ to adjust our predictions.

Concept drift occurs when the statistical properties of the target variable change over time, meaning the true mapping itself changes:

$$P(Y|X) \text{ changes over time}$$

Unlike covariate shift, the relationship between features and labels is non-stationary. An example is fraud detection, where fraud patterns evolve over time to mimic normal transactions. To handle concept drift, AI systems must implement continuous monitoring. A common metric used to detect distribution shifts in features is the Population Stability Index (PSI):

$$\text{PSI} = \sum_{b=1}^{B} \left( P_{\text{actual}, b} - P_{\text{expected}, b} \right) \times \ln\left( \frac{P_{\text{actual}, b}}{P_{\text{expected}, b}} \right)$$

where $b$ represents bins of the feature values, $P_{\text{actual}}$ is the sample distribution observed in production, and $P_{\text{expected}}$ is the reference training sample distribution. A PSI value greater than 0.25 indicates a significant distribution shift, requiring immediate model retraining, domain adaptation, or online gradient updates.