Feature Scaling: Z-Score Standardization

Z-Score Standardization transforms features to have zero mean and unit variance — the most widely used scaling method in ML. Unlike Min-Max, it does not bound values to a specific range, making it more robust to moderate outliers.

The Standardization Formula

For each value x: z = (x − \\mu) / \\sigma, where \\mu is the training column mean and \\sigma is its standard deviation. The result has mean 0 and standard deviation 1, though there is no fixed range.

Applying StandardScaler

<pre><code class="language-python">from sklearn.preprocessing import StandardScaler scaler = StandardScaler() X_train_scaled = scaler.fit_transform(X_train) X_test_scaled = scaler.transform(X_test) print(scaler.mean_) # training set column means print(scaler.scale_) # training set column std devs</pre>

Which Algorithms Require Scaling

Not all algorithms are sensitive to feature scale. Understanding which ones need standardization helps you decide when scaling is worth adding to a pipeline.

Scale-Sensitive vs Scale-Invariant Models

  • Needs scaling: Linear/Logistic Regression, SVMs, KNN, PCA, Neural Networks, K-Means
  • Scale-invariant: Decision Trees, Random Forests, Gradient Boosting (XGBoost, LightGBM), Naive Bayes

Always standardize when using regularized linear models (Ridge, Lasso), as the regularization penalty treats all coefficients equally and assumes equal-scale features.