The Kernel Trick in SVM (Linear, RBF, Polynomial)

The kernel trick allows SVMs to find non-linear decision boundaries by implicitly operating in a transformed high-dimensional feature space — without ever computing the transformation explicitly.

What is the Kernel Trick?

SVM only needs inner products \u27e8\u03c6(x\u1d62), \u03c6(x\u1d63)\u27e9 between feature vectors, not the vectors themselves. A kernel function K(x\u1d62, x\u1d63) computes this inner product in a high-dimensional space efficiently, without ever materialising that space.

Why This Matters

Consider mapping each data point to a space with millions of dimensions to find a linear separator there. Explicitly computing such vectors would be prohibitively expensive. The kernel trick reduces this to evaluating a scalar function K(x\u1d62, x\u1d63) between pairs of original data points.

Common Kernel Functions

scikit-learn's SVC supports several kernels, each suited to different data geometries.

Linear, RBF, and Polynomial Kernels

<pre><code class="language-python">from sklearn.svm import SVC from sklearn.preprocessing import StandardScaler from sklearn.pipeline import make_pipeline from sklearn.datasets import load_digits from sklearn.model_selection import cross_val_score X, y = load_digits(return_X_y=True) for kernel in ["linear", "rbf", "poly"]: clf = make_pipeline(StandardScaler(), SVC(kernel=kernel, C=1.0)) scores = cross_val_score(clf, X, y, cv=5, scoring="accuracy") print(f"{kernel:8s}: {scores.mean():.4f} +/- {scores.std():.4f}")</pre>

Choosing a Kernel

Linear kernel: fast, good for high-dimensional sparse data (e.g., text). RBF (Gaussian) kernel: the default for most problems; has one extra hyperparameter \u03b3 controlling locality. Polynomial kernel: useful for image data where feature interactions matter. When in doubt, try RBF first with a grid search over C and \u03b3.