Vector Magnitude and Direction
Every vector represents a unique geometric identity defined by two fundamental properties: its magnitude (length or size) and its direction (orientation in space). In machine learning, the magnitude often represents the strength of a feature or the confidence of a classification, while the direction represents the core semantic meaning or characteristics of the data point.
Understanding Vector Magnitude: The L² Norm
The magnitude of a vector measures its overall length from the origin. The most common metric for this is the Euclidean Norm or L² Norm, which computes the straight-line distance in a multidimensional Cartesian coordinate system. It is heavily utilized in optimization algorithms, regression loss formulations, and distance-based similarity checks.
Mathematical Definition
For a vector $\mathbf{v} \in \mathbb{R}^n$, the L² norm is denoted as $\|\mathbf{v}\|_2$ and is calculated as the square root of the sum of squared component values:
$$\|\mathbf{v}\|_2 = \sqrt{\sum_{i=1}^n v_i^2} = \sqrt{v_1^2 + v_2^2 + \dots + v_n^2}$$
If we have a vector $\mathbf{v} = [3, 4]^T$, its L² norm is $\sqrt{3^2 + 4^2} = \sqrt{25} = 5$.
Application: Regularization (L² weight decay)
In training neural networks, we want to prevent weights from growing excessively large, which leads to overfitting. By penalizing the L² norm of the weight vector (called Weight Decay or Ridge Regularization), we encourage the network to keep weights small and distributed, smoothing out the decision boundary.
Vector Direction and Unit Vectors
While magnitude tells us the 'intensity' of a vector, its direction represents its core qualitative characteristics. In NLP or recommendation engines, we often care only about direction and want to strip away the magnitude completely. To do this, we convert the vector into a unit vector (or normalize it) so that its magnitude becomes exactly 1.
Vector Normalization
To normalize a non-zero vector $\mathbf{v}$, we divide each of its components by its L² magnitude. The resulting normalized vector $\mathbf{\hat{v}}$ points in the exact same direction but has a length of 1:
$$\mathbf{\hat{v}} = \frac{\mathbf{v}}{\|\mathbf{v}\|_2}$$
For example, normalising $\mathbf{v} = [3, 4]^T$ gives $\mathbf{\hat{v}} = [0.6, 0.8]^T$, whose magnitude is $\sqrt{0.6^2 + 0.8^2} = 1$.
Unit Vectors in Latent Spaces
When matching queries to documents using embeddings, variations in text length can artificially skew vector magnitudes. By normalizing all embedding vectors to unit vectors, we ensure that similarity searches focus purely on semantic alignment (direction) rather than document length (magnitude).
Other Vector Norms (L¹ and L-Infinity)
Different problem domains require different ways of measuring vector size. The L² norm measures straight-line distance, but other norms penalize components differently, leading to distinct behaviors in optimization algorithms.
The L¹ Norm (Manhattan Distance)
The L¹ norm is the sum of absolute values of the vector's components:
$$\|\mathbf{v}\|_1 = \sum_{i=1}^n |v_i|$$
In machine learning, L¹ regularization (Lasso) is widely used because it drives weights to exactly zero, producing a sparse model where only the most critical features are retained.
The Max Norm (L-Infinity Norm)
The L-infinity norm ($L^\infty$) is the maximum absolute value among all components:
$$\|\mathbf{v}\|_\infty = \max_i |v_i|$$
This norm is heavily used in Adversarial Robustness. When evaluating model resilience against attacks (like FSGM), we limit the maximum perturbation applied to any single pixel to a small $L^\infty$ bound to ensure the modification remains imperceptible to humans.