Singular Value Decomposition (SVD) Basics
SVD is frequently called the 'Crown Jewel' of linear algebra. While Eigen-decomposition only works on square matrices, SVD can factorize any matrix into three distinct components. It is the ultimate tool for data compression, noise removal, and understanding hidden relationships.
The Three-Way Split
SVD breaks a matrix $A$ into three parts: $A = U\Sigma V^T$.
The Components
- $U$ (Left Singular Vectors): Corresponds to the 'columns' or categories of people/items.
- $\Sigma$ (Singular Values): A diagonal matrix representing the 'strength' or importance of each feature.
- $V^T$ (Right Singular Vectors): Corresponds to the 'rows' or individual traits.
Modern AI Applications: LoRA
SVD isn't just an old textbook theory—it's at the cutting edge of AI today.
Low-Rank Adaptation (LoRA)
When experts 'fine-tune' massive models like Llama-3, they don't update all billions of parameters. They use SVD to find a 'low-rank' approximation of the weights and only update that. This allows users to train powerful models on consumer-grade hardware by only updating ~1% of the total math.
Noise Reduction
By setting the smallest singular values in $\Sigma$ to zero, we can strip away the 'noise' from a dataset (like static in an audio file or blur in an image) and reconstruct only the 'signal'.