Matrices: 2D Grids of Data
A matrix is a two-dimensional grid of numbers arranged in rows and columns. While a vector represents a single point or a list of features, a matrix represents an entire collection of vectors, or more profoundly, a linear transformation that can scale, rotate, and warp an entire coordinate space. In modern AI, matrices are both the containers of massive datasets and the operators that shape neural networks.
Structure, Dimensions, and Layout
A matrix is formally defined by its dimensions, shape, and indexing rules. Understanding these foundational mechanics is crucial for debugging tensor shape mismatches in machine learning code.
Matrix Dimensions
An $m \times n$ matrix (read 'm by n') has $m$ rows and $n$ columns. We write a matrix $\mathbf{A}$ as:
$$\mathbf{A} = \begin{bmatrix} a_{11} & a_{12} & \dots & a_{1n} \\ a_{21} & a_{22} & \dots & a_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{m1} & a_{m2} & \dots & a_{mn} \end{bmatrix}$$
where $a_{ij}$ represents the element located at row $i$ and column $j$.
Special Matrix Shapes
- Square Matrix: A matrix with an equal number of rows and columns ($m = n$).
- Diagonal Matrix: A square matrix where all entries off the main diagonal are zero. It acts as an efficient multi-dimensional scaling operator.
- Symmetric Matrix: A square matrix that is equal to its transpose ($a_{ij} = a_{ji}$), widely seen in distance matrices and correlation representations.
Matrices as Spatial Transformations
Beyond being simple static grids of numbers, a matrix is a dynamic operator. When you multiply a vector $\mathbf{x}$ by a matrix $\mathbf{A}$, you are transforming $\mathbf{x}$ into a new vector $\mathbf{y}$ in a different coordinate space.
Linear Transformations
The equation $\mathbf{y} = \mathbf{A}\mathbf{x}$ represents a linear transformation. The columns of matrix $\mathbf{A}$ define where the original standard basis vectors (like $[1, 0]^T$ and $[0, 1]^T$ in 2D) land after the transformation. By scaling, rotating, or shearing these basis vectors, the matrix transforms the entire coordinate grid.
Deep Learning as Manifold Untangling
A neural network is essentially a stack of these matrix transformations. As data passes through the layers, the matrices continuously rotate, stretch, and warp the high-dimensional space. The goal is to untangle complex, overlapping datasets so that a simple linear decision boundary can separate them at the final layer.
Matrices in Machine Learning Workflows
Matrices are the native language of GPU hardware. By representing data and parameters as matrices, we can perform massive calculations in parallel.
The Design Matrix (X)
In machine learning, we stack multiple data samples into a single matrix called the Design Matrix $\mathbf{X}$. Each row represents a single data instance (e.g., a customer profile), and each column represents a specific feature (e.g., age, income). Stacking data in this format allows us to process an entire batch of inputs with a single matrix equation.
Layer Weight Matrices (W)
A neural network layer consists of many neurons. Instead of calculating each neuron's activation vector-by-vector, we stack all their individual weight vectors into a single Weight Matrix $\mathbf{W}$. The output activation vector is computed in a single unified operation:
$$\mathbf{a} = \sigma(\mathbf{W}\mathbf{x} + \mathbf{b})$$
This algebraic layout is perfect for GPU processing units, which multiply massive grids of numbers in hardware in parallel.