Matrix Addition and Scalar Multiplication

The simplest way to manipulate a matrix is through addition and scalar scaling. These operations are 'entry-wise', meaning they act on every individual cell independently.

Matrix Addition

If two matrices have the same dimension, they can be added. This is used in AI to combine 'layers' of information or update parameters.

The Rule of Alignment

Matrix $A$ + Matrix $B$ is only possible if both are $m \times n$. It creates matrix $C$, where $c_{ij} = a_{ij} + b_{ij}$.

Scalar Multiplication

Multiplying a matrix by a single number (scalar) scales every element in the grid. This is how we 'dampen' or 'amplify' a signal in a neural network.

Learning Rate Application

When we multiply an update matrix by the Learning Rate, we are performing scalar multiplication to control the speed of the model's growth.