Vector Addition and Subtraction

Vector addition combines two or more vectors into a single resultant vector. Algebraically, this is done by adding corresponding components. Geometrically, it represents joining paths end-to-end to find a final destination.

To add two vectors $\mathbf{u}$ and $\mathbf{v}$ in $\mathbb{R}^n$, they must have the exact same dimensions. The addition is performed element-by-element: $$\mathbf{u} + \mathbf{v} = \begin{bmatrix} u_1 \\ u_2 \\ \vdots \\ u_n \end{bmatrix} + \begin{bmatrix} v_1 \\ v_2 \\ \vdots \\ v_n \end{bmatrix} = \begin{bmatrix} u_1 + v_1 \\ u_2 + v_2 \\ \vdots \\ u_n + v_n \end{bmatrix}$$ This simple operation scales linearly with the number of dimensions, making it extremely fast to run on GPUs.

Geometrically, to add vector $\mathbf{v}$ to vector $\mathbf{u}$, you draw $\mathbf{u}$ starting from the origin, and then draw $\mathbf{v}$ starting from the tip of $\mathbf{u}$. The resultant vector $\mathbf{u} + \mathbf{v}$ is the straight arrow pointing from the origin directly to the tip of $\mathbf{v}$. This forms the diagonal of a parallelogram if both start at the origin.

Vector subtraction is the addition of a negated vector. Algebraically, it computes the differences between corresponding features. Geometrically, subtraction determines the vector that points from the tip of the second vector to the tip of the first, representing a spatial offset or correction.

Subtracting $\mathbf{v}$ from $\mathbf{u}$ is defined component-wise: $$\mathbf{u} - \mathbf{v} = \begin{bmatrix} u_1 - v_1 \\ u_2 - v_2 \\ \vdots \\ u_n - v_n \end{bmatrix}$$ If $\mathbf{u}$ represents a target location and $\mathbf{v}$ represents a current estimated location, then $\mathbf{u} - \mathbf{v}$ is the displacement vector needed to bridge the gap.

In machine learning, vector subtraction is the foundation of error calculation. If $\mathbf{y}$ is a vector of ground-truth labels and $\mathbf{\hat{y}}$ is a vector of predictions, the error vector $\mathbf{e}$ is calculated as: $$\mathbf{e} = \mathbf{y} - \mathbf{\hat{y}}$$ This error vector represents both the magnitude and the precise direction of the mistake, guiding optimizer updates during backpropagation.

One of the most striking discoveries in modern AI is that vectors trained in neural representation spaces exhibit algebraic properties matching real-world concepts. This is called Semantic Vector Arithmetic.

In Word2Vec models, words are embedded in high-dimensional vector spaces. These spaces capture relational concepts. For example, subtracting the vector for 'Man' from 'King' isolates the abstract vector of 'Royalty'. Adding this to 'Woman' lands remarkably close to the vector for 'Queen': $$\mathbf{v}_{\text{King}} - \mathbf{v}_{\text{Man}} + \mathbf{v}_{\text{Woman}} \approx \mathbf{v}_{\text{Queen}}$$ This proves that vector arithmetic can manipulate real-world meanings.

Generative AI models use vector arithmetic to edit images and audio. In a face generation GAN, we can isolate a 'smiling' vector by subtracting the average neutral face vector from the average smiling face vector. Adding this isolated vector to a new face vector forces the model to render a smiling version of that person.