The Dot Product: Measuring Similarity

The dot product (also called the scalar product) takes two equal-length vectors and returns a single scalar number. It has two equivalent definitions—one algebraic (how computers calculate it) and one geometric (how we visualize it).

Algebraically, the dot product is the sum of the products of the corresponding components of two vectors $\mathbf{u}$ and $\mathbf{v}$: $$\mathbf{u} \cdot \mathbf{v} = \sum_{i=1}^n u_i v_i = u_1 v_1 + u_2 v_2 + \dots + u_n v_n$$ For example, if $\mathbf{u} = [1, 3]^T$ and $\mathbf{v} = [4, 2]^T$, the dot product is $(1 \times 4) + (3 \times 2) = 4 + 6 = 10$.

Geometrically, the dot product is the product of the magnitudes of the vectors and the cosine of the angle $\theta$ between them: $$\mathbf{u} \cdot \mathbf{v} = \|\mathbf{u}\|_2 \|\mathbf{v}\|_2 \cos(\theta)$$ This tells us that the dot product is equivalent to projecting one vector onto another and multiplying their lengths. If the vectors are perpendicular (orthogonal), $\cos(90^\circ) = 0$, resulting in a dot product of exactly 0.

The dot product represents how well aligned two vectors are in space. The sign and magnitude of the dot product provide an instant indicator of their relative orientation.

- Positive Dot Product ($\theta < 90^\circ$): The vectors point in generally similar directions. - Zero Dot Product ($\theta = 90^\circ$): The vectors are orthogonal, meaning they are completely independent and share no features in common. - Negative Dot Product ($\theta > 90^\circ$): The vectors point in opposing directions, representing inverse correlations.

Standard dot products are influenced by vector magnitudes. To measure pure directional alignment regardless of vector lengths, we use Cosine Similarity, which normalizes the dot product by the product of the magnitudes: $$\text{Cosine Similarity} = \cos(\theta) = \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\|_2 \|\mathbf{v}\|_2}$$ This guarantees a value between -1 and 1, representing perfect opposition and perfect alignment respectively. It is the primary metric for document retrieval and vector databases.

The dot product is the primary operation executed during both forward inference and backpropagation in deep neural networks.

In an artificial neural network, a single neuron computes its activation by taking the dot product of the input feature vector $\mathbf{x}$ and its learned weight vector $\mathbf{w}$, adding a bias scalar $b$, and passing it to an activation function $\sigma$: $$a = \sigma(\mathbf{w} \cdot \mathbf{x} + b)$$ The weight vector determines which features the neuron is looking for; the dot product measures how closely the input aligns with that ideal template.

Modern Large Language Models (LLMs) like GPT-4 rely on the self-attention mechanism, which determines how much focus a word should place on other words in a sequence. This is calculated using the query vector $\mathbf{q}$ and key vector $\mathbf{k}$: $$\text{Attention}(\mathbf{q}, \mathbf{k}) = \text{softmax}\left(\frac{\mathbf{q} \cdot \mathbf{k}}{\sqrt{d_k}}\right)$$ By taking the dot product $\mathbf{q} \cdot \mathbf{k}$, the model measures how closely the query aligns with each key, enabling context-aware word relationships.