Partial Derivatives: Handling Multiple Variables
In basic single-variable calculus, we study functions where one output depends on a single input. However, in machine learning, we deal with multi-variable functions where the output (loss) depends on millions, or even billions, of inputs (weights and biases). Partial derivatives allow us to dissect these high-dimensional systems by measuring how the function changes with respect to one specific variable while holding all other variables constant.
The Mechanics of Partial Differentiation
Computing a partial derivative is structurally identical to single-variable differentiation. The key difference is that we treat all variables other than the one we are differentiating with respect to as static constants.
Notation and Calculation
We use the 'curly d' symbol ($\partial$) to denote partial derivatives. For a function $f(x, y)$, the partial derivative with respect to $x$ is written as $\frac{\partial f}{\partial x}$. If $f(x, y) = x^2 + xy + y^2$, then $\frac{\partial f}{\partial x} = 2x + y$, because $y^2$ is treated as a constant (derivative is 0) and $y$ in $xy$ acts as a constant coefficient.
Geometric Interpretation
Imagine a three-dimensional mountain landscape where height is the output and horizontal coordinates are $x$ and $y$. The partial derivative $\frac{\partial f}{\partial x}$ represents the slope of the path if you walk strictly in the East-West direction, while $\frac{\partial f}{\partial y}$ represents the slope if you walk strictly North-South.
Partial Derivatives in Machine Learning
Neural networks are composed of individual weights that must be adjusted independently. Partial derivatives provide the mathematical tool to isolate the effect of each individual weight on the overall error.
Isolating Parameters
If our loss function is $L(w_1, w_2, \dots, w_n)$, we calculate the partial derivative $\frac{\partial L}{\partial w_i}$ for each weight $w_i$. This tells us whether increasing that specific weight will increase or decrease the overall error, independent of all other weights in the network.
High-Dimensional Tuning
Because backpropagation computes these partial derivatives for every single parameter, we can fine-tune massive neural networks by making precise, targeted adjustments to each connection, rather than relying on global adjustments.