The Degradation Problem in Deep Networks

Deeper networks should theoretically perform better than shallow ones, but instead exhibit declining training and test performance.

Intuitively, deeper networks should perform better than shallower ones because they have higher representational capacity. If we take a shallow model and append identity layers to build a deeper counterpart, the deeper model should match or exceed the shallower model's performance.

However, experiments show that as network depth increases, training accuracy saturates and then degrades rapidly. This degradation is not caused by overfitting, as the training error increases alongside the validation error. This optimization failure indicates that deep architectures are difficult to optimize.

During backpropagation, gradients are multiplied recursively through the layers. If the weights are initialized poorly or the activation functions saturate, the gradients can shrink exponentially (vanishing gradients) or grow exponentially (exploding gradients) as they flow back to early layers.

While Batch Normalization and careful initialization help mitigate vanishing gradients, they do not resolve the degradation problem. The optimization difficulty persists even when gradients flow correctly, indicating that deep identity mappings are difficult for standard networks to learn.

The loss landscapes of deep architectures become highly non-convex, hindering standard gradient descent methods.

As networks grow deeper, the optimization loss surface becomes highly non-convex and rugged, with a large number of local minima, saddle points, and flat plateaus. Calculating gradients across these dimensions becomes chaotic.

Optimization algorithms struggle to navigate these complex surfaces, getting trapped in local minima or slowing down on plateaus. This complexity explains why deeper networks fail to converge to optimal configurations using standard optimization, resulting in higher training error.

In a standard network layer, learning an identity mapping—where the output matches the input \\(f(x) = x\\)—requires setting the weights and biases to precise values. This mapping is necessary for deep networks to propagate features without distortion.

Because neural layers are parameterized to learn complex transformations, optimization algorithms struggle to converge to identity mappings. Residual networks address this issue by reformulating the layers to learn residual mappings instead, resolving the degradation problem.

We can diagnose optimization issues in deep networks by monitoring gradient flows and parameter updates in PyTorch.

Monitoring the norm of the gradients during training is a key diagnostic step for identifying optimization bottlenecks. If the gradient norms of early layers approach zero while deeper layers have healthy gradients, the network is suffering from vanishing gradients.

Calling this function inside the training loop allows developers to verify that gradients are propagating correctly. If the norms are extremely small, adjusting initialization scales or adding skip connections may be necessary.

Visualizing the loss landscape of a model reveals the complexity of the optimization path. Shallow networks have smooth, convex-like loss surfaces that are easy for optimizers to navigate. Deep networks without skip connections have highly chaotic, sharp landscapes, which leads to high optimization error.

Adding residual connections flattens the loss landscape, turning a chaotic surface into a smooth, convex-like bowl. This change in the optimization terrain is why residual networks train faster and do not suffer from the degradation problem.