ResNet (Residual Networks) and Skip Connections

Residual learning reformulates stacked layers to fit residual mappings instead of direct underlying representations.

Instead of forcing a stack of layers to fit a desired underlying mapping \\(H(x)\\), ResNet reformulates the layers to learn a residual mapping \\(F(x) := H(x) - x\\). The original mapping is cast into \\(F(x) + x\\), which is implemented by adding a skip connection (or identity shortcut) that feeds the input \\(x\\) directly to the output of the convolutional layers:

\\(H(x) = F(x, \\{W_i\\}) + x\\)

This formulation assumes that optimizing a residual mapping is much easier than optimizing the original, unreferenced mapping. If the optimal mapping is an identity, it is much easier for optimizer algorithms to push the residual weights \\(F(x)\\) to zero than to train a stack of non-linear layers to behave like an identity mapping from scratch. This simple formulation completely resolves the degradation problem, allowing networks to scale to hundreds or thousands of layers.

When the input \\(x\\) and the output \\(F(x)\\) have the same dimensions, they can be added element-wise directly. However, if the convolutional layers change the spatial dimensions (via stride) or the channel depth, a simple addition is impossible.

In these cases, a projection shortcut is used to adjust dimensions. This is typically implemented using a 1x1 convolution that projects the input \\(x\\) to match the target channels and spatial dimensions:

\\(H(x) = F(x, \\{W_i\\}) + W_s x\\)

where \\(W_s\\) represents the projection weights. This projection ensures that tensors match in size for the final addition operation.

Deeper ResNet architectures transition from basic double-conv blocks to triple-conv bottleneck designs to control parameter scaling.

Shallow ResNet models (like ResNet-18 and ResNet-34) use the "Basic Block," which consists of two stacked 3x3 convolutional layers. Deep ResNet models (like ResNet-50, ResNet-101, and ResNet-152) use the "Bottleneck Block" to control computational costs.

The bottleneck block uses three stacked convolutions: a 1x1 convolution to reduce channels, a 3x3 convolution, and a final 1x1 convolution to restore the original channel depth. This design reduces parameters and FLOPs, allowing the network to be much deeper without overloading memory.

The introduction of skip connections has a profound impact on the optimization process. By bypassing layers, skip connections create paths where gradients can flow directly from the final layers to the early layers during backpropagation, bypassing the vanishing gradient problem.

Additionally, mathematical analysis shows that skip connections flatten the loss landscape, turning a rugged, chaotic surface into a smooth, convex-like bowl. This makes training stable and allows convergence using standard optimizers like SGD, even at extreme depths.

Let's build a custom ResNet Basic Block and Bottleneck Block in PyTorch, commenting on shape dimensions.

The code below shows how to implement a basic Residual Block with a projection shortcut in PyTorch.

In this block, the shortcut path uses a 1x1 convolution when stride is 2, adjusting the channel size from 64 to 128 and matching spatial sizes for the element-wise addition step.

We can extend this modular design to implement a Bottleneck Block, which is crucial for building larger architectures like ResNet-50.

The bottleneck block uses 1x1 convolutions to project the input into a lower channel space before running the 3x3 convolution, saving computations. The final 1x1 layer restores the channels to 4x expansion size.