The AlexNet Breakthrough (2012)

AlexNet introduced key changes to network structures, including non-saturating activations and multi-GPU parallelization.

Prior to AlexNet, deep networks relied on saturating activations like Tanh or Sigmoid. AlexNet introduced the Rectified Linear Unit (ReLU), defined as \\(f(x) = \\max(0, x)\\), which does not saturate for positive inputs. This prevented the vanishing gradient problem, allowing the model to train significantly faster.

To handle the massive size of the ImageNet dataset, AlexNet was split across two NVIDIA GTX 580 GPUs. The GPUs communicated at specific layers, parallelizing the training of the 60 million parameters. This demonstrated the power of GPU-accelerated tensor arithmetic for training deep networks.

AlexNet used Local Response Normalization (LRN) layers after the first two convolutions. LRN is a normalization layer that models lateral inhibition, a biological phenomenon where active neurons suppress the activity of neighboring neurons. The formula is:

\\(b_{x,y}^i = \\frac{a_{x,y}^i}{\\left(k + \\alpha \\sum_{j=\\max(0, i-n/2)}^{\\min(N-1, i+n/2)} (a_{x,y}^j)^2\\right)^\\beta}\\)

While LRN was thought to improve generalization, subsequent architectures found that standardizing features using Batch Normalization was more effective. LRN is rarely used in modern CNNs, but it remains an important step in the history of normalization techniques.

With millions of parameters, AlexNet relied on dropout and extensive image augmentations to stabilize generalization.

With 60 million parameters, AlexNet was highly prone to overfitting. To prevent this, the designers introduced Dropout in the first two fully connected layers with a probability of 0.5. Dropout randomly zeroes out neurons during training, forcing the network to learn redundant representations.

This regularization technique significantly reduced overfitting, bridging the generalization gap between training and validation accuracy. It prevented co-adaptation of features, ensuring that neurons could generalize independently.

In addition to dropout, AlexNet relied heavily on data augmentation. During training, the images were randomly cropped from 256x256 to 224x224 and horizontally flipped. The researchers also used PCA-based color jittering (altering RGB intensities along the principal components of the ImageNet training set).

These augmentations expanded the dataset size artificially, making the model robust to shifts in scale, orientation, and color intensity, which was critical for achieving a low top-5 error rate on ImageNet.

Let's write a complete PyTorch implementation of AlexNet and review its kernel size parameters.

The code below shows how to build the AlexNet architecture in PyTorch, reflecting its stacked convolutional filters, MaxPool layers, and dropout-regularized dense classifier.

This implementation maps the inputs through the feature extractor and classifier pipelines. Setting inplace=True in ReLUs saves GPU memory by modifying tensors in-place, which was a critical technique for early hardware limits.

AlexNet uses large 11x11 and 5x5 kernels in its early layers to capture spatial context directly from high-resolution inputs.

While large kernels provide wide receptive fields, they are computationally expensive and contain a large number of parameters. Subsequent architectures (like VGG) replaced these large filters with stacks of smaller 3x3 kernels to improve parameters and representational depth.