Deep Learning for Video: Frame-by-Frame vs. 3D Convolutions
Deep learning for video classification balances temporal modeling accuracy with computational overhead, using either frame-by-frame 2D extractors or 3D spatio-temporal convolutions.
Frame-by-Frame (2D CNN + RNN)
Frame-by-frame processing extracts spatial details first before modeling temporal sequence flows.
Spatial Feature Extraction
In the frame-by-frame approach, a pre-trained 2D CNN (like ResNet) is used to extract spatial feature vectors from each video frame independently. The video is treated as a sequence of static images, and the temporal dimension is ignored during this stage.
This extraction produces a sequence of feature vectors of shape \\((T, D)\\), where \\(T\\) is the number of frames and \\(D\\) is the feature dimension, capturing spatial patterns at each time step.
Sequential Modeling
After extracting the sequence of spatial feature vectors, they are passed to a recurrent neural network (such as an LSTM or GRU) or a Transformer to model the temporal transitions.
This recurrent stage allows the model to capture temporal dynamics, such as recognizing that a person is sitting down rather than standing up. This approach is highly parameter-efficient because the heavy 2D CNN weights are frozen.
3D Spatio-Temporal Convolutions
3D Convolutions slide kernels across space and time concurrently to capture action details directly.
Simultaneous Feature Extraction
3D Convolutions (C3D, I3D) process the spatial and temporal dimensions concurrently using 3D kernels. A 3D kernel of size \\(K_T \\times K_H \\times K_W\\) slides across all three dimensions, capturing local motion and spatial layouts in a single step.
This simultaneous extraction allows the model to learn complex spatio-temporal features, such as tracking how the shape of a hand changes as it performs a gesture, preserving temporal context.
Computation and Optimization Challenges
While 3D CNNs achieve high accuracy, they are computationally expensive and require massive GPU memory. The number of parameters and FLOPs scales linearly with the temporal window size, making training slow.
To mitigate this, modern networks use factorized (2+1)D convolutions, which separate spatial and temporal operations to reduce complexity, speeding up convergence.
PyTorch Implementation Comparison
Let's implement a 2D CNN + LSTM video model and a 3D CNN model in PyTorch to compare architectures.
2D CNN + LSTM Classifier
The code below shows how to build a 2D CNN + LSTM video classifier in PyTorch, processing frames sequentially.
<pre><code class="language-python">import torch import torch.nn as nn from torchvision import models class ConvLSTMVideo(nn.Module): def __init__(self, num_classes): super().__init__() resnet = models.resnet18(pretrained=True) self.feature_extractor = nn.Sequential(*list(resnet.children())[:-1]) self.lstm = nn.LSTM(input_size=512, hidden_size=256, num_layers=1, batch_first=True) self.classifier = nn.Linear(256, num_classes) def forward(self, x): b, t, c, h, w = x.shape # Merge batch and temporal dimensions for spatial features x = x.view(b * t, c, h, w) features = self.feature_extractor(x) # [b * t, 512, 1, 1] features = features.view(b, t, 512) # Shape: [b, t, 512] lstm_out, _ = self.lstm(features) # Classify using final hidden state output logits = self.classifier(lstm_out[:, -1, :]) return logits x = torch.randn(2, 8, 3, 224, 224) # batch=2, frames=8, RGB images model = ConvLSTMVideo(num_classes=5) out = model(x) print("2D CNN + LSTM shape:", out.shape) # [2, 5]</pre>In this setup, we flatten the batch and temporal dimensions to run the spatial feature extractor in parallel. The resulting feature sequence is then passed to the LSTM to extract motion dynamics.
3D CNN Classifier
Below is the implementation of a 3D CNN classifier in PyTorch, processing spatial and temporal dimensions concurrently.
<pre><code class="language-python">import torch import torch.nn as nn class Conv3DVideo(nn.Module): def __init__(self, num_classes): super().__init__() self.features = nn.Sequential( # Input shape: [batch, channels, frames, height, width] nn.Conv3d(3, 16, kernel_size=3, padding=1), nn.ReLU(), nn.MaxPool3d(kernel_size=2), nn.AdaptiveAvgPool3d((1, 1, 1)) ) self.classifier = nn.Linear(16, num_classes) def forward(self, x): x = self.features(x) x = torch.flatten(x, 1) # [batch, 16] return self.classifier(x) x = torch.randn(2, 3, 8, 112, 112) # batch=2, channels=3, frames=8 model = Conv3DVideo(num_classes=5) out = model(x) print("3D CNN output shape:", out.shape) # [2, 5]</pre>The 3D CNN model processes video clips directly as a 5D tensor, sliding 3D filters across the temporal and spatial channels concurrently, capturing spatio-temporal features in a single forward pass.