Feature Extraction vs. Fine-tuning Guidelines

The optimal transfer strategy is determined by dataset size and similarity between source and target visual spaces.

The size of the target dataset is the primary constraint. If the target dataset is extremely small (e.g., fewer than 1000 images), fine-tuning deep layers will lead to severe overfitting. In this scenario, feature extraction is the only viable option.

If the target dataset is large (e.g., over 100,000 images), the model can support fine-tuning of all layers without risk of overfitting. The abundance of data allows the pre-trained weights to be adjusted to the target domain safely, improving final classification accuracy.

The similarity between the pre-trained dataset (e.g., ImageNet) and the target dataset (e.g., satellite imagery) determines the relevance of the features.

If the domains are highly similar (e.g., classifying dog breeds), the pre-trained filters are highly relevant, and feature extraction is sufficient. If the domains differ significantly (e.g., classifying medical ultrasound scans), the low-level edge features are still useful, but the high-level semantic shapes differ completely, requiring deep fine-tuning.

Understanding scenario-based guidelines enables developers to select training options that balance computation and accuracy.

The decision matrix can be divided into four scenarios. Scenario 1: Small dataset, similar domain. Use feature extraction; train only the classifier head to avoid overfitting. Scenario 2: Large dataset, similar domain. Use fine-tuning with a low learning rate; since data is abundant, this will improve performance. Scenario 3: Small dataset, different domain. This is the hardest case. Freezing the feature extractor is sub-optimal because the features are not relevant, but fine-tuning leads to overfitting. A common compromise is to freeze early layers and fine-tune only the final few blocks. Scenario 4: Large dataset, different domain. Fine-tune the entire network from pre-trained weights; the pre-trained initialization speeds up convergence compared to random weights.

Using this structured approach ensures that resources are allocated efficiently, matching target data traits.

Feature extraction is computationally cheaper because gradients are only computed for the classification head, reducing memory usage and training time.

Fine-tuning requires computing gradients for all parameters and updating optimizer states, which increases memory consumption and slows down execution. In cloud deployments, choosing feature extraction reduces training costs significantly.

We can verify parameter freeze states programmatically to ensure our training setup aligns with transfer guidelines.

The code below shows how to log the trainable parameter status of a model, helping verify that the chosen transfer learning strategy is implemented correctly.

In this diagnostics run, only the weights in the replaced linear classifier head require gradients, demonstrating a feature extraction setup where 99.95% of weights are locked.

Below is the setup showing how to filter out frozen parameters when passing them to the optimizer.

Filtering parameters prevents the optimizer from creating momentum buffers for frozen weights, saving memory during backpropagation.