Denoising Autoencoders
Denoising Autoencoders (DAEs) are trained to reconstruct clean data from intentionally corrupted inputs. This process forces the network to learn the underlying structure of the data manifold to remove noise, preventing it from learning a trivial identity mapping.
Noise Injection Mechanics
Corrupting inputs before they enter the encoder forces the network to learn robust, noise-invariant features.
Robust Feature Extraction
In a standard autoencoder, there is a risk that the network will simply memorize the training data, especially if the bottleneck layer is not sufficiently narrow. A Denoising Autoencoder (DAE) addresses this by altering the training objective. Instead of training the model to copy its input \\( \\mathbf{x} \\), we corrupt the input to \\( \\mathbf{\\tilde{x}} \\) using a stochastic corruption process \\( q(\\mathbf{\\tilde{x}} \\mid \\mathbf{x}) \\).
The encoder processes the noisy input: \\( \\mathbf{z} = f(\\mathbf{\\tilde{x}}) \\), and the decoder attempts to reconstruct the original, clean input: \\( \\mathbf{\\hat{x}} = g(\\mathbf{z}) \\). Because the input is corrupted, the model cannot learn a simple identity mapping. To minimize the reconstruction loss \\( L(\\mathbf{x}, \\mathbf{\\hat{x}}) \\), the network must learn the relationships between features, allowing it to denoise the input and reconstruct missing information.
Noise Types
Several corruption processes can be used to train DAEs, depending on the data type. The most common is additive Gaussian noise, where we add random values drawn from a normal distribution: \\( \\mathbf{\\tilde{x}} = \\mathbf{x} + \\mathbf{\\epsilon} \\), where \\( \\mathbf{\\epsilon} \\sim \\mathcal{N}(0, \\sigma^2 \\mathbf{I}) \\). This noise type is useful for continuous inputs like audio and sensor data.
For binary or normalized inputs, masking noise is often used. In this setup, a fraction \\( \\alpha \\) of the input features are randomly set to 0. Another alternative is salt-and-pepper noise, where values are randomly set to their minimum (0) or maximum (1) values. Each noise type forces the model to leverage surrounding contextual information to reconstruct the corrupted dimensions.
PyTorch Denoising Autoencoder
Implementing a denoising autoencoder in PyTorch requires adding a noise injection step in the forward pass or training loop.
Model and Noise Pipeline
The following PyTorch code implements a DAE that adds Gaussian noise during training and reconstructs the clean input:
<pre><code class="language-python">import torch import torch.nn as nn class DenoisingAutoencoder(nn.Module): def __init__(self, input_dim, hidden_dim, noise_factor=0.3): super().__init__() self.noise_factor = noise_factor # Encoder self.encoder = nn.Sequential( nn.Linear(input_dim, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, hidden_dim // 2) ) # Decoder self.decoder = nn.Sequential( nn.Linear(hidden_dim // 2, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, input_dim) ) def forward(self, x): # Apply noise only during training if self.training: noise = torch.randn_like(x) * self.noise_factor x_corrupted = x + noise else: x_corrupted = x latent = self.encoder(x_corrupted) reconstructed = self.decoder(latent) return reconstructed, x_corrupted # Example usage model = DenoisingAutoencoder(input_dim=64, hidden_dim=32, noise_factor=0.25) x_clean = torch.rand(4, 64) # Clean input recon, corrupted = model(x_clean) # Loss is calculated against the clean input, NOT the corrupted input loss_fn = nn.MSELoss() loss = loss_fn(recon, x_clean) print("Reconstruction Loss:", loss.item())</pre>Loss Function
The loss function of a DAE is defined as the expectation of the reconstruction error over the training data and noise distribution: \\( L_{DAE} = \\mathbb{E}_{\mathbf{x} \\sim p_{data}, \\mathbf{\\tilde{x}} \\sim q(\mathbf{\\tilde{x}} \\mid \\mathbf{x})} [||\\mathbf{x} - g(f(\\mathbf{\\tilde{x}}))||^2] \\). It is critical that the target of the loss function is the original clean input \\( \\mathbf{x} \\), rather than the corrupted input \\( \\mathbf{\\tilde{x}} \\).
If the model were evaluated against the corrupted input, it would learn to output the noise itself, defeating the purpose of the denoising task. By comparing the output to the clean input, the optimizer adjusts the weights to remove the noise and project the inputs back toward the clean data distribution.
Manifold Learning and Regularization
Denoising autoencoders learn to project off-manifold points back onto the data manifold, serving as a regularized compression system.
Projection onto the Data Manifold
Denoising autoencoders have a clear geometric interpretation. The training data points lie on a low-dimensional manifold embedded in a high-dimensional space. Injecting noise moves the data points off this manifold. The DAE is trained to map these noisy, off-manifold points back onto the nearest point on the clean manifold.
This process defines a projection operator. The model learns a vector field that points from low-probability regions of the input space (where noisy points reside) toward high-probability regions (where the clean data manifold lies). This vector field approximates the score of the data distribution, establishing a link between DAEs and score-based generative models.
Overfitting Mitigation
In standard autoencoders, if the hidden layers have too much capacity, the model can overfit by learning an identity mapping that fails to compress the data. The noise injection in DAEs acts as a regularizer, preventing this behavior. Because the network must reconstruct clean values from corrupted inputs, it cannot simply pass inputs through the bottleneck unchanged.
This regularizing effect allows DAEs to be overcomplete (where the latent space is larger than the input space) while still learning useful representations. Overcomplete DAEs are useful for learning sparse, high-dimensional features that are linearly separable in downstream tasks.