What is an Autoencoder?

An autoencoder is an unsupervised neural network trained to copy its input to its output. By constraining the network with a narrow bottleneck layer, the autoencoder is forced to learn a compressed, low-dimensional representation of the input data.

Reconstruction and the Bottleneck

Autoencoders compress inputs into a latent space through an encoder and reconstruct them using a decoder, relying on a narrow bottleneck to extract key features.

Encoder-Decoder Architecture

An autoencoder is divided into two sub-networks: the encoder and the decoder. The encoder is a function \\( \\mathbf{z} = f(\\mathbf{x}) \\) that maps the input vector \\( \\mathbf{x} \\in \\mathbb{R}^D \\) to a low-dimensional latent space vector \\( \\mathbf{z} \\in \\mathbb{R}^d \\) (where \\( d \\ll D \\)). The layer representing \\( \\mathbf{z} \\) is called the bottleneck. The decoder is a function \\( \\mathbf{\\hat{x}} = g(\\mathbf{z}) \\) that maps the latent representation back to the original input space, yielding the reconstruction \\( \\mathbf{\\hat{x}} \\in \\mathbb{R}^D \\).

The entire network is trained by minimizing a reconstruction loss \\( L(\\mathbf{x}, \\mathbf{\\hat{x}}) \\), which measures the difference between the input and the reconstructed output. If the network had unlimited capacity, it would learn the identity function \\( g(f(\\mathbf{x})) = \\mathbf{x} \\) by copying inputs directly to outputs. The bottleneck constraint prevents this trivial solution, forcing the network to extract the most important underlying factors of variation in the dataset.

The Undercomplete Autoencoder

When the dimension of the latent space \\( d \\) is strictly smaller than the input dimension \\( D \\), the autoencoder is called undercomplete. This constraint forces the model to capture the most salient features of the training data. If the data lies on a lower-dimensional manifold within the high-dimensional input space, the undercomplete autoencoder learns the coordinates of this manifold.

By limiting the information capacity of the bottleneck, the network acts as a lossy compression algorithm. The model discards noise and fine details while retaining the global structure. If \\( d \\) is set too large, the network can overfit and copy noise, whereas if \\( d \\) is set too small, the reconstructed outputs will be overly blurry and lose critical structures.

PyTorch Implementation and Training

A custom PyTorch autoencoder implements the linear mappings and reconstruction loss optimization for unsupervised training.

Linear Autoencoder

The following PyTorch code implements a basic fully connected autoencoder with an undercomplete bottleneck layer:

<pre><code class="language-python">import torch import torch.nn as nn class LinearAutoencoder(nn.Module): def __init__(self, input_dim, latent_dim): super().__init__() # Encoder projects high-dim to bottleneck self.encoder = nn.Sequential( nn.Linear(input_dim, 64), nn.ReLU(), nn.Linear(64, latent_dim) # Latent bottleneck ) # Decoder reconstructs bottleneck to high-dim self.decoder = nn.Sequential( nn.Linear(latent_dim, 64), nn.ReLU(), nn.Linear(64, input_dim) ) def forward(self, x): # x shape: [batch_size, input_dim] latent = self.encoder(x) # [batch_size, latent_dim] reconstructed = self.decoder(latent) # [batch_size, input_dim] return reconstructed, latent # Example instantiation model = LinearAutoencoder(input_dim=128, latent_dim=10) x = torch.randn(8, 128) # Batch size of 8 recon, latent = model(x) print("Latent representation shape:", latent.shape) # [8, 10] print("Reconstructed output shape:", recon.shape) # [8, 128]</pre>

Loss Formulation

The choice of reconstruction loss depends on the nature of the input data. For continuous real-valued inputs (such as raw audio signals or physical measurements), Mean Squared Error (MSE) loss is the standard selection: \\( L_{MSE} = \\frac{1}{N} \\sum ||\\mathbf{x}_i - \\mathbf{\\hat{x}}_i||^2 \\). MSE penalizes large errors heavily, encouraging the model to reconstruct global outlines accurately.

For inputs normalized to the range \\( [0, 1] \\) (such as pixel intensities in normalized images), Binary Cross-Entropy (BCE) loss can be used as an alternative: \\( L_{BCE} = -\\frac{1}{N} \\sum [x_{i,j} \\log \\hat{x}_{i,j} + (1 - x_{i,j}) \\log (1 - \\hat{x}_{i,j})] \\). BCE often leads to faster convergence for images because its gradients are stronger when predictions are far from target intensities.

Latent Representations and Dimension Reduction

Linear autoencoders map to the same subspace as PCA, while non-linear activations enable modeling of complex data manifolds.

Relationship to PCA

There is a direct mathematical connection between autoencoders and Principal Component Analysis (PCA). If a linear autoencoder (meaning all activation functions are identity mappings) is trained using Mean Squared Error loss on centered data, the latent representations learn the same subspace spanned by the first \\( d \\) principal components of the data.

However, the weight vectors learned by the autoencoder will not necessarily be orthogonal, unlike the principal axes in PCA. The encoder and decoder weights adjust to span the same subspace, but the representations can be a rotated and scaled version of the PCA coordinates. Thus, a linear autoencoder is equivalent to PCA in terms of reconstruction capacity, but differs in implementation details and orthogonality constraints.

Non-linear Manifold Learning

When non-linear activation functions (such as ReLU or Sigmoid) are added to the hidden layers, the autoencoder generalizes beyond linear PCA. The network can learn highly complex, non-linear manifolds. A manifold is a topological space that locally resembles Euclidean space but globally can be curved and folded (like a crumpled sheet of paper).

The non-linear encoder learns to "unroll" this manifold, projecting the curved input space into a flat coordinate system in the latent space. The decoder then learns the inverse mapping, wrapping the flat latent coordinates back to the complex data structure. This capability makes non-linear autoencoders far more powerful than PCA for complex datasets like natural images and speech signals.