Overview
Direct Answer
Weight initialisation is the process of assigning initial numerical values to the learnable parameters of a neural network prior to training. The choice of initialisation strategy directly influences convergence speed, final model performance, and the probability of reaching poor local minima.
How It Works
Different initialisation schemes assign parameter values according to statistical distributions tailored to network architecture. Common approaches include Xavier (Glorot) initialisation, which scales values based on the number of neurons in connected layers, and He initialisation, which adjusts variance for networks using ReLU activations. The goal is to maintain stable gradient flow throughout backpropagation by preventing activations from becoming excessively large or small.
Why It Matters
Poor initialisation can cause training to stall, diverge, or converge slowly, increasing computational cost and time-to-deployment. Appropriate initialisation reduces the risk of vanishing or exploding gradients, enabling faster convergence and better generalisation—critical factors in resource-constrained production environments.
Common Applications
Weight initialisation is applied across convolutional neural networks for image classification, recurrent networks for sequential data processing, and transformer models for natural language understanding. Medical imaging, autonomous systems, and recommendation engines all depend on effective initialisation to achieve reliable performance.
Key Considerations
Optimal initialisation strategies vary by activation function, network depth, and architecture type; no single approach is universally optimal. Transfer learning and pre-trained models circumvent initialisation challenges but introduce dependency on source domain similarity.
Cross-References(2)
More in Deep Learning
Diffusion Model
Generative ModelsA generative model that learns to reverse a gradual noising process, generating high-quality samples from random noise.
Pipeline Parallelism
ArchitecturesA form of model parallelism that splits neural network layers across devices and pipelines micro-batches through stages, maximising hardware utilisation during training.
Fine-Tuning
Language ModelsThe process of adapting a pre-trained model to a specific task by continuing training on a smaller task-specific dataset, transferring learned representations to new domains.
Exploding Gradient
ArchitecturesA problem where gradients grow exponentially during backpropagation, causing unstable weight updates and training failure.
Word Embedding
Language ModelsDense vector representations of words where semantically similar words are mapped to nearby points in vector space.
Mamba Architecture
ArchitecturesA selective state space model that achieves transformer-level performance with linear-time complexity by incorporating input-dependent selection mechanisms into the recurrence.
Attention Head
Training & OptimisationAn individual attention computation within a multi-head attention layer that learns to focus on different aspects of the input, with outputs concatenated for richer representations.
Residual Connection
Training & OptimisationA skip connection that adds a layer's input directly to its output, enabling gradient flow through deep networks and allowing training of architectures with hundreds of layers.