Sigmoid Function — Technology Wiki

Overview

Direct Answer

The sigmoid function is a mathematical activation function that transforms any input value into an output between 0 and 1 using the formula 1/(1+e^-x). It is particularly suited for binary classification tasks where outputs must represent probabilities.

How It Works

The function applies an exponential curve that produces smooth, differentiable outputs across its entire domain. As input values increase, the output asymptotically approaches 1; as they decrease, it approaches 0. This S-shaped curve enables neural networks to learn non-linear decision boundaries whilst maintaining gradient flow during backpropagation.

Why It Matters

Sigmoid enables binary classification outputs that directly correspond to probability estimates, critical for applications requiring calibrated confidence scores rather than arbitrary scaled values. Its mathematical properties support efficient training in shallow networks and remain standard in output layers for two-class prediction problems.

Common Applications

Common uses include medical diagnosis systems outputting disease probability, credit risk assessment producing default likelihood scores, and email spam detection yielding classification confidence. It remains the default activation for logistic regression implementations in enterprise analytics platforms.

Key Considerations

The function suffers from vanishing gradient problems in deep networks, making it less suitable for hidden layers in modern architectures. Its output range constraint can cause saturation, slowing convergence during training when gradients near 0 or 1.

Cross-References(1)

Deep Learning

Activation Function

Related in Training & Optimisation

Self-Attention

An attention mechanism where each element in a sequence attends to all other elements to compute its representation.

Multi-Head Attention

An attention mechanism that runs multiple attention operations in parallel, capturing different types of relationships.

Residual Network

A deep neural network architecture using skip connections that allow gradients to flow directly through layers, enabling very deep networks.

Layer Normalisation

A normalisation technique that normalises across the features of each individual sample rather than across the batch.

Dropout

A regularisation technique that randomly deactivates neurons during training to prevent co-adaptation and reduce overfitting.

Activation Function

A mathematical function applied to neural network outputs to introduce non-linearity, enabling the learning of complex patterns.

ReLU

Rectified Linear Unit — an activation function that outputs the input directly if positive, otherwise outputs zero.

Softmax Function

An activation function that converts a vector of numbers into a probability distribution, commonly used in multi-class classification.

Positional Encoding

A technique that injects information about the position of tokens in a sequence into transformer architectures.

Gradient Clipping

A technique that caps gradient values during training to prevent the exploding gradient problem.

Mixed Precision Training

Training neural networks using both 16-bit and 32-bit floating-point arithmetic to speed up computation while maintaining accuracy.

Rotary Positional Encoding

A position encoding method that encodes absolute position with a rotation matrix and naturally incorporates relative position information into attention computations.

More in Deep Learning

Encoder-Decoder Architecture

Architectures

A neural network design where an encoder processes input into a fixed representation and a decoder generates output from it.

Variational Autoencoder

Architectures

A generative model that learns a probabilistic latent space representation, enabling generation of new data samples.

Generative Adversarial Network

Generative Models

A framework where two neural networks compete — a generator creates synthetic data while a discriminator evaluates its authenticity.

Deep Learning

Architectures

A subset of machine learning using neural networks with multiple layers to learn hierarchical representations of data.

Pooling Layer

Architectures

A neural network layer that reduces spatial dimensions by aggregating values, commonly using max or average operations.

Pretraining

Architectures

Training a model on a large general dataset before fine-tuning it on a specific downstream task.

Weight Decay

Architectures

A regularisation technique that penalises large model weights during training by adding a fraction of the weight magnitude to the loss function, preventing overfitting.

Data Parallelism

Architectures

A distributed training strategy that replicates the model across multiple devices and divides training data into batches processed simultaneously, synchronising gradients after each step.