Sigmoid Function — Technology Wiki

Overview

Direct Answer

The sigmoid function is a mathematical activation function that transforms any input value into an output between 0 and 1 using the formula 1/(1+e^-x). It is particularly suited for binary classification tasks where outputs must represent probabilities.

How It Works

The function applies an exponential curve that produces smooth, differentiable outputs across its entire domain. As input values increase, the output asymptotically approaches 1; as they decrease, it approaches 0. This S-shaped curve enables neural networks to learn non-linear decision boundaries whilst maintaining gradient flow during backpropagation.

Why It Matters

Sigmoid enables binary classification outputs that directly correspond to probability estimates, critical for applications requiring calibrated confidence scores rather than arbitrary scaled values. Its mathematical properties support efficient training in shallow networks and remain standard in output layers for two-class prediction problems.

Common Applications

Common uses include medical diagnosis systems outputting disease probability, credit risk assessment producing default likelihood scores, and email spam detection yielding classification confidence. It remains the default activation for logistic regression implementations in enterprise analytics platforms.

Key Considerations

The function suffers from vanishing gradient problems in deep networks, making it less suitable for hidden layers in modern architectures. Its output range constraint can cause saturation, slowing convergence during training when gradients near 0 or 1.

Cross-References(1)

Deep Learning

Activation Function

Related in Training & Optimisation

Self-Attention

An attention mechanism where each element in a sequence attends to all other elements to compute its representation.

Multi-Head Attention

An attention mechanism that runs multiple attention operations in parallel, capturing different types of relationships.

Residual Network

A deep neural network architecture using skip connections that allow gradients to flow directly through layers, enabling very deep networks.

Layer Normalisation

A normalisation technique that normalises across the features of each individual sample rather than across the batch.

Dropout

A regularisation technique that randomly deactivates neurons during training to prevent co-adaptation and reduce overfitting.

Activation Function

A mathematical function applied to neural network outputs to introduce non-linearity, enabling the learning of complex patterns.

ReLU

Rectified Linear Unit — an activation function that outputs the input directly if positive, otherwise outputs zero.

Softmax Function

An activation function that converts a vector of numbers into a probability distribution, commonly used in multi-class classification.

Positional Encoding

A technique that injects information about the position of tokens in a sequence into transformer architectures.

Gradient Clipping

A technique that caps gradient values during training to prevent the exploding gradient problem.

Mixed Precision Training

Training neural networks using both 16-bit and 32-bit floating-point arithmetic to speed up computation while maintaining accuracy.

Rotary Positional Encoding

A position encoding method that encodes absolute position with a rotation matrix and naturally incorporates relative position information into attention computations.

More in Deep Learning

Word Embedding

Language Models

Dense vector representations of words where semantically similar words are mapped to nearby points in vector space.

Generative Adversarial Network

Generative Models

A framework where two neural networks compete — a generator creates synthetic data while a discriminator evaluates its authenticity.

Fine-Tuning

Architectures

The process of taking a pretrained model and further training it on a smaller, task-specific dataset.

Knowledge Distillation

Architectures

A model compression technique where a smaller student model learns to mimic the behaviour of a larger teacher model.

Vanishing Gradient

Architectures

A problem in deep networks where gradients become extremely small during backpropagation, preventing earlier layers from learning.

Autoencoder

Architectures

A neural network trained to encode input data into a compressed representation and then decode it back to reconstruct the original.

Key-Value Cache

Architectures

An optimisation in autoregressive transformer inference that stores previously computed key and value tensors to avoid redundant computation during sequential token generation.

State Space Model

Architectures

A sequence modelling architecture based on continuous-time dynamical systems that processes long sequences with linear complexity, offering an alternative to attention-based transformers.