Overview
Direct Answer
Gradient boosting is an ensemble machine learning method that constructs models sequentially, with each successive model trained to predict and correct the residual errors left by the combined predictions of all previous models. The technique uses gradient descent optimisation to minimise a loss function across iterations.
How It Works
The algorithm initialises with a base learner, then iteratively fits new weak learners (typically decision trees) to the negative gradient of the loss function computed on training data. Each new model is weighted and added to the ensemble, with subsequent models focusing on instances or residuals where the existing ensemble performs poorly. Learning rates control the contribution of each addition, balancing model complexity against convergence speed.
Why It Matters
Gradient boosting achieves state-of-the-art predictive accuracy across classification and regression tasks, often outperforming alternative ensemble methods on tabular data. Organisations deploy it for high-stakes applications requiring robust generalisation, including credit risk assessment, fraud detection, and customer churn prediction, where incremental accuracy improvements directly translate to measurable business value.
Common Applications
Applications span financial services for loan default prediction, e-commerce for demand forecasting, healthcare for patient outcome modelling, and insurance for claims assessment. XGBoost, LightGBM, and CatBoost represent widely adopted open-source implementations used across industries for competition benchmarks and production systems.
Key Considerations
The sequential training process is computationally expensive and harder to parallelise than batch ensemble methods. Practitioners must carefully tune hyperparameters including learning rate, tree depth, and iteration count to avoid overfitting whilst maintaining interpretability.
Referenced By2 terms mention Gradient Boosting
Other entries in the wiki whose definition references Gradient Boosting — useful for understanding how this concept connects across Machine Learning and adjacent domains.
More in Machine Learning
UMAP
Unsupervised LearningUniform Manifold Approximation and Projection — a dimensionality reduction technique for visualisation and general non-linear reduction.
Automated Machine Learning
MLOps & ProductionThe end-to-end automation of the machine learning pipeline including feature engineering, model selection, hyperparameter tuning, and deployment, making ML accessible to non-experts.
Active Learning
MLOps & ProductionA machine learning approach where the algorithm interactively queries a user or oracle to label new data points.
Data Augmentation
Feature Engineering & SelectionTechniques that artificially increase the size and diversity of training data through transformations like rotation, flipping, and cropping.
Matrix Factorisation
Unsupervised LearningA technique that decomposes a matrix into constituent matrices, widely used in recommendation systems and dimensionality reduction.
Bagging
Advanced MethodsBootstrap Aggregating — an ensemble method that trains multiple models on random subsets of data and averages their predictions.
Content-Based Filtering
Unsupervised LearningA recommendation approach that suggests items similar to those a user has previously liked, based on item attributes.
Mini-Batch
Training TechniquesA subset of the training data used to compute a gradient update during stochastic gradient descent.