Overview
Direct Answer
Feature selection is the process of identifying and selecting a subset of input variables that are most predictive or relevant for a machine learning model, while eliminating redundant, irrelevant, or noisy attributes. This differs from dimensionality reduction in that it retains interpretable original variables rather than transforming them.
How It Works
Selection methods operate through three primary approaches: filter methods evaluate variable importance independently using statistical measures; wrapper methods assess subsets by training models iteratively; and embedded methods select features during model training itself, such as regularisation-based approaches. Each method ranks or scores variables based on their contribution to predictive performance or correlation with the target output.
Why It Matters
Reducing input dimensionality decreases computational cost, training time, and model complexity whilst often improving generalisation and interpretability. In regulated industries, fewer variables simplify compliance documentation and explainability requirements. Smaller feature sets also mitigate the curse of dimensionality and reduce storage requirements in resource-constrained deployments.
Common Applications
Healthcare applications use feature selection to identify clinically relevant biomarkers from high-dimensional genomic or imaging datasets. Financial institutions apply it to credit risk models where only the most predictive variables are retained for regulatory reporting. Text classification and natural language processing tasks benefit significantly by selecting informative words or embeddings from vocabularies of millions of potential features.
Key Considerations
The optimal feature subset is often task-specific and dataset-dependent; techniques that perform well on one problem may not transfer directly to another. Over-aggressive feature removal risks discarding subtle but collectively important signals, whilst retaining too many variables undermines the efficiency and interpretability benefits.
Cross-References(1)
Referenced By2 terms mention Feature Selection
Other entries in the wiki whose definition references Feature Selection — useful for understanding how this concept connects across Machine Learning and adjacent domains.
More in Machine Learning
Markov Decision Process
Reinforcement LearningA mathematical framework for modelling sequential decision-making where outcomes are partly random and partly controlled.
Cross-Validation
Training TechniquesA resampling technique that partitions data into subsets, training on some and validating on others to assess model generalisation.
Deep Reinforcement Learning
Reinforcement LearningCombining deep neural networks with reinforcement learning to enable agents to learn complex decision-making from raw sensory input.
Underfitting
Training TechniquesWhen a model is too simple to capture the underlying patterns in the data, resulting in poor performance on both training and test data.
Bias-Variance Tradeoff
Training TechniquesThe balance between a model's ability to minimise bias (error from assumptions) and variance (sensitivity to training data fluctuations).
Dimensionality Reduction
Unsupervised LearningTechniques that reduce the number of input variables in a dataset while preserving essential information and structure.
Semi-Supervised Learning
Advanced MethodsA learning approach that combines a small amount of labelled data with a large amount of unlabelled data during training.
Ridge Regression
Training TechniquesA regularised regression technique that adds an L2 penalty term to prevent overfitting by constraining coefficient magnitudes.