ML Interview Q Series: What Feature Selection methods do you know?
📚 Browse the full ML Interview series here.
Comprehensive Explanation
Feature selection is the process of identifying and selecting a subset of relevant features for use in model construction. This reduces the dimensionality of the input space, speeds up training, improves model interpretability, and can help mitigate overfitting. There are several approaches commonly categorized as filter methods, wrapper methods, and embedded methods. Each approach has different trade-offs in terms of computational efficiency, effectiveness, and interpretability.
Filter Methods
Filter methods evaluate features according to a statistical measure that reflects how informative or relevant they are with respect to the target variable. These methods are typically faster compared to wrapper or embedded methods because they do not rely on iteratively training a specific predictive model on the data; rather, they rely on certain data-derived statistics or relationships.
Examples include:
Correlation-based selection, where one calculates correlation (e.g., Pearson correlation coefficient) between each feature and the target. Features exceeding a certain absolute correlation threshold may be selected.
Mutual information-based selection, which measures how much knowing a feature reduces uncertainty about the target variable.
Statistical tests such as chi-square for categorical variables or ANOVA F-test for comparing mean differences across classes.
Below is the core formula for the Pearson correlation coefficient between a feature x and the target y:
Here:
x_i is the i-th value of the feature x.
y_i is the i-th value of the target y.
n is the number of data points.
bar{x} is the mean of x over all samples.
bar{y} is the mean of y over all samples.
r_{xy} is the correlation coefficient in the range [-1, 1]. A value close to 1 or -1 indicates high linear correlation (positive or negative), while a value close to 0 indicates low linear correlation.
Filter methods are simple and fast, but they can miss complex feature interactions since each feature is often considered in isolation relative to the target.
Wrapper Methods
Wrapper methods search through subsets of features, building and evaluating a model (like a regression or classification model) as a “wrapper.” They try different feature combinations, train a model on each subset, and then select the subset that yields the best performance metric. They can capture feature interactions better than filter methods but can be computationally expensive for large feature sets.
Common examples:
Forward selection, which starts with no features and iteratively adds the feature that most improves model performance.
Backward elimination, which starts with all features and iteratively removes the least impactful feature.
Recursive Feature Elimination (RFE), which trains a model (often linear or tree-based), then ranks features by importance and iteratively eliminates the least important ones.
Embedded Methods
Embedded methods learn which features are the most important during the model training process itself. They incorporate feature selection directly into the construction of the model.
Examples include:
L1-regularized linear or logistic regression (Lasso), where the L1 penalty tends to push coefficients of irrelevant features to zero.
Tree-based models (such as Random Forests or Gradient Boosted Trees) that can rank features according to how effectively they split the data. Features that produce minimal reduction in the loss function can be considered less relevant.
A classic example is training a Lasso regression. The L1 penalty shrinks coefficients to absolute zero when they are not contributing strongly to minimizing the loss, thus effectively performing feature selection.
Practical Example in Python
Below is a quick snippet that demonstrates using a filter-based feature selection approach in scikit-learn (for demonstration):
import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
# Synthetic data
X, y = make_classification(n_samples=1000, n_features=20,
n_informative=5, n_redundant=2,
random_state=42)
# Filter method: ANOVA F-test
selector = SelectKBest(score_func=f_classif, k=5)
X_new = selector.fit_transform(X, y)
# Train a model on the reduced set
X_train, X_test, y_train, y_test = train_test_split(X_new, y,
test_size=0.2,
random_state=42)
model = LogisticRegression()
model.fit(X_train, y_train)
print("Accuracy:", model.score(X_test, y_test))
This snippet demonstrates how to perform a feature selection step based on the ANOVA F-test, keep the top 5 features, and then train a logistic regression on that reduced feature set.
Potential Follow-up Questions
How do you decide when to apply filter, wrapper, or embedded methods?
Filter methods are fast and are often a good first pass when you want to quickly reduce dimensionality before moving on to more computationally expensive methods. Wrapper methods are particularly useful when model performance is paramount and you can afford the extra computation. Embedded methods are advantageous if you are training a model with built-in regularization or feature importance mechanisms, as they streamline feature selection into the model building process.
A deeper point is that sometimes a combination is used: first filter out clearly irrelevant features, then apply a more expensive wrapper or embedded method on the reduced feature set.
What are the practical pitfalls of each feature selection approach?
Filter methods might remove features that are only informative when combined with other features. Wrapper methods can be very computationally expensive, especially with large numbers of features, and they may overfit if your dataset is small. Embedded methods rely heavily on the assumptions of the underlying model. For instance, L1-regularization is helpful, but if the relationships between features and target are non-linear, a linear model might not capture them well.
In real-world scenarios, it is crucial to validate that the selected features generalize well beyond the training data. Cross-validation is key to avoid overfitting in the selection process.
How does regularization-based feature selection differ from dimension reduction techniques like PCA?
Regularization-based feature selection aims to preserve the original features but shrink or remove irrelevant ones. Dimension reduction methods like PCA transform the feature space into a new coordinate system (principal components), which can make interpretability more challenging because you no longer have the original feature dimensions. Feature selection is usually preferred when interpretability with original features is essential, whereas dimension reduction might be ideal for purely predictive tasks with high-dimensional data.
How would you handle correlated features in the context of feature selection?
When features are correlated, filter methods like correlation with the target might inadvertently select features that are duplicates of each other in terms of their information content. One common approach is to analyze pairwise correlation among features and remove redundant ones that have high correlation with others, while ensuring that you also maintain coverage of features that are highly correlated with the target.
In wrapper or embedded methods, correlated features may not pose as big an issue because the model can decide which correlated features to keep (for instance, Lasso can zero out some of the correlated coefficients).
How might you measure the effectiveness of a chosen subset of features?
One approach is to perform cross-validation for different feature subsets and observe the performance metric (e.g., accuracy, F1-score, RMSE). The chosen feature subset should yield good generalization performance across folds. Additionally, checking the stability of feature subsets across different folds can help ensure that the selection is not excessively dependent on random variations in the training data.
If interpretability is important, domain experts can be consulted to confirm that the selected features make sense in context. Features that are identified as critical by domain expertise often align with stable subsets discovered by robust feature selection methods.
Below are additional follow-up questions
How do you handle feature selection in highly imbalanced classification tasks?
One of the key challenges in imbalanced classification is that standard metrics such as accuracy can be misleading. In a scenario where 99% of data belong to one class, a naive model predicting the majority class for all inputs could achieve 99% accuracy without learning any meaningful distinctions. Therefore, traditional filter or wrapper methods that rely on accuracy or similar metrics might highlight features useful for distinguishing the majority class but ignore important minority-class patterns.
In practice, one strategy is to use specialized metrics (like F1-score, AUC-ROC, precision-recall AUC) during the feature selection process. For instance, if you are using a wrapper method, you could select subsets of features based on how well a model performs using precision-recall AUC. For filter methods, you might rely on measures (e.g., mutual information) that consider joint distributions of features and target, ensuring you do not discard low-frequency (but critical) patterns.
Pitfalls include:
Accidentally discarding features correlated with minority-class characteristics if one only uses majority-class-driven metrics.
Overfitting to the minority class if the dataset is very small or the same minority samples are repeatedly used in a cross-validation setting.
Careful cross-validation strategies (like stratified folds) and synthetic data augmentation (SMOTE, for instance) can help preserve minority-class signals when deciding which features are genuinely relevant.
How do you handle feature selection for time-series or sequence data?
Time-series data often has temporal dependence, which means selecting features cannot be treated as an entirely independent process across time. Techniques that ignore temporal structure can result in data leakage if future values (or transformations involving future points) are used when selecting features for earlier timesteps.
A common approach is to:
Maintain the chronological order. Split the dataset into training, validation, and test sets by time instead of random splits.
Use time-based cross-validation methods (e.g., rolling or expanding windows) to ensure that performance metrics are calculated by only using past data to predict the future.
Incorporate specialized features such as lag features, rolling averages, or exponential moving averages. When performing feature selection, evaluate the relevancy of these lag-based features in the context of a predictive model for future points.
A subtle issue is how far into the future you need to predict. If you are building features with a large time lag but you only need near-future predictions, those features might be less relevant. On the other hand, shorter-lag features might be more predictive but also more prone to transient noise. Balancing these factors is essential.
How do you perform feature selection when you have very high-dimensional data but relatively few samples?
In scenarios such as genomics, text classification with limited labeled examples, or certain image-based problems, the number of features can be much larger than the number of samples. Here, overfitting is a primary concern, and the search space for wrapper methods can become enormous.
Approaches typically used:
Strong regularization (L1 or L1+L2 penalties) in an embedded method can help reduce overfitting by shrinking irrelevant coefficients to zero.
Filter methods (mutual information or correlation-based thresholds) can quickly prune obviously uninformative features before applying more advanced methods. This is sometimes referred to as a hybrid approach where you first apply a simple filter to reduce dimensionality drastically and then use a more computationally expensive wrapper or embedded approach on the remaining subset.
Domain knowledge can be extremely beneficial. For instance, in genomics, certain genes or biomarkers may be known to be more relevant.
Pitfalls include:
Aggressively pruning features might discard subtle but important signals when the sample size is small.
Naively applying model-based methods on extremely high-dimensional data can lead to severe overfitting, especially if cross-validation folds are not large enough or not properly stratified.
What is the difference between forward feature selection and backward feature elimination in terms of real-world performance?
Forward feature selection starts with an empty set of features and iteratively adds one feature at a time, picking the feature that yields the best performance improvement at each step. Backward elimination starts with all features and iteratively removes the least useful feature.
In practice:
Forward selection can be more efficient if you expect only a small number of features to be relevant, because it might converge to a good subset without searching the entire space of features. However, forward selection might not revisit previously chosen features to see if they still remain optimal in the presence of newly selected ones.
Backward elimination can be more computationally expensive for very large feature sets, because it starts from the entire set. That said, it can be more thorough in identifying interdependencies among features since it sees how the removal of a feature impacts performance in the presence of all others.
A subtle issue is that neither forward nor backward approaches guarantee finding the globally optimal set of features—they can get stuck in local optima. With forward selection, once you add a suboptimal feature early, you may not remove it later. With backward elimination, you might remove a feature early that becomes beneficial in combination with a different subset. Cross-validation and domain expertise are often used to mitigate these limitations.
How would you handle missing data in the feature selection process?
When data contains missing values, certain filter methods (like correlation) or wrapper methods that rely on model training can fail if the missing data is not handled properly. For example, if you simply drop rows with missing values, you might lose valuable information and inadvertently bias your dataset.
Common strategies:
Imputation: Fill in missing values with mean, median, or more sophisticated methods (like k-nearest neighbors imputation). However, if a large fraction of the data is missing for a certain feature, that feature may no longer be reliable. Conversely, consistent missingness patterns can be informative in themselves if missingness is correlated with the target.
Model-based imputation within a cross-validation framework. Ensure that imputation is done separately in each fold to prevent data leakage.
Treat “missingness” as a separate category if it carries meaningful information. For instance, a patient missing a test result might be clinically informative in a medical dataset.
A subtle pitfall is performing imputation on the entire dataset before splitting into train and test sets. This can cause data leakage. The recommended approach is to split your data into training and validation/test sets first, fit the imputation technique on the training set, and then apply it to the validation/test sets.
How do you determine the right threshold or number of features to select in filter-based methods?
In filter-based approaches, one often picks a threshold (for correlation, mutual information, chi-square test, etc.) or specifies the number of top-ranked features. Deciding this threshold or k can be tricky.
Practical approaches:
Use domain knowledge to guide how many features are realistically relevant to the problem.
Apply cross-validation: for each candidate number of features k in a range, select the top k features, train a model, and measure the performance. Then pick the k that yields the best performance on the validation set.
Look for an “elbow” or plateau in a plot of validation performance vs. the number of features.
A pitfall arises if one sets the threshold solely based on statistical significance without considering multicollinearity, redundancy, or the final model’s performance. Another subtlety is that any threshold chosen purely on the training set might not generalize well to unseen data. Employing cross-validation remains a robust way to mitigate these issues.
How can you incorporate domain knowledge into feature selection?
Domain knowledge can drastically improve the efficiency and interpretability of feature selection. Instead of treating the problem as a purely statistical exercise, one can rely on expert insights to identify which features are known to be clinically relevant, physically meaningful, or historically proven to have predictive power.
Ways to incorporate:
Start with a small set of features that domain experts believe are crucial. Then expand iteratively if performance suggests more coverage is needed.
Use domain constraints. For example, if certain features must appear together because they measure related phenomena, treat them as a unit in wrapper methods.
Qualitatively review the top features from a filter-based or embedded approach with domain experts to see if they make sense from a scientific perspective.
A subtlety: occasionally, domain knowledge might favor older, established views that might overlook novel interactions discovered by data-driven methods. Hence, a combination of domain insights and data-driven selection often yields the best results.
How do you handle feature selection when using deep neural networks?
Neural networks, especially deep architectures, can learn complex feature representations automatically, making explicit feature selection less common than in shallow modeling. However, in some scenarios (like interpretability requirements or extremely high-dimensional input with limited data), selecting a smaller subset of features can still be beneficial.
Possible approaches:
Employ an embedded strategy using techniques such as a “sparse neural network” with L1 regularization on input weights to drive insignificant connections toward zero. This approach is conceptually similar to Lasso.
Use a filter or wrapper method prior to the neural network. You might start with correlation-based or mutual information-based filtering to remove obviously redundant or irrelevant features, then feed the remaining subset to a deep model.
For interpretability, one can use post-hoc methods (e.g., feature attribution methods like Integrated Gradients or SHAP) to identify which inputs strongly influence the network’s output. This insight might help you prune away unimportant features in subsequent training cycles.
A pitfall is that neural networks often require large datasets to generalize well. Overly aggressive feature selection might hamper the network’s ability to learn complex representations, especially if the selection discards relevant but subtle information.
Is feature selection always necessary for large datasets containing many features?
Not always. Some robust models (like Random Forests or gradient-boosted trees) handle large, sparse feature spaces without explicit feature selection. These models can downweight or ignore irrelevant features if they do not yield performance improvements during splits. Similarly, certain regularization schemes in neural networks or linear models reduce the impact of irrelevant features implicitly.
However, there are still reasons to consider feature selection in big data contexts:
Computational efficiency: fewer features lead to faster model training and inference.
Interpretability: domain experts find it easier to reason about a smaller set of features.
Data storage and pipeline simplicity: storing fewer features can reduce memory and latency requirements in production.
Pitfalls occur if you assume that a model “automatically handles” thousands of irrelevant features. In practice, large amounts of noisy features can confuse learning, especially if your data is not as big as you think or if there is excessive correlation. You might inadvertently hamper the model’s ability to converge to a good solution.
What are some common mistakes people make when integrating feature selection in a machine learning pipeline?
Some mistakes include:
Performing feature selection outside of a proper cross-validation loop. For instance, if you select features based on the entire dataset before splitting into train and validation sets, you risk data leakage.
Over-relying on a single metric (like accuracy) when the problem requires more nuanced evaluation (e.g., a skewed dataset might need recall or F1-score).
Discarding features purely based on univariate statistical tests. This ignores interactions among features. Some features might only become predictive when used alongside others.
Ignoring data preparation details (e.g., normalization, encoding) during feature selection. If your pipeline first selects features and then standardizes them differently per fold, you can introduce inconsistency and cause model performance to degrade unexpectedly in real-world scenarios.
A subtle edge case is online or streaming data, where new features and new data arrive continuously. People might pick a feature set once, and then over time, the data distribution shifts, rendering those features less effective. Ongoing monitoring is essential in such environments.