ML Interview Q Series: What techniques do you use to handle outliers in a dataset?

📚 Browse the full ML Interview series here.

Comprehensive Explanation

Outliers are observations that lie significantly far from the overall distribution of the data. These points can bias statistical measures and model training if not handled appropriately. The exact approach for dealing with outliers depends on the nature of the data, the goals of the analysis, and how critical those extreme points are. In certain real-world applications, outliers might be valid data points offering unique insights, whereas in other scenarios, they could be erroneous or unrepresentative artifacts that should be removed or transformed.

Identifying Outliers

It is generally important to investigate outliers in multiple ways before deciding on a strategy. Here are some common practices:

Visual Examination Visualizing data through box plots, histograms, scatter plots, or pair plots can often give a quick idea of any data anomalies or extreme values.

IQR-Based Method A widely used approach involves the Interquartile Range (IQR), which is the difference between the 75th percentile (Q3) and the 25th percentile (Q1). Points that lie outside [Q1 - k * IQR, Q3 + k * IQR] are considered outliers, where k is often chosen as 1.5 or 3, depending on how strict you want to be.

In this formula, Q1 is the first quartile, Q3 is the third quartile, and 1.5 is a common multiplier to set the threshold for classifying outliers (although 3 can also be used for a more lenient filter).

Z-Score Method Another popular method is to calculate how many standard deviations an observation is away from the mean. If an observation’s z-score is above a certain threshold (often 3), it can be considered an outlier. The z-score can be computed as

where x is the data point, μ is the mean of the distribution, and σ is its standard deviation. Values of z above or below the threshold could be flagged as outliers.

Domain Knowledge Statistical thresholds are often just a starting point. Real-world knowledge about the data is crucial for determining which outliers might be legitimate rare events versus which could be noise or errors.

Approaches to Dealing with Outliers

Removal of Outliers In cases where outliers are clearly due to measurement errors or data corruption, removing them might be justified. However, dropping outliers indiscriminately can result in loss of important information, especially if the dataset is small or if those points are inherently valid.

Capping (Winsorization) Instead of outright removing outliers, one can cap or floor them at certain percentiles (for instance, setting any value above the 99th percentile to the 99th percentile value). This preserves the data structure while limiting the impact of extreme values.

Transformation Applying transformations such as log, Box-Cox, or power transforms can reduce the influence of large values by making the distribution more symmetric. For example, applying a log transform is common when dealing with data that span multiple orders of magnitude.

Using Robust Models and Metrics Some models (like tree-based methods) are less sensitive to outliers. Also, robust metrics such as median absolute deviation or Huber loss can down-weight the effect of extreme outliers.

Sample Python Implementation

import numpy as np
import pandas as pd

# Suppose we have a DataFrame called df with numerical column 'feature'
df = pd.DataFrame({
    'feature': [10, 12, 500, 14, 15, 16, 600, 18, 19, 20, 21, 22, 30]
})

# 1. IQR-based approach
Q1 = df['feature'].quantile(0.25)
Q3 = df['feature'].quantile(0.75)
IQR = Q3 - Q1

lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR

# Outliers
outliers = df[(df['feature'] < lower_bound) | (df['feature'] > upper_bound)]
print("Outliers (IQR method):")
print(outliers)

# 2. Z-Score-based approach
mean_val = df['feature'].mean()
std_val = df['feature'].std()

df['zscore'] = (df['feature'] - mean_val) / std_val
z_threshold = 3
z_outliers = df[abs(df['zscore']) > z_threshold]
print("\nOutliers (Z-score method):")
print(z_outliers)

# 3. Capping approach
upper_cap = df['feature'].quantile(0.95)
df_capped = df.copy()
df_capped.loc[df_capped['feature'] > upper_cap, 'feature'] = upper_cap
print("\nData after capping at 95th percentile:")
print(df_capped)

In this example, you can see how to identify outliers using IQR and z-score methods, then optionally apply a capping strategy.

Handling Outliers in Different Contexts

Time-Series Data Outliers in time-series data can have various causes such as sensor malfunction or seasonal spikes. One approach is to apply rolling statistical measures, or domain-driven thresholds, to detect anomalies at each time step. Another is to use specialized algorithms (e.g., ARIMA or LSTM-based models with anomaly detection layers) that capture temporal correlations.

High-Dimensional Data When dealing with high-dimensional features, metrics like Euclidean distance may lose their effectiveness because most points become equidistant in high dimensions. Techniques such as isolation forests or DBSCAN with carefully tuned parameters can sometimes be more effective, as they rely on different principles for outlier identification.

Practical Considerations It is critical to treat training and test sets consistently. If you remove or transform outliers in the training set, apply the same transformations to the test set. Also, re-check outliers after feature engineering because new features or scaling might change which points appear extreme.

Follow-Up Questions

How do you decide whether to keep or remove an outlier?

One must rely on domain-specific insights. If an outlier corresponds to a plausible real-world scenario (for instance, a valid rare event), removing it might cause the model to overlook important tail behavior. However, if further investigation reveals that the outlier is due to a sensor malfunction or data entry error, removing it is justified. Always analyze outliers thoroughly to determine their legitimacy before deciding on a strategy.

Why might you choose capping over removing outliers?

Capping (or winsorizing) modifies the extreme values to a less extreme percentile without dropping them entirely, thus preserving the size of the dataset and avoiding gaps in the distribution. This can be especially beneficial when the dataset is not very large, because you don’t reduce your sample size and still limit the potential skew introduced by extreme points.

When is a log transform helpful in handling outliers?

A log transform is often useful when the data spans several orders of magnitude, or when the distribution is heavily right-skewed. By compressing the scale of large values more than small ones, the log transform reduces the relative impact of outliers and can make the distribution more symmetric. This is commonly applied to data such as financial transactions, biological data, or frequency counts where the spread can be very large.

Why might tree-based models be more robust to outliers?

Tree-based methods (like random forests or gradient boosting) split the data based on feature thresholds. Since these splits depend on sorting feature values rather than assuming a particular distance metric or distribution, extreme values do not drastically affect the decision boundaries. Consequently, outliers do not have as large an impact on the overall model as they would with models that rely on distance calculations (like linear or nearest-neighbor methods).

Are outliers always detrimental?

Not necessarily. Outliers might sometimes represent events that are rare but very important, like fraud detection or device failures. If your model is intended to capture such scenarios, removing these points could cause the model to miss critical patterns. Conversely, if outliers are indeed invalid observations or measurement errors, they can degrade model performance.

How can you handle outliers in cross-validation or hyperparameter tuning?

One strategy is to consistently apply your outlier-handling method (e.g., capping, removal, or transformation) within each fold of cross-validation. This ensures that the approach is not biased by the overall dataset statistics. For example, if you remove or transform outliers, do so inside each fold’s training subset, and then apply the same transformation rules to the corresponding validation subset.

How do you handle outliers in unsupervised learning tasks?

In unsupervised settings, there might not be clear labels to indicate correct or incorrect classes. Approaches like isolation forest, DBSCAN, or robust clustering (e.g., using distance metrics that are less sensitive to outliers) can be particularly helpful. You might also consider dimension reduction methods (like PCA) to project high-dimensional data into a space where outliers are more readily identified.

By considering these various approaches and techniques, one can more effectively address outliers in a range of datasets, leading to better data quality and model performance.

Below are additional follow-up questions

How do you handle outliers in very high-dimensional datasets where traditional distance-based methods become less effective?

In high-dimensional space, notions of distance can become counterintuitive. Many outlier detection methods rely on distances (e.g., Euclidean) or density-based measures to flag extreme points. However, as dimension grows, all points tend to become roughly equally distant from each other, making standard distance-based methods much less effective.

One approach is to first perform dimensionality reduction using PCA or t-SNE to capture major variance in fewer dimensions. After projecting the data into a lower-dimensional representation, it can be more tractable to apply outlier detection methods such as DBSCAN, isolation forests, or robust clustering. Another strategy is to analyze features separately or in smaller subsets, using domain knowledge to group related features and detect outliers within each group. A key pitfall is assuming that a univariate approach for each feature individually will be sufficient, because certain points might appear normal in each feature but still be anomalous when considering the joint distribution of multiple features. Regular communication with domain experts can help identify which features are truly relevant, reducing dimensionality effectively while preserving important variance.

Edge cases include situations where most features are sparse or binary, making typical methods (like PCA) less informative. In such cases, specialized dimensionality reduction or factorization methods that handle sparse data more gracefully (e.g., Non-negative Matrix Factorization) could be more appropriate.

How would you manage outliers in a streaming or real-time data pipeline where new data arrives continuously?

When data is streaming in real time, outlier handling must be both efficient and adaptive. A common approach is to maintain a running window (e.g., fixed-size or sliding window) and perform incremental computations of summary statistics (like running mean or median) to spot outliers on the fly. Another strategy is to use online algorithms such as incremental versions of DBSCAN or isolation forests that update their internal structure as new data arrives.

A pitfall in streaming environments is that the distribution itself might shift over time (concept drift). An outlier in one time window may become normal in a later window. Therefore, it is important to adapt thresholds dynamically to reflect the latest data distribution. Additionally, because decisions often need to be made in near real time, you must carefully balance the frequency of updates with computational constraints. This is especially challenging in domains like fraud detection, where you might not want to miss newly emerging patterns of rare but genuine fraud cases.

How do you identify and deal with outliers in ordinal or categorical features?

For ordinal data, outliers might be values that break the usual order pattern by being unexpectedly high or low compared to neighboring ranks. A simple detection technique is to calculate frequency distributions within each ordinal level and see if any level is extremely rare relative to adjacent levels. Another approach is to use specialized metrics that respect the order (such as weighted distance) to quantify how distant a point’s rank is from the bulk of the data.

For purely categorical features, one cannot directly compute a mean or standard deviation. Instead, unusual or rare categories (with extremely low frequency) might be considered potential outliers. Practitioners sometimes merge these rare categories into an “Other” class to prevent the model from overfitting to highly unique but uninformative categories. A downside is that this merging can obscure potentially relevant minority patterns if those categories are actually meaningful. Hence, domain knowledge is essential to ensure you are not discarding rare but valid events.

Is there any scenario where you would artificially introduce or amplify outliers for data augmentation?

While it might seem counterintuitive, there can be limited scenarios—particularly in anomaly or fraud detection—where you have few examples of outliers but wish to better train a model to detect them. In such a case, synthetic minority over-sampling methods (e.g., SMOTE-like adaptations for outlier scenarios) can be used to boost rare events. This approach can help a model learn boundaries around the outlier class. However, randomly generating “extreme values” that do not faithfully reflect real anomalies can degrade model performance or introduce unrealistic data points.

One pitfall is incorrectly labeling artificially created points as genuine anomalies. If the synthetic outliers do not capture realistic patterns or correlation structures, models trained on them may have a distorted notion of what constitutes an outlier. Additionally, oversampling any kind of outlier-like data must be done with caution to avoid overshadowing real data patterns.

What are robust statistical measures of central tendency or variability, and why might they be preferred for outlier detection?

Robust statistics, such as the median (instead of the mean) or the median absolute deviation (MAD) for dispersion (instead of the standard deviation), reduce the influence of extreme values. For instance, the median absolute deviation is calculated by measuring the absolute deviations from the median, rather than from the mean, making it less sensitive to outliers.

A potential pitfall when using these robust measures is that they might overlook certain types of outliers if the distribution itself is multimodal or has heavy tails. In other words, robust metrics can sometimes mask interesting behavior because they fundamentally aim to dampen the effect of outliers. However, in many practical cases where you suspect that extreme values might be spurious or due to measurement errors, robust statistics can provide a more stable baseline for further outlier analysis.

When would you rely more on domain knowledge than on purely statistical measures for outlier detection?

Statistical thresholds (like z-scores or IQR) are useful generic tools, but they might not always capture the real-world context. For example, in medical datasets, certain lab test values might appear extreme by normal statistical standards but are clinically plausible due to specific conditions. Conversely, a small deviation from the mean might actually be extremely significant in some highly controlled manufacturing process. Relying on domain experts ensures that outliers discovered by a purely statistical method are indeed anomalies and not legitimate phenomena.

A subtle pitfall is that domain experts can have biases toward what they consider “normal.” Sometimes a data-driven approach might detect an outlier that reveals an emerging trend unknown to experts. Balancing domain insights with data-based methods is crucial: confirm that outliers are genuine anomalies (or not) without entirely discarding what the data reveals.

What are good practices for data scaling or transformation in the presence of outliers?

Scaling methods like standardization (subtract mean, divide by standard deviation) can be heavily impacted by extreme values. Hence, robust scaling—where you transform data based on median and IQR—is often a better option. Alternatively, applying a suitable transformation (e.g., log) can compress high-valued data, reducing skew.

A potential edge case is when you have negative values but still want to apply a log transform. A workaround is to shift the entire distribution by adding a constant so that all values are positive, but this shift can change relationships among features if not done carefully. Additionally, if outliers are extremely large or small, the same scaling method for all data might not be optimal. In such cases, you might do selective transformation just for specific features known to have heavily skewed distributions.

Could ignoring outliers create ethical or regulatory concerns in certain domains?

In fields like finance, healthcare, or autonomous driving, ignoring outliers can lead to significant risks. An extremely rare patient outcome in a medical study might be indicative of a dangerous side effect. Overlooking an unusual transaction in financial data might allow fraud to go undetected. Regulatory bodies may even require explicit consideration of edge cases to ensure safety and compliance.

An important pitfall is failing audits if it’s discovered that certain critical observations were removed without justification. For regulated industries, you might need to document your outlier handling process thoroughly. Merely labeling a point as an outlier and discarding it could omit an essential event with legal or safety ramifications.