ML Interview Q Series: How can less Training Data give Higher Accuracy?
📚 Browse the full ML Interview series here.
Comprehensive Explanation
A seemingly counterintuitive observation is that sometimes reducing the size of your training set can lead to higher accuracy. While the general rule in machine learning is that more data helps improve model performance, there are scenarios where less data actually helps or, at least, appears to. This situation usually arises when the additional data introduces noise, mislabeled examples, or domain mismatch. It can also occur in highly imbalanced or specialized domains where carefully curated, high-quality data outperforms a much larger but noisy dataset.
Overfitting, Bias-Variance Considerations, and Data Quality
Models learn from patterns in the training data, but if that data contains a lot of misleading signals or inconsistencies, the model can end up fitting those spurious patterns. This phenomenon is closely related to the bias-variance decomposition in learning:
In the expression above, E[...] indicates the expected error of the model across the data distribution. Bias(\hat{f}(x)) represents how far the model’s average predictions deviate from the true values. Var(\hat{f}(x)) captures the variability of the model predictions for different training sets. And ε is the irreducible noise in the data that no model can fully capture.
If you keep adding noisy or low-quality data, the variance term can increase significantly because the model starts fitting inconsistent signals. If some portion of the data has incorrect labels or out-of-domain examples, removing that portion (thus effectively “using less data”) might improve overall accuracy by reducing harmful noise and variance.
Mislabeled and Out-of-Domain Data
In large datasets, especially those collected from crowdsourced or automated mechanisms, mislabeled examples may appear. If these labels are consistently wrong or are random and the proportion is large enough, training on them can degrade model performance. An example is language data that includes contradictory human annotations or images that are tagged incorrectly. Trimming down to a smaller, higher-quality subset can yield better accuracy.
Out-of-domain data is another source of trouble. Suppose your task is to classify medical images from a specific device, but your dataset includes images captured from multiple devices with vastly different characteristics. Training on everything might confuse the model, while focusing on the single relevant domain might yield higher accuracy on your in-domain test set.
Imbalanced Class Distributions and Curated Subsets
Sometimes the “less data” that remains after a certain form of filtering or balancing is much more representative. If you have a highly imbalanced dataset, selectively sampling to ensure balanced classes can mean throwing away a lot of data from the over-represented classes. The overall size of the dataset decreases, yet the performance on important metrics can actually go up.
Illustrative Code Example
Below is a conceptual Python snippet to illustrate how one might compare a full noisy dataset to a reduced, cleaner subset. The example is contrived, but it demonstrates how removing noisy data can sometimes lead to better performance:
import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
# Generate synthetic data
X, y = make_classification(n_samples=2000, n_features=10,
n_informative=8, n_redundant=2,
random_state=42)
# Introduce noise in some portion of the labels
rng = np.random.RandomState(42)
noisy_indices = rng.choice(range(len(y)), size=300, replace=False)
y_noisy = y.copy()
y_noisy[noisy_indices] = 1 - y_noisy[noisy_indices] # Flip labels
# Split into train/test
X_train_full, X_test, y_train_full, y_test_full = train_test_split(X, y_noisy,
test_size=0.3,
random_state=42)
# Option 1: Train on full noisy dataset
clf_full = RandomForestClassifier(random_state=42)
clf_full.fit(X_train_full, y_train_full)
y_pred_full = clf_full.predict(X_test)
accuracy_full = accuracy_score(y_test_full, y_pred_full)
# Option 2: Filter out some "noisy" data by index (hypothetically identified)
# For demonstration, let's just remove the known noisy indices
mask_clean = np.ones(len(X_train_full), dtype=bool)
# Imagine we identified half of the noisy indices somehow
noisy_indices_subset = noisy_indices[:150]
for ni in noisy_indices_subset:
# We need to ensure ni matches our train split; here we assume direct overlap
if ni < len(X_train_full):
mask_clean[ni] = False
X_train_less = X_train_full[mask_clean]
y_train_less = y_train_full[mask_clean]
clf_less = RandomForestClassifier(random_state=42)
clf_less.fit(X_train_less, y_train_less)
y_pred_less = clf_less.predict(X_test)
accuracy_less = accuracy_score(y_test_full, y_pred_less)
print("Accuracy with full noisy data: ", accuracy_full)
print("Accuracy with less but cleaner data: ", accuracy_less)
In practice, you might see the model trained on the smaller (but cleaner) dataset achieving higher accuracy than the model trained on the full noisy set. Of course, in a real-world scenario, identifying and filtering out noisy data is not always trivial.
Possible Follow-Up Questions
Could there be other reasons why smaller datasets might perform better?
Yes, besides noise and label corruption, there can be scenarios involving distribution shifts, where part of the dataset is from a distribution that is not relevant to the task at hand. Excluding that portion effectively makes the training data “smaller,” but also more consistent with the real-world distribution you aim to generalize to.
Another aspect involves simpler models that can overfit large amounts of noisy or heterogeneous data but may do reasonably well with a smaller, more coherent subset.
How do we identify the portion of the training data that should be removed?
It can be challenging to identify and remove low-quality or out-of-domain data without inadvertently discarding valuable information. Methods might include:
Performing anomaly detection on features to detect outliers that are suspect.
Training a model and identifying data points that consistently contribute to high loss or misclassification.
Using domain experts to manually label or verify certain subsets.
Applying cross-validation and measuring data influence or model interpretability methods (like influence functions) to see which samples degrade performance.
Can active learning help to choose the most informative subset?
Yes, active learning strategies are designed to iteratively request labels for the most “informative” or “uncertain” samples. By focusing on a subset of critical examples, the model can often reach higher accuracy with fewer training instances. This approach is particularly useful when data labeling is expensive or when large parts of the dataset provide redundant information.
Does reducing training data always improve accuracy?
No. Typically, more data is beneficial as long as it aligns well with the true data distribution and has minimal mislabeling. Reducing the dataset helps only when you remove harmful aspects, such as noisy or out-of-domain samples. If your data is of uniformly high quality, removing a random portion generally hurts performance by reducing the overall coverage of the data distribution.
How can we ensure that smaller subsets are not creating bias?
When you remove a portion of your dataset, you risk skewing the distribution. This can lead to unintended biases if the removed data contains sub-populations essential for the model’s generalization. It’s crucial to ensure that you remove data systematically, checking that all important demographics or feature distributions remain adequately represented. Techniques such as stratified sampling help preserve proportions of different classes or relevant attributes.
What if cleaning the data is not an option?
Sometimes you must work with data as-is, especially if labeling resources are scarce. In these cases, techniques like robust training procedures, noise-robust loss functions, or advanced regularization can mitigate the effects of noisy data. Gathering more data of higher quality (if feasible) is often an effective strategy. Additionally, you might use unsupervised or semi-supervised learning methods to help identify problematic samples before fully retraining a supervised model.
What real-world pitfalls could arise in large datasets?
In large-scale real-world datasets, some pitfalls include:
Drifting data distributions over time (e.g., concept drift in time-series or user-generated data).
Substantial annotation noise due to crowdsourcing or automated labeling.
Hidden biases that become amplified in large datasets, leading to worse outcomes for underrepresented groups.
Difficulties in maintaining data pipelines, ensuring consistent labeling and cleaning processes.
These challenges highlight why “bigger is better” is not always the full story. High-quality curation and understanding of data collection processes can sometimes provide more gains than simply adding more unverified data.
Below are additional follow-up questions
What are the trade-offs in choosing a more complex model on a smaller, cleaner dataset versus a simpler model on a larger, noisier dataset?
When selecting a modeling strategy, there is a natural tension between model complexity and data quality. A complex model trained on a smaller but higher-quality dataset might generalize better if noise is minimal, because that smaller dataset can focus the model on strong, consistent patterns. However, highly complex models can overfit when trained on insufficient data, particularly if there are not enough samples to capture the true diversity of the underlying distribution.
On the other hand, using a simpler model on a large, noisy dataset might help in two ways:
A simpler model can be more robust to outliers and random noise, provided there is enough data to smooth over inconsistencies.
If the noise is not too overwhelming, the larger dataset may allow the model to identify broad trends despite the inaccuracies.
Pitfalls and edge cases include:
A mismatch between model capacity and data size: a complex model might memorize small data subsets without learning generalizable patterns.
Over-simplification when noise is extensive: if a dataset is extremely noisy and you still use a simpler model, you may underfit, missing important details hidden within the data.
Shifts in data distribution: a complex model might inadvertently learn artifacts that only occur in the smaller set, whereas a simpler model on a larger set might capture more generalizable features.
How can early stopping or regularization help mitigate negative effects of noisy data without discarding it?
Early stopping and regularization strategies (like L2 regularization, dropout in neural networks, or smaller maximum tree depth in decision trees) limit how deeply or how precisely models can fit the data. This can prevent overfitting to noisy training points. The training process is guided to find a balance between fitting the majority pattern and avoiding excessive responsiveness to outliers or mislabeled samples.
Potential pitfalls arise when:
The regularization is too strong, causing underfitting and ignoring genuine structure.
Early stopping is applied too early, preventing the model from learning essential complex relationships.
The noise might not be uniformly distributed. For instance, certain classes might exhibit more mislabeled points, leading to systematic performance degradation despite regularization.
Could transferring knowledge from a large dataset to a smaller curated dataset be beneficial, and how do we ensure effective transfer?
Transfer learning allows a model trained on a large, possibly noisy dataset to capture general features that can then be adapted or fine-tuned on a smaller, high-quality dataset. This approach is especially popular in deep learning for computer vision or NLP, where pretrained models on massive datasets are refined for specific tasks with relatively few clean samples.
Important considerations:
If the large dataset is from a completely different domain, some representations may not generalize well to the smaller dataset, causing negative transfer.
The fine-tuning process must be carefully managed to avoid overfitting, especially when the smaller dataset is quite limited.
Hyperparameter choices, such as learning rate and number of epochs for fine-tuning, can greatly influence whether the transfer is successful or leads to under/overfitting.
How do we systematically measure the effect of removing certain subsets of the training data on overall performance?
A systematic approach involves iteratively or selectively removing portions of data and evaluating performance changes on a validation or hold-out test set. Methods include:
Cross-validation with subset removal: each fold trains a model with different subsets removed to gauge consistency of the improvement (or decline).
Influence functions or other data-centric methods: these can quantify how much each training instance affects the final model parameters.
Model interpretability and error analysis: by examining which samples the model misclassifies, you can isolate suspicious or low-quality subsets.
Edge cases:
Removing data may appear beneficial on a short-term validation set but cause a reduction in long-term robustness if those removed examples carry rare but crucial patterns.
If there is concept drift, the “noisy” data might actually represent new trends, and discarding it would degrade the model’s ability to adapt to future shifts.
When might a smaller dataset miss important complexities that are beneficial to generalization?
A key risk is that smaller datasets are less likely to capture the full diversity of the real data distribution. If an important but rare phenomenon occurs in only a small fraction of the examples, removing too many data points (or never having enough in the first place) can cause the model to miss essential signals. For instance, in fraud detection tasks, fraudulent events are often scarce. Retaining those outliers, even if they appear noisy, can be vital to the model’s ability to detect fraud.
Pitfalls:
Overlooking edge cases that matter in real-world scenarios (e.g., safety-critical applications).
Biasing the model toward more frequent patterns at the expense of minority classes.
How can domain adaptation or domain generalization techniques help instead of simply discarding out-of-domain data?
Rather than removing data that seems irrelevant, domain adaptation techniques aim to align the feature distributions or extract domain-invariant representations across different domains. If you have out-of-domain samples, you might:
Use adversarial domain adaptation to encourage the network to learn features that cannot distinguish between domains.
Perform data augmentation that bridges differences between domains.
Weight training examples differently based on how well they match your target domain.
Potential pitfalls:
Overcomplicating the training pipeline and increasing risk of hyperparameter mismanagement.
Failing to identify subtle domain differences that remain after adaptation, leading to partial or inadequate alignment.
If the out-of-domain data is too large or too divergent, the adaptation process might degrade performance on the target domain without robust procedures in place.
If you are under legal or compliance constraints that limit certain data usage, how do you maintain model performance?
In domains like healthcare or finance, legal regulations (e.g., HIPAA, GDPR) might require data anonymization or prohibit using personally identifiable data. You could:
Use differential privacy to train models while protecting sensitive information.
Resort to synthetic data generation that mirrors the statistical properties of restricted data but does not contain actual private details.
Collaborate with legal teams to define permissible aggregated or derived features that are still predictive without revealing confidential elements.
Pitfalls:
Synthetic data might introduce artificial patterns or fail to capture important correlations, decreasing real-world performance.
Removing too many features can result in feature distribution drift, requiring re-engineering or more complex modeling techniques.
How might we quantify or estimate the label noise or out-of-domain fraction in a dataset so we can systematically remove it?
One approach is to compute consistency or agreement among multiple models or repeated cross-validation folds. Instances that are frequently misclassified by a diverse set of models can be flagged as suspiciously labeled. Another approach is unsupervised clustering in feature space to detect out-of-domain clusters.
Edge cases include:
Rare classes that consistently confuse models, leading them to be incorrectly flagged as noisy.
Data shifts over time might make older data “look noisy,” when in reality it represents a previous distribution that still retains some importance.
Could oversampling or undersampling imbalanced datasets inadvertently remove valuable data or create synthetic noise?
Yes, oversampling or undersampling techniques aim to adjust class distributions, but they can introduce new issues. In undersampling, valuable minority samples might be discarded if they share certain feature characteristics with the majority class. In oversampling, methods such as SMOTE can create synthetic examples that do not represent real-world scenarios, potentially confusing the model.
Pitfalls:
Oversampling can magnify label noise if noisy points in the minority class are duplicated or used to generate synthetic examples.
Improper undersampling might remove critical outliers or borderline examples that are essential for learning the decision boundary.
Balancing the trade-off between actual coverage of the data distribution and artificially induced sampling is non-trivial.
What are common best practices in designing data pipelines or data cleaning pipelines to reduce reliance on discarding data post-hoc?
Building robust pipelines that catch labeling errors and anomalies early is crucial. Best practices involve:
Implementing thorough logging and versioning of data to track when and how data was collected, labeled, or modified.
Establishing automated data validation checks to detect anomalies, missing values, or unexpected distributions before training begins.
Conducting iterative labeling or human-in-the-loop verification on suspicious samples flagged by unsupervised or model-based anomaly detection.
Maintaining feedback loops from the deployed model: real-world errors or user feedback can be incorporated to refine labels and discover new edge cases.
Pitfalls:
Overburdening the pipeline with excessive checks might cause delays or false positives, leading to valid data being discarded.
Scaling these practices for extremely large datasets can be challenging; parallelization and distributed systems become necessary to manage data efficiently.
A lack of domain expertise can lead to incorrectly labeling normal samples as outliers or failing to catch subtle, domain-specific noise.