ML Interview Q Series: You have a large number of hypotheses to evaluate and plan to conduct multiple t-tests. What factors do you need to consider in this scenario?

📚 Browse the full ML Interview series here.

Comprehensive Explanation

When numerous hypotheses are tested using t-tests (or any other hypothesis testing framework), the critical concern is how to handle the phenomenon often described as the “multiple comparisons problem.” The more tests you conduct, the higher the likelihood that you encounter a statistically significant result purely by chance, leading to an increase in the overall (or “familywise”) Type I error rate. Here is a detailed look at the key considerations:

Multiple Comparisons Problem

If each t-test uses the same alpha (for instance, 0.05), running hundreds of t-tests substantially inflates the probability of rejecting at least one true null hypothesis by chance. This scenario is often called “alpha inflation,” since the probability of observing false positives accumulates rapidly as the number of tests grows.

Methods for Controlling the Error Rate

There are several statistical strategies to keep the false positives in check:

Bonferroni Correction

One of the simplest, yet more conservative, approaches to control the familywise error rate is the Bonferroni method. Instead of using alpha (say 0.05) for each hypothesis, you adjust the significance threshold to alpha' = alpha / m, where m is the total number of tests.

alpha is the original significance level (often 0.05). m is the total number of hypotheses you test. alpha' is the new adjusted per-test threshold that ensures the familywise Type I error remains controlled at alpha.

This approach is straightforward and widely used. However, it can be overly conservative, potentially increasing Type II errors (false negatives), particularly when the number of hypotheses is very large and some of those hypotheses truly reflect real effects.

Holm Correction (Holm–Bonferroni)

A more powerful method than a strict Bonferroni correction is the Holm procedure. The idea is to:

1. Sort the p-values in ascending order.

2. Compare the smallest p-value with alpha/m, the next smallest with alpha/(m-1), and so forth.

3. As soon as you encounter a p-value that fails the threshold, you stop rejecting further hypotheses.

This approach controls the familywise error rate but typically has more power than the original Bonferroni method, because it does not uniformly penalize all p-values from the start.

False Discovery Rate (FDR) Control

In many research settings, controlling the fraction of false positives among all positives is more important than guaranteeing no false positives whatsoever. This fraction is known as the false discovery rate (FDR). A common procedure for FDR control is the Benjamini–Hochberg method, which provides a higher power in detecting true effects.

p_{(k)} is the k-th smallest p-value among the m tests. m is the total number of hypotheses tested. alpha is the pre-specified FDR level (commonly 0.05). k is the rank index of the sorted p-values.

You find the largest k such that this condition holds, and reject all hypotheses whose p-values are less than or equal to p_{(k)}. This ensures that on average, the proportion of false rejections does not exceed alpha.

Statistical Power Considerations

When you apply stringent corrections (like Bonferroni), you reduce Type I errors at the expense of potentially missing true effects (increased Type II errors). Hence, if you expect many true signals among your hypotheses, you might lean toward an FDR-control approach for a better balance between discovery and controlling false positives.

Practical Example in Python

import numpy as np
from statsmodels.stats.multitest import multipletests

# Suppose we have a list of p-values from multiple t-tests
p_values = [0.001, 0.04, 0.02, 0.045, 0.0013, 0.10, 0.0005]  # etc.

# Bonferroni correction
reject_bonferroni, pvals_corr_bonferroni, _, _ = multipletests(p_values, alpha=0.05, method='bonferroni')

# Holm correction
reject_holm, pvals_corr_holm, _, _ = multipletests(p_values, alpha=0.05, method='holm')

# Benjamini–Hochberg (FDR) correction
reject_bh, pvals_corr_bh, _, _ = multipletests(p_values, alpha=0.05, method='fdr_bh')

print("Bonferroni corrected p-values:", pvals_corr_bonferroni)
print("Holm corrected p-values:", pvals_corr_holm)
print("Benjamini-Hochberg (FDR) corrected p-values:", pvals_corr_bh)

This snippet demonstrates how to correct p-values with Bonferroni, Holm, and Benjamini–Hochberg approaches. Depending on the context, you can pick the method that best aligns with your tolerance for Type I vs. Type II errors.

What Happens in Real-World Settings?

In real-world data analysis, it’s fairly common to test large numbers of hypotheses simultaneously (e.g., genome-wide association studies, feature selections in high-dimensional data, or large-scale screening experiments). Without multiple comparisons correction, you’d see a flurry of seemingly significant findings that do not replicate. Approaches like FDR control are therefore essential for deriving meaningful discoveries.

Potential Pitfalls

• Overly Conservative Thresholds: Bonferroni can lead to excessive false negatives if many hypotheses are truly positive.

• P-Hacking: If analysts select the “best” results among many tests, that artificially inflates significance. Good multiple testing procedures and pre-registration of analysis plans help mitigate this risk.

• Misinterpretation of Adjusted p-values: Adjusted p-values can be misunderstood as their unadjusted counterparts. Always clarify whether p-values are raw or have undergone correction.

Common Follow-Up Questions

How do you choose between Bonferroni-type corrections and FDR-based corrections?

In settings where even a single false positive could be highly problematic (e.g., confirming a novel drug’s serious side effect), the Bonferroni or Holm–Bonferroni familywise error control methods may be preferable because they minimize the risk of any false positives. On the other hand, if you can tolerate a small proportion of false positives in favor of identifying as many actual positives as possible (e.g., exploratory gene expression studies), then FDR (e.g., Benjamini–Hochberg) is more appropriate because it provides higher power.

Could sample size impact how we handle multiple testing corrections?

Yes, because statistical power generally grows with sample size, you can detect smaller effect sizes more reliably with larger samples. If your samples are small, aggressive corrections like Bonferroni can make it difficult to detect true effects. In large-sample scenarios, the penalty of multiple testing is somewhat mitigated by higher power. That said, you still must correct for multiple testing to avoid spurious findings.

Are there any advanced methods beyond these standard corrections?

Several more nuanced techniques exist, including hierarchical testing procedures (where you test sets of hypotheses in a certain sequence to improve power) and Bayesian approaches (e.g., empirical Bayes shrinkage of p-values). Researchers in fields such as genomics or neuroscience often incorporate domain knowledge or correlation structures among tests to refine multiple testing corrections. This can yield fewer false negatives while still controlling error rates.

When would you use permutation testing or bootstrap methods in this context?

Permutation or bootstrap methods can help estimate empirical distributions of test statistics when theoretical assumptions (e.g., normality, independence) are in doubt or the data are high-dimensional. You can run permutations of your data, compute test statistics under these permutations, and derive an empirical null distribution for each test. Then you apply a multiple comparisons correction on top of the empirical p-values. This is often done, for instance, in fMRI studies where strong spatial correlations exist among thousands of voxels.

Could correlation among hypotheses reduce or increase the need for correction?

Strong correlation among tested hypotheses can influence the effective number of independent tests. In some correction methods, if tests are assumed to be independent, the corrections might be too conservative when variables are highly correlated. Advanced methods account for correlation structure among hypotheses to refine the effective alpha adjustment, thus potentially reducing false negatives.

How to handle a scenario where new hypotheses are added after initial tests?

Adding new hypotheses mid-analysis can complicate the correction for multiple comparisons. Ideally, all hypotheses should be decided upon beforehand (prospective design) so that a single correction method can be applied. If new hypotheses are added after seeing some initial results, that can be considered data snooping or post-hoc exploration. In such cases, it is essential to document clearly which hypotheses were pre-specified and which emerged post-hoc, applying corrections separately or using more sophisticated sequential testing designs that account for the new tests.

All these considerations, from the fundamental requirement to control Type I error to the nuanced selection of correction methods, are crucial in high-dimensional or large-scale statistical analyses.

Below are additional follow-up questions

What if the data fails to meet normality assumptions or sample sizes are significantly unbalanced across tests?

When performing multiple t-tests, the standard t-test assumptions (normal distribution of residuals, equal variances, and independent samples) can be violated in real-world data. In such cases, standard t-tests may produce inaccurate p-values, and any subsequent multiple-comparison corrections (such as Bonferroni, Holm, or FDR-based) would be applying adjustments on potentially biased p-values. For instance, if you have a severely skewed distribution or a small group size in one of the tests, you could face inflated Type I or Type II errors before you even do the correction. In these scenarios, a non-parametric alternative (like the Mann–Whitney U test) or bootstrap-based methods might be more appropriate, provided you also account for multiple testing with those non-parametric p-values. A specific pitfall arises if some tests meet assumptions while others do not—merging parametric and non-parametric p-values can complicate the correction. Ideally, one would choose a consistent framework across tests or use a robust approach (e.g., a rank-based test) that can handle violations gracefully. Edge cases include extremely large outliers that break normality entirely, leading to inflated variance estimates and thus artificially large p-values in a typical t-test.

Could domain knowledge be used to group or cluster hypotheses before applying corrections?

There are situations where you have hundreds of hypotheses that are not all independent or that logically group together based on domain considerations (e.g., testing gene sets in genomics or feature subsets in high-dimensional models). Clustering hypotheses by shared characteristics or correlation structures can be advantageous. If a subset of hypotheses is known to be correlated, you can analyze them collectively (e.g., via a multivariate test or hierarchical modeling) rather than running separate univariate tests for each hypothesis. A pitfall is mislabeling or misgrouping hypotheses, which could either hide real effects or inflate significance for one group. Also, if each cluster is tested for significance and then further sub-tests are run within the cluster, you risk multiple comparisons inflation at multiple levels (between clusters and within clusters). A structured approach that accounts for each layer of testing is essential.

How can hierarchical or multi-level modeling help in multiple comparison scenarios?

When data naturally follows a multi-level structure (for example, repeated measurements within subjects, nested data such as students within classes, or gene expressions grouped by biological pathways), hierarchical models can borrow strength across levels. Instead of independently testing each hypothesis, these models handle group-level effects and individual-level variations together. One major advantage is that partial pooling in Bayesian hierarchical methods or mixed-effects models (in frequentist frameworks) can shrink effect estimates toward a group mean, thus reducing the variability in estimates and mitigating false positives. However, a subtle pitfall is the assumption about how random effects are distributed and whether the hierarchical structure truly captures the correlation among hypotheses. Mis-specified hierarchical structures or ignoring correlation across levels can undermine the validity of your inferences.

What if multiple tests show borderline significance after standard corrections?

If many tests show p-values that hover just around the threshold for significance, interpreting those results becomes tricky. A borderline p-value might reflect a true but small effect size, or it could be due to random fluctuations in the data. It is dangerous to sift through borderline results and selectively highlight ones that cross the threshold, as this practice can exacerbate publication bias. A subtle but common edge case is “p-hacking,” where analysts nudge borderline results below the significance threshold by adjusting the analysis pipeline. Another challenge is that borderline p-values often invite post-hoc subgroup explorations: once you find a near-significant result, you might be tempted to slice the data further, inadvertently creating even more hypotheses. In such scenarios, it’s better to pre-register your analysis, consider confidence intervals rather than just p-values, or use Bayesian approaches to quantify evidence.

In practical applications like A/B testing with continuous deployment, how do multiple comparisons issues arise?

In continuous A/B testing, one might run numerous experiments over time—each testing different features, user segments, or variations. Even though each experiment might use a straightforward t-test, the accumulation of tests can inflate the overall risk of false positives when you look at your entire experimentation program. An additional pitfall arises if results are monitored repeatedly in real time (i.e., optional stopping): each time an experiment’s partial results are checked, it effectively adds another “test,” thus inflating Type I error if you stop at the first sign of significance. This can lead to implementing features that seem beneficial but are not truly so. To mitigate these risks, techniques such as sequential testing (adjusting the alpha threshold at each interim analysis) or Bayesian adaptive experimentation are often used.

How do missing data or imputation methods interact with multiple testing procedures?

In large datasets with many hypotheses, missing data are common. Imputation can introduce uncertainty in the dataset that standard t-tests rarely incorporate explicitly. If you run many separate t-tests on imputed datasets, the p-values may not capture the extra variability caused by imputation. A subtle edge case is when data are not missing at random but are missing in a pattern correlated to the outcome. In that case, standard imputation methods might be biased, leading to systematically inflated or deflated test statistics. Combining multiple imputation (where you create several imputed versions of the dataset) with multiple comparison corrections can be challenging. You would need to pool results across all imputed sets while controlling for familywise or false discovery rates, which is computationally more demanding and prone to misinterpretation if done hastily.

Are there scenarios where it might be acceptable not to apply multiple comparisons corrections at all?

Some exploratory or hypothesis-generating research contexts allow for “fishing expeditions,” where the objective is to identify potentially interesting leads for future study rather than to confirm definitive effects. In such exploratory phases, strict control of the Type I error rate might be relaxed, acknowledging that many of the findings will need further validation. The risk is that you might present preliminary results as if they have confirmatory significance. This can lead to inflated claims in the literature or misinformed business decisions if the results are taken at face value. A best practice is to clearly label findings as exploratory and conduct confirmatory tests with formal corrections in follow-up analyses.