ML Interview Q Series: Controlling False Positives in Multiple T-Tests via Bonferroni Correction

Browse all the Probability Interview Questions here.

Suppose you are running a large number of experiments, each involving a t-test. What key considerations should be taken into account when interpreting the results?

Short Compact solution

When many t-tests are run separately, the likelihood of seeing a few tests show significance purely by random chance rises. For example, if you carry out 100 t-tests at a significance level of 0.05, you can expect about five tests to appear significant even if none actually are. This leads to an inflated probability of a Type I error (incorrectly rejecting a null hypothesis). A common way to mitigate this issue is to use the Bonferroni correction, which involves dividing the significance threshold by the total number of tests (for instance, using 0.05 / 100 = 0.0005 instead of 0.05). Although this method helps reduce false positives, it may increase the likelihood of Type II errors, where real differences might be missed due to a more stringent threshold.

Comprehensive Explanation

Multiple Comparisons Concern

When you carry out a t-test on a single hypothesis using a standard significance level, such as 0.05, you are typically comfortable with up to a 5% risk of committing a Type I error. However, if you run dozens or even hundreds of these tests on different hypotheses, the chance of obtaining at least one seemingly significant outcome just by luck becomes quite large. This is often referred to as the multiple comparisons or multiple testing problem. It leads to inflated false-positive rates because even random noise can cross the threshold in at least a few tests.

Type I and Type II Errors

The trade-off is that the threshold for each test becomes very stringent. When mm is large, the Bonferroni method can become overly conservative. This makes it much harder to identify true effects, thereby increasing the chance of a Type II error. In settings where only a few hypotheses are likely to be true, Bonferroni can be beneficial. However, in large-scale testing scenarios (for example, genomics, high-dimensional data analysis), Bonferroni may be too strict.

Alternative Multiple Comparison Methods

There are several other techniques designed to manage the multiple comparisons problem more flexibly while keeping a tight handle on false positives. One commonly used approach is the Benjamini–Hochberg procedure, which focuses on controlling the False Discovery Rate (FDR) rather than the Family-Wise Error Rate (FWER) targeted by Bonferroni. By controlling the proportion of false discoveries among all declared findings, the Benjamini–Hochberg procedure typically provides more statistical power than Bonferroni when the number of tests is large, albeit at the cost of allowing some acceptable proportion of false positives.

In more advanced settings, researchers might use permutation-based techniques or hierarchical testing methods to further refine their control of false positives while preserving the ability to detect true signals.

Practical Workflow in Python

Many Python libraries, including statsmodels, provide built-in functions for multiple testing corrections. A typical workflow might involve computing p-values for all tests, then using a function like “statsmodels.stats.multitest.multipletests” to correct for multiple comparisons. An example in Python:

import numpy as np
from statsmodels.stats.multitest import multipletests

# Suppose we have an array of p-values from running multiple tests
p_values = np.array([0.04, 0.001, 0.03, 0.055, 0.002, 0.6, 0.07, 0.001])

# Apply Bonferroni correction
reject, pvals_corrected, alpha_corrected, _ = multipletests(p_values, alpha=0.05, method='bonferroni')

print("Original p-values:", p_values)
print("Adjusted p-values:", pvals_corrected)
print("Rejected hypotheses:", reject)
print("Adjusted alpha:", alpha_corrected)

This approach automatically adjusts the significance level for you (in this case, via Bonferroni correction), helps you decide which hypotheses to reject, and provides corrected p-values.

Behavior in Large-Scale Testing

In scenarios involving hundreds or thousands of simultaneous tests, it is often unrealistic to require every result to pass a Bonferroni-corrected threshold to declare it significant because the effective alpha becomes extremely small. This leads to a large number of missed discoveries. Consequently, many practitioners use FDR-based methods, especially when exploring multiple hypotheses in areas like bioinformatics, marketing experiments, or A/B testing for large-scale feature rollouts.

Concerns About Test Independence

All these corrections typically assume tests are independent or not strongly dependent. If you have correlated hypotheses, the standard corrections can be either too lenient or too harsh, depending on the correlation structure. Some procedures do provide ways to handle such dependencies. In practice, it is crucial to understand how the tests are correlated before settling on a specific correction method.

Real-World Example

Imagine a company running numerous A/B tests on a platform to evaluate different new features (varying color schemes, button placements, recommended items, etc.). Without controlling for multiple comparisons, the platform might record many spurious “winners” even if none of the new features actually improve user engagement. Conversely, if they apply a very strict Bonferroni correction to hundreds of tests, they might end up concluding that none of the new features provide any benefit, even if a few of them truly do. Balancing the likelihood of false positives with the need to detect genuinely impactful changes is a critical aspect of designing such large-scale experiments.

Potential Follow-up Questions

How does the Bonferroni correction compare to controlling the False Discovery Rate (FDR) with the Benjamini–Hochberg method?

Bonferroni is focused on controlling the Family-Wise Error Rate (FWER), ensuring that the probability of at least one false positive remains below the chosen significance level. This is often overly conservative for large datasets. The Benjamini–Hochberg (BH) method controls the False Discovery Rate, which is the expected proportion of false positives among all declared positives. Because of this different target, BH is generally more powerful than Bonferroni when many tests are performed, while permitting a controlled fraction of false positives.

What are potential pitfalls of using Bonferroni if many of the tests are correlated?

Bonferroni assumes that either the tests are independent or at least not so strongly dependent as to break its worst-case bounds. If your hypotheses are highly correlated, Bonferroni can become far too conservative, leading to an extremely small alpha for each test and a higher rate of false negatives. An alternative approach is to use resampling or permutation tests that can account for the correlation structure, or use specialized multiple comparison methods designed for dependent tests.

Why might it still be necessary to consider statistical power when applying multiple comparison corrections?

Statistical power refers to the probability of correctly rejecting the null hypothesis when it is false. By lowering the alpha threshold through Bonferroni or other conservative corrections, you decrease the likelihood of detecting true effects. Especially in scenarios where the signal is small or moderate, a more stringent threshold can cause real findings to be missed. As a result, you need to ensure your study or experiment is designed with enough sample size and strong enough effects so that even with conservative corrections, you still have adequate power to detect meaningful differences.

How can you decide when Bonferroni is appropriate versus more sophisticated corrections?

Bonferroni is intuitive and a good protective measure if you have only a few tests or very high stakes for Type I errors. However, for exploratory experiments involving large-scale testing—where missing important signals might be as problematic as falsely claiming significance—methods like Benjamini–Hochberg are often preferred to balance controlling false positives with retaining power.

What happens if you want to run a pre-planned set of contrasts among different groups?

If you have a small set of planned contrasts, Bonferroni or other adjustments might be suitable because you already have a precise idea about which comparisons matter. However, if you decide post hoc to look at many different possible comparisons, you should apply a multiple comparisons correction that accounts for this broader scope.

Do large-scale experiments always require strictly adjusting p-values?

Not always. In some cases, investigators might prefer to risk a higher overall false-positive rate if it means they can discover novel findings worth exploring in more rigorous follow-up experiments. As long as you acknowledge the trade-off and plan a second stage of validation, you can sometimes use more liberal thresholds. In production systems, however, stricter controls are usually needed to prevent wrong decisions based on random noise.

Below are additional follow-up questions

If you have repeated measures for each subject (i.e., multiple measurements from the same individual), how does that affect multiple comparison procedures?

Repeated measures can introduce within-subject correlations, which violate the assumption that each test is independent. Traditional corrections like Bonferroni treat each test as isolated, potentially overcompensating when the data points are not truly independent. In this context, a few considerations come into play:

1. Within-Subject Correlation Each subject’s repeated measurements tend to be more alike than measurements from different subjects. If you naively apply Bonferroni, you do not account for this correlation, leading to an overly strict threshold that can produce more false negatives.

2. Mixed Effects Models Instead of running separate t-tests for each time point or condition, a common practice is to use mixed effects models that incorporate both fixed effects (e.g., treatment groups, time) and random effects (e.g., per-subject random intercepts). By capturing the within-subject correlation structure, mixed models can give more accurate estimates and p-values. If you still want to correct for multiple tests within such a framework, you can adjust the resulting p-values or confidence intervals, but the model-based approach might reduce the number of comparisons.

3. Repeated Measures ANOVA For comparing multiple time points or conditions for the same subjects, repeated measures ANOVA is often used. If you find an overall significant effect, you can conduct post-hoc comparisons with methods like the Tukey HSD or other corrections that better account for repeated observations.

4. Pitfall: Overly Conservative Adjustments Applying straightforward multiple comparison corrections can become so conservative that you may not detect real differences. Instead, you might resort to specialized tests or hierarchical models that handle repeated measures more gracefully.

Are there scenarios where it might be beneficial to skip multiple comparison corrections entirely?

While it might be unusual to skip corrections completely, there are scenarios where researchers may choose to accept a higher risk of Type I errors for practical or strategic reasons:

1. Exploratory Research In early-stage or exploratory experiments where the primary goal is to generate hypotheses, strict control of false positives may be secondary. Researchers might decide to allow more false alarms, provided these findings are followed by confirmatory studies with stricter controls.

2. High Cost of Missed Discoveries In contexts where missing a real effect is significantly more detrimental than a false positive (e.g., important medical breakthroughs, major industrial innovation), some might prioritize reducing Type II errors. They then accept a higher chance of false positives, subject to subsequent verification.

3. Multiple Testing Not Strictly 'Family-Wise' If each test addresses a distinct question with no logical grouping into a single family (e.g., separate business experiments with no overlap in user populations), a global correction might be overly conservative. Researchers could justify analyzing each test independently if they truly represent unrelated hypotheses.

4. Pitfall: Overinterpretation Even if you skip formal multiple testing corrections, you must be transparent that your reported p-values do not account for the inflation of false positives. There is a real risk that others might interpret your findings as more robust than they truly are.

How do effect sizes factor into multiple comparisons, and can they mitigate some pitfalls?

When you only rely on p-values, you risk focusing exclusively on statistical significance without considering practical importance. Effect sizes can add context:

1. Quantifying Practical Significance An effect size (e.g., Cohen’s d for mean differences or partial eta-squared for ANOVA) helps determine whether a statistically significant difference is large enough to matter in real-world settings. Even if a test remains significant after Bonferroni or another correction, it might have a negligible effect size, suggesting limited practical value.

2. Guiding Research Priorities After correcting for multiple comparisons, a handful of tests might remain significant. Comparing their effect sizes can help you prioritize which differences to investigate further. Tests with both low p-values and large effect sizes are more likely to be of genuine interest.

3. Power and Sample Size In experiments with multiple hypotheses, a larger sample might be required not only to pass stricter thresholds but also to reliably estimate moderate to small effect sizes. Balancing Type I error control with the detection of meaningful effect sizes is often a resource allocation decision (time, money, participants, etc.).

4. Pitfall: Large Sample Bias Large samples can detect even tiny, irrelevant effects as “highly significant.” Reporting both the p-value and effect size ensures the interpretation does not overstate the practical value of small but statistically significant differences.

In large-scale studies, how do you handle missing data across many simultaneous tests?

Missing data can occur in different ways (missing completely at random, missing at random, missing not at random) and can complicate multiple comparisons:

1. Imputation Strategies Simple methods like mean or median imputation across each test can bias the results when the pattern of missingness is not random. More advanced imputation (e.g., multiple imputation) may better preserve the distribution of the missing values.

2. Complete-Case Analysis Sometimes you only include subjects or samples without missing values in any test. While this can simplify analysis, you risk dropping large portions of the data, leading to a decrease in power, especially detrimental when adjusting for multiple comparisons.

3. Hierarchical Modeling Bayesian or frequentist hierarchical models may simultaneously estimate missing data and handle multiple comparisons. By jointly modeling the entire dataset, you can shrink estimates for each test toward a global mean or pattern in a principled way.

4. Pitfall: Varied Missingness Across Tests With hundreds of tests, certain tests might have more missing data than others. If you treat them all equally, you may apply the same correction method but end up with tests that have distinct statistical power or reliability. This discrepancy complicates interpreting results and comparing them.

How do you approach multiple comparisons when running non-parametric tests for distributions that deviate from normality?

Non-parametric tests often come into play for data that violate normality, or when sample sizes are small:

1. Multiple Non-Parametric Hypothesis Testing The conceptual challenge is the same: each test might have its own Type I error rate. Methods such as Bonferroni still apply to the resulting p-values. A Kruskal–Wallis test can replace ANOVA, and a Wilcoxon test can replace t-tests. You can adjust those non-parametric test p-values in the same manner.

2. Permutation Methods For non-parametric data, permutation-based methods are common. You randomly shuffle labels to approximate the distribution of test statistics under the null. By doing this for all tests collectively, you can derive an empirical distribution that already accounts for multiple comparisons. This approach can be more powerful if your data are correlated or exhibit specific distributional shapes.

3. Pitfall: Reduced Statistical Power Non-parametric tests can be less powerful than parametric counterparts if the data are close to normal. Applying them in conjunction with multiple testing corrections further lowers power. You need to ensure you have enough data to overcome this double penalty.

4. False Discovery Rate Adjustments FDR-based procedures still apply. The primary difference is that you feed the procedure with p-values from non-parametric tests instead of standard t-tests.

When you continually monitor a single hypothesis over time (interim analyses), do you need multiple comparison corrections?

Even though it is one hypothesis, repeated “peeking” at the data over time effectively introduces multiple comparisons:

1. Alpha Spending or Group Sequential Methods Methods like alpha-spending functions allocate parts of the overall Type I error budget across multiple interim analyses. The significance threshold is adjusted at each interim look to preserve the overall alpha level.

2. Pitfall: Overstating Early Significance Stopping the experiment as soon as a result crosses a certain p-value threshold inflates Type I error if not adjusted. This can lead to prematurely claiming a significant effect that might not hold up in the final analysis.

3. Data-Dependent Stopping If you base your decision on partial results before your predefined sample size is reached, the effect might appear large early on but regress to the mean with more data. Properly planned interim analyses with sequential boundaries or Bayesian updating can mitigate this risk.

How does the definition of a “family” of hypotheses affect your choice of correction?

The notion of a “family” of tests is somewhat subjective and depends on study design:

1. Pre-Specified Family If you clearly define a set of tests all aimed at the same overarching question, they form a “family,” and family-wise error rate control (like Bonferroni) is standard. For example, testing multiple endpoints in a clinical trial often calls for controlling the FWER to protect against claiming a treatment effect if none exist.

2. Exploratory vs. Confirmatory In exploratory settings, you might have broad sets of tests you were not sure you would run initially. They might or might not form a cohesive family. A less stringent correction approach (like FDR) may be more appropriate.

3. Pitfall: Arbitrary Grouping If you artificially break a large set of tests into smaller families, you may incorrectly reduce the penalty for multiple comparisons. Conversely, if you lump all tests into one very large family, you may over-penalize some genuinely separate groups of hypotheses.

How should you handle multiple comparisons when comparing several machine learning models in cross-validation?

Comparing multiple algorithms across the same cross-validation splits or datasets poses a multiple testing scenario:

1. Nemenyi Test or Friedman Test For multiple model comparisons, non-parametric rank-based methods such as Friedman’s test followed by a Nemenyi post-hoc comparison can be used. They consider the correlated nature of cross-validation folds.

2. Pitfall: Overlapping Training Sets If your cross-validation folds share data in a way that leads to correlated performance estimates, pairwise t-tests with naive Bonferroni might not be appropriate. In contrast, specialized tests like the Dietterich 5x2cv test or the Alpaydin Combined 5x2cv procedure address correlation in cross-validation.

3. Family-Wise vs. FDR If you test many models, controlling the family-wise error rate might be overly strict. Some prefer an FDR-based approach—only a small fraction of declared “best” models can be false positives. But FDR is less common for model comparison than for, say, feature selection. It really depends on your goals.

Can Bayesian methods circumvent the multiple comparisons problem, and if so, how?

In a Bayesian framework, rather than controlling p-values, you focus on posterior probabilities of parameters or hypotheses:

1. Partial Pooling in Hierarchical Bayesian Models When estimating multiple effects simultaneously (e.g., multiple group differences), partial pooling naturally shrinks the posterior estimates toward an overall mean. This shrinkage can serve a similar role to multiple comparison corrections, reducing the chance that random noise in any one test stands out as a large effect.

2. Bayes Factors In place of p-values, you can compute Bayes factors comparing the null hypothesis to the alternative. However, running many Bayes factor computations can still yield spurious signals if you go on a “fishing expedition.” Some Bayesian statisticians thus suggest hierarchical approaches that let you define priors that shrink improbable hypotheses.

3. Pitfall: Informative Priors If you have poorly chosen or overly vague priors, you might not get enough shrinkage to counteract multiple testing inflation. Bayesian analysis does not automatically solve multiple comparisons—it requires careful modeling.

4. Interpretation Shifts Instead of focusing on whether each result crosses a threshold (like p < 0.05), you might interpret posterior intervals or probabilities. A well-designed hierarchical Bayesian model can mitigate the false positive risk by pooling information across tests, but it is not a simple “turnkey solution.”