ML Interview Q Series: Unexpected A/B Test Results: Why a $10 Incentive Lowered User Response Rates

Browse all the Probability Interview Questions here.

Suppose you run an experiment to see how giving a $10 incentive affects the probability that users respond, and you find that the group offered $10 only responds 30% of the time, while the group getting no incentive responds 50% of the time. Why might this occur and how could you refine the experiment for better accuracy?

Comprehensive Explanation

Potential Reasons for Lower Response in the Incentivized Group

There can be several explanations for why a group offered $10 has a 30% response rate, which is paradoxically lower than the control group’s 50% rate.

Signaling and Suspicion Effect If users perceive the offer of $10 as “too good to be true,” they might become skeptical. This suspicion can lower overall engagement if people suspect spam, phishing, or hidden conditions attached to receiving the reward.

Differences in the Sample Populations It is possible the randomization process did not truly produce comparable groups. For instance, the treatment group might unintentionally contain users who are generally less inclined to respond, or it might have been exposed to a different user interface or messaging flow, leading to a lower response rate.

Inadequate Framing of the Incentive The wording or format of presenting the $10 incentive can matter. If the $10 reward is not clearly explained or if it appears as an intrusive pop-up, the reaction might be negative, causing drop-offs.

Timing and Communication Issues If the incentive was offered at a point in the user experience where participants are least motivated, the reward might fail to outweigh the inconvenience. Alternatively, the control group’s request might have appeared at a more favorable time or with simpler instructions.

Insufficient Sample Size or Randomization Sometimes the difference arises simply due to experimental noise and sample imbalance. If the number of participants in each group is too small, chance alone can produce unexpected outcomes.

Key Statistical Concepts for Understanding Proportions

When dealing with response rates, you typically look at the proportion of successes (in this case, responses) in each group. One basic measure is the sample proportion:

Where x is the number of participants who responded, and n is the total number of participants in the group. For the control group, you might have x_control and n_control, while for the treatment group, x_treatment and n_treatment.

To compare two proportions, an analyst might look at the difference:

If you want to assess statistical significance, you can perform a two-proportion z-test, which involves computing a standard error of the difference between two proportions. The standard error can be approximated by:

Where hat{p}_{pooled} is the combined proportion of responses across both groups. This helps determine whether the observed difference could be due to chance or if it suggests a true underlying effect.

Strategies to Improve Experimental Design

Ensure Proper Randomization Use robust randomization methods to assign participants to treatment or control groups. Double-check that participants are balanced across relevant demographics, user behavior patterns, and other factors that might influence response likelihood.

Pilot Different Reward Levels and Messaging Test different ways of presenting the $10. For example, an early pilot study could randomly vary the message wording and see if a clearer explanation of the reward’s legitimacy mitigates suspicion.

Validate Timing and Delivery Mechanisms Examine how, when, and where the incentive is introduced. If the $10 pop-up appears at a moment when users are focused on other tasks, it might reduce participation rather than encourage it. Adjust the experiment so that the reward is offered at a logical point in user flow.

Split Test for UI or UX Differences Confirm that both groups have an identical user experience, aside from the reward. Even small design differences (e.g., layout, color scheme) can skew response rates independently of the financial incentive.

Check Sufficient Sample Sizes Calculate needed sample sizes in advance. If your experiment is too small, random noise could overshadow the true effect. Larger sample sizes lower the variance of the estimates, making it easier to reliably detect genuine differences.

Pre-Testing and Qualitative Feedback Before running a large-scale A/B test, do smaller usability sessions or qualitative tests to identify potential confusion or trust issues related to the incentive. Gather direct feedback on how participants perceive the reward.

Example Code Snippet for Analyzing Difference in Proportions

import statsmodels.api as sm

# Hypothetical data
# control_responses: number of users who responded in the control group
# control_n: total number of users in the control group
# treatment_responses: number of users who responded in the treatment group
# treatment_n: total number of users in the treatment group

control_responses = 500
control_n = 1000
treatment_responses = 300
treatment_n = 1000

# Calculate observed proportions
control_prop = control_responses / control_n
treatment_prop = treatment_responses / treatment_n

# Perform two-proportion z-test
count = [treatment_responses, control_responses]
nobs = [treatment_n, control_n]
z_stat, p_value = sm.stats.proportions_ztest(count, nobs)

print("Treatment proportion:", treatment_prop)
print("Control proportion:", control_prop)
print("Z-statistic:", z_stat)
print("P-value:", p_value)

In this snippet, z_stat is the test statistic for the difference in proportions, while p_value indicates whether the difference is statistically significant, given a significance threshold like 0.05.

Possible Follow-up Questions

How could selection bias manifest in this scenario, and how do we mitigate it?

Selection bias may occur if individuals in the $10 incentive group differ systematically from those in the control group. For instance, maybe the treatment group automatically includes certain high-value or more engaged users, or conversely a subset of users who are typically less trusting. This mismatch can lead to misleading conclusions about the effect of the reward on response rates. To mitigate selection bias:

Use purely random assignment so each user has the same probability of joining the treatment or control. Stratify randomization by important factors (e.g., region, user activity level) if you suspect these factors influence response. Monitor group characteristics after assignment to ensure that they remain comparable.

Why might an overall negative effect appear even if the incentive is beneficial in principle?

There might be confounding factors that overshadow the incentive’s true impact. Users may dislike the sign-up process required to claim the $10, or the message might be poorly timed or worded, leading to drop-offs. Alternatively, the $10 might be insufficient for the perceived effort or time investment, or it might raise concern about sharing personal information. All of these issues can accumulate to produce a net negative response rate, even though, in principle, a financial incentive could be motivating.

If we see the same result after re-running the experiment with better design, what could be the interpretation?

If the experiment consistently shows a lower response rate with the incentive even after careful controls, it suggests that the incentive is genuinely detrimental under the conditions tested. The negative effect might be rooted in user psychology (e.g., “If they’re paying me, this must be spam.”), communication issues, or cultural norms around receiving payments. The interpretation would be that the particular monetary offer or how it is presented simply does not resonate positively with the user base.

How can we handle the possibility of novelty effects over time?

Novelty effects occur when users respond differently simply because something is new or unexpected. In a longer-running study, the effect of an incentive can change as users become accustomed to it. One way to manage novelty effects is by running your experiment for a sufficient duration so that initial excitement or suspicion stabilizes. Then you can observe sustained behavior rather than short-term bursts or drops in engagement that might misrepresent true behavior over time.

Below are additional follow-up questions

Could running multiple experiments simultaneously dilute or distort the results in this reward-based experiment?

If other interventions run in parallel (for example, testing a new interface design or a different communication channel), those simultaneous changes may confound your reward-based experiment. In such scenarios, users in either the treatment or control group might also be part of another experiment, creating interactions that mask or exaggerate the effects of the $10 incentive. One approach to mitigate this is to maintain mutually exclusive user buckets for each experiment. By ensuring that no user belongs to more than one test at a time, you can isolate the effect of the $10 reward without interference from other concurrent interventions.

Nevertheless, there are practical realities (like time constraints and large user populations) that often necessitate parallel testing. In those cases, factoring in each user’s exposure to other experiments becomes essential. Analysts can incorporate variables indicating participation in other tests when performing a regression or logistic model, thereby controlling for confounding factors.

In what ways could users’ prior experiences influence the experiment outcomes, and how can you account for that?

Past experiences with incentives, surveys, or similar studies can greatly color user responses. For instance, users who have previously encountered spammy or misleading “reward” offers may be more reluctant to trust a $10 incentive. Conversely, some users who have had positive experiences might respond quickly, inflating response rates.

One strategy to deal with these differences is to track each user’s history of interacting with similar programs or promotions. If your system identifies which users have participated in past reward-based studies, you can randomize treatment and control within different segments. You might, for example, stratify by “never participated in a rewards experiment before” vs. “previously participated in a rewards experiment,” ensuring that the distribution of these user types is balanced across treatment and control groups.

How might limited-time offers or urgency messaging affect user perception of the reward?

If the $10 incentive is framed as “limited time only” or accompanied by urgent language, some users might respond more quickly. Others, however, may view urgency tactics as suspicious or pushy. This can skew results in unpredictable ways. Urgent offers sometimes boost response rates among certain cohorts while generating skepticism in others, leading to overall variance in outcomes.

A best practice is to test urgency versus non-urgency in separate experiments or sub-treatments. By doing so, you isolate whether the core monetary reward drives the response or whether the urgency messaging is the primary contributor to changes in user behavior. You can also collect qualitative feedback (like short survey responses) that reveals user attitudes about the urgency aspect specifically.

What if high-value users feel alienated by a small reward that seems trivial for their usual spending or engagement level?

Users who regularly transact large sums or have higher lifetime value (LTV) to the platform might interpret a $10 incentive as minimal or even insulting, causing them to ignore the offer. Conversely, newer or lower-engagement users might view $10 as more appealing. If the experiment does not segment users based on their typical spending or engagement level, the impact of the $10 might be diluted—or, in some cases, reversed—by the presence of high-value users who find the amount inconsequential.

A refinement here is to personalize incentive levels based on user LTV or past behavior. You might offer a larger reward to high-value segments to ensure the amount is meaningful. Another approach is to track redemption or response rates by user tier (e.g., new vs. loyal users) and analyze how each segment responds differently. This segmentation sheds light on whether the negative or positive effect is uniform or concentrated in a specific user subset.

How do you address potential gaming or fraud if users attempt to exploit the $10 reward?

Any monetary incentive brings the risk that some individuals might exploit the system—creating multiple accounts or artificially inflating responses. If that occurs, the experiment’s metrics become skewed by fraudulent activities rather than genuine user intent.

Mitigations include implementing strict identity checks, rate-limiting how often a user can receive the reward, or requiring a valid payment method or verified email before the user qualifies for the $10. You can also monitor for anomalous patterns (e.g., a sudden spike in new account sign-ups shortly after the incentive is launched). If suspicious behavior is detected, exclude those data points from your analysis or flag them for manual review. Adding friction—like requiring a short questionnaire or additional account verification—can discourage fraudulent sign-ups, although too much friction may also reduce legitimate user participation.

Could there be cultural or regional nuances affecting the perception of a $10 incentive?

Some cultures may perceive direct cash rewards as more suspicious or less desirable than discounts, loyalty points, or charitable donations. In other regions, a direct cash incentive might be more acceptable and even preferred. If your user base is global or spans multiple regions, the $10 incentive might resonate differently across various demographic groups.

To address this, run geographically segmented experiments. Evaluate how the reward performs in different cultural contexts. You might discover that a voucher or in-app credit is more effective in certain regions, while direct cash is more effective in others. If you choose a one-size-fits-all approach for a global user population, you risk seeing mixed or contradictory results driven by cultural nuances rather than the fundamental appeal of the incentive.

What if you need to generalize these results to a completely different user population or a new product feature?

Generalizing from one set of users or one product environment to another can be problematic if the factors driving response rates differ significantly. For instance, an e-commerce setting where users are accustomed to coupons and promotions might show a positive effect, whereas a professional networking site’s users might find monetary incentives unusual or distracting. Extrapolating success (or failure) from one platform to another could lead to misguided strategic decisions.

To mitigate this external validity problem, replicate the experiment in multiple contexts. By running smaller-scale tests across different product lines, markets, or demographic segments, you build a body of evidence about whether the $10 incentive consistently yields a certain response pattern or if its effectiveness depends heavily on context-specific factors. This multi-context approach prevents overfitting your conclusions to a single user population.

How can you ensure that the presence of the reward doesn’t overshadow the content or purpose of the request itself?

When an incentive is introduced, users might focus primarily on the reward rather than the substance of what you’re asking them to do. This can lead to superficial participation if users are simply aiming to collect the money without genuinely engaging with the survey, feedback, or whatever request is at hand.

In such cases, measuring user engagement quality is critical. You could include a brief comprehension check or require minimal but meaningful interaction that demonstrates the user understood the request. Tracking metrics like “time spent reading the instructions” or “thoughtful completion of free-text fields” can reveal whether the $10 is genuinely motivating quality participation or merely enticing users to click through quickly. If you notice a drop in the quality of responses, it might be more effective to redesign the incentive to reward thoughtful completion rather than just a binary “submit” action.

How can platform trust and brand reputation play a role in response rates for the $10 incentive?

If the platform already enjoys strong trust and good reputation, a $10 reward might be taken at face value, prompting higher participation. However, on a platform with weaker brand reputation, an out-of-the-blue cash incentive might trigger suspicion. Users might fear scams or privacy breaches, resulting in a lower response rate than the control group.

To mitigate this, consider gradually introducing the incentive or coupling it with a transparent brand message, such as: “This reward is a way to thank you for your time. No hidden terms, guaranteed.” Additionally, giving examples or testimonials from real users who have received and benefited from the incentive in the past can alleviate skepticism. If brand trust is a known issue, you may want to do a trust-building campaign first before launching a monetary-based experiment.

Could the design of the control group inadvertently nudge people to respond more frequently?

Sometimes the control group’s messaging or offer is simpler and more straightforward, thus generating a higher response rate. Even without an actual reward, the minimal friction or the clarity of instructions in the control version might prove more compelling. In contrast, any added steps or disclaimers required for the $10 reward could drive people away.

A technique to detect this is to conduct a small test: present the exact same messaging to both groups except remove the mention of money in the control group. If the control group still outperforms, it may indicate that your $10 mention actually complicates the user experience. Reducing friction—such as by auto-enrolling users or simplifying terms—might then help the treatment group catch up or exceed the control’s performance.