ML Case-study Interview Question: Optimizing E-commerce Authorization Buffers with Regression Discontinuity Design

Browse all the ML Case-Studies here.

Case-Study question

A fast-growing ecommerce platform provides different authorization buffer amounts on customers’ credit cards at checkout to accommodate post-checkout alterations and replacements. The current policy rounds the total up to the nearest 5 dollars once a base buffer percentage is applied. This often leads to sudden increases in the authorization amount for orders that fall just above a 5-dollar threshold. The platform suspects these spikes cause card declines and deter customers from placing orders. They want a data-driven approach to decide on the optimal buffer policy that balances reducing unpaid amounts and minimizing card declines. How would you proceed with your analysis, recommend a policy change, and validate that recommendation?

In-Depth Solution

Regression Discontinuity Approach

Use a regression discontinuity design (RDD) around each 5-dollar threshold. Orders that just cross a threshold get a higher authorization buffer, while orders just below it get a lower buffer. Arrival amounts just above or below the threshold are plausibly similar in user characteristics, aside from the higher authorization buffer.

Collect data on order total, authorization buffer, card decline rate, re-authorizations, final paid amounts, and user outcomes (e.g., completed orders). For each threshold, estimate the local average treatment effect (LATE) of the higher buffer on card declines, unpaid amounts, and final orders. The key assumption is that users do not manipulate the exact order amount to avoid crossing a threshold.

Trade-Off Formula

Examine incremental orders versus incremental cost. Compare the gains from lower card declines or more completed orders to the potential costs of unpaid amounts or re-authorization attempts. This trade-off can be framed as:

Here, guardrail is a monetary value assigned to the downside risk of an additional unpaid amount or re-authorization. If the expression is greater than zero, lowering the buffer near that threshold is beneficial, and vice versa.

Applying to the Buffer Policy

Analyze each threshold where the buffer jumps by 5 dollars. For those points, measure the discontinuity in card declines, cost offsets (unpaid amounts, re-authorization attempts), and final order completions. If the incremental orders exceed guardrail * (additional cost), then reducing the buffer is justified.

Run an A/B experiment where you reduce upward rounding at selected thresholds. Monitor any changes in card decline rates, order completion, and final paid amounts. If results align with the RDD findings and confirm that lowered buffers generate net gains, expand the new policy.

Implementation Example in Python

import pandas as pd
import statsmodels.formula.api as smf

# Suppose df has columns:
# 'distance_from_threshold', 'decline_rate', and 'above_threshold' (1 or 0)

# Focus on a narrow window around the threshold
df_narrow = df[df['distance_from_threshold'].abs() < 2.0]

# Run a regression to estimate the discontinuity
model = smf.ols('decline_rate ~ above_threshold + distance_from_threshold',
                data=df_narrow).fit()
print(model.summary())

This simple outline regresses decline_rate on the indicator for crossing the threshold and the running variable (distance_from_threshold). The coefficient on above_threshold estimates the jump at the threshold.

Scaling Beyond Local Effects

Local effects near thresholds may not extrapolate to all users. Validate any global policy shift with an expanded A/B test or multi-armed bandit approach that tests multiple buffer levels. Continue refining thresholds based on cost trade-offs and user segments, since some customers may tolerate higher holds.

Possible Follow-Up Questions

How do you test the core assumption of the RDD (no manipulation at the threshold)?

Inspect the density of orders around the threshold. If there is a suspicious dip or spike exactly at the cutoff, suspect manipulation. Show a histogram of order amounts near each 5-dollar threshold. If the distribution is smooth, the core assumption likely holds.

How do you reconcile LATE estimates with global policy changes?

RDD provides a local effect for users clustered near the threshold. These users might differ from users with orders far from any threshold. Validate the final global policy by running A/B tests on the entire user base. Compare LATE estimates to average treatment effects (ATE). If they align, you can be more confident in generalizing.

What if you do not find a significant discontinuity but still see a business need to reduce friction?

Try a direct experiment. If no natural threshold creates a suitable discontinuity, design a randomized controlled trial that tests different buffer levels. Estimate cost trade-offs and measure changes in final orders. Use multi-armed bandit methods if you want to dynamically allocate buffer levels to promising variants.

How do you decide the guardrail in practice?

Collaborate with Finance and Business teams to compute long-term value per order. Weigh that value against risk from unpaid amounts or secondary holds. Choose a guardrail that reflects realistic opportunity costs and potential damage from declines. This ensures your trade-off equation aligns with business goals.

Could you tailor the buffer policy for different user segments?

Segment users based on payment history, order sizes, or purchase frequencies. For low-risk users, smaller buffers might boost order conversions. For riskier profiles, higher buffers might be safer. Adaptive policies can be introduced, then A/B tested or validated with an RDD approach if natural thresholds exist (e.g., high-frequency users might cross certain thresholds more often).

How would you monitor post-deployment performance?

Track metrics such as new card declines, re-authorization attempts, unpaid amounts, average order values, and user churn. Compare to the pre-deployment baseline. Use holdout groups if possible to confirm the effect. Keep iterating if you see unintended shifts in other metrics or user behaviors.

---