ML Case-study Interview Question: Scoring & Orchestrating Multi-Strategy Push Notifications for Personalized E-commerce Engagement.
Browse all the ML Case-Studies here.
Case-Study question
You are working for a large e-commerce and payment platform that wants to boost user engagement through targeted push notifications. The product team is eager to show users relevant discounts and store suggestions without flooding them. Multiple strategies exist, each with its own method of selecting users and content. You must design a system that selects the best message for each user from among all possible strategies. How would you build a solution that identifies, scores, and reconciles these possible notifications, then sends the most relevant one to each user?
Your solution must address:
How you would structure the orchestration so multiple strategies compete without overwhelming users.
How you would adapt your approach when certain notifications outperform others.
How you would integrate a technique that detects user behavior patterns (e.g., interest in one store category leading to a purchase in a related category).
How you would link data from online browsing to nearby physical stores that might interest the user based on prior online activity.
Which metrics you would use to measure success.
Detailed Solution
Overall System Structure
Propose a modular pipeline with the following components: a Target step to define which stores, items, or offers should be notified. An Audience step to find all users who might be interested. A Filter step to remove invalid pairs (users already contacted or not meeting eligibility). A Score step to pick the single best store (or best offer) per user under each strategy. A Conciliation step to select which strategy wins for each user. Then send the chosen notification at an optimal time.
Notification Strategies
Some strategies focus on specific purchase patterns. Others use data from the user’s online search behavior to recommend nearby physical stores.
Market Basket Affinity
Identify pairs of merchant categories that commonly follow one another in a short time frame. Define a ratio that compares the probability of a user buying from MCC2 given a purchase in MCC1, against the baseline probability of MCC2. A simple approach:
MCC1 is the category of the first purchase. MCC2 is the category of the potential second purchase. P(MCC2 | MCC1) is the probability that a user buys from MCC2 soon after MCC1. P(MCC2) is the probability of a user buying from MCC2 in general. When the ratio exceeds 1, a follow-up purchase in MCC2 is more likely than average. Use a threshold to decide which MCC2 is most relevant.
When a user purchases in MCC1, prompt them about stores in MCC2 categories. Keep it selective to avoid spam.
From Online to Offline
Bridge the user’s online browsing data to physical stores offering similar or identical items. Identify whether a user viewed or added to cart certain products but did not complete the purchase. Tie these product categories to local stores. Estimate a “frequent location” from historical geodata when available. Offer the user a physical store that sells that product category at a discount. If the item is a high-interest product, prompt them to visit a nearby store.
Orchestration and Scoring
Each strategy yields user-store pairs. Remove duplicates for users contacted too recently or not eligible. Assign a score for each pair. The scoring approach can be a heuristic (e.g., user distance or discount amount) or a predictive model (e.g., user click probability). Select the top-scoring store per user within each strategy. Then run a conciliation step across strategies to choose the single best candidate.
Sending the Notification
Send the push only at a time the user is likely to engage. Personalize the notification text based on the strategy. Insert store name, category, discount details, or a direct link to encourage action.
Measuring Effectiveness
Track open rate or click-to-open rate. Compare new strategies against a baseline to confirm improved engagement.
Example Python Snippet
import pandas as pd
def select_best_pairs(user_store_df, score_column="score"):
# user_store_df has columns: user_id, store_id, strategy, score
# Sort by user_id then score descending
user_store_df = user_store_df.sort_values(["user_id", score_column], ascending=[True, False])
# Select top row per user_id-strategy
top_by_strategy = user_store_df.groupby(["user_id", "strategy"], as_index=False).head(1)
# Re-score or re-rank if needed, then pick best strategy for each user
# For simplicity, pick the highest score among strategies
top_by_strategy = top_by_strategy.sort_values(["user_id", score_column], ascending=[True, False])
final_df = top_by_strategy.groupby("user_id", as_index=False).head(1)
return final_df
# Example usage
data = [
{"user_id":1, "store_id":101, "strategy":"basket", "score":0.85},
{"user_id":1, "store_id":202, "strategy":"online_offline", "score":0.77},
{"user_id":2, "store_id":303, "strategy":"basket", "score":0.90},
{"user_id":2, "store_id":404, "strategy":"online_offline", "score":0.95},
]
df = pd.DataFrame(data)
best_for_each_user = select_best_pairs(df)
print(best_for_each_user)
This snippet picks the best store per user across strategies. The approach can be extended with more complex ranking.
How do you handle data drift in the models?
Adapt your models over time. Retrain on recent user interactions. Track engagement metrics to see if predictions degrade. If open rates drop unexpectedly, investigate external factors, such as changes in user habits or external events that shift purchasing behavior. Maintain a retraining pipeline that automatically refreshes your data and model parameters on a schedule.
Why not use traditional conditional probability for the second purchase?
The business found that many users do not make a second purchase soon after the first. Traditional P(MCC2 | MCC1) / P(MCC2) might overestimate the likelihood of sequential purchases. Instead, it is better to compare P(MCC2 | MCC1) against the baseline within a population that actually made a second purchase, then measure that ratio. This highlights pairs of categories with strong sequential affinity.
How do you ensure the push notifications are well orchestrated?
Define a global pipeline that merges all strategy results before sending any push. Include a consolidation mechanism that ensures only one notification per user in a given period. Score each candidate to find the top choice. Reconcile across all strategies, so the user receives the most relevant push. Keep a log of recent notifications to avoid sending multiple alerts too frequently.
Which metrics do you use to measure success, and why?
Open rate or click-to-open rate is straightforward. If the user actually sees and clicks, it shows relevance. Conversion rate can be the next step: how many notifications lead to a transaction or desired action. Retention and user satisfaction scores matter when you want long-term acceptance of notifications.
How do you prevent push fatigue?
Limit how often a user receives notifications. Include a cooldown period. Expand your scoring approach to penalize users who have ignored multiple notifications. Track user engagement history. Temporarily suppress notifications if a user shows repeated disinterest. Balance new strategies with user comfort.
Provide a code snippet for the scoring function
import numpy as np
def calculate_affinity_score(purchase_prob, baseline_prob):
# Simple ratio-based score
if baseline_prob == 0:
return 0
return purchase_prob / baseline_prob
def get_store_score(distance, discount, affinity_score):
# Combine multiple factors
# Higher is better, so for distance we invert it
distance_factor = max(0, 1 - (distance / 5.0)) # Example scaling
discount_factor = discount / 100.0
return affinity_score * 0.5 + distance_factor * 0.3 + discount_factor * 0.2
# Example usage
purchase_prob = 0.02
baseline_prob = 0.01
affinity_score = calculate_affinity_score(purchase_prob, baseline_prob)
final_score = get_store_score(distance=1.5, discount=10, affinity_score=affinity_score)
print("Final score:", final_score)
This combines an affinity ratio with distance and discount. Tuning the weights depends on experimentation and business goals.