ML Interview Q Series: How would you reliably forecast next year's revenue as part of a tech company's prediction team?

📚 Browse the full ML Interview series here.

Comprehensive Explanation

Forecasting next year’s revenue for a large tech platform typically involves blending historical data patterns, robust modeling approaches (ranging from time series techniques to more sophisticated deep learning models), and the incorporation of relevant external factors. The process includes gathering data, cleaning and preprocessing, selecting an appropriate model, and then validating the model’s performance. Below is a step-by-step breakdown of how one might approach such a forecast.

Data Gathering and Preprocessing

• Collect historical revenue data and ensure proper formatting (e.g., date columns, daily/weekly/monthly aggregated revenue).

• Clean the data by handling missing values, outliers, and any anomalies such as one-time adjustments or special promotions.

• Conduct data transformations: for example, log transforms if the revenue scale varies widely or to stabilize the variance.

Exploratory Analysis

• Plot historical revenue trends to understand seasonal effects (e.g., holiday spikes, cyclical user behavior).

• Check for stationarity if considering classical time series approaches. Stationarity means statistical properties like mean and variance remain stable over time.

• Investigate correlations with external variables (e.g., ad spend, marketing campaigns, user growth rate, macroeconomic indicators).

Modeling Techniques

Classical Time Series Models

If a classical time series approach is chosen, one might use an ARIMA (Autoregressive Integrated Moving Average) or SARIMA (Seasonal ARIMA) model. ARIMA is characterized by combining autoregressive terms, differencing (to achieve stationarity), and moving average components.

Below is a representative formula for a general ARIMA model, showing how the current value y_t depends on its previous values and errors:

Here:

• c is a constant term.

• phi_1, phi_2, ... are the autoregressive parameters describing how past revenue values influence the current revenue.

• theta_1, ... are the moving average parameters indicating how past errors (differences between predicted and actual values) influence the current revenue.

• epsilon_t is the white noise error term at time t, assumed to be uncorrelated with past values.

These models often work well if the time series is stationary or can be made stationary by differencing. In many real-world settings, additional terms can be introduced to handle seasonality, known as SARIMA.

Machine Learning Approaches

For more flexibility, regression-based methods or tree-based models (Random Forest, Gradient Boosted Trees) may be used. One could incorporate a large number of features (e.g., user engagement metrics, marketing spend, monthly user growth).

Deep Learning Approaches

If there is a highly non-linear relationship or complex seasonality/trends, neural networks such as LSTM (Long Short-Term Memory) or Transformer-based models can capture long-term dependencies in revenue data. These methods typically require more data, careful hyperparameter tuning, and robust validation.

External Variables and Feature Engineering

• Include relevant external data, such as macroeconomic indicators (GDP growth, ad market trends) or internal platform metrics (active user metrics, engagement rates) as additional features to improve the model’s forecast accuracy.

• Perform feature engineering to derive more interpretable signals (e.g., rolling averages, cyclical encoding of time for capturing weekly or monthly seasonality).

Model Validation and Evaluation

• Split historical data into training and validation periods.

• Evaluate models using errors such as Mean Absolute Error (MAE), Mean Absolute Percentage Error (MAPE), or Root Mean Square Error (RMSE).

• Select the model or ensemble of models that performs best on validation data.

• Continuously monitor the model’s performance in production and update or retrain as new data arrives or when major shifts occur (e.g., policy changes, changes in user behavior).

Implementation in Python Example

Below is a short Python code snippet using Prophet (a popular library from Facebook/Meta) for illustrative purposes:

import pandas as pd
from prophet import Prophet

# Assume the CSV contains two columns: 'ds' (date) and 'y' (revenue).
df = pd.read_csv('historical_revenue_data.csv')

model = Prophet(
    # Additional parameters can be specified here, for example: seasonality_mode='multiplicative'
)
model.fit(df)

# Forecast for 365 days into the future
future = model.make_future_dataframe(periods=365)
forecast = model.predict(future)

# Inspect the forecast
print(forecast[['ds', 'yhat', 'yhat_lower', 'yhat_upper']])

This approach automatically handles seasonality and trend components. For many production forecasts, multiple models are tested and compared to ensure the best performance.

What if there is irregular seasonality or sudden changes in user behavior?

When seasonality is irregular (e.g., new advertising product launches or platform changes), purely historical approaches can fail. You can:

• Segment the time series by different product lines or user segments if they have distinct behaviors.

• Incorporate calendar-based or event-based features (e.g., product release dates, major ad campaigns).

• Reassess model assumptions frequently, especially if the market is rapidly changing.

What if the data is not stationary?

Classical time series models like ARIMA assume stationarity. You have a few choices:

• Apply differencing, which subtracts the previous value from the current value, to remove trends.

• Use transformations such as logs to stabilize variance.

• Switch to models (e.g., Prophet, certain machine learning algorithms) that don’t require strict stationarity.

How would you incorporate macroeconomic indicators?

Macroeconomic or external variables, such as competitor data, GDP growth, or overall online advertising expenditures, can be included as regressors (i.e., additional covariates) in many forecasting models. In Prophet, for example, you can add regressors:

df['macro_indicator'] = <SOME_MACRO_VALUE>

model.add_regressor('macro_indicator')
model.fit(df)

Including these indicators can improve the ability to handle large-scale market shifts.

How do you deal with uncertainty in the forecast?

Every forecast includes an element of uncertainty. You can:

• Use confidence intervals (e.g., yhat_lower, yhat_upper in Prophet) to present a range of possible outcomes.

• Employ scenario analysis, where you model different potential futures (e.g., optimistic, baseline, and pessimistic).

• Combine forecasts from multiple models (an ensemble) to reduce variance and mitigate over-reliance on any single method.

How would you address sudden shocks or black swan events?

Events such as policy changes, unexpected economic recessions, or global crises cannot always be captured with standard time series or machine learning models trained purely on historical data. Potential steps:

• Incorporate external data sources or leading indicators that might anticipate shocks (e.g., real-time user engagement data).

• Use domain knowledge to create intervention models that manually adjust for known events (e.g., if your platform enters a new country, you might estimate an increase in users and account for it explicitly).

• Continuously retrain or fine-tune models as new data capturing the shock’s effects becomes available.

How can you communicate results to non-technical executives?

Executives are primarily interested in revenue projections and their key drivers. Suggested communication tactics:

• Present central forecast estimates along with clear confidence intervals.

• Provide visualizations (e.g., line charts showing predicted vs. actual revenue) to simplify complex modeling details.

• Highlight the biggest factors influencing the forecast (e.g., marketing budget changes, user growth metrics).

• Emphasize the limitations of the model and any assumptions made in the forecast.

By combining domain expertise, robust modeling, and thorough evaluation, one can deliver a reasoned and data-driven revenue forecast that accounts for historical patterns, external drivers, and potential sources of uncertainty.

Below are additional follow-up questions

What if the historical data is extremely short or comes from a brand-new product line?

When the available revenue data spans only a few months (or even weeks), traditional time series approaches may suffer due to insufficient data. A few strategies to mitigate this challenge are:

• Transfer Learning from Related Data: If historical data from similar or related product lines (or geographies) is available, you can apply a transfer learning approach. The idea is to initially train a model on the related product’s larger dataset, then fine-tune on the short dataset.

• Bayesian Methods: Bayesian frameworks can incorporate prior distributions over parameters, which helps regularize estimates when data is sparse. For instance, hierarchical Bayesian models can share strength among multiple related time series.

• Expert Knowledge: In many real-world scenarios, domain experts might have an estimate of baseline demand or growth rate based on anecdotal evidence or competitor data. Integrating expert priors can help guide predictions when the data itself is minimal.

• Caution with Overfitting: If the dataset is extremely short, the risk of memorizing noise (overfitting) is high. Cross-validation becomes tricky with limited data, so you must consider simpler models or data augmentation strategies.

Pitfall: Believing that more complex models automatically yield better forecasts can be misleading with insufficient data, because they can latch onto random fluctuations as if they were patterns.

How would you handle strong outliers in the revenue data?

Revenue outliers might occur due to sudden advertising spikes, erroneous entries, or atypical promotional events. Proper handling is essential to avoid skewing model parameters.

• Robust Statistical Techniques: Methods like robust regression (e.g., Huber loss) or ARIMA models with robust outlier detection can reduce the impact of extreme points.

• Special Event Markers: Label certain days or periods as “events” in your model. Some forecasting frameworks (like Prophet) allow custom event regressors or holiday effects, which is useful if the outliers correspond to known events.

• Winsorizing/Trimming: In some contexts, you can cap values above a certain percentile to manage extreme outliers. However, be cautious: if legitimate extreme values do occur regularly, capping might distort the forecast.

• Data Review and Correction: Confirm that the outlier is genuine revenue data and not a data-logging error. If an entry is invalid, correct it or remove it from the dataset.

Pitfall: Simply dropping outliers without understanding their cause can remove critical clues about actual revenue surges or drops, thereby losing valuable signals.

How do you approach real-time forecasting in a streaming context?

In some businesses, revenue data (e.g., from ad impressions or subscription transactions) arrives continuously and needs prompt forecasting updates.

• Streaming Frameworks: Use systems like Apache Kafka, Apache Flink, or Spark Streaming to handle incoming data in near real-time.

• Online Learning Algorithms: Algorithms that update their parameters incrementally (e.g., online gradient descent or incremental boosting) allow continuous adaptation to new data without retraining from scratch.

• Sliding Windows: Maintain a rolling window of the most recent data if older data becomes less relevant. This helps the model to focus on the latest patterns and adapt to shifting trends quickly.

• Latency vs. Complexity Trade-Off: More complex deep learning models may not be able to update in real-time if they require large batch computations. Simpler or approximate models might be better when speed is critical.

Pitfall: If the online model drifts too rapidly, you might chase short-term noise. If it adapts too slowly, it could fail to capture abrupt changes. Hyperparameter tuning for adaptivity is crucial in streaming scenarios.

What if you need to forecast at multiple hierarchical levels (e.g., product-level and company-level revenue)?

Large companies often want forecasts at various levels: overall company revenue, product line revenue, and geographical revenue.

• Top-Down vs. Bottom-Up:

  • In a top-down approach, you forecast the total revenue first, then allocate it to subcategories based on historical proportions. This can be simpler but may overlook changing distribution patterns among subcategories.

  • In a bottom-up approach, you forecast at the subcategory level and sum forecasts to get total revenue. This might capture richer signals but can introduce aggregation inconsistencies.

• Hierarchical Forecasting: Certain algorithms (e.g., the forecast reconciliation approach from the HierarchicalForecast library) combine forecasts from multiple levels and reconcile them so they align with the hierarchy constraints (e.g., summing subcategories should match the overall category).

• Cross-Validation for Hierarchies: Evaluate consistency across multiple levels. A model might perform well for each product line in isolation but fail at the aggregated corporate level, or vice versa.

Pitfall: If each subcategory has different seasonalities or trends, a naive top-down approach might poorly capture the nuances of each line. Reconciling forecasts after they are made at multiple levels helps ensure internal consistency.

How do you detect when your model is drifting or no longer valid?

Models can become stale if market conditions, user behavior, or platform features shift.

• Continuous Monitoring: Track metrics such as Mean Absolute Percentage Error (MAPE) and prediction residuals in near real-time or after each forecast cycle. Large spikes in residuals could indicate concept drift.

• Change Point Detection: Statistical tests (like the CUSUM algorithm) or machine learning-based changepoint detection can help identify structural breaks in the time series.

• Champion-Challenger Approach: Keep an alternate model (the “challenger”) that is retrained more frequently. Compare its performance to the currently deployed model (the “champion”). If the challenger starts outperforming the champion, transition to the challenger.

• Retraining Schedules: Periodically retrain the model on the latest data. The frequency depends on business cadence (weekly, monthly, or after notable changes).

Pitfall: Overreacting to every short-term variation can lead to frequent retraining. Striking a balance between model stability and adaptation is crucial.

How do you handle changes in data definition, such as a revised way of calculating revenue?

Occasionally, the business may alter how revenue is recorded or define new revenue segments. For example, they might include subscription upgrades differently than before.

• Parallel Data Collection: For a transitional period, collect data both in the old and the new definition so you can align historical records with the new approach.

• Mapping/Bridging: If the new definition is a known transformation of the old metric (e.g., an added surcharge or commission structure), you can mathematically map old data to match the new definition.

• Separate Models: In some cases, it might be better to start a new time series for the new revenue definition. The old data could be used as a reference for general patterns but might not perfectly align with the new revenue concept.

• Communication with Stakeholders: Ensure that executives and other teams understand that previous forecasts and new forecasts are not directly comparable if the metric itself has changed.

Pitfall: Simply stitching together old and new definitions without accounting for the structural difference can cause major inaccuracies in forecasts and might mislead decision-makers.

How do you incorporate advanced state-space or structural time series models for revenue forecasting?

A structural time series model can explicitly model components like trend, seasonality, and external regressors in a state-space framework. A common representation uses the observation equation and transition equation:

Below that, the state vector alpha_t evolves over time according to:

alpha_{t+1} = T_t * alpha_t + R_t * eta_t

Where:

• y_t is the observed revenue at time t in text format.

• alpha_t is the state vector capturing trend, seasonality, and other hidden factors.

• Z_t is the observation matrix (or vector) that maps the hidden states to the observed data.

• T_t is the transition matrix defining how states evolve over time.

• R_t is the control matrix mapping process noise (eta_t) into the state space.

• epsilon_t and eta_t are noise terms (observation noise and state noise).

Steps to implement such models:

• Define Components: Choose which components to explicitly model (e.g., a local linear trend, one or more seasonal components, cyclical terms, external regressors).

• Parameter Estimation: Estimate model parameters (e.g., the transition matrix T_t) typically via maximum likelihood or Bayesian inference.

• Interpretability: Structural models can reveal how each component (trend, seasonality, external factors) contributes to overall revenue.

• Use of Libraries: Tools like statsmodels (in Python) provide structural or state-space time series capabilities through the SARIMAX or UnobservedComponents classes.

Pitfall: If the model is specified with too many components or states, it can become over-parameterized, leading to unstable estimates, especially with limited data.