ML Interview Q Series: How can you verify if a linear model is overfitting its training data?

📚 Browse the full ML Interview series here.

Comprehensive Explanation

Overfitting in linear models is often indicated by an excessive focus on fitting the training set’s noise or idiosyncrasies. The primary way to detect overfitting is to compare model performance on data seen during training and on unseen data (validation or test set). If the model shows a significant performance drop when evaluated on the unseen data relative to the training data, that is a common sign of overfitting.

One way to quantify performance is via a loss metric. A frequently used loss function in linear models is the Mean Squared Error (MSE). Below is the core formula in its standard form, which is central to evaluating how well a linear model fits the data:

where y_i represents the true target value for the i-th instance in the dataset, and hat{y}_i denotes the model’s predicted value for that instance. N is the total number of data points in the evaluated set. The lower the MSE, the closer the predictions are to the ground truth on that set. Overfitting can be inferred if the MSE is very low on the training data but significantly higher on the test or validation data.

Another approach is to look at metrics such as R-squared (R^2), but it is often more insightful to compare these metrics on training vs. unseen data rather than looking at a single score.

There are also visual checks for overfitting, such as plotting residuals. If the residuals on training data appear random (which is typically good) but on test data show large or systematic deviations, this also suggests overfitting. In simpler linear models, overfitting usually arises when one includes far too many features (especially polynomial terms or interactions) or when the model has not been regularized and is forced to fit small fluctuations in the training data.

Cross-validation is another robust method to detect overfitting in linear models. By splitting the training data into multiple folds and repeatedly training and validating, you gain a more reliable measure of how the model would perform on unseen data. A stable model that is not overfitting tends to show similar performance metrics across the various folds.

Using simpler linear models (fewer parameters) or adding regularization (L2 regularization in Ridge Regression, L1 regularization in Lasso Regression, or a combination of both in Elastic Net) can mitigate overfitting. These methods reduce the magnitude or number of model coefficients, preventing them from fitting random noise in the training data.

How do you interpret the discrepancy between training and test scores to confirm overfitting in linear models?

If the error on the training set is significantly lower than on the test set, it indicates that the model is overly specialized to training patterns that do not generalize well. A large gap between training and test performance (e.g., training R^2 is close to 1 but test R^2 is much lower) points to overfitting.

In practical scenarios, you might see a near-zero training MSE, but a high test MSE, which reflects the same phenomenon. By systematically adjusting model complexity (for example, adding or removing polynomial features) and comparing training vs. validation errors, you can confirm if a model is truly overfitting or if there is another issue (like high variance in the dataset or data leakage).

How can cross-validation help detect overfitting in linear models?

Cross-validation divides the training set into smaller subsets called folds. In k-fold cross-validation, you train on (k-1) folds and validate on the remaining fold. You repeat this process k times so that each fold serves as a validation set exactly once. The average performance across these folds is a more reliable indicator of the model’s ability to generalize.

When a linear model overfits, the validation performance in some folds will likely degrade significantly compared to the training performance on those same folds. A consistent pattern of significantly higher training accuracy (or lower training error) than validation accuracy (or error) across the folds suggests overfitting. Because cross-validation uses different splits, it helps confirm that overfitting is not just a fluke in a single train-test split.

What strategies mitigate overfitting in linear models?

One approach is to reduce the model’s complexity, for example by eliminating irrelevant features. This ensures the model does not “memorize” noise. Another approach is adding regularization, which penalizes large coefficient values. L2 regularization (Ridge) shrinks coefficients, thereby stabilizing them, while L1 regularization (Lasso) can drive some coefficients to zero, effectively performing feature selection.

Other strategies include acquiring more data (if feasible), applying dimensionality reduction techniques (like PCA before regression), or employing techniques like dropout in neural networks (though dropout is more specific to deep learning architectures rather than classical linear models). Early stopping can also help when using gradient-based optimization: you monitor the validation loss and stop training once it starts to deteriorate while the training loss continues to decrease.

Could you explain the role of adjusted R-squared in detecting overfitting in linear models?

If you add more features to a linear model, the regular R^2 metric often increases, even if these features have little actual predictive value. Adjusted R-squared corrects this by penalizing the addition of features that do not significantly improve the model. While not a perfect solution, a noticeable discrepancy between R^2 and adjusted R^2 can signal that additional features are not truly enhancing the model but instead may be pushing it to overfit. If the adjusted R^2 does not increase in tandem with R^2, it indicates the model might be overfitting with those extra features.

Does multicollinearity play a role in overfitting for linear models?

Multicollinearity occurs when features are highly correlated with each other, making it difficult for the model to distinguish their individual effects. Although high multicollinearity does not always imply overfitting, it can inflate the variances of the coefficient estimates. This inflation might lead to overly complex models that are sensitive to small changes in the training data. Regularization helps here as well, by shrinking correlated coefficients and reducing the risk of the model latching onto noise.

How do you implement a simple check for overfitting in a linear regression model using Python?

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

# Suppose X, y are your features and target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = LinearRegression()
model.fit(X_train, y_train)

y_pred_train = model.predict(X_train)
y_pred_test = model.predict(X_test)

mse_train = mean_squared_error(y_train, y_pred_train)
mse_test = mean_squared_error(y_test, y_pred_test)

print("Training MSE:", mse_train)
print("Testing MSE:", mse_test)

If you see a much smaller MSE on the training set than on the test set, it typically indicates overfitting. You could extend this approach by incorporating cross-validation through scikit-learn’s cross_val_score function for a more thorough check.