ML Interview Q Series: In PCA, what procedure is used to identify the first principal component axis?

📚 Browse the full ML Interview series here.

Comprehensive Explanation

Principal Component Analysis (PCA) extracts directions of maximum variance in high-dimensional data. The first principal component is the direction (or axis) along which the data exhibits the greatest spread (variance). Once this direction is found, the data can be projected onto this axis to reduce dimensionality while retaining as much of that variation as possible. In more detail:

• Centering the Data PCA typically requires the data to be mean-centered. This means that each feature has its mean subtracted so that the transformed data has mean 0. If x_i is the i-th data sample (row vector) and bar{x} is the overall mean, the centered sample is x_i - bar{x}.

• Covariance Matrix From the centered data, one computes the sample covariance matrix S (sometimes denoted Sigma). If the data matrix has n samples and d features, each row vector is a sample, then:

  • The covariance matrix is computed to quantify how each dimension co-varies with the others.

• The First Principal Component The first principal component is the eigenvector of the covariance matrix that corresponds to the largest eigenvalue. Another way to see this is that we look for the unit vector w that maximizes the variance of the data when projected onto w.

  

  Here, S is the covariance matrix. The solution is the eigenvector w that has the largest eigenvalue. In words, we are finding the unit-length direction w in which the data’s variance is maximum.

  • Eigenvalue Decomposition Once you solve the above optimization, you obtain w, which is the first principal component direction (the eigenvector). The largest eigenvalue indicates the magnitude of this maximum variance.

  • Implementation in Practice In practice, many implementations rely on Singular Value Decomposition (SVD) of the data matrix or on directly performing an eigen-decomposition on the covariance matrix. Popular libraries (like scikit-learn in Python) let you compute these principal components with just a few lines of code.

  import numpy as np
  from sklearn.decomposition import PCA
  
  X = np.array([
      [2.5, 2.4],
      [0.5, 0.7],
      [2.2, 2.9],
      [1.9, 2.2],
      [3.1, 3.0],
      [2.3, 2.7]
  ])
  
  pca_model = PCA(n_components=1)
  pca_model.fit(X)
  first_component = pca_model.components_[0]
  print("First Principal Component Axis:", first_component)
  

  Here, the variable first_component contains the direction vector corresponding to the first principal component.

What Are Some Possible Follow-up Questions?

What Happens If We Skip Centering the Data?

If the data is not centered, the direction of maximum variance might not align with the intended principal component directions. Not centering can shift the covariance structure and lead to incorrect components. For example, if the data has a large nonzero mean in one dimension, that dimension might dominate the variance, overshadowing other important variability.

Why Is the First Principal Component Associated with the Largest Eigenvalue?

The eigenvalues of the covariance matrix measure the amount of variance captured by their corresponding eigenvectors. The largest eigenvalue signifies the maximum variance direction in the data. By ordering the eigenvalues from largest to smallest, you get the sequence of principal components in descending order of captured variance.

How Do We Select the Number of Principal Components to Keep?

A common approach is to look at the cumulative variance explained by the principal components. You can sum up the eigenvalues (each eigenvalue divided by the total sum of eigenvalues gives the fraction of variance explained). Then, you choose the smallest number of components that cover a satisfactory threshold of total variance (for example, 90% or 95%).

What if the Data Has Very High Dimensionality Compared to the Number of Samples?

When the number of features is very large compared to the number of samples, the covariance matrix can be rank-deficient. This often happens in situations like genomics or image processing. In such cases, it might be more stable to compute PCA via SVD on the data matrix rather than by forming the covariance matrix directly. SVD-based PCA can handle high-dimensional data more numerically reliably.

Is PCA Sensitive to Outliers?

Yes. Because PCA is based on the variance-covariance structure, outliers can heavily influence the direction of the largest variance. A single outlier might significantly shift the axis. In practice, robust PCA methods (which use robust estimates of covariance) or outlier removal/mitigation techniques can help.

Does PCA Work with Categorical Variables?

PCA is inherently a linear method that operates on continuous data. For categorical features, one might use one-hot encoding or some form of embedding, but the interpretation of principal components for purely categorical features can become less straightforward. In many real-world cases, dimensionality reduction on categorical variables is addressed through alternative factorization methods or specialized models.

How Does PCA Differ from Linear Discriminant Analysis (LDA)?

PCA is unsupervised and focuses on capturing maximum variance in the data, irrespective of class labels. LDA is supervised and seeks directions that best separate multiple classes in a labeled dataset. The objectives and the criteria for extracting components differ. PCA might fail to highlight low-variance directions that are crucial for class discrimination, whereas LDA explicitly maximizes class separation.

Can We Use PCA for Data Visualization?

Yes. One of the most common uses of PCA is for visualizing high-dimensional data in 2D or 3D. By projecting data points onto the first two or three principal components, it is often possible to see some structure, clusters, or trends within the data that were not apparent in the original high-dimensional space. However, one must be cautious about overinterpreting these plots, as the lower-dimensional representation might omit subtle information present in other components.

How Does SVD Relate to Eigen-Decomposition in PCA?

SVD of the data matrix X (of dimension n x d) can be written as X = U * Sigma * V^T. The columns of V (right singular vectors) correspond to the directions of maximal variance, which align with the eigenvectors of the covariance matrix. The singular values in Sigma squared are proportional to the eigenvalues of the covariance matrix. This is a numerically stable way to compute PCA in practice, especially for large, sparse, or tall datasets.

Potential Edge Cases or Pitfalls

• Data Not Scaled. If different features have vastly different scales, the largest-scale features may dominate the variance. Standardizing or normalizing features is often performed before PCA.

• Multicollinearity. When features are highly correlated, the effective dimensionality might be smaller than it appears. PCA can help mitigate multicollinearity but also can be sensitive if done incorrectly (e.g., not centering).

• Interpretability. PCA components are linear combinations of original features, which can sometimes be hard to interpret if many features are involved or the domain knowledge is lacking.

All of these factors highlight practical considerations that one must keep in mind when using PCA in real-world settings.