PCA Algorithm

1. Input: data matrix (\(n\) observations and \(p\) features) \(X\in\mathbb{R}^{n\times p}\).

2. Standardize: for each feature \(j\), compute mean \(\mu_j\) and standard deviation \(s_j\), then set for each entry \(\widetilde X_{i,j}=(X_{i,j}-\mu_j)/s_j\) to ensure zero mean and unit variance for each feature.

3. Covariance: calculate the (empirical) covariance matrix \(\Sigma=\frac{1}{n-1}\,\widetilde X^\top\widetilde X\).

4. Eigen-decomposition: After this we perform eigen-decomposition, \(\Sigma=Q\,\Lambda\,Q^\top\), which represents the covariance as a diagonal matrix of eigenvalues but in the eigenvector basis. This gives the variances of the data projected into that space, i.e. the covariance matrix of \(\widetilde X\,Q\).

5. Order and select: Sort eigenvalues \(\lambda_1\ge\cdots\ge\lambda_p\), choose \(r\) (see below for how), and form \(Q_r=[v_1,\dots,v_r]\).

6. Project: Compute the scores, which are the projection of the original data into the eigenbasis space:

   \[ Z = \widetilde X\,Q_r\ \]

Shortly: The idea here is that we standardize the data so that each feature has mean 0 and unit variance, then calculate the covariance matrix. After this we perform eigen-decomposition, which represents the covariance as a diagonal matrix of eigenvalues in the eigenvector basis. This gives the covariance matrix of the data projected into that space, i.e. the covariance matrix of \(\widetilde X\,Q\). The columns of \(\widetilde X\,Q\) are the so called scores, and we retain only those corresponding to the largest eigenvalues (we retain matrix \(Z\) as defined above).