Principal Component Analysis (PCA)

Principal Component Analysis: Why It Matters and Where It Lives

Principal Component Analysis (PCA) is a linear dimensionality reduction technique that transforms a dataset with many correlated variables into a smaller set of uncorrelated variables — called principal components — while retaining as much of the original information as possible. It sounds technical. But once you understand what it actually does, it becomes one of the most intuitive ideas in all of statistics.

Here’s the core intuition. Imagine measuring 50 different features about 10,000 university students — GPA, test scores, attendance, hours studied, income, distance from campus, and so on. Many of these variables are highly correlated: students who study more tend to get better grades, tend to have higher attendance, tend to score better. The 50 variables are not all telling you 50 different things. Much of the information is redundant. PCA finds those underlying patterns — the real dimensions of variation — and expresses your data in terms of them. Instead of 50 correlated variables, you might end up with 5 principal components that capture 90% of the variance and are completely uncorrelated with each other. Understanding descriptive and inferential statistics is a useful foundation before tackling PCA, since both concepts inform how PCA processes and summarizes data.

PCA was invented in 1901 by Karl Pearson, a British mathematician and statistician who is also credited with developing the Pearson correlation coefficient and the chi-square test. It was later independently developed and named by Harold Hotelling, an American statistician, in the 1930s. For most of its history, PCA was a niche statistical tool. The explosion of computing power and big data in the late 20th and early 21st centuries turned it into a foundational technique — taught in every serious data science program, from Stanford University‘s machine learning courses to MIT’s statistical learning curriculum, from the London School of Economics to University College London’s data science programs.

PCA sits at the intersection of linear algebra, statistics, and data science. It draws on concepts like covariance matrices, eigenvectors, eigenvalues, and singular value decomposition — but you don’t need a graduate degree in mathematics to use it effectively. What you do need is a clear understanding of what each step does and why. That’s exactly what this guide provides. Understanding correlation between variables is a critical prerequisite, since PCA is fundamentally about identifying and restructuring correlated information in a dataset.

What Is Dimensionality Reduction?

Dimensionality reduction is the process of reducing the number of features (variables, dimensions) in a dataset while preserving as much meaningful information as possible. It addresses one of the most persistent problems in data science: datasets with more features than can be practically managed, visualized, or modeled. When a dataset has 200 features, visualizing it is impossible. Training a machine learning model on it is slow and prone to overfitting. Many of those 200 features may be redundant or noisy. Understanding the nature of quantitative data clarifies why high-dimensional quantitative datasets are where PCA does its most important work.

Dimensionality reduction methods fall into two broad categories: feature selection (choosing a subset of the original features) and feature extraction (creating new features from combinations of the original ones). PCA is a feature extraction method — it creates new variables (principal components) that are linear combinations of all original variables. This distinction matters: PCA doesn’t discard variables; it transforms them.

Where PCA Is Used: A Quick Overview

PCA appears across virtually every field that handles high-dimensional data. In genomics, it is used to identify population structure from thousands of genetic markers. In neuroscience, it reduces the complexity of neural activity patterns recorded from hundreds of electrodes simultaneously. In finance, it identifies underlying risk factors from correlated asset returns. In computer vision, it compresses image data for storage and pattern recognition. In social science, it reduces survey response data into underlying attitudinal dimensions. Each of these applications shares the same core mathematical operation — but the interpretation and implementation differ by domain. Factor analysis, a related but distinct method, is also widely used in social science and psychology for similar purposes — the distinction matters and is covered later in this guide.

The Mathematics Behind PCA: Variance, Covariance, and Eigendecomposition

You can use PCA without deriving it from scratch. But knowing the mathematics — even at a conceptual level — makes you a far more effective practitioner. It tells you why you standardize, why you compute a covariance matrix, and why eigenvectors point in the directions that matter. This section covers the key mathematical concepts without unnecessary formalism. Understanding variance and expected values is directly foundational here — PCA is, at its core, a method for redistributing and capturing variance.

Variance and Covariance: The Starting Point

Variance measures how spread out a single variable is around its mean. High variance means the data is widely distributed; low variance means it clusters near the mean. PCA’s goal is to find the directions in which a dataset has the most variance — because high variance equals high information content. Directions with low variance are mostly noise.

Covariance measures how two variables vary together. Positive covariance means they tend to increase and decrease together; negative covariance means one increases as the other decreases; zero covariance means they are linearly independent. The covariance matrix is an n×n matrix (where n is the number of variables) that contains the covariance between every pair of variables. It is the core input to PCA — everything that follows is a mathematical transformation of this matrix.

When variables are highly correlated, their covariance is high, and the covariance matrix will reflect strong off-diagonal values. PCA exploits this structure. By finding the eigenvectors of the covariance matrix, PCA identifies the axes that capture the most covariance — that is, the directions along which the data varies most coherently.

Eigenvectors and Eigenvalues: The Heart of PCA

This is the mathematical core of PCA, and it’s simpler than it looks. An eigenvector of a matrix is a special vector that, when the matrix is applied to it, doesn’t change its direction — it only gets scaled. The eigenvalue is the scaling factor: it tells you how much the vector was stretched or compressed. For the covariance matrix in PCA, eigenvectors point in the directions of maximum variance in the data. Eigenvalues tell you how much variance exists in those directions.

This orthogonality — the fact that principal components are perpendicular to each other — is what makes them uncorrelated. By definition, if two directions are orthogonal, they share no information. This is a crucial property: unlike the original correlated variables, principal components contain no redundant information.

Singular Value Decomposition (SVD): The Practical Algorithm

In practice, most software — including Python’s scikit-learn and R’s prcomp function — does not compute PCA by explicitly building and decomposing a covariance matrix. Instead, it uses Singular Value Decomposition (SVD), a matrix factorization technique that is numerically more stable and computationally more efficient, especially for large datasets. SVD decomposes the data matrix X directly into three matrices: U (left singular vectors), Σ (diagonal matrix of singular values), and Vᵀ (right singular vectors). The right singular vectors are the principal components; the singular values squared, divided by (n−1), give the eigenvalues. The results are mathematically equivalent to the covariance matrix approach but are more numerically reliable for large, high-dimensional datasets.

Explained Variance Ratio

Once you have the eigenvalues, you can calculate the explained variance ratio for each principal component: simply divide each eigenvalue by the sum of all eigenvalues. This gives you the proportion of total variance captured by each component. For example, if PC1 has an eigenvalue of 4.5 and the sum of all eigenvalues is 10, then PC1 explains 45% of the total variance. This ratio is the primary tool for deciding how many components to retain — and it’s what the scree plot visualizes.

Cumulative explained variance — the running sum of explained variance ratios — tells you how much total information is retained as you include more components. A common target is 90–95% cumulative explained variance, though the right threshold depends on your specific application.

How to Perform PCA: A Step-by-Step Walkthrough

Performing Principal Component Analysis correctly requires a specific sequence of steps. Skipping or reordering any of them produces incorrect or misleading results. This section walks through every step with clear explanations of what you’re doing and why.

PCA in Python: Implementation with Scikit-Learn

Principal Component Analysis is implemented in Python most conveniently through scikit-learn, the open-source machine learning library developed at INRIA (France) and now maintained by a global contributor community. Scikit-learn’s PCA class handles standardization-compatible workflows and uses SVD under the hood for numerical stability. Below is a complete, annotated implementation — from data preparation through visualization and interpretation.

Step 1 — Import Libraries and Load Data

# Standard PCA implementation in Python
# Using scikit-learn, pandas, numpy, and matplotlib

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA

# Load your dataset (example: Iris dataset from sklearn)
from sklearn.datasets import load_iris
data = load_iris()
X = data.data          # Feature matrix (150 samples, 4 features)
y = data.target        # Target labels (not used in PCA — it's unsupervised)
feature_names = data.feature_names

Step 2 — Standardize the Data

# Standardize: zero mean, unit variance per feature
# This is essential before PCA — never skip this step

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Verify standardization
print("Mean of each feature:", X_scaled.mean(axis=0).round(5))
# Output: [0.  0.  0.  0.] (approximately zero)
print("Std of each feature:", X_scaled.std(axis=0).round(5))
# Output: [1.  1.  1.  1.] (unit variance)

Step 3 — Fit PCA and Examine Explained Variance

# Fit PCA — first retain all components to examine variance
pca = PCA(n_components=None)  # None = keep all
pca.fit(X_scaled)

# Explained variance ratio per component
evr = pca.explained_variance_ratio_
print("Explained Variance Ratio:", evr.round(4))
# Output: [0.7277  0.2303  0.0366  0.0054]
# PC1 explains 72.8%, PC2 explains 23.0% → cumulative 95.8%

# Cumulative explained variance
cumulative = np.cumsum(evr)
print("Cumulative Variance:", cumulative.round(4))
# Output: [0.7277  0.9580  0.9946  1.0000]

# Scree plot
plt.figure(figsize=(8, 4))
plt.bar(range(1, len(evr)+1), evr, alpha=0.7, label='Individual')
plt.step(range(1, len(evr)+1), cumulative, where='mid', color='red', label='Cumulative')
plt.axhline(y=0.90, color='green', linestyle='--', label='90% threshold')
plt.xlabel('Principal Component'); plt.ylabel('Variance Ratio')
plt.title('Scree Plot'); plt.legend(); plt.show()

Step 4 — Transform Data to Selected Components

# Retain 2 components (explain 95.8% of variance for Iris data)
pca_2d = PCA(n_components=2)
X_pca = pca_2d.fit_transform(X_scaled)
print(f"Original shape: {X_scaled.shape}")    # (150, 4)
print(f"Reduced shape: {X_pca.shape}")      # (150, 2)

# Visualize in 2D — PCA plot
colors = ['#2563EB', '#AA4646', '#7500DE']
plt.figure(figsize=(8, 6))
for i, label in enumerate(data.target_names):
    mask = y == i
    plt.scatter(X_pca[mask, 0], X_pca[mask, 1],
               c=colors[i], label=label, alpha=0.8, s=60)
plt.xlabel(f'PC1 ({evr[0]*100:.1f}% variance)')
plt.ylabel(f'PC2 ({evr[1]*100:.1f}% variance)')
plt.title('PCA: Iris Dataset (2 Components)')
plt.legend(); plt.tight_layout(); plt.show()

# Examine factor loadings (component weights)
loadings = pd.DataFrame(
    pca_2d.components_.T,
    columns=['PC1', 'PC2'],
    index=feature_names
)
print("\nFactor Loadings:\n", loadings.round(3))

Using PCA with n_components as Variance Threshold

# Automatically select components explaining 95% of variance
pca_95 = PCA(n_components=0.95)
X_pca_95 = pca_95.fit_transform(X_scaled)
print(f"Components selected: {pca_95.n_components_}")
# Output: 2 (for Iris data — 2 components explain 95.8%)

Principal Component Analysis in the Real World: Applications Across Fields

PCA’s power comes from its generality — the same mathematical operation produces meaningful results across radically different domains. Understanding these applications builds intuition for when and how to use PCA in your own work.

Genomics and Bioinformatics

PCA is arguably most transformative in genomics, where it handles datasets with millions of genetic variants (single nucleotide polymorphisms, or SNPs) measured across thousands of individuals. Applying PCA to this data reveals population structure — the genetic relationships between individuals that reflect shared ancestry. When researchers at institutions like the Broad Institute or the Wellcome Sanger Institute plot the first two principal components of a large genomic dataset, individuals cluster by geographic ancestry with remarkable precision. This PCA-based approach to population stratification is now standard in genome-wide association studies (GWAS).

Finance and Risk Management

In finance, PCA is used to identify underlying risk factors from correlated asset returns. Stock prices within the same sector tend to move together. PCA extracts the latent factors driving this comovement. The first principal component of equity returns almost always corresponds to the broad market factor. Subsequent components capture sector-specific or macro factors (interest rate sensitivity, inflation exposure, etc.). Fixed income portfolio managers at firms like BlackRock, Vanguard, and Goldman Sachs use PCA to identify key risk factors in bond portfolios and hedge against specific interest rate exposures.

Image Compression and Computer Vision

One of the earliest and most visually intuitive applications of PCA is image compression. The famous Eigenfaces method, developed at MIT by Matthew Turk and Alex Pentland in 1991, used PCA to represent human face images efficiently for recognition. Each face is expressed as a linear combination of “eigenfaces” — the principal components of a face image dataset. Using only the top 50–100 eigenfaces (instead of tens of thousands of pixels), faces can be reconstructed with high fidelity and recognized efficiently.

Neuroscience and fMRI Data

Brain imaging with functional MRI (fMRI) generates data from 50,000–100,000 voxels simultaneously, measured over hundreds or thousands of time points. PCA reduces this massively high-dimensional data to a manageable number of components. At research centers like the NIH, UCL’s Wellcome Centre for Human Neuroimaging, and the Stanford Human Performance Laboratory, PCA-derived components are used to separate signal from noise in brain imaging data and identify resting-state brain networks.

Climate Science and Meteorology

In climate science, PCA is known as Empirical Orthogonal Function (EOF) analysis. The El Niño–Southern Oscillation (ENSO), the leading mode of tropical Pacific sea surface temperature variability, was identified and characterized using EOF analysis. NOAA and the UK Met Office routinely use EOF/PCA to analyze climate model outputs, satellite observations, and reanalysis datasets.

PCA vs. Factor Analysis, t-SNE, UMAP, and LDA: When to Use Each

PCA is not the only dimensionality reduction technique, and it is not always the right one. Understanding when PCA is the appropriate choice — and when t-SNE, UMAP, LDA, or Factor Analysis serves better — is a critical skill for any student or practitioner working with high-dimensional data.

PCA vs. Factor Analysis

This comparison causes more confusion than any other. Both methods reduce dimensionality; both produce components or factors from a set of correlated variables. PCA makes no assumptions about underlying structure — it is a pure mathematical transformation that finds directions of maximum variance. Factor Analysis assumes that observed variables are caused by a smaller number of latent (unmeasured, theoretical) factors plus unique variance.

PCA vs. t-SNE

t-SNE (t-Distributed Stochastic Neighbor Embedding), developed by Laurens van der Maaten and Geoffrey Hinton at the University of Toronto (2008), is a non-linear dimensionality reduction method designed specifically for visualization. Where PCA finds directions of maximum variance globally, t-SNE focuses on preserving local neighborhood structure. This makes t-SNE exceptional for revealing clusters in complex datasets. The trade-offs: t-SNE is slow for large datasets, is non-deterministic, cannot be applied to new data without refitting, and does not preserve global structure. PCA is preferred for preprocessing; t-SNE is preferred for final visualization of complex cluster structures.

PCA vs. UMAP

UMAP (Uniform Manifold Approximation and Projection) is a newer non-linear method that addresses many of t-SNE’s limitations. UMAP is significantly faster, preserves both local and global structure better, and can produce a reusable transformation that can be applied to new data. For preprocessing before machine learning, PCA still holds advantages: it is interpretable, linear, and computationally trivial for most dataset sizes.

PCA vs. LDA

Linear Discriminant Analysis (LDA) is a supervised dimensionality reduction technique. Unlike PCA, which maximizes variance without considering class labels, LDA maximizes the separation between known classes. LDA is the right choice when you have labeled data and your goal is classification. In practice, LDA and PCA are often compared in classification preprocessing: PCA may retain more total variance but LDA’s class-discriminating components often produce better classification performance.

PCA Assumptions, Limitations, and Common Mistakes

PCA is powerful, but it is not universally applicable. Applying it without understanding its assumptions leads to misleading results that can propagate through entire analyses.

Assumption 1: Linearity

PCA assumes that the principal components are linear combinations of the original variables. If the meaningful structure in your data is non-linear — for example, if your data lies on a curved manifold in high-dimensional space — PCA will fail to capture it. For non-linear structures, Kernel PCA, t-SNE, or UMAP are more appropriate.

Assumption 2: Large Variance = Important Information

PCA equates high variance with high information content. This is often true — but not always. If a variable has high variance due to noise or measurement error, PCA will treat it as informative. Conversely, variables with low variance — even if they contain crucial information — will be deprioritized. PCA results should always be validated against domain knowledge.

Assumption 3: Interpretability Trade-Off

Each principal component is a linear combination of all original variables. This makes individual components hard to interpret — you can’t say “PC1 represents feature 7.” If interpretability is paramount, Factor Analysis or sparse PCA may be more appropriate.

Assumption 4: Sensitivity to Outliers

Because PCA maximizes variance, and outliers have extreme values that inflate variance, outliers can pull principal components significantly out of alignment with the true underlying structure. Robust PCA methods — including those using L1 norms or explicit outlier decomposition — address this problem for datasets with known outlier contamination.

Assumption 5: Scale and Measurement Invariance

PCA results change if you change the scale of your variables. Running PCA on income measured in dollars versus thousands of dollars produces different components if data is not standardized first. Standardization makes PCA invariant to arbitrary units of measurement and is nearly always the right choice.

Assumption 6: Missing Data

Standard PCA cannot handle missing data. Rows with any missing values must be removed or imputed before running PCA. Probabilistic PCA (PPCA), developed by Tipping and Bishop at Microsoft Research (UK), extends PCA to handle missing data via the EM algorithm and is a more principled approach when substantial missing data exists.

Using PCA as a Machine Learning Preprocessing Step

One of PCA’s most valuable roles is as a preprocessing step in machine learning pipelines — transforming input features before feeding them into classification, regression, or clustering algorithms. This is particularly important for algorithms sensitive to the curse of dimensionality (k-nearest neighbors, support vector machines), algorithms that assume feature independence (naïve Bayes), and situations where training time and memory are constraints.

Building a PCA Pipeline with scikit-learn

# Complete ML pipeline: StandardScaler → PCA → Classifier
# Using scikit-learn Pipeline to prevent data leakage

from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.datasets import load_iris
import numpy as np

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)

# Build pipeline — PCA inside pipeline prevents data leakage
pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('pca',    PCA(n_components=0.95)),   # retain 95% variance
    ('clf',    LogisticRegression(random_state=42))
])

cv_scores = cross_val_score(pipe, X_train, y_train, cv=5, scoring='accuracy')
print(f"CV Accuracy: {cv_scores.mean():.3f} ± {cv_scores.std():.3f}")

pipe.fit(X_train, y_train)
test_acc = pipe.score(X_test, y_test)
print(f"Test Accuracy: {test_acc:.3f}")
n_comp = pipe.named_steps['pca'].n_components_
print(f"Components selected: {n_comp}")

How to Choose the Right Number of Principal Components

One of the most practically important decisions in applying PCA is selecting how many components to retain. Retaining too few means losing important information. Retaining too many defeats the purpose of dimensionality reduction and can reintroduce noise.

Method 1: The Scree Plot

A scree plot graphs eigenvalues (or explained variance ratios) on the y-axis against component number on the x-axis. You look for an “elbow” — a point where the curve bends sharply and begins to flatten. Components before the elbow capture substantial variance; components after represent diminishing returns. The limitation: the elbow is often ambiguous or gradual, making the visual judgment subjective.

Method 2: Explained Variance Threshold

Select the minimum number of components needed to explain a specified proportion of total variance — typically 80%, 90%, or 95%, depending on how much information loss is acceptable. This is the most intuitive and widely used approach in applied machine learning and data science.

Method 3: Kaiser’s Rule (Eigenvalue > 1)

Kaiser’s Rule retains components whose eigenvalues exceed 1 when PCA is run on standardized data. The logic: if a component’s eigenvalue is less than 1, it explains less variance than a single original variable. Kaiser’s Rule is simple and widely implemented as a default in statistical software like SPSS and SAS. Its limitation: simulation studies show it consistently overestimates the number of meaningful components for wide matrices.

Method 4: Parallel Analysis

Parallel Analysis (PA) is currently the most statistically rigorous method. It generates random datasets with the same dimensions as your real data, computes PCA on each, then retains only components whose eigenvalues exceed the 95th percentile of eigenvalues from the random data. This directly tests whether each component captures more structure than expected by chance. It is increasingly preferred over Kaiser’s Rule in published research.

Variants of PCA: Kernel PCA, Sparse PCA, Incremental PCA, and More

Standard PCA has spawned a family of extensions that address its limitations for specific contexts. Understanding these variants broadens your toolkit and helps you select the most appropriate method for complex real-world data situations.

Kernel PCA

Kernel PCA extends standard PCA to capture non-linear structure using the kernel trick — implicitly mapping data into a high-dimensional feature space where non-linear relationships become linear, then performing standard PCA in that space. The most common kernels are the radial basis function (RBF/Gaussian) kernel and polynomial kernel. Implemented in scikit-learn as sklearn.decomposition.KernelPCA.

Sparse PCA

Sparse PCA adds an L1 penalty to the component loadings, constraining many loadings toward zero. This produces components with sparse loading structures — each component is influenced by only a small subset of the original variables. This dramatically improves interpretability. Formalized by Zou, Hastie, and Tibshirani (Stanford University) and implemented in scikit-learn as sklearn.decomposition.SparsePCA.

Incremental PCA

Incremental PCA computes PCA on data that arrives in batches rather than all at once — essential when the full dataset is too large to fit in memory. It processes one mini-batch at a time, updating component estimates incrementally. Implemented in scikit-learn as sklearn.decomposition.IncrementalPCA.

Probabilistic PCA (PPCA)

Probabilistic PCA, developed by Tipping and Bishop at Microsoft Research Cambridge, reformulates PCA as a probabilistic latent variable model. It assumes the observed data is generated by a low-dimensional latent variable with Gaussian noise. This framework enables principled handling of missing data via the EM algorithm and uncertainty quantification.

Robust PCA

Robust PCA, developed by Emmanuel Candès (Stanford University) and collaborators, decomposes a data matrix into a low-rank component (the true underlying structure) plus a sparse component (outliers or corruptions). This makes it highly effective for surveillance video analysis, financial data with extreme returns, and medical imaging with artifact contamination.

Frequently Asked Questions: Principal Component Analysis