Unlocking Data Insights: How Principal Component Analysis Simplifies Complex Variables

When performing data analysis, many variables increase complexity. Principal Component Analysis (PCA) is a dimensionality‑reduction technique that transforms multiple variables into a few principal components, compressing data, reducing noise, and enabling visualization.

Basic Idea of PCA: Maximum Variance Theory

PCA replaces the original n features with a smaller set of m features that (1) maximize sample variance and (2) remain mutually independent. The new features are linear combinations of the original ones, providing a new framework for interpretation.

Let a random variable represent observations; if we can find a weight vector that maximizes variance, the variance—reflecting data differences—captures the greatest variation of the variables. A constraint (e.g., unit‑length) is required to avoid trivial infinite solutions.

Under this constraint, the optimal solution is a unit vector in p‑dimensional space, representing a “direction” – the principal component direction. Because one component cannot represent all p variables, additional components are sought, each orthogonal to the previous ones, ensuring zero covariance between them.

Key Points

1) Results are affected by the scale of variables; therefore, standardize data before using the covariance or correlation matrix.

2) In practice, select a small number of components (usually no more than 5‑6) that together explain 70%‑80% of the variance (cumulative contribution rate).

Project 2‑D data onto 1‑D while preserving as much original information as possible.

Maximize dispersion after projection; larger variance indicates more information.

Find a direction that maximizes variance of the projected data.

Dimensionality reduction is a change of basis in linear space.

Case Study

In defining clothing standards, measurements of six body dimensions (height, sitting height, chest circumference, arm length, rib circumference, waist circumference) were taken from 128 adult males.

Step 1: Standardize the raw data (subtract mean, divide by standard deviation) and compute the correlation (or covariance) matrix.

Covariance and Correlation

Covariance measures the overall error between two variables, while variance is a special case when the variables are identical. The Pearson correlation coefficient is defined as the normalized covariance.

Step 2: Compute eigenvalues and eigenvectors of the correlation matrix.

The table below shows the first three eigenvalues, eigenvectors, and their contribution rates.

Eigenvalues are ordered from largest to smallest, and the corresponding eigenvectors follow the same order. The first three principal components (after standardization) are identified.

Contribution Rate Formula

The proportion of total variance explained by the k‑th principal component is its contribution rate. The cumulative contribution rate is the sum of individual rates, equal to the sum of the corresponding eigenvalues.

Step 3: Choose the number of components based on cumulative contribution (commonly ≥85%).

Interpretation: The first component mainly reflects overall body size, the second captures shape or slimness, and the third relates to arm length. Not all components can always be meaningfully interpreted.

Step‑by‑Step Summary

Given a dataset with p variables and n samples:

Standardize the data (subtract mean, divide by standard deviation).

Compute the covariance (or correlation) matrix.

Obtain eigenvalues and eigenvectors of this matrix.

Form a matrix of eigenvectors ordered by decreasing eigenvalues.

Calculate the first k principal components using the top k eigenvalues and eigenvectors.

Apply the components for tasks such as principal component regression, normality assessment, outlier detection, and identifying multicollinearity.