Linear Algebra Background for Factor Analysis

We now summarize the key concepts from Linear Algebra that are necessary to perform principal component analysis and factor analysis. Additional details can be found in Linear Algebra and Advanced Matrix Topics.

Eigenvalues and eigenvectors

Definition 1: Given a square k ×  k matrix A, an eigenvalue of A is a scalar λ such that det (A – λI) = 0, where I is the k ×  k identity matrix. A non-zero k × 1 column vector X is an eigenvector that corresponds to eigenvalue λ provided AX = λX.

Since any scalar multiple of an eigenvector is also an eigenvector, commonly we consider unit eigenvectors (i.e. an eigenvector whose length is 1). If X = [x_i] is an eigenvector corresponding to λ, then X/||X|| is a unit eigenvector corresponding to λ, where ||X|| = .

Eigenvalues and eigenvectors of a square matrix can be constructed in Excel using a variety of approaches. In Excel’s Goal Seek and Solver we show how to find eigenvalues using Excel’s Solver capability; we can then find the corresponding eigenvectors using Gaussian Elimination (although there are some limits to this approach). We also show how to calculate eigenvalues and eigenvectors using QR Factorization (see Orthogonal Vectors and Matrices and Spectral Decomposition).

Worksheet Functions

When you need to find eigenvalues and/or eigenvectors you can use either of these techniques within Excel, but because these methods are complicated and time-consuming, we suggest that you use the following Real Statistics array functions (see Eigenvalues and Eigenvectors for more details).

Real Statistics Functions: The Real Statistics Resource Pack provides the following array functions, where R1 is a k × k range in Excel.

eigVALSym(R1): Produces a 1 × k array containing the eigenvalues of the matrix in range R1. These eigenvalues are listed in decreasing absolute value order.

eigVECTSym(R1): Produces a row with the eigenvalues as for eigVAL(R1). Below each eigenvalue is a unit eigenvector corresponding to this eigenvalue. Thus the output from eigVECTSym(R1) is a (k+1) × k array.

The eigenvectors produced by eigVECTSym(R1) are all orthogonal, as described in Definition 8 of Matrix Operations. See Figure 5 of Principal Component Analysis for an example of the output from the eigVECTSym function.

Key properties

Every square k × k matrix has at most k (real) eigenvalues (see Eigenvalues and Eigenvectors). If A is symmetric then it has k eigenvalues, although these don’t need to be distinct (see Symmetric Matrices). It turns out that the eigenvalues for covariance and correlation matrices are always non-negative (see Positive Definite Matrices).

Spectral Decomposition Theorem

Theorem 1: Let A be a symmetric n × n matrix, then A has a spectral decomposition A = CDC^T where C is an n × n matrix whose columns are unit eigenvectors C₁, …, C_n corresponding to the eigenvalues λ₁, …, λ_n of A and D is the n × n diagonal matrix whose main diagonal consists of λ₁, …, λ_n.

This is Property 1 in Spectral Decomposition. We will use this theorem to carry out principal component analysis and factor analysis. In fact, the key form of the theorem that we will use is that A can be expressed as

References

Golub, G. H., Van Loan, C. F. (1996) Matrix computations. 3rd ed. Johns Hopkins University Press

Searle, S. R. (1982) Matrix algebra useful for statistics. Wiley

Perry, W. L. (1988) Elementary linear algebra. McGraw-Hill

Fasshauer, G. (2015) Linear algebra.
https://math.iit.edu/~fass/532_handouts.html

Lambers, J. (2010) Numerical linear algebra
https://www.yumpu.com/en/document/view/41276350