Positive Definite Matrices

Definition 1: An n × n symmetric matrix A is positive definite if for any n × 1 column vector X ≠ 0, X^TAX > 0. A is positive semidefinite if for any n × 1 column vector X, X^TAX ≥ 0.

Observation: Note that if A = [a_ij] and X = [x_i], then

If we set X to be the column vector with x_k = 1 and x_i = 0 for all i ≠ k, then X^TAX = a_kk, and so if A is positive definite, then a_kk > 0, which means that all the entries in the diagonal of A are positive. Similarly, if A is positive semidefinite then all the elements in its diagonal are non-negative.

Property 1: If B is an m × n matrix, then A = B^TB is symmetric

Proof: If B = [b_ij] is an m × n  matrix then A = B^TB = [a_kj] is an n × n matrix where a_kj = . A is symmetric since by Property 1 of Matrix Operations, A^T = (B^TB)^T = B^T(B^T)^T = B^TB = A.

Observation: If X = [x_i] is an m × 1 column vector, then X^TX = .

Property 2: If B is an m × n  matrix, then A = B^TB is positive semidefinite.

Proof: As we observed in Property 1, A is a symmetric n × n matrix. For any n × 1 column vector X, BX is an m × 1 column vector [c_i] where c_i = , and so

Property 3: If B is an m × n matrix of rank n where n ≤ m, then A = B^TB is a positive definite matrix.

Proof: From the proof of Property 2, we know that X^TAX = for any n × 1 column vector X. Now let X be any non-null n × 1 column vector. If all the  are zero, then BX = 0. But by Property 3 of Matrix Rank, if follows that X = 0, which is a contradiction. Since BX ≠ 0, at least one of the c_i ≠ 0, and so > 0, which means that X^TAX =  > 0, and so A is positive definite.

Property 4: The following are equivalent for a symmetric n × n matrix A:

1. A is positive semidefinite
2. There is a matrix U such that A = U^TU
3. All the eigenvalues of A are non-negative

Proof: Assume (c) and show (b). Since A is symmetric, by Theorem 1 of Spectral Decomposition, A has a spectral decomposition A = CDC^T where D consists of the eigenvalues λ₁, …, λ_n of A. By assumption these are all non-negative, and so there exists the diagonal matrix D^½ whose main diagonal consists of , …, . Since D^½D^½ = D, we have

and so the desired matrix is U = (CD^½)^T.

Assume (b) and show (a). Let X be any n × 1 column vector. Then

Assume (a) and show (c). Let A be positive semidefinite and let X be an eigenvector corresponding to eigenvalue λ. Since A is positive semidefinite, X^TAX ≥ 0. Since X is an eigenvector corresponding to λ, AX = λX, and so 0 ≤ X^TAX = X^TλX = λX^TX. Since X^TX = ||X|| > 0, it follows that λ ≥ 0.

Property 5: The following are equivalent for a symmetric n × n matrix A:

1. A is positive definite
2. There is an invertible matrix U such that A = U^TU
3. All the eigenvalues of A are positive

Proof: Assume (c) and show (b). Since A is symmetric, by Theorem 1 of Spectral Decomposition, A has a spectral decomposition A = CDC^T where D consists of the eigenvalues λ₁, …, λ_n of A. By assumption these are all positive, and so there exists the diagonal matrix D½ whose main diagonal consists of , …, . Since D^½D^½ = D, we have

and so the desired matrix is U = (CD^½)^T provided we can show that U is invertible. Now C is an orthogonal matrix and so C^-1 = C^T. Since D^½ is a diagonal matrix det D^½ = the product of the elements on the diagonal. Since all the elements on the main diagonal are positive, it follows that det D^½ ≠ 0, and so D^½ is invertible. Thus U is invertible with inverse ((D^½)^-1C^T)^T, which is CE, where E = the diagonal matrix whose main diagonal consists of the elements , …, 

Assume (b) and show (a). Let X be any n × 1 column vector. Then

If ||UX||² = 0 then UX = 0. Since U is invertible, X = U^-1UX = 0, which is a contradiction. Thus X^TAX = ||UX||² > 0.

Assume (a) and show (c). Let A be positive definite and let X be an eigenvector corresponding to eigenvalue λ. Since A is positive definite, X^TAX > 0. Since X is an eigenvector corresponding to λ, AX = λX, and so 0 < X^TAX = X^TλX = λX^TX. Since X^TX = ||X|| > 0, it follows that λ > 0.

Property 6: The determinant of a positive definite matrix is positive. Furthermore, a positive semidefinite matrix is positive definite if and only if it is invertible.

Proof: The first assertion follows from Property 1 of Eigenvalues and Eigenvectors and Property 5. The second follows from the first and Property 4 of Linear Independent Vectors.

Observation: If A is a positive semidefinite matrix, it is symmetric, and so it makes sense to speak about the spectral decomposition of A.

Definition 2: If A is a positive semidefinite matrix, then the square root of A, denoted A^½, is defined to be the n × n matrix CD^½C^T where C is as defined in Definition 1 of Symmetric matrices and D^½ is the diagonal matrix whose main diagonal consists of , …, .

Property 7: If A is a positive semidefinite matrix, then A^½ is a symmetric matrix and A = A^½A^½

Proof:

Since a diagonal matrix is symmetric, we have

Property 8: Any covariance matrix is positive semidefinite. If the covariance matrix is invertible then it is positive definite.

Proof: We will show the proof for the sample covariance n × n matrix S for X. The proof for a population matrix is similar. Note that

where X = [x_ij] is a k × n matrix such that for each i, {x_ij : 1  ≤ j ≤ n} is a random sample for the random variable x_i. Now let Y be any n x 1 column vector. Thus

Now the following matrices can be represented as a dot product, which evaluate to the same scalar c_i

Thus

which shows that any covariance matrix is positive semidefinite. The second assertion follows from Property 6.

Observation: A consequence of Property 4 and 8 is that all the eigenvalues of a covariance (or correlation) matrix are non-negative real numbers.

Real Statistics Function: The Real Statistics Resource Pack provides the following array function, where R1 is a k × k array.

MSQRT(R1): Produces a k × k array which is the square root of the matrix represented by range R1

Example 1: Find the square root of the matrix in range A4:C6 of Figure 1.

Figure 1 – Square root of a matrix

Range A9:C9 contains the eigenvalues of matrix A and range A10:C12 contains the corresponding eigenvectors (which are repeated as matrix C). These can be calculated using eVECTORS(A4:C6). D^½ is a diagonal matrix whose main diagonal consists of the square roots of the eigenvalues.

The square root of A is therefore given in range I4:K6, calculated by the array formula

=MMULT(E4:G6,MMULT(E9:G11,E14:G16))

The same result can be achieved using the array formula =MSQRT(A4:C6).

Note that the spectral decomposition A = CDC^T is captured by the array formula

=MMULT(E4:G6,MMULT(DIAGONAL(A9:C9),E14:G16))