Positive Definite Matrices

Positive definite and semidefinite matrices

Definition 1: An n × n symmetric matrix A is positive definite if for any n × 1 column vector X ≠ 0, X^TAX > 0. A is positive semidefinite if for any n × 1 column vector X, X^TAX ≥ 0.

Observation: Note that if A = [a_ij] and X = [x_i], then

If we set X to be the column vector with x_k = 1 and x_i = 0 for all i ≠ k, then X^TAX = a_kk, and so if A is positive definite, then a_kk > 0, which means that all the entries in the diagonal of A are positive. Similarly, if A is positive semidefinite then all the elements in its diagonal are non-negative.

Property 1: If B is an m × n matrix, then A = B^TB is symmetric

Proof: If B = [b_ij] is an m × n  matrix then A = B^TB = [a_kj] is an n × n matrix where a_kj = . A is symmetric since by Property 1 of Matrix Operations, A^T = (B^TB)^T = B^T(B^T)^T = B^TB = A.

Observation: If X = [x_i] is an m × 1 column vector, then X^TX = .

Property 2: If B is an m × n  matrix, then A = B^TB is positive semidefinite.

Proof: As we observed in Property 1, A is a symmetric n × n matrix. For any n × 1 column vector X, BX is an m × 1 column vector [c_i] where c_i = , and so

Property 3: If B is an m × n matrix of rank n where n ≤ m, then A = B^TB is a positive definite matrix.

Proof: From the proof of Property 2, we know that X^TAX = for any n × 1 column vector X. Now let X be any non-null n × 1 column vector. If all the  are zero, then BX = 0. But by Property 3 of Matrix Rank, if follows that X = 0, which is a contradiction. Since BX ≠ 0, at least one of the c_i ≠ 0, and so > 0, which means that X^TAX =  > 0, and so A is positive definite.

Equivalent positive semidefinite properties

Property 4: The following are equivalent for a symmetric n × n matrix A:

1. A is positive semidefinite
2. There is a matrix U such that A = U^TU
3. All the eigenvalues of A are non-negative

Proof: Assume (c) and show (b). Since A is symmetric, by Property 1 of Spectral Decomposition, A has a spectral decomposition A = CDC^T where D consists of the eigenvalues λ₁, …, λ_n of A. By assumption these are all non-negative, and so there exists the diagonal matrix D^½ whose main diagonal consists of , …, . Since D^½D^½ = D, we have

and so the desired matrix is U = (CD^½)^T.

Assume (b) and show (a). Let X be any n × 1 column vector. Then

Assume (a) and show (c). Let A be positive semidefinite and let X be an eigenvector corresponding to eigenvalue λ. Since A is positive semidefinite, X^TAX ≥ 0. Since X is an eigenvector corresponding to λ, AX = λX, and so 0 ≤ X^TAX = X^TλX = λX^TX. Finally, since X^TX = ||X|| > 0, it follows that λ ≥ 0.

Equivalent positive definite properties

Property 5: The following are equivalent for a symmetric n × n matrix A:

1. A is positive definite
2. There is an invertible matrix U such that A = U^TU
3. All the eigenvalues of A are positive

Proof: Assume (c) and show (b). Since A is symmetric, by Property 1 of Spectral Decomposition, A has a spectral decomposition A = CDC^T where D consists of the eigenvalues λ₁, …, λ_n of A. By assumption these are all positive, and so there exists the diagonal matrix D^½ whose main diagonal consists of , …, . Since D^½D^½ = D, we have

and so the desired matrix is U = (CD^½)^T provided we can show that U is invertible. Now C is an orthogonal matrix and so C^-1 = C^T. Since D^½ is a diagonal matrix det D^½ = the product of the elements on the diagonal. Since all the elements on the main diagonal are positive, it follows that det D^½ ≠ 0, and so D^½ is invertible. Thus U is invertible with inverse ((D^½)^-1C^T)^T, which is CE, where E = the diagonal matrix whose main diagonal consists of the elements , …, 

Assume (b) and show (a). Let X be any n × 1 column vector. Then

If ||UX||² = 0 then UX = 0. Since U is invertible, X = U^-1UX = 0, which is a contradiction. Thus X^TAX = ||UX||² > 0.

Assume (a) and show (c). Let A be positive definite and let X be an eigenvector corresponding to eigenvalue λ. Since A is positive definite, X^TAX > 0. Since X is an eigenvector corresponding to λ, AX = λX, and so 0 < X^TAX = X^TλX = λX^TX. Finally, since X^TX = ||X|| > 0, it follows that λ > 0.

Determinant

Property 6: The determinant of a positive definite matrix is positive. Furthermore, a positive semidefinite matrix is positive definite if and only if it is invertible.

Proof: The first assertion follows from Property 1 of Eigenvalues and Eigenvectors and Property 5. The second follows from the first and Property 4 of Determinant of a Square Matrix.

Observation: A positive semidefinite matrix A is symmetric, and so it makes sense to speak about the spectral decomposition of A.

Square root of a positive semidefinite matrix

Definition 2: If A is a positive semidefinite matrix, then the square root of A, denoted A^½, is defined to be the n × n matrix CD^½C^T where C is as defined in Definition 1 of Symmetric matrices and D^½ is the diagonal matrix whose main diagonal consists of , …, .

Property 7: If A is a positive semidefinite matrix, then A^½ is a symmetric matrix and A = A^½A^½

Proof:

Since a diagonal matrix is symmetric, we have

Covariance matrix

Property 8: Any covariance matrix is positive semidefinite. If the covariance matrix is invertible then it is positive definite.

Proof: We will show the proof for the sample covariance n × n matrix S for X. The proof for a population matrix is similar. Note that

where X = [x_ij] is a k × n matrix such that for each i, {x_ij : 1  ≤ j ≤ n} is a random sample for the random variable x_i. Now let Y be any n x 1 column vector. Thus

Now the following matrices can be represented as a dot product, which evaluate to the same scalar c_i

Thus

which shows that any covariance matrix is positive semidefinite. The second assertion follows from Property 6.

Observation: A consequence of Property 4 and 8 is that all the eigenvalues of a covariance (or correlation) matrix are non-negative real numbers.

Worksheet Function

Real Statistics Function: The Real Statistics Resource Pack provides the following array function, where R1 is a k × k array.

MSQRT(R1): Produces a k × k array which is the square root of the matrix represented by range R1

Example

Example 1: Find the square root of the matrix in range A4:C6 of Figure 1.

Figure 1 – Square root of a matrix

Range A9:C9 contains the eigenvalues of matrix A and range A10:C12 contains the corresponding eigenvectors (which are repeated as matrix C). These can be calculated using eVECTORS(A4:C6). D^½ is a diagonal matrix whose main diagonal consists of the square roots of the eigenvalues.

The square root of A is therefore given in range I4:K6, calculated by the array formula

=MMULT(E4:G6,MMULT(E9:G11,E14:G16))

The same result can be achieved using the array formula =MSQRT(A4:C6).

Note that the spectral decomposition A = CDC^T is captured by the array formula

=MMULT(E4:G6,MMULT(DIAGONAL(A9:C9),E14:G16))

Examples Workbook

Click here to download the Excel workbook with the examples described on this webpage.

References

Golub, G. H., Van Loan, C. F. (1996) Matrix computations. 3rd ed. Johns Hopkins University Press

Searle, S. R. (1982) Matrix algebra useful for statistics. Wiley

Perry, W. L. (1988) Elementary linear algebra. McGraw-Hill

Fasshauer, G. (2015) Linear algebra.
https://math.iit.edu/~fass/532_handouts.html

Lambers, J. (2010) Numerical linear algebra
https://www.yumpu.com/en/document/view/41276350