Multiple Regression Analysis Theory

Properties from Multiple Regression using Matrices

Property 1:

Proof: Define the n × 1 vector E such that E = Y − XB. The goal of the least-squares method is to find the (k+1) × 1 vector such that E^TE is minimized. Now note that

This last equality is true since B^TX^TY  is a scalar and so

Setting the first derivative of E^TE equal to zero, we get

Thus

and so

This is essentially another proof of Theorem 1 of Least Squares for Multiple Regression.

Property 2:

Proof: This follows from Property 1 and Definition 1 of Multiple Regression using Matrices since

Property A: The hat matrix H is symmetric and idempotent, i.e. H^T = H and HH = H. The same is true for I − H.

Proof: First we observe that (X^TX)^-1 is symmetric since

Now

Also

This shows symmetry. We now show idempotency.

Property 3: B is an unbiased estimator of β, i.e. E[B] = β

Proof: By Property 1

But

and so we have

Thus

since it is assumed that E[B] = β.

Property 4: The covariance matrix of B can be represented by

Proof: Based on the assumption that var(ε_i) = σ² for all i and cov(ε_i, ε_i) = 0 for all i ≠ j, it follows that cov(ε) = E[εε^T] = σ²I. As we have seen in the proof of Property 3

Thus by Property 3

Hence

Properties from Multiple Regression in Excel

Property B: For any n × n matrix A and n × 1 vectors Y and Z

Proof:

Thus

Since Tr(E[C]) = E[Tr(C)] for any square matrix C, Tr(CD) = Tr(DC) and b = Tr(b) for any scalar b, we have

Consequently

and so

Property C:

Proof: We first note that by Property 1

By Property 3 of Multiple Regression Analysis in Excel and the above equality

Property 4: MS_Res is an unbiased estimator of σ², the variance of the error terms ε_i

Proof: By Property 1 and C

By Property B

where
But

and so

Now

Since Y = Xβ + ε, it follows that E[Y] = E[Xβ] + E[ε] = Xβ, and so we have

Putting it all together we have

Finally