Logistic Regression using Newton’s Method Detailed

Maximizing the log-likelihood statistic

Property 1: The maximum of the log-likelihood statistic occurs when

Proof: Let

where the y_i are considered constants from the sample and the p_i are defined as follows:

Here

which is the odds ratio (see Definition 3 of Basic Concepts of Logistic Regression). Now let

To make our notation simpler we will define x_i0 = 1 for all i, and so we have

Thus

Also, note that

The maximum value of ln L occurs where the partial derivatives are equal to 0. We first note that

Thus

The maximum of ln L occurs when

for all j, completing the proof.

Newton’s method for logistic regression

To find the values of the coefficients b_i we need to solve the equations of Property 1.

We do this iteratively using Newton’s method (see Definition 2 and Property 2 of Newton’s Method), as described in the following property.

Property 2: Let B = [b_j] be the (k+1) × 1 column vector of logistic regression coefficients, let Y = [y_i] be the n × 1 column vector of observed outcomes of the dependent variable, let X be the n × (k+1) design matrix, let P = [p_i] be the n × 1 column vector of predicted values of success and V = [v_i] be the n × n matrix where v_i = p_i(1–p_i). Then if B₀ is an initial guess of B and for all m we define the following iteration

then for m sufficiently large, B_m+1 ≈ B_m, and so B_m is a reasonable estimate of the coefficient vector.

Proof: Define

where x_i0 = 1. We now calculate the partial derivatives of the f_j.

Let v_i = p_i(1–p_i) and using the terminology of Definition 2 of Newton’s Method, define

Now

where X is the design matrix (see Definition 3 of Multiple Regression Least Squares),  Y is the column matrix with elements y_i and P is the column matrix with elements p_i. Let V = the diagonal matrix with the elements v_i on the main diagonal. Then

We can now use Newton’s method to find B, namely define the k × 1 column vectors P_m and B_m and the (k+1) × (k+1) square matrices V_m and J_m as follows based on the values of P, F, V, and J described above.

Then for sufficiently large m, F(B_m) = 0, which is equivalent to the statement of the property.

Reference

Shalizi, C. (2009) Logistic regression and Newton’s method. Data mining class
https://www.stat.cmu.edu/~cshalizi/350/lectures/26/lecture-26.pdf