Binary logistic regression can give poor results when the two classes are perfectly separated by a linear decision boundary.

Binary logistic regression can give poor results when the two classes are perfectly separated by a linear decision boundary. One way to address this problem is to use the Lasso applied to logistic regression.

Write the likelihood function for the logistic regression problem in terms of x; y; 0 and1. Assume for simplicity that we have n observations and only one variable (i.e. xi is a real number, for i = 1; : : : ; n).

Recall that the logistic regression coefficients βˆ0 ,βˆ1 are obtained by maximizing L(βˆ0, βˆ1). Suppose that all of the xi corresponding to yi = 0 are negative, all other xi are positive. In this case, note that we can get L(βˆ0, βˆ1) arbitrarily close to 1. Explain why this means that βˆ0 and βˆ1 are undefined.

Inspired  by  the  Lasso,  suggest  a  way  to  modify the  negative  log-likelihood function

− log L(β0, β1) so that βˆ0 and βˆ1 become defined even in the separable case above. Justify your answer.