Hi,
What are some good reasons to get convinced on the fact that logistic regression is a “linear” model although it is based on a “non-linear” sigmoid/logistic function?
In regression, a linear model is of the form:
y_pred = X @ coef
where y_pred are the prediction of the model, X are the input data, coef are the weights of the linear model and @ is the matrix multiplication operator.
In the regression setting, y_pred will be able to take any value in the range [-inf, +inf].
In the easiest classification framework that is known as binary classification, the target y to be predicted will take two potential values {0, 1}. Therefore, we could adapt the linear regression problem to a binary classification problem if we use a function that map the range [-inf, +inf] into the range [0, 1].
Such a function is the logistic function and the model prediction become:
y_pred = 1 / (1 - np.exp(- X @ coef))
So what should be noted here is that y_pred can take any value in the range [0, 1] that corresponds to the probability of belonging to the class 1. We can then threshold this probability at a 0.5 threshold such that if y_pred < 0.5, we output class 0 and y_pred >= 0.5, we output class 1.
Therefore, logistic regression is still a linear model. It only used a linear function to map the y_pred into the target values defined in a binary classification problem.
3 Likes
ok, thanks for the reply. To make it short and convincingly understandable.
"Logistic regression is called a linear model despite its hypothesis being non-linear due to
→ “X @ coef” is a linear combination of features like the linear regression, that results in
→ “Decision boundary @ threshold” being linear.
Please rectify if I am making a wrong statement.
We are not talking about the condition when “X @ coef” uses combination of non-linear features that results in non-linear decision boundary.
Yes, exactly.
Indeed, we discuss this particular case in a later section.