Conceptual question about lineartiy

M4. Linear Models M4. Linear regression with non-linear link between
#1

When engineering new features on the column vector data, by performing something like: x = np.concatenate([data, data ** 2, data ** 3], axis=1), and then fitting a sklearn.linear_model.LinearRegression(x,y) with the corresponding target y, why is this approach still called a linear fit? By fitting or training a linear model, I understand that we try to describe the data with a linear model of the form y = m*x + b, finding an approximation for the slope m, and the intercept b. In reality, we seem to be approximating the coefficients m, n, k of the polynomial y = m*x + n*x*x + k*x*x*x. So why is this still seen as a linear regression? Because we are still only inverting a matrix, or why? How should I be thinking this conceptually?

Any explanation that would help clarify my interpretation would be greatly appreciated.

Cheers,
c4rlos

#2

Basically, the model is still a linear model of the form y = X @ coef. In some words, this model is still a linear combination of the input feature to provide a prediction.

What changed then is that a feature was modified to be linked non-linearly with the target y.

So conceptually, the algorithm used to find coef is still the same as in the linear regression case.

#3

Thanks for the answer.

To understand a bit better: is the algorithm inverting a Vandermonde matrix to find the coefficients? or what is it doing to find the coefficients?

#4

The naive solution of the linear model would be:

np.linalg.inv(X.T @ X) @ X.T @ y

In practice, we use NumPy and the following function: numpy.linalg.lstsq — NumPy v1.22 Manual

I supposed that it uses lapack under the hood that should use something like SVD to make the decomposition and find the solution.

1 Like
#5

I get it, thanks again!

#6

Good day! What does @ symbol means here? Is it matrix multiplication or what?

#7

It is matrix multiplication indeed.

1 Like