I have some douts concerning under-fitting and over-fitting. I have gone through the provided link : under-fitting vs over-fitting
I plotted this graphic using a modified version of the code.
Since we know the true function, we see that degree 4 has a better fit than degree 2. The larger error of degree 2 is then due to the underfitting because the model is not flexible enough.
I would argue that degree 2 and degree 4 fits perform more or less the same, as the MSE distributions have a large overlap.
The blue and orange lines are closer with degree 4 and the orange being the true generative model, I would say that degree 4 is better. Without having access to this information and only having the MSE scores then I would agree with your statement.
By looking at the match with the ground truth curve, I would also say that 2 is (slightly) underfitting, compared to the the degree 4 model that is a very good match.
The MSE seems to be quite higher than for the degree 4 model but it’s not possible to tell if the degree 4 model’s improvement is related to matching unstructured noise in the training. To conclude about under fitting vs over fitting and the best trade-off without having access to the ground truth orange curve, one would require to use cross-validation to compute both the cross-validated train and test scores for each model and look at the gap between the two and use the model with the best test scores to identify the best trade-off.
