Question related to gridsearch and model evaluation

Module 4. Linear Models Modelling with a non-linear relationship data-targ
#1

Hi guys,

First of all,
Congrats on the great content on this MOOC. I’ve learnt a lot, since I started dedicated to this class.
My question is:
For any model we create with or without pipeline, I am not sure whether I should use all the data in the gridsearch for exemple, or just the train set, and them use test set for the predictions. After that, should I use all the data and target on model evaluation.
For each step I will use the inner cross validation on grid search, and them use gridsearch on cross validate with "outer"cv to make sure my best model isnt biased.

My doubt is, if we split the data on train/test set, where should I use the test set. Only, when I will make the predictions with my best model ?

#2

Thank you for your kind words and I am very glad you’ve learnt that much. The standard steps to take are the following:

  • Split data on train / test set.
  • Perform model selection & tuning on a sub-sample of the training set (namely the validation set) which can be obtained with CV, for instance.
  • Score your model on the test set.
  • When trying to make predictions out of your present set, you may need to use your full set for training with the model that performed the best.

I hope that makes things clearer!

#3

Hi, ArturoAmorQ,
Thanks for your replay, let me add numbered steps to make things clear to myself, please feel free to correct me if I am mistaken.
1 - split data on train/test set.
1.1 Put away the test set
1.2 keep the train set

2 - Perform model selection and tuning using the training set with nested cross-validation procedure.
3 - with the final model obtained by automated search and after we make sure the statistical performance
4 - Score the final model using the test set ?
5 - I am not sure if I need to use the full set for make predictions using the best model ?

If I correct understand the discussion about evaluation model, once I train and choose best hyper-parameters with train set, and then use full set of data for evaluation the model and find the generalization error. Once I am sure, I got the best model possible so I can use the test set (my data set unseen) for make predictions and score.