Hello,
I am wondering if to compare both list of scores, we should (or not) order them first to compare the best score of one to the best score of the other, etc… I actually did that spontaneously but then I was not that sure it was relevant or not. But clearly, ordering the scores changes significantly the result (from 7/10 to 9/10).
Thanks for any comment on that point.
Regards,
Pierre
Hello @PTocquin,
We compare fold-to-fold to compare the models on the same basis i.e. being trained and tested on the same subsets of data.
Imagine for instance that one fold has a lot of outliers and one of the models is particularly robust when learning them but another one isn’t. Then the former would perform better but just in a very particular case (which is by definition is more the exception than the rule) whereas the overall performance of the not-so-robust but ok-performing model might be better for the rest of the splits.
At the end of the day, it’s more consistent to compare things on the same basis.
Best,
Arturo.
Thank you Arturo.