Access model_grid_search.cv_results_ with nested cross-validation

Module 7. Evaluating model performance Nested cross-validation
#1

Hi everyone,
Is it possible to access the grid search results when using a nested cross_validation?

I want to use Nested cross-validation and be able to use heatmap representation to show combinations of hyperparameters.
Thanks in advance.

#2

Here I put an example:

import numpy as np
import pandas as pd
from sklearn.linear_model import Ridge
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import GridSearchCV

alphas = np.logspace(-1, 2.5, num=3)
model = make_pipeline(StandardScaler(), Ridge())
search_cv = GridSearchCV(model, param_grid={"ridge__alpha": alphas})

from sklearn.linear_model import Ridge
from sklearn.model_selection import GridSearchCV

from sklearn.model_selection import cross_validate
from sklearn.model_selection import RepeatedKFold

cv = RepeatedKFold(n_splits=3, n_repeats=3, random_state=0)
cv_results = cross_validate(
    search_cv, X_train, y_train, cv=cv, return_estimator=True, n_jobs=-1)
pd.DataFrame(cv_results["estimator"][0].cv_results_)

I show how to run a grid-search that is provided to an external cross-validation. At the end, I am accessing to the grid-search cross-validated results of the first fold of the outer cross-validation.

2 Likes
#3

Thanks a lot for your answer. It is what I was looking for.

#4

Hi everyone,
I am using this question, to close a few questions at this end of course,
the inner cv is inside de GridSearch object right ? And you are checking the statistical performance with cross-validated, using cv_outer ?
Following the ML workflow, if we are satisfied with the results from cross-validate object, should I use the test set to access the predictions and the metrics from my best estimator ? And those metrics are which I should expect close to production ?
ps: In this way it is more clear to me that create a function as I saw in couple articles over internet, like medium for example.