I am not sure if this was intended, but when I ran the code for not shuffling the data during split & LeaveOneGroupOut & TimeSeriesSplit sections, I got a negative R2 value that also exceeds 1? I could not figure out what’s wrong as I don’t think I changed any code. Restarting the kernel and re-running, or re-running individual cells produce very wild variations in R2 score (e.g. for the final TimeSeriesSplit code cell, I could even get R2 of -123.00 +/- 510.86.)
data_train, data_test, target_train, target_test = train_test_split(
data, target, shuffle=False, random_state=0,
)
regressor.fit(data_train, target_train)
target_predicted = regressor.predict(data_test)
target_predicted = pd.Series(target_predicted, index=target_test.index)
test_score = r2_score(target_test, target_predicted)
print(f"The R2 on this single split is: {test_score:.2f}")
The R2 on this single split is: -2.21
from sklearn.model_selection import LeaveOneGroupOut
groups = quotes.index.to_period("Q")
cv = LeaveOneGroupOut()
test_score = cross_val_score(regressor, data, target,
cv=cv, groups=groups, n_jobs=-1)
print(f"\nThe mean R2 is: "
f"{test_score.mean():.2f} +/- {test_score.std():.2f}")
The mean R2 is: -1.11 +/- 1.95
from sklearn.model_selection import TimeSeriesSplit
cv = TimeSeriesSplit(n_splits=groups.nunique())
test_score = cross_val_score(regressor, data, target,
cv=cv, groups=groups, n_jobs=-1)
print(f"The mean R2 is: "
f"{test_score.mean():.2f} +/- {test_score.std():.2f}")
The mean R2 is: -1.39 +/- 1.94
