How to pass scorer function along with default scorer string?

Module 7. Evaluating model performance Classification metrics
#1

I want to do something like this:

tree = DecisionTreeClassifier()
myscorer = make_scorer(score_func=precision_score, pos_label="donated")
cross_validate(tree, data, target, cv=5, scoring=[myscorer, 'balanced_accuracy'])

But this throws an error saying ValueError: The list/tuple elements must be unique strings of predefined scorers. One or more of the elements were callables. Use a dict of score name mapped to the scorer callable. Got [make_scorer(precision_score, pos_label=donated), 'balanced_accuracy']

I then tried doing this:

tree = DecisionTreeClassifier()
scores_dict = {
    'myscorer': make_scorer(score_func=precision_score, pos_label="donated"),
    'balanced_accuracy': make_scorer(balanced_accuracy_score, pos_label="donated") ,
    'accuracy': make_scorer(accuracy_score, pos_label="donated")
}
cross_validate(tree, data, target, cv=5, scoring=scores_dict)

It works but returns NaNs for all metrics. Can someone help spot what I am doing wrong here?

UPDATE:

So it turns out one doesn’t need to pass pos_label for balanced_accuracy and accuracy. This works now:

tree = DecisionTreeClassifier()
scores_dict = {
    'myscorer': make_scorer(score_func=precision_score, pos_label="donated"),
    'balanced_accuracy': make_scorer(balanced_accuracy_score),
    'accuracy': make_scorer(accuracy_score)
}
cross_validate(tree, data, target, cv=5, scoring=scores_dict)
1 Like