<h4 dir="auto">Description</h4>
<p dir="auto">model_selection.cross_val_score explicitly blocks multiple scores despite calling cross_validate underneath the hood. Essentially, we have two identical functions but have manually blocked cross_val_score from accepting multiple scores.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="def cross_val_score(estimator, X, y=None, groups=None, scoring=None, cv=None,
                    n_jobs=1, verbose=0, fit_params=None,
                    pre_dispatch='2*n_jobs'):

   # To ensure multimetric format is not supported
    scorer = check_scoring(estimator, scoring=scoring)

    cv_results = cross_validate(estimator=estimator, X=X, y=y, groups=groups,
                                scoring={'score': scorer}, cv=cv,
                                return_train_score=False,
                                n_jobs=n_jobs, verbose=verbose,
                                fit_params=fit_params,
                                pre_dispatch=pre_dispatch)
    return cv_results['test_score']"><pre class="notranslate"><span class="pl-k">def</span> <span class="pl-en">cross_val_score</span>(<span class="pl-s1">estimator</span>, <span class="pl-v">X</span>, <span class="pl-s1">y</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-s1">groups</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-s1">scoring</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-s1">cv</span><span class="pl-c1">=</span><span class="pl-c1">None</span>,
                    <span class="pl-s1">n_jobs</span><span class="pl-c1">=</span><span class="pl-c1">1</span>, <span class="pl-s1">verbose</span><span class="pl-c1">=</span><span class="pl-c1">0</span>, <span class="pl-s1">fit_params</span><span class="pl-c1">=</span><span class="pl-c1">None</span>,
                    <span class="pl-s1">pre_dispatch</span><span class="pl-c1">=</span><span class="pl-s">'2*n_jobs'</span>):

   <span class="pl-c"># To ensure multimetric format is not supported</span>
    <span class="pl-s1">scorer</span> <span class="pl-c1">=</span> <span class="pl-en">check_scoring</span>(<span class="pl-s1">estimator</span>, <span class="pl-s1">scoring</span><span class="pl-c1">=</span><span class="pl-s1">scoring</span>)

    <span class="pl-s1">cv_results</span> <span class="pl-c1">=</span> <span class="pl-en">cross_validate</span>(<span class="pl-s1">estimator</span><span class="pl-c1">=</span><span class="pl-s1">estimator</span>, <span class="pl-v">X</span><span class="pl-c1">=</span><span class="pl-v">X</span>, <span class="pl-s1">y</span><span class="pl-c1">=</span><span class="pl-s1">y</span>, <span class="pl-s1">groups</span><span class="pl-c1">=</span><span class="pl-s1">groups</span>,
                                <span class="pl-s1">scoring</span><span class="pl-c1">=</span>{<span class="pl-s">'score'</span>: <span class="pl-s1">scorer</span>}, <span class="pl-s1">cv</span><span class="pl-c1">=</span><span class="pl-s1">cv</span>,
                                <span class="pl-s1">return_train_score</span><span class="pl-c1">=</span><span class="pl-c1">False</span>,
                                <span class="pl-s1">n_jobs</span><span class="pl-c1">=</span><span class="pl-s1">n_jobs</span>, <span class="pl-s1">verbose</span><span class="pl-c1">=</span><span class="pl-s1">verbose</span>,
                                <span class="pl-s1">fit_params</span><span class="pl-c1">=</span><span class="pl-s1">fit_params</span>,
                                <span class="pl-s1">pre_dispatch</span><span class="pl-c1">=</span><span class="pl-s1">pre_dispatch</span>)
    <span class="pl-k">return</span> <span class="pl-s1">cv_results</span>[<span class="pl-s">'test_score'</span>]</pre></div>
<p dir="auto">It creates confusion and duplication of APIs. Open to discussion but can we perhaps consider:</p>
<ol dir="auto">
<li>re-evaluate previous discussion in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="176272358" data-permission-text="Title is private" data-url="https://github.com/scikit-learn/scikit-learn/issues/7388" data-hovercard-type="pull_request" data-hovercard-url="/scikit-learn/scikit-learn/pull/7388/hovercard" href="https://github.com/scikit-learn/scikit-learn/pull/7388">#7388</a> and make cross_val_score accept both single and multiple scores and callable, None?</li>
<li>Or alternatively make error message more explicit when passing multiple scores to cross_val_score.</li>
</ol>
<h4 dir="auto">Steps/Code to Reproduce</h4>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)
lr = LogisticRegression()

cross_val_score(lr, X, y, scoring=['accuracy', 'f1'])"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">model_selection</span> <span class="pl-k">import</span> <span class="pl-s1">cross_val_score</span>
<span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">linear_model</span> <span class="pl-k">import</span> <span class="pl-v">LogisticRegression</span>
<span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">datasets</span> <span class="pl-k">import</span> <span class="pl-s1">load_breast_cancer</span>

<span class="pl-v">X</span>, <span class="pl-s1">y</span> <span class="pl-c1">=</span> <span class="pl-en">load_breast_cancer</span>(<span class="pl-s1">return_X_y</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
<span class="pl-s1">lr</span> <span class="pl-c1">=</span> <span class="pl-v">LogisticRegression</span>()

<span class="pl-en">cross_val_score</span>(<span class="pl-s1">lr</span>, <span class="pl-v">X</span>, <span class="pl-s1">y</span>, <span class="pl-s1">scoring</span><span class="pl-c1">=</span>[<span class="pl-s">'accuracy'</span>, <span class="pl-s">'f1'</span>])</pre></div>
<h4 dir="auto">Expected Results</h4>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; cross_val_score(lr, X, y, scoring=['accuracy', 'f1'])
{'fit_time': array([0.00580502, 0.00351   , 0.00399804]), 'score_time': array([0.00100303, 0.00081301, 0.00084615]), 'test_accuracy': array([0.93684211, 0.96842105, 0.94179894]), 'test_f1': array([0.95081967, 0.97520661, 0.9527897 ])}"><pre class="notranslate"><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-en">cross_val_score</span>(<span class="pl-s1">lr</span>, <span class="pl-v">X</span>, <span class="pl-s1">y</span>, <span class="pl-s1">scoring</span><span class="pl-c1">=</span>[<span class="pl-s">'accuracy'</span>, <span class="pl-s">'f1'</span>])
{<span class="pl-s">'fit_time'</span>: <span class="pl-en">array</span>([<span class="pl-c1">0.00580502</span>, <span class="pl-c1">0.00351</span>   , <span class="pl-c1">0.00399804</span>]), <span class="pl-s">'score_time'</span>: <span class="pl-en">array</span>([<span class="pl-c1">0.00100303</span>, <span class="pl-c1">0.00081301</span>, <span class="pl-c1">0.00084615</span>]), <span class="pl-s">'test_accuracy'</span>: <span class="pl-en">array</span>([<span class="pl-c1">0.93684211</span>, <span class="pl-c1">0.96842105</span>, <span class="pl-c1">0.94179894</span>]), <span class="pl-s">'test_f1'</span>: <span class="pl-en">array</span>([<span class="pl-c1">0.95081967</span>, <span class="pl-c1">0.97520661</span>, <span class="pl-c1">0.9527897</span> ])}</pre></div>
<h4 dir="auto">Actual Results</h4>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; cross_val_score(lr, X, y, scoring=['accuracy', 'f1'])
Traceback (most recent call last):
  File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;
  File &quot;/Users/mingli/GitHub/personal/scikit-learn/sklearn/model_selection/_validation.py&quot;, line 349, in cross_val_score
    scorer = check_scoring(estimator, scoring=scoring)
  File &quot;/Users/mingli/GitHub/personal/scikit-learn/sklearn/metrics/scorer.py&quot;, line 305, in check_scoring
    &quot; None. %r was passed&quot; % scoring)
ValueError: scoring value should either be a callable, string or None. ['accuracy', 'f1'] was passed"><pre class="notranslate"><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-en">cross_val_score</span>(<span class="pl-s1">lr</span>, <span class="pl-v">X</span>, <span class="pl-s1">y</span>, <span class="pl-s1">scoring</span><span class="pl-c1">=</span>[<span class="pl-s">'accuracy'</span>, <span class="pl-s">'f1'</span>])
<span class="pl-v">Traceback</span> (<span class="pl-s1">most</span> <span class="pl-s1">recent</span> <span class="pl-s1">call</span> <span class="pl-s1">last</span>):
  <span class="pl-v">File</span> <span class="pl-s">"&lt;stdin&gt;"</span>, <span class="pl-s1">line</span> <span class="pl-c1">1</span>, <span class="pl-s1">in</span> <span class="pl-c1">&lt;</span><span class="pl-s1">module</span><span class="pl-c1">&gt;</span>
  <span class="pl-v">File</span> <span class="pl-s">"/Users/mingli/GitHub/personal/scikit-learn/sklearn/model_selection/_validation.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">349</span>, <span class="pl-s1">in</span> <span class="pl-s1">cross_val_score</span>
    <span class="pl-s1">scorer</span> <span class="pl-c1">=</span> <span class="pl-en">check_scoring</span>(<span class="pl-s1">estimator</span>, <span class="pl-s1">scoring</span><span class="pl-c1">=</span><span class="pl-s1">scoring</span>)
  <span class="pl-v">File</span> <span class="pl-s">"/Users/mingli/GitHub/personal/scikit-learn/sklearn/metrics/scorer.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">305</span>, <span class="pl-s1">in</span> <span class="pl-s1">check_scoring</span>
    <span class="pl-s">" None. %r was passed"</span> <span class="pl-c1">%</span> <span class="pl-s1">scoring</span>)
<span class="pl-v">ValueError</span>: <span class="pl-s1">scoring</span> <span class="pl-s1">value</span> <span class="pl-s1">should</span> <span class="pl-s1">either</span> <span class="pl-s1">be</span> <span class="pl-s1">a</span> <span class="pl-s1">callable</span>, <span class="pl-s1">string</span> <span class="pl-c1">or</span> <span class="pl-c1">None</span>.<span class="pl-s1"></span> [<span class="pl-s">'accuracy'</span>, <span class="pl-s">'f1'</span>] <span class="pl-s1">was</span> <span class="pl-s1">passed</span></pre></div>
<h4 dir="auto">Versions</h4>
<p dir="auto">Darwin-17.4.0-x86_64-i386-64bit<br>
Python 3.6.4 (default, Mar 22 2018, 13:54:22)<br>
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)]<br>
NumPy 1.14.2<br>
SciPy 1.0.1<br>
Scikit-Learn 0.20.dev0</p>
<p dir="auto">As illustrated in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="462798760" data-permission-text="Title is private" data-url="https://github.com/scikit-learn/scikit-learn/issues/14230" data-hovercard-type="pull_request" data-hovercard-url="/scikit-learn/scikit-learn/pull/14230/hovercard" href="https://github.com/scikit-learn/scikit-learn/pull/14230">#14230</a> tests can contain unreachable code which then produces false positives.</p>
<p dir="auto">Looking at codecov results for test files is probably a good start. For instance, there may be issue in,</p>
<ul dir="auto">
<li><a href="https://codecov.io/gh/scikit-learn/scikit-learn/src/master/sklearn/feature_extraction/tests/test_text.py#L279" rel="nofollow">test_countvectorizer_custom_vocabulary_repeated_indices</a>, <a href="https://codecov.io/gh/scikit-learn/scikit-learn/src/master/sklearn/feature_extraction/tests/test_text.py#L291" rel="nofollow">test_countvectorizer_custom_vocabulary_gap_index</a></li>
</ul>
<p dir="auto">(I have checked only a few test files).</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/VarIr/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/VarIr">@VarIr</a> in case you are interested in looking into this (possibly in another PR)..</p>