<h4 dir="auto">Description</h4>
<p dir="auto">For binary tasks with wrapped down custom scoring functions by the <code class="notranslate">metrics.make_scorerer</code>-scheme <code class="notranslate">roc_auc_score</code> is behaving unexpectedly. When requiring the probability output from a binary classifier, which is a shape( n , 2) object, while the training/testing lables are an expected shape (n, ) input, A scoring will fail.<br>
However, the binary task and internal handling of different y shapes is incidentially correctly understood by <code class="notranslate">metrics.log_loss</code> by internal evaluations, but <code class="notranslate">roc_auc_score</code> currently fails at this. This especially cumbersome, if the scoring function is wrapped down in a <code class="notranslate">cross_val_score</code> and a <code class="notranslate">make_scorer</code> much deeper in the code with possible nested pipelines etc, where the automatic correct evaluation of this particular metric is required.</p>
<h4 dir="auto">Steps/Code to Reproduce</h4>
<p dir="auto">This should illustrate what is failing</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from sklearn.metrics import roc_auc_score, log_loss
y_true0 = np.array([False, False, True, True])
y_true1 = ~y_true0
y_true = np.matrix([y_true0, y_true1]).T

y_proba0 = np.array([0.1, 0.4, 0.35, 0.8]) #predict_proba component [:,0]
y_proba1 = 1 - y_proba0 #predict_proba component [:,1]
y_proba = np.matrix([y_proba0, y_proba1]).T #as obtained by classifier.predict_proba()

log_loss(y_true1, y_proba1) # compute for positive class component &gt;&gt;&gt; OK
log_loss(y_true, y_proba) # compute for all class &gt;&gt;&gt; OK
log_loss(y_true1, y_proba) # compute for mixed component &gt;&gt;&gt; OK

roc_auc_score(y_true1, y_proba1) # compute for positive class component &gt;&gt;&gt; OK
roc_auc_score(y_true, y_proba) # compute for all class &gt;&gt;&gt; OK
roc_auc_score(y_true1, y_proba) # compute for mixed component &gt;&gt;&gt; FAIL: bad input shape (4, 2)

#last above line is source of error in this snippet: of binary classification and scoring task
from sklearn.datasets import make_hastie_10_2
from sklearn.ensemble import GradientBoostingClassifier

X, y = make_hastie_10_2(random_state=0)
X_train, X_test = X[:2000], X[2000:]
y_train, y_test = y[:2000], y[2000:]

clf = GradientBoostingClassifier(n_estimators=100, learning_rate=1.0,
    max_depth=1, random_state=0).fit(X_train, y_train)

from sklearn.metrics import make_scorer, roc_auc_score
make_scorer(roc_auc_score, needs_proba=True)(clf, X_test, y_test) # &gt;&gt;&gt; FAIL: bad input shape (1000, 2)
#compare
make_scorer(log_loss, greater_is_better=True, needs_proba=True)(clf, X_test, y_test) # &gt;&gt;&gt; OK"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">metrics</span> <span class="pl-k">import</span> <span class="pl-s1">roc_auc_score</span>, <span class="pl-s1">log_loss</span>
<span class="pl-s1">y_true0</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>([<span class="pl-c1">False</span>, <span class="pl-c1">False</span>, <span class="pl-c1">True</span>, <span class="pl-c1">True</span>])
<span class="pl-s1">y_true1</span> <span class="pl-c1">=</span> <span class="pl-c1">~</span><span class="pl-s1">y_true0</span>
<span class="pl-s1">y_true</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">matrix</span>([<span class="pl-s1">y_true0</span>, <span class="pl-s1">y_true1</span>]).<span class="pl-v">T</span>

<span class="pl-s1">y_proba0</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>([<span class="pl-c1">0.1</span>, <span class="pl-c1">0.4</span>, <span class="pl-c1">0.35</span>, <span class="pl-c1">0.8</span>]) <span class="pl-c">#predict_proba component [:,0]</span>
<span class="pl-s1">y_proba1</span> <span class="pl-c1">=</span> <span class="pl-c1">1</span> <span class="pl-c1">-</span> <span class="pl-s1">y_proba0</span> <span class="pl-c">#predict_proba component [:,1]</span>
<span class="pl-s1">y_proba</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">matrix</span>([<span class="pl-s1">y_proba0</span>, <span class="pl-s1">y_proba1</span>]).<span class="pl-v">T</span> <span class="pl-c">#as obtained by classifier.predict_proba()</span>

<span class="pl-en">log_loss</span>(<span class="pl-s1">y_true1</span>, <span class="pl-s1">y_proba1</span>) <span class="pl-c"># compute for positive class component &gt;&gt;&gt; OK</span>
<span class="pl-en">log_loss</span>(<span class="pl-s1">y_true</span>, <span class="pl-s1">y_proba</span>) <span class="pl-c"># compute for all class &gt;&gt;&gt; OK</span>
<span class="pl-en">log_loss</span>(<span class="pl-s1">y_true1</span>, <span class="pl-s1">y_proba</span>) <span class="pl-c"># compute for mixed component &gt;&gt;&gt; OK</span>

<span class="pl-en">roc_auc_score</span>(<span class="pl-s1">y_true1</span>, <span class="pl-s1">y_proba1</span>) <span class="pl-c"># compute for positive class component &gt;&gt;&gt; OK</span>
<span class="pl-en">roc_auc_score</span>(<span class="pl-s1">y_true</span>, <span class="pl-s1">y_proba</span>) <span class="pl-c"># compute for all class &gt;&gt;&gt; OK</span>
<span class="pl-en">roc_auc_score</span>(<span class="pl-s1">y_true1</span>, <span class="pl-s1">y_proba</span>) <span class="pl-c"># compute for mixed component &gt;&gt;&gt; FAIL: bad input shape (4, 2)</span>

<span class="pl-c">#last above line is source of error in this snippet: of binary classification and scoring task</span>
<span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">datasets</span> <span class="pl-k">import</span> <span class="pl-s1">make_hastie_10_2</span>
<span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">ensemble</span> <span class="pl-k">import</span> <span class="pl-v">GradientBoostingClassifier</span>

<span class="pl-v">X</span>, <span class="pl-s1">y</span> <span class="pl-c1">=</span> <span class="pl-en">make_hastie_10_2</span>(<span class="pl-s1">random_state</span><span class="pl-c1">=</span><span class="pl-c1">0</span>)
<span class="pl-v">X_train</span>, <span class="pl-v">X_test</span> <span class="pl-c1">=</span> <span class="pl-v">X</span>[:<span class="pl-c1">2000</span>], <span class="pl-v">X</span>[<span class="pl-c1">2000</span>:]
<span class="pl-s1">y_train</span>, <span class="pl-s1">y_test</span> <span class="pl-c1">=</span> <span class="pl-s1">y</span>[:<span class="pl-c1">2000</span>], <span class="pl-s1">y</span>[<span class="pl-c1">2000</span>:]

<span class="pl-s1">clf</span> <span class="pl-c1">=</span> <span class="pl-v">GradientBoostingClassifier</span>(<span class="pl-s1">n_estimators</span><span class="pl-c1">=</span><span class="pl-c1">100</span>, <span class="pl-s1">learning_rate</span><span class="pl-c1">=</span><span class="pl-c1">1.0</span>,
    <span class="pl-s1">max_depth</span><span class="pl-c1">=</span><span class="pl-c1">1</span>, <span class="pl-s1">random_state</span><span class="pl-c1">=</span><span class="pl-c1">0</span>).<span class="pl-en">fit</span>(<span class="pl-v">X_train</span>, <span class="pl-s1">y_train</span>)

<span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">metrics</span> <span class="pl-k">import</span> <span class="pl-s1">make_scorer</span>, <span class="pl-s1">roc_auc_score</span>
<span class="pl-en">make_scorer</span>(<span class="pl-s1">roc_auc_score</span>, <span class="pl-s1">needs_proba</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)(<span class="pl-s1">clf</span>, <span class="pl-v">X_test</span>, <span class="pl-s1">y_test</span>) <span class="pl-c"># &gt;&gt;&gt; FAIL: bad input shape (1000, 2)</span>
<span class="pl-c">#compare</span>
<span class="pl-en">make_scorer</span>(<span class="pl-s1">log_loss</span>, <span class="pl-s1">greater_is_better</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, <span class="pl-s1">needs_proba</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)(<span class="pl-s1">clf</span>, <span class="pl-v">X_test</span>, <span class="pl-s1">y_test</span>) <span class="pl-c"># &gt;&gt;&gt; OK</span></pre></div>
<h4 dir="auto">Expected Results</h4>
<p dir="auto"><code class="notranslate">roc_auc_score</code> should behave in a similar way as <code class="notranslate">log_loss</code>, guessing the binary classification task and handle different shape input correctly</p>
<h4 dir="auto">Versions</h4>
<p dir="auto">Windows-10-10.0.15063-SP0<br>
Python 3.6.1 |Anaconda 4.4.0 (64-bit)| (default, May 11 2017, 13:25:24) [MSC v.1900 64 bit (AMD64)]<br>
NumPy 1.13.3<br>
SciPy 0.19.1<br>
Scikit-Learn 0.19.1</p>
<p dir="auto">Related to <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="25636854" data-permission-text="Title is private" data-url="https://github.com/scikit-learn/scikit-learn/issues/2755" data-hovercard-type="issue" data-hovercard-url="/scikit-learn/scikit-learn/issues/2755/hovercard" href="https://github.com/scikit-learn/scikit-learn/issues/2755">#2755</a> and <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="175273552" data-permission-text="Title is private" data-url="https://github.com/scikit-learn/scikit-learn/issues/7346" data-hovercard-type="issue" data-hovercard-url="/scikit-learn/scikit-learn/issues/7346/hovercard" href="https://github.com/scikit-learn/scikit-learn/issues/7346">#7346</a> .</p>
<p dir="auto"><code class="notranslate">BaseEstimator.get_params</code> has logic which:</p>
<ul dir="auto">
<li>sets a warnings filter to always show DeprecationWarnings. Always. Even if they're not from scikit-learn, or triggered outside of <code class="notranslate">get_params</code>.</li>
<li>refuses to return from <code class="notranslate">get_params</code> a parameter (as defined by <code class="notranslate">BaseEstimator._get_param_names</code> to be a constructor parameter) whose attribute access raises a <code class="notranslate">DeprecationWarning</code>.</li>
</ul>
<p dir="auto">I find both these behaviours wrong. Clearly modifying the global warning filter setting is being a bad citizen (and doing so once per parameter is even weirder, even if duplicates are ignored by the warnings module), and we use a more circumspect formulation in <code class="notranslate">sklearn.__init__</code> which only affects warnings coming from <code class="notranslate">sklearn.*</code>.</p>
<p dir="auto">While we rarely use <code class="notranslate">DeprecationWarnings</code> on parameter access (e.g. through a <code class="notranslate">property</code> with the deprecated parameter's name) within Scikit-learn itself, we <em>do</em> tend to allow parameters that are in the process of deprecation to have effect (e.g. <code class="notranslate">SGDClassifier.n_iter</code>), to be displayed by <code class="notranslate">get_params</code>, and to be set by <code class="notranslate">set_params</code>. (No doubt this <code class="notranslate">get_params</code> implementation was written before some of the "do not validate during construction" API policies were clear.) Whose code are we going to break if we just include the deprecated parameters in <code class="notranslate">get_params</code>??</p>
<p dir="auto">I understand that <code class="notranslate">get_params</code> should not show any <code class="notranslate">DeprecationWarning</code>. And I understand that silencing them will in all cases run risks in terms of multithreading. (I also think we need to clarify the risks of solving <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="25636854" data-permission-text="Title is private" data-url="https://github.com/scikit-learn/scikit-learn/issues/2755" data-hovercard-type="issue" data-hovercard-url="/scikit-learn/scikit-learn/issues/2755/hovercard" href="https://github.com/scikit-learn/scikit-learn/issues/2755">#2755</a> with locking, as I think it really should be solved ASAP. But it's a separate issue.)</p>
<p dir="auto">Ping <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ogrisel/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ogrisel">@ogrisel</a>.</p>