<p dir="auto"><strong>Easy:</strong></p>
<ul dir="auto">
<li>in test_common, check that the ValueError raise has a useful error message. (see sparse test for an example)</li>
<li>put as many of the "specific" tests in test_clustering, test_transformers, ... into test_non_meta_estimators.</li>
</ul>
<p dir="auto"><strong>Not so easy:</strong></p>
<ul dir="auto">
<li>calling <code class="notranslate">fit</code> forgets the previous model if any</li>
<li><del>check how classifiers handle only one class being present</del></li>
<li><del>test how models handle non-float input (does uint8 cause overflows?)</del></li>
</ul>
<p dir="auto"><strong>Things done</strong></p>
<p dir="auto">We should factorize common tests in a new file <code class="notranslate">test_common.py</code> (or maybe <code class="notranslate">test_input.py</code>?). Things to check:</p>
<ul dir="auto">
<li><del>can pickle the object</del></li>
<li><del>raise an exception when data contains nans</del></li>
<li><del>raise an exception for invalid input (e.g., <code class="notranslate">np.matrix</code> or <code class="notranslate">sp.csr_matrix</code> if dense only implementation)</del></li>
<li><del>raise an exception if <code class="notranslate">n_features</code> is not the same in <code class="notranslate">fit</code> and <code class="notranslate">predict</code> or <code class="notranslate">transform</code></del></li>
<li><del><code class="notranslate">__repr__</code> and <code class="notranslate">clone</code> work</del></li>
<li><del>check that we can pickle and unpickle estimators.</del></li>
<li><del>check that all classifiers have a <code class="notranslate">classes_</code> attribute (needs some fixes)</del></li>
</ul>
<hr>
<p dir="auto">Edit by <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/amueller/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/amueller">@amueller</a>!</p>
<p dir="auto">Edit by <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/GaelVaroquaux/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/GaelVaroquaux">@GaelVaroquaux</a> on Aug 13th 2014 to reflect progress in the codebase.</p>
<p dir="auto">I hope this is not a duplicate issue, but it's something I've been thinking about for a while.  It would be nice to create a utility function to generate learning curves: both for hyperparameter value vs. score/error, and for number of training samples vs. score/error.  It's something I do by-hand very often.  I think an interface similar to grid search would work well, with an interface that looks like this:</p>
<p dir="auto">Some setup code:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from sklearn.linear_model import RidgeClassifier
clf = RidgeClassifier()

from sklearn.datasets import load_digits
digits = load_digits()
X, y = digits.data, digits.target"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">linear_model</span> <span class="pl-k">import</span> <span class="pl-v">RidgeClassifier</span>
<span class="pl-s1">clf</span> <span class="pl-c1">=</span> <span class="pl-v">RidgeClassifier</span>()

<span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">datasets</span> <span class="pl-k">import</span> <span class="pl-s1">load_digits</span>
<span class="pl-s1">digits</span> <span class="pl-c1">=</span> <span class="pl-en">load_digits</span>()
<span class="pl-v">X</span>, <span class="pl-s1">y</span> <span class="pl-c1">=</span> <span class="pl-s1">digits</span>.<span class="pl-s1">data</span>, <span class="pl-s1">digits</span>.<span class="pl-s1">target</span></pre></div>
<p dir="auto">Here's the first function we should create (arguments have the same meaning as in <code class="notranslate">GridSearchCV</code>)</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="alpha_range = np.logspace(-2, 2, 50)
train_score, test_score = validation_curve(clf,
                               param_grid={'alpha':alpha_range},
                               scoring='accuracy', cv=10)

import matplotlib.pyplot as plt
plt.semilogx(alpha_range, train_score, label='train')
plt.semilogx(alpha_range, test_score, label='test')"><pre class="notranslate"><span class="pl-s1">alpha_range</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">logspace</span>(<span class="pl-c1">-</span><span class="pl-c1">2</span>, <span class="pl-c1">2</span>, <span class="pl-c1">50</span>)
<span class="pl-s1">train_score</span>, <span class="pl-s1">test_score</span> <span class="pl-c1">=</span> <span class="pl-en">validation_curve</span>(<span class="pl-s1">clf</span>,
                               <span class="pl-s1">param_grid</span><span class="pl-c1">=</span>{<span class="pl-s">'alpha'</span>:<span class="pl-s1">alpha_range</span>},
                               <span class="pl-s1">scoring</span><span class="pl-c1">=</span><span class="pl-s">'accuracy'</span>, <span class="pl-s1">cv</span><span class="pl-c1">=</span><span class="pl-c1">10</span>)

<span class="pl-k">import</span> <span class="pl-s1">matplotlib</span>.<span class="pl-s1">pyplot</span> <span class="pl-k">as</span> <span class="pl-s1">plt</span>
<span class="pl-s1">plt</span>.<span class="pl-en">semilogx</span>(<span class="pl-s1">alpha_range</span>, <span class="pl-s1">train_score</span>, <span class="pl-s1">label</span><span class="pl-c1">=</span><span class="pl-s">'train'</span>)
<span class="pl-s1">plt</span>.<span class="pl-en">semilogx</span>(<span class="pl-s1">alpha_range</span>, <span class="pl-s1">test_score</span>, <span class="pl-s1">label</span><span class="pl-c1">=</span><span class="pl-s">'test'</span>)</pre></div>
<p dir="auto">Here's the second function we should create (same argument meanings):</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="N_range = np.arange(10, int(0.8 * X.shape[0]), 10)
train_score, test_score = learning_curve(clf, N_range,
                               scoring='accuracy', cv=10)

plt.plot(N_range, train_score, label='train')
plt.plot(N_range, test_score, label='test')"><pre class="notranslate"><span class="pl-v">N_range</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">arange</span>(<span class="pl-c1">10</span>, <span class="pl-en">int</span>(<span class="pl-c1">0.8</span> <span class="pl-c1">*</span> <span class="pl-v">X</span>.<span class="pl-s1">shape</span>[<span class="pl-c1">0</span>]), <span class="pl-c1">10</span>)
<span class="pl-s1">train_score</span>, <span class="pl-s1">test_score</span> <span class="pl-c1">=</span> <span class="pl-en">learning_curve</span>(<span class="pl-s1">clf</span>, <span class="pl-v">N_range</span>,
                               <span class="pl-s1">scoring</span><span class="pl-c1">=</span><span class="pl-s">'accuracy'</span>, <span class="pl-s1">cv</span><span class="pl-c1">=</span><span class="pl-c1">10</span>)

<span class="pl-s1">plt</span>.<span class="pl-en">plot</span>(<span class="pl-v">N_range</span>, <span class="pl-s1">train_score</span>, <span class="pl-s1">label</span><span class="pl-c1">=</span><span class="pl-s">'train'</span>)
<span class="pl-s1">plt</span>.<span class="pl-en">plot</span>(<span class="pl-v">N_range</span>, <span class="pl-s1">test_score</span>, <span class="pl-s1">label</span><span class="pl-c1">=</span><span class="pl-s">'test'</span>)</pre></div>
<p dir="auto">I use this sort of thing all the time both in tutorials and in practice; it would be nice to have this functionality available in a convenience routine.  Any thoughts on this?</p>