<p dir="auto">Following Bergstra NIPS 2011 (<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jaberg/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jaberg">@jaberg</a> for the intimates) it would be great to have a GridSearch-like object implementing Gaussian-Process based hyper-parameter optimization.</p>
<p dir="auto">To keep things simple, we would optimize only on continuous parameter, i.e. in a hyper-cube. The API would specify an initial bounding box, and also whether each parameter should be varied in the log-domain or in the linear-domain.</p>
<p dir="auto">Some notes from a discussion with James Bergstra:</p>
<ul dir="auto">
<li>The algorithm goes mostly as
<pre class="notranslate"></pre></li>
<li>Compute the score on a set of random points in the specified hyper-cube</li>
<li>For i in budget:<br>
2.a    fit Gaussian Process to the pairs (parameters, scores)<br>
2.b    find the parameters that optimizes the expectation of getting a lower score then the best currently available. For this, optimize it using a black-box scipy optimizer, such as the simulated annealing).<br>
</li>
<li>A lot of the gain comes from sequentially choosing the points, so, as we don't have a queue mechanism, we should do this sequentially for now. Parallel can be done in internal cross-validations, or to compute initial points.</li>
</ul>
<p dir="auto">Remark: to computing the expectancy of (X, y) with Gaussian process, we should overide score to use something more clever than the standard LinearRegression score.</p>
<p dir="auto">I've found that I cannot use pipelines if I wish to use the label encoder. In the following I wish to build a pipeline that first encodes the label and then constructs a one-hot encoding from that labelling.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from sklearn.preprocessing import  OneHotEncoder, LabelEncoder
from sklearn.pipeline import make_pipeline
import numpy as np

X = np.array(['cat', 'dog', 'cow', 'cat', 'cow', 'dog'])

enc = LabelEncoder()
hot = OneHotEncoder()

pipe = make_pipeline(enc, hot)
pipe.fit_transform(X)"><pre class="notranslate"><code class="notranslate">from sklearn.preprocessing import  OneHotEncoder, LabelEncoder
from sklearn.pipeline import make_pipeline
import numpy as np

X = np.array(['cat', 'dog', 'cow', 'cat', 'cow', 'dog'])

enc = LabelEncoder()
hot = OneHotEncoder()

pipe = make_pipeline(enc, hot)
pipe.fit_transform(X)
</code></pre></div>
<p dir="auto">However, the following error is returned:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="lib/python2.7/site-packages/sklearn/pipeline.pyc in _pre_transform(self, X, y, **fit_params)
    117         for name, transform in self.steps[:-1]:
    118             if hasattr(transform, &quot;fit_transform&quot;):
--&gt; 119                 Xt = transform.fit_transform(Xt, y, **fit_params_steps[name])
    120             else:
    121                 Xt = transform.fit(Xt, y, **fit_params_steps[name]) \

TypeError: fit_transform() takes exactly 2 arguments (3 given)"><pre class="notranslate"><code class="notranslate">lib/python2.7/site-packages/sklearn/pipeline.pyc in _pre_transform(self, X, y, **fit_params)
    117         for name, transform in self.steps[:-1]:
    118             if hasattr(transform, "fit_transform"):
--&gt; 119                 Xt = transform.fit_transform(Xt, y, **fit_params_steps[name])
    120             else:
    121                 Xt = transform.fit(Xt, y, **fit_params_steps[name]) \

TypeError: fit_transform() takes exactly 2 arguments (3 given)
</code></pre></div>
<p dir="auto">It seems that the problem is that the fit method for label encoder only takes a y argument, whereas the pipeline assumes that it will take an X and an optional y.</p>