<p dir="auto">Minimal example:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import numpy as np
from sklearn.model_selection import cross_val_score, GridSearchCV
from sklearn.base import ClassifierMixin, BaseEstimator

class Dummy(ClassifierMixin, BaseEstimator):
    def __init__(self, answer=1):
        self.answer = answer

    def fit(self, X, y=None):
        return self

    def predict(self, X):
        return np.ones(X.shape[0], dtype='int') * self.answer

n_samples, n_features = 500, 8
X = np.random.randn(n_samples, n_features)
y = np.random.randint(0, 2, n_samples)

dummy = Dummy()
gcv = GridSearchCV(dummy, {'answer': [0, 1]}, cv=5, iid=False, n_jobs=1)
cross_val_score(gcv, X, y, cv=5, n_jobs=5)

# BrokenProcessPool: A task has failed to un-serialize.
# Please ensure that the arguments of the function are all picklable."><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">model_selection</span> <span class="pl-k">import</span> <span class="pl-s1">cross_val_score</span>, <span class="pl-v">GridSearchCV</span>
<span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">base</span> <span class="pl-k">import</span> <span class="pl-v">ClassifierMixin</span>, <span class="pl-v">BaseEstimator</span>

<span class="pl-k">class</span> <span class="pl-v">Dummy</span>(<span class="pl-v">ClassifierMixin</span>, <span class="pl-v">BaseEstimator</span>):
    <span class="pl-k">def</span> <span class="pl-en">__init__</span>(<span class="pl-s1">self</span>, <span class="pl-s1">answer</span><span class="pl-c1">=</span><span class="pl-c1">1</span>):
        <span class="pl-s1">self</span>.<span class="pl-s1">answer</span> <span class="pl-c1">=</span> <span class="pl-s1">answer</span>

    <span class="pl-k">def</span> <span class="pl-en">fit</span>(<span class="pl-s1">self</span>, <span class="pl-v">X</span>, <span class="pl-s1">y</span><span class="pl-c1">=</span><span class="pl-c1">None</span>):
        <span class="pl-k">return</span> <span class="pl-s1">self</span>

    <span class="pl-k">def</span> <span class="pl-en">predict</span>(<span class="pl-s1">self</span>, <span class="pl-v">X</span>):
        <span class="pl-k">return</span> <span class="pl-s1">np</span>.<span class="pl-en">ones</span>(<span class="pl-v">X</span>.<span class="pl-s1">shape</span>[<span class="pl-c1">0</span>], <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s">'int'</span>) <span class="pl-c1">*</span> <span class="pl-s1">self</span>.<span class="pl-s1">answer</span>

<span class="pl-s1">n_samples</span>, <span class="pl-s1">n_features</span> <span class="pl-c1">=</span> <span class="pl-c1">500</span>, <span class="pl-c1">8</span>
<span class="pl-v">X</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">randn</span>(<span class="pl-s1">n_samples</span>, <span class="pl-s1">n_features</span>)
<span class="pl-s1">y</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">randint</span>(<span class="pl-c1">0</span>, <span class="pl-c1">2</span>, <span class="pl-s1">n_samples</span>)

<span class="pl-s1">dummy</span> <span class="pl-c1">=</span> <span class="pl-v">Dummy</span>()
<span class="pl-s1">gcv</span> <span class="pl-c1">=</span> <span class="pl-v">GridSearchCV</span>(<span class="pl-s1">dummy</span>, {<span class="pl-s">'answer'</span>: [<span class="pl-c1">0</span>, <span class="pl-c1">1</span>]}, <span class="pl-s1">cv</span><span class="pl-c1">=</span><span class="pl-c1">5</span>, <span class="pl-s1">iid</span><span class="pl-c1">=</span><span class="pl-c1">False</span>, <span class="pl-s1">n_jobs</span><span class="pl-c1">=</span><span class="pl-c1">1</span>)
<span class="pl-en">cross_val_score</span>(<span class="pl-s1">gcv</span>, <span class="pl-v">X</span>, <span class="pl-s1">y</span>, <span class="pl-s1">cv</span><span class="pl-c1">=</span><span class="pl-c1">5</span>, <span class="pl-s1">n_jobs</span><span class="pl-c1">=</span><span class="pl-c1">5</span>)

<span class="pl-c"># BrokenProcessPool: A task has failed to un-serialize.</span>
<span class="pl-c"># Please ensure that the arguments of the function are all picklable.</span></pre></div>
<p dir="auto">Full traceback in details.</p>
<p dir="auto">Interestingly, it does not fail when:</p>
<ul dir="auto">
<li>calling <code class="notranslate">cross_val_score</code> with <code class="notranslate">n_jobs=1</code>.</li>
<li>calling <code class="notranslate">cross_val_score</code> directly on <code class="notranslate">dummy</code>, without <code class="notranslate">GridSearchCV</code>.</li>
<li>using a imported classifier, as <code class="notranslate">LogisticRegression</code>, or even the same <code class="notranslate">Dummy</code> custom classifier but imported from another file.</li>
</ul>
<p dir="auto">This is a joblib 0.12 issue, different from <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="366882492" data-permission-text="Title is private" data-url="https://github.com/scikit-learn/scikit-learn/issues/12289" data-hovercard-type="issue" data-hovercard-url="/scikit-learn/scikit-learn/issues/12289/hovercard" href="https://github.com/scikit-learn/scikit-learn/issues/12289">#12289</a> or <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="370340698" data-permission-text="Title is private" data-url="https://github.com/scikit-learn/scikit-learn/issues/12389" data-hovercard-type="issue" data-hovercard-url="/scikit-learn/scikit-learn/issues/12389/hovercard" href="https://github.com/scikit-learn/scikit-learn/issues/12389">#12389</a>. <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ogrisel/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ogrisel">@ogrisel</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/tomMoral/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/tomMoral">@tomMoral</a></p>
<details>
<div class="highlight highlight-text-python-traceback notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="Traceback (most recent call last):
  File &quot;/cal/homes/tdupre/work/src/joblib/joblib/externals/loky/process_executor.py&quot;, line 393, in _process_worker
    call_item = call_queue.get(block=True, timeout=timeout)
  File &quot;/cal/homes/tdupre/miniconda3/envs/py36/lib/python3.6/multiprocessing/queues.py&quot;, line 113, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'Dummy' on &lt;module 'sklearn.externals.joblib.externals.loky.backend.popen_loky_posix' from '/cal/homes/tdupre/work/src/scikit-learn/sklearn/externals/joblib/externals/loky/backend/popen_loky_posix.py'&gt;
'''

The above exception was the direct cause of the following exception:

BrokenProcessPool                         Traceback (most recent call last)
~/work/src/script_csc/condition_effect/test.py in &lt;module&gt;()
     32 
     33     # fails
---&gt; 34     cross_val_score(gcv, X, y, cv=5, n_jobs=5)
     35     &quot;&quot;&quot;
     36     BrokenProcessPool: A task has failed to un-serialize.

~/work/src/scikit-learn/sklearn/model_selection/_validation.py in cross_val_score(estimator, X, y, groups, scoring, cv, n_jobs, verbose, fit_params, pre_dispatch, error_score)
    384                                 fit_params=fit_params,
    385                                 pre_dispatch=pre_dispatch,
--&gt; 386                                 error_score=error_score)
    387     return cv_results['test_score']
    388 

~/work/src/scikit-learn/sklearn/model_selection/_validation.py in cross_validate(estimator, X, y, groups, scoring, cv, n_jobs, verbose, fit_params, pre_dispatch, return_train_score, return_estimator, error_score)
    232             return_times=True, return_estimator=return_estimator,
    233             error_score=error_score)
--&gt; 234         for train, test in cv.split(X, y, groups))
    235 
    236     zipped_scores = list(zip(*scores))

~/work/src/joblib/joblib/parallel.py in __call__(self, iterable)
    996 
    997             with self._backend.retrieval_context():
--&gt; 998                 self.retrieve()
    999             # Make sure that we get a last message telling us we are done
   1000             elapsed_time = time.time() - self._start_time

~/work/src/joblib/joblib/parallel.py in retrieve(self)
    899             try:
    900                 if getattr(self._backend, 'supports_timeout', False):
--&gt; 901                     self._output.extend(job.get(timeout=self.timeout))
    902                 else:
    903                     self._output.extend(job.get())

~/work/src/joblib/joblib/_parallel_backends.py in wrap_future_result(future, timeout)
    519         AsyncResults.get from multiprocessing.&quot;&quot;&quot;
    520         try:
--&gt; 521             return future.result(timeout=timeout)
    522         except LokyTimeoutError:
    523             raise TimeoutError()

~/miniconda3/envs/py36/lib/python3.6/concurrent/futures/_base.py in result(self, timeout)
    403                 raise CancelledError()
    404             elif self._state == FINISHED:
--&gt; 405                 return self.__get_result()
    406             else:
    407                 raise TimeoutError()

~/miniconda3/envs/py36/lib/python3.6/concurrent/futures/_base.py in __get_result(self)
    355     def __get_result(self):
    356         if self._exception:
--&gt; 357             raise self._exception
    358         else:
    359             return self._result

BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable."><pre class="notranslate">Traceback (most recent call last):
  File <span class="pl-s">"/cal/homes/tdupre/work/src/joblib/joblib/externals/loky/process_executor.py"</span>, line <span class="pl-c1">393</span>, in <span class="pl-en">_process_worker</span>
    call_item <span class="pl-k">=</span> call_queue.get(<span class="pl-v">block</span><span class="pl-k">=</span><span class="pl-c1">True</span>, <span class="pl-v">timeout</span><span class="pl-k">=</span>timeout)
  File <span class="pl-s">"/cal/homes/tdupre/miniconda3/envs/py36/lib/python3.6/multiprocessing/queues.py"</span>, line <span class="pl-c1">113</span>, in <span class="pl-en">get</span>
    <span class="pl-k">return</span> _ForkingPickler.loads(res)
<span class="pl-en">AttributeError</span>: <span class="pl-s">Can't get attribute 'Dummy' on &lt;module 'sklearn.externals.joblib.externals.loky.backend.popen_loky_posix' from '/cal/homes/tdupre/work/src/scikit-learn/sklearn/externals/joblib/externals/loky/backend/popen_loky_posix.py'&gt;</span>
'''

The above exception was the direct cause of the following exception:

BrokenProcessPool                         Traceback (most recent call last)
~/work/src/script_csc/condition_effect/test.py in &lt;module&gt;()
     <span class="pl-c1">32</span> 
     <span class="pl-c1">33</span>     <span class="pl-c"><span class="pl-c">#</span> fails</span>
---&gt; 34     cross_val_score(gcv, X, y, cv=5, n_jobs=5)
     <span class="pl-c1">35</span>     <span class="pl-s"><span class="pl-pds">"""</span></span>
     <span class="pl-c1">36</span>     BrokenProcessPool: A task has failed to un<span class="pl-k">-</span>serialize.

~/work/src/scikit-learn/sklearn/model_selection/_validation.py in cross_val_score(estimator, X, y, groups, scoring, cv, n_jobs, verbose, fit_params, pre_dispatch, error_score)
    <span class="pl-c1">384</span>                                 fit_params<span class="pl-k">=</span>fit_params,
    <span class="pl-c1">385</span>                                 pre_dispatch<span class="pl-k">=</span>pre_dispatch,
--&gt; 386                                 error_score=error_score)
    <span class="pl-c1">387</span>     <span class="pl-k">return</span> cv_results[<span class="pl-s"><span class="pl-pds">'</span>test_score<span class="pl-pds">'</span></span>]
    <span class="pl-c1">388</span> 

~/work/src/scikit-learn/sklearn/model_selection/_validation.py in cross_validate(estimator, X, y, groups, scoring, cv, n_jobs, verbose, fit_params, pre_dispatch, return_train_score, return_estimator, error_score)
    <span class="pl-c1">232</span>             return_times<span class="pl-k">=</span><span class="pl-c1">True</span>, return_estimator<span class="pl-k">=</span>return_estimator,
    <span class="pl-c1">233</span>             error_score<span class="pl-k">=</span>error_score)
--&gt; 234         for train, test in cv.split(X, y, groups))
    <span class="pl-c1">235</span> 
    <span class="pl-c1">236</span>     zipped_scores <span class="pl-k">=</span> <span class="pl-c1">list</span>(<span class="pl-c1">zip</span>(<span class="pl-k">*</span>scores))

~/work/src/joblib/joblib/parallel.py in __call__(self, iterable)
    <span class="pl-c1">996</span> 
    <span class="pl-c1">997</span>             <span class="pl-k">with</span> <span class="pl-c1">self</span>._backend.retrieval_context():
--&gt; 998                 self.retrieve()
    <span class="pl-c1">999</span>             <span class="pl-c"><span class="pl-c">#</span> Make sure that we get a last message telling us we are done</span>
   1000             elapsed_time = time.time() - self._start_time

~/work/src/joblib/joblib/parallel.py in retrieve(self)
    <span class="pl-c1">899</span>             <span class="pl-k">try</span>:
    <span class="pl-c1">900</span>                 <span class="pl-k">if</span> <span class="pl-c1">getattr</span>(<span class="pl-c1">self</span>._backend, <span class="pl-s"><span class="pl-pds">'</span>supports_timeout<span class="pl-pds">'</span></span>, <span class="pl-c1">False</span>):
--&gt; 901                     self._output.extend(job.get(timeout=self.timeout))
    <span class="pl-c1">902</span>                 <span class="pl-k">else</span>:
    <span class="pl-c1">903</span>                     <span class="pl-c1">self</span>._output.extend(job.get())

~/work/src/joblib/joblib/_parallel_backends.py in wrap_future_result(future, timeout)
    <span class="pl-c1">519</span>         AsyncResults.get <span class="pl-k">from</span> multiprocessing.<span class="pl-s"><span class="pl-pds">"""</span></span>
    <span class="pl-c1">520</span>         <span class="pl-k">try</span>:
--&gt; 521             return future.result(timeout=timeout)
    <span class="pl-c1">522</span>         <span class="pl-k">except</span> LokyTimeoutError:
    <span class="pl-c1">523</span>             <span class="pl-k">raise</span> <span class="pl-c1">TimeoutError</span>()

~/miniconda3/envs/py36/lib/python3.6/concurrent/futures/_base.py in result(self, timeout)
    <span class="pl-c1">403</span>                 <span class="pl-k">raise</span> CancelledError()
    <span class="pl-c1">404</span>             <span class="pl-k">elif</span> <span class="pl-c1">self</span>._state <span class="pl-k">==</span> <span class="pl-c1">FINISHED</span>:
--&gt; 405                 return self.__get_result()
    <span class="pl-c1">406</span>             <span class="pl-k">else</span>:
    <span class="pl-c1">407</span>                 <span class="pl-k">raise</span> <span class="pl-c1">TimeoutError</span>()

~/miniconda3/envs/py36/lib/python3.6/concurrent/futures/_base.py in __get_result(self)
    <span class="pl-c1">355</span>     <span class="pl-k">def</span> <span class="pl-en">__get_result</span>(<span class="pl-smi"><span class="pl-smi">self</span></span>):
    <span class="pl-c1">356</span>         <span class="pl-k">if</span> <span class="pl-c1">self</span>._exception:
--&gt; 357             raise self._exception
    <span class="pl-c1">358</span>         <span class="pl-k">else</span>:
    <span class="pl-c1">359</span>             <span class="pl-k">return</span> <span class="pl-c1">self</span>._result

<span class="pl-en">BrokenProcessPool</span>: <span class="pl-s">A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.</span></pre></div></details>
<p dir="auto">I'm sure this has been proposed before, but I haven't been able to find a duplicate issue.  Anyways, the idea is to enable <code class="notranslate">StratifiedKFold</code> in <code class="notranslate">model_selection</code> to handle multilabel (multiclass with non-mututally exclusive categories) data.  If the number of labels is small, then ordinary stratified sampling on all possible combinations should work, otherwise one consideration might be to use the iterative approach suggested in <a href="http://lpis.csd.auth.gr/publications/sechidis-ecmlpkdd-2011.pdf" rel="nofollow">http://lpis.csd.auth.gr/publications/sechidis-ecmlpkdd-2011.pdf</a>.</p>