<h3 dir="auto">Describe the bug</h3>
<p dir="auto">When the input is a Pandas DataFrame with multiple dtypes in the columns and the output is also Pandas, the dtypes aren't preserved but cast to a common type.</p>
<p dir="auto">I believe this happens because <code class="notranslate">check_array</code> does this cast:</p>
<p dir="auto"></p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/scikit-learn/scikit-learn/blob/188267212cb5459bfba947c9ece083c0b5f63518/sklearn/utils/validation.py#L788">scikit-learn/sklearn/utils/validation.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
         Line 788
      in
      <a data-pjax="true" class="commit-tease-sha" href="/scikit-learn/scikit-learn/commit/188267212cb5459bfba947c9ece083c0b5f63518">1882672</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L788" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="788"></td>
          <td id="LC788" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-s1">dtype_orig</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">result_type</span>(<span class="pl-c1">*</span><span class="pl-s1">dtypes_orig</span>) </td>
        </tr>
    </tbody></table>
  </div>
</div>
<p></p>
<p dir="auto">And transforms it into a NumPy array. But then when transformed back to a Pandas DataFrame, the dtypes aren't recovered:</p>
<p dir="auto"></p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/scikit-learn/scikit-learn/blob/188267212cb5459bfba947c9ece083c0b5f63518/sklearn/utils/_set_output.py#L60">scikit-learn/sklearn/utils/_set_output.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
         Line 60
      in
      <a data-pjax="true" class="commit-tease-sha" href="/scikit-learn/scikit-learn/commit/188267212cb5459bfba947c9ece083c0b5f63518">1882672</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L60" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="60"></td>
          <td id="LC60" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-k">return</span> <span class="pl-s1">pd</span>.<span class="pl-v">DataFrame</span>(<span class="pl-s1">data_to_wrap</span>, <span class="pl-s1">index</span><span class="pl-c1">=</span><span class="pl-s1">index</span>, <span class="pl-s1">columns</span><span class="pl-c1">=</span><span class="pl-s1">columns</span>, <span class="pl-s1">copy</span><span class="pl-c1">=</span><span class="pl-c1">False</span>) </td>
        </tr>
    </tbody></table>
  </div>
</div>
<p></p>
<h3 dir="auto">Steps/Code to Reproduce</h3>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import pandas as pd
from sklearn.feature_selection import VarianceThreshold

df = pd.DataFrame({&quot;a&quot;: [True, False, True], &quot;b&quot;: [1, 2, 3]})
print(df)
print(VarianceThreshold().set_output(transform=&quot;pandas&quot;).fit_transform(df))

print((df.dtypes == out.dtypes).all())"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">pandas</span> <span class="pl-k">as</span> <span class="pl-s1">pd</span>
<span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">feature_selection</span> <span class="pl-k">import</span> <span class="pl-v">VarianceThreshold</span>

<span class="pl-s1">df</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">DataFrame</span>({<span class="pl-s">"a"</span>: [<span class="pl-c1">True</span>, <span class="pl-c1">False</span>, <span class="pl-c1">True</span>], <span class="pl-s">"b"</span>: [<span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">3</span>]})
<span class="pl-en">print</span>(<span class="pl-s1">df</span>)
<span class="pl-en">print</span>(<span class="pl-v">VarianceThreshold</span>().<span class="pl-en">set_output</span>(<span class="pl-s1">transform</span><span class="pl-c1">=</span><span class="pl-s">"pandas"</span>).<span class="pl-en">fit_transform</span>(<span class="pl-s1">df</span>))

<span class="pl-en">print</span>((<span class="pl-s1">df</span>.<span class="pl-s1">dtypes</span> <span class="pl-c1">==</span> <span class="pl-s1">out</span>.<span class="pl-s1">dtypes</span>).<span class="pl-en">all</span>())</pre></div>
<h3 dir="auto">Expected Results</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="       a  b
0   True  1
1  False  2
2   True  3
       a  b
0   True  1
1  False  2
2   True  3
True"><pre class="notranslate"><code class="notranslate">       a  b
0   True  1
1  False  2
2   True  3
       a  b
0   True  1
1  False  2
2   True  3
True
</code></pre></div>
<h3 dir="auto">Actual Results</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="       a  b
0   True  1
1  False  2
2   True  3
   a  b
0  1  1
1  0  2
2  1  3
False"><pre class="notranslate"><code class="notranslate">       a  b
0   True  1
1  False  2
2   True  3
   a  b
0  1  1
1  0  2
2  1  3
False
</code></pre></div>
<h3 dir="auto">Versions</h3>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="System:
    python: 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0]
executable: .../bin/python
   machine: Linux-5.15.0-71-generic-x86_64-with-glibc2.35
Python dependencies:
      sklearn: 1.2.2
          pip: 23.1.2
   setuptools: 65.5.1
        numpy: 1.24.3
        scipy: 1.10.1
       Cython: None
       pandas: 2.0.1
   matplotlib: 3.7.1
       joblib: 1.2.0
threadpoolctl: 3.1.0
Built with OpenMP: True
threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: .../lib/python3.10/site-packages/numpy.libs/libopenblas64_p-r0-15028c96.3.21.so
        version: 0.3.21
threading_layer: pthreads
   architecture: Haswell
    num_threads: 12
       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: .../lib/python3.10/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 12
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: .../lib/python3.10/site-packages/scipy.libs/libopenblasp-r0-41284840.3.18.so
        version: 0.3.18
threading_layer: pthreads
   architecture: Haswell
    num_threads: 12"><pre class="notranslate">System:
    python: 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0]
executable: .../bin/python
   machine: Linux-5.15.0-71-generic-x86_64-with-glibc2.35
Python dependencies:
      sklearn: 1.2.2
          pip: 23.1.2
   setuptools: 65.5.1
        numpy: 1.24.3
        scipy: 1.10.1
       Cython: None
       pandas: 2.0.1
   matplotlib: 3.7.1
       joblib: 1.2.0
threadpoolctl: 3.1.0
Built with OpenMP: True
threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: .../lib/python3.10/site-packages/numpy.libs/libopenblas64_p-r0-15028c96.3.21.so
        version: 0.3.21
threading_layer: pthreads
   architecture: Haswell
    num_threads: 12
       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: .../lib/python3.10/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 12
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: .../lib/python3.10/site-packages/scipy.libs/libopenblasp-r0-41284840.3.18.so
        version: 0.3.18
threading_layer: pthreads
   architecture: Haswell
    num_threads: 12</pre></div>
<p dir="auto"><code class="notranslate">Scorer</code> objects currently provide an interface that returns a scalar score given an estimator and test data. This is necessary for <code class="notranslate">*SearchCV</code> to calculate a mean score across folds, and determine the best score among parameters.</p>
<p dir="auto">This is very hampering in terms of the diagnostic information available from a cross-fold validation or parameter exploration, which one can see by comparing to the catalogue of <code class="notranslate">metrics</code> that includes: precision and recall with F-score; scores for each of multiple classes as well as an aggregate; and error distributions (i.e. PR-curve or confusion matrix). <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/solomonm/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/solomonm">@solomonm</a> (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="12773836" data-permission-text="Title is private" data-url="https://github.com/scikit-learn/scikit-learn/issues/1837" data-hovercard-type="issue" data-hovercard-url="/scikit-learn/scikit-learn/issues/1837/hovercard" href="https://github.com/scikit-learn/scikit-learn/issues/1837">#1837</a>) and I (ML, an implementation within <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="11954190" data-permission-text="Title is private" data-url="https://github.com/scikit-learn/scikit-learn/issues/1768" data-hovercard-type="pull_request" data-hovercard-url="/scikit-learn/scikit-learn/pull/1768/hovercard" href="https://github.com/scikit-learn/scikit-learn/pull/1768">#1768</a>) have independently sought Precision and Recall to be returned from cross-validation routines when F1 is used as the cross-validation objective; <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/eickenberg/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/eickenberg">@eickenberg</a> on <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="8452341" data-permission-text="Title is private" data-url="https://github.com/scikit-learn/scikit-learn/issues/1381" data-hovercard-type="pull_request" data-hovercard-url="/scikit-learn/scikit-learn/pull/1381/hovercard" href="https://github.com/scikit-learn/scikit-learn/pull/1381#commitcomment-2607318">#1381 (comment)</a> posed a concern regarding array of scores corresponding to multiple targets.</p>
<p dir="auto">I thought it deserved an Issue of its own to solidify the argument and its solution.</p>
<p dir="auto">Some design options:</p>
<ol dir="auto">
<li>Allow multiple scorers to be provided to <code class="notranslate">cross_val_score</code> or <code class="notranslate">*SearchCV</code> (henceforth <code class="notranslate">CVEvaluator</code>), with one specified as the objective. But since the <code class="notranslate">Scorer</code> generally calls <code class="notranslate">estimator.{predict,decision_function,predict_proba}</code>, each scorer would repeat this work.</li>
<li>Separate the objective and non-objective metrics as parameters to <code class="notranslate">CVEvaluator</code>: the <code class="notranslate">scoring</code> parameter remains as it is and a <code class="notranslate">diagnostics</code> parameter provides a callable with similar (same?) arguments as <code class="notranslate">Scorer</code>, but returning a dict. This means that the prediction work is repeated but not necessarily as many times as there are metrics. This diagnostics callable is more flexible and perhaps could be passed the training data as well as the test data.</li>
<li>Continue to use the <code class="notranslate">scoring</code> parameter, but allow the <code class="notranslate">Scorer</code> to return a dict with a special key for the objective score. This would need to be handled by the caller. For backwards compatibility, no existing scorers would change their behaviour of returning a float. This ensures no repeated prediction work.</li>
<li>Add an additional method to the <code class="notranslate">Scorer</code> interface that generates a set of named outputs (as with <code class="notranslate">calc_names</code> proposed in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="12773836" data-permission-text="Title is private" data-url="https://github.com/scikit-learn/scikit-learn/issues/1837" data-hovercard-type="issue" data-hovercard-url="/scikit-learn/scikit-learn/issues/1837/hovercard" href="https://github.com/scikit-learn/scikit-learn/issues/1837">#1837</a>), again with a special key for the objective score. This allows users to continue using <code class="notranslate">scoring='f1'</code> but get back precision and recall for free.</li>
</ol>
<p dir="auto">Note that 3. and 4. potentially allow for any set of metrics to be composed into a scorer without redundant prediction work (and 1. allows composition with highly redundant prediction work).</p>
<p dir="auto">Comments, critiques and suggestions are very welcome.</p>