<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="/tmp# ls -lt sklearn-test* | head
-rw------- 1 buildslave buildslave 213 Jan 14 01:07 sklearn-test8Yh7h6.gz
-rw------- 1 buildslave buildslave 209 Jan 14 01:07 sklearn-testbux42U.bz2
-rw------- 1 buildslave buildslave 209 Jan 14 00:50 sklearn-testpv8EZp.bz2
-rw------- 1 buildslave buildslave 208 Jan 14 00:50 sklearn-testzv3J8Y.gz
-rw------- 1 buildslave buildslave 209 Jan 14 00:49 sklearn-testn2VN9G.bz2
-rw------- 1 buildslave buildslave 213 Jan 14 00:49 sklearn-testQ2Rs8V.gz
-rw------- 1 buildslave buildslave 209 Jan 14 00:20 sklearn-testDuiS27.bz2
-rw------- 1 buildslave buildslave 208 Jan 14 00:20 sklearn-testte9W4r.gz
-rw------- 1 buildslave buildslave 213 Jan 14 00:19 sklearn-testvJ8UnC.gz
-rw------- 1 buildslave buildslave 209 Jan 14 00:19 sklearn-testYTaIeW.bz2
root@lingual:/tmp# bzcat sklearn-testbux42U.bz2 | less
# comment
# note: the next line contains a tab
1.0 3:2.5          11:-5.2 16:1.5 # and an inline comment
2.0 6:1.0 13:-3
# another comment
3.0 21:27
4.0 2:1.234567890123456e10 # double precision value
1.0     # empty line, all zeros
2.0 3:0 # explicit zeros"><pre class="notranslate"><code class="notranslate">/tmp# ls -lt sklearn-test* | head
-rw------- 1 buildslave buildslave 213 Jan 14 01:07 sklearn-test8Yh7h6.gz
-rw------- 1 buildslave buildslave 209 Jan 14 01:07 sklearn-testbux42U.bz2
-rw------- 1 buildslave buildslave 209 Jan 14 00:50 sklearn-testpv8EZp.bz2
-rw------- 1 buildslave buildslave 208 Jan 14 00:50 sklearn-testzv3J8Y.gz
-rw------- 1 buildslave buildslave 209 Jan 14 00:49 sklearn-testn2VN9G.bz2
-rw------- 1 buildslave buildslave 213 Jan 14 00:49 sklearn-testQ2Rs8V.gz
-rw------- 1 buildslave buildslave 209 Jan 14 00:20 sklearn-testDuiS27.bz2
-rw------- 1 buildslave buildslave 208 Jan 14 00:20 sklearn-testte9W4r.gz
-rw------- 1 buildslave buildslave 213 Jan 14 00:19 sklearn-testvJ8UnC.gz
-rw------- 1 buildslave buildslave 209 Jan 14 00:19 sklearn-testYTaIeW.bz2
root@lingual:/tmp# bzcat sklearn-testbux42U.bz2 | less
# comment
# note: the next line contains a tab
1.0 3:2.5          11:-5.2 16:1.5 # and an inline comment
2.0 6:1.0 13:-3
# another comment
3.0 21:27
4.0 2:1.234567890123456e10 # double precision value
1.0     # empty line, all zeros
2.0 3:0 # explicit zeros
</code></pre></div>
<h4 dir="auto">Description</h4>
<p dir="auto"><code class="notranslate">KBinsDiscretizer</code> with <code class="notranslate">strategy='quantile'</code> is producing duplicate bins when used on data which don't have uniform distribution.</p>
<h4 dir="auto">Steps/Code to Reproduce</h4>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import numpy as np
np.random.randint(1, size=(10,1))
X1 = np.random.randint(1, size=(10,1))
X2 = np.random.randint(4, size=(5,1))

X = np.concatenate([X1, X2], axis=0)

from sklearn.preprocessing import KBinsDiscretizer
transformer = KBinsDiscretizer(encode='ordinal')
transformer.fit(X)

transformer.bin_edges_
# Output: array([array([0., 0., 0., 0., 1., 3.])], dtype=object)

transformer.transform(X)
# Output: 
# array([[3.],
#        [3.],
#        [3.],
#        [3.],
#        [3.],
#        [3.],
#        [3.],
#        [3.],
#        [3.],
#        [3.],
#        [4.],
#        [4.],
#        [4.],
#        [4.],
#        [4.]])"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">randint</span>(<span class="pl-c1">1</span>, <span class="pl-s1">size</span><span class="pl-c1">=</span>(<span class="pl-c1">10</span>,<span class="pl-c1">1</span>))
<span class="pl-v">X1</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">randint</span>(<span class="pl-c1">1</span>, <span class="pl-s1">size</span><span class="pl-c1">=</span>(<span class="pl-c1">10</span>,<span class="pl-c1">1</span>))
<span class="pl-v">X2</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">randint</span>(<span class="pl-c1">4</span>, <span class="pl-s1">size</span><span class="pl-c1">=</span>(<span class="pl-c1">5</span>,<span class="pl-c1">1</span>))

<span class="pl-v">X</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">concatenate</span>([<span class="pl-v">X1</span>, <span class="pl-v">X2</span>], <span class="pl-s1">axis</span><span class="pl-c1">=</span><span class="pl-c1">0</span>)

<span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">preprocessing</span> <span class="pl-k">import</span> <span class="pl-v">KBinsDiscretizer</span>
<span class="pl-s1">transformer</span> <span class="pl-c1">=</span> <span class="pl-v">KBinsDiscretizer</span>(<span class="pl-s1">encode</span><span class="pl-c1">=</span><span class="pl-s">'ordinal'</span>)
<span class="pl-s1">transformer</span>.<span class="pl-en">fit</span>(<span class="pl-v">X</span>)

<span class="pl-s1">transformer</span>.<span class="pl-s1">bin_edges_</span>
<span class="pl-c"># Output: array([array([0., 0., 0., 0., 1., 3.])], dtype=object)</span>

<span class="pl-s1">transformer</span>.<span class="pl-en">transform</span>(<span class="pl-v">X</span>)
<span class="pl-c"># Output: </span>
<span class="pl-c"># array([[3.],</span>
<span class="pl-c">#        [3.],</span>
<span class="pl-c">#        [3.],</span>
<span class="pl-c">#        [3.],</span>
<span class="pl-c">#        [3.],</span>
<span class="pl-c">#        [3.],</span>
<span class="pl-c">#        [3.],</span>
<span class="pl-c">#        [3.],</span>
<span class="pl-c">#        [3.],</span>
<span class="pl-c">#        [3.],</span>
<span class="pl-c">#        [4.],</span>
<span class="pl-c">#        [4.],</span>
<span class="pl-c">#        [4.],</span>
<span class="pl-c">#        [4.],</span>
<span class="pl-c">#        [4.]])</span></pre></div>
<h4 dir="auto">Actual Results</h4>
<p dir="auto">The first three bins are duplicates. They are not used in the output. Even if I change the <code class="notranslate">n_bins</code> to 3 or 4, even then the duplicate bins are generated and then not used.</p>
<h4 dir="auto">Expected Results</h4>
<p dir="auto">I understand that:</p>
<ol dir="auto">
<li>The duplicate bins are present because the <code class="notranslate">'strategy'</code> used is 'quantile' and <code class="notranslate">n_bins</code> is fixed.</li>
<li>The bins are not used in output because the internal code is using <code class="notranslate">numpy.isclose</code> and <code class="notranslate">numpy.digitize</code>.</li>
</ol>
<p dir="auto">So is there a scope of removing the duplicate bins after fitting with a warning?</p>
<h4 dir="auto">Versions</h4>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="System:
    python: 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 17:14:51)  [GCC 7.2.0]
executable: ~/anaconda3/.../bin/python
   machine: Linux-4.15.0-20-generic-x86_64-with-debian-buster-sid

BLAS:
    macros: SCIPY_MKL_H=None, HAVE_CBLAS=None
  lib_dirs: ~/anaconda3/.../lib
cblas_libs: mkl_rt, pthread

Python deps:
       pip: 18.1
setuptools: 40.2.0
   sklearn: 0.20.1
     numpy: 1.15.4
     scipy: 1.1.0
    Cython: 0.29
    pandas: 0.23.4"><pre class="notranslate"><code class="notranslate">System:
    python: 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 17:14:51)  [GCC 7.2.0]
executable: ~/anaconda3/.../bin/python
   machine: Linux-4.15.0-20-generic-x86_64-with-debian-buster-sid

BLAS:
    macros: SCIPY_MKL_H=None, HAVE_CBLAS=None
  lib_dirs: ~/anaconda3/.../lib
cblas_libs: mkl_rt, pthread

Python deps:
       pip: 18.1
setuptools: 40.2.0
   sklearn: 0.20.1
     numpy: 1.15.4
     scipy: 1.1.0
    Cython: 0.29
    pandas: 0.23.4
</code></pre></div>