<p dir="auto">Just quite simply, the 20news dataset in sklearn is noisy!<br>
Beware just using this dataset out-of-the-box<br>
For example, in "test" there are impossibly noisy samples (length of tokens &lt; 5):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="SHORT text found: [['Not', 'this', 'again'], ['talk.politics.misc']]
SHORT text found: [['SUBSCRIBE', 'XPERT', 'skji@eve.kaist.ac.kr'], ['comp.windows.x']]
SHORT text found: [['1B', 'Career', 'DON', 'MATTINGLY'], ['rec.sport.baseball']]
SHORT text found: [['please', 'subscrive', 'me', 'rpicas@porto.inescn.pt'], ['comp.windows.x']]
SHORT text found: [['very', 'good'], ['sci.med']]
SHORT text found: [['alt.religion.spam'], ['talk.religion.misc']]
SHORT text found: [['BULLSHIT'], ['sci.med']]
SHORT text found: [['cheek'], ['rec.autos']]
SHORT text found: [['unsubscribe'], ['comp.windows.x']]
SHORT text found: [['shut', 'up', 'andi'], ['talk.politics.mideast']]
SHORT text found: [['unsubscribe'], ['comp.windows.x']]
SHORT text found: [['please', 'subscrive', 'me', 'rpicas@porto.inescn.pt'], ['comp.windows.x']]
SHORT text found: [['Brandon', 'Wise', 'bwise@nyx.cs.du.edu'], ['comp.os.ms-windows.misc']]
SHORT text found: [['unfortunately', 'not', 'Legalize', 'Freedom'], ['talk.religion.misc']]
SHORT text found: [['urn', 'll', 'he', 'uckers'], ['talk.politics.guns']]
SHORT text found: [['Test'], ['misc.forsale']]
SHORT text found: [['Harvested', 'to', 'order'], ['sci.med']]
SHORT text found: [['Does', 'Autodesk', 'has', 'BBS'], ['comp.graphics']]
SHORT text found: [['ignore'], ['rec.motorcycles']]
SHORT text found: [['exit'], ['sci.space']]
SHORT text found: [['Ditto', 'Great', 'post', 'joe.kusmierczak@mail.trincoll.edu'], ['talk.politics.guns']]
SHORT text found: [['Ken'], ['misc.forsale']]
SHORT text found: [['this', 'is', 'test', 'Erme'], ['comp.sys.mac.hardware']]
SHORT text found: [['Israel', 'Happy', '45th', 'Birthday'], ['talk.politics.mideast']]
SHORT text found: [['Just', 'kidding'], ['talk.politics.mideast']]
SHORT text found: [['Inguiry', 'by', 'address:er1@eridan.chuvashia.su'], ['misc.forsale']]
SHORT text found: [['400included', 'shipping'], ['misc.forsale']]
SHORT text found: [['Bo', 'Bilinsky'], ['rec.sport.baseball']]
SHORT text found: [['This', 'is', 'test', 'Thanks'], ['misc.forsale']]
SHORT text found: [['Please', 'get', 'REAL', 'life'], ['sci.space']]
SHORT text found: [['subscribe', 'comp.graphics', 'quit'], ['comp.graphics']]
SHORT text found: [['this', 'is', 'tesrt'], ['talk.politics.misc']]
SHORT text found: [['Narrative', 'narrative', 'narrative', 'C.B'], ['talk.politics.misc']]
SHORT text found: [['test'], ['rec.motorcycles']]
SHORT text found: [['AM', 'Satan'], ['soc.religion.christian']]
SHORT text found: [['subscribe', 'grape@nswses.navy.mil'], ['comp.windows.x']]
SHORT text found: [['yxy4145@usl.edu', 'Thanks', 'lot'], ['sci.electronics']]
SHORT text found: [['David'], ['rec.sport.hockey']]
SHORT text found: [['Rawley', 'Eastwick'], ['rec.sport.baseball']]
SHORT text found: [['art'], ['comp.sys.mac.hardware']]
SHORT text found: [['this', 'is', 'test'], ['comp.sys.mac.hardware']]
SHORT text found: [['Good', 'luck'], ['rec.autos']]
SHORT text found: [['Need', 'say', 'more'], ['rec.sport.hockey']]
SHORT text found: [['critus'], ['talk.religion.misc']]
SHORT text found: [[&quot;It's&quot;, 'only', 'test', 'message'], ['rec.sport.hockey']]
SHORT text found: [['hello', 'testing'], ['rec.autos']]
SHORT text found: [['Inguiry', 'by', 'address:er1@eridan.chuvashia.su'], ['misc.forsale']]
SHORT text found: [['and', 'VGA', 'monitor', 'e-mail'], ['misc.forsale']]
SHORT text found: [['subscribe', 'min@stella.skku.ac.kr'], ['comp.windows.x']]"><pre class="notranslate"><code class="notranslate">SHORT text found: [['Not', 'this', 'again'], ['talk.politics.misc']]
SHORT text found: [['SUBSCRIBE', 'XPERT', 'skji@eve.kaist.ac.kr'], ['comp.windows.x']]
SHORT text found: [['1B', 'Career', 'DON', 'MATTINGLY'], ['rec.sport.baseball']]
SHORT text found: [['please', 'subscrive', 'me', 'rpicas@porto.inescn.pt'], ['comp.windows.x']]
SHORT text found: [['very', 'good'], ['sci.med']]
SHORT text found: [['alt.religion.spam'], ['talk.religion.misc']]
SHORT text found: [['BULLSHIT'], ['sci.med']]
SHORT text found: [['cheek'], ['rec.autos']]
SHORT text found: [['unsubscribe'], ['comp.windows.x']]
SHORT text found: [['shut', 'up', 'andi'], ['talk.politics.mideast']]
SHORT text found: [['unsubscribe'], ['comp.windows.x']]
SHORT text found: [['please', 'subscrive', 'me', 'rpicas@porto.inescn.pt'], ['comp.windows.x']]
SHORT text found: [['Brandon', 'Wise', 'bwise@nyx.cs.du.edu'], ['comp.os.ms-windows.misc']]
SHORT text found: [['unfortunately', 'not', 'Legalize', 'Freedom'], ['talk.religion.misc']]
SHORT text found: [['urn', 'll', 'he', 'uckers'], ['talk.politics.guns']]
SHORT text found: [['Test'], ['misc.forsale']]
SHORT text found: [['Harvested', 'to', 'order'], ['sci.med']]
SHORT text found: [['Does', 'Autodesk', 'has', 'BBS'], ['comp.graphics']]
SHORT text found: [['ignore'], ['rec.motorcycles']]
SHORT text found: [['exit'], ['sci.space']]
SHORT text found: [['Ditto', 'Great', 'post', 'joe.kusmierczak@mail.trincoll.edu'], ['talk.politics.guns']]
SHORT text found: [['Ken'], ['misc.forsale']]
SHORT text found: [['this', 'is', 'test', 'Erme'], ['comp.sys.mac.hardware']]
SHORT text found: [['Israel', 'Happy', '45th', 'Birthday'], ['talk.politics.mideast']]
SHORT text found: [['Just', 'kidding'], ['talk.politics.mideast']]
SHORT text found: [['Inguiry', 'by', 'address:er1@eridan.chuvashia.su'], ['misc.forsale']]
SHORT text found: [['400included', 'shipping'], ['misc.forsale']]
SHORT text found: [['Bo', 'Bilinsky'], ['rec.sport.baseball']]
SHORT text found: [['This', 'is', 'test', 'Thanks'], ['misc.forsale']]
SHORT text found: [['Please', 'get', 'REAL', 'life'], ['sci.space']]
SHORT text found: [['subscribe', 'comp.graphics', 'quit'], ['comp.graphics']]
SHORT text found: [['this', 'is', 'tesrt'], ['talk.politics.misc']]
SHORT text found: [['Narrative', 'narrative', 'narrative', 'C.B'], ['talk.politics.misc']]
SHORT text found: [['test'], ['rec.motorcycles']]
SHORT text found: [['AM', 'Satan'], ['soc.religion.christian']]
SHORT text found: [['subscribe', 'grape@nswses.navy.mil'], ['comp.windows.x']]
SHORT text found: [['yxy4145@usl.edu', 'Thanks', 'lot'], ['sci.electronics']]
SHORT text found: [['David'], ['rec.sport.hockey']]
SHORT text found: [['Rawley', 'Eastwick'], ['rec.sport.baseball']]
SHORT text found: [['art'], ['comp.sys.mac.hardware']]
SHORT text found: [['this', 'is', 'test'], ['comp.sys.mac.hardware']]
SHORT text found: [['Good', 'luck'], ['rec.autos']]
SHORT text found: [['Need', 'say', 'more'], ['rec.sport.hockey']]
SHORT text found: [['critus'], ['talk.religion.misc']]
SHORT text found: [["It's", 'only', 'test', 'message'], ['rec.sport.hockey']]
SHORT text found: [['hello', 'testing'], ['rec.autos']]
SHORT text found: [['Inguiry', 'by', 'address:er1@eridan.chuvashia.su'], ['misc.forsale']]
SHORT text found: [['and', 'VGA', 'monitor', 'e-mail'], ['misc.forsale']]
SHORT text found: [['subscribe', 'min@stella.skku.ac.kr'], ['comp.windows.x']]
</code></pre></div>
<p dir="auto">I found that <code class="notranslate">feature_selection</code> based on <code class="notranslate">f_classif</code> (i.e. F-test) can break if one feature has a constant value (e.g. all zeros).</p>
<p dir="auto">The best way to test this is to add a column of all zeros to <code class="notranslate">X</code> in the example <code class="notranslate">plot_feature_selection.py</code>, e.g.: see the column <code class="notranslate">dummy</code> below</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="# Some noisy data not correlated
E = np.random.uniform(0, 0.1, size=(len(iris.data), 20))

# Add the noisy data to the informative features
X = np.hstack((iris.data, E))

dummy = np.zeros((X.shape[0],3))
X = np.hstack((iris.data, E, dummy))
y = iris.target"><pre class="notranslate"><code class="notranslate"># Some noisy data not correlated
E = np.random.uniform(0, 0.1, size=(len(iris.data), 20))

# Add the noisy data to the informative features
X = np.hstack((iris.data, E))

dummy = np.zeros((X.shape[0],3))
X = np.hstack((iris.data, E, dummy))
y = iris.target
</code></pre></div>
<p dir="auto"><code class="notranslate">f_classif</code> already throws a warning whenever multiple columns (multiple features) are duplicates of each other. It may be a good idea to also warn the user when a feature is constant across all instances.</p>