<p dir="auto">I have several groupby problems crop up regularly at work**, and almost always have to write custom functions for operating on individual groups - which are then very slow to execute.</p>
<p dir="auto">Looking for solutions, I came across<br>
<a href="http://esantorella.com/2016/06/16/groupby/" rel="nofollow">http://esantorella.com/2016/06/16/groupby/</a><br>
and the update<br>
<a href="https://github.com/esantorella/hdfe/blob/master/groupby.py">https://github.com/esantorella/hdfe/blob/master/groupby.py</a></p>
<p dir="auto">I had to adapt my function a little bit to be compatible with numpy- instead of pandas-indexing, but in one case I tested today, this sped up my code by <em>more than a factor 100</em>, and this wasn't even for the full dataset. In fact (for ~5mio rows), the pandas-native version didn't finish in 2h, whereas the adaptation based on the code linked above ran in ~35sec.</p>
<p dir="auto">It would be very nice if such optimisations would be taken care of directly by pandas, instead of having to work around like this.</p>
<p dir="auto">** one example among many: process data from source, deduplicate based on a given signature, process further, keep deduplicated record. Updates to this record need to recalculate any time one of the constituent records has changed</p>
<h4 dir="auto">Code Sample, a copy-pastable example if possible</h4>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import pandas as pd
d = {'one' : pd.Series([1, 2, 3], index=['a', 'b', 'c'])}
df = pd.DataFrame(d)
dados.to_clipboard(index=False, header=False, line_terminator=&quot;\r\n&quot;)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">pandas</span> <span class="pl-k">as</span> <span class="pl-s1">pd</span>
<span class="pl-s1">d</span> <span class="pl-c1">=</span> {<span class="pl-s">'one'</span> : <span class="pl-s1">pd</span>.<span class="pl-v">Series</span>([<span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">3</span>], <span class="pl-s1">index</span><span class="pl-c1">=</span>[<span class="pl-s">'a'</span>, <span class="pl-s">'b'</span>, <span class="pl-s">'c'</span>])}
<span class="pl-s1">df</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">DataFrame</span>(<span class="pl-s1">d</span>)
<span class="pl-s1">dados</span>.<span class="pl-en">to_clipboard</span>(<span class="pl-s1">index</span><span class="pl-c1">=</span><span class="pl-c1">False</span>, <span class="pl-s1">header</span><span class="pl-c1">=</span><span class="pl-c1">False</span>, <span class="pl-s1">line_terminator</span><span class="pl-c1">=</span><span class="pl-s">"<span class="pl-cce">\r</span><span class="pl-cce">\n</span>"</span>)</pre></div>
<h4 dir="auto">Problem description</h4>
<p dir="auto">I just upgraded to <code class="notranslate">pandas 0.23.0</code> and this code on my program broke. This is likelly related to <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="324030680" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/21104" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/21104/hovercard" href="https://github.com/pandas-dev/pandas/issues/21104">#21104</a>.<br>
According to <code class="notranslate">DataFrame.to_clipboard</code> documentation, it would pass any <code class="notranslate">**kwargs</code> down to <code class="notranslate">DataFrame.to_csv</code>, which has the <code class="notranslate">line_terminator</code> argument and used to work fine.</p>
<p dir="auto">It seems that it now uses <code class="notranslate">DataFrame.to_string</code> albeit undocumented. This function does not support the same set of arguments as <code class="notranslate">DataFrame.to_csv</code> which is bound to cause some bugs.</p>
<p dir="auto">I get the following error:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; df.to_clipboard(index=False, header=False, line_terminator=&quot;\r\n&quot;)
Traceback (most recent call last):
  File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;
  File &quot;C:\Users\u5en\ArqProg\Anaconda3\lib\site-packages\pandas\core\generic.py&quot;, line 2246, in to_clipboard
    clipboards.to_clipboard(self, excel=excel, sep=sep, **kwargs)
  File &quot;C:\Users\u5en\ArqProg\Anaconda3\lib\site-packages\pandas\io\clipboards.py&quot;, line 117, in to_clipboard
    objstr = obj.to_string(**kwargs)
TypeError: to_string() got an unexpected keyword argument 'line_terminator'"><pre class="notranslate"><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">df</span>.<span class="pl-en">to_clipboard</span>(<span class="pl-s1">index</span><span class="pl-c1">=</span><span class="pl-c1">False</span>, <span class="pl-s1">header</span><span class="pl-c1">=</span><span class="pl-c1">False</span>, <span class="pl-s1">line_terminator</span><span class="pl-c1">=</span><span class="pl-s">"<span class="pl-cce">\r</span><span class="pl-cce">\n</span>"</span>)
<span class="pl-v">Traceback</span> (<span class="pl-s1">most</span> <span class="pl-s1">recent</span> <span class="pl-s1">call</span> <span class="pl-s1">last</span>):
  <span class="pl-v">File</span> <span class="pl-s">"&lt;stdin&gt;"</span>, <span class="pl-s1">line</span> <span class="pl-c1">1</span>, <span class="pl-s1">in</span> <span class="pl-c1">&lt;</span><span class="pl-s1">module</span><span class="pl-c1">&gt;</span>
  <span class="pl-v">File</span> <span class="pl-s">"C:\Users\u5en\ArqProg\Anaconda3\lib\site-packages\pandas\core\generic.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">2246</span>, <span class="pl-s1">in</span> <span class="pl-s1">to_clipboard</span>
    <span class="pl-s1">clipboards</span>.<span class="pl-en">to_clipboard</span>(<span class="pl-s1">self</span>, <span class="pl-s1">excel</span><span class="pl-c1">=</span><span class="pl-s1">excel</span>, <span class="pl-s1">sep</span><span class="pl-c1">=</span><span class="pl-s1">sep</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)
  <span class="pl-v">File</span> <span class="pl-s">"C:\Users\u5en\ArqProg\Anaconda3\lib\site-packages\pandas\io\clipboards.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">117</span>, <span class="pl-s1">in</span> <span class="pl-s1">to_clipboard</span>
    <span class="pl-s1">objstr</span> <span class="pl-c1">=</span> <span class="pl-s1">obj</span>.<span class="pl-en">to_string</span>(<span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)
<span class="pl-v">TypeError</span>: <span class="pl-en">to_string</span>() <span class="pl-s1">got</span> <span class="pl-s1">an</span> <span class="pl-s1">unexpected</span> <span class="pl-s1">keyword</span> <span class="pl-s1">argument</span> <span class="pl-s">'line_terminator'</span></pre></div>
<p dir="auto">I also note that <code class="notranslate">Series.to_clipboard</code> seems to <em><strong>silently ignore</strong></em> the <code class="notranslate">line_terminator</code> parameter, which also is not good.</p>
<h4 dir="auto">Output of <code class="notranslate">pd.show_versions()</code></h4>
<details>
<h2 dir="auto">[paste the output of <code class="notranslate">pd.show_versions()</code> here below this line]<br>
INSTALLED VERSIONS</h2>
<p dir="auto">commit: None<br>
python: 3.6.5.final.0<br>
python-bits: 32<br>
OS: Windows<br>
OS-release: 7<br>
machine: AMD64<br>
processor: Intel64 Family 6 Model 42 Stepping 7, GenuineIntel<br>
byteorder: little<br>
LC_ALL: None<br>
LANG: None<br>
LOCALE: None.None</p>
<p dir="auto">pandas: 0.23.0<br>
pytest: 3.5.1<br>
pip: 10.0.1<br>
setuptools: 39.1.0<br>
Cython: 0.28.2<br>
numpy: 1.14.3<br>
scipy: 1.1.0<br>
pyarrow: None<br>
xarray: None<br>
IPython: 6.4.0<br>
sphinx: 1.7.4<br>
patsy: 0.5.0<br>
dateutil: 2.7.3<br>
pytz: 2018.4<br>
blosc: None<br>
bottleneck: 1.2.1<br>
tables: 3.4.3<br>
numexpr: 2.6.5<br>
feather: None<br>
matplotlib: 2.2.2<br>
openpyxl: 2.5.3<br>
xlrd: 1.1.0<br>
xlwt: 1.3.0<br>
xlsxwriter: 1.0.4<br>
lxml: 4.2.1<br>
bs4: 4.6.0<br>
html5lib: 1.0.1<br>
sqlalchemy: 1.2.7<br>
pymysql: None<br>
psycopg2: None<br>
jinja2: 2.10<br>
s3fs: None<br>
fastparquet: None<br>
pandas_gbq: None<br>
pandas_datareader: None</p>
</details>