<p dir="auto">With pandas v0.8.0b1</p>
<p dir="auto">In [90]: u<br>
Out[90]:<br>
2012-05-08 01:45:00     86<br>
2012-05-08 01:50:00    170<br>
2012-05-08 01:55:00    130<br>
2012-05-08 02:00:00    206<br>
2012-05-08 02:05:00     52<br>
2012-05-08 02:10:00      4<br>
Freq: 5T</p>
<p dir="auto">In [91]: u.append(u)</p>
<p dir="auto">Traceback (most recent call last):<br>
File "", line 1, in <br>
File "C:\Python27\lib\site-packages\pandas\core\series.py", line 1566, in append<br>
return concat(to_concat, ignore_index=False, verify_integrity=True)<br>
File "C:\Python27\lib\site-packages\pandas\tools\merge.py", line 836, in concat<br>
verify_integrity=verify_integrity)<br>
File "C:\Python27\lib\site-packages\pandas\tools\merge.py", line 895, in <strong>init</strong><br>
self.new_axes = self._get_new_axes()<br>
File "C:\Python27\lib\site-packages\pandas\tools\merge.py", line 1066, in _get_new_axes<br>
concat_axis = self._get_concat_axis()<br>
File "C:\Python27\lib\site-packages\pandas\tools\merge.py", line 1097, in _get_concat_axis<br>
self._maybe_check_integrity(concat_axis)<br>
File "C:\Python27\lib\site-packages\pandas\tools\merge.py", line 1106, in _maybe_check_integrity<br>
% str(overlap))<br>
Exception: Indexes have overlapping values: [1970-01-16 225:45:00, 1970-01-16 225:50:00, 1970-01-16 225:55:00, 1970-01-16 226:00:00, 1970-01-16 226:05:00, 1970-01-16 226:10:00]</p>
<h4 dir="auto">Code Sample, a copy-pastable example if possible</h4>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="# Your code here
import pandas as pd
import numpy as np


def processCols(x):
    x_dtype = str(x.dtype)
    if x_dtype=='float64':
        y = x.fillna(x.mean())  # replace nan in col with mean
        return y
    elif x_dtype=='category':
        x_top = x.describe().top
        y = x.fillna(str(x_top))  # replace nan in col with most occuring cat
        return y
    elif x_dtype=='int64': #do nothing for now
        return x
    else:  # do nothing
        return x


# Test snippet

#pid,typeid,bnum,bcat
#41,A,1.0,3D
#42,A,2.0,3D
#43,B,4.5,3E
#44,A,4.0,3D
#45,A,5.0,4D
#46,B,3.0,3E
#47,A,1.1,4D
#48,B,2.0,3F
#49,A,,
#50,A,,
#51,B,,
#52,B,,

rawData = pd.DataFrame({'pid' : np.array([41,42,43,44,45,46,47,48,49,50,51,52], dtype='int64'),
                        'typeid' : pd.Categorical([&quot;A&quot;,&quot;A&quot;,&quot;B&quot;,&quot;A&quot;,&quot;A&quot;,&quot;B&quot;,&quot;A&quot;,&quot;B&quot;,&quot;A&quot;,&quot;A&quot;,&quot;B&quot;,&quot;B&quot;]),
                        'bnum' : np.array([1,2,4.5,4,5,3,1.1,2,np.nan,np.nan,np.nan,np.nan],dtype='float64'),
                        'bcat' : pd.Categorical([&quot;3D&quot;,&quot;3D&quot;,&quot;3E&quot;,&quot;3D&quot;,&quot;4D&quot;,&quot;3E&quot;,&quot;4D&quot;,&quot;3F&quot;,np.nan,np.nan,np.nan,np.nan])
                       })
                                       

groupedBy = rawData.groupby('typeid')
        
for name, group in groupedBy:
    print('By Groups:')
    print(name)
    print('Before nan correction')
    print(group)
    transformed = group.transform(processCols)
    print()
    print('After nan correction')
    print(transformed)
    print()
        
# Now process all at once to compare 
print()     
print('transformed all at once')  
transformed_all_at_once = groupedBy.transform(processCols)
print(transformed_all_at_once)"><pre class="notranslate"><span class="pl-c"># Your code here</span>
<span class="pl-k">import</span> <span class="pl-s1">pandas</span> <span class="pl-k">as</span> <span class="pl-s1">pd</span>
<span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>


<span class="pl-k">def</span> <span class="pl-en">processCols</span>(<span class="pl-s1">x</span>):
    <span class="pl-s1">x_dtype</span> <span class="pl-c1">=</span> <span class="pl-en">str</span>(<span class="pl-s1">x</span>.<span class="pl-s1">dtype</span>)
    <span class="pl-k">if</span> <span class="pl-s1">x_dtype</span><span class="pl-c1">==</span><span class="pl-s">'float64'</span>:
        <span class="pl-s1">y</span> <span class="pl-c1">=</span> <span class="pl-s1">x</span>.<span class="pl-en">fillna</span>(<span class="pl-s1">x</span>.<span class="pl-en">mean</span>())  <span class="pl-c"># replace nan in col with mean</span>
        <span class="pl-k">return</span> <span class="pl-s1">y</span>
    <span class="pl-k">elif</span> <span class="pl-s1">x_dtype</span><span class="pl-c1">==</span><span class="pl-s">'category'</span>:
        <span class="pl-s1">x_top</span> <span class="pl-c1">=</span> <span class="pl-s1">x</span>.<span class="pl-en">describe</span>().<span class="pl-s1">top</span>
        <span class="pl-s1">y</span> <span class="pl-c1">=</span> <span class="pl-s1">x</span>.<span class="pl-en">fillna</span>(<span class="pl-en">str</span>(<span class="pl-s1">x_top</span>))  <span class="pl-c"># replace nan in col with most occuring cat</span>
        <span class="pl-k">return</span> <span class="pl-s1">y</span>
    <span class="pl-k">elif</span> <span class="pl-s1">x_dtype</span><span class="pl-c1">==</span><span class="pl-s">'int64'</span>: <span class="pl-c">#do nothing for now</span>
        <span class="pl-k">return</span> <span class="pl-s1">x</span>
    <span class="pl-k">else</span>:  <span class="pl-c"># do nothing</span>
        <span class="pl-k">return</span> <span class="pl-s1">x</span>


<span class="pl-c"># Test snippet</span>

<span class="pl-c">#pid,typeid,bnum,bcat</span>
<span class="pl-c">#41,A,1.0,3D</span>
<span class="pl-c">#42,A,2.0,3D</span>
<span class="pl-c">#43,B,4.5,3E</span>
<span class="pl-c">#44,A,4.0,3D</span>
<span class="pl-c">#45,A,5.0,4D</span>
<span class="pl-c">#46,B,3.0,3E</span>
<span class="pl-c">#47,A,1.1,4D</span>
<span class="pl-c">#48,B,2.0,3F</span>
<span class="pl-c">#49,A,,</span>
<span class="pl-c">#50,A,,</span>
<span class="pl-c">#51,B,,</span>
<span class="pl-c">#52,B,,</span>

<span class="pl-s1">rawData</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">DataFrame</span>({<span class="pl-s">'pid'</span> : <span class="pl-s1">np</span>.<span class="pl-en">array</span>([<span class="pl-c1">41</span>,<span class="pl-c1">42</span>,<span class="pl-c1">43</span>,<span class="pl-c1">44</span>,<span class="pl-c1">45</span>,<span class="pl-c1">46</span>,<span class="pl-c1">47</span>,<span class="pl-c1">48</span>,<span class="pl-c1">49</span>,<span class="pl-c1">50</span>,<span class="pl-c1">51</span>,<span class="pl-c1">52</span>], <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s">'int64'</span>),
                        <span class="pl-s">'typeid'</span> : <span class="pl-s1">pd</span>.<span class="pl-v">Categorical</span>([<span class="pl-s">"A"</span>,<span class="pl-s">"A"</span>,<span class="pl-s">"B"</span>,<span class="pl-s">"A"</span>,<span class="pl-s">"A"</span>,<span class="pl-s">"B"</span>,<span class="pl-s">"A"</span>,<span class="pl-s">"B"</span>,<span class="pl-s">"A"</span>,<span class="pl-s">"A"</span>,<span class="pl-s">"B"</span>,<span class="pl-s">"B"</span>]),
                        <span class="pl-s">'bnum'</span> : <span class="pl-s1">np</span>.<span class="pl-en">array</span>([<span class="pl-c1">1</span>,<span class="pl-c1">2</span>,<span class="pl-c1">4.5</span>,<span class="pl-c1">4</span>,<span class="pl-c1">5</span>,<span class="pl-c1">3</span>,<span class="pl-c1">1.1</span>,<span class="pl-c1">2</span>,<span class="pl-s1">np</span>.<span class="pl-s1">nan</span>,<span class="pl-s1">np</span>.<span class="pl-s1">nan</span>,<span class="pl-s1">np</span>.<span class="pl-s1">nan</span>,<span class="pl-s1">np</span>.<span class="pl-s1">nan</span>],<span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s">'float64'</span>),
                        <span class="pl-s">'bcat'</span> : <span class="pl-s1">pd</span>.<span class="pl-v">Categorical</span>([<span class="pl-s">"3D"</span>,<span class="pl-s">"3D"</span>,<span class="pl-s">"3E"</span>,<span class="pl-s">"3D"</span>,<span class="pl-s">"4D"</span>,<span class="pl-s">"3E"</span>,<span class="pl-s">"4D"</span>,<span class="pl-s">"3F"</span>,<span class="pl-s1">np</span>.<span class="pl-s1">nan</span>,<span class="pl-s1">np</span>.<span class="pl-s1">nan</span>,<span class="pl-s1">np</span>.<span class="pl-s1">nan</span>,<span class="pl-s1">np</span>.<span class="pl-s1">nan</span>])
                       })
                                       

<span class="pl-s1">groupedBy</span> <span class="pl-c1">=</span> <span class="pl-s1">rawData</span>.<span class="pl-en">groupby</span>(<span class="pl-s">'typeid'</span>)
        
<span class="pl-k">for</span> <span class="pl-s1">name</span>, <span class="pl-s1">group</span> <span class="pl-c1">in</span> <span class="pl-s1">groupedBy</span>:
    <span class="pl-en">print</span>(<span class="pl-s">'By Groups:'</span>)
    <span class="pl-en">print</span>(<span class="pl-s1">name</span>)
    <span class="pl-en">print</span>(<span class="pl-s">'Before nan correction'</span>)
    <span class="pl-en">print</span>(<span class="pl-s1">group</span>)
    <span class="pl-s1">transformed</span> <span class="pl-c1">=</span> <span class="pl-s1">group</span>.<span class="pl-en">transform</span>(<span class="pl-s1">processCols</span>)
    <span class="pl-en">print</span>()
    <span class="pl-en">print</span>(<span class="pl-s">'After nan correction'</span>)
    <span class="pl-en">print</span>(<span class="pl-s1">transformed</span>)
    <span class="pl-en">print</span>()
        
<span class="pl-c"># Now process all at once to compare </span>
<span class="pl-en">print</span>()     
<span class="pl-en">print</span>(<span class="pl-s">'transformed all at once'</span>)  
<span class="pl-s1">transformed_all_at_once</span> <span class="pl-c1">=</span> <span class="pl-s1">groupedBy</span>.<span class="pl-en">transform</span>(<span class="pl-s1">processCols</span>)
<span class="pl-en">print</span>(<span class="pl-s1">transformed_all_at_once</span>)</pre></div>
<h4 dir="auto">Problem description</h4>
<p dir="auto">When using groupby.transform and a custom function on a full data frame, the groupby key does not get returned in the final glued-together output.  However, if this same process is done by iterating over the groups, the individual data frames do return the groupby key.  If the groupby key is getting removed, then it requires an external join to reintroduce it.  However, the individual replacements for the nan (which is the what the custom function does) is correct in both cases.  Thanks.</p>
<p dir="auto">[this should explain <strong>why</strong> the current behaviour is a problem and why the expected output is a better solution.]</p>
<p dir="auto"><strong>Note</strong>: We receive a lot of issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates!</p>
<p dir="auto"><strong>Note</strong>: Many problems can be resolved by simply upgrading <code class="notranslate">pandas</code> to the latest version. Before submitting, please check if that solution works for you. If possible, you may want to check if <code class="notranslate">master</code> addresses this issue, but that is not necessary.</p>
<h4 dir="auto">Expected Output</h4>
<h4 dir="auto">Output of <code class="notranslate">pd.show_versions()</code></h4>
<details>
<h2 dir="auto">[paste the output of <code class="notranslate">pd.show_versions()</code> here below this line]<br>
INSTALLED VERSIONS</h2>
<p dir="auto">commit: None<br>
python: 3.5.3.final.0<br>
python-bits: 64<br>
OS: Darwin<br>
OS-release: 16.7.0<br>
machine: x86_64<br>
processor: i386<br>
byteorder: little<br>
LC_ALL: None<br>
LANG: None<br>
LOCALE: None.None</p>
<p dir="auto">pandas: 0.20.3<br>
pytest: None<br>
pip: 9.0.1<br>
setuptools: 36.0.1<br>
Cython: None<br>
numpy: 1.13.1<br>
scipy: 0.19.1<br>
xarray: None<br>
IPython: 6.1.0<br>
sphinx: 1.6.2<br>
patsy: None<br>
dateutil: 2.6.1<br>
pytz: 2017.2<br>
blosc: None<br>
bottleneck: None<br>
tables: None<br>
numexpr: None<br>
feather: None<br>
matplotlib: 2.0.2<br>
openpyxl: None<br>
xlrd: None<br>
xlwt: None<br>
xlsxwriter: None<br>
lxml: None<br>
bs4: None<br>
html5lib: 0.9999999<br>
sqlalchemy: None<br>
pymysql: None<br>
psycopg2: None<br>
jinja2: 2.9.6<br>
s3fs: None<br>
pandas_gbq: None<br>
pandas_datareader: None</p>
</details>