<p dir="auto">I'm grouping rows by column investor id. within apply, when a group has more than 1 row, i'm copying a value in the second row in the group into the a column of the first row in that group. I'm using groupby with apply but the assembled result is incorrect - all rows except row 1 show the same investor B - seems like a bug?</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="In [1]: import pandas as pd
In [2]: import numpy as np

In [3]: df=pd.DataFrame({'InvestorID':[6,6,17,17,19,19,30,40],
   ...:                  'Investor':['AAA','AAA','BBB','BBB','CCC','CCC','EEE','FFF'],
   ...:                  'CurrentPeriodEnd':['7/31/2014','5/31/2014','7/31/2014','5/31/2014','7/31/2014','5/31/2014','7/31/2014','7/31/2014'],
   ...:                  'PriorPeriodEnd':np.nan*8})

In [4]: df
Out[4]: 
  CurrentPeriodEnd Investor  InvestorID  PriorPeriodEnd
0        7/31/2014      AAA           6             NaN
1        5/31/2014      AAA           6             NaN
2        7/31/2014      BBB          17             NaN
3        5/31/2014      BBB          17             NaN
4        7/31/2014      CCC          19             NaN
5        5/31/2014      CCC          19             NaN
6        7/31/2014      EEE          30             NaN
7        7/31/2014      FFF          40             NaN"><pre class="notranslate"><code class="notranslate">In [1]: import pandas as pd
In [2]: import numpy as np

In [3]: df=pd.DataFrame({'InvestorID':[6,6,17,17,19,19,30,40],
   ...:                  'Investor':['AAA','AAA','BBB','BBB','CCC','CCC','EEE','FFF'],
   ...:                  'CurrentPeriodEnd':['7/31/2014','5/31/2014','7/31/2014','5/31/2014','7/31/2014','5/31/2014','7/31/2014','7/31/2014'],
   ...:                  'PriorPeriodEnd':np.nan*8})

In [4]: df
Out[4]: 
  CurrentPeriodEnd Investor  InvestorID  PriorPeriodEnd
0        7/31/2014      AAA           6             NaN
1        5/31/2014      AAA           6             NaN
2        7/31/2014      BBB          17             NaN
3        5/31/2014      BBB          17             NaN
4        7/31/2014      CCC          19             NaN
5        5/31/2014      CCC          19             NaN
6        7/31/2014      EEE          30             NaN
7        7/31/2014      FFF          40             NaN
</code></pre></div>
<p dir="auto">this function updates the first positional row in a group with values from the second row<br>
where a group has 2 rows or more</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="In [5]: def f(x):
   ...:     if len(x) &gt;1:
   ...:         x.iloc[0,x.columns.get_loc('PriorPeriodEnd')]=x.iloc[1,x.columns.get_loc('CurrentPeriodEnd')] 
   ...:     return x.iloc[0]

In [6]: df.groupby(['InvestorID']).apply(f).reset_index(drop=True)"><pre class="notranslate"><code class="notranslate">In [5]: def f(x):
   ...:     if len(x) &gt;1:
   ...:         x.iloc[0,x.columns.get_loc('PriorPeriodEnd')]=x.iloc[1,x.columns.get_loc('CurrentPeriodEnd')] 
   ...:     return x.iloc[0]

In [6]: df.groupby(['InvestorID']).apply(f).reset_index(drop=True)
</code></pre></div>
<p dir="auto">this is not an expected result - the investor column should not have repeating BBB values</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Out[6]: 
  CurrentPeriodEnd Investor  InvestorID PriorPeriodEnd
0        7/31/2014      AAA           6      5/31/2014
1        7/31/2014      BBB          17      5/31/2014
2        7/31/2014      BBB          19      5/31/2014
3        7/31/2014      BBB          30      5/31/2014
4        7/31/2014      BBB          40      5/31/2014"><pre class="notranslate"><code class="notranslate">Out[6]: 
  CurrentPeriodEnd Investor  InvestorID PriorPeriodEnd
0        7/31/2014      AAA           6      5/31/2014
1        7/31/2014      BBB          17      5/31/2014
2        7/31/2014      BBB          19      5/31/2014
3        7/31/2014      BBB          30      5/31/2014
4        7/31/2014      BBB          40      5/31/2014
</code></pre></div>
<p dir="auto">the function below copies the first row first before making updates and it works as expected</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="In [7]: def f(x):
   ...:     top_row=pd.Series(x.iloc[0])
   ...:     if len(x) &gt;1:
   ...:         top_row.iloc[x.columns.get_loc('PriorPeriodEnd')]=x.iloc[1,x.columns.get_loc('CurrentPeriodEnd')] 
   ...:     return top_row"><pre class="notranslate"><code class="notranslate">In [7]: def f(x):
   ...:     top_row=pd.Series(x.iloc[0])
   ...:     if len(x) &gt;1:
   ...:         top_row.iloc[x.columns.get_loc('PriorPeriodEnd')]=x.iloc[1,x.columns.get_loc('CurrentPeriodEnd')] 
   ...:     return top_row
</code></pre></div>
<p dir="auto">this is what the result should look like</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="In [8]: df.groupby(['InvestorID']).apply(f).reset_index(drop=True)
Out[8]: 
  CurrentPeriodEnd Investor  InvestorID PriorPeriodEnd
0        7/31/2014      AAA           6      5/31/2014
1        7/31/2014      BBB          17      5/31/2014
2        7/31/2014      CCC          19      5/31/2014
3        7/31/2014      EEE          30            NaN
4        7/31/2014      FFF          40            NaN"><pre class="notranslate"><code class="notranslate">In [8]: df.groupby(['InvestorID']).apply(f).reset_index(drop=True)
Out[8]: 
  CurrentPeriodEnd Investor  InvestorID PriorPeriodEnd
0        7/31/2014      AAA           6      5/31/2014
1        7/31/2014      BBB          17      5/31/2014
2        7/31/2014      CCC          19      5/31/2014
3        7/31/2014      EEE          30            NaN
4        7/31/2014      FFF          40            NaN
</code></pre></div>
<p dir="auto">When trying to insert/append a subclass (or composition) of a pandas Series into a DataFrame, any and all of the 'extra' functions that come with my subclass (or composition) are stripped and a Series is created:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="In [7]: df = read_csv('some/data/from/file.csv')

In [8]: sp = SpatialSeries(df.the_geom) # SpatialSeries is subclass, the_geom is spatial location (WKT)

In [9]: type(sp)
Out[9]: spseries.SpatialSeries

In [10]: type(df)
Out[10]: pandas.core.frame.DataFrame

In [11]: df['geoms'] = sp

In [12]: type(df['geoms'])
Out[12]: pandas.core.series.Series"><pre class="notranslate"><span class="pl-v">In</span> [<span class="pl-c1">7</span>]: <span class="pl-s1">df</span> <span class="pl-c1">=</span> <span class="pl-en">read_csv</span>(<span class="pl-s">'some/data/from/file.csv'</span>)

<span class="pl-v">In</span> [<span class="pl-c1">8</span>]: <span class="pl-s1">sp</span> <span class="pl-c1">=</span> <span class="pl-v">SpatialSeries</span>(<span class="pl-s1">df</span>.<span class="pl-s1">the_geom</span>) <span class="pl-c"># SpatialSeries is subclass, the_geom is spatial location (WKT)</span>

<span class="pl-v">In</span> [<span class="pl-c1">9</span>]: <span class="pl-en">type</span>(<span class="pl-s1">sp</span>)
<span class="pl-v">Out</span>[<span class="pl-c1">9</span>]: <span class="pl-s1">spseries</span>.<span class="pl-v">SpatialSeries</span>

<span class="pl-v">In</span> [<span class="pl-c1">10</span>]: <span class="pl-en">type</span>(<span class="pl-s1">df</span>)
<span class="pl-v">Out</span>[<span class="pl-c1">10</span>]: <span class="pl-s1">pandas</span>.<span class="pl-s1">core</span>.<span class="pl-s1">frame</span>.<span class="pl-v">DataFrame</span>

<span class="pl-v">In</span> [<span class="pl-c1">11</span>]: <span class="pl-s1">df</span>[<span class="pl-s">'geoms'</span>] <span class="pl-c1">=</span> <span class="pl-s1">sp</span>

<span class="pl-v">In</span> [<span class="pl-c1">12</span>]: <span class="pl-en">type</span>(<span class="pl-s1">df</span>[<span class="pl-s">'geoms'</span>])
<span class="pl-v">Out</span>[<span class="pl-c1">12</span>]: <span class="pl-s1">pandas</span>.<span class="pl-s1">core</span>.<span class="pl-s1">series</span>.<span class="pl-v">Series</span></pre></div>
<p dir="auto">I suspect that for the most part, this kind of behaviour is useful, however, I need the extra functions and classes associated with SpatialSeries, and I'd rather not have to subclass DataFrame to create a special DataFrame that allows this. It looks like the culprit is here in frame.py at lines 1761-1772:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="    def _set_item(self, key, value):
        &quot;&quot;&quot;
        Add series to DataFrame in specified column.

        If series is a numpy-array (not a Series/TimeSeries), it must be the
        same length as the DataFrame's index or an error will be thrown.

        Series/TimeSeries will be conformed to the DataFrame's index to
        ensure homogeneity.
        &quot;&quot;&quot;
        value = self._sanitize_column(key, value)
        NDFrame._set_item(self, key, value)"><pre class="notranslate">    <span class="pl-k">def</span> <span class="pl-en">_set_item</span>(<span class="pl-s1">self</span>, <span class="pl-s1">key</span>, <span class="pl-s1">value</span>):
        <span class="pl-s">"""</span>
<span class="pl-s">        Add series to DataFrame in specified column.</span>
<span class="pl-s"></span>
<span class="pl-s">        If series is a numpy-array (not a Series/TimeSeries), it must be the</span>
<span class="pl-s">        same length as the DataFrame's index or an error will be thrown.</span>
<span class="pl-s"></span>
<span class="pl-s">        Series/TimeSeries will be conformed to the DataFrame's index to</span>
<span class="pl-s">        ensure homogeneity.</span>
<span class="pl-s">        """</span>
        <span class="pl-s1">value</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">_sanitize_column</span>(<span class="pl-s1">key</span>, <span class="pl-s1">value</span>)
        <span class="pl-v">NDFrame</span>.<span class="pl-en">_set_item</span>(<span class="pl-s1">self</span>, <span class="pl-s1">key</span>, <span class="pl-s1">value</span>)</pre></div>
<p dir="auto">I particular, I'm looking at <code class="notranslate">value = self._sanitize_column(key, value)</code>, which appears to use <code class="notranslate">np.asarray(value)</code> before it returns the input array (even if the input column is a Series). Is there any way to avoid this behaviour? Or alternatively, a better way to implement this so that useful subclasses can be used within a DataFrame? I hope I'm not missing something simple/vital here?</p>
<p dir="auto">FYI:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="In [13]: pandas.__version__
Out[13]: '0.8.0b1'"><pre class="notranslate"><span class="pl-v">In</span> [<span class="pl-c1">13</span>]: <span class="pl-s1">pandas</span>.<span class="pl-s1">__version__</span>
<span class="pl-v">Out</span>[<span class="pl-c1">13</span>]: <span class="pl-s">'0.8.0b1'</span></pre></div>