<p dir="auto">Hi all,</p>
<p dir="auto">I have a strange issue with pandas 0.9, I think it's a bug.  I'm trying to use dataframe.join() and it works well on a random dataframe, but not on a dataframe created from my simulation result.  The code below shows that join() on the second dataframe blows up the index and the result is completely wrong.</p>
<p dir="auto">To run this code you need this file: <a href="https://dl.dropbox.com/u/6200325/mydf.dataframe" rel="nofollow">https://dl.dropbox.com/u/6200325/mydf.dataframe</a> in your work folder.  The script below can also be downloaded here: <a href="https://dl.dropbox.com/u/6200325/BugJoin.py" rel="nofollow">https://dl.dropbox.com/u/6200325/BugJoin.py</a></p>
<p dir="auto">This is the result I get:</p>
<p dir="auto">In [17]: run -i 'C:\Workspace\Python\Tests\BugJoin.py'</p>
<p dir="auto">Before:<br>
&lt;class 'pandas.core.frame.DataFrame'&gt;<br>
DatetimeIndex: 100000 entries, 2012-01-01 00:00:00 to 2023-05-29 15:00:00<br>
Freq: H<br>
Empty DataFrame</p>
<p dir="auto">&lt;class 'pandas.core.frame.DataFrame'&gt;<br>
DatetimeIndex: 100000 entries, 2012-01-01 00:00:00 to 2023-05-29 15:00:00<br>
Freq: H<br>
Data columns:<br>
0    100000  non-null values<br>
dtypes: float64(1)</p>
<p dir="auto">After:<br>
&lt;class 'pandas.core.frame.DataFrame'&gt;<br>
DatetimeIndex: 100000 entries, 2012-01-01 00:00:00 to 2023-05-29 15:00:00<br>
Freq: H<br>
Data columns:<br>
0    100000  non-null values<br>
dtypes: float64(1)</p>
<p dir="auto">Before:<br>
&lt;class 'pandas.core.frame.DataFrame'&gt;<br>
DatetimeIndex: 108355 entries, 2010-01-01 00:00:00 to 2011-01-01 00:00:00<br>
Empty DataFrame</p>
<p dir="auto">&lt;class 'pandas.core.frame.DataFrame'&gt;<br>
DatetimeIndex: 108355 entries, 2010-01-01 00:00:00 to 2011-01-01 00:00:00<br>
Data columns:<br>
SID0000    108355  non-null values<br>
dtypes: float64(1)</p>
<p dir="auto">After:<br>
&lt;class 'pandas.core.frame.DataFrame'&gt;<br>
DatetimeIndex: 4054807 entries, 2010-01-09 16:00:00 to 2010-05-17 15:55:42<br>
Data columns:<br>
SID0000    4054807  non-null values<br>
dtypes: float64(1)</p>
<p dir="auto">This is the code from the script:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import numpy as np
import pandas as pd
from scipy.integrate import cumtrapz

df1=pd.DataFrame(np.random.rand(1e5), 
     index=pd.date_range('2012-01-01', freq='H', periods=1e5))

df2=pd.load('mydf.dataframe')

for dataframe in [df1, df2]:

    cum = pd.DataFrame(index=dataframe.index)
    for c in dataframe.columns:
        # we need to remove the empty values for the cumtrapz function to work
        ts = dataframe[c].dropna()

        tscum = pd.DataFrame(data=cumtrapz(ts.values, ts.index.asi8/1e9, initial=0),
                         index=ts.index, 
                         columns=[c])
        print '\nBefore:'
        print cum, '\n'
        print tscum, '\n'

        cum=cum.join(tscum, how='left')

        print 'After:'
        print cum"><pre class="notranslate"><code class="notranslate">import numpy as np
import pandas as pd
from scipy.integrate import cumtrapz

df1=pd.DataFrame(np.random.rand(1e5), 
     index=pd.date_range('2012-01-01', freq='H', periods=1e5))

df2=pd.load('mydf.dataframe')

for dataframe in [df1, df2]:

    cum = pd.DataFrame(index=dataframe.index)
    for c in dataframe.columns:
        # we need to remove the empty values for the cumtrapz function to work
        ts = dataframe[c].dropna()

        tscum = pd.DataFrame(data=cumtrapz(ts.values, ts.index.asi8/1e9, initial=0),
                         index=ts.index, 
                         columns=[c])
        print '\nBefore:'
        print cum, '\n'
        print tscum, '\n'

        cum=cum.join(tscum, how='left')

        print 'After:'
        print cum
</code></pre></div>
<p dir="auto">I propose that <code class="notranslate">itertuples()</code> should return <code class="notranslate">collections.namedtuple</code> objects, a drop-in replacement for the standard tuple but with the benefit of having named fields. I have tested the following with Python 3.4 (only slight changes compared to the current implementation).</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="def itertuples(self, index=True):
    arrays = []
        if index:
            arrays.append(self.index)
            fields = [&quot;Index&quot;] + list(self.columns)
        else:
            fields = self.columns

        itertuple = collections.namedtuple(&quot;Itertuple&quot;, fields, rename=True)

        # use integer indexing because of possible duplicate column names
        arrays.extend(self.iloc[:, k] for k in range(len(self.columns)))

        return (itertuple(*row) for row in zip(*arrays))"><pre class="notranslate"><span class="pl-k">def</span> <span class="pl-en">itertuples</span>(<span class="pl-s1">self</span>, <span class="pl-s1">index</span><span class="pl-c1">=</span><span class="pl-c1">True</span>):
    <span class="pl-s1">arrays</span> <span class="pl-c1">=</span> []
        <span class="pl-k">if</span> <span class="pl-s1">index</span>:
            <span class="pl-s1">arrays</span>.<span class="pl-en">append</span>(<span class="pl-s1">self</span>.<span class="pl-s1">index</span>)
            <span class="pl-s1">fields</span> <span class="pl-c1">=</span> [<span class="pl-s">"Index"</span>] <span class="pl-c1">+</span> <span class="pl-en">list</span>(<span class="pl-s1">self</span>.<span class="pl-s1">columns</span>)
        <span class="pl-k">else</span>:
            <span class="pl-s1">fields</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-s1">columns</span>

        <span class="pl-s1">itertuple</span> <span class="pl-c1">=</span> <span class="pl-s1">collections</span>.<span class="pl-en">namedtuple</span>(<span class="pl-s">"Itertuple"</span>, <span class="pl-s1">fields</span>, <span class="pl-s1">rename</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)

        <span class="pl-c"># use integer indexing because of possible duplicate column names</span>
        <span class="pl-s1">arrays</span>.<span class="pl-en">extend</span>(<span class="pl-s1">self</span>.<span class="pl-s1">iloc</span>[:, <span class="pl-s1">k</span>] <span class="pl-k">for</span> <span class="pl-s1">k</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-en">len</span>(<span class="pl-s1">self</span>.<span class="pl-s1">columns</span>)))

        <span class="pl-k">return</span> (<span class="pl-en">itertuple</span>(<span class="pl-c1">*</span><span class="pl-s1">row</span>) <span class="pl-k">for</span> <span class="pl-s1">row</span> <span class="pl-c1">in</span> <span class="pl-en">zip</span>(<span class="pl-c1">*</span><span class="pl-s1">arrays</span>))</pre></div>
<p dir="auto">Example</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="In [3]: df = pd.DataFrame({'col1': [1, 2], 'col2': [0.1, 0.2]}, index=['a', 'b'])

In [4]: for row in df.itertuples():
   ...:     print(row)
   ...:     
Itertuple(Index='a', col1=1, col2=0.10000000000000001)
Itertuple(Index='b', col1=2, col2=0.20000000000000001)

In [5]: row.Index, row.col1, row.col2
Out[5]: ('b', 2, 0.20000000000000001)"><pre class="notranslate"><code class="notranslate">In [3]: df = pd.DataFrame({'col1': [1, 2], 'col2': [0.1, 0.2]}, index=['a', 'b'])

In [4]: for row in df.itertuples():
   ...:     print(row)
   ...:     
Itertuple(Index='a', col1=1, col2=0.10000000000000001)
Itertuple(Index='b', col1=2, col2=0.20000000000000001)

In [5]: row.Index, row.col1, row.col2
Out[5]: ('b', 2, 0.20000000000000001)
</code></pre></div>
<p dir="auto">There is no performance overhead. I'm not sure about the compatibility for older versions of Python, though. The <code class="notranslate">rename</code> parameter is needed for renaming disallowed field names and duplicate identifiers to standard position-based identifiers, and this feature was added in Python 2.7/3.1.</p>