<p dir="auto">pandas version: 0.8.2.dev-742d7fb</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import pandas
import pandas.util.testing as put
index = put.makeDateIndex(4)
index.isin(index) # returns array([False, False, False, False], dtype=bool)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">pandas</span>
<span class="pl-k">import</span> <span class="pl-s1">pandas</span>.<span class="pl-s1">util</span>.<span class="pl-s1">testing</span> <span class="pl-k">as</span> <span class="pl-s1">put</span>
<span class="pl-s1">index</span> <span class="pl-c1">=</span> <span class="pl-s1">put</span>.<span class="pl-en">makeDateIndex</span>(<span class="pl-c1">4</span>)
<span class="pl-s1">index</span>.<span class="pl-en">isin</span>(<span class="pl-s1">index</span>) <span class="pl-c"># returns array([False, False, False, False], dtype=bool)</span></pre></div>
<p dir="auto">It works fine with pandas 0.7.3, i.e. it returns an array of True instead. This works fine as well with a pandas.Int64Index. <code class="notranslate">index.isin(index)</code> just calls <code class="notranslate">pandas.lib.ismember(index._array_values(), set(index))</code> but I am afraid I lack the skills to understand what goes wrong in the cython code.</p>
<p dir="auto">Just for some background, the original problem I bumped into:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import numpy as np
import pandas
ts = pandas.TimeSeries(data=[1,2,3], index=[ np.datetime64(x) for x in ['2000', '2000', '2002'] ])
ts.ix[ts.index] # returns an empty timeseries whereas ts[ts.index] works fine"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-k">import</span> <span class="pl-s1">pandas</span>
<span class="pl-s1">ts</span> <span class="pl-c1">=</span> <span class="pl-s1">pandas</span>.<span class="pl-v">TimeSeries</span>(<span class="pl-s1">data</span><span class="pl-c1">=</span>[<span class="pl-c1">1</span>,<span class="pl-c1">2</span>,<span class="pl-c1">3</span>], <span class="pl-s1">index</span><span class="pl-c1">=</span>[ <span class="pl-s1">np</span>.<span class="pl-en">datetime64</span>(<span class="pl-s1">x</span>) <span class="pl-k">for</span> <span class="pl-s1">x</span> <span class="pl-c1">in</span> [<span class="pl-s">'2000'</span>, <span class="pl-s">'2000'</span>, <span class="pl-s">'2002'</span>] ])
<span class="pl-s1">ts</span>.<span class="pl-s1">ix</span>[<span class="pl-s1">ts</span>.<span class="pl-s1">index</span>] <span class="pl-c"># returns an empty timeseries whereas ts[ts.index] works fine</span></pre></div>
<p dir="auto">I tracked it down to <code class="notranslate">ts.ix._get_item_iterable</code> which has a special case for when the index has duplicates which use pandas.TimeSeries.isin.</p>
<p dir="auto">As an aside it looks like pandas.core.common._asarray_tuplesafe(index) is wrong when using numpy 1.6 so that ts.ix[ts.index] still would return an empty array if this bug was fixed, but I guess that's a numpy 1.6 datetime64 bug which I have seen mentioned in a few other places.</p>
<p dir="auto">When assigning a <code class="notranslate">timedelta64</code> array to a subset of a new column of a <code class="notranslate">DataFrame</code>, missing data is not filled with <code class="notranslate">NaT</code> as expected; rather, the new column is cast to <code class="notranslate">float64</code> and <code class="notranslate">NaN</code> is used instead.  This cast does not usually occur when all values are present, <em>except</em> when there are already <code class="notranslate">float64</code> columns but no <code class="notranslate">timedelta64</code> columns in the <code class="notranslate">DataFrame</code> <em>and</em> indexing is done through <code class="notranslate">.ix</code> or <code class="notranslate">.loc</code>.</p>
<p dir="auto">It's possible these should be two separate issues.</p>
<p dir="auto">There are a lot of issues involving <code class="notranslate">NaT</code> in the issue tracker; I'm not 100% sure that this isn't a duplicate.  (Nor am I 100% sure this isn't intended behavior, but if it is I'd expect it to be documented more prominently.)</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="
import numpy as np
import pandas as pd

one_hour = 60*60*10**9

temp = pd.DataFrame({}, index=pd.date_range('2014-1-1', periods=4))
temp['A'] = np.array([1*one_hour]*4, dtype='m8[ns]')
temp.loc[:,'B'] = np.array([2*one_hour]*4, dtype='m8[ns]')
temp.loc[:3,'C'] = np.array([3*one_hour]*3, dtype='m8[ns]')
temp.ix[:,'D'] = np.array([4*one_hour]*4, dtype='m8[ns]')
temp.ix[:3,'E'] = np.array([5*one_hour]*3, dtype='m8[ns]')
temp['F'] = np.timedelta64('NaT')
temp.ix[:-1,'F'] = np.array([6*one_hour]*3, dtype='m8[ns]')

temp
#                   A        B             C        D             E        F
#2014-01-01 01:00:00 02:00:00  1.080000e+13 04:00:00  1.800000e+13 06:00:00
#2014-01-02 01:00:00 02:00:00  1.080000e+13 04:00:00  1.800000e+13 06:00:00
#2014-01-03 01:00:00 02:00:00  1.080000e+13 04:00:00  1.800000e+13 06:00:00
#2014-01-04 01:00:00 02:00:00           NaN 04:00:00           NaN      NaT
# 
# [4 rows x 6 columns]

temp = pd.DataFrame({}, index=pd.date_range('2014-1-1', periods=4))
# Partial assignment converts
temp.ix[:-1,'A'] = np.array([1*one_hour]*3, dtype='m8[ns]')
# DataFrame is all floats; converts
temp.ix[:,'B'] = np.array([2*one_hour]*4, dtype='m8[ns]')
# .ix and .loc behave the same
temp.loc[:,'C'] = np.array([3*one_hour]*4, dtype='m8[ns]')
# straight column assignment doesn't convert
temp['D'] = np.array([4*one_hour]*4, dtype='m8[ns]')
# Now there are timedeltas; doesn't convert
temp.ix[:,'E'] = np.array([5*one_hour]*4, dtype='m8[ns]')
# .ix and .loc still behave the same
temp.loc[:,'F'] = np.array([6*one_hour]*4, dtype='m8[ns]')

temp
#                        A             B             C        D        E  \
#2014-01-01  3.600000e+12  7.200000e+12  1.080000e+13 04:00:00 05:00:00   
#2014-01-02  3.600000e+12  7.200000e+12  1.080000e+13 04:00:00 05:00:00   
#2014-01-03  3.600000e+12  7.200000e+12  1.080000e+13 04:00:00 05:00:00   
#2014-01-04           NaN  7.200000e+12  1.080000e+13 04:00:00 05:00:00   
# 
#                   F  
#2014-01-01 06:00:00  
#2014-01-02 06:00:00  
#2014-01-03 06:00:00  
#2014-01-04 06:00:00  
# 
# [4 rows x 6 columns]

temp = pd.DataFrame({}, index=pd.date_range('2014-1-1', periods=4))
# No columns yet, no conversion
temp.ix[:,'A'] = np.array([2*one_hour]*4, dtype='m8[ns]')
#                   A
#2014-01-01 02:00:00
#2014-01-02 02:00:00
#2014-01-03 02:00:00
#2014-01-04 02:00:00
# 
# [4 rows x 1 columns]
"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-k">import</span> <span class="pl-s1">pandas</span> <span class="pl-k">as</span> <span class="pl-s1">pd</span>

<span class="pl-s1">one_hour</span> <span class="pl-c1">=</span> <span class="pl-c1">60</span><span class="pl-c1">*</span><span class="pl-c1">60</span><span class="pl-c1">*</span><span class="pl-c1">10</span><span class="pl-c1">**</span><span class="pl-c1">9</span>

<span class="pl-s1">temp</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">DataFrame</span>({}, <span class="pl-s1">index</span><span class="pl-c1">=</span><span class="pl-s1">pd</span>.<span class="pl-en">date_range</span>(<span class="pl-s">'2014-1-1'</span>, <span class="pl-s1">periods</span><span class="pl-c1">=</span><span class="pl-c1">4</span>))
<span class="pl-s1">temp</span>[<span class="pl-s">'A'</span>] <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>([<span class="pl-c1">1</span><span class="pl-c1">*</span><span class="pl-s1">one_hour</span>]<span class="pl-c1">*</span><span class="pl-c1">4</span>, <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s">'m8[ns]'</span>)
<span class="pl-s1">temp</span>.<span class="pl-s1">loc</span>[:,<span class="pl-s">'B'</span>] <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>([<span class="pl-c1">2</span><span class="pl-c1">*</span><span class="pl-s1">one_hour</span>]<span class="pl-c1">*</span><span class="pl-c1">4</span>, <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s">'m8[ns]'</span>)
<span class="pl-s1">temp</span>.<span class="pl-s1">loc</span>[:<span class="pl-c1">3</span>,<span class="pl-s">'C'</span>] <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>([<span class="pl-c1">3</span><span class="pl-c1">*</span><span class="pl-s1">one_hour</span>]<span class="pl-c1">*</span><span class="pl-c1">3</span>, <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s">'m8[ns]'</span>)
<span class="pl-s1">temp</span>.<span class="pl-s1">ix</span>[:,<span class="pl-s">'D'</span>] <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>([<span class="pl-c1">4</span><span class="pl-c1">*</span><span class="pl-s1">one_hour</span>]<span class="pl-c1">*</span><span class="pl-c1">4</span>, <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s">'m8[ns]'</span>)
<span class="pl-s1">temp</span>.<span class="pl-s1">ix</span>[:<span class="pl-c1">3</span>,<span class="pl-s">'E'</span>] <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>([<span class="pl-c1">5</span><span class="pl-c1">*</span><span class="pl-s1">one_hour</span>]<span class="pl-c1">*</span><span class="pl-c1">3</span>, <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s">'m8[ns]'</span>)
<span class="pl-s1">temp</span>[<span class="pl-s">'F'</span>] <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">timedelta64</span>(<span class="pl-s">'NaT'</span>)
<span class="pl-s1">temp</span>.<span class="pl-s1">ix</span>[:<span class="pl-c1">-</span><span class="pl-c1">1</span>,<span class="pl-s">'F'</span>] <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>([<span class="pl-c1">6</span><span class="pl-c1">*</span><span class="pl-s1">one_hour</span>]<span class="pl-c1">*</span><span class="pl-c1">3</span>, <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s">'m8[ns]'</span>)

<span class="pl-s1">temp</span>
<span class="pl-c">#                   A        B             C        D             E        F</span>
<span class="pl-c">#2014-01-01 01:00:00 02:00:00  1.080000e+13 04:00:00  1.800000e+13 06:00:00</span>
<span class="pl-c">#2014-01-02 01:00:00 02:00:00  1.080000e+13 04:00:00  1.800000e+13 06:00:00</span>
<span class="pl-c">#2014-01-03 01:00:00 02:00:00  1.080000e+13 04:00:00  1.800000e+13 06:00:00</span>
<span class="pl-c">#2014-01-04 01:00:00 02:00:00           NaN 04:00:00           NaN      NaT</span>
<span class="pl-c"># </span>
<span class="pl-c"># [4 rows x 6 columns]</span>

<span class="pl-s1">temp</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">DataFrame</span>({}, <span class="pl-s1">index</span><span class="pl-c1">=</span><span class="pl-s1">pd</span>.<span class="pl-en">date_range</span>(<span class="pl-s">'2014-1-1'</span>, <span class="pl-s1">periods</span><span class="pl-c1">=</span><span class="pl-c1">4</span>))
<span class="pl-c"># Partial assignment converts</span>
<span class="pl-s1">temp</span>.<span class="pl-s1">ix</span>[:<span class="pl-c1">-</span><span class="pl-c1">1</span>,<span class="pl-s">'A'</span>] <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>([<span class="pl-c1">1</span><span class="pl-c1">*</span><span class="pl-s1">one_hour</span>]<span class="pl-c1">*</span><span class="pl-c1">3</span>, <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s">'m8[ns]'</span>)
<span class="pl-c"># DataFrame is all floats; converts</span>
<span class="pl-s1">temp</span>.<span class="pl-s1">ix</span>[:,<span class="pl-s">'B'</span>] <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>([<span class="pl-c1">2</span><span class="pl-c1">*</span><span class="pl-s1">one_hour</span>]<span class="pl-c1">*</span><span class="pl-c1">4</span>, <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s">'m8[ns]'</span>)
<span class="pl-c"># .ix and .loc behave the same</span>
<span class="pl-s1">temp</span>.<span class="pl-s1">loc</span>[:,<span class="pl-s">'C'</span>] <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>([<span class="pl-c1">3</span><span class="pl-c1">*</span><span class="pl-s1">one_hour</span>]<span class="pl-c1">*</span><span class="pl-c1">4</span>, <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s">'m8[ns]'</span>)
<span class="pl-c"># straight column assignment doesn't convert</span>
<span class="pl-s1">temp</span>[<span class="pl-s">'D'</span>] <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>([<span class="pl-c1">4</span><span class="pl-c1">*</span><span class="pl-s1">one_hour</span>]<span class="pl-c1">*</span><span class="pl-c1">4</span>, <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s">'m8[ns]'</span>)
<span class="pl-c"># Now there are timedeltas; doesn't convert</span>
<span class="pl-s1">temp</span>.<span class="pl-s1">ix</span>[:,<span class="pl-s">'E'</span>] <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>([<span class="pl-c1">5</span><span class="pl-c1">*</span><span class="pl-s1">one_hour</span>]<span class="pl-c1">*</span><span class="pl-c1">4</span>, <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s">'m8[ns]'</span>)
<span class="pl-c"># .ix and .loc still behave the same</span>
<span class="pl-s1">temp</span>.<span class="pl-s1">loc</span>[:,<span class="pl-s">'F'</span>] <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>([<span class="pl-c1">6</span><span class="pl-c1">*</span><span class="pl-s1">one_hour</span>]<span class="pl-c1">*</span><span class="pl-c1">4</span>, <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s">'m8[ns]'</span>)

<span class="pl-s1">temp</span>
<span class="pl-c">#                        A             B             C        D        E  \</span>
<span class="pl-c">#2014-01-01  3.600000e+12  7.200000e+12  1.080000e+13 04:00:00 05:00:00   </span>
<span class="pl-c">#2014-01-02  3.600000e+12  7.200000e+12  1.080000e+13 04:00:00 05:00:00   </span>
<span class="pl-c">#2014-01-03  3.600000e+12  7.200000e+12  1.080000e+13 04:00:00 05:00:00   </span>
<span class="pl-c">#2014-01-04           NaN  7.200000e+12  1.080000e+13 04:00:00 05:00:00   </span>
<span class="pl-c"># </span>
<span class="pl-c">#                   F  </span>
<span class="pl-c">#2014-01-01 06:00:00  </span>
<span class="pl-c">#2014-01-02 06:00:00  </span>
<span class="pl-c">#2014-01-03 06:00:00  </span>
<span class="pl-c">#2014-01-04 06:00:00  </span>
<span class="pl-c"># </span>
<span class="pl-c"># [4 rows x 6 columns]</span>

<span class="pl-s1">temp</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">DataFrame</span>({}, <span class="pl-s1">index</span><span class="pl-c1">=</span><span class="pl-s1">pd</span>.<span class="pl-en">date_range</span>(<span class="pl-s">'2014-1-1'</span>, <span class="pl-s1">periods</span><span class="pl-c1">=</span><span class="pl-c1">4</span>))
<span class="pl-c"># No columns yet, no conversion</span>
<span class="pl-s1">temp</span>.<span class="pl-s1">ix</span>[:,<span class="pl-s">'A'</span>] <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>([<span class="pl-c1">2</span><span class="pl-c1">*</span><span class="pl-s1">one_hour</span>]<span class="pl-c1">*</span><span class="pl-c1">4</span>, <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s">'m8[ns]'</span>)
<span class="pl-c">#                   A</span>
<span class="pl-c">#2014-01-01 02:00:00</span>
<span class="pl-c">#2014-01-02 02:00:00</span>
<span class="pl-c">#2014-01-03 02:00:00</span>
<span class="pl-c">#2014-01-04 02:00:00</span>
<span class="pl-c"># </span>
<span class="pl-c"># [4 rows x 1 columns]</span></pre></div>