<p dir="auto">Came a across this very odd problem while trying to get difference between two columns in a DataFrame of type datetime64. The following simple example illustrates the problem:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="df = pd.DataFrame({'A': pd.date_range('20150301', '20150304'), 
               'B': pd.date_range('20150303', '20150306')}, 
               index = [1, 2, 3, 3])
df
    A           B
1   2015-03-01  2015-03-03
2   2015-03-02  2015-03-04
3   2015-03-03  2015-03-05
3   2015-03-04  2015-03-06

df.B - df.A
1   2 days
2   2 days
3   2 days
3   1 days
3   3 days
3   2 days"><pre class="notranslate"><code class="notranslate">df = pd.DataFrame({'A': pd.date_range('20150301', '20150304'), 
               'B': pd.date_range('20150303', '20150306')}, 
               index = [1, 2, 3, 3])
df
    A           B
1   2015-03-01  2015-03-03
2   2015-03-02  2015-03-04
3   2015-03-03  2015-03-05
3   2015-03-04  2015-03-06

df.B - df.A
1   2 days
2   2 days
3   2 days
3   1 days
3   3 days
3   2 days
</code></pre></div>
<p dir="auto">dtype: timedelta64[ns]</p>
<p dir="auto">What happens generally is that the output contains an entry for the difference in every pair in the Cartesian product of the rows containing the duplicate index. I learned this the hard way on a df with 30K+ rows and only 50 distinct indices ending up trying to compute a series with 385 million+ entries, quickly freezing my computer.</p>
<p dir="auto">On a related note, using the eval method to compute the difference throws the following error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="df.eval('B - A')
ValueError: unkown type timedelta64[ns]"><pre class="notranslate"><code class="notranslate">df.eval('B - A')
ValueError: unkown type timedelta64[ns]
</code></pre></div>
<p dir="auto">I have a hd5store that accumulates data by batch. However the batches overlap and I want to  just add the new rows..<br>
The data is in a table with a major index of event timestamp. I can assume the new batch has complete data for the first day so all I need to do is to remove all rows with a timestamp &gt;= min_new_date.</p>
<p dir="auto">I am using python 2.7, pandas  0.11.0.dev-b41dc91  on Windows 7 64bit  and lget the following error however I tried to pass in the date  In this example I hard code the date string to simplify the example.</p>
<p dir="auto">dstore = pyt.HDFStore(hdfile)</p>
<p dir="auto">dstore<br>
Out[39]:<br>
&lt;class 'pandas.io.pytables.HDFStore'&gt;<br>
File path: Y:\HAVI_MONITORING\Log_Scan_Files\ROPLogHistComp2.h5<br>
/files              series       (shape-&gt;[1])<br>
/logdata            frame_table  (typ-&gt;appendable_multi,nrows-&gt;1148964,ncols-&gt;13,indexers-&gt;[index],dc-&gt;[FileId,Start,Wob,Start_date,Type_1])<br>
/meta               frame        (shape-&gt;[1,1])<br>
/stores             frame_table  (typ-&gt;appendable,nrows-&gt;1123,ncols-&gt;20,indexers-&gt;[index])</p>
<h2 dir="auto">dstore.remove('logdata','major_axis&gt;=20130327')</h2>
<p dir="auto">Exception                                 Traceback (most recent call last)<br>
 in ()<br>
----&gt; 1 dstore.remove('logdata','major_axis&gt;=20130327')</p>
<p dir="auto">C:\Python27\lib\site-packages\pandas\io\pytables.pyc in remove(self, key, where, start, stop)<br>
546             if not s.is_table:<br>
547                 raise Exception('can only remove with where on objects written as tables')<br>
--&gt; 548             return s.delete(where = where, start=start, stop=stop)<br>
549<br>
550     def append(self, key, value, columns=None, **kwargs):</p>
<p dir="auto">C:\Python27\lib\site-packages\pandas\io\pytables.pyc in delete(self, where, *_kwargs)<br>
2748         # create the selection<br>
2749         table = self.table<br>
-&gt; 2750         self.selection = Selection(self, where, *_kwargs)<br>
2751         values = self.selection.select_coords()<br>
2752</p>
<p dir="auto">C:\Python27\lib\site-packages\pandas\io\pytables.pyc in <strong>init</strong>(self, table, where, start, stop, **kwargs)<br>
3275             self.coordinates = where.values<br>
3276         else:<br>
-&gt; 3277             self.terms = self.generate(where)<br>
3278<br>
3279             # create the numexpr &amp; the filter</p>
<p dir="auto">C:\Python27\lib\site-packages\pandas\io\pytables.pyc in generate(self, where)<br>
3304<br>
3305         queryables = self.table.queryables()<br>
-&gt; 3306         return [Term(c, queryables=queryables) for c in where]<br>
3307<br>
3308     def select(self):</p>
<p dir="auto">C:\Python27\lib\site-packages\pandas\io\pytables.pyc in <strong>init</strong>(self, field, op, value, queryables)<br>
3132<br>
3133         if len(self.q):<br>
-&gt; 3134             self.eval()<br>
3135<br>
3136     def <strong>str</strong>(self):</p>
<p dir="auto">C:\Python27\lib\site-packages\pandas\io\pytables.pyc in eval(self)<br>
3158<br>
3159         if not self.is_valid:<br>
-&gt; 3160             raise Exception("query term is not valid [%s]" % str(self))<br>
3161<br>
3162         # convert values if we are in the table</p>
<p dir="auto">Exception: query term is not valid [field-&gt;major_axis,op-&gt;&gt;=,value-&gt;['20130327']]<br>
C:\Python27\lib\site-packages\pandas\io\pytables.pyc in <strong>init</strong>(self, table, where, start, stop, **kwargs)<br>
3275             self.coordinates = where.values<br>
3276         else:<br>
-&gt; 3277             self.terms = self.generate(where)<br>
3278<br>
3279             # create the numexpr &amp; the filter</p>
<p dir="auto">C:\Python27\lib\site-packages\pandas\io\pytables.pyc in generate(self, where)<br>
3304<br>
3305         queryables = self.table.queryables()<br>
-&gt; 3306         return [Term(c, queryables=queryables) for c in where]<br>
3307<br>
3308     def select(self):</p>
<p dir="auto">C:\Python27\lib\site-packages\pandas\io\pytables.pyc in <strong>init</strong>(self, field, op, value, queryables)<br>
3132<br>
3133         if len(self.q):<br>
-&gt; 3134             self.eval()<br>
3135<br>
3136     def <strong>str</strong>(self):</p>
<p dir="auto">C:\Python27\lib\site-packages\pandas\io\pytables.pyc in eval(self)<br>
3158<br>
3159         if not self.is_valid:<br>
-&gt; 3160             raise Exception("query term is not valid [%s]" % str(self))<br>
3161<br>
3162         # convert values if we are in the table</p>
<p dir="auto">Exception: query term is not valid [field-&gt;major_axis,op-&gt;&gt;=,value-&gt;['20130327']]</p>