<p dir="auto">Hi,<br>
I noticed that datetime parsing for large sql/csv tables is really slow. Would it be acceptable to use a technique where repeated calculations are cached? For example, instead of:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="def parse_date(date_str) :
    return datetime.datetime.strptime(date_str,FMT)
def parse_date_col(str_col) :
    return [parse_date(date_str) for date_str in str_col]"><pre class="notranslate"><span class="pl-k">def</span> <span class="pl-en">parse_date</span>(<span class="pl-s1">date_str</span>) :
    <span class="pl-k">return</span> <span class="pl-s1">datetime</span>.<span class="pl-s1">datetime</span>.<span class="pl-en">strptime</span>(<span class="pl-s1">date_str</span>,<span class="pl-v">FMT</span>)
<span class="pl-k">def</span> <span class="pl-en">parse_date_col</span>(<span class="pl-s1">str_col</span>) :
    <span class="pl-k">return</span> [<span class="pl-en">parse_date</span>(<span class="pl-s1">date_str</span>) <span class="pl-k">for</span> <span class="pl-s1">date_str</span> <span class="pl-c1">in</span> <span class="pl-s1">str_col</span>]</pre></div>
<p dir="auto">use</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="def parse_date(date_str) :
    return datetime.datetime.strptime(date_str,FMT)
def parse_date_col(str_col) :
    cache = dict()
    for date_str in str_col :
        if date_str not in cache :
            cache[date_str] = parse_date(date_str)
    return [cache[date_str] for date_str in str_col]"><pre class="notranslate"><span class="pl-k">def</span> <span class="pl-en">parse_date</span>(<span class="pl-s1">date_str</span>) :
    <span class="pl-k">return</span> <span class="pl-s1">datetime</span>.<span class="pl-s1">datetime</span>.<span class="pl-en">strptime</span>(<span class="pl-s1">date_str</span>,<span class="pl-v">FMT</span>)
<span class="pl-k">def</span> <span class="pl-en">parse_date_col</span>(<span class="pl-s1">str_col</span>) :
    <span class="pl-s1">cache</span> <span class="pl-c1">=</span> <span class="pl-en">dict</span>()
    <span class="pl-k">for</span> <span class="pl-s1">date_str</span> <span class="pl-c1">in</span> <span class="pl-s1">str_col</span> :
        <span class="pl-k">if</span> <span class="pl-s1">date_str</span> <span class="pl-c1">not</span> <span class="pl-c1">in</span> <span class="pl-s1">cache</span> :
            <span class="pl-s1">cache</span>[<span class="pl-s1">date_str</span>] <span class="pl-c1">=</span> <span class="pl-en">parse_date</span>(<span class="pl-s1">date_str</span>)
    <span class="pl-k">return</span> [<span class="pl-s1">cache</span>[<span class="pl-s1">date_str</span>] <span class="pl-k">for</span> <span class="pl-s1">date_str</span> <span class="pl-c1">in</span> <span class="pl-s1">str_col</span>]</pre></div>
<p dir="auto">The reason this works is that string hashing / comparison / dictionary insertion is much much faster than strptime.</p>
<p dir="auto">For tables where dates are repeated many times this can result in orders of magnitude speedup.</p>
<p dir="auto">Thanks<br>
Charles</p>
<p dir="auto">I just upgrage from 0.11 to 0.12 version. And meet  dataframe rename error caused by upgrading.  (It worked well in 0.11) .</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; df4
                 TClose      RT    TExg
STK_ID RPT_Date                        
600809 20130331   22.02  0.0454  0.0422

&gt;&gt;&gt; df5
                 STK_ID  RPT_Date STK_Name  TClose
STK_ID RPT_Date                                   
600809 20120930  600809  20120930     山西汾酒   38.05
       20121231  600809  20121231     山西汾酒   41.66
       20130331  600809  20130331     山西汾酒   30.01

&gt;&gt;&gt; k=pd.merge(df4, df5, how='inner', left_index=True, right_index=True)
&gt;&gt;&gt; k
                 TClose_x      RT    TExg  STK_ID  RPT_Date STK_Name  TClose_y
STK_ID RPT_Date                                                               
600809 20130331     22.02  0.0454  0.0422  600809  20130331     山西汾酒     30.01

&gt;&gt;&gt; k.rename(columns={'TClose_x':'TClose', 'TClose_y':'QT_Close'})
Traceback (most recent call last):
  File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;
  File &quot;d:\Python27\lib\site-packages\pandas\core\base.py&quot;, line 40, in __repr__
    return str(self)
  File &quot;d:\Python27\lib\site-packages\pandas\core\base.py&quot;, line 20, in __str__
    return self.__bytes__()
  File &quot;d:\Python27\lib\site-packages\pandas\core\base.py&quot;, line 32, in __bytes__
    return self.__unicode__().encode(encoding, 'replace')
  File &quot;d:\Python27\lib\site-packages\pandas\core\frame.py&quot;, line 668, in __unicode__
    self.to_string(buf=buf)
  File &quot;d:\Python27\lib\site-packages\pandas\core\frame.py&quot;, line 1556, in to_string
    formatter.to_string()
  File &quot;d:\Python27\lib\site-packages\pandas\core\format.py&quot;, line 294, in to_string
    strcols = self._to_str_columns()
  File &quot;d:\Python27\lib\site-packages\pandas\core\format.py&quot;, line 239, in _to_str_columns
    str_columns = self._get_formatted_column_labels()
  File &quot;d:\Python27\lib\site-packages\pandas\core\format.py&quot;, line 435, in _get_formatted_column_labels
    dtypes = self.frame.dtypes
  File &quot;d:\Python27\lib\site-packages\pandas\core\frame.py&quot;, line 1696, in dtypes
    return self.apply(lambda x: x.dtype)
  File &quot;d:\Python27\lib\site-packages\pandas\core\frame.py&quot;, line 4416, in apply
    return self._apply_standard(f, axis)
  File &quot;d:\Python27\lib\site-packages\pandas\core\frame.py&quot;, line 4491, in _apply_standard
    raise e
TypeError: (&quot;'NoneType' object is not iterable&quot;, u'occurred at index TExg')

&gt;&gt;&gt; df4.dtypes
TClose    float64
RT        float64
TExg      float64
dtype: object

&gt;&gt;&gt; df5.dtypes
STK_ID       object
RPT_Date     object
STK_Name     object
TClose      float64
dtype: object
&gt;&gt;&gt; "><pre class="notranslate"><code class="notranslate">&gt;&gt;&gt; df4
                 TClose      RT    TExg
STK_ID RPT_Date                        
600809 20130331   22.02  0.0454  0.0422

&gt;&gt;&gt; df5
                 STK_ID  RPT_Date STK_Name  TClose
STK_ID RPT_Date                                   
600809 20120930  600809  20120930     山西汾酒   38.05
       20121231  600809  20121231     山西汾酒   41.66
       20130331  600809  20130331     山西汾酒   30.01

&gt;&gt;&gt; k=pd.merge(df4, df5, how='inner', left_index=True, right_index=True)
&gt;&gt;&gt; k
                 TClose_x      RT    TExg  STK_ID  RPT_Date STK_Name  TClose_y
STK_ID RPT_Date                                                               
600809 20130331     22.02  0.0454  0.0422  600809  20130331     山西汾酒     30.01

&gt;&gt;&gt; k.rename(columns={'TClose_x':'TClose', 'TClose_y':'QT_Close'})
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "d:\Python27\lib\site-packages\pandas\core\base.py", line 40, in __repr__
    return str(self)
  File "d:\Python27\lib\site-packages\pandas\core\base.py", line 20, in __str__
    return self.__bytes__()
  File "d:\Python27\lib\site-packages\pandas\core\base.py", line 32, in __bytes__
    return self.__unicode__().encode(encoding, 'replace')
  File "d:\Python27\lib\site-packages\pandas\core\frame.py", line 668, in __unicode__
    self.to_string(buf=buf)
  File "d:\Python27\lib\site-packages\pandas\core\frame.py", line 1556, in to_string
    formatter.to_string()
  File "d:\Python27\lib\site-packages\pandas\core\format.py", line 294, in to_string
    strcols = self._to_str_columns()
  File "d:\Python27\lib\site-packages\pandas\core\format.py", line 239, in _to_str_columns
    str_columns = self._get_formatted_column_labels()
  File "d:\Python27\lib\site-packages\pandas\core\format.py", line 435, in _get_formatted_column_labels
    dtypes = self.frame.dtypes
  File "d:\Python27\lib\site-packages\pandas\core\frame.py", line 1696, in dtypes
    return self.apply(lambda x: x.dtype)
  File "d:\Python27\lib\site-packages\pandas\core\frame.py", line 4416, in apply
    return self._apply_standard(f, axis)
  File "d:\Python27\lib\site-packages\pandas\core\frame.py", line 4491, in _apply_standard
    raise e
TypeError: ("'NoneType' object is not iterable", u'occurred at index TExg')

&gt;&gt;&gt; df4.dtypes
TClose    float64
RT        float64
TExg      float64
dtype: object

&gt;&gt;&gt; df5.dtypes
STK_ID       object
RPT_Date     object
STK_Name     object
TClose      float64
dtype: object
&gt;&gt;&gt; 
</code></pre></div>