<p dir="auto">Hi team,</p>
<p dir="auto">I have been having <strong>issues with pandas memory management</strong>. Specifically, there is an (at least for me) unavoidable peak of memory which occurs when attempting to <strong>remove variables</strong> from a data set. It should be (almost) free! I am getting rid of part of the data, but it still needs to allocate a big amount of memory producing MemoryErrors.</p>
<p dir="auto">Just to give you a little bit of context, I am working with a DataFrame which contains 33M of rows and 500  columns (just a big one!), almost all of them numeric, in a machine with 360GB of RAM. The whole data set fits in memory and I can successfully apply some transformations to the variables. The problem comes when I need to drop a 10% of the columns contained in the table. It just produces a <strong>big peak of memory</strong> leading to a <code class="notranslate">MemoryError</code>. Before performing this operation, there are more than 80GB of memory available!.</p>
<p dir="auto">I tried to use the following methods for removing the columns and all of them failed.</p>
<ul dir="auto">
<li><code class="notranslate">drop()</code> with or without <code class="notranslate">inplace</code> parameter</li>
<li><code class="notranslate">pop()</code></li>
<li><code class="notranslate">reindex()</code></li>
<li><code class="notranslate">reindex_axis()</code></li>
<li><code class="notranslate">del df[column]</code> in a loop over the columns to be removed</li>
<li><code class="notranslate">__delitem__(column)</code> in a loop over the columns to be removed</li>
<li><code class="notranslate">pop()</code> and <code class="notranslate">drop()</code> in a loop over the columns to be removed.</li>
<li>I also tried to reasign the columns overwritting the data frame using indexing with <code class="notranslate">loc() </code>and <code class="notranslate">iloc()</code> but it does not help.</li>
</ul>
<p dir="auto">I found that the drop method with inplace is the most efficient one but it still generates a huge peak.</p>
<p dir="auto">I would like to discuss if there is there any way of implementing (or is it already implemented by any chance) a method for more efficiently removing variables without generating more memory consumption...</p>
<p dir="auto">Thank you<br>
Iván</p>
<p dir="auto">Hi,<br>
I reopen <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="275933864" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/18422" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/18422/hovercard" href="https://github.com/pandas-dev/pandas/issues/18422">#18422</a> with  copy-pastable example</p>
<h4 dir="auto">Code Sample, a copy-pastable example if possible</h4>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import pandas as pd

idx1 = pd.date_range('2011-01-01', periods=3, freq='H', tz='Europe/Paris')
idx2 = pd.date_range(start=idx1[0], end=idx1[-1], freq='H')

df1 = pd.DataFrame({'a': [1, 2, 3]}, index=idx1)
df2 = pd.DataFrame({'b': [1, 2, 3]}, index=idx2)

res = pd.concat([df1, df2], axis=1)

print(df1.index.tzinfo)
print(df2.index.tzinfo)
print(res.index.tzinfo)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">pandas</span> <span class="pl-k">as</span> <span class="pl-s1">pd</span>

<span class="pl-s1">idx1</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-en">date_range</span>(<span class="pl-s">'2011-01-01'</span>, <span class="pl-s1">periods</span><span class="pl-c1">=</span><span class="pl-c1">3</span>, <span class="pl-s1">freq</span><span class="pl-c1">=</span><span class="pl-s">'H'</span>, <span class="pl-s1">tz</span><span class="pl-c1">=</span><span class="pl-s">'Europe/Paris'</span>)
<span class="pl-s1">idx2</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-en">date_range</span>(<span class="pl-s1">start</span><span class="pl-c1">=</span><span class="pl-s1">idx1</span>[<span class="pl-c1">0</span>], <span class="pl-s1">end</span><span class="pl-c1">=</span><span class="pl-s1">idx1</span>[<span class="pl-c1">-</span><span class="pl-c1">1</span>], <span class="pl-s1">freq</span><span class="pl-c1">=</span><span class="pl-s">'H'</span>)

<span class="pl-s1">df1</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">DataFrame</span>({<span class="pl-s">'a'</span>: [<span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">3</span>]}, <span class="pl-s1">index</span><span class="pl-c1">=</span><span class="pl-s1">idx1</span>)
<span class="pl-s1">df2</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">DataFrame</span>({<span class="pl-s">'b'</span>: [<span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">3</span>]}, <span class="pl-s1">index</span><span class="pl-c1">=</span><span class="pl-s1">idx2</span>)

<span class="pl-s1">res</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-en">concat</span>([<span class="pl-s1">df1</span>, <span class="pl-s1">df2</span>], <span class="pl-s1">axis</span><span class="pl-c1">=</span><span class="pl-c1">1</span>)

<span class="pl-en">print</span>(<span class="pl-s1">df1</span>.<span class="pl-s1">index</span>.<span class="pl-s1">tzinfo</span>)
<span class="pl-en">print</span>(<span class="pl-s1">df2</span>.<span class="pl-s1">index</span>.<span class="pl-s1">tzinfo</span>)
<span class="pl-en">print</span>(<span class="pl-s1">res</span>.<span class="pl-s1">index</span>.<span class="pl-s1">tzinfo</span>)</pre></div>
<h4 dir="auto">Output</h4>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Europe/Paris
Europe/Paris
UTC"><pre class="notranslate"><code class="notranslate">Europe/Paris
Europe/Paris
UTC
</code></pre></div>
<h4 dir="auto">Problem description</h4>
<p dir="auto">pd.concat reset the tz-aware index to UTC</p>
<p dir="auto">This seems to come from the two different representation of time zone 'Europe/Paris' as this ipython output shows :</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="In [23]: df1.index.tzinfo
Out[23]: &lt;DstTzInfo 'Europe/Paris' LMT+0:09:00 STD&gt;

In [24]: df2.index.tzinfo
Out[24]: &lt;DstTzInfo 'Europe/Paris' CET+1:00:00 STD&gt;

In [26]: res.index.tzinfo
Out[26]: &lt;UTC&gt;"><pre class="notranslate"><code class="notranslate">In [23]: df1.index.tzinfo
Out[23]: &lt;DstTzInfo 'Europe/Paris' LMT+0:09:00 STD&gt;

In [24]: df2.index.tzinfo
Out[24]: &lt;DstTzInfo 'Europe/Paris' CET+1:00:00 STD&gt;

In [26]: res.index.tzinfo
Out[26]: &lt;UTC&gt;
</code></pre></div>
<h4 dir="auto">Expected Output</h4>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Europe/Paris
Europe/Paris
Europe/Paris"><pre class="notranslate"><code class="notranslate">Europe/Paris
Europe/Paris
Europe/Paris
</code></pre></div>
<h4 dir="auto">Output of <code class="notranslate">pd.show_versions()</code></h4>
<details>
<h2 dir="auto">INSTALLED VERSIONS</h2>
<p dir="auto">commit: None<br>
python: 3.6.2.final.0<br>
python-bits: 64<br>
OS: Windows<br>
OS-release: 10<br>
machine: AMD64<br>
processor: Intel64 Family 6 Model 94 Stepping 3, GenuineIntel<br>
byteorder: little<br>
LC_ALL: None<br>
LANG: None<br>
LOCALE: None.None</p>
<p dir="auto">pandas: 0.21.0<br>
pytest: None<br>
pip: 9.0.1<br>
setuptools: 28.8.0<br>
Cython: None<br>
numpy: 1.13.1<br>
scipy: None<br>
pyarrow: None<br>
xarray: None<br>
IPython: None<br>
sphinx: None<br>
patsy: None<br>
dateutil: 2.6.1<br>
pytz: 2017.2<br>
blosc: None<br>
bottleneck: None<br>
tables: None<br>
numexpr: None<br>
feather: None<br>
matplotlib: 2.0.2<br>
openpyxl: None<br>
xlrd: None<br>
xlwt: None<br>
xlsxwriter: None<br>
lxml: None<br>
bs4: 4.6.0<br>
html5lib: None<br>
sqlalchemy: None<br>
pymysql: None<br>
psycopg2: None<br>
jinja2: 2.9.6<br>
s3fs: None<br>
fastparquet: None<br>
pandas_gbq: None<br>
pandas_datareader: None</p>
</details>