<p dir="auto">When try to access labels (<code class="notranslate">.loc</code>) by using a specific list of <code class="notranslate">Timestamp</code> objects or a <code class="notranslate">DatetimeIndex</code> object (See attached csv file), the resulting index is returned in UTC offset but the original timezone is not removed.</p>
<p dir="auto">This seems to happen only in very specific cases, when the index passed to <code class="notranslate">.loc</code> contains labels that do not exist in the DataFrame and also contains duplicates.</p>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/yuval-jether/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/yuval-jether">@yuval-jether</a></p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import pandas as pd
import pytz
idx = pd.date_range('2011-01-01', '2017-10-01 00:00:00', freq='h', tz='America/Chicago')
s = pd.Series(np.random.rand(len(idx)), index=idx)
i = pd.to_datetime(['2017-09-30 06:00:00', '2017-09-30 23:00:00', '2017-09-30 23:00:00', '2017-10-01 00:00:00',]).tz_localize('America/Chicago')
s.loc[i]

Out[39]: 
2017-09-30 06:00:00-05:00    0.380138
2017-09-30 23:00:00-05:00    0.774696
2017-09-30 23:00:00-05:00    0.774696
2017-10-01 00:00:00-05:00    0.728027
dtype: float64

# Added a label that does not exist in the Series
import pandas as pd
import pytz
idx = pd.date_range('2011-01-01', '2017-10-01 00:00:00', freq='h', tz='America/Chicago')
s = pd.Series(np.random.rand(len(idx)), index=idx)
i = pd.to_datetime(['2017-09-30 06:00:00', '2017-09-30 23:00:00', '2017-09-30 23:00:00', '2017-10-01 00:00:00', '2017-10-01 01:00:00',]).tz_localize('America/Chicago')
s.loc[i]

Out[40]: 
2017-09-30 11:00:00-05:00    0.645350
2017-10-01 04:00:00-05:00    0.099323
2017-10-01 04:00:00-05:00    0.099323
2017-10-01 05:00:00-05:00    0.037136
2017-10-01 06:00:00-05:00         NaN
dtype: float64
"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">pandas</span> <span class="pl-k">as</span> <span class="pl-s1">pd</span>
<span class="pl-k">import</span> <span class="pl-s1">pytz</span>
<span class="pl-s1">idx</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-en">date_range</span>(<span class="pl-s">'2011-01-01'</span>, <span class="pl-s">'2017-10-01 00:00:00'</span>, <span class="pl-s1">freq</span><span class="pl-c1">=</span><span class="pl-s">'h'</span>, <span class="pl-s1">tz</span><span class="pl-c1">=</span><span class="pl-s">'America/Chicago'</span>)
<span class="pl-s1">s</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">Series</span>(<span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">rand</span>(<span class="pl-en">len</span>(<span class="pl-s1">idx</span>)), <span class="pl-s1">index</span><span class="pl-c1">=</span><span class="pl-s1">idx</span>)
<span class="pl-s1">i</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-en">to_datetime</span>([<span class="pl-s">'2017-09-30 06:00:00'</span>, <span class="pl-s">'2017-09-30 23:00:00'</span>, <span class="pl-s">'2017-09-30 23:00:00'</span>, <span class="pl-s">'2017-10-01 00:00:00'</span>,]).<span class="pl-en">tz_localize</span>(<span class="pl-s">'America/Chicago'</span>)
<span class="pl-s1">s</span>.<span class="pl-s1">loc</span>[<span class="pl-s1">i</span>]

<span class="pl-v">Out</span>[<span class="pl-c1">39</span>]: 
<span class="pl-c1">2017</span><span class="pl-c1">-</span><span class="pl-c1">09</span><span class="pl-c1">-</span><span class="pl-c1">30</span> <span class="pl-c1">06</span>:<span class="pl-c1">00</span>:<span class="pl-c1">00</span><span class="pl-c1">-</span><span class="pl-c1">05</span>:<span class="pl-c1">00</span>    <span class="pl-c1">0.380138</span>
<span class="pl-c1">2017</span><span class="pl-c1">-</span><span class="pl-c1">09</span><span class="pl-c1">-</span><span class="pl-c1">30</span> <span class="pl-c1">23</span>:<span class="pl-c1">00</span>:<span class="pl-c1">00</span><span class="pl-c1">-</span><span class="pl-c1">05</span>:<span class="pl-c1">00</span>    <span class="pl-c1">0.774696</span>
<span class="pl-c1">2017</span><span class="pl-c1">-</span><span class="pl-c1">09</span><span class="pl-c1">-</span><span class="pl-c1">30</span> <span class="pl-c1">23</span>:<span class="pl-c1">00</span>:<span class="pl-c1">00</span><span class="pl-c1">-</span><span class="pl-c1">05</span>:<span class="pl-c1">00</span>    <span class="pl-c1">0.774696</span>
<span class="pl-c1">2017</span><span class="pl-c1">-</span><span class="pl-c1">10</span><span class="pl-c1">-</span><span class="pl-c1">01</span> <span class="pl-c1">00</span>:<span class="pl-c1">00</span>:<span class="pl-c1">00</span><span class="pl-c1">-</span><span class="pl-c1">05</span>:<span class="pl-c1">00</span>    <span class="pl-c1">0.728027</span>
<span class="pl-s1">dtype</span>: <span class="pl-s1">float64</span>

<span class="pl-c"># Added a label that does not exist in the Series</span>
<span class="pl-k">import</span> <span class="pl-s1">pandas</span> <span class="pl-k">as</span> <span class="pl-s1">pd</span>
<span class="pl-k">import</span> <span class="pl-s1">pytz</span>
<span class="pl-s1">idx</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-en">date_range</span>(<span class="pl-s">'2011-01-01'</span>, <span class="pl-s">'2017-10-01 00:00:00'</span>, <span class="pl-s1">freq</span><span class="pl-c1">=</span><span class="pl-s">'h'</span>, <span class="pl-s1">tz</span><span class="pl-c1">=</span><span class="pl-s">'America/Chicago'</span>)
<span class="pl-s1">s</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">Series</span>(<span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">rand</span>(<span class="pl-en">len</span>(<span class="pl-s1">idx</span>)), <span class="pl-s1">index</span><span class="pl-c1">=</span><span class="pl-s1">idx</span>)
<span class="pl-s1">i</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-en">to_datetime</span>([<span class="pl-s">'2017-09-30 06:00:00'</span>, <span class="pl-s">'2017-09-30 23:00:00'</span>, <span class="pl-s">'2017-09-30 23:00:00'</span>, <span class="pl-s">'2017-10-01 00:00:00'</span>, <span class="pl-s">'2017-10-01 01:00:00'</span>,]).<span class="pl-en">tz_localize</span>(<span class="pl-s">'America/Chicago'</span>)
<span class="pl-s1">s</span>.<span class="pl-s1">loc</span>[<span class="pl-s1">i</span>]

<span class="pl-v">Out</span>[<span class="pl-c1">40</span>]: 
<span class="pl-c1">2017</span><span class="pl-c1">-</span><span class="pl-c1">09</span><span class="pl-c1">-</span><span class="pl-c1">30</span> <span class="pl-c1">11</span>:<span class="pl-c1">00</span>:<span class="pl-c1">00</span><span class="pl-c1">-</span><span class="pl-c1">05</span>:<span class="pl-c1">00</span>    <span class="pl-c1">0.645350</span>
<span class="pl-c1">2017</span><span class="pl-c1">-</span><span class="pl-c1">10</span><span class="pl-c1">-</span><span class="pl-c1">01</span> <span class="pl-c1">04</span>:<span class="pl-c1">00</span>:<span class="pl-c1">00</span><span class="pl-c1">-</span><span class="pl-c1">05</span>:<span class="pl-c1">00</span>    <span class="pl-c1">0.099323</span>
<span class="pl-c1">2017</span><span class="pl-c1">-</span><span class="pl-c1">10</span><span class="pl-c1">-</span><span class="pl-c1">01</span> <span class="pl-c1">04</span>:<span class="pl-c1">00</span>:<span class="pl-c1">00</span><span class="pl-c1">-</span><span class="pl-c1">05</span>:<span class="pl-c1">00</span>    <span class="pl-c1">0.099323</span>
<span class="pl-c1">2017</span><span class="pl-c1">-</span><span class="pl-c1">10</span><span class="pl-c1">-</span><span class="pl-c1">01</span> <span class="pl-c1">05</span>:<span class="pl-c1">00</span>:<span class="pl-c1">00</span><span class="pl-c1">-</span><span class="pl-c1">05</span>:<span class="pl-c1">00</span>    <span class="pl-c1">0.037136</span>
<span class="pl-c1">2017</span><span class="pl-c1">-</span><span class="pl-c1">10</span><span class="pl-c1">-</span><span class="pl-c1">01</span> <span class="pl-c1">06</span>:<span class="pl-c1">00</span>:<span class="pl-c1">00</span><span class="pl-c1">-</span><span class="pl-c1">05</span>:<span class="pl-c1">00</span>         <span class="pl-v">NaN</span>
<span class="pl-s1">dtype</span>: <span class="pl-s1">float64</span></pre></div>
<h4 dir="auto">Output of <code class="notranslate">pd.show_versions()</code></h4>
<details>
<h2 dir="auto">pd.show_versions()<br>
2017-10-30 03:15:08 [pip.utils] [DEBUG]  lzma module is not available<br>
2017-10-30 03:15:08 [pip.vcs] [DEBUG]  Registered VCS backend: git<br>
2017-10-30 03:15:08 [pip.vcs] [DEBUG]  Registered VCS backend: hg<br>
2017-10-30 03:15:08 [pip.pep425tags] [DEBUG]  Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect<br>
2017-10-30 03:15:08 [pip.pep425tags] [DEBUG]  Config variable 'WITH_PYMALLOC' is unset, Python ABI tag may be incorrect<br>
2017-10-30 03:15:08 [pip.pep425tags] [DEBUG]  Config variable 'Py_UNICODE_SIZE' is unset, Python ABI tag may be incorrect<br>
2017-10-30 03:15:08 [pip.pep425tags] [DEBUG]  Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect<br>
2017-10-30 03:15:08 [pip.pep425tags] [DEBUG]  Config variable 'WITH_PYMALLOC' is unset, Python ABI tag may be incorrect<br>
2017-10-30 03:15:08 [pip.pep425tags] [DEBUG]  Config variable 'Py_UNICODE_SIZE' is unset, Python ABI tag may be incorrect<br>
2017-10-30 03:15:08 [pip.vcs] [DEBUG]  Registered VCS backend: svn<br>
2017-10-30 03:15:09 [pip.vcs] [DEBUG]  Registered VCS backend: bzr<br>
INSTALLED VERSIONS</h2>
<p dir="auto">commit: None<br>
python: 2.7.12.final.0<br>
python-bits: 64<br>
OS: Windows<br>
OS-release: 8.1<br>
machine: AMD64<br>
processor: Intel64 Family 6 Model 60 Stepping 3, GenuineIntel<br>
byteorder: little<br>
LC_ALL: None<br>
LANG: None<br>
LOCALE: None.None<br>
pandas: 0.20.3<br>
pytest: 3.0.7<br>
pip: 9.0.1<br>
setuptools: 36.5.0<br>
Cython: None<br>
numpy: 1.13.1<br>
scipy: 0.18.1<br>
xarray: 0.9.6<br>
IPython: 5.5.0<br>
sphinx: None<br>
patsy: 0.4.1<br>
dateutil: 2.6.1<br>
pytz: 2017.2<br>
blosc: None<br>
bottleneck: None<br>
tables: None<br>
numexpr: 2.6.2<br>
feather: None<br>
matplotlib: 2.0.2<br>
openpyxl: 2.4.8<br>
xlrd: 1.0.0<br>
xlwt: 1.2.0<br>
xlsxwriter: None<br>
lxml: 3.8.0<br>
bs4: 4.5.1<br>
html5lib: 0.999999999<br>
sqlalchemy: None<br>
pymysql: None<br>
psycopg2: None<br>
jinja2: 2.9.6<br>
s3fs: None<br>
pandas_gbq: None<br>
pandas_datareader: 0.4.0</p>
</details>
<p dir="auto">Perhaps there is some canonical way of doing this that I'm not aware of, but from time to time I find myself wanting to group more than one frame by a key, and apply some function to the result. At the moment, I'm doing this in a somewhat roundabout way:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="results = []
for key, t1 in table1.groupby(level='id'):
    try:
        t2 = table2.xs(key, level='id')
    except KeyError:
        t2 = None
    try:
        t3 = table3.xs(key, level='id')
    except KeyError:
        t3 = None
    # ... etc
    r = some_func(t1, t2, t3)
    results.append(r)
results = pd.concat(results)"><pre class="notranslate"><code class="notranslate">results = []
for key, t1 in table1.groupby(level='id'):
    try:
        t2 = table2.xs(key, level='id')
    except KeyError:
        t2 = None
    try:
        t3 = table3.xs(key, level='id')
    except KeyError:
        t3 = None
    # ... etc
    r = some_func(t1, t2, t3)
    results.append(r)
results = pd.concat(results)
</code></pre></div>
<p dir="auto">I realize that often the easiest thing to do is just to merge the frames, but there are cases where that doesn't really make sense (e.g. non-compatible multiindexes that share one level). I imagine <code class="notranslate">pd.merge(t1, t2, on="id")</code> already does something like this to line up tables - do you guys think there is any value in exposing a standard way of doing this? Perhaps something like:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pd.multi_frame_groupby([t1, t2, t3, t4], level='id').apply(some_func)"><pre class="notranslate"><code class="notranslate">pd.multi_frame_groupby([t1, t2, t3, t4], level='id').apply(some_func)
</code></pre></div>
<p dir="auto">Or, given that the full behaviour of such a groupby object is non trivial, perhaps just a generator for frame tuples?</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="gen = pd.multi_frame_groupby([t1, t2, t3, t4], level='id')
pd.concat([some_func(*t) for t in gen])"><pre class="notranslate"><code class="notranslate">gen = pd.multi_frame_groupby([t1, t2, t3, t4], level='id')
pd.concat([some_func(*t) for t in gen])
</code></pre></div>