<p dir="auto">This is an attempt to simplify/streamline internal API that has been brewing inside my head for quite a while. It does mean a significant overhaul and may take time, but it may prove worth the while. I'm putting it here for discussion ahead of time to make sure the effort isn't wasted for going in the wrong direction.</p>
<h3 dir="auto">Idea</h3>
<p dir="auto">The idea is simple: "augmented take" operation — with -1 taking from nowhere and creating a new column — is enough to express any reindexing/merging/joining that may happen at Index level.  So, lower levels of API that do the heavy lifting may be relieved from the burden of operating on labels and keeping them in sync. This will make them more self-contained with the following benefits:</p>
<ul dir="auto">
<li>simplify implementation</li>
<li>regularize data-handling operations making them more dependable, no more "oh no, I have a duplicate/timestamp/period/multiindex/etc. label in the index, all slicing operations are now 10x slower", this happened to me more times than I'm proud of.</li>
<li>simplify contributions from new developers, no more tracking down all the code paths up to the public API to fix one small check.</li>
<li>make code more test- and benchmark-friendly, no more exponential growth in number of tests/benchmarks for each new feature</li>
<li>declaring an API will simplify developing and maintaining "unconventional" storages (sparse, categorical, compressed, etc.)</li>
</ul>
<p dir="auto">There's a <a href="https://github.com/pydata/pandas/issues/162" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/162/hovercard">three-year-old ticket</a> ticket that mentions a similar (if not the same) idea. As mentioned there, this <em>may</em> break pickles and other deserialization and thus it will require a separate legacy deserialization compatibility layer.</p>
<p dir="auto"><a href="https://github.com/pydata/pandas/issues/163" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/163/hovercard">Another ticket</a> mentions moving Block &amp; BlockManager internals to cython level and dropping axes dependency will definitely facilitate that.</p>
<h3 dir="auto">Goals</h3>
<p dir="auto">The end goal is to have internals layered as follows:</p>
<ul dir="auto">
<li><strong>Block</strong>: a <em>proper</em> homogenous ndarray
<ul dir="auto">
<li>think numpy.ndarray with all necessary fixes/workarounds</li>
<li>datatype inference</li>
<li>support for custom pandas datatypes</li>
<li>typical (slice, concatenate) and pandas-specific (take-with-insert) operations</li>
</ul>
</li>
<li><strong>RemappableBlock</strong>: homogeneous ndarray that supports "remapping" one of its axes
<ul dir="auto">
<li>think <em>Block</em> + <em>ref_locs</em></li>
<li><em>ref_locs</em> should be Int64Index (platform-int-index?, also <a href="https://github.com/pydata/pandas/issues/939" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/939/hovercard">RangeIndex</a> will help)</li>
<li><em>Block</em> instances may be shared between <em>RemappableBlocks</em></li>
<li>optimizations for no-remapping mode of work (think <em>SingleBlockManager</em>)</li>
<li>FIXME: better name?</li>
</ul>
</li>
<li><strong>BlockManager</strong>: a <em>proper</em> heterogeneous ndarray
<ul dir="auto">
<li>external interface is similar to that of <strong>Block</strong></li>
<li>can share <em>RemappableBlocks</em></li>
</ul>
</li>
<li><strong>NDFrame</strong>: labeled heterogeneous ndarray
<ul dir="auto">
<li>more or less equivalent to current <em>NDFrames</em></li>
<li>merging/joining/reindexing only appears at this level</li>
</ul>
</li>
</ul>
<h3 dir="auto">Deliverables</h3>
<ul dir="auto">
<li>Stage 1 <strong>DONE</strong>
<ul dir="auto">
<li>make ref_locs primary source of information (leaving items/ref_items in place to back it up and avoid breakage)</li>
<li>port merging/joining internals to loc-based implementation (there's quite a number of hacks a.t.m. that make this non-trivial)</li>
<li>drop Block <code class="notranslate">items</code> &amp; <code class="notranslate">ref_items</code> fields (ensure io backward compatibility!)</li>
<li>fix performance issues &amp; integrate with mainline</li>
</ul>
</li>
<li>Stage 2: TBA</li>
</ul>
<p dir="auto">I found that segfault that messed up my HDFStore earlier.</p>
<p dir="auto">You'll need to csv files:</p>
<p dir="auto"><a href="https://www.dropbox.com/s/55gcpkp3b8byfka/df1.csv" rel="nofollow">https://www.dropbox.com/s/55gcpkp3b8byfka/df1.csv</a><br>
<a href="https://www.dropbox.com/s/mqqhfgocdg4qggp/df2.csv" rel="nofollow">https://www.dropbox.com/s/mqqhfgocdg4qggp/df2.csv</a></p>
<p dir="auto">To reproduce:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="In [1]: df1 = pd.read_csv('df1.csv', index_col=[0,1,2])

In [2]: df2 = pd.read_csv('df2.csv', index_col=[0,1,2])

In [3]: df2.loc[df1.index]
Segmentation fault: 11
(pandas-dev)dhcp80fff527:Desktop tom$ "><pre class="notranslate"><span class="pl-v">In</span> [<span class="pl-c1">1</span>]: <span class="pl-s1">df1</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-en">read_csv</span>(<span class="pl-s">'df1.csv'</span>, <span class="pl-s1">index_col</span><span class="pl-c1">=</span>[<span class="pl-c1">0</span>,<span class="pl-c1">1</span>,<span class="pl-c1">2</span>])

<span class="pl-v">In</span> [<span class="pl-c1">2</span>]: <span class="pl-s1">df2</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-en">read_csv</span>(<span class="pl-s">'df2.csv'</span>, <span class="pl-s1">index_col</span><span class="pl-c1">=</span>[<span class="pl-c1">0</span>,<span class="pl-c1">1</span>,<span class="pl-c1">2</span>])

<span class="pl-v">In</span> [<span class="pl-c1">3</span>]: <span class="pl-s1">df2</span>.<span class="pl-s1">loc</span>[<span class="pl-s1">df1</span>.<span class="pl-s1">index</span>]
<span class="pl-v">Segmentation</span> <span class="pl-s1">fault</span>: <span class="pl-c1">11</span>
(<span class="pl-s1">pandas</span><span class="pl-c1">-</span><span class="pl-s1">dev</span>)<span class="pl-s1">dhcp80fff527</span>:<span class="pl-v">Desktop</span> <span class="pl-s1">tom</span>$ </pre></div>
<p dir="auto">I've never debugged a segfault before, but I gather there are some <a href="https://wiki.python.org/moin/DebuggingWithGdb" rel="nofollow">methods</a>.  Can anyone reproduce this?</p>
<p dir="auto">Also, that's a valid use of <code class="notranslate">.loc</code> right?</p>
<p dir="auto">In [1]: pd.<strong>version</strong><br>
Out[1]: '0.12.0-1084-ged64416'</p>
<p dir="auto">In [2]: np.<strong>version</strong><br>
Out[2]: '1.7.1'</p>