<h4 dir="auto">Problem description</h4>
<p dir="auto">pd.read_hdf() segfaults when called from multiple threads in read only mode.</p>
<p dir="auto">From the documentation I expected that reading a HDF5 file in another thread would work.  In the example each file is mapped out to its own thread and the thread is responsible for opening and closing the file.  Note that when I used HDF5 files with compression the segfault happened everytime.  When I was making the minimal example, I used no compression and fround it does not segfault about 80% of the time.  The test code shows that when complib is set, but complevel is 0 (i.e no compression) the failure rate is 100%.</p>
<p dir="auto">This problem is also reported in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="131691514" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/12236" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/12236/hovercard" href="https://github.com/pandas-dev/pandas/issues/12236">#12236</a></p>
<h4 dir="auto">Test Code</h4>
<p dir="auto">Note this fails reliably when complib is not None.  When complib=None the failure rate is ~ 20% for me in limited testing.</p>
<p dir="auto">This is another example of # but shows effect of compression settings on the issue</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import numpy as np
import pandas as pd
import concurrent.futures as cf
                                # Failure rate
                                # - Fails with Segmentation fault
COMPLIB, COMPLEVEL = None, 0     # 2 out of 10 tests
COMPLIB, COMPLEVEL = 'zlib', 9   # 3 out of 3 tests
COMPLIB, COMPLEVEL = 'zlib', 0   # 3 out of 3 tests
COMPLIB, COMPLEVEL = 'blosc', 0  # 3 out of 3 tests
COMPLIB, COMPLEVEL = 'blosc', 1  # 3 out of 3 tests
COMPLIB, COMPLEVEL = 'lzo', 0    # 3 out of 3 tests
COMPLIB, COMPLEVEL = 'bzip2', 0  # 3 out of 3 tests

# create test data
a = np.random.rand(1000000)
b = np.random.rand(1000000)

d1 = pd.DataFrame(dict(a=a*1, b=b*1))
d2 = pd.DataFrame(dict(a=a*2, b=b*2))
d3 = pd.DataFrame(dict(a=a*3, b=b*3))
d4 = pd.DataFrame(dict(a=a*4, b=b*4))

d1.to_hdf('d1.h5', '/data', complib=COMPLIB, complevel=COMPLEVEL)
d2.to_hdf('d2.h5', '/data', complib=COMPLIB, complevel=COMPLEVEL)
d3.to_hdf('d3.h5', '/data', complib=COMPLIB, complevel=COMPLEVEL)
d4.to_hdf('d4.h5', '/data', complib=COMPLIB, complevel=COMPLEVEL)

files = ['d{}.h5'.format(i) for i in range(1, 5)]

# map reads out to threads
e = cf.ThreadPoolExecutor()
futures = [e.submit(pd.read_hdf, f, '/data/', mode='r') for f in files]
"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-k">import</span> <span class="pl-s1">pandas</span> <span class="pl-k">as</span> <span class="pl-s1">pd</span>
<span class="pl-k">import</span> <span class="pl-s1">concurrent</span>.<span class="pl-s1">futures</span> <span class="pl-k">as</span> <span class="pl-s1">cf</span>
                                <span class="pl-c"># Failure rate</span>
                                <span class="pl-c"># - Fails with Segmentation fault</span>
<span class="pl-v">COMPLIB</span>, <span class="pl-v">COMPLEVEL</span> <span class="pl-c1">=</span> <span class="pl-c1">None</span>, <span class="pl-c1">0</span>     <span class="pl-c"># 2 out of 10 tests</span>
<span class="pl-v">COMPLIB</span>, <span class="pl-v">COMPLEVEL</span> <span class="pl-c1">=</span> <span class="pl-s">'zlib'</span>, <span class="pl-c1">9</span>   <span class="pl-c"># 3 out of 3 tests</span>
<span class="pl-v">COMPLIB</span>, <span class="pl-v">COMPLEVEL</span> <span class="pl-c1">=</span> <span class="pl-s">'zlib'</span>, <span class="pl-c1">0</span>   <span class="pl-c"># 3 out of 3 tests</span>
<span class="pl-v">COMPLIB</span>, <span class="pl-v">COMPLEVEL</span> <span class="pl-c1">=</span> <span class="pl-s">'blosc'</span>, <span class="pl-c1">0</span>  <span class="pl-c"># 3 out of 3 tests</span>
<span class="pl-v">COMPLIB</span>, <span class="pl-v">COMPLEVEL</span> <span class="pl-c1">=</span> <span class="pl-s">'blosc'</span>, <span class="pl-c1">1</span>  <span class="pl-c"># 3 out of 3 tests</span>
<span class="pl-v">COMPLIB</span>, <span class="pl-v">COMPLEVEL</span> <span class="pl-c1">=</span> <span class="pl-s">'lzo'</span>, <span class="pl-c1">0</span>    <span class="pl-c"># 3 out of 3 tests</span>
<span class="pl-v">COMPLIB</span>, <span class="pl-v">COMPLEVEL</span> <span class="pl-c1">=</span> <span class="pl-s">'bzip2'</span>, <span class="pl-c1">0</span>  <span class="pl-c"># 3 out of 3 tests</span>

<span class="pl-c"># create test data</span>
<span class="pl-s1">a</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">rand</span>(<span class="pl-c1">1000000</span>)
<span class="pl-s1">b</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">rand</span>(<span class="pl-c1">1000000</span>)

<span class="pl-s1">d1</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">DataFrame</span>(<span class="pl-en">dict</span>(<span class="pl-s1">a</span><span class="pl-c1">=</span><span class="pl-s1">a</span><span class="pl-c1">*</span><span class="pl-c1">1</span>, <span class="pl-s1">b</span><span class="pl-c1">=</span><span class="pl-s1">b</span><span class="pl-c1">*</span><span class="pl-c1">1</span>))
<span class="pl-s1">d2</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">DataFrame</span>(<span class="pl-en">dict</span>(<span class="pl-s1">a</span><span class="pl-c1">=</span><span class="pl-s1">a</span><span class="pl-c1">*</span><span class="pl-c1">2</span>, <span class="pl-s1">b</span><span class="pl-c1">=</span><span class="pl-s1">b</span><span class="pl-c1">*</span><span class="pl-c1">2</span>))
<span class="pl-s1">d3</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">DataFrame</span>(<span class="pl-en">dict</span>(<span class="pl-s1">a</span><span class="pl-c1">=</span><span class="pl-s1">a</span><span class="pl-c1">*</span><span class="pl-c1">3</span>, <span class="pl-s1">b</span><span class="pl-c1">=</span><span class="pl-s1">b</span><span class="pl-c1">*</span><span class="pl-c1">3</span>))
<span class="pl-s1">d4</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">DataFrame</span>(<span class="pl-en">dict</span>(<span class="pl-s1">a</span><span class="pl-c1">=</span><span class="pl-s1">a</span><span class="pl-c1">*</span><span class="pl-c1">4</span>, <span class="pl-s1">b</span><span class="pl-c1">=</span><span class="pl-s1">b</span><span class="pl-c1">*</span><span class="pl-c1">4</span>))

<span class="pl-s1">d1</span>.<span class="pl-en">to_hdf</span>(<span class="pl-s">'d1.h5'</span>, <span class="pl-s">'/data'</span>, <span class="pl-s1">complib</span><span class="pl-c1">=</span><span class="pl-v">COMPLIB</span>, <span class="pl-s1">complevel</span><span class="pl-c1">=</span><span class="pl-v">COMPLEVEL</span>)
<span class="pl-s1">d2</span>.<span class="pl-en">to_hdf</span>(<span class="pl-s">'d2.h5'</span>, <span class="pl-s">'/data'</span>, <span class="pl-s1">complib</span><span class="pl-c1">=</span><span class="pl-v">COMPLIB</span>, <span class="pl-s1">complevel</span><span class="pl-c1">=</span><span class="pl-v">COMPLEVEL</span>)
<span class="pl-s1">d3</span>.<span class="pl-en">to_hdf</span>(<span class="pl-s">'d3.h5'</span>, <span class="pl-s">'/data'</span>, <span class="pl-s1">complib</span><span class="pl-c1">=</span><span class="pl-v">COMPLIB</span>, <span class="pl-s1">complevel</span><span class="pl-c1">=</span><span class="pl-v">COMPLEVEL</span>)
<span class="pl-s1">d4</span>.<span class="pl-en">to_hdf</span>(<span class="pl-s">'d4.h5'</span>, <span class="pl-s">'/data'</span>, <span class="pl-s1">complib</span><span class="pl-c1">=</span><span class="pl-v">COMPLIB</span>, <span class="pl-s1">complevel</span><span class="pl-c1">=</span><span class="pl-v">COMPLEVEL</span>)

<span class="pl-s1">files</span> <span class="pl-c1">=</span> [<span class="pl-s">'d{}.h5'</span>.<span class="pl-en">format</span>(<span class="pl-s1">i</span>) <span class="pl-k">for</span> <span class="pl-s1">i</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-c1">1</span>, <span class="pl-c1">5</span>)]

<span class="pl-c"># map reads out to threads</span>
<span class="pl-s1">e</span> <span class="pl-c1">=</span> <span class="pl-s1">cf</span>.<span class="pl-v">ThreadPoolExecutor</span>()
<span class="pl-s1">futures</span> <span class="pl-c1">=</span> [<span class="pl-s1">e</span>.<span class="pl-en">submit</span>(<span class="pl-s1">pd</span>.<span class="pl-s1">read_hdf</span>, <span class="pl-s1">f</span>, <span class="pl-s">'/data/'</span>, <span class="pl-s1">mode</span><span class="pl-c1">=</span><span class="pl-s">'r'</span>) <span class="pl-k">for</span> <span class="pl-s1">f</span> <span class="pl-c1">in</span> <span class="pl-s1">files</span>]</pre></div>
<h4 dir="auto">Expected Output</h4>
<p dir="auto">a list of 4 futures that can be</p>
<h4 dir="auto">Output of <code class="notranslate">pd.show_versions()</code></h4>
<details>
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.0.final.0
python-bits: 64
OS: Darwin
OS-release: 15.6.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
<p dir="auto">pandas: 0.19.2<br>
nose: None<br>
pip: 9.0.1<br>
setuptools: 28.8.0<br>
Cython: None<br>
numpy: 1.12.0<br>
scipy: None<br>
statsmodels: None<br>
xarray: None<br>
IPython: 5.1.0<br>
sphinx: None<br>
patsy: None<br>
dateutil: 2.6.0<br>
pytz: 2016.10<br>
blosc: None<br>
bottleneck: None<br>
tables: 3.3.0<br>
numexpr: 2.6.1<br>
matplotlib: 2.0.0<br>
openpyxl: None<br>
xlrd: None<br>
xlwt: None<br>
xlsxwriter: None<br>
lxml: None<br>
bs4: None<br>
html5lib: 0.9999999<br>
httplib2: None<br>
apiclient: None<br>
sqlalchemy: None<br>
pymysql: None<br>
psycopg2: None<br>
jinja2: 2.9.4<br>
boto: None<br>
pandas_datareader: None</p>
</details>
<p dir="auto">Crash Report</p>
<details>
python(7164,0x700000c10000) malloc: *** error for object 0x7f8ba3655e60: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6
</details>
<p dir="auto">I was about to answer an SO question giving the standard "use <code class="notranslate">lambda</code> or <code class="notranslate">functools.partial</code>" to create a new function with bound arguments when to my surprise my example didn't work with the OP's code.  After some experimentation, it turns out to be because we don't always get the same return type.  For example:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="df = pd.DataFrame(np.arange(5, dtype=np.int64), 
                  index=pd.DatetimeIndex(start='2014/01/01', periods=5, freq='d'))

def f(x,a=1):
    print(type(x))
    return int(isinstance(x, pd.DataFrame)) + 1000*a"><pre class="notranslate"><code class="notranslate">df = pd.DataFrame(np.arange(5, dtype=np.int64), 
                  index=pd.DatetimeIndex(start='2014/01/01', periods=5, freq='d'))

def f(x,a=1):
    print(type(x))
    return int(isinstance(x, pd.DataFrame)) + 1000*a
</code></pre></div>
<p dir="auto">After which:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; df.resample(&quot;M&quot;, how=f)
&lt;class 'pandas.core.series.Series'&gt;
               0
2014-01-31  1000
&gt;&gt;&gt; df.resample(&quot;M&quot;, how=lambda x: f(x,5))
&lt;class 'pandas.core.series.Series'&gt;
               0
2014-01-31  5000
&gt;&gt;&gt; df.resample(&quot;M&quot;, how=partial(f, a=9))
&lt;class 'pandas.core.frame.DataFrame'&gt;
               0
2014-01-31  9001"><pre class="notranslate"><code class="notranslate">&gt;&gt;&gt; df.resample("M", how=f)
&lt;class 'pandas.core.series.Series'&gt;
               0
2014-01-31  1000
&gt;&gt;&gt; df.resample("M", how=lambda x: f(x,5))
&lt;class 'pandas.core.series.Series'&gt;
               0
2014-01-31  5000
&gt;&gt;&gt; df.resample("M", how=partial(f, a=9))
&lt;class 'pandas.core.frame.DataFrame'&gt;
               0
2014-01-31  9001
</code></pre></div>
<p dir="auto"><code class="notranslate">how</code> shouldn't care about how the function was constructed, only whether it's callable, and we should be getting the same result (whether Series or DataFrame) fed into <code class="notranslate">how</code> in every case.</p>