<p dir="auto">Summary: there is some non-deterministic crash while doing <code class="notranslate">make check</code> that is easier for me to reproduce when I set environment variable <code class="notranslate">RUST_THREADS=2</code> during the <code class="notranslate">make check</code>.  This is on a build with both <code class="notranslate">--enable-debug</code> and <code class="notranslate">--enable-optimize</code>; I am not sure if removing those options actually makes the bug go away, or merely makes it harder to reproduce.  (Update: <code class="notranslate">--enable-optimize</code> is not relevant; it can be toggled on/off and the bug persists.)</p>
<p dir="auto">It appears to have been injected by commit <a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/rust-lang/rust/commit/41dcec2fe16e272016ae77d10a6a5ff3a737f192/hovercard" href="https://github.com/rust-lang/rust/commit/41dcec2fe16e272016ae77d10a6a5ff3a737f192"><tt>41dcec2</tt></a></p>
<p dir="auto">This is a merge commit by bors; its parent supplied by bors is: <a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/rust-lang/rust/commit/137d1fb210a844a76f89d7355a1aaf9f7a88af33/hovercard" href="https://github.com/rust-lang/rust/commit/137d1fb210a844a76f89d7355a1aaf9f7a88af33"><tt>137d1fb</tt></a></p>
<p dir="auto">Things seem to work fine on the parent.</p>
<p dir="auto">﻿<br>
I found during my attempts to bisect the history that I could not trust our build infrastructure to reliably update LLVM source and/or rebuild the object code (perhaps for LLVM or perhaps elsewhere), so I got in the habit of deleting the both the LLVM source tree and the entire build tree (and re-running configure) during the bisection.  This of course makes the build take a bit longer.  (But maybe now that I have narrowed it to this particular merge, perhaps one can be less conservative with how much cleanup one does before testing.)</p>
<p dir="auto">So, to reproduce, here is the command I was running in the build tree:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="export RUST_THREADS=2 ; git log -1 ; echo ; echo ; rm -rf ../src/llvm/ ; rm -rf * ; echo ; echo ; /Users/pnkfelix/Dev/Mozilla/rust.git/configure --enable-debug --enable-optimize  --enable-clang --prefix=~/opt/rust-dbg ; remake clean clean-llvm ; time make check "><pre class="notranslate"><code class="notranslate">export RUST_THREADS=2 ; git log -1 ; echo ; echo ; rm -rf ../src/llvm/ ; rm -rf * ; echo ; echo ; /Users/pnkfelix/Dev/Mozilla/rust.git/configure --enable-debug --enable-optimize  --enable-clang --prefix=~/opt/rust-dbg ; remake clean clean-llvm ; time make check 
</code></pre></div>
<p dir="auto">(There is certainly steps that can be omitted from the above, like the <code class="notranslate">git log -1</code>, which was for my own logging purposes while I bisected.  <code class="notranslate">remake</code> is the fork of GNU make I often use: <a href="http://bashdb.sourceforge.net/remake/" rel="nofollow">remake</a>, but that is almost certainly orthogonal.)</p>
<p dir="auto">I wish I had better information (either a more specific commit from the originating branch, or a narrow test test), but as of yet I have not been able to gather such.  I'm mostly filing this so that I keep a log of the crashes and any other detail I manage to gather.</p>
<p dir="auto">A few of the crashes I saw occurred while testing <code class="notranslate">rustpkg</code>, and included this as the output:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="  rustc: for the -arm-enable-ehabi option: may only occur zero or one times!
  rustc: for the -arm-enable-ehabi-descriptors option: may only occur zero or one times!"><pre class="notranslate"><code class="notranslate">  rustc: for the -arm-enable-ehabi option: may only occur zero or one times!
  rustc: for the -arm-enable-ehabi-descriptors option: may only occur zero or one times!
</code></pre></div>
<p dir="auto">shortly before crashing; from what I can tell from skimming the source code, there is no way we should be seeing that message from tests on an Intel Mac, unless some internal state has been corrupted.  But the failure is non-deterministic, so this does not happen reliably.</p>
<p dir="auto">A few samples of the sorts of crashes that one sees while running <code class="notranslate">make check</code>, some while using <code class="notranslate">RUST_LOG</code> to get more info (I have more complete logs available on request); I had to post process some of the output, replacing raw bytes unacceptable to this text box e.g. \342\200\224 with character strings representing those byte sequences:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="test rt::io::net::tcp::test::bind_error ... ignored
test rt::io::extensions::test::read_bytes ... ok
test result::tests::chain_failure ... ok
rust: ~&quot;waking up task for 140313321592696&quot;
test rt::io::extensions::test::push_bytes_partial ... ok
test rt::io::mem::test::test_mem_writer ... okrust: ~&quot;waking up task for 140313322615688&quot;
rust: ~&quot;waking up task for 140313318415048&quot;
error opening /dev/urandom: Too many open files"><pre class="notranslate"><code class="notranslate">test rt::io::net::tcp::test::bind_error ... ignored
test rt::io::extensions::test::read_bytes ... ok
test result::tests::chain_failure ... ok
rust: ~"waking up task for 140313321592696"
test rt::io::extensions::test::push_bytes_partial ... ok
test rt::io::mem::test::test_mem_writer ... okrust: ~"waking up task for 140313322615688"
rust: ~"waking up task for 140313318415048"
error opening /dev/urandom: Too many open files
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="test rt::io::extensions::test::read_to_end ... ok
test rt::io::extensions::test::read_byte_0_bytes ... ok
rust: ~&quot;blocked = 0 this = 7f9e45c08be0&quot;
rust: ~&quot;blocked = 7f9e45c08be0 this = 7f9e45c08be0 old_task = 0&quot;
rust: ~&quot;woke up, p.state = Full&quot;
rust: ~&quot;no data available on &amp;mut {header: {state: Blocked, blocked_task: (0x7f9e45c08be0 as *()), buffer: (0x7f9e45a038c0 as *())}, payload: None}, going to sleep.&quot;
task failed at 'assertion failed: handle.is_not_null()', /Users/pnkfelix/Dev/Mozilla/rust.git/src/libstd/rt/uv/mod.rs:100
e
There was a night when winds from unknown spaces whirled us irresistibly into
limitless vacum beyond all thought and entity. Perceptions of the most
maddeningly untransmissible sort thronged upon us; perceptions of infinity
which at the time convulsed us with joy, yet which are now partly lost to my
memory and partly incapable of presentation to others.

fatal runtime error: assertion failed: void_sched.is_not_null()"><pre class="notranslate"><code class="notranslate">test rt::io::extensions::test::read_to_end ... ok
test rt::io::extensions::test::read_byte_0_bytes ... ok
rust: ~"blocked = 0 this = 7f9e45c08be0"
rust: ~"blocked = 7f9e45c08be0 this = 7f9e45c08be0 old_task = 0"
rust: ~"woke up, p.state = Full"
rust: ~"no data available on &amp;mut {header: {state: Blocked, blocked_task: (0x7f9e45c08be0 as *()), buffer: (0x7f9e45a038c0 as *())}, payload: None}, going to sleep."
task failed at 'assertion failed: handle.is_not_null()', /Users/pnkfelix/Dev/Mozilla/rust.git/src/libstd/rt/uv/mod.rs:100
e
There was a night when winds from unknown spaces whirled us irresistibly into
limitless vacum beyond all thought and entity. Perceptions of the most
maddeningly untransmissible sort thronged upon us; perceptions of infinity
which at the time convulsed us with joy, yet which are now partly lost to my
memory and partly incapable of presentation to others.

fatal runtime error: assertion failed: void_sched.is_not_null()
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="test rt::comm::test::oneshot_single_thread_send_then_recv ... ok
test rt::io::mem::test::test_mem_reader ... ok
rust: ~&quot;blocked = 0 this = 7fea01519590&quot;
rust: ~&quot;blocked = 7fea01519590 this = 7fea01519590 old_task = 0&quot;
rust: ~&quot;no data available on &amp;mut {header: {state: Blocked, blocked_task: (0x7fea01519590 as *()), buffer: (0x7fea0150ec00 as *())}, payload: None}, going to sleep.&quot;
rust: ~&quot;waking up task for 140643020157512&quot;
test rt::io::net::tcp::test::connect_error ... ok
rust: ~&quot;blocked = 0 this = 7fea02a022a0&quot;
test rt::io::extensions::test::push_bytes_partial ... ok
rust: ~&quot;blocked = 7fea02a022a0 this = 7fea02a022a0 old_task = 0&quot;
rust: ~&quot;blocked = 0 this = 7fea02b15670&quot;
test rt::comm::test::oneshot_single_thread_send_port_close ... ok
rust: ~&quot;blocked = 0 this = 7fea02a04750&quot;
rust: ~&quot;blocked = 7fea02b15670 this = 7fea02b15670 old_task = 0&quot;
test rt::io::extensions::test::read_to_end_error ... ok
rust: ~&quot;no data available on &amp;mut {header: {state: Blocked, blocked_task: (0x7fea02a022a0 as *()), buffer: (0x7fea01418ce0 as *())}, payload: None}, going to sleep.&quot;
rust: ~&quot;blocked = 7fea02a04750 this = 7fea02a04750 old_task = 0&quot;
rust: ~&quot;blocked = 0 this = 7fea01519890&quot;
rust: ~&quot;no data available on &amp;mut {header: {state: Blocked, blocked_task: (0x7fea02b15670 as *()), buffer: (0x7fea02b154e0 as *())}, payload: None}, going to sleep.&quot;
test rt::io::extensions::test::read_bytes ... okrust: ~&quot;blocked = 7fea01519890 this = 7fea01519890 old_task = 0&quot;
rust: ~&quot;no data available on &amp;mut {header: {state: Blocked, blocked_task: (0x7fea02a04750 as *()), buffer: (0x7fea02a00800 as *())}, payload: None}, going to sleep.&quot;
rust: ~&quot;no data available on &amp;mut {header: {state: Blocked, blocked_task: (0x7fea01519890 as *()), buffer: (0x7fea01609fa0 as *())}, payload: None}, going to sleep.&quot;
test rt::comm::test::oneshot_single_thread_close_port_first ... ok
rust: ~&quot;blocked = 0 this = 7fea01601c60&quot;
rust: ~&quot;blocked = 7fea01601c60 this = 7fea01601c60 old_task = 0&quot;
rust: ~&quot;no data available on &amp;mut {header: {state: Blocked, blocked_task: (0x7fea01601c60 as *()), buffer: (0x7fea0280f020 as *())}, payload: None}, going to sleep.&quot;
(libuv) Failed to create kqueue (24)


Instead of the poems I had hoped for, there came only a shuddering blackness
and ineffable loneliness; and I saw at last a fearful truth which no one had
ever dared to breathe before \342\200\224 the unwhisperable secret of secrets \342\200\224 The fact
that this city of stone and stridor is not a sentient perpetuation of Old New
York as London is of Old London and Paris of Old Paris, but that it is in fact
quite dead, its sprawling body imperfectly embalmed and infested with queer
animate things which have nothing to do with it as it was in life.

fatal runtime error: no scheduler"><pre class="notranslate"><code class="notranslate">test rt::comm::test::oneshot_single_thread_send_then_recv ... ok
test rt::io::mem::test::test_mem_reader ... ok
rust: ~"blocked = 0 this = 7fea01519590"
rust: ~"blocked = 7fea01519590 this = 7fea01519590 old_task = 0"
rust: ~"no data available on &amp;mut {header: {state: Blocked, blocked_task: (0x7fea01519590 as *()), buffer: (0x7fea0150ec00 as *())}, payload: None}, going to sleep."
rust: ~"waking up task for 140643020157512"
test rt::io::net::tcp::test::connect_error ... ok
rust: ~"blocked = 0 this = 7fea02a022a0"
test rt::io::extensions::test::push_bytes_partial ... ok
rust: ~"blocked = 7fea02a022a0 this = 7fea02a022a0 old_task = 0"
rust: ~"blocked = 0 this = 7fea02b15670"
test rt::comm::test::oneshot_single_thread_send_port_close ... ok
rust: ~"blocked = 0 this = 7fea02a04750"
rust: ~"blocked = 7fea02b15670 this = 7fea02b15670 old_task = 0"
test rt::io::extensions::test::read_to_end_error ... ok
rust: ~"no data available on &amp;mut {header: {state: Blocked, blocked_task: (0x7fea02a022a0 as *()), buffer: (0x7fea01418ce0 as *())}, payload: None}, going to sleep."
rust: ~"blocked = 7fea02a04750 this = 7fea02a04750 old_task = 0"
rust: ~"blocked = 0 this = 7fea01519890"
rust: ~"no data available on &amp;mut {header: {state: Blocked, blocked_task: (0x7fea02b15670 as *()), buffer: (0x7fea02b154e0 as *())}, payload: None}, going to sleep."
test rt::io::extensions::test::read_bytes ... okrust: ~"blocked = 7fea01519890 this = 7fea01519890 old_task = 0"
rust: ~"no data available on &amp;mut {header: {state: Blocked, blocked_task: (0x7fea02a04750 as *()), buffer: (0x7fea02a00800 as *())}, payload: None}, going to sleep."
rust: ~"no data available on &amp;mut {header: {state: Blocked, blocked_task: (0x7fea01519890 as *()), buffer: (0x7fea01609fa0 as *())}, payload: None}, going to sleep."
test rt::comm::test::oneshot_single_thread_close_port_first ... ok
rust: ~"blocked = 0 this = 7fea01601c60"
rust: ~"blocked = 7fea01601c60 this = 7fea01601c60 old_task = 0"
rust: ~"no data available on &amp;mut {header: {state: Blocked, blocked_task: (0x7fea01601c60 as *()), buffer: (0x7fea0280f020 as *())}, payload: None}, going to sleep."
(libuv) Failed to create kqueue (24)


Instead of the poems I had hoped for, there came only a shuddering blackness
and ineffable loneliness; and I saw at last a fearful truth which no one had
ever dared to breathe before \342\200\224 the unwhisperable secret of secrets \342\200\224 The fact
that this city of stone and stridor is not a sentient perpetuation of Old New
York as London is of Old London and Paris of Old Paris, but that it is in fact
quite dead, its sprawling body imperfectly embalmed and infested with queer
animate things which have nothing to do with it as it was in life.

fatal runtime error: no scheduler
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="test rt::io::extensions::test::push_bytes_eof ... ok
test rt::comm::test::oneshot_single_thread_try_send_closed ... ok
test rt::io::file::super_simple_smoke_test_lets_go_read_some_files_and_have_a_good_time ... ignored
rust: ~&quot;no data available on &amp;mut {header: {state: Blocked, blocked_task: (0x7f81dbe030a0 as *()), buffer: (0x7f81ddc011b0 as *())}, payload: None}, going to sleep.&quot;
test rt::io::flate::test::smoke_test ... ignored
rust: ~&quot;\&quot;Trap: pushing handler to TLS\&quot;&quot;
rust: ~&quot;blocked = 0 this = 7f81ddb03a20&quot;
rust: ~&quot;blocked = 7f81ddb03a20 this = 7f81ddb03a20 old_task = 0&quot;
rust: ~&quot;\&quot;Condition.raise: found handler\&quot;&quot;
rust: ~&quot;\&quot;Guard: popping handler from TLS\&quot;&quot;
rust: ~&quot;no data available on &amp;mut {header: {state: Blocked, blocked_task: (0x7f81ddb03a20 as *()), buffer: (0x7f81ddb025e0 as *())}, payload: None}, going to sleep.&quot;
rust: ~&quot;woke up, p.state = Full&quot;
test rt::comm::test::megapipe_stress ... okrust: ~&quot;blocked = 0 this = 7f81dd901580&quot;
rust: ~&quot;blocked = 0 this = 7f81dda01dd0&quot;
rust: ~&quot;woke up, p.state = Full&quot;
rust: ~&quot;\&quot;read 0 bytes. trying again\&quot;&quot;
s ... ok
test rt::io::extensions::test::read_bytes_eof ... ok
rust: ~&quot;blocked = 7f81dd901580 this = 7f81dd901580 old_task = 0&quot;
test rt::comm::test::oneshot_single_thread_try_recv_open ... okrust: ~&quot;blocked = 7f81dda01dd0 this = 7f81dda01dd0 old_task = 0&quot;
rust: ~&quot;\&quot;Trap: pushing handler to TLS\&quot;&quot;
rust: ~&quot;no data available on &amp;mut {header: {state: Blocked, blocked_task: (0x7f81dd901580 as *()), buffer: (0x7f81ddc028a0 as *())}, payload: None}, going to sleep.&quot;
rust: ~&quot;\&quot;Condition.raise: found handler\&quot;&quot;
rust: ~&quot;no data available on &amp;mut {header: {state: Blocked, blocked_task: (0x7f81dda01dd0 as *()), buffer: (0x7f81dbe06a10 as *())}, payload: None}, going to sleep.&quot;
rust: ~&quot;\&quot;Condition.raise: found handler\&quot;&quot;
rust: ~&quot;\&quot;Guard: popping handler from TLS\&quot;&quot;
rust: ~&quot;\&quot;Trap: pushing handler to TLS\&quot;&quot;
rust: ~&quot;\&quot;Condition.raise: found handler\&quot;&quot;
rust: ~&quot;\&quot;Condition.raise: found no handler\&quot;&quot;
rust: task failed at 'Unhandled condition: read_error: {kind: OtherIoError, desc: &quot;Placeholder error. You shouldn\'t be seeing this&quot;, detail: None}', /Users/pnkfelix/Dev/Mozilla/rust.git/src/libstd/condition.rs:43
test rt::io::net::http::test::smoke_test ... ignored
test rt::io::net::tcp::test::bind_error ... ignored
test rt::io::extensions::test::read_byte_0_bytes ... ok
test rt::comm::test::oneshot_multi_task_recv_then_close ... ok
rust: ~&quot;blocked = 0 this = 7f81dbe08970&quot;
rust: ~&quot;blocked = 7f81dbe08970 this = 7f81dbe08970 old_task = 0&quot;
test rt::io::extensions::test::read_byte ... ok
rust: ~&quot;no data available on &amp;mut {header: {state: Blocked, blocked_task: (0x7f81dbe08970 as *()), buffer: (0x7f81ddb07b80 as *())}, payload: None}, going to sleep.&quot;
rust: ~&quot;waking up task for 140195716426264&quot;
~&quot;\&quot;Trap: pushing handler to TLS\&quot;&quot;"><pre class="notranslate"><code class="notranslate">test rt::io::extensions::test::push_bytes_eof ... ok
test rt::comm::test::oneshot_single_thread_try_send_closed ... ok
test rt::io::file::super_simple_smoke_test_lets_go_read_some_files_and_have_a_good_time ... ignored
rust: ~"no data available on &amp;mut {header: {state: Blocked, blocked_task: (0x7f81dbe030a0 as *()), buffer: (0x7f81ddc011b0 as *())}, payload: None}, going to sleep."
test rt::io::flate::test::smoke_test ... ignored
rust: ~"\"Trap: pushing handler to TLS\""
rust: ~"blocked = 0 this = 7f81ddb03a20"
rust: ~"blocked = 7f81ddb03a20 this = 7f81ddb03a20 old_task = 0"
rust: ~"\"Condition.raise: found handler\""
rust: ~"\"Guard: popping handler from TLS\""
rust: ~"no data available on &amp;mut {header: {state: Blocked, blocked_task: (0x7f81ddb03a20 as *()), buffer: (0x7f81ddb025e0 as *())}, payload: None}, going to sleep."
rust: ~"woke up, p.state = Full"
test rt::comm::test::megapipe_stress ... okrust: ~"blocked = 0 this = 7f81dd901580"
rust: ~"blocked = 0 this = 7f81dda01dd0"
rust: ~"woke up, p.state = Full"
rust: ~"\"read 0 bytes. trying again\""
s ... ok
test rt::io::extensions::test::read_bytes_eof ... ok
rust: ~"blocked = 7f81dd901580 this = 7f81dd901580 old_task = 0"
test rt::comm::test::oneshot_single_thread_try_recv_open ... okrust: ~"blocked = 7f81dda01dd0 this = 7f81dda01dd0 old_task = 0"
rust: ~"\"Trap: pushing handler to TLS\""
rust: ~"no data available on &amp;mut {header: {state: Blocked, blocked_task: (0x7f81dd901580 as *()), buffer: (0x7f81ddc028a0 as *())}, payload: None}, going to sleep."
rust: ~"\"Condition.raise: found handler\""
rust: ~"no data available on &amp;mut {header: {state: Blocked, blocked_task: (0x7f81dda01dd0 as *()), buffer: (0x7f81dbe06a10 as *())}, payload: None}, going to sleep."
rust: ~"\"Condition.raise: found handler\""
rust: ~"\"Guard: popping handler from TLS\""
rust: ~"\"Trap: pushing handler to TLS\""
rust: ~"\"Condition.raise: found handler\""
rust: ~"\"Condition.raise: found no handler\""
rust: task failed at 'Unhandled condition: read_error: {kind: OtherIoError, desc: "Placeholder error. You shouldn\'t be seeing this", detail: None}', /Users/pnkfelix/Dev/Mozilla/rust.git/src/libstd/condition.rs:43
test rt::io::net::http::test::smoke_test ... ignored
test rt::io::net::tcp::test::bind_error ... ignored
test rt::io::extensions::test::read_byte_0_bytes ... ok
test rt::comm::test::oneshot_multi_task_recv_then_close ... ok
rust: ~"blocked = 0 this = 7f81dbe08970"
rust: ~"blocked = 7f81dbe08970 this = 7f81dbe08970 old_task = 0"
test rt::io::extensions::test::read_byte ... ok
rust: ~"no data available on &amp;mut {header: {state: Blocked, blocked_task: (0x7f81dbe08970 as *()), buffer: (0x7f81ddb07b80 as *())}, payload: None}, going to sleep."
rust: ~"waking up task for 140195716426264"
~"\"Trap: pushing handler to TLS\""
</code></pre></div>
<p dir="auto">E.g. every <code class="notranslate">Examples</code> heading on <a href="http://doc.rust-lang.org/nightly/std/primitive.str.html" rel="nofollow">http://doc.rust-lang.org/nightly/std/primitive.str.html</a> has the same id (<code class="notranslate">examples</code>) and all link to <a href="http://doc.rust-lang.org/nightly/std/primitive.str.html#examples" rel="nofollow">http://doc.rust-lang.org/nightly/std/primitive.str.html#examples</a> . Rustdoc should adjust those internals to ensure the links are unique (e.g. <code class="notranslate">examples</code>, <code class="notranslate">examples-1</code>, <code class="notranslate">examples-2</code>, ...).</p>