<p dir="auto">I am using TF2.0 latest nightly build and I am trying to train LSTM model for text classification on very large dataset of 16455928 sentences. For embedding layer in the model, I have a vocab size of 366856 and I used 1000 as embedding dimension value in it, on which the 2 GPUs(Tesla T4 from Google) ran out of memory.<br>
Since I can not lower the size of vocabulary (maybe there is a way), so I used lower value for embedding dimension (100) on which the model starts training. Now my question is if there is a way I can use higher value of embedding dimension?. Maybe by putting set of layers of my model on different GPUs, if so then what is the way in TF2.0? Also, will using more number GPUs help? Thank you!</p>
<p dir="auto"><em>Please make sure that this is a bug. As per our <a href="https://github.com/tensorflow/tensorflow/blob/master/ISSUES.md">GitHub Policy</a>, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template</em></p>
<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes</li>
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04):<br>
Mac Sierra</li>
<li>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:</li>
<li>TensorFlow installed from (source or binary):<br>
binary</li>
<li>TensorFlow version (use command below):<br>
v1.12.0-rc2-3-ga6d8ffae09 1.12.0</li>
<li>Python version:<br>
3.6.5</li>
<li>Bazel version (if compiling from source):</li>
<li>GCC/Compiler version (if compiling from source):</li>
<li>CUDA/cuDNN version:<br>
no Cuda</li>
<li>GPU model and memory:<br>
not using GPU</li>
</ul>
<p dir="auto">You can collect some of this information using our environment capture <a href="https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh">script</a><br>
You can also obtain the TensorFlow version with<br>
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"</p>
<p dir="auto"><strong>Describe the current behavior</strong><br>
I'm trying Magenta from Google Brain. When I run <code class="notranslate">python onsets_frames_transcription_transcribe.py --acoustic_run_dir /Users/lorenzori/Downloads/maestro_checkpoint  ~/Downloads/test_audio.wav</code> the script starts, then it ends with:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="/Users/lorenzori/virtualenvs/test-audio/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  return _inspect.getargspec(target)
/Users/lorenzori/virtualenvs/test-audio/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  return _inspect.getargspec(target)
/Users/lorenzori/virtualenvs/test-audio/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  return _inspect.getargspec(target)
2019-01-10 17:49:57.458805: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
INFO:tensorflow:Restoring parameters from /Users/lorenzori/Downloads/maestro_checkpoint/train/model.ckpt-maestro
INFO:tensorflow:Starting transcription for /Users/lorenzori/Downloads/test_audio.wav...
INFO:tensorflow:Processing file...
INFO:tensorflow:Running inference...
[1]    20612 bus error  python onsets_frames_transcription_transcribe.py --acoustic_run_dir"><pre class="notranslate"><code class="notranslate">/Users/lorenzori/virtualenvs/test-audio/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  return _inspect.getargspec(target)
/Users/lorenzori/virtualenvs/test-audio/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  return _inspect.getargspec(target)
/Users/lorenzori/virtualenvs/test-audio/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  return _inspect.getargspec(target)
2019-01-10 17:49:57.458805: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
INFO:tensorflow:Restoring parameters from /Users/lorenzori/Downloads/maestro_checkpoint/train/model.ckpt-maestro
INFO:tensorflow:Starting transcription for /Users/lorenzori/Downloads/test_audio.wav...
INFO:tensorflow:Processing file...
INFO:tensorflow:Running inference...
[1]    20612 bus error  python onsets_frames_transcription_transcribe.py --acoustic_run_dir
</code></pre></div>
<p dir="auto">Using lldb I obtain the following info:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Process 19280 stopped
* thread #25, stop reason = EXC_BAD_ACCESS (code=2, address=0x700001382000)
    frame #0: 0x0000000110a24805 libopenblasp-r0.3.0.dev.dylib`dgemm_thread_tn + 1541
libopenblasp-r0.3.0.dev.dylib`dgemm_thread_tn:
-&gt;  0x110a24805 &lt;+1541&gt;: xchgq  %rdi, -0x40(%rsi)
    0x110a24809 &lt;+1545&gt;: xorl   %edi, %edi
    0x110a2480b &lt;+1547&gt;: xchgq  %rdi, (%rsi)
    0x110a2480e &lt;+1550&gt;: addq   $0x200, %rsi              ; imm = 0x200
Target 0: (python) stopped.
(lldb) bt
* thread #25, stop reason = EXC_BAD_ACCESS (code=2, address=0x700001382000)
  * frame #0: 0x0000000110a24805 libopenblasp-r0.3.0.dev.dylib`dgemm_thread_tn + 1541
    frame #1: 0x00000001108f3e26 libopenblasp-r0.3.0.dev.dylib`cblas_dgemm + 854
    frame #2: 0x0000000105564285 multiarray.cpython-36m-darwin.so`cblas_matrixproduct + 4917
    frame #3: 0x0000000105529d37 multiarray.cpython-36m-darwin.so`PyArray_MatrixProduct2 + 215
    frame #4: 0x000000010552ed1f multiarray.cpython-36m-darwin.so`array_matrixproduct + 191
    frame #5: 0x00000001000d1cbe Python`_PyCFunction_FastCallDict + 463
    frame #6: 0x00000001001362d6 Python`call_function + 489
    frame #7: 0x000000010012f18b Python`_PyEval_EvalFrameDefault + 4811
    frame #8: 0x0000000100136a38 Python`_PyEval_EvalCodeWithName + 1719
    frame #9: 0x000000010013713b Python`fast_function + 218
    frame #10: 0x00000001001362ad Python`call_function + 448
    frame #11: 0x000000010012f224 Python`_PyEval_EvalFrameDefault + 4964
    frame #12: 0x00000001001373db Python`_PyFunction_FastCall + 121
    frame #13: 0x00000001001362ad Python`call_function + 448
    frame #14: 0x000000010012f18b Python`_PyEval_EvalFrameDefault + 4811
    frame #15: 0x0000000100136a38 Python`_PyEval_EvalCodeWithName + 1719
    frame #16: 0x000000010013730b Python`_PyFunction_FastCallDict + 449
    frame #17: 0x0000000100099f21 Python`_PyObject_FastCallDict + 196
    frame #18: 0x0000000100182073 Python`partial_call + 258
    frame #19: 0x0000000100099da2 Python`PyObject_Call + 101
    frame #20: 0x000000010012f3f4 Python`_PyEval_EvalFrameDefault + 5428
    frame #21: 0x0000000100136a38 Python`_PyEval_EvalCodeWithName + 1719
    frame #22: 0x000000010013730b Python`_PyFunction_FastCallDict + 449
    frame #23: 0x0000000100099f21 Python`_PyObject_FastCallDict + 196
    frame #24: 0x000000010009a044 Python`_PyObject_Call_Prepend + 156
    frame #25: 0x0000000100099da2 Python`PyObject_Call + 101
    frame #26: 0x00000001000e4460 Python`slot_tp_call + 50
    frame #27: 0x0000000100099da2 Python`PyObject_Call + 101
    frame #28: 0x00000001212b078e _pywrap_tensorflow_internal.so`tensorflow::PyFuncOp::Compute(tensorflow::OpKernelContext*) + 974
    frame #29: 0x000000012bd82422 libtensorflow_framework.so`tensorflow::(anonymous namespace)::ExecutorState::Process(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long) + 6690
    frame #30: 0x000000012bd895ba libtensorflow_framework.so`std::__1::__function::__func&lt;std::__1::__bind&lt;void (tensorflow::(anonymous namespace)::ExecutorState::*)(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long), tensorflow::(anonymous namespace)::ExecutorState*, tensorflow::(anonymous namespace)::ExecutorState::TaggedNode const&amp;, long long&amp;&gt;, std::__1::allocator&lt;std::__1::__bind&lt;void (tensorflow::(anonymous namespace)::ExecutorState::*)(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long), tensorflow::(anonymous namespace)::ExecutorState*, tensorflow::(anonymous namespace)::ExecutorState::TaggedNode const&amp;, long long&amp;&gt; &gt;, void ()&gt;::operator()() + 58
    frame #31: 0x000000012bdde824 libtensorflow_framework.so`Eigen::NonBlockingThreadPoolTempl&lt;tensorflow::thread::EigenEnvironment&gt;::WorkerLoop(int) + 1876
    frame #32: 0x000000012bdddfd4 libtensorflow_framework.so`std::__1::__function::__func&lt;tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function&lt;void ()&gt;)::'lambda'(), std::__1::allocator&lt;tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function&lt;void ()&gt;)::'lambda'()&gt;, void ()&gt;::operator()() + 52
    frame #33: 0x000000012be00070 libtensorflow_framework.so`void* std::__1::__thread_proxy&lt;std::__1::tuple&lt;std::__1::function&lt;void ()&gt; &gt; &gt;(void*) + 96
    frame #34: 0x00007fffb96b893b libsystem_pthread.dylib`_pthread_body + 180
    frame #35: 0x00007fffb96b8887 libsystem_pthread.dylib`_pthread_start + 286
    frame #36: 0x00007fffb96b808d libsystem_pthread.dylib`thread_start + 13"><pre class="notranslate"><code class="notranslate">Process 19280 stopped
* thread #25, stop reason = EXC_BAD_ACCESS (code=2, address=0x700001382000)
    frame #0: 0x0000000110a24805 libopenblasp-r0.3.0.dev.dylib`dgemm_thread_tn + 1541
libopenblasp-r0.3.0.dev.dylib`dgemm_thread_tn:
-&gt;  0x110a24805 &lt;+1541&gt;: xchgq  %rdi, -0x40(%rsi)
    0x110a24809 &lt;+1545&gt;: xorl   %edi, %edi
    0x110a2480b &lt;+1547&gt;: xchgq  %rdi, (%rsi)
    0x110a2480e &lt;+1550&gt;: addq   $0x200, %rsi              ; imm = 0x200
Target 0: (python) stopped.
(lldb) bt
* thread #25, stop reason = EXC_BAD_ACCESS (code=2, address=0x700001382000)
  * frame #0: 0x0000000110a24805 libopenblasp-r0.3.0.dev.dylib`dgemm_thread_tn + 1541
    frame #1: 0x00000001108f3e26 libopenblasp-r0.3.0.dev.dylib`cblas_dgemm + 854
    frame #2: 0x0000000105564285 multiarray.cpython-36m-darwin.so`cblas_matrixproduct + 4917
    frame #3: 0x0000000105529d37 multiarray.cpython-36m-darwin.so`PyArray_MatrixProduct2 + 215
    frame #4: 0x000000010552ed1f multiarray.cpython-36m-darwin.so`array_matrixproduct + 191
    frame #5: 0x00000001000d1cbe Python`_PyCFunction_FastCallDict + 463
    frame #6: 0x00000001001362d6 Python`call_function + 489
    frame #7: 0x000000010012f18b Python`_PyEval_EvalFrameDefault + 4811
    frame #8: 0x0000000100136a38 Python`_PyEval_EvalCodeWithName + 1719
    frame #9: 0x000000010013713b Python`fast_function + 218
    frame #10: 0x00000001001362ad Python`call_function + 448
    frame #11: 0x000000010012f224 Python`_PyEval_EvalFrameDefault + 4964
    frame #12: 0x00000001001373db Python`_PyFunction_FastCall + 121
    frame #13: 0x00000001001362ad Python`call_function + 448
    frame #14: 0x000000010012f18b Python`_PyEval_EvalFrameDefault + 4811
    frame #15: 0x0000000100136a38 Python`_PyEval_EvalCodeWithName + 1719
    frame #16: 0x000000010013730b Python`_PyFunction_FastCallDict + 449
    frame #17: 0x0000000100099f21 Python`_PyObject_FastCallDict + 196
    frame #18: 0x0000000100182073 Python`partial_call + 258
    frame #19: 0x0000000100099da2 Python`PyObject_Call + 101
    frame #20: 0x000000010012f3f4 Python`_PyEval_EvalFrameDefault + 5428
    frame #21: 0x0000000100136a38 Python`_PyEval_EvalCodeWithName + 1719
    frame #22: 0x000000010013730b Python`_PyFunction_FastCallDict + 449
    frame #23: 0x0000000100099f21 Python`_PyObject_FastCallDict + 196
    frame #24: 0x000000010009a044 Python`_PyObject_Call_Prepend + 156
    frame #25: 0x0000000100099da2 Python`PyObject_Call + 101
    frame #26: 0x00000001000e4460 Python`slot_tp_call + 50
    frame #27: 0x0000000100099da2 Python`PyObject_Call + 101
    frame #28: 0x00000001212b078e _pywrap_tensorflow_internal.so`tensorflow::PyFuncOp::Compute(tensorflow::OpKernelContext*) + 974
    frame #29: 0x000000012bd82422 libtensorflow_framework.so`tensorflow::(anonymous namespace)::ExecutorState::Process(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long) + 6690
    frame #30: 0x000000012bd895ba libtensorflow_framework.so`std::__1::__function::__func&lt;std::__1::__bind&lt;void (tensorflow::(anonymous namespace)::ExecutorState::*)(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long), tensorflow::(anonymous namespace)::ExecutorState*, tensorflow::(anonymous namespace)::ExecutorState::TaggedNode const&amp;, long long&amp;&gt;, std::__1::allocator&lt;std::__1::__bind&lt;void (tensorflow::(anonymous namespace)::ExecutorState::*)(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long), tensorflow::(anonymous namespace)::ExecutorState*, tensorflow::(anonymous namespace)::ExecutorState::TaggedNode const&amp;, long long&amp;&gt; &gt;, void ()&gt;::operator()() + 58
    frame #31: 0x000000012bdde824 libtensorflow_framework.so`Eigen::NonBlockingThreadPoolTempl&lt;tensorflow::thread::EigenEnvironment&gt;::WorkerLoop(int) + 1876
    frame #32: 0x000000012bdddfd4 libtensorflow_framework.so`std::__1::__function::__func&lt;tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function&lt;void ()&gt;)::'lambda'(), std::__1::allocator&lt;tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function&lt;void ()&gt;)::'lambda'()&gt;, void ()&gt;::operator()() + 52
    frame #33: 0x000000012be00070 libtensorflow_framework.so`void* std::__1::__thread_proxy&lt;std::__1::tuple&lt;std::__1::function&lt;void ()&gt; &gt; &gt;(void*) + 96
    frame #34: 0x00007fffb96b893b libsystem_pthread.dylib`_pthread_body + 180
    frame #35: 0x00007fffb96b8887 libsystem_pthread.dylib`_pthread_start + 286
    frame #36: 0x00007fffb96b808d libsystem_pthread.dylib`thread_start + 13
</code></pre></div>
<p dir="auto"><strong>Describe the expected behavior</strong><br>
The script should run</p>
<p dir="auto"><strong>Code to reproduce the issue</strong><br>
python onsets_frames_transcription_transcribe.py --acoustic_run_dir &lt;checkpoint_dir&gt; &lt;wav_file&gt;</p>